Doesn't that just trade one symmetric secret for another? (your password)
I suppose it's a little better because you probably haven't written your password on the side of your card, but everytime I have to go through it it feels like I'm getting phished. Also, SSL seems kind of messy for the job. You've already got the processor as a third party, now the CA's are a fourth party, plus whoever gets to install certs on that device as a fifth...
You could just have the card sign the transaction and have the merchant send that signature to the clearinghouse. For online orders, your phone could just be the payment terminal, and still the secret on the card is the signing key.
But none of that actually helps unless you deprecate the insecure stuff.