If you use a public key encryption, the 'other' party has your public key and you have your private key. In your situation, the other party is your ssh server, so that one will have the public key.
With regards to the last question:
Yes, they can. But a good passphrase will prevent that. My passphrase is about 20 characters. Assuming that there are roughly 72 characters (uppercase, lowercase, numbers and the stuff above the numbers), you have 1.4 e37 possible combinations (72^20). Assume as well that a brute force attack can do 10 e6 attacks per second (feasable with current PCs), it will take 44,447,088,893,843,884,113,135 years (can somebody please pronounce that) to hack.
Even if you only use lower case (26^20 possible combinations), it takes
63,191,745,608,858 years.
So I think that I'm reasonable safe
Of course the first hit can be bingo, but it might also be the last one.