If you get
really paranoid, you might take a look at
portknocking. It's a way of denying ssh access from anyone, until a (usually scripted) attempt is made to connect to various closed ports in a specific sequence, then a connection is allowed from that IP for a given amount of time.
That should help you sleep a little more easily.
Enjoy!
--- Cerbere