If you look through the manpages for ssh, ssh_config and sshd_config, you may find some things that can help as well. For example, you can prefix your public key in the authorized_keys list with "from=<address>" to only accept a connection from a certain address. This will help protect the server if the private key of the client is stolen. Also password protect your private key with a good passphrase. You can unlock it ahead of time once, for convenience:
eval $(ssh-agent)
ssh-add
<passphrase>
Since this unlocks the private key on the client, you can ssh to different servers that have your public key. You won't need to enter the passphrase in again from the same terminal.
