As said above, ssh can use public key encryption.
Whenever you generate ssh keys (or any public key for that matter) there is a public key and a private key, or asymmetric key.
This works in a special way:
The public key can encrypt any information so that only the private key may decrypt it. Note the public key cannot decode information encoded by the public key (very important)
The private key can sign any information and anyone with the public key may decode it.
This permits two important things.
First:
People using the public key know that only one person can access it.
People decoding the signed information know that only one person could of created it.
Why is this relevant to ssh? ssh works a bit like this:
The server has a copy of the clients public key (authorized keys), the client has a copy of the servers public key (known hosts)
The client sends a request to the server, encrypting it using the servers public key. No one but the server may access it.
The server obtains it, decrypts the message with its private key, and encrypts a symmetric key (same password to encode/decode)
It sends this to the client who then decrypts it. It attempts to connect to the server using the symmetric key.
If everything is ok, a encrypted connection is made.
Hopefully I wrote that clearly, what it means is this:
Anyone can have the public key, technically - you could
post a public key and a public key encrypted message to a pastebin and expect total privacy.
The private key however, you must protect. If it is compromised, they are fully trusted to be the correct person.
If someone copies the public key and attempts to make a connection, the server will encrypt the symmetric key to the same public key, and the (lame) cracker will be foiled.
So there is no need to lock down the public key (or as you described - hashes) as it is only useful to the owner of the private key.
Permissions must reflect this, I believe if the private key (~/.ssh/id_rsa for example) has permissions above 600, ssh will refuse to run using it, for good reason.