As far as I understand it, the passphrase is used as one of the parameters for generating the private key. There is also a random part (I thing it's called salt) that contributes to both private and public key. If you rerun ssh-keygen, you'll end up with a different public and private keys. Changing the passphrase will only affect the private key.
The point of cryptographic hashes that if you know the password, it is easy to compute the hash, but if you know the hash, it is impossible (or, at least very difficult) to find the password. So, even if someone knows the hash, they're still a long way from breaking in. Still, setting the read permissions on your private key file so that nobody else can read it doesn't hurt.