You should use a pass phrase to protect your private key. If you use ssh-agent, you only have to enter the pass-phrase once per shell session.
This will unlock your private key and you can scp or login without a pass phrase.
I don't know how you have the ssh server configured, but if you have both public key authentication and password authentication, I think it will ask for a password before trying public key authentication. You would allow an attacker to use a brute force attack. Stick with public key authentication.
I would also suggest using "AllowUsers", disable root logins and only allow Protocol 2.