Ok lets say someone is trying to brute force login their way to your box.

How would you setup a log system that would log it down?

Have a look at the file /etc/syslog.conf and its man page with man syslog.conf. At default settings, my box here (Slackware) logs failed logins at the terminal to /var/log/secure:
It also logs failed ssh attempts to /var/log/messages:
Were you looking at ways of sending your log files to a remote server in case the attacker got in?

Well I was thinking about something like this.

It may sound a little complicated

ok so log information about failed logs into a log file. Then have a cron job to monitor that file (every 10 min or so) and then if that file exceeds a certain length (+5 lines) then it will shut off the sshd service for an hour or so. Then have a cron job (that runs every 6 hours) that checks to see if sshd is off and then turns it on.

That way it prevents a brute force attack by literally disabling ssh for a few hours.

Is this possbile?

It looks possible... But if you only allow logins that use public keys (no passwords) you don't have to worry about this. Well, not until the public key stuff is cracked and spotted in the wild

I have never actually understood the whole system behind public keys. How do you setup a public key automated login system between your computer and the server.

Also how would you be able to setup an automated login authentication system between two servers, so that they could exchange files for redundancy?

Once you have your keys generated with ssh-keygen (there's a howto at http://sial.org/howto/openssh/publickey-auth/ as well as man ssh-keygen), you can set up an ssh agent. The idea is that you provide your key's passphrase to the agent at the start of your session and it provides the passphrase for all ssh connections to the server. There's similar software for Windows clients - have a look at putty. For configuring the server, there needs to be a line in /etc/ssh/sshd_config that says PasswordAuthentication no

You could use ssh forwarding for a connection between two servers so that they pass data over the encrypted channel. It depends on what you need it to as to how much work is involved. There are plenty of examples via google for various setups. For example, I use ssh to encrypt my vnc session over the LAN at work.