Today one of my linux servers, which functions primarily as mail, and mail related web sites (squirrelmail, a few custom apps, etc), went down. After rebooting I went in to find this.

Looks like I have been getting lots of the failed ssh requests on "test, admin, user" etc. that everyone is talking about. Then I noticed a login this morning that shouldn't have happened.

Long story short, it was a mistake on my part, a username "steve" that had the same password, that got setup long ago, and I forgot about.

I am guessing that they got in through the a bruteforce?

Can't find to many signs, except my root account was renamed "rot". Obviously this is a compromised box, and I will be reinstalling, as well as changed the passwords to every other box I have.

Any ideas if it was anything more malicious than a brute force? And how can I try and check how the person got root priveldges. The .bash_history for the steve account was erased.

Immediatly after the ssh session, i got kernal modprobe net-pf-14 errors.

Help, I am starting to feel a little confused.

If your /etc/passwd entry has root as rot, or in order to load modules, or really mess with kernelspace to cause it to send errors, you would need root. Best alternative is to copy off your few config files, check them thoroughly, and reinstall from known good media.

I am guessing that they got in through the a bruteforce?
If 'steve' was supposed to be logging in, then I'd say it's highly likely

The .bash_history for the steve account was erased.
If it was truly erased and not just 'unset', you might have some luck with one of the various undelete programs available for linux. Several of the forensic linux distros have more advanced recovery software as well, though they are not very user friendly. I'd check roots/rots bash_history as well (if it exists).

Immediatly after the ssh session, i got kernal modprobe net-pf-14 errors.
Commonly see that with ptrace exploits. Are you runnning an updated kernel?

Obviously this is a compromised box, and I will be reinstalling, as well as changed the passwords to every other box I have.
Sounds like it. Take redmoustache's advice and format the drive (low-level format with '0's), reinstall from trusted media and do yourself a favor and don't use weak passwords (though I imagine that doesn't need to be said).

So I am quite sure they broke in through my "steve" account (boy I feel dumb, every password I have is complex, and this one just slipped through the crack.)

It looks like they did in fact do the ptrace exploit. So I am going to the data center to get the box and rebuild. This was a mail server, so nothing overly "secure" was compromised. However the whole server crashed shortly after this was run, so did the hacker actually do much, or did he make the system unstable, and just kill it?

Also, in my netstat, I kept seeing the hackers IP on a syn_sent, and now its gone. Is there an easy way to block the IP from making any TCP/IP connection? And was he trying to get back in? I changed all the password immediatly, have I killed his hole, and that's why he wasn't getting a connection?

However the whole server crashed shortly after this was run, so did the hacker actually do much, or did he make the system unstable, and just kill it?
Well he clearly had root level access and enough time to change roots username as well as to scrub the logs, so I 'd wager that they had enough access to modify a lot of things. Unfortunately once someone has root, it can be extremely difficult to fully determine what's been modifed on the system as well as the actions they've taken on the system. If you had installed a tool like tripwire, it might be able a little more info. If you're using an rpm based system, you can try using rpm -Va . Though rootkits or trojan binaries can potentially defeat either of those. So a rebuild is going to be the smartest (and only) option.

Also, in my netstat, I kept seeing the hackers IP on a syn_sent, and now its gone. Is there an easy way to block the IP from making any TCP/IP connection?
If it was in a syn_sent state, then it was likely a connection originating from the compromised box (could be the cracker directly using it or some kind of bot app making the system 'dial home'. You can try using iptables to block the IP in both INPUT and OUTPUT chains, but again a rootkit or trojaned iptables command will make those in-effective as well.