For the past couple weeks, I've been working furiously on securing the newly reinstalled system. I had decided to use a new hard drive since the other one was beginning to make some abnormally high-pitched noises signalling it's iminent death. After installing Mandrake 10.1, I was satisfied that the problem was gone and I could move on. But, curiosity got the best of me, thanks to nnsg.
I swapped the old hard drive back in so I could do a little forensics work and find out what had happened. I had set up the machine with very loose security (ie. none to speak of, besides the "High" security level provided by msec) because I used it as a learning tool to learn Linux. Well, I installed and ran chkrootkit which revealed a possible Madalin root kit. I'm fairly new to security, so already my adrenaline was pumping. I knew I had an intruder.
I tried running netstat to see anyone that was connected. Nothing abnormal showed up. So, I checked chkrootkit's output again and found that a couple programs had been changed (ls and netstat). I promptly downloaded and installed the appropriate rpms to replace those commands. Netstat then showed an odd connection from port 6667, which is IRC. Interesting. I recorded the IP, then ran netstat again. The 6667 connection was not there anymore. I reinstalled netstat again and the connection showed up. A few seconds later, it was gone again. Either the hacker was there, ready, or a cron job was doing the work of covering his tracks. So, I ran the hacked netstat with a few more options and |more, and when I hit 'q' to end the listing from more, I got an odd error message (more on that later). (One note: during all this, the machine was connected to the Internet because the only way I can administer the system conveniently is remotely from home. I couldn't unplug the cat5 cable to keep the hacker out.)
Netstat had showed that the 6667 connection was supposedly started by kjournald. Running 'ps ax' showed nothing unusual. So, I reinstalled the package containing ps. After running ps again, I saw ./kjournald in the output. That was the backdoor, apparently.
That odd error message from netstat revealed a very unusual directory as well as many 'grep -v' statements meant to exclude certain ports from the output. The error only showed up after piping to more and stopping with 'q'. So, I went to the directory and it turned out to be where the hacker had set up shop. One directory had all the system commands that had replaced mine, including some I didn't know had been changed. The others included an IRC program and the kjournald executable.
So, I backed up all my logs (besides the ones that had been removed by the hacker) and all the files in the hacker's directory. That way, after all this, I could thoroughly scan the logs and hacker-files to squeeze out any information relating to how he got in and any other identifying characteristics (like other IPs).
With the backup complete, I killed the kjournald process and (after reinstalling netstat yet again), saw that the connection was no longer there. I then deleted the hacker's files and checked my cron jobs to make sure kjournald wouldn't start again. Nothing happened. So, I ate, had some coffee, and came back 15 minutes later to see that a new connection had been made by a different IP. Netstat showed no process name attached to the connection, but there was a PID. Shortly after discovering the IP and PID with netstat, netstat was again replaced with a hacked version (I knew that because I ran chkrootkit before and after running netstat, and after, it said netstat was infected). I checked /proc/PID/exe to see what started the process. Right now, I can't remember the name of the process, but that's not important. So, I killed the process and right away, there it was again. I did that several times. Apparently, this ticked the hacker off, because all of a sudden, I started losing all my major shell commands, including rpm which I'd been using to reinstall hacked programs. It was so weird to use a command and 1 second later see, "Command not found." Panic!! Luckily, the shutdown command was still there. I tried shutdown -h now, but after the initial message saying the system was going down, nothing else happened. I was logged out, but the system kept going, and I couldn't log in again. He locked me out of my own system.
I tried calling the place where the system resides to have someone unplug the cat5 cable for me, but no one was there. In about an hour or so, I was there myself and unplugged it. Ahh, relief.
After doing some investigating, I noticed some log entries about unusual telnet traffic. Why did I ever install telnet in the first place? I have concluded that the hacker got in via telnet, but I'm really not sure because I don't have enough information. I found that the original IP was from Amsterdam. I also found 3 other IP's. Now, my objective will be to contact the ISP's of those IP's to report the incident.
So, now that the system is reinstalled, I have taken the time to learn IPTables, PSAD, msec, Tripwire, and more so I can prevent such an attack in the future. I will bet that as soon as the system is live on the Internet again, I will receive some scans/hack attempts from those IP's. This time, I'm ready (or more ready than before, at least).
And that's my story.
