I have been notified that someone has been using my server to attempt to gain access other machines on the internet. The administrators of those targeted machines sent log data showing times and IP addresses. When I look at my own authlog and syslog entries I can can find no evidence of anything unusual happening at the times it occured. I am looking for some suggestions on how to find and identify who is doing this on my machine, whether it be one of my own users or a hacker.

My OS is Debian woody with oidentd running.

Could you post some of the logs they sent you (remove their IPs as well as yours before posting)? Do you see any abnormal processes running? Any odd logins when you run last -i ? Anything abnormal in root's and other users .bash_histories? Anything abnormal in netstat -pantu or lsof -i.

If you find any evidence indicating that their logs are correct, you should immediately take the machine offline.

Here is one log entry that was sent. I've changed the IP
address to xx.xxx.xx.xx and the domain name to mydomain.net.
There are other log entires that were sent too. I will
post those in a subsequent post.

email & log:


We have detected hacking attempts originating from the
following IP Address: xx.xxx.xx.xx
Please notify this person of you acceptable use policy. I
have also notified your upstream provider.


Processing Initiated: Thu Jan 6 04:02:00 2005
Date Range Processed: yesterday
Detail Level of Output: 0
Logfiles for Host: mail
###########################################################
admin/password from xx.xxx.xx.xx: 2 Time(s)
guest/password from xx.xxx.xx.xx: 1 Time(s)
root/password from xx.xxx.xx.xx: 27 Time(s)
user/password from xx.xxx.xx.xx: 1 Time(s)

Jan 5 00:00:31 mail sshd(pam_unix)[25658]: check pass;
user unknown
Jan 5 00:00:31 mail sshd(pam_unix)[25658]: authentication
failure; logname= uid=0 euid=0 tty=NODEVssh ruser=
rhost=mydomain.net
Jan 5 00:00:34 mail sshd(pam_unix)[25660]: check pass;
user unknown
Jan 5 00:00:34 mail sshd(pam_unix)[25660]: authentication
failure; logname= uid=0 euid=0 tty=NODEVssh ruser=
rhost=mydomain.net
Jan 5 00:00:37 mail sshd(pam_unix)[25662]: check pass;
user unknown
Jan 5 00:00:37 mail sshd(pam_unix)[25662]: authentication
failure; logname= uid=0 euid=0 tty=NODEVssh ruser=
rhost=mydomain.net
Jan 5 00:00:40 mail sshd(pam_unix)[25664]: check pass;
user unknown
Jan 5 00:00:40 mail sshd(pam_unix)[25664]: authentication
failure; logname= uid=0 euid=0 tty=NODEVssh ruser=
rhost=mydomain.net
Jan 5 00:00:43 mail sshd(pam_unix)[25666]: check pass;
user unknown
Jan 5 00:00:43 mail sshd(pam_unix)[25666]: authentication
failure; logname= uid=0 euid=0 tty=NODEVssh ruser=
rhost=mydomain.net
Jan 5 00:00:47 mail sshd(pam_unix)[25668]: authentication
failure; logname= uid=0 euid=0 tty=NODEVssh ruser=
rhost=mydomain.net user=root
Jan 5 00:00:50 mail sshd(pam_unix)[25670]: authentication
failure; logname= uid=0 euid=0 tty=NODEVssh ruser=
rhost=mydomain.net user=root
Jan 5 00:00:53 mail sshd(pam_unix)[25672]: authentication
failure; logname= uid=0 euid=0 tty=NODEVssh ruser=
rhost=mydomain.net user=root
Jan 5 00:00:56 mail sshd(pam_unix)[25674]: check pass;
user unknown
Jan 5 00:00:56 mail sshd(pam_unix)[25674]: authentication
failure; logname= uid=0 euid=0 tty=NODEVssh ruser=
rhost=mydomain.net
Jan 5 00:00:59 mail sshd(pam_unix)[25676]: check pass;
user unknown
Jan 5 00:00:59 mail sshd(pam_unix)[25676]: authentication
failure; logname= uid=0 euid=0 tty=NODEVssh ruser=
rhost=mydomain.net
Jan 5 00:01:03 mail sshd(pam_unix)[25680]: check pass;
user unknown
Jan 5 00:01:03 mail sshd(pam_unix)[25680]: authentication
failure; logname= uid=0 euid=0 tty=NODEVssh ruser=
rhost=mydomain.net
Jan 5 00:01:06 mail sshd(pam_unix)[25682]: check pass;
user unknown
Jan 5 00:01:06 mail sshd(pam_unix)[25682]: authentication
failure; logname= uid=0 euid=0 tty=NODEVssh ruser=
rhost=mydomain.net
Jan 5 00:01:09 mail sshd(pam_unix)[25684]: authentication
failure; logname= uid=0 euid=0 tty=NODEVssh ruser=
rhost=mydomain.net user=root
Jan 5 00:01:13 mail sshd(pam_unix)[25686]: authentication
failure; logname= uid=0 euid=0 tty=NODEVssh ruser=
rhost=mydomain.net user=root
Jan 5 00:01:17 mail sshd(pam_unix)[25688]: authentication
failure; logname= uid=0 euid=0 tty=NODEVssh ruser=
rhost=mydomain.net user=root
Jan 5 00:01:21 mail sshd(pam_unix)[25690]: authentication
failure; logname= uid=0 euid=0 tty=NODEVssh ruser=
rhost=mydomain.net user=root
Jan 5 00:01:24 mail sshd(pam_unix)[25692]: authentication
failure; logname= uid=0 euid=0 tty=NODEVssh ruser=
rhost=mydomain.net user=root
Jan 5 00:01:28 mail sshd(pam_unix)[25694]: authentication
failure; logname= uid=0 euid=0 tty=NODEVssh ruser=
rhost=mydomain.net user=root
Jan 5 00:01:31 mail sshd(pam_unix)[25697]: authentication
failure; logname= uid=0 euid=0 tty=NODEVssh ruser=
rhost=mydomain.net user=root
Jan 5 00:01:34 mail sshd(pam_unix)[25699]: authentication
failure; logname= uid=0 euid=0 tty=NODEVssh ruser=
rhost=mydomain.net user=root
Jan 5 00:01:38 mail sshd(pam_unix)[25701]: authentication
failure; logname= uid=0 euid=0 tty=NODEVssh ruser=
rhost=mydomain.net user=root
Jan 5 00:01:41 mail sshd(pam_unix)[25703]: authentication
failure; logname= uid=0 euid=0 tty=NODEVssh ruser=
rhost=mydomain.net user=root
Jan 5 00:01:47 mail sshd(pam_unix)[25705]: authentication
failure; logname= uid=0 euid=0 tty=NODEVssh ruser=
rhost=mydomain.net user=root
Jan 5 00:01:50 mail sshd(pam_unix)[25707]: authentication
failure; logname= uid=0 euid=0 tty=NODEVssh ruser=
rhost=mydomain.net user=root
Jan 5 00:01:56 mail sshd(pam_unix)[25709]: authentication
failure; logname= uid=0 euid=0 tty=NODEVssh ruser=
rhost=mydomain.net user=root
Jan 5 00:01:59 mail sshd(pam_unix)[25711]: authentication
failure; logname= uid=0 euid=0 tty=NODEVssh ruser=
rhost=mydomain.net user=root
Jan 5 00:02:08 mail sshd(pam_unix)[25713]: authentication
failure; logname= uid=0 euid=0 tty=NODEVssh ruser=
rhost=mydomain.net user=root
Jan 5 00:02:15 mail sshd(pam_unix)[25715]: authentication
failure; logname= uid=0 euid=0 tty=NODEVssh ruser=
rhost=mydomain.net user=root
Jan 5 00:02:18 mail sshd(pam_unix)[25717]: authentication
failure; logname= uid=0 euid=0 tty=NODEVssh ruser=
rhost=mydomain.net user=root
Jan 5 00:02:21 mail sshd(pam_unix)[25719]: authentication
failure; logname= uid=0 euid=0 tty=NODEVssh ruser=
rhost=mydomain.net user=root
Jan 5 00:02:24 mail sshd(pam_unix)[25721]: authentication
failure; logname= uid=0 euid=0 tty=NODEVssh ruser=
rhost=mydomain.net user=root
Jan 5 00:02:27 mail sshd(pam_unix)[25723]: authentication
failure; logname= uid=0 euid=0 tty=NODEVssh ruser=
rhost=mydomain.net user=root
Jan 5 00:02:30 mail sshd(pam_unix)[25726]: authentication
failure; logname= uid=0 euid=0 tty=NODEVssh ruser=
rhost=mydomain.net user=root
Jan 5 00:02:33 mail sshd(pam_unix)[25728]: authentication
failure; logname= uid=0 euid=0 tty=NODEVssh ruser=
rhost=mydomain.net user=root
Jan 5 00:02:37 mail sshd(pam_unix)[25730]: authentication
failure; logname= uid=0 euid=0 tty=NODEVssh ruser=
rhost=mydomain.net user=root
Jan 5 00:02:50 mail sshd(pam_unix)[25732]: authentication
failure; logname= uid=0 euid=0 tty=NODEVssh ruser=
rhost=mydomain.net user=root

Please use the correct terminology.

Hackers write source code, often for the open source community.

Crackers break into computers unlawfully.

The major media apparently is ignorant of this fact.

Jan 5 02:11:16 mail sshd[97067]: Failed password for admin from
xx.xxx.xx.xx port 34496 ssh2
Jan 5 02:11:16 mail sshd[97069]: Failed password for admin from
xx.xxx.xx.xx port 34513 ssh2
Jan 5 02:11:17 mail sshd[97071]: Failed password for admin from
xx.xxx.xx.xx port 34533 ssh2
Jan 5 02:11:18 mail sshd[97073]: Failed password for admin from
xx.xxx.xx.xx port 34557 ssh2
Jan 5 02:11:24 mail sshd[97085]: Failed password for root from xx.xxx.xx.xx
port 34747 ssh2
Jan 5 02:11:25 mail sshd[97090]: Failed password for root from xx.xxx.xx.xx
port 34689 ssh2
Jan 5 02:11:26 mail sshd[97096]: Failed password for root from xx.xxx.xx.xx
port 34824 ssh2

And it went on like that, for 7 minutes, over 500 attempts/ It appears that they are running some sort of password cracking program. So I guess that's what I've got to look for.

This might be handy to run.

chkrootkit is a tool to locally check for signs of a rootkit

Looks like a brutessh2 scan. This is not something you can spoof easily, so it was almost certainly run from your system (assuming the xx.xx.xx.xx addess in the logs was your IP). So you should definitely disconect the system from the network to prevent any further attacks from originating from your system. Do not put it back online until you are certain it's clean.

Running chkrootkit is a good start. Check the system thoroughly for abnormal files or directories (esp hidden ones). Also, the brutessh tool is often loaded onto systems that were successfully compromised using the tool, so definitely check last -i for strange logins (do you have secure passwords on the system?). Also please post the output of netstat -pantu and lsof -i.

First, I have been running chkrootkit regularly, with its output mailed to me daily. Have been for awhile. Nothing there to raise any concerns.

The last -i suggestion was a good one. I went over all those logins and inspected them closely and compared IP addresses and I was able to find some logins that didn't fit. In fact he was actively logged in at the time. I got rid of him, changed passwords, and removed his voluminous files (which were all .hidden) to a secure place for closer inspection. I just hope he didn't leave any hidden backdoors. If he did I am prepared to take the box offline and reinstall if necessary.

Any comments about Tripwire? I'm seriously considering installing it. Any tips on a debian apt-get install of Tripwire? Anyone know of any entries to make in the apt sources.list file for it?

I know the difference between hackers and crackers. Not all "hackers" are quite as benign as suggested, and there is a subspecies of hacker that *does* attempt to effect unauthorized entry into systems, just for the sport of it, and not for any other malicious purpose.

Thanks for the help/suggestions.

First, I have been running chkrootkit regularly, with its output mailed to me daily. Have been for awhile. Nothing there to raise any concerns.
Chkrootkit is designed primarily to identify common rootkits, backdoors, and sniffers. Other software and scripts will easily avoid detection, so a clean bill of health from chkrootkit is by no means a guarantee (as you can see).

The last -i suggestion was a good one. I went over all those logins and inspected them closely and compared IP addresses and I was able to find some logins that didn't fit. In fact he was actively logged in at the time. I got rid of him, changed passwords, and removed his voluminous files (which were all .hidden) to a secure place for closer inspection.
Might want to look through these for a file of vulnerable targets that were identified (and possibly compromised) from your machine. Likely be something like vuln.txt, but you can basically look for anything that is a list of IPs + succesfull usernames/password combos. Might also find some other tidbits like IRC conversations, interesting cracking tools, etc.


I just hope he didn't leave any hidden backdoors. If he did I am prepared to take the box offline and reinstall if necessary.
That is the problem indeed. One of the basic rules in security is that if your system has been compromised, then a complete format and reinstall is absolutely necessasry. It's possible to hide multiple backdoors and kernel modules on a system that are virtually undetectable to most security tools. So changing passwords may be of zero help once someone has already gained access to your system. Unfortunately a reinstall is the only way to be certain that the system is clean.

Any comments about Tripwire? I'm seriously considering installing it.
Great tool if you put it on a freshly installed system before it gets connected to a network. Not as effective if it's already networked. Lousy if the system has already been compromised (like locking the front door once all your stuff's already been stolen). Also not a 100% effective defense, so don't completely rely on it.

Aside from that, you should take the address that you identified in the last -i output and send him/her and their ISP a polite letter letting them know that their may have been compromised. More often than not, the owner is likely unaware and is being used as a proxy (unfortunately you can probably relate). Also contact the list of those identifed as vulnerable targets and let them know that they may have been compromised as well.

Once you get you new machine up, install a file integrity checker like tripwire, AIDE, Samhain. Do some general hardening (turn off services). Turn on a firewall. Use good passwords and encrypt sensitive traffic. And keep up with patching.