Possible system compromise (slackware linux 10.2, apache 1.3.33, OpenSSL 0.9.7g)

Noido · 05-10-2006, 09:09 AM

I've recently been noticing lots of failed password attempts in my messages log, (/var/log/messages), I'm pretty sure it's a dictionary attack but I thought it hadn't worked, until now. I woke up this morning (may 10th) and my PC had rebooted during the night. The log tells me it was around 3:30 am:

Code:

May  9 22:55:55 chidori sshd[11286]: Invalid user desktop from 62.233.144.26
May  9 22:55:55 chidori sshd[11286]: Failed password for invalid user desktop from 62.233.144.26 port 2959 ssh2
May  9 22:55:56 chidori sshd[11289]: Invalid user workshop from 62.233.144.26
May  9 22:55:56 chidori sshd[11289]: Failed password for invalid user workshop from 62.233.144.26 port 2999 ssh2
May  9 22:55:56 chidori sshd[11292]: Invalid user mailnull from 62.233.144.26
May  9 22:55:56 chidori sshd[11292]: Failed password for invalid user mailnull from 62.233.144.26 port 3034 ssh2
May  9 22:55:57 chidori sshd[11295]: Invalid user nfsnobody from 62.233.144.26
May  9 22:55:57 chidori sshd[11295]: Failed password for invalid user nfsnobody from 62.233.144.26 port 3377 ssh2
May  9 22:55:58 chidori sshd[11298]: Invalid user rpcuser from 62.233.144.26
May  9 22:55:58 chidori sshd[11298]: Failed password for invalid user rpcuser from 62.233.144.26 port 3408 ssh2
May  9 22:55:58 chidori sshd[11301]: Failed password for rpc from 62.233.144.26 port 3472 ssh2
May  9 22:55:59 chidori sshd[11304]: Invalid user gopher from 62.233.144.26
May  9 22:55:59 chidori sshd[11304]: Failed password for invalid user gopher from 62.233.144.26 port 3511 ssh2
May  9 23:13:22 chidori -- MARK --
May  9 23:33:22 chidori -- MARK --
May  9 23:53:22 chidori -- MARK --
May 10 00:13:22 chidori -- MARK --
May 10 00:33:22 chidori -- MARK --
May 10 00:53:22 chidori -- MARK --
May 10 01:13:22 chidori -- MARK --
May 10 01:33:22 chidori -- MARK --
May 10 01:53:22 chidori -- MARK --
May 10 02:13:22 chidori -- MARK --
May 10 02:33:22 chidori -- MARK --
May 10 02:53:22 chidori -- MARK --
May 10 03:13:22 chidori -- MARK --
May 10 03:29:59 chidori syslogd 1.4.1: restart.

I've got more dictionary attacks today around 12:50, from different IP addresses, so I assume they're all zombie hosts.

I ran rkhunter and got the following warnings etc.:

scanning for hidden files... [ Warning! ]
-----------
/etc/.pwd.lock
/etc/.java
-----------

and it also tells me that Apache 1.3.33 and OpenSSL 0.9.7g are Vulerable and root access is possible.

My question is: how can I be sure that i'm not compromised, and which logs do I check to see why it shut down?

Thanks very much,

Noido

ssfrstlstnm · 05-10-2006, 09:37 AM

I can't help you to find out if you have been compromised, but you may want to take a look at denyhosts. It might save you the trouble of this problem next time.

Samoth · 05-10-2006, 09:38 AM

First off, in the future you must make sure that all passwords are completely random. Second, if you have to reinstall because you aren't sure, try installing tripwire before you hook your comp back up to the net. I am not very familiar with system compromises but I thought that you could see if they succeeded in logging in. If they did...I would suggest reinstalling.

Noido · 05-10-2006, 09:45 AM

My passwords _are_ random, and i'm pretty sure that the dictionary attack didn't do it, because it's still going. I'd still like to know where I can find logs which will tell me why it shut down...

Thanks

Noido

Samoth · 05-10-2006, 10:00 AM

you sure there isn't anything in /var/log/messages? that is the place they would be. Also, there are a number of logs in /var/log so try them.

Noido · 05-10-2006, 10:27 AM

The only other log entry that I can find around that time that I'm unsure of is the following:

Code:

64.62.190.36 - - [10/May/2006:14:16:22 +0000] "POST http://64.62.190.36:6667/ HTTP/1.0" 200 1025
64.62.190.36 - - [10/May/2006:14:16:22 +0000] "CONNECT 64.62.190.36:6667 HTTP/1.0" 200 719

from the apache access_log

Any ideas what this might be?

Oh, and I've looked through /var/log/messages, it's all dictionary attacks (well, most of it) except for the part where it boots, I just posted a snippet earlier

Other than this, i'm at a loss

Hangdog42 · 05-10-2006, 01:36 PM

I'll preface this with stating that I'm a security amateur, but as far as your ssh logs go, there is nothing there that indicates a successful attack. All of the login attempts failed. Anyone running an ssh server on port 22 is going to get a ton of those (see the sticky thread at the top of this forum for info on what you can/should do about these).

The restart entry isn't a system reboot, it is the syslog deamon restart. Given that it happened at 3:30AM, I would strongly suspect that it is logrotate running its daily cron job. Have a look at root's crontab file and see what time logrotate is running.

Now the http access strikes me as kind of funny because it is occurring on port 6667 and it seems to be entirely internal (the requesting IP and target IP are the same). Have you installed any software that would listen on that port? Something is definitely listenting there because the request was to that port and the 200 is a successful response.

Other suggestions:

Have a look at the output of netstat -pantu and see what is listening. You should be able to account for all of the ports listed. If not, post the output.

Look at the output of last and see if there are logins you can't account for.

The downside is that if you have been compromised, it is possible that these commands (and the system logs) have been altered to hide the compromise. The chkrootkit response may not indicate a compromise since I think it is sensitive to hidden directories and files in certain system locations. Try running it again and run rkhunter for a second opinion.

<EDIT>
Ooops, I see you did run rkhunter. Try running chkrootkit and see if it turns up anything.
</EDIT>

Do you have a firewall on this machine? If so, posting it may be helpful.

Finally, it won't help now, but in the future, it would be a good idea to run a file integrity checking system like Aide, Samhain or Tripwire. However, you need to run those on systems you are positive are clean.

Noido · 05-11-2006, 10:13 AM

Ok: I've done pretty much all you said, and I've come up with:

1) cron jobs running at 4:30, also it *was* a reboot, I just didn't post all of the log (sorry) it's here:

Code:

May 10 03:29:59 chidori syslogd 1.4.1: restart.
May 10 03:30:01 chidori kernel: klogd 1.4.1, log source = /proc/kmsg started.
May 10 03:30:01 chidori kernel: BIOS-provided physical RAM map:
May 10 03:30:01 chidori kernel: 253MB LOWMEM available.
May 10 03:30:01 chidori kernel: Initializing CPU#0
May 10 03:30:01 chidori kernel: Memory: 254020k/260060k available (1832k kernel code, 5656k reserved, 607k data, 120k init, 0k highmem)
May 10 03:30:01 chidori kernel: Dentry cache hash table entries: 32768 (order: 6, 262144 bytes)
May 10 03:30:01 chidori kernel: Inode cache hash table entries: 16384 (order: 5, 131072 bytes)
May 10 03:30:01 chidori kernel: Mount cache hash table entries: 512 (order: 0, 4096 bytes)
May 10 03:30:01 chidori kernel: Buffer cache hash table entries: 16384 (order: 4, 65536 bytes)
May 10 03:30:01 chidori kernel: CPU: L1 I cache: 16K, L1 D cache: 16K
May 10 03:30:01 chidori kernel: CPU: L2 cache: 256K
May 10 03:30:01 chidori kernel: Enabling fast FPU save and restore... done.
May 10 03:30:01 chidori kernel: Enabling unmasked SIMD FPU exception support... done.
May 10 03:30:01 chidori kernel: Checking 'hlt' instruction... OK.
May 10 03:30:01 chidori kernel: PCI: PCI BIOS revision 2.10 entry at 0xfbfee, last bus=1
May 10 03:30:01 chidori kernel: PCI: Using configuration type 1
May 10 03:30:01 chidori kernel: PCI: Probing PCI hardware
May 10 03:30:01 chidori kernel: PCI: Using IRQ router PIIX/ICH [8086/2440] at 00:1f.0
May 10 03:30:01 chidori kernel: Linux NET4.0 for Linux 2.4
May 10 03:30:01 chidori kernel: Based upon Swansea University Computer Society NET3.039
May 10 03:30:01 chidori kernel: VFS: Disk quotas vdquot_6.5.1
May 10 03:30:01 chidori kernel: Journalled Block Device driver loaded
May 10 03:30:01 chidori kernel: Serial driver version 5.05c (2001-07-08) with HUB-6 MANY_PORTS MULTIPORT SHARE_IRQ SERIAL_PCI enabled
May 10 03:30:01 chidori kernel: ttyS00 at 0x03f8 (irq = 4) is a 16550A
May 10 03:30:01 chidori kernel: ttyS01 at 0x02f8 (irq = 3) is a 16550A
May 10 03:30:01 chidori kernel: Real Time Clock Driver v1.10f
May 10 03:30:01 chidori kernel: Floppy drive(s): fd0 is 1.44M
May 10 03:30:01 chidori kernel: FDC 0 is a National Semiconductor PC87306
May 10 03:30:01 chidori kernel: loop: loaded (max 8 devices)
May 10 03:30:01 chidori kernel: Uniform Multi-Platform E-IDE driver Revision: 7.00beta4-2.4
May 10 03:30:01 chidori kernel: ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
May 10 03:30:01 chidori kernel: ICH2: IDE controller at PCI slot 00:1f.1
May 10 03:30:01 chidori kernel: ICH2: chipset revision 2
May 10 03:30:01 chidori kernel: ICH2: not 100%% native mode: will probe irqs later
May 10 03:30:01 chidori kernel:     ide0: BM-DMA at 0xffa0-0xffa7, BIOS settings: hda:DMA, hdb:pio
May 10 03:30:01 chidori kernel:     ide1: BM-DMA at 0xffa8-0xffaf, BIOS settings: hdc:DMA, hdd:pio
May 10 03:30:01 chidori kernel: hda: 78165360 sectors (40021 MB) w/2048KiB Cache, CHS=4865/255/63, UDMA(100)
May 10 03:30:01 chidori kernel: hdc: ATAPI 32X CD-ROM CD-R/RW drive, 4096kB Cache, UDMA(33)
May 10 03:30:01 chidori kernel: Uniform CD-ROM driver Revision: 3.12
May 10 03:30:01 chidori kernel: Partition check:
May 10 03:30:01 chidori kernel:  hda: hda1 hda2 < hda5 hda6 hda7 hda8 >
May 10 03:30:01 chidori kernel: SCSI subsystem driver Revision: 1.00
May 10 03:30:01 chidori kernel: md: linear personality registered as nr 1
May 10 03:30:01 chidori kernel: md: raid0 personality registered as nr 2
May 10 03:30:01 chidori kernel: md: raid1 personality registered as nr 3
May 10 03:30:01 chidori kernel: md: raid5 personality registered as nr 4
May 10 03:30:01 chidori kernel: raid5: measuring checksumming speed
May 10 03:30:01 chidori kernel: md: md driver 0.90.0 MAX_MD_DEVS=256, MD_SB_DISKS=27

(EDIT: I'd just like to note that that's not *all* of it)

It definitely rebooted, because I woke up in the morning and all my processes had been killed (IRC, and X, etc.)

my netstat -pantu gives me:

Code:

tcp        0      0 0.0.0.0:37              0.0.0.0:*               LISTEN      2493/inetd
tcp        0      0 0.0.0.0:587             0.0.0.0:*               LISTEN      2513/sendmail: acce
tcp        0      0 0.0.0.0:6000            0.0.0.0:*               LISTEN      30095/X
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      2567/httpd
tcp        0      0 0.0.0.0:113             0.0.0.0:*               LISTEN      2493/inetd
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      19384/sshd
tcp        0      0 0.0.0.0:25              0.0.0.0:*               LISTEN      2513/sendmail: acce

udp        0      0 0.0.0.0:512             0.0.0.0:*                           2493/inetd
udp        0      0 0.0.0.0:37              0.0.0.0:*                           2493/inetd
udp        0      0 0.0.0.0:68              0.0.0.0:*                           244/dhcpcd

I'm not sure what the apache CONNECT and POST things were about, but they aren't internal IPs (I tracerouted them)

chkrootkit didn't find anything at all, which suggests it's fine, but the rebooting still puzzles me, and I don't see what could have caused it to restart.

Hangdog42 · 05-11-2006, 12:02 PM

I guess that aside from the odd re-boot and the http connection, I'm not seeing anything that raises any red flags. You obviously have some services listening, but there don't appear to be any established connections and there isn't anything unusual about the listening programs (by the way, you did run that as root, didn't you?). Chkrootkit coming back clean is a good sign.

At this point the choice is really yours and depends on your personal level of paranoia. If you want to be absolutely sure your machine is clean, a complete re-install is in order. There just isn't any other way to be sure. As I said before, one of the problems with actually being cracked is that the crackers can be very good at hiding their tracks. If you do go this route, I would strongly suggest getting Aide, Samhain or Tripwire onto that box immediately as one of those programs could tell you if files have been changed.

However, if you don't want to re-install, I would keep a very close eye on the box and spend some serious time reading the thread unSpawn has stickied at the top of this forum. There is lots of good stuff there on how to monitor a box as well has what you can do to harden it.

Noido · 05-11-2006, 03:07 PM

Thanks for all the advice, i'll install tripwire &c. immediately. I would reinstall, but I'll need to wait a while until I can back up some data.

Thanks very much for all your help

~Noido