After years of Windows I finally decided to try a UNIX system, so far I have been really impressed and have wondered why I havent done it before.

But.......

Today I got an email from my ISP saying :

Your server's switchport has been de-rated to 10 Mb/s because your server
began generating an out-bound storm of packets. This type of event usually
indicates a compromise in security.


It does seem completely out of the ordinary as the max I have ever seem off this server is 1gig a day for photo swap with a balanced inbound and outbound. None of the site get visited much at the moment.

Yesterday was 10 gig inbound and 64 meg out!

I am running IPchains with a tight level of security, none of the websites have a high level of traffic and telnet is not enabled. I think I made one mistake in making one single dir 777.

The problem is I cant find any large files on the system that relate to this inbound traffic and I cant find anything looking amiss! IS there any possible way that this traffic which must be some sort of hack attempt might not have got past the firewall... explaining why I can see any odd files. Alternatively can anyone provide me with any guidance and show me where I should look for more info on what might have happend to the server!!!

Thanks

Toby

First you need to find out what is running on your box and who is connected:

ps -ef or ps -aux will tell you what is running

netstat -an will tell you who is connected and on what service port

My question to you is what is this box supposed to be doing? And is your ISP limiting bandwidth?

[root@RedHat73-ENSIM root]# ps -ef
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 11:37 ? 00:00:04 init
root 2 1 0 11:37 ? 00:00:00 [keventd]
root 3 1 0 11:37 ? 00:00:02 [ksoftirqd_CPU0]
root 4 1 0 11:37 ? 00:00:05 [kswapd]
root 5 1 0 11:37 ? 00:01:57 [kscand]
root 6 1 0 11:37 ? 00:00:00 [bdflush]
root 7 1 0 11:37 ? 00:00:00 [kupdated]
root 8 1 0 11:37 ? 00:00:00 [mdrecoveryd]
root 12 1 0 11:37 ? 00:00:08 [kjournald]
root 92 1 0 11:37 ? 00:00:00 [khubd]
root 190 1 0 11:41 ? 00:00:00 [kjournald]
ntp 856 1 0 11:41 ? 00:00:00 ntpd -U ntp -g
root 914 1 0 11:41 ? 00:00:03 syslogd -m 0 -a /home/virtual/FI
root 934 1 0 11:41 ? 00:00:00 klogd
named 996 1 0 11:41 ? 00:00:00 named -u named
named 1000 996 0 11:41 ? 00:00:00 named -u named
named 1001 1000 0 11:41 ? 00:00:00 named -u named
named 1002 1000 0 11:41 ? 00:00:00 named -u named
named 1003 1000 0 11:41 ? 00:00:00 named -u named
root 1019 1 0 11:41 ? 00:00:00 /usr/sbin/sshd
root 1039 1 0 11:41 ? 00:00:00 sshd -f /etc/ssh/sshd-rb_config
root 1074 1 0 11:41 ? 00:00:00 xinetd -stayalive -pidfile /var/
root 1121 1 0 11:42 ? 00:00:00 /bin/sh /usr/bin/safe_mysqld --d
mysql 1173 1121 0 11:42 ? 00:00:00 /usr/libexec/mysqld --defaults-f
mysql 1176 1173 0 11:42 ? 00:00:00 /usr/libexec/mysqld --defaults-f
mysql 1177 1176 0 11:42 ? 00:00:00 /usr/libexec/mysqld --defaults-f
mysql 1186 1176 0 11:42 ? 00:00:00 /usr/libexec/mysqld --defaults-f
root 1278 1 0 11:42 ? 00:00:01 /usr/sbin/httpd -f /etc/httpd/co
postgres 1350 1 0 11:42 ? 00:00:00 /usr/bin/postmaster
postgres 1370 1350 0 11:42 ? 00:00:00 postgres: stats buffer process
postgres 1371 1370 0 11:42 ? 00:00:00 postgres: stats collector proces
root 1381 1 0 11:42 ? 00:00:00 proftpd: (accepting connections)
root 1500 1 0 11:42 ? 00:00:00 /usr/sbin/ocwhttpd -DSSL -d /usr
nobody 1502 1500 0 11:42 ? 00:00:00 /usr/sbin/fcgi-pm -DSSL -d /usr
nobody 1503 1500 0 11:42 ? 00:00:00 /usr/sbin/ocwhttpd -DSSL -d /usr
nobody 1504 1500 0 11:42 ? 00:00:00 /usr/sbin/ocwhttpd -DSSL -d /usr
nobody 1506 1500 0 11:42 ? 00:00:00 /usr/sbin/ocwhttpd -DSSL -d /usr
nobody 1507 1500 0 11:42 ? 00:00:00 /usr/sbin/ocwhttpd -DSSL -d /usr
nobody 1508 1500 0 11:42 ? 00:00:00 /usr/sbin/ocwhttpd -DSSL -d /usr
root 1527 1 0 11:42 ? 00:00:04 /usr/bin/ensim-python /usr/lib/o
root 1553 1 0 11:42 ? 00:00:00 /usr/sbin/bandwidth_manager
root 1571 1 0 11:42 ? 00:00:00 crond
root 1589 1 0 11:42 ? 00:00:04 /usr/bin/perl /sbin/poprelayd -d
daemon 1626 1 0 11:42 ? 00:00:00 /usr/sbin/atd
root 1649 1 0 11:42 tty1 00:00:00 /sbin/mingetty tty1
root 1650 1 0 11:42 tty2 00:00:00 /sbin/mingetty tty2
root 1651 1 0 11:42 tty3 00:00:00 /sbin/mingetty tty3
root 1652 1 0 11:42 tty4 00:00:00 /sbin/mingetty tty4
root 1653 1 0 11:42 tty5 00:00:00 /sbin/mingetty tty5
root 1654 1 0 11:42 tty6 00:00:00 /sbin/mingetty tty6
nobody 6271 1500 0 13:01 ? 00:00:00 /usr/sbin/ocwhttpd -DSSL -d /usr
root 6272 1527 0 13:01 ? 00:00:00 /usr/bin/ensim-python /usr/lib/o
root 6273 6272 0 13:01 ? 00:00:03 /usr/bin/ensim-python /usr/lib/o
root 6274 6272 0 13:01 ? 00:00:00 /usr/bin/ensim-python /usr/lib/o
root 6275 6272 0 13:01 ? 00:00:00 /usr/bin/ensim-python /usr/lib/o
root 6276 6272 0 13:01 ? 00:00:00 /usr/bin/ensim-python /usr/lib/o
root 6277 6272 0 13:01 ? 00:00:04 /usr/bin/ensim-python /usr/lib/o
nobody 6310 1500 0 13:01 ? 00:00:00 /usr/sbin/ocwhttpd -DSSL -d /usr
root 7722 1 0 13:18 ? 00:00:02 sendmail: accepting connections
root 7781 1278 0 13:18 ? 00:00:00 /usr/local/sbin/cronolog /home/v
root 7782 1278 0 13:18 ? 00:00:00 /usr/local/sbin/cronolog /home/v
root 7783 1278 0 13:18 ? 00:00:00 /usr/local/sbin/cronolog /home/v
root 7784 1278 0 13:18 ? 00:00:00 /usr/local/sbin/cronolog /home/v
root 7785 1278 0 13:18 ? 00:00:00 /usr/local/sbin/cronolog /home/v
root 7786 1278 0 13:18 ? 00:00:00 /usr/local/sbin/cronolog /home/v
root 7787 1278 0 13:18 ? 00:00:00 /usr/local/sbin/cronolog /home/v
apache 7788 1278 0 13:18 ? 00:00:00 /usr/sbin/httpd -f /etc/httpd/co
apache 7789 1278 0 13:18 ? 00:00:00 /usr/sbin/httpd -f /etc/httpd/co
apache 7790 1278 0 13:18 ? 00:00:00 /usr/sbin/httpd -f /etc/httpd/co
apache 7791 1278 0 13:18 ? 00:00:00 /usr/sbin/httpd -f /etc/httpd/co
apache 7792 1278 0 13:18 ? 00:00:00 /usr/sbin/httpd -f /etc/httpd/co
apache 9716 1278 0 13:58 ? 00:00:00 /usr/sbin/httpd -f /etc/httpd/co
apache 14883 1278 0 15:31 ? 00:00:00 /usr/sbin/httpd -f /etc/httpd/co
apache 14885 1278 0 15:31 ? 00:00:00 /usr/sbin/httpd -f /etc/httpd/co
root 17224 7722 0 16:19 ? 00:00:00 sendmail: server [216.218.102.20
root 17245 17224 0 16:19 ? 00:00:00 sendmail: ./i6UFJsA17245 [216.21
root 17511 1019 0 16:24 ? 00:00:00 /usr/sbin/sshd
root 17522 17511 0 16:25 pts/0 00:00:00 -bash
root 17564 7722 0 16:25 ? 00:00:00 sendmail: server node-c-105a.a20
root 17574 17522 0 16:25 pts/0 00:00:00 ps -ef


Are the processes running. It is a basic webserver with apache and MYSQL
Using an ensim control pannel. Nothing too fancy!

Do any of these processes look suspect?

Ok you say this is a simple webserver but 2 things, actually more jump out at me:

A) Why are you running BIND?



B) Is there a need to run a mail server?




Is this a personal deal or a corporate production server??

It's personal, set-up for my wife's business, allows me to create a site for her unlce too! Seems a good deal at $80 a month and it gave me the chance to try out a few things on a cheap box. There are 5 domains pointing to it at the moment.

We need the mail server running on that machine but it only allows relay from a couple of sites.

I have only ever developed windows boxes through my job, so I am a newbi to Linux, but I would have thought the same security features apply. It's been running for 2 months withno problems.

Not sure why Bind is running I am using the ISP's system.

I was looking for a straight forward install as the ISP specified and setup, using enim to manage everything (including the webserver). The only linux I changed was the ipchains to secure it more thoroughly.

One very strange thing it is doing now is that if I try to view one of the websites unless the html dir has permissions for other set too execute I cant view the webpages.

thanks for the help it is very appreciated!

Toby

Your ISP says its a large amount of outbound traffic and since you have proftp running i would think you are being used as a FTP server. You could also be being used to flood someone else in a DoS attack. But you said you're getting a lot of inbound traffic which makes me think its a DoS attack against you... It sounds more likely that they would be using you as a FTP server or to launch a DoS attack against someone else, are you sure you're getting a lot of inbound traffic, and its not outbound?

Definitely inbound here is the log:

Very wierd about browsing the website, the permissions are shot. surely I should not need to have other set to execute inorder to browse an HTML page!

starting ending in out total
07/01/2004 00:00 07/01/2004 23:59 155 4,556 4,711
07/02/2004 00:00 07/02/2004 23:59 112 4,521 4,634
07/03/2004 00:00 07/03/2004 23:59 6,504 5,723 12,227
07/04/2004 00:00 07/04/2004 23:59 1,364 8,339 9,703
07/05/2004 00:00 07/05/2004 23:59 4,838 6,124 10,962
07/06/2004 00:00 07/06/2004 23:59 942 4,859 5,801
07/07/2004 00:00 07/07/2004 23:59 577 4,620 5,196
07/08/2004 00:00 07/08/2004 23:59 3,832 6,532 10,364
07/09/2004 00:00 07/09/2004 23:59 96,173 92,095 188,269
07/10/2004 00:00 07/10/2004 23:59 4,978 4,971 9,949
07/11/2004 00:00 07/11/2004 23:59 21,705 14,133 35,838
07/12/2004 00:00 07/12/2004 23:59 8,824 6,082 14,906
07/13/2004 00:00 07/13/2004 23:59 22,669 7,355 30,023
07/14/2004 00:00 07/14/2004 23:59 1,695 6,258 7,952
07/15/2004 00:00 07/15/2004 23:59 5,321 9,874 15,195
07/16/2004 00:00 07/16/2004 23:59 21,359 48,192 69,551
07/17/2004 00:00 07/17/2004 23:59 17,852 45,023 62,875
07/18/2004 00:00 07/18/2004 23:59 19,821 48,018 67,839
07/19/2004 00:00 07/19/2004 23:59 19,605 50,293 69,898
07/20/2004 00:00 07/20/2004 23:59 1,031,113 1,046,965 2,078,079
07/21/2004 00:00 07/21/2004 23:59 30,716 58,198 88,914
07/22/2004 00:00 07/22/2004 23:59 1,013,922 1,101,656 2,115,578
07/23/2004 00:00 07/23/2004 23:59 494,724 539,153 1,033,877
07/24/2004 00:00 07/24/2004 23:59 69,400 86,215 155,616
07/25/2004 00:00 07/25/2004 23:59 59,785 82,239 142,024
07/26/2004 00:00 07/26/2004 23:59 89,877 227,060 316,937
07/27/2004 00:00 07/27/2004 23:59 95,033 80,321 175,354
07/28/2004 00:00 07/28/2004 23:59 58,855 68,492 127,347
07/29/2004 00:00 07/29/2004 23:59 9,904,902 64,288 9,969,190

Replace /bin/ls from CD, or put a copy from CD (or ftp, or some safe source, like another identical system) somewhere and use it (try "alias ls /wherever/you/put/it/ls" for instance) instead of the one in /bin (for now).

Then, go check out find. Make sure it's original. Check the timestamp on it. If it's been modified in the past day or two (or three or four, etc) replace it.

Then, do something like:
find / -mtime -# -print
where # is the number of days back to when you know the system was safe. This will produce a listing of files that have been modified in the last # days. The further you go back, the longer the list is likely to be, but if you don't go back far enough, you might miss the intrusion (assuming there was one). In any case, the listing could be long, so you might want to direct the output to a file (by tacking "> /tmp/list.txt" to the end of the command line). The just open the file in a text editor.

If the system was compromised, this will be a VERY effective way to find out, not because it is technically rigorous (you can manually change the timestamps on files with a single command), but because most ppl are just too lazy to cover their tracks by updating timestamps.

The list SHOULD include a bunch of files in /var and wherever your webserver logfiles are.

Things it should definitely NOT include:
any directory whose filename starts with a dot (like /var/tmp/.whatever)
anything in /etc /usr /bin /sbin or anything in /dev that isn't an actual device


This is NOT a complete plan-of-attack, but it's a start.

Good luck, and keep us posted.

-Cengiz

Actually, you do need the html dirs other's perms set to execute. Otherwise, the httpd process won't be able to change to it, to serve out pages. This is true especially if the httpd process and the html directory have different owner/group permissions.

A few of other things:

I. Get rid of the FTP server. For a number of reasons:
1) its insecure.
2) you said this server is for web.
3) you are already running sshd. It can handle secure ftp transfers.

II. If it exists, get rid of the telnet entry in your inetd.conf file. Same reasons for FTP, you don't need clear text telnet sessions if you are already running sshd.

III. Double check your sendmail settings. They are probably using your server as a relay for spam.

I'd just like to point out you seem to have postgres running as well as mysql.
Also, ocwhttpd as well as apache's httpd processes.
You could try downloading www.chkrootkit.org and running that.