LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices



Reply
 
Search this Thread
Old 02-01-2012, 11:15 AM   #1
juodojiakis
LQ Newbie
 
Registered: Jan 2012
Posts: 2

Rep: Reputation: Disabled
/tmp/ repeatedly infected: how to trace (and disable?) server IRC connections?


Hello everyone,

I am volunteering as an administrator for a small history club forum. We're being hosted on a virtual private server, so all administrative tasks fall on our shoulders. However, none of us are good at Linux, so I apologize for the stupid questions incoming and I hope they'll be easy to answer.

In short, we're being infected. To my knowledge, someone is repeatedly infecting /tmp/ and /var/tmp/ with Perl.Shellbot.B (this one comes in many names), disguised as unix.txt or similar system-looking files. This Shellbot then spawns numerous processes (usr/bin/httpd, usr/sbin/ateam...) under user apache, which bring the CPU to ridiculuos loads and the server has to be restarted. after I kill the processes and delete the files, tmp stays clean for some time, but at some point i noticed a "sh -c cd /tmp/ wget (a perl file in some distant domain)" command in the process list.

so, I read in your forum about securing /tmp/ by moving it to a new filesystem with noexec and other security attributes. i also setup a hourly cronjob to delete all files in /tmp/. also disabled apache and root shell access.

despite that, i got email from my host that my server was used for a DoS attack (once again, must be something in /tmp/). The support admin suggested that we should "inspect our server's connections to IRC servers", but the job is left to us, server-newbie historicians.
and here I come to you - is there any way to:
a) trace the mentioned IRC connections?
b) (if possible) disable those connections?

Linux newbies and history experts will greatly appreciate your help. promise to send you nice photos of World War 2!

------------------------------------------
System details: CentOS Linux 4.7, Linux 2.6.32, Webmin installed 1.441 installed. Hardware - 1GB RAM, plenty of HDD space and Intel(R) Xeon(R) CPU E5645 @ 2.40GHz (1438 MHz).

Last edited by juodojiakis; 02-01-2012 at 11:17 AM. Reason: forgot system details
 
Click here to see the post LQ members have rated as the most helpful post in this thread.
Old 02-01-2012, 12:55 PM   #2
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,744
Blog Entries: 54

Rep: Reputation: 2973Reputation: 2973Reputation: 2973Reputation: 2973Reputation: 2973Reputation: 2973Reputation: 2973Reputation: 2973Reputation: 2973Reputation: 2973Reputation: 2973
Quote:
Originally Posted by juodojiakis View Post
I apologize for the stupid questions
No apologies necessary: we've all been new to Linux at one point.
OTOH if you don't learn from mistakes, that would be stupid :-]


Quote:
Originally Posted by juodojiakis View Post
after I kill the processes and delete the files, tmp stays clean for some time, but at some point i noticed a "sh -c cd /tmp/ wget (a perl file in some distant domain)" command in the process list. so, I read in your forum about securing /tmp/ by moving it to a new filesystem with noexec and other security attributes. i also setup a hourly cronjob to delete all files in /tmp/. also disabled apache and root shell access.
Here you're treating symptoms and not the cause. The problem often is with software running in the web stack that has either problematic access permissions, home brewn software that doesn't properly escape user input, stale software versions or software that should not be accessible from the 'net at all.


Quote:
Originally Posted by juodojiakis View Post
i got email from my host that my server was used for a DoS attack (..)
a) trace the mentioned IRC connections?
b) (if possible) disable those connections?
Something like:
Code:
/sbin/iptables -A OUTPUT -m tcp -m multiport 6660:7000 -m state --state NEW -j LOG --log-prefix "IRC "
/sbin/iptables -A OUTPUT -m tcp -m multiport 6660:7000 -m state --state NEW -j DROP
...but best start by:
- listing process information
Code:
( /bin/ps acxfwwwe 2>&1; /usr/sbin/lsof -Pwln 2>&1; /bin/ls -al /var/spool/cron 2>&1; /bin/netstat -anpe 2>&1; /usr/bin/lastlog 2>&1; /usr/bin/last 2>&1; /usr/bin/who -a 2>&1; /sbin/iptables -nvxL ) > /path/to/data.txt
- checking your logs for any anomalies or clues (and attaching or posting them),
- checking user homes and any directories any software in the web stack can write to for rogue files.


Quote:
Originally Posted by juodojiakis View Post
CentOS Linux 4.7, Linux 2.6.32, Webmin installed 1.441 installed.
That's a mortal mistake. Instead of Centos 4 you should be using Centos-5.8 (current AFAIK) and Webmin is at version 1.580. I wonder what more outdated software is accessible from the 'net...

Last edited by unSpawn; 02-01-2012 at 01:55 PM. Reason: //Use code tags ;-p
 
2 members found this post helpful.
Old 02-01-2012, 01:15 PM   #3
business_kid
Guru
 
Registered: Jan 2006
Location: Ireland
Distribution: Slackware & Android
Posts: 6,624

Rep: Reputation: 585Reputation: 585Reputation: 585Reputation: 585Reputation: 585Reputation: 585
Welcome to L

Not the expert here, but there is rkhunter for that sort of time in your life. I'd try that for forensics. (root kit hunter). This command
Quote:
chmod 0666 /tmp
lets nobody execute files from there. That may cause trouble, so chmod 0766 /tmp might be better.

If you are running things as root, stop. If you have an old system, your servers (anything ending in d, e.g. ircd) might be open to exploits. What's your distribution?

Do you use Perl scripts? If not, why not uninstall Perl?

Also, if you don't need stuff running, stop it. Particularly in /etc/inetd.conf (man inetd.conf)
 
1 members found this post helpful.
Old 02-01-2012, 01:27 PM   #4
anomie
Senior Member
 
Registered: Nov 2004
Location: Texas
Distribution: RHEL, Scientific Linux, Debian, Fedora, Lubuntu, FreeBSD
Posts: 3,930
Blog Entries: 5

Rep: Reputation: Disabled
@business_kid: Instead of changing /tmp permissions, another approach is:
Code:
# mount -o remount,noexec /tmp
And I'd follow unSpawn's very excellent suggestions. (Although it would be nice if he'd use code tags. )
 
Old 02-01-2012, 09:52 PM   #5
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,744
Blog Entries: 54

Rep: Reputation: 2973Reputation: 2973Reputation: 2973Reputation: 2973Reputation: 2973Reputation: 2973Reputation: 2973Reputation: 2973Reputation: 2973Reputation: 2973Reputation: 2973
Quote:
Originally Posted by business_kid View Post
there is rkhunter for that sort of time in your life. I'd try that for forensics.
RKH is of limited use in that sort of case (if I described his problem properly). Running RKH does in no way constitute doing forensics.


Quote:
Originally Posted by business_kid View Post
If you are running things as root, stop. If you have an old system, your servers (anything ending in d, e.g. ircd) might be open to exploits. What's your distribution?
Services are often started as root and then drop privileges to a lesser-privileged user. Wrt the rest you should read my first reply again.
 
Old 02-01-2012, 10:04 PM   #6
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,563
Blog Entries: 29

Rep: Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179
chmod 0666 or 0766 /tmp will cause all sorts of breakage
 
Old 02-02-2012, 05:10 AM   #7
juodojiakis
LQ Newbie
 
Registered: Jan 2012
Posts: 2

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by unSpawn View Post
Here you're treating symptoms and not the cause. The problem often is with software running in the web stack that has either problematic access permissions, home brewn software that doesn't properly escape user input, stale software versions or software that should not be accessible from the 'net at all.
well, we've been lazy in our server-side updates, but at least we don't install anything we're not sure about. Is the only way out of this - reinstalling CentOS?

Code:
/sbin/iptables -A OUTPUT -m tcp -m multiport 6660:7000 -m state --state NEW -j LOG --log-prefix "IRC "
/sbin/iptables -A OUTPUT -m tcp -m multiport 6660:7000 -m state --state NEW -j DROP
Somehow my iptables don't like this and reply
Try `iptables -h' or 'iptables --help' for more information.
Bad argument `6660:7000'



Quote:
Originally Posted by unSpawn View Post
...but best start by:
- listing process information
Code:
( /bin/ps acxfwwwe 2>&1; /usr/sbin/lsof -Pwln 2>&1; /bin/ls -al /var/spool/cron 2>&1; /bin/netstat -anpe 2>&1; /usr/bin/lastlog 2>&1; /usr/bin/last 2>&1; /usr/bin/who -a 2>&1; /sbin/iptables -nvxL ) > /path/to/data.txt
- checking your logs for any anomalies or clues (and attaching or posting them),
- checking user homes and any directories any software in the web stack can write to for rogue files.
so I get up, open 'Running Processes' and there it is - "sh wget...", trying to download stuff, and it's child process "php spread.txt", both stuck. (/tmp/ is empty) i hope the process information shows anything:
Code:
 4987 ?        S      0:15  \_ httpd
 6487 ?        S      0:00  |   \_ sh
 6492 ?        S      0:00  |       \_ php
(both are children of some httpd, otherwise indistuingishable from other httpds)

and a lot of established connections to various ports (see attachment). can you make out anything from that?

as to other logs - I've already checked them, but couldn't see anything suspicious. i'll go through them again later today.
the software is kept to a minimum, and I'd be glad to uninstall anything you suggest. i'm running e107 CMS (quite up-to-date), which does have several 777 directories, but i checked them immediately and didn't see anything.

business_kid and anomie: i left tmp attributes at 777. It's already mounted as a new filesystem with noexec, nosuid and similar limitations

as to perl - i'm always willing to uninstall stuff! e107 CMS needs only PHP, but will Webmin run without Perl?

once again, thanks for the help!

Last edited by unSpawn; 02-02-2012 at 11:15 AM. Reason: //Removed attachment wrt exposing machine IP address
 
Old 02-02-2012, 08:05 AM   #8
salasi
Senior Member
 
Registered: Jul 2007
Location: Directly above centre of the earth, UK
Distribution: SuSE, plus some hopping
Posts: 3,919

Rep: Reputation: 779Reputation: 779Reputation: 779Reputation: 779Reputation: 779Reputation: 779Reputation: 779
Quote:
Originally Posted by unSpawn View Post
That's a mortal mistake. Instead of Centos 4 you should be using Centos-5.8 (current AFAIK) and Webmin is at version 1.580. I wonder what more outdated software is accessible from the 'net...
As far as currency of Centos is concerned:

Quote:
Each CentOS version is supported for 7 years (by means of security updates).
(Ref http://wiki.centos.org/)
(I'm not quite clear if this 7 years from the date of release of the major version, ie, x.0, or whether it is from the release date of major.minor version, but in any case...)

According to distrowatch, release dates for Centos versions were:
Code:
2.0     2004/05/24
3.9     2007/07/27
4.9     2011/03/03
5.7     2011/09/13
6.2     2011/12/20
So, technically, it looks like any of those versions, with the exception of 2.0 could be currently kept up-to-date. Practically, you wouldn't want to advise someone to go back to 3.9 as a sensible measure, but 4.9, 5.7 and 6.2 could all be possible.

What I think that we have got here is a version that could have been updated, but wasn't rather than something that couldn't have been updated. You may want to have a look at http://wiki.centos.org/FAQ/General?h...t=%28update%29 for the update information.


Quote:
Originally Posted by unSpawn View Post

Something like:
Code:
/sbin/iptables -A OUTPUT -m tcp -m multiport 6660:7000 -m state --state NEW -j LOG --log-prefix "IRC "
/sbin/iptables -A OUTPUT -m tcp -m multiport 6660:7000 -m state --state NEW -j DROP
Personally, I'd always put some kind of rate limiting in logging, just 'coz I'm a bit paranoid about feeling smug about having done the right thing, and finding out later that I've actually created a new vulnerability, because a high load on logging can cause a problem, that wasn't originally present. Not that this is one of the urgent-est things to attend to, however.

Last edited by salasi; 02-02-2012 at 08:06 AM.
 
1 members found this post helpful.
Old 02-02-2012, 08:28 AM   #9
salasi
Senior Member
 
Registered: Jul 2007
Location: Directly above centre of the earth, UK
Distribution: SuSE, plus some hopping
Posts: 3,919

Rep: Reputation: 779Reputation: 779Reputation: 779Reputation: 779Reputation: 779Reputation: 779Reputation: 779
Quote:
Originally Posted by juodojiakis View Post

Code:
/sbin/iptables -A OUTPUT -m tcp -m multiport 6660:7000 -m state --state NEW -j LOG --log-prefix "IRC "
/sbin/iptables -A OUTPUT -m tcp -m multiport 6660:7000 -m state --state NEW -j DROP
Somehow my iptables don't like this and reply
Try `iptables -h' or 'iptables --help' for more information.
Bad argument `6660:7000'

According to my copy of Iptables tutorial 1.2.2, Oskar Andreasson (...download from frozentux.net...everyone should have a copy...absolutely priceless for occasions like this...not light bedtime reading, but this is no time for light bedtime reading) P190, &ff (table 10-23):
Multiport must be followed by one of
--source-port
--destination-port
--port
where source-port and destination-port do what you'd probably expect* and --port could matches if either source-port or destination-port match. "It (the --port match) can take a maximum of 15 ports and can only be used in conjunction with -p tcp and -p udp. Note that the --port match will only match packets coming in from and going to the same port, for example, port 80 to port 80, port 110 to port 110, and so on."

* If, by 'what you'd expect' you also meant 'has a limit of 15 ports'. There are times when reading the friendly manual is not a sign of weakness.
 
1 members found this post helpful.
Old 02-02-2012, 09:52 AM   #10
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,744
Blog Entries: 54

Rep: Reputation: 2973Reputation: 2973Reputation: 2973Reputation: 2973Reputation: 2973Reputation: 2973Reputation: 2973Reputation: 2973Reputation: 2973Reputation: 2973Reputation: 2973
Thumbs up

Quote:
Originally Posted by salasi View Post
I'm not quite clear if this 7 years from the date of release of the major version
AFAIK The North-American Vendor supports a ten year cycle for major release from the date of the first release. http://wiki.centos.org/FAQ/General#h...91e1dde5b75e6d lists EOL for CentOS-4 as receiving updates until Feb 29, 2012*.


Quote:
Originally Posted by salasi View Post
What I think that we have got here is a version that could have been updated, but wasn't
Agree.


Quote:
Originally Posted by salasi View Post
Personally, I'd always put some kind of rate limiting in logging, just 'coz I'm a bit paranoid about feeling smug about having done the right thing, and finding out later that I've actually created a new vulnerability, because a high load on logging can cause a problem, that wasn't originally present. Not that this is one of the urgent-est things to attend to, however.
Agree, but resource-wise rather disk than CPU IMHO.


Quote:
Originally Posted by salasi View Post
Multiport must be followed by one of (..) --destination-port
Thanks, been posting a bit too quickly.
 
Old 02-02-2012, 10:08 AM   #11
Noway2
Senior Member
 
Registered: Jul 2007
Distribution: Ubuntu 10.10, Slackware 64-current
Posts: 2,124

Rep: Reputation: 776Reputation: 776Reputation: 776Reputation: 776Reputation: 776Reputation: 776Reputation: 776
Looking at your text file of the log output, I see the following connection information:
Code:
tcp        0      0 79.98.25.151:53155          69.168.77.37:6668           ESTABLISHED 48         912383286  6492/php
This is a connection made from the PHP program, launched by a shell interface, from httpd and it is connecting to a typical IRC port range. Looking over your process list, I am not seeing anything particular out of the ordinary for a typical web and email server with a MySQL backend. It does look like material is being uploaded to your /tmp folder and being executed. Here I agree with unSpawn, that setting noexec on this location will be treating the symptom, not addressing the cause of getting the file to upload.

For a point of reference, 69.168.77.37 is a co-lo hosting provider in Utah called Fibernet, and is undoubtedly another "infected" machine of some sort.

What I don't like is the fact that they are running a shell. There are two paths that I would take at this point, one investigate and evaluate your web applications. As unSpawn pointed out previously, home brew PHP (or other web stack code, e.g. PERL) can be a common source of problems as well as outdated CMS and server applications. This would be the first thing to look at. The second thing is that based upon the information, it looks like httpd is an entry vector, that was possibly hijacked into giving a cmd shell, so lets start with that. What version are you running of Apache, PHP, and other web stack applications, e.g. webmin, PHP, MySQL, etc? What I am getting at here is, are you running a version with a known exploit that could account for this?
 
Old 02-02-2012, 11:13 AM   #12
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,744
Blog Entries: 54

Rep: Reputation: 2973Reputation: 2973Reputation: 2973Reputation: 2973Reputation: 2973Reputation: 2973Reputation: 2973Reputation: 2973Reputation: 2973Reputation: 2973Reputation: 2973
Quote:
Originally Posted by juodojiakis View Post
as to perl - i'm always willing to uninstall stuff! e107 CMS needs only PHP, but will Webmin run without Perl?
Best not uninstall or install anything right now. Any (well-meant) advice wrt cleaning up, post-incident hardening and such is rather premature.


Quote:
Originally Posted by juodojiakis View Post
so I get up, open 'Running Processes' and there it is - "sh wget...", trying to download stuff, and it's child process "php spread.txt", both stuck. (/tmp/ is empty) i hope the process information shows anything
Only in that it reinforced what I thought the source could be.


Quote:
Originally Posted by juodojiakis View Post
Is the only way out of this - reinstalling CentOS?
No, we'll get to that after we've tackled your priority #1 problem.


Quote:
Originally Posted by juodojiakis View Post
as to other logs - I've already checked them, but couldn't see anything suspicious. i'll go through them again later today.
Recursively copy, 'scp -r' or rsync /var/log to another machine, install Logwatch there, move the logs to /var/log (or edit the logwatch.conf) and run 'logwatch --numeric --detail 5 --service all --range All --archives --print' for clues?


Quote:
Originally Posted by juodojiakis View Post
i'm running e107 CMS (quite up-to-date), which does have several 777 directories, but i checked them immediately and didn't see anything.
"quite up-to-date" doesn't cut it: we need a version as well because e107 CMS before version 1.0.0 (released 28/Dec/2011) has several vulns, see http://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=e107 .


Quote:
Originally Posted by juodojiakis View Post
and a lot of established connections to various ports (see attachment). can you make out anything from that?
Lots of connections to 174.59.20.100 which is listed as a Malware Controller, a connection to 69.168.77.37 (Malware Controller as well), one to 84.32.250.241 (but only for the process calling "index.cgi" so that might be a false positive) and one to 94.125.182.253 (a "Site associated with email abuse or spam, Malware scan and infect source"). Your machine runs MySQL on any interface (should be confined to localhost), POP3 and IMAP (Dovecot does allow you to run SSL-ized services), FTP (plain text auth), SSH, Sendmail, HTTPS (unused?), Sysfence and like you said Webmin (unclear if access is restricted). On top of this all there seem to be no active firewall rules.


I suggest, in the following order, you:
- stop the web server,
- stop and disable any services you don't need during clean-up (MySQL, FTP, Dovecot and Sendmail),
- apply basic firewall rules,
- update what needs to be updated,
- verify all files on your file system,
- harden the machine
and only then expose it to the 'net again.
 
Old 02-03-2012, 10:37 PM   #13
abefroman
Senior Member
 
Registered: Feb 2004
Location: lost+found
Distribution: CentOS
Posts: 1,278

Rep: Reputation: 53
Quote:
Originally Posted by Noway2 View Post
Looking at your text file of the log output, I see the following connection information:
Code:
tcp        0      0 79.98.25.151:53155          69.168.77.37:6668           ESTABLISHED 48         912383286  6492/php
Look at process 6492 some more.

If you have mod_status install, it will be something like:
Code:
wget 127.0.0.1/server-status
Also run:
Code:
cat /proc/6492/cmdline;cat /proc/6492/environ
HTH
 
1 members found this post helpful.
Old 02-04-2012, 12:57 PM   #14
rhbegin
Member
 
Registered: Oct 2003
Location: Arkansas, NWA
Distribution: Fedora/CentOS/SL6
Posts: 381

Rep: Reputation: 23
Quote:
Originally Posted by unSpawn View Post
No apologies necessary: we've all been new to Linux at one point.
OTOH if you don't learn from mistakes, that would be stupid :-]



Here you're treating symptoms and not the cause. The problem often is with software running in the web stack that has either problematic access permissions, home brewn software that doesn't properly escape user input, stale software versions or software that should not be accessible from the 'net at all.



Something like:
Code:
/sbin/iptables -A OUTPUT -m tcp -m multiport 6660:7000 -m state --state NEW -j LOG --log-prefix "IRC "
/sbin/iptables -A OUTPUT -m tcp -m multiport 6660:7000 -m state --state NEW -j DROP
...but best start by:
- listing process information
Code:
( /bin/ps acxfwwwe 2>&1; /usr/sbin/lsof -Pwln 2>&1; /bin/ls -al /var/spool/cron 2>&1; /bin/netstat -anpe 2>&1; /usr/bin/lastlog 2>&1; /usr/bin/last 2>&1; /usr/bin/who -a 2>&1; /sbin/iptables -nvxL ) > /path/to/data.txt
- checking your logs for any anomalies or clues (and attaching or posting them),
- checking user homes and any directories any software in the web stack can write to for rogue files.



That's a mortal mistake. Instead of Centos 4 you should be using Centos-5.8 (current AFAIK) and Webmin is at version 1.580. I wonder what more outdated software is accessible from the 'net...


Thanks for the listing process commands, it is very helpful, I am putting these in my work notes.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Trace number of connections & threads in apache server sushantchawla2005 Linux - Server 3 05-16-2010 09:23 AM
[SOLVED] Connections to IRC server, and it's not me mrkorb Linux - Security 12 02-09-2010 03:01 AM
how can i trace which process is creating these irc bots bangsters Linux - Security 13 10-04-2009 01:18 AM
help: Disable HTTP Trace / TRACK Methods Paris Heng Linux - Server 3 03-20-2009 04:15 AM
How to trace and disable the HTTP TRACE method in Apache 1.3.33 with FreeBSD? SomnathG Linux - Security 1 11-11-2008 10:41 AM


All times are GMT -5. The time now is 03:42 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration