LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices

Reply
 
Search this Thread
Old 01-22-2013, 01:49 PM   #1
compix
LQ Newbie
 
Registered: Jan 2013
Distribution: CentOS
Posts: 17

Rep: Reputation: Disabled
Determining the file/application used by a network connection.


Hello All,

I am not a Linux expert, but not a newbie either. I am recently experiencing an interesting problem that I just wanted to share and get your opinions.

A few days ago one of my server's IP, which I use as a shared web server for a bunch of websites, got blacklisted. The server is a Linux server running CentOS with cPanel control panel.

I have found the note below in the database which was blacklisted my IP.

Quote:

This IP is infected with, or is NATting for a machine infected with Win32/Zbot (Microsoft).

This was detected by observing this IP attempting to make contact to a Zeus Command and Control server, with contents unique to Zeus C&C command protocols.

Zbot is known by other names: Wsnpoem (Symantec) and most commonly as Zeus.

Zbot/Zeus is a banking trojan, and specializes in stealing personal information (passwords, account information, etc) from interactions with banking sites through the use of "formgrabs".

This was detected by a TCP/IP connection from X.X.X.X on port 58955 going to IP address 87.255.51.229 (the sinkhole) on port 80.
The botnet command and control domain for this connection was "zxcqsd1rfasga.com".

This detection corresponds to a connection at 2013-01-10 15:20:20 (GMT - this timestamp is believed accurate to within one second).

These infections are rated as a "severe threat" by Microsoft. It is a trojan downloader, and can download and execute ANY software on the infected computer.

You will need to find and eradicate the infection before delisting the IP address.

We strongly recommend that you DO NOT simply firewall off connections to the sinkhole IP addresses given above. Those IP addresses are of sinkholes operated by malware researchers. In other words, it's a "sensor" (only) run by "the good guys". The bot "thinks" its a command and control server run by the spambot operators but it isn't. It DOES NOT actually download anything, and is not a threat. If you firewall the sinkhole addresses, your IPs will remain infected, and they will STILL be delivering your users/customers personal information, including banking information to the criminal bot operators.

If you do choose to firewall these IPs, PLEASE instrument your firewall to tell you which internal machine is connecting to them so that you can identify the infected machine yourself and fix it.

We are enhancing the instructions on how to find these infections, and more information will be given here as it becomes available.

Virtually all detections made by the CBL are of infections that do NOT leave any "tracks" for you to find in your mail server logs. This is even more important for the viruses described here - these detections are made on network-level detections of malicious behaviour and may NOT involve malicious email being sent.

This means: if you have port 25 blocking enabled, do not take this as indication that your port 25 blocking isn't working.

The links above may help you find this infection. You can also consult Advanced Techniques for other options and alternatives. NOTE: the Advanced Techniques link focuses on finding port 25(SMTP) traffic. With "sinkhole malware" detections such as this listing, we aren't detecting port 25 traffic, we're detecting traffic on other ports. Therefore, when reading Advanced Techniques, you will need to consider all ports, not just SMTP.

Pay very close attention: Most of these trojans have extremely poor detection rates in current Anti-Virus software. For example, Ponmocup is only detected by 3 out of 49 AV tools queried at Virus Total.

Thus: having your anti-virus software doesn't find anything doesn't prove that you're not infected.

While we regret having to say this, downloaders will generally download many different malicious payloads. Even if an Anti-Virus product finds and removes the direct threat, they will not have detected or removed the other malicious payloads. For that reason, we recommend recloning the machine - meaning: reformatting the disks on the infected machine, and re-installing all software from known-good sources.
The interesting thing is, this Zeus thing seems to be working only Windows based machines. So it should not be possible for this to happen on a Linux server in my opinion. The only thing that I can think of is, a proxy script or something that an account might be using which might have been used by an infected Windows computer. However, this is the point where I am lost as I am not sure how I can detect this script on server, and I am even not sure if there is any.

When I check the active connections on the server during a day(via netstat), I can see multiple connections to remote servers over port 80 but I am not sure if I can say that all these connections are caused by the same thing? (I guess they might be Wordpress caching stuff etc., might they?)

I was wondering if this can be tracked through this way or another way? Is my assumption correct? I would really appreciate any comments/opinions on this whether be positive or negative.

Thank you.
 
Old 01-22-2013, 03:58 PM   #2
lykwydchykyn
Member
 
Registered: Mar 2006
Location: Tennessee, USA
Distribution: Debian, Ubuntu
Posts: 135

Rep: Reputation: 36
Running netstat with the -p option (must be root to do this) will show you what process is associated with each network connection. Have you tried that already?
 
Old 01-22-2013, 04:35 PM   #3
compix
LQ Newbie
 
Registered: Jan 2013
Distribution: CentOS
Posts: 17

Original Poster
Rep: Reputation: Disabled
Yes, tried that.

Quote:
root@server [~]# netstat -anp | grep 80
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 5393/httpd
tcp 0 0 SERVERS_IP_ADDRESS:80 REMOTE_IP_ADDRESS:56004 ESTABLISHED 11520/httpd
tcp 0 0 SERVERS_IP_ADDRESS:80 REMOTE_IP_ADDRESS:25190 FIN_WAIT2 -
tcp 0 0 SERVERS_IP_ADDRESS:993 REMOTE_IP_ADDRESS:38056 ESTABLISHED 11427/imap-login
tcp 0 0 SERVERS_IP_ADDRESS:80 REMOTE_IP_ADDRESS:25185 FIN_WAIT2 -
tcp 0 0 SERVERS_IP_ADDRESS:80 REMOTE_IP_ADDRESS:25187 FIN_WAIT2 -
tcp 0 1 SERVERS_IP_ADDRESS:32986 REMOTE_IP_ADDRESS:80 SYN_SENT 12878/php
tcp 0 1 SERVERS_IP_ADDRESS:37569 REMOTE_IP_ADDRESS:80 SYN_SENT 12908/php
tcp 0 0 SERVERS_IP_ADDRESS:80 REMOTE_IP_ADDRESS:25189 FIN_WAIT2 -
tcp 0 0 SERVERS_IP_ADDRESS:80 REMOTE_IP_ADDRESS:25186 FIN_WAIT2 -
tcp 0 0 SERVERS_IP_ADDRESS:993 REMOTE_IP_ADDRESS:33980 ESTABLISHED 11426/imap-login
unix 2 [ ] DGRAM 5380154 830/auditd
After that, I have tried the command below, thinking that the lsof can show me the file used by that PID.

However, I only see the root web directory of the cPanel account with lsof. And when I check there, there I see just regular Wordpress files.

Quote:
root@server[~]# lsof -p 12878
php 12878 CPANEL_USERNAME cwd DIR 202,1 4096 6525245 /home/CPANEL_USERNAME/public_html
php 12878 CPANEL_USERNAME rtd DIR 202,1 4096 2 /
php 12878 CPANEL_USERNAME txt REG 202,1 24901084 9977892 /usr/bin/php
php 12878 CPANEL_USERNAME mem REG 202,1 17892 985545 /lib/libdl-2.12.so
php 12878 CPANEL_USERNAME mem REG 202,1 350318 1082342 /opt/tidy/lib/libtidy-0.99.so.0.0.0
php 12878 CPANEL_USERNAME mem REG 202,1 462574 1082329 /opt/libmcrypt/lib/libmcrypt.so.4.4.8
php 12878 CPANEL_USERNAME mem REG 202,1 1608420 6065878 /usr/lib/libcrypto.so.1.0.0
php 12878 CPANEL_USERNAME mem REG 202,1 67800 6066393 /usr/lib/libXpm.so.4.11.0
php 12878 CPANEL_USERNAME mem REG 202,1 158736 6066203 /usr/lib/libpng12.so.0.49.0
php 12878 CPANEL_USERNAME mem REG 202,1 674744 5373981 /usr/lib/libaspell.so.15.1.4
php 12878 CPANEL_USERNAME mem REG 202,1 354976 6065880 /usr/lib/libssl.so.1.0.0
php 12878 CPANEL_USERNAME mem REG 202,1 621356 6066184 /usr/lib/libfreetype.so.6.3.22
php 12878 CPANEL_USERNAME mem REG 202,1 141668 6066199 /usr/lib/libjpeg.so.62.0.0
php 12878 CPANEL_USERNAME mem REG 202,1 267344 1016372 /opt/pcre/lib/libpcre.so.0.0.1
php 12878 CPANEL_USERNAME mem REG 202,1 44744 5373964 /usr/lib/libicuio.so.42.1
php 12878 CPANEL_USERNAME mem REG 202,1 75332 985585 /lib/libz.so.1.2.3
php 12878 CPANEL_USERNAME mem REG 202,1 39676 985567 /lib/librt-2.12.so
php 12878 CPANEL_USERNAME mem REG 202,1 254016 985643 /lib/libgssapi_krb5.so.2.2
php 12878 CPANEL_USERNAME mem REG 202,1 379105 1049266 /opt/curlssl/lib/libcurl.so.4.2.0
php 12878 CPANEL_USERNAME mem REG 202,1 201740 985605 /lib/libidn.so.11.6.1
php 12878 CPANEL_USERNAME mem REG 202,1 9536 985641 /lib/libkeyutils.so.1.3
php 12878 CPANEL_USERNAME mem REG 202,1 38376 985543 /lib/libcrypt-2.12.so
php 12878 CPANEL_USERNAME mem REG 202,1 1272672 6066385 /usr/lib/libX11.so.6.3.0
php 12878 CPANEL_USERNAME mem REG 202,1 131080 985459 /lib/i686/nosegneg/libpthread-2.12.so
php 12878 CPANEL_USERNAME mem REG 202,1 714629 1082357 /opt/xslt/lib/libxslt.so.1.1.27
php 12878 CPANEL_USERNAME mem REG 202,1 103384 985565 /lib/libresolv-2.12.so
php 12878 CPANEL_USERNAME mem REG 202,1 13836 985596 /lib/libcom_err.so.2.1
php 12878 CPANEL_USERNAME mem REG 202,1 152734 6605547 /usr/local/lib/php/extensions/no-debug-non-zts-20090626/pdo_mysql.so
php 12878 CPANEL_USERNAME mem REG 202,1 58704 985555 /lib/libnss_files-2.12.so
php 12878 CPANEL_USERNAME mem REG 202,1 35304 6066487 /usr/lib/libltdl.so.7.2.1
php 12878 CPANEL_USERNAME mem REG 202,1 120672 986712 /lib/libgcc_s-4.4.6-20120305.so.1
php 12878 CPANEL_USERNAME mem REG 202,1 73536 6605239 /usr/local/lib/php/extensions/no-debug-non-zts-20090626/ixed.5.3.lin
php 12878 CPANEL_USERNAME mem REG 202,1 25592 985553 /lib/libnss_dns-2.12.so
php 12878 CPANEL_USERNAME mem REG 202,1 70464 985591 /lib/libbz2.so.1.0.4
php 12878 CPANEL_USERNAME mem REG 202,1 117228 6066359 /usr/lib/libxcb.so.1.1.0
php 12878 CPANEL_USERNAME mem REG 202,1 50816 985625 /lib/libpam.so.0.82.2
php 12878 CPANEL_USERNAME mem REG 202,1 300676 985532 /lib/libfreebl3.so
php 12878 CPANEL_USERNAME mem REG 202,1 120780 985472 /lib/libselinux.so.1
php 12878 CPANEL_USERNAME mem REG 202,1 141016 985465 /lib/ld-2.12.so
php 12878 CPANEL_USERNAME mem REG 202,1 38608 985651 /lib/libkrb5support.so.0.1
php 12878 CPANEL_USERNAME mem REG 202,1 986620 6605549 /usr/local/lib/php/extensions/no-debug-non-zts-20090626/sqlite.so
php 12878 CPANEL_USERNAME mem REG 202,1 8500 6066270 /usr/lib/libXau.so.6.0.0
php 12878 CPANEL_USERNAME mem REG 202,1 161084 985639 /lib/libexpat.so.1.5.2
php 12878 CPANEL_USERNAME mem REG 202,1 173812 985647 /lib/libk5crypto.so.3.1
php 12878 CPANEL_USERNAME mem REG 202,1 112224 985621 /lib/libaudit.so.1.0.0
php 12878 CPANEL_USERNAME mem REG 202,1 2848 5373983 /usr/lib/libpspell.so.15.1.4
php 12878 CPANEL_USERNAME mem REG 202,1 113908 985549 /lib/libnsl-2.12.so
php 12878 CPANEL_USERNAME mem REG 202,1 872700 985649 /lib/libkrb5.so.3.3
php 12878 CPANEL_USERNAME mem REG 202,1 200024 985547 /lib/libm-2.12.so
php 12878 CPANEL_USERNAME mem REG 202,1 283826 6605250 /usr/local/lib/php/extensions/no-debug-non-zts-20090626/pdo.so
php 12878 CPANEL_USERNAME mem REG 202,1 233120 1082390 /opt/xslt/lib/libexslt.so.0.8.16
php 12878 CPANEL_USERNAME mem REG 202,1 1338252 5373972 /usr/lib/libicuuc.so.42.1
php 12878 CPANEL_USERNAME mem REG 202,1 1678608 5373962 /usr/lib/libicui18n.so.42.1
php 12878 CPANEL_USERNAME mem REG 202,1 16010708 5373960 /usr/lib/libicudata.so.42.1
php 12878 CPANEL_USERNAME mem REG 202,1 866928 6881329 /usr/local/IonCube/ioncube_loader_lin_5.3.so
php 12878 CPANEL_USERNAME mem REG 202,1 930192 6065840 /usr/lib/libstdc++.so.6.0.13
php 12878 CPANEL_USERNAME mem REG 202,1 917366 6525081 /usr/local/Zend/lib/Guard-5.5.0/php-5.3.x/ZendGuardLoader.so
php 12878 CPANEL_USERNAME mem REG 202,1 4951028 6068867 /usr/lib/libmysqlclient.so.16.0.0
php 12878 CPANEL_USERNAME mem REG 202,1 1919076 985530 /lib/i686/nosegneg/libc-2.12.so
php 12878 CPANEL_USERNAME mem REG 202,1 1965464 6605548 /usr/local/lib/php/extensions/no-debug-non-zts-20090626/pdo_sqlite.so
php 12878 CPANEL_USERNAME mem REG 202,1 3466413 1049427 /opt/xml2/lib/libxml2.so.2.9.0
php 12878 CPANEL_USERNAME DEL REG 0,4 282558464 /SYSV00000000
php 12878 CPANEL_USERNAME 0r FIFO 0,8 0t0 16043481 pipe
 
Old 01-22-2013, 06:34 PM   #4
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,462
Blog Entries: 54

Rep: Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899
Quote:
Originally Posted by compix View Post
A few days ago one of my server's IP
That's an understatement: the report says it got listed nearly two weeks ago.
Linux may be free to use but using it is not free of responsibilities.


Quote:
Originally Posted by compix View Post
a proxy script or something that an account might be using which might have been used by an infected Windows computer.
What I find interesting is the reason your web server is (perceived to be) (ab)used for proxying traffic. AFAIK Zeus (and I haven't read that much really) isn't that "intelligent" to modify a users proxy settings and while I should not speculate about these things there's two generic options: somebody deliberately configured their machine to use yours for proxying purposes or somebody "discovered" yours as an open proxy.

First of let's try to log and mitigate things. Download the ipblocklist from https://zeustracker.abuse.ch/blocklist.php and create an egress filter using 'ipset':
Code:
ipset create BLOCK hash:net
cat zeus_ipblocklist.txt | xargs -iX ipset add BLOCK 'X'
review the list got loaded OK:
Code:
ipset list BLOCK | less
then add these logging and blocking rules at their appropriate position ('iptables -t filter --line-numbers -nvxL OUTPUT;' to review) in the filter table OUTPUT chain:
Code:
iptables -t filter -A OUTPUT -p ALL -m state --state NEW -m set --match-set BLOCK dst -j LOG --log-prefix "OUT_zeus "
iptables -t filter -A OUTPUT -p ALL -m state --state NEW -m set --match-set BLOCK dst -j DROP
*Note the list is refreshed about ever twenty four hours so you probably want to automate downloading the new list, cleaning stale addresses and loading new ones into list. And please don't mistake this set of blocking rules for a "solution".

Secondly out-of-the-box Apache installations still come with all LSO's enabled including proxying ('grep "^LoadModule.proxy" httpd.conf;') so it would be good to review your configuration for what you allow. That goes for the firewall and other services as well. We don't know the location of the web server (could you be willingly proxying for your own LAN?) or if you're running a proxy service too. And like you said an interpreter-based proxy script or other "fun" could be running. It can be grepped for or better: use RFx' Linux Malware Detect ClamAV rules (just load them on the clamscan command line) on your own and your customers homes and all docroots. While you're at it review the servers system and daemon logs for anomalies (or use Logwatch for generating leads), review your own and your customers homes for anything seemingly odd and while you inspect things don't leave your own local machine(s) out.

Third, yes, you can list (I prefer 'lsof -Pwln -a -i;') processes and network traffic but without ways to correlate it (remote domain names or IP addresses, Zeus agent signatures, process details that mark it suspicious or rogue) I wonder how efficient that will be. (I made a web log post about traffic correlation here but I hope you find the cause without having to resort to that kind of stuff.) Sure you should inspect processes but IMHO you shouldn't focus on remote ports. Besides there's no way telling if traffic will be constant or transient. If your investigation turns up nothing you could install Snort as Emerging Threats, zeustracker and SourceFire include Zeus signatures and sniff traffic but I'd go for quick wins first.
 
1 members found this post helpful.
Old 01-23-2013, 06:38 PM   #5
compix
LQ Newbie
 
Registered: Jan 2013
Distribution: CentOS
Posts: 17

Original Poster
Rep: Reputation: Disabled
Hi unSpawn, that was very helpful! Thank you very much for the idea(s).
 
Old 01-23-2013, 07:00 PM   #6
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,462
Blog Entries: 54

Rep: Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899
I noticed you marked this thread solved without posting how you solved the case. This does not leave any clues for others who might face the same problem in the future and it's a rather unsatisfactory ending for those who love to troubleshoot and diagnose things. So I would appreciate it if you would reciprocate and offer us a quick run-through of that measures you took and of your findings,
TIA.
 
Old 01-29-2013, 09:48 AM   #7
compix
LQ Newbie
 
Registered: Jan 2013
Distribution: CentOS
Posts: 17

Original Poster
Rep: Reputation: Disabled
Hi unSpawn, you are right I should let everyone know with what I did. I was just way too hasty while working on this problem and completely forgot this thread. Sorry about that.

Here's what I did;

I have tried running LMD (Linux Malware Detect). It did found about 8-10 (some were same) and I have cleaned all of them.
I have created an egress filter as advised and monitored the logs for about 2-3 days and have not noticed any outgoing connection to Zeus trackers. (I also created an update script to update blocklist every 24 hours.) I have not been blacklisted by the database so far.
As for your Apache configuration suggestion, this server has a default(almost) cPanel/WHM Apache configuration and I confirmed that the proxy module was not enabled/loaded. I have also my firewall tightly configured, just allows regular incoming/outgoing HTTP, HTTPS, POP(S), IMAP(S), DNS and some cPanel/WHM specific port connections.

I'll keep monitoring this and will update this thread if I notice anything new.

Thanks again!
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Determining address for UDP sendto calls after a prior TCP connection sh86 Programming 1 02-09-2010 10:53 AM
Determining the network throughput speed kaplan71 Linux - Networking 1 01-13-2010 05:57 PM
Simple application to write IP addresses on a network to a file n0kx Linux - Software 3 09-12-2007 04:35 AM
Application are very slow due to lack of Network connection winxandlinx Linux - Networking 2 02-18-2007 10:35 PM
determining if shell is local connection or remote SaxyWeed Linux - General 3 01-26-2004 11:17 PM


All times are GMT -5. The time now is 11:16 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration