Find process which generates TCP packets

darkxer0x · 12-17-2010, 12:21 PM

Hi!
My machine is trying to communicate with another computer. I´ve blocked the traffic with this machine with iptables (input and output traffic), but I want to find the origin of this traffic. There´re 90% of probabilities it´s a trojan, and I want to find it.
I have logged the packets with iptables (and then dropped), but with this I don´t know the proccess source.
I´ve tried with netstat -o, but I don´t get nothing.

How can I see the Process source (i.e. the PID) of this traffic?
The traffic are TCP packets, with SYN flagged active (my machine is trying to establish a connection with that IP).

druuna · 12-17-2010, 12:27 PM

Hi,

Maybe this will help:

Code:

netstat -plan | more

Does need to be run as root to see all processes.

Hope this helps.

unSpawn · 12-17-2010, 02:06 PM

...or as root run 'lsof -Pwlni tcp'. If nothing works and the process doesn't appear to be in the process table (beware of processes changing their argv[0] to something seemingly innocuous) then I suggest you shut down the machine and boot a Live CD. That won't get you the traffic but it does give you a better chance to find out what's wrong. If you reply details about the machine (distro, release, kernel, accessible services, applications in the web stack if any) would be welcome.

darkxer0x · 12-18-2010, 07:23 AM

Thanks to both of you.
With netstat -plan :
tcp 0 1 my_machine:53835 89.187.53.226:8086 SYN_SENT 3838/apache2

With lsof -Pwlni tcp:
apache2 3838 33 15u IPv4 46969 TCP my_machine:53836->89.187.53.226:8086 (SYN_SENT)

I think I've a problem. My apache is trying to establish a connection with this IP (and is the only one).
I've done a google of this ip, reverse lookup (dig -x), and nothing. Maybe one of my CMS is infected. I suspect of zen photo, because it appears in apache log, connections between this IP and setup.php of zen photo.

Thanks again for your help!

unSpawn · 12-18-2010, 08:18 AM

Quote:

Originally Posted by darkxer0x

Code:

apache2    3838       33   15u  IPv4  46969       TCP my_machine:53836->89.187.53.226:8086 (SYN_SENT)

Does 'readlink -f /proc/3838/exe' show the process to be your regular Apache binary? If not then what? If it is Apache, what anomalies do your access and error log show?

Quote:

Originally Posted by darkxer0x

I think I've a problem. My apache is trying to establish a connection with this IP (and is the only one). I've done a google of this ip, reverse lookup (dig -x), and nothing.

It's registered as being in the Republica Moldova?

Quote:

Originally Posted by darkxer0x

Maybe one of my CMS is infected. I suspect of zen photo, because it appears in apache log, connections between this IP and setup.php of zen photo.

Please don't talk about issues but instead post exact error messages, log lines, your exact version of Zenphoto, if you protected the setup directory through mod_security or .htaccess files, et cetera.

unixfool · 12-18-2010, 08:39 AM

This thread is marked as solved? I see no solution or remediation. :/

darkxer0x · 12-18-2010, 12:07 PM

Quote:

Originally Posted by unixfool

This thread is marked as solved? I see no solution or remediation. :/

I asked "Find process which generates TCP packets", and this is solved. I marked the post as solved because if one person search "find process generate TCP packet", can see this post, and that question is solved.

Quote:

Originally Posted by unSpawn

Does 'readlink -f /proc/3838/exe' show the process to be your regular Apache binary? If not then what? If it is Apache, what anomalies do your access and error log show?

Yes, it's the binary of apache:

Code:

readlink -f /proc/30411/exe
/usr/sbin/apache2

The same apache as others web process. And /usr/sbin/apache2 is OK, because I've run debsums:

Code:

debsums -a apache2-mpm-prefork
/usr/sbin/apache2                                                             OK

About the anomalies, I have this in access.log:

Code:

./error.log.1:[Tue Dec 07 21:31:28 2010] [error] [client 89.187.53.226] File does not exist: /var/www/zp-core
./error.log:[Fri Dec 17 03:48:52 2010] [error] [client 89.187.53.226] File does not exist: /var/www/zp-core
./access.log:89.187.53.226 - - [17/Dec/2010:03:48:52 +0100] "GET /zp-core/setup.php HTTP/1.0" 404 215 "-" "Wget/1.11.4 Red Hat modified"
./access.log:89.187.53.226 - - [17/Dec/2010:03:48:52 +0100] "GET /fotos/zp-core/setup.php HTTP/1.0" 200 5929 "-" "Wget/1.11.4 Red Hat modified"
./access.log:89.187.53.226 - - [17/Dec/2010:03:48:53 +0100] "POST /fotos/zp-core/setup.php HTTP/1.0" 200 6367 "-" "Wget/1.11.4 Red Hat modified"
./access.log:89.187.53.226 - - [17/Dec/2010:03:48:54 +0100] "POST /fotos/zp-core/setup.php HTTP/1.0" 200 5929 "-" "Wget/1.11.4 Red Hat modified"
./access.log:89.187.53.226 - - [17/Dec/2010:03:48:54 +0100] "POST /fotos/ HTTP/1.0" 200 9865 "-" "Wget/1.11.4 Red Hat modified"
./access.log.1:89.187.53.226 - - [07/Dec/2010:21:31:27 +0100] "POST / HTTP/1.0" 200 175 "-" "Wget/1.11.4 Red Hat modified"
./access.log.1:89.187.53.226 - - [07/Dec/2010:21:31:27 +0100] "POST /fotos/ HTTP/1.0" 200 9859 "-" "Wget/1.11.4 Red Hat modified"
./access.log.1:89.187.53.226 - - [07/Dec/2010:21:31:28 +0100] "GET /zp-core/setup.php HTTP/1.0" 404 215 "-" "Wget/1.11.4 Red Hat modified"

I'm not the maintainer of the web, so I didn't know that setup.php of the CMS zen photo was not deleted. When I saw this, I deleted. If you run setup.php, you can see all the version details os the machine (OS version, apache and php version...). Although you can't execute because is password protected.

Quote:

Originally Posted by unSpawn

It's registered as being in the Republica Moldova?

Yes. The reputation of the IP-range is not very good. See this post

Quote:

Originally Posted by unSpawn

Please don't talk about issues but instead post exact error messages, log lines, your exact version of Zenphoto, if you protected the setup directory through mod_security or .htaccess files, et cetera.

I've post above the acces.log with the lines of that IP. About zen photo, the version is 1.2.2 [2983]. Setup directory is not protected, but if you want to install, you have to enter a password in setup.php. It's as zen photo "worked" by default.

I'm going to talk to the web maintaner to update all the web software's versions of the machine. This version of zen photo is from 2008...

unSpawn · 12-18-2010, 01:07 PM

Quote:

Originally Posted by darkxer0x

I'm going to talk to the web maintaner to update all the web software's versions of the machine. This version of zen photo is from 2008...

Remember you started this thread because Apache was making connections (instead of just answering) it should not. Depending on (apparent lack of) maintenance and (mis-)configuration of the system and how low this has been gone undetected it would be better to first investigate and ensure integrity of the system before removing and installing software. Also please note that while deleting things is an understandable reflex it is not automagically the good one (think then act) because deleting things also wipes out any traces. Basically that would only benefit the miscreants.

darkxer0x · 12-18-2010, 02:23 PM

Quote:

Originally Posted by unSpawn

Remember you started this thread because Apache was making connections (instead of just answering) it should not.

I started this thread beacause an unknown process was making connections. Now, I know it´s apache.

Quote:

Originally Posted by unSpawn

Depending on (apparent lack of) maintenance and (mis-)configuration of the system and how low this has been gone undetected it would be better to first investigate and ensure integrity of the system before removing and installing software. Also please note that while deleting things is an understandable reflex it is not automagically the good one (think then act) because deleting things also wipes out any traces. Basically that would only benefit the miscreants.

The file I deleted, setup.php, was only moved out of the web directory. And the first thing I did was see what it contained. And it contained nothing stranged. And the modification time was the same as the others files in setup directory.
As you said, it would be better to first investigate and ensure integrity of the system. I´m investating the source, but I have to ensure the integrity of the system. That´s why I´m going to update all the software. I´ll make a backup of all the current files, but having old software it´s not good for the integrity of the system.

The first thing I did was search the IP in all the web files, but it didn´t appear anything.
I´ve got the backtrace of the process, and I get:

Code:

#0  0x00007fa747889b9f in poll () from /lib/libc.so.6
#1  0x00007fa741095871 in php_network_connect_socket ()
   from /usr/lib/apache2/modules/libphp5.so
#2  0x00007fa741095b5d in php_network_connect_socket_to_host ()
   from /usr/lib/apache2/modules/libphp5.so
#3  0x00007fa7410a4e6f in ?? () from /usr/lib/apache2/modules/libphp5.so
#4  0x00007fa740ec962f in ?? () from /usr/lib/apache2/modules/libphp5.so
#5  0x00007fa741098d99 in ?? () from /usr/lib/apache2/modules/libphp5.so
#6  0x00000000025a8c98 in ?? ()
#7  0x0000000000000012 in ?? ()
#8  0x0000000000000000 in ?? ()

And these are the files used:

Code:

 lsof -p 3838
COMMAND  PID     USER   FD   TYPE DEVICE    SIZE    NODE NAME
apache2 3838 www-data  cwd    DIR    8,1    4096 4022268 /var/www/fotos
apache2 3838 www-data  rtd    DIR    8,1    4096       2 /
apache2 3838 www-data  txt    REG    8,1  418032 5792716 /usr/sbin/apache2
apache2 3838 www-data  mem    REG    8,1         5820799 /usr/lib/php5/20060613/pdo_mysql.so (path inode=5820360)
apache2 3838 www-data  mem    REG    8,1         5820795 /usr/lib/php5/20060613/pdo.so (path inode=5820727)
apache2 3838 www-data  mem    REG    8,1         5820800 /usr/lib/php5/20060613/mysqli.so (path inode=5820359)
apache2 3838 www-data  mem    REG    8,1         5820801 /usr/lib/php5/20060613/mysql.so (path inode=5820357)
apache2 3838 www-data  mem    REG    8,1   29472 5792275 /usr/lib/libltdl.so.3.1.6
apache2 3838 www-data  mem    REG    8,1  174680 5792278 /usr/lib/libmcrypt.so.4.4.7
apache2 3838 www-data  mem    REG    8,1         5820798 /usr/lib/php5/20060613/mcrypt.so (path inode=5820361)
apache2 3838 www-data  mem    REG    8,1   20096 5790877 /usr/lib/libXdmcp.so.6.0.0
apache2 3838 www-data  mem    REG    8,1    9824 5790875 /usr/lib/libXau.so.6.0.0
apache2 3838 www-data  mem    REG    8,1  114176 5790879 /usr/lib/libxcb.so.1.0.0
apache2 3838 www-data  mem    REG    8,1    5896 5790881 /usr/lib/libxcb-xlib.so.0.0.0
apache2 3838 www-data  mem    REG    8,1  200344 5792264 /usr/lib/libfontconfig.so.1.3.0
apache2 3838 www-data  mem    REG    8,1  139392 5792266 /usr/lib/libjpeg.so.62.0.0
apache2 3838 www-data  mem    REG    8,1         5792269 /usr/lib/libpng12.so.0.27.0 (path inode=5788001)
apache2 3838 www-data  mem    REG    8,1   69272 5792271 /usr/lib/libXpm.so.4.11.0
apache2 3838 www-data  mem    REG    8,1 1096880 5790883 /usr/lib/libX11.so.6.2.0
apache2 3838 www-data  mem    REG    8,1         5792262 /usr/lib/libfreetype.so.6.3.18 (path inode=5787997)
apache2 3838 www-data  mem    REG    8,1  290976 5792280 /usr/lib/libt1.so.5.1.2
apache2 3838 www-data  mem    REG    8,1  271120 5792273 /usr/lib/libgd.so.2.0.0
apache2 3838 www-data  mem    REG    8,1         5820797 /usr/lib/php5/20060613/gd.so (path inode=5820725)
apache2 3838 www-data  mem    REG    8,1         2981641 /lib/libnss_files-2.7.so (path inode=2979845)
apache2 3838 www-data  mem    REG    8,1         2981637 /lib/libnss_nis-2.7.so (path inode=2979847)
apache2 3838 www-data  mem    REG    8,1         2981630 /lib/libnss_compat-2.7.so (path inode=2979842)
apache2 3838 www-data  mem    REG    8,1    8472 5820985 /usr/lib/apache2/modules/mod_userdir.so
apache2 3838 www-data  mem    REG    8,1    6968 5820974 /usr/lib/apache2/modules/mod_suexec.so
apache2 3838 www-data  mem    REG    8,1   19544 5820970 /usr/lib/apache2/modules/mod_status.so
apache2 3838 www-data  mem    REG    8,1  173152 5820967 /usr/lib/apache2/modules/mod_ssl.so
apache2 3838 www-data  mem    REG    8,1   11256 5820959 /usr/lib/apache2/modules/mod_setenvif.so
apache2 3838 www-data  mem    REG    8,1   61464 5820987 /usr/lib/apache2/modules/mod_rewrite.so
apache2 3838 www-data  mem    REG    8,1         5791681 /usr/lib/libxml2.so.2.6.32 (path inode=5789861)
apache2 3838 www-data  mem    REG    8,1   66224 2979510 /lib/libbz2.so.1.0.4
apache2 3838 www-data  mem    REG    8,1         5820796 /usr/lib/apache2/modules/libphp5.so (path inode=5820726)
apache2 3838 www-data  mem    REG    8,1   32056 5820980 /usr/lib/apache2/module                                  s/mod_negotiation.so
apache2 3838 www-data  mem    REG    8,1   17072 5820968 /usr/lib/apache2/modules/mod_mime.so
apache2 3838 www-data  mem    REG    8,1   23992 5819508 /usr/lib/apache2/modules/mod_ldap_userdir.so
apache2 3838 www-data  mem    REG    8,1   49232 5820949 /usr/lib/apache2/modules/mod_ldap.so
apache2 3838 www-data  mem    REG    8,1    7152 5820951 /usr/lib/apache2/modules/mod_env.so
apache2 3838 www-data  mem    REG    8,1    7600 5820981 /usr/lib/apache2/modules/mod_dir.so
apache2 3838 www-data  mem    REG    8,1   20184 5820948 /usr/lib/apache2/modules/mod_deflate.so
apache2 3838 www-data  mem    REG    8,1   24048 5820976 /usr/lib/apache2/modules/mod_cgi.so
apache2 3838 www-data  mem    REG    8,1   34200 5820971 /usr/lib/apache2/modules/mod_autoindex.so
apache2 3838 www-data  mem    REG    8,1    6936 5820930 /usr/lib/apache2/modules/mod_authz_user.so
apache2 3838 www-data  mem    REG    8,1    8664 5820928 /usr/lib/apache2/modules/mod_authz_host.so
apache2 3838 www-data  mem    REG    8,1    9440 5820929 /usr/lib/apache2/modules/mod_authz_groupfile.so
apache2 3838 www-data  mem    REG    8,1    5664 5820934 /usr/lib/apache2/modules/mod_authz_default.so
apache2 3838 www-data  mem    REG    8,1   26264 5820933 /usr/lib/apache2/module                                  s/mod_authnz_ldap.so
apache2 3838 www-data  mem    REG    8,1    6968 5820761 /usr/lib/apache2/modules/mod_authn_file.so
apache2 3838 www-data  mem    REG    8,1    8536 5820935 /usr/lib/apache2/modules/mod_auth_basic.so
apache2 3838 www-data  mem    REG    8,1   12280 5820986 /usr/lib/apache2/modules/mod_alias.so
apache2 3838 www-data  mem    REG    8,1    7736 2981621 /lib/libkeyutils-1.2.so
apache2 3838 www-data  mem    REG    8,1         5790814 /usr/lib/libkrb5support.so.0.1 (path inode=5789725)
apache2 3838 www-data  mem    REG    8,1         5790811 /usr/lib/libk5crypto.so.3.1 (path inode=5789835)
apache2 3838 www-data  mem    REG    8,1  421512 5789845 /usr/lib/libgcrypt.so.1 1.4.4
apache2 3838 www-data  mem    REG    8,1   65064 5789872 /usr/lib/libtasn1.so.3.0.15
apache2 3838 www-data  mem    REG    8,1   93504 5789136 /usr/lib/libz.so.1.2.3.3
apache2 3838 www-data  mem    REG    8,1         2981625 /lib/libm-2.7.so (path inode=2979839)
apache2 3838 www-data  mem    REG    8,1         2981629 /lib/libnsl-2.7.so (path inode=2979841)
apache2 3838 www-data  mem    REG    8,1         5790815 /usr/lib/libgssapi_krb5.so.2.2 (path inode=5789726)
apache2 3838 www-data  mem    REG    8,1   11648 2979560 /lib/libcom_err.so.2.1
apache2 3838 www-data  mem    REG    8,1         5790810 /usr/lib/libkrb5.so.3.3(path inode=5789858)
apache2 3838 www-data  mem    REG    8,1         5789955 /usr/lib/libcrypto.so.0.9.8 (path inode=5789312)
apache2 3838 www-data  mem    REG    8,1         5789956 /usr/lib/libssl.so.0.9.8 (path inode=5789313)
apache2 3838 www-data  mem    REG    8,1         5789860 /usr/lib/libgnutls.so.26.4.6 (path inode=5789258)
apache2 3838 www-data  mem    REG    8,1  107360 5791669 /usr/lib/libsasl2.so.2.0.22
apache2 3838 www-data  mem    REG    8,1         2981635 /lib/libresolv-2.7.so (path inode=2979850)
apache2 3838 www-data  mem    REG    8,1         2981631 /lib/libdl-2.7.so (path inode=2979838)
apache2 3838 www-data  mem    REG    8,1         2981627 /lib/libcrypt-2.7.so (path inode=2979836)
apache2 3838 www-data  mem    REG    8,1         2981634 /lib/librt-2.7.so (pathbinode=2979851)
apache2 3838 www-data  mem    REG    8,1   15344 2979565 /lib/libuuid.so.1.2
apache2 3838 www-data  mem    REG    8,1  169776 5791617 /usr/lib/libexpat.so.1.5.2
apache2 3838 www-data  mem    REG    8,1  483832 5787783 /usr/lib/libsqlite3.so.0.8.6
apache2 3838 www-data  mem    REG    8,1         5791622 /usr/lib/libmysqlclient_r.so.15.0.0 (path inode=5792404)
apache2 3838 www-data  mem    REG    8,1         5791625 /usr/lib/libpq.so.5.1 (path inode=5790219)
apache2 3838 www-data  mem    REG    8,1 1348776 5788687 /usr/lib/libdb-4.6.so
apache2 3838 www-data  mem    REG    8,1   61000 5791671 /usr/lib/liblber-2.4.so.2.1.0
apache2 3838 www-data  mem    REG    8,1  290456 5791672 /usr/lib/libldap_r-2.4.so.2.1.0
apache2 3838 www-data  mem    REG    8,1         2981633 /lib/libc-2.7.so (path inode=2979833)
apache2 3838 www-data  mem    REG    8,1         2981644 /lib/libpthread-2.7.so(path inode=2979828)
apache2 3838 www-data  mem    REG    8,1         5791615 /usr/lib/libapr-1.so.0.2.12 (path inode=5789911)
apache2 3838 www-data  mem    REG    8,1         5791644 /usr/lib/libaprutil-1.so.0.2.12 (path inode=5790221)
apache2 3838 www-data  mem    REG    8,1  162816 5788481 /usr/lib/libpcre.so.3.12.1
apache2 3838 www-data  mem    REG    8,1         2981640 /lib/ld-2.7.so (path inode=2979830)
apache2 3838 www-data  DEL    REG    0,8            8641 /dev/zero
apache2 3838 www-data  DEL    REG    0,8            8639 /dev/zero
apache2 3838 www-data  DEL    REG    0,8            8636 /dev/zero
apache2 3838 www-data  mem    REG    8,1   13168 5789847 /usr/lib/libgpg-error.so.0.3.0
apache2 3838 www-data    0r   CHR    1,3             550 /dev/null
apache2 3838 www-data    1w   CHR    1,3             550 /dev/null
apache2 3838 www-data    2w   REG    8,1   71496 4022092 /var/log/apache2/error.log
apache2 3838 www-data    3u  IPv4 107416             TCP my_machine:59269->89.187.53.226:8086 (SYN_SENT)
apache2 3838 www-data    4u  sock    0,4            8618 can't identify protocol
apache2 3838 www-data    6u  sock    0,4            8622 can't identify protocol
apache2 3838 www-data    7r  FIFO    0,6            8635 pipe
apache2 3838 www-data    8w  FIFO    0,6            8635 pipe
apache2 3838 www-data    9w   REG    8,1       0 4021961 /var/log/apache2/other_vhosts_access.log
apache2 3838 www-data   10w   REG    8,1  184397 4022007 /var/log/apache2/ssl_access.log
apache2 3838 www-data   11w   REG    8,1 1610398 4022090 /var/log/apache2/access.log
apache2 3838 www-data   12w   REG    8,1       0 4022087 /var/run/apache2/ssl_mutex (deleted)
apache2 3838 www-data   13u  0000    0,7       0      13 anon_inode
apache2 3838 www-data   14u  sock    0,4            8789 can't identify protocol

Zen photo is under /var/www/fotos, the same CWD (current working directory) as process 3838. What a coincidence!! xDD
Of course port 8086 is open in 89.187.53.226...

Thanks again, unSpawn

unSpawn · 12-19-2010, 11:16 AM

Quote:

Originally Posted by darkxer0x

The file I deleted, setup.php, was only moved out of the web directory. And the first thing I did was see what it contained. And it contained nothing stranged. And the modification time was the same as the others files in setup directory.

All good then.

Quote:

Originally Posted by darkxer0x

I´ll make a backup of all the current files, but having old software it´s not good for the integrity of the system.

I see no reason for doing that unless you have somehow established the machine was compromised, and then making backups would be done solely for investigative purposes. Certainly not for restoring the machine to a certain (compromised) state later on...

Quote:

Originally Posted by darkxer0x

but I have to ensure the integrity of the system. That´s why I´m going to update all the software.

What I mean is that you have to make certain before you update. Updating itself does not ensure there are no issues and that integrity is intact.