TTYMON high CPU load

SoX|OK · 09-07-2009, 03:06 AM

Hi @ll,

hopefully I'm in the correct forum, other please move this thread.

I'm nearly new in using LINUX and now I recognized a "problem" on my Debian server.

If I open TOP, I can see that the process ttymon (command ttymon tymon) uses more than 50% of the CPU. Any idea what happened or what should I do?

In the past I never recognized this process and I have no clue why it uses so much CPU-time.

Any help would be appreciated

UPDATE:
I just recognized another strange thing:
I see that also perl have a high CPU-load and that several commans are running like:
./nn
./nnn
./nnu
./nni

Any idea regarding this?

Regards,
OK

unSpawn · 09-07-2009, 05:03 AM

Quote:

Originally Posted by SoX|OK

I'm nearly new in using LINUX and now I recognized a "problem" on my Debian server. If I open TOP, I can see that the process ttymon (command ttymon tymon) uses more than 50% of the CPU. Any idea what happened or what should I do? In the past I never recognized this process and I have no clue why it uses so much CPU-time.

UPDATE:
I just recognized another strange thing:
I see that also perl have a high CPU-load and that several commans are running like:
./nn
./nnn
./nnu
./nni

By default ttymon should be a terminal port monitor. However in your case it may be collateral from a compromise. We'll help you find out if it is, to what point they've gotten and what the damage is. But before doing anything it is crucial you read the CERT Intruder Detection Checklist (old): http://web.archive.org/web/200801092...checklist.html.

I'll outline in short what we need to do:
- You need to read the Intruder Detection Checklist so you know what to do later on,
- Save open files (\lsof -P -w -n), process (\ps ax -o ppid,pid,uid,cmd --sort=uid) and network data (\netstat -anpe) listings to a location where you do not overwrite data or pipe through ssh,
- Gain control by shutting down all non-essential services and raising the firewall to only allow SSH traffic to and from your management IP (range). If the machine is local to you and you have any doubts power down the box and only boot with a Live CD unless you tell us the machine is remote (tell us),
- Tell us when you first encountered this situation,
- Tell us if the server exhibited any odd behaviour in the past,
- Tell us (attach logs?) if you have recent logs from running Chkrootkit / Rootkit Hunter or Aide / Samhain (or Integrit, Osiris or even tripwire) (don't install anything),
- Tell us what services the server runs and if it is a LAMP machine also what runs on top of Perl/PHP/Ruby (forum, web log, awstats) and possibly their versions,
- Tell us (attach output?) what the /tmp, /var/tmp and webserver docroot directories hold (find /dirname -ls),

After you've answered those question (don't install or delete anything) we'll move on to preparing backups for reference (not reuse) and investigate further (if necessary) using (changes in or logged reports about) system authentication data, any IDS logs (Snort, Prelude, router logs, filesystem integrity checkers, package manager if good enough), all system, daemon and firewall logs, temp files, unusual (setuid root) files, user shell histories. When you report back include any information, hints, hunches or gut feelings you think would help. Please attach logs if possible, else please use BB code tags to preserve formatting and efficient reading. And please ask before doing things if you have any doubts.

// Please note this thread will be moved to the Linux Security forum RSN.

SoX|OK · 09-07-2009, 05:39 AM

Hi unSpawn,

I just read the CERT Intruder Detection Checklist.

"\lsof -P -w -n" <- does not work

Code:

debian31m:/home/joerg# \netstat -anpe
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode       PID/Program name
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      101        5859        2731/mysqld
tcp        0      0 0.0.0.0:113             0.0.0.0:*               LISTEN      0          5990        2786/inetd
tcp        0      0 0.0.0.0:4949            0.0.0.0:*               LISTEN      0          6890        3095/munin-node
tcp        0      0 0.0.0.0:21              0.0.0.0:*               LISTEN      0          6072        2837/vsftpd
tcp        0      0 0.0.0.0:1110            0.0.0.0:*               LISTEN      0          5369        2241/sshd
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      0          5593        2524/exim4
tcp        0      0 0.0.0.0:65534           0.0.0.0:*               LISTEN      0          7046        3154/ttyload
tcp        0    420 [localaddress]:1110      [remoteaddress]:19144      ESTABLISHED 0          6789        3037/sshd: joerg [p
tcp6       0      0 :::80                   :::*                    LISTEN      0          6780        3038/apache2
tcp6       0      0 :::1110                 :::*                    LISTEN      0          5367        2241/sshd
udp        0      0 0.0.0.0:500             0.0.0.0:*                           0          5328        2230/openvpn
udp        0      0 0.0.0.0:501             0.0.0.0:*                           0          5311        2221/openvpn
udp        0      0 0.0.0.0:502             0.0.0.0:*                           0          5294        2212/openvpn
udp        0      0 0.0.0.0:503             0.0.0.0:*                           0          5267        2203/openvpn
udp        0      0 192.168.75.9:123        0.0.0.0:*                           0          6104        2846/ntpd
udp        0      0 10.0.0.1:123            0.0.0.0:*                           0          6103        2846/ntpd
udp        0      0 10.0.0.5:123            0.0.0.0:*                           0          6102        2846/ntpd
udp        0      0 10.0.0.9:123            0.0.0.0:*                           0          6101        2846/ntpd
udp        0      0 88.198.32.131:123       0.0.0.0:*                           0          6100        2846/ntpd
udp        0      0 127.0.0.1:123           0.0.0.0:*                           0          6099        2846/ntpd
udp        0      0 0.0.0.0:123             0.0.0.0:*                           0          6092        2846/ntpd
udp6       0      0 fe80::213:d3ff:fea8:123 :::*                                0          6098        2846/ntpd
udp6       0      0 ::1:123                 :::*                                0          6097        2846/ntpd
udp6       0      0 :::123                  :::*                                0          6093        2846/ntpd
raw        0      0 0.0.0.0:1               0.0.0.0:*               7           0          7050        3157/ttymon
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node   PID/Program name    Path
unix  2      [ ACC ]     STREAM     LISTENING     5860     2731/mysqld         /var/run/mysqld/mysqld.sock
unix  2      [ ]         DGRAM                    2339     1024/udevd          @/org/kernel/udev/udevd
unix  10     [ ]         DGRAM                    5235     2170/syslogd        /dev/log
unix  3      [ ]         STREAM     CONNECTED     7213     3037/sshd: joerg [p
unix  3      [ ]         STREAM     CONNECTED     7212     3209/0
unix  2      [ ]         DGRAM                    7211     3037/sshd: joerg [p
unix  2      [ ]         DGRAM                    6089     2846/ntpd
unix  2      [ ]         DGRAM                    5858     2733/logger
unix  2      [ ]         DGRAM                    5326     2230/openvpn
unix  2      [ ]         DGRAM                    5309     2221/openvpn
unix  2      [ ]         DGRAM                    5292     2212/openvpn
unix  2      [ ]         DGRAM                    5265     2203/openvpn
unix  2      [ ]         DGRAM                    5247     2179/klogd

Code:

debian31m:/home/joerg# \ps ax -o ppid,pid,uid,cmd --sort=uid
 PPID   PID   UID CMD
    0     1     0 init [2]
    0     2     0 [kthreadd]
    2     3     0 [migration/0]
    2     4     0 [ksoftirqd/0]
    2     5     0 [watchdog/0]
    2     6     0 [events/0]
    2     7     0 [khelper]
    2    39     0 [kblockd/0]
    2    41     0 [kacpid]
    2    42     0 [kacpi_notify]
    2   121     0 [ksuspend_usbd]
    2   127     0 [khubd]
    2   130     0 [kseriod]
    2   167     0 [pdflush]
    2   168     0 [pdflush]
    2   169     0 [kswapd0]
    2   170     0 [aio/0]
    2   351     0 [xfslogd/0]
    2   352     0 [xfsdatad/0]
    2   353     0 [xfs_mru_cache]
    2   379     0 [ata/0]
    2   380     0 [ata_aux]
    2   383     0 [scsi_eh_0]
    2   385     0 [scsi_eh_1]
    2   902     0 [md0_raid1]
    2   910     0 [md1_raid1]
    2   943     0 [kjournald]
    1  1024     0 udevd --daemon
    2  1730     0 [kpsmoused]
    2  1741     0 [kstriped]
    2  1763     0 [ksnapd]
    1  2170     0 /sbin/syslogd
    1  2179     0 /sbin/klogd -x
    1  2203     0 /usr/sbin/openvpn --writepid /var/run/openvpn.privat.pid --daemon ovpn-privat --status /var/run/openvpn.privat.status 10 -
    1  2212     0 /usr/sbin/openvpn --writepid /var/run/openvpn.home.pid --daemon ovpn-home --status /var/run/openvpn.home.status 10 --cd /e
    1  2221     0 /usr/sbin/openvpn --writepid /var/run/openvpn.stef.pid --daemon ovpn-stef --status /var/run/openvpn.stef.status 10 --cd /e
    1  2230     0 /usr/sbin/openvpn --writepid /var/run/openvpn.work.pid --daemon ovpn-work --status /var/run/openvpn.work.status 10 --cd /e
    1  2241     0 /usr/sbin/sshd
    1  2692     0 /bin/sh /usr/bin/mysqld_safe
 2692  2733     0 logger -p daemon.err -t mysqld_safe -i -t mysqld
    1  2786     0 /usr/sbin/inetd
    1  2837     0 /usr/sbin/vsftpd
    1  2856     0 /sbin/mdadm --monitor --pid-file /var/run/mdadm/monitor.pid --daemonise --scan --syslog
    1  2884     0 /usr/sbin/cron
 2241  3037     0 sshd: joerg [priv]
    1  3038     0 /usr/sbin/apache2 -k start
    1  3095     0 /usr/sbin/munin-node
    1  3124     0 /sbin/getty 38400 tty1
    1  3126     0 /sbin/getty 38400 tty2
    1  3128     0 /sbin/getty 38400 tty3
    1  3129     0 /sbin/getty 38400 tty4
    1  3130     0 /sbin/getty 38400 tty5
    1  3131     0 /sbin/getty 38400 tty6
    1  3154     0 /sbin/ttyload -q
    1  3157     0 ttymon tymon
 3210  3216     0 su
 3216  3217     0 bash
 3217  3326     0 ps ax -o ppid,pid,uid,cmd --sort=uid
    1  2864     1 /usr/sbin/atd
 3038  3090    33 /usr/sbin/apache2 -k start
 3038  3091    33 /usr/sbin/apache2 -k start
 3038  3092    33 /usr/sbin/apache2 -k start
 3038  3093    33 /usr/sbin/apache2 -k start
 3038  3094    33 /usr/sbin/apache2 -k start
 2692  2731   101 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-external-locking --por
    1  2524   102 /usr/sbin/exim4 -bd -q30m
    1  2846   110 /usr/sbin/ntpd -p /var/run/ntpd.pid -u 110:109 -g
 3037  3209  1000 sshd: joerg@pts/0
 3209  3210  1000 -bash

Code:

debian31m:/home/joerg# find /tmp -ls
13303809    4 drwxrwxrwt   4 root     root         4096 Sep  7 12:20 /tmp
13402561    4 drwxrwxrwt   2 root     root         4096 Sep  7 12:20 /tmp/.X11-unix
13402562    4 drwxrwxrwt   2 root     root         4096 Sep  7 12:20 /tmp/.ICE-unix

Code:

debian31m:/home/joerg# find /var/tmp -ls
4292678    4 drwxrwxrwt   3 root     root         4096 Sep  7 11:47 /var/tmp
4292679    4 drwxrwxrwt   2 root     root         4096 Sep 23  2008 /var/tmp/vi.recover

As far as I know, nothing from this (Chkrootkit / Rootkit Hunter or Aide / Samhain (or Integrit, Osiris or even tripwire)) is installed.

I recognized this problem today the first time and have it on two different servers. Both server are used for HLDS-Counterstrike, one runs also a VPN,BNC and Eggdrop-bots - both run also apache2, mysql and PHP.
One server is located in Nuremberg and the other in Frankfurt, so I do not have direct access.

The shhd port is moved to a non-standard-port and only ports are allowed in the firewall, that belong to specific programs (hlds, bnc and so on).

After a reboot of both servers, the "./nn" tasks and perl run fine - I could not see any problem with them anymore but ttymon is still using 50% of the CPU load on one of the servers (the one with VPN, BNC, eggdrop bots, apache2, mysql & php).

SoX|OK · 09-07-2009, 05:40 AM

I also checked , as mentioned in the link, the passwd file and could not find any new account nor an account that has the same access/grouplevel as root.

Tinkster · 09-07-2009, 05:43 AM

Moved: This thread is more suitable in <SECURITY> and has been moved accordingly to help your thread/question get the exposure it deserves.

SoX|OK · 09-07-2009, 05:50 AM

Info:
cat /etc/debian_version
5.0.3

cat /proc/version
Linux version 2.6.26.1 (root@debian31m) (gcc version 4.3.2 (Debian 4.3.2-1.1) ) #1 SMP PREEMPT Tue Feb 24 12:47:30 CET 2009

unSpawn · 09-07-2009, 09:59 AM

Quote:

Originally Posted by SoX|OK

I just read the CERT Intruder Detection Checklist.

Read and executed it, by the looks of it.

Quote:

Originally Posted by SoX|OK

"\lsof -P -w -n" <- does not work

Find out if it is installed. Only if the installed binary matches the packages checksum execute 'lsof' (might need to prefix the path).

Quote:

Originally Posted by SoX|OK

Code:

debian31m:/home/joerg# \netstat -anpe
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode       PID/Program name
raw        0      0 0.0.0.0:1               0.0.0.0:*               7           0          7050        3157/ttymon

Code:

debian31m:/home/joerg# \ps ax -o ppid,pid,uid,cmd --sort=uid
 PPID   PID   UID CMD
    1  3154     0 /sbin/ttyload -q
    1  3157     0 ttymon tymon

Notice how in 'netstat' output the protocol is "raw" and the UID is "root". Notice how in 'ps' output the parent PID is 1 (init) and again the UID is "root". ttyload in combination with ttymon leads me to believe we're looking at evidence of the SHV4 (or 5) rootkit. This means you have a root compromise on your hands, so at this point all bets are off. If you have off-site backups, assess if they are tainted or not.

Quote:

Originally Posted by SoX|OK

After a reboot of both servers, the "./nn" tasks and perl run fine

I asked you not to perform anything other than list details. By rebooting you have permanently destroyed "evidence" of the intruder like open network connections and process details.

I'd like to point out that root compromises are one of the worst things that can happen to a GNU/Linux server. You will now have the opportunity to (re-)prioritize tasks, and in this order: mitigate, collect "evidence" (all logs and auth data), shut down servers, reinstall from scratch. At this point the first thing you must do is mitigate the situation by shutting down all non-essential services and raising the firewall to only allow SSH traffic to and from your management IP.
- I also must point out that deliberately not mitigating the situation is a hostile act towards other Internet users.
- Be aware that if you skip the evidence collecting stage you loose the opportunity to find out how the intruder gained entrance.
- To properly manage your expectations: for root compromises there will be no "cleanup" stage to "restore" your server, even if we do find the point(s) of entry and even if your backups are not tainted.

SoX|OK · 09-07-2009, 11:05 AM

The reboot was done before I post my thread here

A login via root and ssh is not possible as far as I know, because for sshd root login is denied.

All services running on that host (vpn, hlds, apache, ftp, ZNC, eggdrop,...) are now stopped and all ports closed.

I also checked the logfiles (syslog, lastlog, apache-logs,authlog,...) and could not find anything strange, except in the authlog there are several entries like:

Quote:

Sep 7 18:15:01 debian31m CRON[4372]: pam_unix(cron:session): session opened for user munin by (uid=0)
Sep 7 18:15:01 debian31m CRON[4372]: pam_unix(cron:session): session closed for user munin
Sep 7 18:15:01 debian31m CRON[4371]: pam_unix(cron:session): session opened for user root by (uid=0)
Sep 7 18:15:01 debian31m CRON[4371]: pam_unix(cron:session): session closed for user root
Sep 7 18:15:01 debian31m CRON[4375]: pam_unix(cron:session): session opened for user root by (uid=0)
Sep 7 18:15:01 debian31m CRON[4375]: pam_unix(cron:session): session closed for user root
Sep 7 18:17:01 debian31m CRON[4414]: pam_unix(cron:session): session opened for user root by (uid=0)
Sep 7 18:17:01 debian31m CRON[4414]: pam_unix(cron:session): session closed for user root
Sep 7 18:20:01 debian31m CRON[4541]: pam_unix(cron:session): session opened for user root by (uid=0)
Sep 7 18:20:01 debian31m CRON[4541]: pam_unix(cron:session): session closed for user root

So if I understand you correct, the way to fix it is a reinstall of the server?

unSpawn · 09-07-2009, 11:44 AM

Quote:

Originally Posted by SoX|OK

The reboot was done before I post my thread here

I see.

Quote:

Originally Posted by SoX|OK

A login via root and ssh is not possible as far as I know, because for sshd root login is denied. All services running on that host (vpn, hlds, apache, ftp, ZNC, eggdrop,...) are now stopped and all ports closed.

OK. Are there any other users with admin access to the machine? Are there any other users who can access the machine right now? Did you also kill the ttyload and ttymon processes?

Quote:

Originally Posted by SoX|OK

I also checked the logfiles (syslog, lastlog, apache-logs,authlog,...) and could not find anything strange,

How far do the logs go back? Do you know what to look for? If you would want a second opinion you're invited to tarball all your logs, compress 'em, (host them or upload it to some free host) and email me the D/L location (else contact me by email).

Quote:

Originally Posted by SoX|OK

except in the authlog there are several entries like

Checked /etc/crontab and the user crontabs in /var/spool/cron?

Quote:

Originally Posted by SoX|OK

So if I understand you correct, the way to fix it is a reinstall of the server?

In the end, yes. I understand that setting up and maintaining a server represents an investment in terms of money and time but a root compromise means you can no longer trust the machine, and irrepairably so. However it would be good to find out how they got in. Else you might reinstall from scratch allowing the same vulnerability to reappear.

SoX|OK · 09-07-2009, 11:55 AM

There is no other account that have admin-rights. Only root and only I use the root-account, no one else should know the password.
All other accounts are personalized and only these can access the server via ssh or ftp and the ssh port is moved to another port.

ttyload and ttymon are still active, should I kill them?

All logfiles have a minimum of 7 days - some much longer.
Sent a download link via mail to unSpawn.

Code:

debian31m:/var/log# cat /etc/crontab
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# m h dom mon dow user  command
17 *   * * * root    cd / && run-parts --report /etc/cron.hourly
25 6   * * * root  test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
47 6   * * 7 root  test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
52 6   1 * * root  test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )
#

All user-crontabs look fine, as I set them up with no change.

Absolutly confirmed, makes more sense to find out how the came in instead of setting up the server again and run into the same mistake

Another thing, do you think that my server is compromised because of the ttymon and ttyload thing or because of the perl and ./nn thing? Because I recognized that also on my second server but there is everything working fine a reboot and I haven't recognized the ttymon/ttyload before the reboot. So I'm a littl frightend that maybe both servers were compromised

SoX|OK · 09-07-2009, 12:26 PM

I think I found something

auth.log.6:
Jul 22 06:25:01 debian31m su[19327]: Successful su for nobody by root

passwd:
nobody:x:65534:65534:nobody:/nonexistent:/bin/sh

65534 is the same as the port where ttymon/ttyload is listening, right?

nobody is also in /etc/passwd but as far as I know, I never setup an account like "nobody" - could that be my "problem"?

unSpawn · 09-07-2009, 12:37 PM

Quote:

Originally Posted by SoX|OK

There is no other account that have admin-rights. Only root and only I use the root-account, no one else should know the password.

SHV5 means (among others) a trojaned /bin/login so yes, somebody else will know the root account password.

Quote:

Originally Posted by SoX|OK

ttyload and ttymon are still active, should I kill them?

Yes, but remove the ttyload and ttymon lines first from/etc/inittab.

I'll wait for the log link.

Quote:

Originally Posted by SoX|OK

auth.log.6: Jul 22 06:25:01 debian31m su[19327]: Successful su for nobody by root

Yes, that does not look right. Auth data may be tampered with, does 'last' show data about this login as well?

Quote:

Originally Posted by SoX|OK

passwd: nobody:x:65534:65534:nobody:/nonexistent:/bin/sh
65534 is the as the port where ttymon/ttyload is listening, right?

Right. And 'ttyload' is a SSH daemon.

Quote:

Originally Posted by SoX|OK

nobody is also in /etc/passwd but as far as I know, I never setup an account like "nobody" - could that be my "problem"?

Each GNU/Linux system will have an inert (meaning it doesn't have a login shell) "nobody" system account but as far as I know it should have an UID way below 500.

SoX|OK · 09-07-2009, 12:51 PM

Killed 2 processes (ttymon and ttyload) and removed ttyload from the inittab.

I sent the link via this page, no idea if you recieve an email or a message here on the page

Yes, also the actual logfiles show the login for nobody and always at 06:25:01 am.

I searched "nobody" with google and bing, and someone recognized that this is normal and should tell that root granted su for user nobody, normally a cronjob (e.g. exim4-base) - could that be?

Also the config in passwd would be normal for a debian etch installation.
Checked my workstation here with a nearly fresh install and no internet-connection - found the same entry for user nobody.

unSpawn · 09-07-2009, 08:03 PM

OK. I have gotten your tarball and I only had time for a quick look at the Apache logs which proved to be quite interesting. If believed not tampered with then access.log starts on 2008/Sep/07 and error.log starts on 2008/Aug/31. Having loads of logs helps reconstruct timelines and it is nice to have.

If you decompress and collate all Apache error.logs and run

Code:

egrep -vie "(\[notice\]|w00t|not.exist|dynamic.library|negotiated|unable.to.stat|overlaps|Invalid|resuming|qualified|skipping|denied:.access|NameVirtualHost|SANS)"

on it then on 2009/Jul/06 a successful attack is started ("200 OK" response) downloading an unknown script to "/tmp/90.txt", likely a perl backdoor, and at later dates a prefab vmsplice exploit, some C code and a host of perl-based backdoor scripts. While crackers still obfuscate files by using seemingly innocuous (and therefore unblocked) extensions like ".jpg" they show here they openly download "c" code. Also the failures to find results "(cracked.txt|/tmp/cracker/find.txt): No such file or directory" and compilation errors ('nou.c:') shows some toolkits aren't working properly and provide at least some form of "early warning" if logs are actually read.

2009/Jul/06 correlates with Apache access.logs

Code:

grep 'config.inc.php'  access.logs | egrep -vie "(HTTP\/1\.[0-1]\" [3,4]0[0-9] )"

showing HTTP "/phpmyadmin/config.inc.php?c=" GET requests which AFAIK points to PMASA-2009-3 of 2009/Mar/24 / CVE-2009-1151 of 2009/Mar/26 aka the PhpMyAdmin remote code execution exploit. (And at least one unconfirmed source showed that on 2009/Jul/29 Debian Etch phpmyadmin 2.9.* still was unpatched.)

And there you have it: publicly accessable toolage that should not be (it does have "admin" in the name for a reason) and a vulnerable version to boot. I'll have a more thorough look at things since I haven't found the SHV5 rootkit entries but I'd say the ability to bring in compiled exploits, compiling code and executing tools undetected for over a month wraps things up pretty much.

HTH

SoX|OK · 09-08-2009, 02:51 AM

Hi There,

so if I understand everything correct, then my problem was the outdated phpmyadmin. The attacker used a bug in the installed version to run some script, right?

Any idea if this is my only failure? Or should we look into something else as well?

Normally I check for updates regulaly, I have no clue why phpmyadmin wasn't updated