hard disk memory usage weirldy high

nikooo777 · 04-02-2010, 01:55 PM

hello! i am having a problem that i would call a bit "important" with my server.
so, from last 3 weeks the used space of my hard disk (RAID I) started growing up.
i have 2 x 1 tb HDD working on RAID I and i did not install anything those weeks.
the space just started changing from 90 GB till 580 GB. now the situation is stable there but i think it's not normal.
the bandwidth usage is low (like 120 gb in 2 months) and i am running 6 counter strike gameservers, a forum, a very little website and some local stuffs...
a friend of mine told me that my server could have been hacked but i am afraid it did... some useful informations:
when i reboot the server the used space goes down again to ~100 GB and then it starts going up again.

i cant really find where all those files are located:

Quote:

box:/opt/lampp# du /* -hs | grep [0-9]G | sort -rn | head -10
du: cannot access `/proc/21190/task/21190/fd/4': No such file or directory
du: cannot access `/proc/21190/task/21190/fdinfo/4': No such file or directory
du: cannot access `/proc/21190/fd/4': No such file or directory
du: cannot access `/proc/21190/fdinfo/4': No such file or directory
8.8G /home
2.8G /var

i checked all directories with du -h and the biggest are those...
i really have no idea what to do :/
ill post also the list of the processes running on my system incase anyone would need them.

thank you in advice

Quote:

box:/opt/lampp# ps ax
PID TTY STAT TIME COMMAND
1 ? Ss 0:09 init [2]
2 ? S< 0:00 [kthreadd]
3 ? S< 0:00 [migration/0]
4 ? S< 0:07 [ksoftirqd/0]
5 ? S< 0:00 [migration/1]
6 ? S< 0:08 [ksoftirqd/1]
7 ? S< 0:00 [migration/2]
8 ? S< 0:09 [ksoftirqd/2]
9 ? S< 0:00 [migration/3]
10 ? S< 0:14 [ksoftirqd/3]
11 ? S< 0:00 [events/0]
12 ? S< 0:00 [events/1]
13 ? S< 0:00 [events/2]
14 ? S< 0:00 [events/3]
15 ? S< 0:00 [cpuset]
16 ? S< 0:00 [khelper]
19 ? S< 0:00 [netns]
22 ? S< 0:00 [async/mgr]
237 ? S< 0:00 [kblockd/0]
238 ? S< 0:01 [kblockd/1]
239 ? S< 0:00 [kblockd/2]
240 ? S< 0:00 [kblockd/3]
242 ? S< 0:00 [kacpid]
243 ? S< 0:00 [kacpi_notify]
244 ? S< 0:00 [kacpi_hotplug]
315 ? S< 0:00 [ata/0]
316 ? S< 0:00 [ata/1]
317 ? S< 0:00 [ata/2]
318 ? S< 0:00 [ata/3]
319 ? S< 0:00 [ata_aux]
324 ? S< 0:00 [ksuspend_usbd]
328 ? S< 0:00 [khubd]
331 ? S< 0:00 [kseriod]
422 ? S 2:30 [pdflush]
423 ? S< 2:03 [kswapd0]
471 ? S< 0:00 [aio/0]
472 ? S< 0:00 [aio/1]
473 ? S< 0:00 [aio/2]
474 ? S< 0:00 [aio/3]
485 ? S< 0:00 [nfsiod]
490 ? S< 0:00 [crypto/0]
491 ? S< 0:00 [crypto/1]
492 ? S< 0:00 [crypto/2]
493 ? S< 0:00 [crypto/3]
690 ? S< 0:00 [scsi_eh_0]
693 ? S< 0:00 [scsi_eh_1]
697 ? S< 0:00 [scsi_eh_2]
700 ? S< 0:00 [scsi_eh_3]
752 ? S< 0:00 [kpsmoused]
759 ? S< 0:00 [kstriped]
762 ? S< 0:00 [kondemand/0]
763 ? S< 0:00 [kondemand/1]
764 ? S< 0:00 [kondemand/2]
765 ? S< 0:00 [kondemand/3]
793 ? S< 0:00 [usbhid_resumer]
816 ? S< 0:00 [rpciod/0]
817 ? S< 0:00 [rpciod/1]
818 ? S< 0:00 [rpciod/2]
819 ? S< 0:00 [rpciod/3]
1462 ? S< 10:55 [md0_raid1]
1496 ? S< 0:34 [kjournald]
1572 ? S<s 0:00 udevd --daemon
1945 pts/2 Sl+ 88:12 ./srcds_i686 -game cstrike +map gg_aim_shotty -maxplayers 20 -ip 85.114.140.30 -port 27045 -tickrate 100
2416 pts/1 Sl+ 202:47 ./srcds_i686 -game cstrike +map scoutzknivez -maxplayers 24 -ip 85.114.140.30 -port 27025 -tickrate 100 -debug
2704 ? Ss 0:00 /sbin/portmap
2717 ? Ss 0:00 /sbin/rpc.statd
2831 ? Ss 0:00 /sbin/mdadm --monitor --pid-file /var/run/mdadm/monitor.pid --daemonise --scan --syslog
2847 ? Ss 0:00 /usr/sbin/famd -T 0
2896 ? Ss 0:00 /usr/sbin/saslauthd -a pam -c -m /var/spool/postfix/var/run/saslauthd -r -n 5
2897 ? S 0:00 /usr/sbin/saslauthd -a pam -c -m /var/spool/postfix/var/run/saslauthd -r -n 5
2898 ? S 0:00 /usr/sbin/saslauthd -a pam -c -m /var/spool/postfix/var/run/saslauthd -r -n 5
2899 ? S 0:00 /usr/sbin/saslauthd -a pam -c -m /var/spool/postfix/var/run/saslauthd -r -n 5
2900 ? S 0:00 /usr/sbin/saslauthd -a pam -c -m /var/spool/postfix/var/run/saslauthd -r -n 5
2921 ? Ss 0:03 amavisd (master)
2939 ? S 0:00 amavisd (virgin child)
2940 ? S 0:00 amavisd (virgin child)
2950 ? Ssl 0:00 /usr/sbin/named -u bind
3162 ? Ss 1:27 /usr/sbin/clamd
3171 ? S 0:00 /usr/sbin/courierlogger -pid=/var/run/courier/authdaemon/pid -start /usr/lib/courier/courier-authlib/authdaemond
3172 ? S 0:00 /usr/lib/courier/courier-authlib/authdaemond
3182 ? S 0:00 /usr/lib/courier/courier-authlib/authdaemond
3183 ? S 0:00 /usr/lib/courier/courier-authlib/authdaemond
3184 ? S 0:00 /usr/lib/courier/courier-authlib/authdaemond
3185 ? S 0:00 /usr/lib/courier/courier-authlib/authdaemond
3186 ? S 0:00 /usr/lib/courier/courier-authlib/authdaemond
3200 ? Ss 0:02 /usr/sbin/cron
3210 ? Ss 0:00 /usr/bin/dbus-daemon --system
3226 ? Sl 0:55 /usr/bin/python /usr/bin/fail2ban-server -b -s /var/run/fail2ban/fail2ban.sock
3247 ? Ss 0:01 /usr/sbin/ntpd -p /var/run/ntpd.pid -u 103:105 -g
3268 ? S 0:02 /usr/lib/postgresql/8.3/bin/postgres -D /var/lib/postgresql/8.3/main -c config_file=/etc/postgresql/8.3/main/postgresql.conf
3270 ? Ss 0:42 postgres: writer process
3271 ? Ss 0:26 postgres: wal writer process
3272 ? Ss 0:03 postgres: autovacuum launcher process
3273 ? Ss 0:04 postgres: stats collector process
3291 ? Ss 0:00 /usr/sbin/postgrey --pidfile=/var/run/postgrey.pid --daemonize --inet=127.0.0.1:60000
3300 ? Ss 0:30 /usr/sbin/spamd --create-prefs --max-children 5 --helper-home-dir -d --pidfile=/var/run/spamd.pid
3306 ? Ss 0:05 /usr/sbin/sshd
3320 ? S 0:00 /usr/sbin/vsftpd
3358 ? S 0:00 spamd child
3359 ? S 0:00 spamd child
3538 ? Ss 0:02 /usr/bin/freshclam -d --quiet
3549 ? Ss 0:01 /usr/sbin/hald
3550 ? S 0:00 hald-runner
3570 ? S 0:00 hald-addon-input: Listening on /dev/input/event1 /dev/input/event0
3575 ? S 0:00 /usr/lib/hal/hald-addon-cpufreq
3576 ? S 0:00 hald-addon-acpi: listening on acpi kernel interface /proc/acpi/event
3656 ? Ss 0:04 /usr/lib/postfix/master
3669 ? S 0:05 qmgr -l -t fifo -u
3671 ? Ss 163:09 /usr/bin/gkrellmd --pidfile /var/run/gkrellmd.pid
3823 ? Ss 0:05 /usr/sbin/dovecot
3831 ? S 0:08 dovecot-auth
3854 ? S 0:03 imap-login
3855 ? S 0:03 imap-login
3856 ? S 0:03 imap-login
3858 ? Ss 0:03 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf
3860 tty1 Ss+ 0:00 /sbin/getty 38400 tty1
3861 tty2 Ss+ 0:00 /sbin/getty 38400 tty2
3862 tty3 Ss+ 0:00 /sbin/getty 38400 tty3
3863 tty4 Ss+ 0:00 /sbin/getty 38400 tty4
3864 tty5 Ss+ 0:00 /sbin/getty 38400 tty5
3865 tty6 Ss+ 0:00 /sbin/getty 38400 tty6
7426 ? Ss 0:08 /opt/lampp/bin/httpd -k start -DSSL -DPHP5 -E /opt/lampp/logs/error_log
7445 ? S 0:00 /bin/sh /opt/lampp/bin/mysqld_safe --datadir=/opt/lampp/var/mysql --pid-file=/opt/lampp/var/mysql/box.elite-hunterz.info.pid
7572 ? Sl 21:43 /opt/lampp/sbin/mysqld --basedir=/opt/lampp --datadir=/opt/lampp/var/mysql --user=nobody --log-error=/opt/lampp/var/mysql/box.elite-hunterz.info.err --pid-file=/opt/lampp/var/mysql
10740 ? Ss 0:00 SCREEN -A -m -d -S css-server ./srcds_run -game cstrike +map gg_aim_shotty -maxplayers 20 -ip 85.114.140.30 -port 27045 -tickrate 100
10741 pts/2 Ss+ 0:00 /bin/sh ./srcds_run -game cstrike +map gg_aim_shotty -maxplayers 20 -ip 85.114.140.30 -port 27045 -tickrate 100
11238 ? Ss 0:03 ./sbnc
11239 ? S 0:04 ./sbnc --rpc-child
11604 ? S 0:31 ./eggdrop niko.conf
11956 pts/4 Sl+ 67:47 ./srcds_i686 -game cstrike +map glass_war -maxplayers 24 -ip 85.114.140.30 -port 27055 -tickrate 66 -autoupdate -debug
14403 ? Ss 0:00 SCREEN -A -m -d -S css-server ./srcds_run -game cstrike +map de_dust2 -maxplayers 12 -tickrate 100 -ip 85.114.140.30 -port 27035 -debug
14404 pts/3 Ss+ 0:00 /bin/sh ./srcds_run -game cstrike +map de_dust2 -maxplayers 12 -tickrate 100 -ip 85.114.140.30 -port 27035 -debug
14418 pts/3 Sl+ 24:14 ./srcds_i686 -game cstrike +map de_dust2 -maxplayers 12 -tickrate 100 -ip 85.114.140.30 -port 27035 -debug
14761 ? S 0:00 pop3-login
14769 ? S 0:01 pop3-login
14780 ? S 0:00 pop3-login
14781 ? S 0:00 pop3-login
14782 ? S 0:00 pop3-login
14783 ? S 0:00 pop3-login
14784 ? S 0:00 pop3-login
14785 ? S 0:01 pop3-login
14786 ? S 0:00 pop3-login
14787 ? S 0:01 pop3-login
14788 ? S 0:01 pop3-login
14789 ? S 0:00 pop3-login
14790 ? S 0:00 pop3-login
14791 ? S 0:00 pop3-login
14792 ? S 0:00 pop3-login
14793 ? S 0:00 pop3-login
14794 ? S 0:00 pop3-login
14796 ? S 0:00 pop3-login
14799 ? S 0:01 pop3-login
14800 ? S 0:01 pop3-login
15012 ? S 0:11 ./eggdrop eggdrop.conf
15882 ? Ss 0:00 SCREEN -A -m -d -S css-server ./srcds_run -game cstrike +map scoutzknivez -maxplayers 24 -ip 85.114.140.30 -port 27025 -tickrate 100 -debug
15883 pts/1 Ss+ 0:00 /bin/sh ./srcds_run -game cstrike +map scoutzknivez -maxplayers 24 -ip 85.114.140.30 -port 27025 -tickrate 100 -debug
20432 ? Ss 0:00 sshd: root@pts/5
20443 pts/5 Ss 0:00 -bash
20500 ? S 0:00 /opt/lampp/bin/httpd -k start -DSSL -DPHP5 -E /opt/lampp/logs/error_log
20957 pts/0 Sl+ 26:57 ./srcds_i686 -game cstrike +map zm_cbble_b3 -maxplayers 50 -ip 85.114.140.30 -port 27015 -tickrate 66 -debug
21248 ? Ss 0:01 SCREEN -A -m -d -S css-server ./srcds_run -game cstrike +map zm_cbble_b3 -maxplayers 50 -ip 85.114.140.30 -port 27015 -tickrate 66 -debug
21249 pts/0 Ss+ 0:00 /bin/sh ./srcds_run -game cstrike +map zm_cbble_b3 -maxplayers 50 -ip 85.114.140.30 -port 27015 -tickrate 66 -debug
21740 ? S 0:00 cleanup -z -t unix -u -c
22074 ? S 0:07 /usr/bin/perl ./hlstats.pl --configfile=hlstats.conf
22582 ? S 0:00 pickup -l -t fifo -u -c
22673 ? S 0:00 /opt/lampp/bin/httpd -k start -DSSL -DPHP5 -E /opt/lampp/logs/error_log
22692 ? S 0:00 /opt/lampp/bin/httpd -k start -DSSL -DPHP5 -E /opt/lampp/logs/error_log
22729 ? S 0:00 /opt/lampp/bin/httpd -k start -DSSL -DPHP5 -E /opt/lampp/logs/error_log
22731 ? S 0:00 /opt/lampp/bin/httpd -k start -DSSL -DPHP5 -E /opt/lampp/logs/error_log
22737 ? S 0:00 /opt/lampp/bin/httpd -k start -DSSL -DPHP5 -E /opt/lampp/logs/error_log
22781 ? Ss 0:00 SCREEN -A -m -d -S css-server ./srcds_run -game cstrike +map zm_cbble_b3 -maxplayers 10 -ip 85.114.140.30 -port 27065 -tickrate 66 -sv_password eh -autpupdate -debug debug
22782 pts/6 Ss+ 0:00 /bin/sh ./srcds_run -game cstrike +map zm_cbble_b3 -maxplayers 10 -ip 85.114.140.30 -port 27065 -tickrate 66 -sv_password eh -autpupdate -debug debug
22796 pts/6 Sl+ 39:45 ./srcds_i686 -game cstrike +map zm_cbble_b3 -maxplayers 10 -ip 85.114.140.30 -port 27065 -tickrate 66 -sv_password eh -autpupdate -debug debug
22822 ? S 0:00 /opt/lampp/bin/httpd -k start -DSSL -DPHP5 -E /opt/lampp/logs/error_log
22823 ? S 0:00 /opt/lampp/bin/httpd -k start -DSSL -DPHP5 -E /opt/lampp/logs/error_log
22826 ? S 0:00 /opt/lampp/bin/httpd -k start -DSSL -DPHP5 -E /opt/lampp/logs/error_log
22829 ? S 0:00 /opt/lampp/bin/httpd -k start -DSSL -DPHP5 -E /opt/lampp/logs/error_log
22831 ? S 0:00 /opt/lampp/bin/httpd -k start -DSSL -DPHP5 -E /opt/lampp/logs/error_log
22912 ? S 0:00 local -t unix
23436 pts/5 R+ 0:00 ps ax
24619 ? Ss 0:00 SCREEN -A -m -d -S css-server ./srcds_run -game cstrike +map glass_war -maxplayers 24 -ip 85.114.140.30 -port 27055 -tickrate 66 -autoupdate -debug
24620 pts/4 Ss+ 0:00 /bin/sh ./srcds_run -game cstrike +map glass_war -maxplayers 24 -ip 85.114.140.30 -port 27055 -tickrate 66 -autoupdate -debug
25097 ? S 0:01 [pdflush]

ps: my mailserver never worked

rweaver · 04-02-2010, 02:02 PM

Run rkhunter and chkrootkit, doesn't catch it all but should give you an idea. What do you get with a 'df -k' and 'du -sk /* | sort -n'? Is all your software upto date including the web apps? Have you been updating the system via the package manager?

nikooo777 · 04-02-2010, 02:33 PM

hello, so:

Quote:

box:/opt/lampp# df -k
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/md0 960390224 45506804 866098440 5% /
tmpfs 3112020 0 3112020 0% /lib/init/rw
udev 10240 684 9556 7% /dev
tmpfs 3112020 4 3112016 1% /dev/shm

Quote:

box:/opt/lampp# du -sk /* | sort -n
du: cannot access `/proc/24347/task/24347/fd/4': No such file or directory
du: cannot access `/proc/24347/task/24347/fdinfo/4': No such file or directory
du: cannot access `/proc/24347/fd/4': No such file or directory
du: cannot access `/proc/24347/fdinfo/4': No such file or directory
0 /initrd.img
0 /initrd.img.old
0 /proc
0 /selinux
0 /sys
0 /vmlinuz
0 /vmlinuz.old
4 /media
4 /mnt
4 /srv
4 /ssl
4 /webmin-setup.out
16 /lost+found
124 /root
392 /tmp
688 /dev
4104 /sbin
4276 /bin
7920 /xmail
26108 /boot
67064 /lib
121872 /etc
179112 /stats
392796 /opt
1025840 /usr
2902532 /var
9141612 /home

after running rkhunter -c see what happened:
915.90 GB total, 89.92 GB used

i have no idea what i did.
my server is up to date and stable.

E:

the rkhunter gave me those warnings but i dont know what i have to do:

Quote:

[19:41:15] /usr/bin/dpkg [ Warning ]
[19:41:15] Warning: The file properties have changed:
[19:41:15] File: /usr/bin/dpkg
[19:41:15] Current hash: 91c9011cabf1e27516471eec0859d46562eca240
[19:41:15] Stored hash : a7ee4491d8ce6d3b199fd8406e042a966ab391c5
[19:41:15] Current inode: 6389972 Stored inode: 45114100
[19:41:15] Current file modification time: 1268080529
[19:41:15] Stored file modification time : 1263296576

[19:41:15] /usr/bin/dpkg-query [ Warning ]
[19:41:15] Warning: The file properties have changed:
[19:41:15] File: /usr/bin/dpkg-query
[19:41:15] Current inode: 6389970 Stored inode: 45114098
[19:41:15] Current file modification time: 1268080529
[19:41:15] Stored file modification time : 1263296576

[19:41:16] /usr/bin/sudo [ Warning ]

[19:41:17] /sbin/chkconfig [ Warning ]

[19:41:19] /usr/sbin/inetd [ Warning ]

[19:41:19] /usr/sbin/rsyslogd [ Warning ]

[19:41:19] /usr/sbin/unhide [ Warning ]

[19:41:19] /usr/sbin/unhide-linux26 [ Warning ]
[19:41:19] Warning: The file '/usr/sbin/unhide-linux26' exists on the system, but it is not present in the rkhunter.dat file.

[19:28:58] Checking kernel module commands [ Warning ]
[19:28:58] Warning: No output found from the lsmod command or the /proc/modules file:
[19:28:58] /proc/modules output:
[19:28:58] lsmod output:
[19:28:58] Info: Using modules pathname of '/lib/modules/2.6.31.6-myloc'

[19:28:58] Checking for TCP port 6667 [ Warning ]
[19:28:58] Warning: Network TCP port 6667 is being used by /home/server2/srcds_2/srcds_i686. Possible rootkit: Possible rogue IRC bot
Use the 'lsof -i' or 'netstat -an' command to check this.
(this one should be ok. i have an irc relay installed on every gameserver. only thing i dont get is why it reports only server2)

[19:29:00] Checking if SSH root access is allowed [ Warning ]
[19:29:00] Warning: The SSH and rkhunter configuration options should be the same:
[19:29:00] SSH configuration option 'PermitRootLogin': yes
[19:29:00] Rkhunter configuration option 'ALLOW_SSH_ROOT_USER': no

[19:29:00] Checking if SSH protocol v1 is allowed [ Not allowed ] (i think it's ok this one right?)

[19:29:00] Checking for running syslog daemon [ Warning ]
[19:29:00] Warning: The syslog daemon is not running.
(i have syslog-ng)

[19:29:01] Checking version of GnuPG [ Warning ]
[19:29:01] Warning: Application 'gpg', version '1.4.9', is out of date, and possibly a security risk.

[19:29:01] Checking version of Apache [ Warning ]
[19:29:01] Warning: Application 'httpd', version '2.2.9', is out of date, and possibly a security risk.

[19:29:01] Checking version of Bind DNS [ Warning ]
[19:29:01] Warning: Application 'named', version '9.5.1', is out of date, and possibly a security risk.

[19:29:01] Checking version of OpenSSL [ Warning ]
[19:29:01] Warning: Application 'openssl', version '0.9.8g', is out of date, and possibly a security risk.

[19:29:01] Checking version of OpenSSH [ Warning ]
[19:29:01] Warning: Application 'sshd', version '5.1p1', is out of date, and possibly a security risk.

btw i update my system with webmin.