LinuxQuestions.org
Page 1 of 2 1 2
Show 50 post(s) from this thread on one page

LinuxQuestions.org (/questions/)
-   Linux - Server (https://www.linuxquestions.org/questions/linux-server-73/)
-   -   VSFTPD Max CPU - Persistent problem (https://www.linuxquestions.org/questions/linux-server-73/vsftpd-max-cpu-persistent-problem-856454/)

MisterTickle 01-15-2011 05:16 AM
VSFTPD Max CPU - Persistent problem
 
I have been battling an issue with my Gameservers company for 3 weeks to no avail to anything I've tried. Every now and than around the timeframe of 11:30AM to 4PM (this is peak time) multiple VSFTPD processes start up and use max CPU which lags out the box for my clients and makes it impossible for me to access Putty to stop it. I can stop it but, If I start it back up too soon the process starts all over again. This is hurting my business alot and my customers aswell as me are very stressed. The box runs fine other than that. Doesn't really go over 70% CPU usage, ram is low and as long as FTP is off the server runs fine but, than my clients cannot update their files which becomes quite an issue. I have also all of a sudden had an issue with remote locations not being able to access our My SQL databases. I have googled for help on all of these issues and have had none. I really would like to get this sorted today and put it behind me so I can grow my business and keep the clients I already have because, every time I think I fixed it I let my clients know the issues has now been resolved and it never is. I would appreciate any help and hope I posted in the right section.

I have started a thread at WHT I am also seeking help but, I really need this problem resolved so I am seeking alternative sources. (http://www.webhostingtalk.com/showthread.php?t=1014062) - Hope I was allowed to post that if not feel free to remove it it was just to refer to any progress I've made and input of others.

unSpawn 01-15-2011 05:56 AM
Since you're new to Linux you should know some responses are less than perfect. For instance this person writing "You might wanna consider changing to ProFTPD. It's much more lightweight. I'm not sure how it's now, but earlier vsftpd was kinda easy 'exploitable'." is completely clueless as it's exactly the other way around (as this shows). Unless you have some gut feeling your machine is compromised or unless there's evidence of that running Chkrootkit or Rootkit Hunter won't "fix" things.


A multi-layered approach to make FTP traffic less resource-intensive and more robust by:
0) ensuring best Netfilter rule-parsing performance by putting state ESTABLISHED,RELATED traffic before NEW,
1) not mucking around with /proc/sys/net/ip*/*mem* sysctls unless you are certain they do work for you,
2) dropping "bad" traffic using fail2ban,
3) rate-limiting traffic in VsFTPd itself: see the VsFTPd FAQ "Help! Does vsftpd support bandwidth limiting?" entry (also see maximum sessions, maximum concurrent connections per IP and other performance settings and the hints in the TUNING document),
4) using "iptables -m recent" rules as posted there,
5) bandwidth-limiting traffic using iproute ('man tc') (or search for "LARTC wondershaper" as example)
would be best I think.

* This is not a Linux Security issue so I'll be moving this to the Networking forum shortly.

MisterTickle 01-15-2011 07:18 AM
What would be your recommended configurations for fail2ban? I have it running atm actually I installed it a while back in hopes it would fix the issue but, it never did. I still don't understand why 20 processes tend to start up randomly and use up ALL the CPU. It isn't like the box is under too much stress with FTP. Its been under more stress in the past and the problem seems to always happen between 11:30 and 4:00 PM I have limited the connection to 1 per person.

MisterTickle 01-15-2011 07:51 AM
Just did a bunch of the changes and there it goes... again max cpu... again... upset customers again.

Noticed This:


Jan 15 15:08:37 rs238 kernel: possible SYN flooding on port 80. Sending cookies.
Jan 15 15:16:43 rs238 kernel: samp03svr[22945]: segfault at 0000000050f6f962 rip 00000000f7e38c63 rsp 00000000ff8eaea0 error 4
Jan 15 15:32:52 rs238 kernel: possible SYN flooding on port 80. Sending cookies.
Jan 15 15:33:52 rs238 kernel: possible SYN flooding on port 80. Sending cookies.
Jan 15 15:35:52 rs238 last message repeated 2 times
Jan 15 15:36:53 rs238 kernel: possible SYN flooding on port 80. Sending cookies.
Jan 15 15:38:59 rs238 yum: Installed: fail2ban-0.8.2-3.el5.rf.noarch

Oh and btw sendmail seems to be using 37.7% of the CPU or some reason. I've never seen it do that.

unSpawn 01-15-2011 08:52 AM
You seem to have more than one issue and things segfaulting on you isn't good. I don't really like to trace multiple issues all at once but it seems it's unavoidable.
- Please post detailed hardware specs of this machine.
- Post what distribution and release and what software the machine runs and what your web server serves (exactly which forum or tracker or game server software)?
- Post a detailed list of all the issues you have encountered so far so we don't miss information.
- Do attach as plain text the output files in /tmp from running as root:
- package validation (this might take a while to finish): 'rpm -Vva 2>&1 | grep -v '^\.\{8\}' > /tmp/rpm.txt &;'
- effective iptables rule set: '/sbin/iptables-save > /tmp/iptables.txt;'
- locate your vsftpd.conf and substitute "/path/to/": 'grep -v ^# /path/to/vsftpd.conf | grep . > /tmp/vsftpd.txt;'
- locate your httpd.conf and substitute "/path/to/": 'grep -v ^# /path/to/httpd.conf | grep . > /tmp/httpd.txt;'
- connection list: netstat -antlpe 2>&1 | sed -e "s|[0-9]\{1,3\}\.|n.|g" > /tmp/netstat.txt;'
- process list: '/bin/ps axfwwwe 2>&1 > /tmp/ps.txt;'
- process top: 'top -bn1 2>&1 > /tmp/top.txt;'
- kernel tweaks: '/sbin/sysctl -a | egrep -vie "(host|random|cdrom|fs\.|parp)" > /tmp/sysctl.txt;'.

...or as one-liner:
Code:
(rpm -Vva 2>&1|grep -v '^\.\{8\}'; /sbin/iptables-save; grep -v ^# /etc/vsftpd/vsftpd.conf|grep .; grep -v ^# /etc/httpd/conf/httpd.conf | grep .; netstat -antlpe 2>&1|sed -e "s|[0-9]\{1,3\}\.|n.|g"; /bin/ps axfwwwe 2>&1; top -bn1 2>&1; /sbin/sysctl -a | egrep -vie "(host|random|cdrom|fs\.|parp)"; who; last; lastlog; ) > /tmp/output.txt
.

* Note I only anonymized netstat output. Check all logs before you post and run 'sed -i "s|[0-9]\{1,3\}\.|n.|g" /path/to/logfile;' to anonymize them as necessary. Please do not delete lines you can not determine as irrelevant. Please attach a Logwatch report ('logwatch --detail High --range All --archives --numeric --output unformatted --save /tmp/logwatch.txt') if you run Logwatch.

Once we have your output we'll assess health of the box, limit traffic and processes and take it from there.

MisterTickle 01-15-2011 12:35 PM
1 Attachment(s)
Issue 1: Randomly in the middle of the day my clients will report that the servers are acting slow, It will take me some time to get into Putty when I finally do get in it will show either 1 VSFTPD process using 100% CPU or a few all using substantial amounts - This usually happens everyday between 11:30 AM and 4:00 PM and a restart won't make it go away as it will come right back. I need to end it for a while and turn it back on. Have never had it happen in the evening. No CPU problems from general use just the random spikes and its not gradual raise I mean its sudden the second it happens it goes all out and just lags out all my clients gameservers.

Issue 2: I think I recieved some sort of ddos/dos attack this morning I had similar issues to the FTP issue but, I couldn't get into putty at all I ha d to restart the box and when I started it back up there were tons of httpd processes running using quite a bit of CPU I had to end the httpd service to get rid of them.

Issue 3: Clients can't connect to their mysql databases remotely anymore I think I may have caused this when installing CSF Firewall (Which is not running atm btw as it caused issues with the connection between my webserver and dedicated server)

unSpawn 01-15-2011 07:15 PM
Just a quick summary of the log you posted (thanks):
- no iptables (-t filter) rules AT ALL meaning this server runs completely unprotected,
- VsFTPd running with local user access (OK, through PAM) but without any bandwidth, concurrent connection or session limiting,
- HTTPd with all modules loaded including proxy,
- RPC and MySQL listening on eth0 (instead of localhost only),
- 31 SMTP processes on TCP/25 (check /var/log/maillog),
- users server1114, server1129, server444, server578, server526 using 11 processes on TCP/6667 (IRC),
- user server93 running GTA server on port TCP/5554,
- user "toord" has root rights,
- user "toord" has logged in as root,
- superfluous services running (RPC, Avahi, Bluetooth, CUPS),
- CSF component "lfd" still running.

Based on this the first course of action would be to limit inbound traffic and log denied traffic. Iptables rule set to load as root user:
Code:
#!/bin/bash --
# Flush all rules
iptables -F; iptables -X
# Setting default filter policy
iptables -P INPUT DROP; iptables -P FORWARD DROP; iptables -P OUTPUT ACCEPT

# Allow unlimited traffic on loopback
iptables -A INPUT -i lo -j ACCEPT; iptables -A OUTPUT -o lo -j ACCEPT

# Allow inbound existing connections first as this is the majority of traffic
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

# Allow FTP and FTP data
iptables -A INPUT -m state --state NEW -m tcp -p tcp -m multiport --dports 20,21,22 -j ACCEPT

# Allow SSH
iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT

# Allow HTTP and flood-limit access to max 30 requests per minute
iptables -A INPUT -p tcp -m state --state NEW -m tcp --dport 80 -m limit --limit 10/minute --limit-burst 30 -j ACCEPT

# Allow Grand Theft Auto (guessed port assignment)
iptables -A INPUT -m state --state NEW -m tcp -p tcp -m multiport --dports 5554,8888 -j ACCEPT
iptables -A INPUT -m state --state NEW -m tcp -p udp --dport 7777 -j ACCEPT

# Drop ICMP frags but allow some ICMP diagnostic messages:
iptables -A INPUT -p icmp --fragment -j DROP
for ICMPTYPE in echo-reply time-exceeded fragmentation-needed; do
iptables -A INPUT -p icmp --icmp-type ${ICMPTYPE} -j ACCEPT; done

# Catchall rules to log and reject anything else (dropping remote RPC, SMTP, MySQL):
iptables -A INPUT -m state --state NEW -m limit --limit 10/minute --limit-burst 30 -j LOG --log-prefix "INREJ "
iptables -A INPUT -j REJECT --reject-with icmp-host-prohibited

# Log outbound traffic initiation
iptables -A OUTPUT -m state --state NEW -m limit --limit 10/minute --limit-burst 30 -j LOG --log-prefix "OUTNEW "

# End
exit 0
As said the above will allow FTP, FTP data, SSH, HTTP and GTA. Do realize port assignments are taken from what you posted so there might be some breakage though this should not disrupt services you need like SSH. Remote RPC, SMTP and MySQL are dropped. Note you'll have to 'tail -f /var/log/messages|grep "INREJ";' to see any denied inbound traffic and adjust your iptables rules accordingly. We'll fix the rest later on.

In case of breakage set a safeguard beforehand. As root run: 'service atd start && echo "service iptables restart"|/usr/bin/at "now + 5 minutes";'. This will restore your default iptables rule set after 5 minutes after which you can troubleshoot any problems. To load the above rule set just save to file and execute the script. Don't forget to rerun the script after 5 minutes if you can still access the machine because of the safeguard and restart VsFTPd and Sendmail.

syg00 01-15-2011 08:10 PM
Nice, @unSpawn
I must take some time to read through this in more detail.

MisterTickle 01-16-2011 03:02 AM
GTA San Andreas runs on a multitude of ports. The clients select which port they want for there server between 4500 - 21474 so I think I'll need a custom ruleset for that.

Also I believe those servers to be using IRC as there is an IRC plugin that some servers use.

btw Toord is an account my partner made which was basically supposed to have root without using root I guess I'm not 100% sure. I'm also afraid to flush the IPTables as there is one rule to make the webserver work and if it isn't active than they disconnect from each other.

iptables -A INPUT -s 96.30.60.204 -p tcp --destination-port 25 -j DROP

Thats the rule above.

I have also attached maillog the sendmail process has been using quite a bit of CPU lately and the log looked concerning thought you should take a look.

I limited the MAX CLIENTS to 8 and MAX CONNECTIONS to 11 for VSFTPD.

BTW I apperciate all the help.

Edit: maillog was too big but, I uploaded it here - http://www.2shared.com/file/qZRar9ou/maillog.html

I also forgot to give you the specs..

Intel(R) Core(TM) i3 530 @ 2x 2.93GHz + H/T
4GB DDR3 Memory
160 SATAII
Synergy upstream

unSpawn 01-16-2011 06:27 AM
Quote:
Originally Posted by MisterTickle (Post 4225846)
GTA San Andreas runs on a multitude of ports. The clients select which port they want for there server between 4500 - 21474 so I think I'll need a custom ruleset for that.
Use a port range then:
Code:
iptables -A INPUT -m state --state NEW -m tcp -p tcp -m multiport --dports 4500:21474 -j ACCEPT

Quote:
Originally Posted by MisterTickle (Post 4225846)
btw Toord is an account my partner made which was basically supposed to have root without using root I guess I'm not 100% sure.
That's OK for the time being and as long as only you can use it.


Quote:
Originally Posted by MisterTickle (Post 4225846)
I'm also afraid to flush the IPTables as there is one rule to make the web server work and if it isn't active than they disconnect from each other.
That's why the safeguard you should set up kicks in after five minutes: to test and see if everything works. Still you've got to watch /var/log/messages to ensure all is OK. No way around that. But since it's a server we could turn the rule set around and go from default DROP to default ACCEPT if that makes you happy. I'm keeping accept rules to allow for limiting though and I still add a catchall reject rule for any traffic that doesn't match any of the filters:

Code:
#!/bin/bash --
# Flush all rules
iptables -F; iptables -X
# Setting default filter policy
iptables -P INPUT ACCEPT; iptables -P FORWARD DROP; iptables -P OUTPUT ACCEPT

# Allow unlimited traffic on loopback
iptables -A INPUT -i lo -j ACCEPT; iptables -A OUTPUT -o lo -j ACCEPT

# Allow inbound existing connections first as this is the majority of traffic
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

# Allow SSH: ideally this should be limited to a set of IP addresses or ranges or be dealt with by fail2ban
iptables -A INPUT -m state --state NEW -m tcp -p tcp -m multiport --dports 22 -j ACCEPT

# Allow Grand Theft Auto (guessed port assignment)
iptables -A INPUT -m state --state NEW -m tcp -p tcp -m multiport --dports 4500:21474 -j ACCEPT
iptables -A INPUT -m state --state NEW -m tcp -p udp --dport 7777 -j ACCEPT

# Allow FTP and FTP data and flood-limit access to max 30 requests per minute
iptables -A INPUT -m state --state NEW -m tcp -p tcp -m multiport --dports 20,21 -m limit --limit 10/minute --limit-burst 30 -j ACCEPT

# Allow HTTP and flood-limit access to max 30 requests per minute
iptables -A INPUT -p tcp -m state --state NEW -m tcp --dport 80 -m limit --limit 10/minute --limit-burst 30 -j ACCEPT

# Drop ICMP frags but allow some ICMP diagnostic messages:
iptables -A INPUT -p icmp --fragment -j DROP
for ICMPTYPE in echo-reply time-exceeded fragmentation-needed; do
iptables -A INPUT -p icmp --icmp-type ${ICMPTYPE} -j ACCEPT; done

# Catchall rules to log and reject anything else (dropping remote RPC, SMTP, MySQL):
iptables -A INPUT -m state --state NEW -m limit --limit 10/minute --limit-burst 30 -j LOG --log-prefix "INREJ "
iptables -A INPUT -j REJECT --reject-with icmp-host-prohibited

# Log outbound traffic initiation
iptables -A OUTPUT -m state --state NEW -m limit --limit 10/minute --limit-burst 30 -j LOG --log-prefix "OUTNEW "

# End
exit 0
As an example you could also choose to drop only traffic to ports you don't want to be reached over the 'net but in this example there's no limiting:
Code:
#!/bin/bash --
# Flush all rules
iptables -F; iptables -X
# Setting default filter policy
iptables -P INPUT ACCEPT; iptables -P FORWARD DROP; iptables -P OUTPUT ACCEPT
# Own IP address
EXTERNAL=$(hostname -i 2>/dev/null|| { echo "IP lookup error, exiting."; exit 1; })

# Allow unlimited traffic on loopback
iptables -A INPUT -i lo -j ACCEPT; iptables -A OUTPUT -o lo -j ACCEPT

# Allow inbound existing connections first as this is the majority of traffic
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

# Allow MySQL TCP for samp03svr until you figure out how to make samp03svr use local UNIX socket
iptables -A INPUT -m state --state NEW -m tcp -p tcp -s ${EXTERNAL}/32 -d ${EXTERNAL}/32 --dport 3306 -j ACCEPT
# Drop specific services only
iptables -A INPUT -m state --state NEW -m tcp -p tcp -m multiport --dports 111,730,3306 -j LOG --log-prefix "INREJ "
iptables -A INPUT -m state --state NEW -m tcp -p tcp -m multiport --dports 111,730,3306 -j REJECT --reject-with icmp-host-prohibited
# Remove this rule if GTA doesn't use it:
iptables -A INPUT -m state --state NEW -m tcp -p udp ! --dport 7777 -j LOG --log-prefix "INREJ "
iptables -A INPUT -m state --state NEW -m tcp -p udp ! --dport 7777 -j REJECT --reject-with icmp-host-prohibited

# Drop ICMP frags but allow some ICMP diagnostic messages:
iptables -A INPUT -p icmp --fragment -j DROP
for ICMPTYPE in echo-reply time-exceeded fragmentation-needed; do
iptables -A INPUT -p icmp --icmp-type ${ICMPTYPE} -j ACCEPT; done

# End
exit 0
Finally you couldload rules manually but the point is you have to do something.


Quote:
Originally Posted by MisterTickle (Post 4225846)
iptables -A INPUT -s 96.30.60.204 -p tcp --destination-port 25 -j DROP
What?


Quote:
Originally Posted by MisterTickle (Post 4225846)
Also I believe those servers to be using IRC as there is an IRC plugin that some servers use.
OK. Since they don't run IRC daemons on your machine there's no rule needed.

MisterTickle 01-16-2011 06:32 AM
I updated the post when you we're replying. You might want to read it over and when I removed iptables -A INPUT -s 96.30.60.204 -p tcp --destination-port 25 -j DROP from the IP tables by accident it disconnected the web server from accessing the dedicated server and I had to readd that rule.

What should I save the file as and what do I need to type to run it? (I probably sound like a total Linux noob but truly, This wasn't initially what I was supposed to manage my partner left)

And you know something I noticed the VSFTPD process issue the processes are always root or nobody yet if a client with an account is using it will show their username/server name listed in top so it can't be a client doing something directly I wouldn't think. Maybe I can block VSFTPD processes on root since I only use FTP on toord or as a client with a user account.

Edit: I added serlist_deny=YES to my cfg which is supposed to block


root
bin
daemon
adm
lp
sync
shutdown
halt
mail
news
uucp
operator
games
nobody

from running VSFTPD processes. Is there a way to check if the block is working though?

unSpawn 01-16-2011 09:18 AM
Quote:
Originally Posted by MisterTickle (Post 4225846)
I have also attached maillog the sendmail process has been using quite a bit of CPU lately and the log looked concerning thought you should take a look.
Your maillog shows a few things:
- very little email sent by Apache, fail2ban and regular user accounts,
- evidence of a process (probably other than Sendmail) causing ludicrous load problems, highest I saw was 1115 IIRC, making Sendmail halt its own processing until load drops,
- a lot of mail sent by root. If these are warnings sent by email as a convenience then I would disable most and find another way to get alerted. Check your root mailbox and tell us.
Other than that I don't see any problems: very few messages show delivery problem status like "Connection refused" or "Service unavailable".


Quote:
Originally Posted by MisterTickle (Post 4225846)
I limited the MAX CLIENTS to 8 and MAX CONNECTIONS to 11 for VSFTPD.
So your complete conf should look like this now:
Code:
anonymous_enable=NO
local_enable=YES
write_enable=YES
local_umask=003
file_open_mode=0777
dirmessage_enable=YES
xferlog_enable=YES
connect_from_port_20=YES
xferlog_std_format=YES
listen=YES
pam_service_name=vsftpd
tcp_wrappers=YES
passwd_chroot_enable=YES
chroot_local_user=YES
force_dot_files=YES
hide_ids=YES
userlist_deny=YES
# Confine PASV mode to a higher range (easier for firewalling later on)
pasv_min_port=50000
pasv_max_port=60000
# Drop idle sessions after 1 minute and data after 2 minutes
idle_session_timeout=60
data_connection_timeout=120
# Allow a maximum of 10 clients to connect
max_clients=10
# Allow a maximum of 2 sessions per IP address
max_per_ip=2
# Set a 128kbyte/sec rate for authenticated users
local_max_rate=128000
Note I added some with comments. After restarting VsFTPd fail2ban should pick up malicious activity and act on it if you enabled checking FTP in /etc/fail2ban/jail.conf.


Quote:
Originally Posted by MisterTickle (Post 4225846)
BTW I apperciate all the help.
That's OK. It's what we're here for.


Quote:
Originally Posted by MisterTickle (Post 4226018)
I updated the post when you we're replying.
Then next time please mark those changes if I need to respond to them or best create a new post. Let's avoid anything that could lead to confusion.


Quote:
Originally Posted by MisterTickle (Post 4226018)
when I removed iptables -A INPUT -s 96.30.60.204 -p tcp --destination-port 25 -j DROP from the IP tables by accident it disconnected the web server from accessing the dedicated server and I had to readd that rule.
As far as I can see you didn't have any iptables rules loaded from the default location and I didn't anticipate that. What does
Code:
for TABLE in $(cat /proc/net/ip_tables_names); do /sbin/iptables -t ${TABLE} -n -v -x -L --line-numbers; done
return?


Quote:
Originally Posted by MisterTickle (Post 4226018)
What should I save the file as and what do I need to type to run it?
Lets first see if you have any rules loaded.


Quote:
Originally Posted by MisterTickle (Post 4226018)
And you know something I noticed the VSFTPD process issue the processes are always root or nobody yet if a client with an account is using it will show their username/server name listed in top so it can't be a client doing something directly I wouldn't think. Maybe I can block VSFTPD processes on root since I only use FTP on toord or as a client with a user account. Edit: I added serlist_deny=YES to my cfg which is supposed to block root bin daemon adm lp sync shutdown halt mail news uucp operator games nobody from running VSFTPD processes. Is there a way to check if the block is working though?
VsFTPd first process starts as root since it needs to bind to a port below 1024. The nobody processes may be the FTP user entrusted with serving files. Blocking system accounts from accessing FTP is good but as your VsFTP configuration uses PAM and most of the system accounts listed aren't allowed to log in anyway you would see warnings in /var/log/secure.


Quote:
Originally Posted by MisterTickle (Post 4226018)
echo: write error: Broken pipe when I try to run safeguard thing you gave me.
Then the 'at' service (/etc/init.d/atd) might not be installed. Run the command instead in the background as 'sleep 5m && service iptables stop &'. Notice the "stop" as you have no default rules loading from the default location, so this would reset all policies back to ACCEPT and load no rules.

MisterTickle 01-16-2011 09:24 AM
I fixed the broken pipe issue. I type the command in wrong.

Output:

Chain INPUT (policy ACCEPT 15914022 packets, 1334023601 bytes)
num pkts bytes target prot opt in out source destination
1 0 0 fail2ban-VSFTPD tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:21
2 1 39 DROP udp -- * * 77.46.0.0/16 2 13.5.176.155 udp dpt:7782
3 17 679 DROP udp -- * * 178.223.0.0/16 2 13.5.176.155 udp dpt:7782
4 0 0 DROP udp -- * * 93.87.0.0/16 2 13.5.176.155 udp dpt:7782
5 3 117 DROP udp -- * * 109.93.0.0/16 2 13.5.176.155 udp dpt:7782
6 5 195 DROP udp -- * * 79.101.0.0/16 2 13.5.176.155 udp dpt:7782
7 1 39 DROP udp -- * * 93.86.0.0/16 2 13.5.176.155 udp dpt:7782

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
num pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 34621229 packets, 4517872658 bytes)
num pkts bytes target prot opt in out source destination

Chain fail2ban-VSFTPD (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 RETURN all -- * * 0.0.0.0/0 0 .0.0.0/0
Chain PREROUTING (policy ACCEPT 67517 packets, 2822571 bytes)
num pkts bytes target prot opt in out source destination

Chain POSTROUTING (policy ACCEPT 8454 packets, 554497 bytes)
num pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 8454 packets, 554497 bytes)
num pkts bytes target prot opt in out source destination
Chain PREROUTING (policy ACCEPT 15918220 packets, 1334234603 bytes)
num pkts bytes target prot opt in out source destination

Chain INPUT (policy ACCEPT 15914853 packets, 1334096650 bytes)
num pkts bytes target prot opt in out source destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
num pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 34623135 packets, 4518057991 bytes)
num pkts bytes target prot opt in out source destination

Chain POSTROUTING (policy ACCEPT 34623135 packets, 4518057991 bytes)
num pkts bytes target prot opt in out source destination

Oh and I'm also noticing kjournald uses quite a bit of CPU when sendmail does.

MisterTickle 01-16-2011 09:26 AM
evidence of a process (probably other than Sendmail) causing ludicrous load problems, highest I saw was 1115 IIRC, making Sendmail halt its own processing until load drops

Would there be anyway to pinpoint the process?

Also

Jan 16 18:52:25 rs238 sshd[10880]: reverse mapping checking getaddrinfo for ptr110.49.dnion.com failed - POSSIBLE BREAK-IN ATTEMPT!
*Jan 16 18:52:25 rs238 sshd[10880]: Invalid user test1 from 122.11.49.110
*Jan 16 18:52:25 rs238 sshd[10880]: Invalid user test1 from 122.11.49.110
Jan 16 18:52:25 rs238 sshd[10880]: reverse mapping checking getaddrinfo for ptr110.49.dnion.com failed - POSSIBLE BREAK-IN ATTEMPT!
Jan 16 18:52:25 rs238 sshd[10883]: input_userauth_request: invalid user test1
Jan 16 18:52:25 rs238 sshd[10880]: pam_unix(sshd:auth): check pass; user unknown

*Jan 16 18:52:25 rs238 sshd[10880]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=122.11.49.110

unSpawn 01-16-2011 10:57 AM
Quote:
Originally Posted by MisterTickle (Post 4226152)
Code:
Chain INPUT (policy ACCEPT 15914022 packets, 1334023601 bytes)
num      pkts      bytes target    prot opt in    out    source                                                  destination
1          0        0 fail2ban-VSFTPD  tcp  --  *      *      0.0.0.0/0                                              0.0.0.0/0          tcp dpt:21
2          1      39 DROP      udp  --  *      *      77.46.0.0/16        2                                  n.n.n.155      udp dpt:7782
3          17      679 DROP      udp  --  *      *      178.223.0.0/16      2                                  n.n.n.155      udp dpt:7782
4          0        0 DROP      udp  --  *      *      93.87.0.0/16        2                                  n.n.n.155      udp dpt:7782
5          3      117 DROP      udp  --  *      *      109.93.0.0/16        2                                  n.n.n.155      udp dpt:7782
6          5      195 DROP      udp  --  *      *      79.101.0.0/16        2                                  n.n.n.155      udp dpt:7782
7          1      39 DROP      udp  --  *      *      93.86.0.0/16        2                                  n.n.n.155      udp dpt:7782

Chain fail2ban-VSFTPD (1 references)
num      pkts      bytes target    prot opt in    out    source                                                  destination
1          0        0 RETURN    all  --  *      *      0.0.0.0/0            0                                  .0.0.0/0
I don't know where the UDP/7782 rules come from but basically you don't have any iptables rules loaded to protect services (please edit your post and obfuscate your server's IP address always).


Quote:
Originally Posted by MisterTickle (Post 4226152)
Oh and I'm also noticing kjournald uses quite a bit of CPU when sendmail does.
Any process with a relatively low PID and the PPID set to that of "kthread" and having their argv[0] in brackets are in-kernel processes (see 'pgrep -lP $(pgrep kthread'). As the name suggests kjournald are threads (one per mounted ext3 file system) that take care of journaling. As long as running 'top' doesn't show the process to have a state of "D" there's no problem (also see sysctl VM settings, 'hal-disable-polling' and "noatime,nodiratime" mount flags).



Quote:
Originally Posted by MisterTickle (Post 4226153)
Would there be anyway to pinpoint the process?
- process accounting tools like 'lastcomm' and 'sa -cm' might provide crude information if psacct is enabled.
- 'top' (or 'atop 2 -g -C -f') should show if it's a process currently running.
- 'iostat' (or 'atop 2 -g -D -f') might also show clues if disk I/O is a bottleneck.
-
Code:
top -b -n 1 | awk '{if (NR <=7) print; else if ($8 == "D") {print; count++} } END {print "Total status D: "count}'
(code by one of our LQ regulars but I always forget who) should show any process in uninterruptable sleep.
- 'dstat' and 'collectl' can log statistics to file, but even better,
- 'atop' can log statistics to a binary file which means you can interactively review the process table at any given point in time (interval).

Any non-standard tools listed should be in the RPMForge repo.
If you decide to install and run Atop shoot me an email after system load has gone through the roof again so we can discuss where to send the binary log.


All times are GMT -5. The time now is 03:39 PM.
Page 1 of 2 1 2
Show 50 post(s) from this thread on one page