LinuxQuestions.org
Register a domain and help support LQ
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices

Reply
 
Search this Thread
Old 08-31-2007, 12:42 AM   #1
csdhiman
Member
 
Registered: Mar 2007
Posts: 47

Rep: Reputation: 15
ip_conntrack table full


Dear All

I have Fedora Core 1 on Server which has postfix with mysql running on it.
Hardware details are
Dual Intel Xeon CPU 3 Ghz ,
1 Gb Memory
Swap is 3 GB

i get the error messeges on the screen as: ( from syslog )

Aug 30 13:25:46 mailsswl kernel: ip_conntrack: table full, dropping packet.
Aug 30 13:25:50 mailsswl kernel: NET: 66 messages suppressed.
Aug 30 13:25:51 mailsswl kernel: ip_conntrack: table full, dropping packet.
Aug 30 13:25:55 mailsswl kernel: NET: 39 messages suppressed.
Aug 30 13:25:56 mailsswl kernel: ip_conntrack: table full, dropping packet.
Aug 30 13:26:01 mailsswl kernel: NET: 36 messages suppressed.
Aug 30 13:26:02 mailsswl kernel: ip_conntrack: table full, dropping packet.

Check the output of
# cat /proc/net/ip_conntrack | wc -l is
62222

cat /proc/sys/net/ipv4/ip_conntrack_max
65496

What can be wrong pls suggest me ... As the messege said dropping packets is it is dropping mails.
 
Old 09-01-2007, 10:14 AM   #2
felosi
LQ Newbie
 
Registered: Jan 2006
Location: USA
Distribution: CentOS for servers and Ubuntu for desktop
Posts: 25

Rep: Reputation: 15
Look like you are getting syn flooded. You need to find a way to limit connections like using a firewall like csf or apf with dos deflate and you can increase the syn_backlog and contrack_table according to your hardware in /etc/sysctl.conf
You can do sysctl -a | grep contrack
and then add the line to sysctl.conf with the new limit. You may wanna google adjusting that stuff to get the right equation for limits to use per certain amount of memory, etc;
 
Old 09-03-2007, 12:08 AM   #3
csdhiman
Member
 
Registered: Mar 2007
Posts: 47

Original Poster
Rep: Reputation: 15
re

Quote:
Originally Posted by felosi View Post
Look like you are getting syn flooded. You need to find a way to limit connections like using a firewall like csf or apf with dos deflate and you can increase the syn_backlog and contrack_table according to your hardware in /etc/sysctl.conf
You can do sysctl -a | grep contrack
and then add the line to sysctl.conf with the new limit. You may wanna google adjusting that stuff to get the right equation for limits to use per certain amount of memory, etc;
There is no entry for ip_conntrack in sysctl
sysctl -a | grep contrack
no output

what should i use to solve the problem
 
Old 09-03-2007, 08:48 AM   #4
rameshshihora
LQ Newbie
 
Registered: Jul 2007
Posts: 16

Rep: Reputation: 0
Hi,

Just enable tcp_syncookies will solve your problem.

net.ipv4.tcp_syncookies = 1

sysctl -p

Best Regards,
Ramesh Shihora
 
Old 09-03-2007, 11:26 PM   #5
felosi
LQ Newbie
 
Registered: Jan 2006
Location: USA
Distribution: CentOS for servers and Ubuntu for desktop
Posts: 25

Rep: Reputation: 15
mispelled conntrack
sysctl -a | grep contrack
should be
sysctl -a | grep conntrack
 
Old 09-05-2007, 08:04 PM   #6
win32sux
Guru
 
Registered: Jul 2003
Location: Los Angeles
Distribution: Ubuntu
Posts: 9,870

Rep: Reputation: 371Reputation: 371Reputation: 371Reputation: 371
How did you guys reach the conclusion that this was a SYN flood and not just excessive normal traffic? Strikes me as odd to have had concluded that so fast with only the information given. I've seen several cases where Netfilter's state table is overloaded from normal (non-DoS attack) usage. Tweaking maxes and timeouts almost always proved to be the solution in these situations.
 
Old 09-07-2007, 06:50 AM   #7
csdhiman
Member
 
Registered: Mar 2007
Posts: 47

Original Poster
Rep: Reputation: 15
Dear All
The output of sysctl -a | grep conntrack is

net.ipv4.ip_conntrack_max = 65496
net.ipv4.netfilter.ip_conntrack_generic_timeout = 600
net.ipv4.netfilter.ip_conntrack_icmp_timeout = 30
net.ipv4.netfilter.ip_conntrack_udp_timeout_stream = 180
net.ipv4.netfilter.ip_conntrack_udp_timeout = 30
net.ipv4.netfilter.ip_conntrack_tcp_timeout_close = 10
net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait = 120
net.ipv4.netfilter.ip_conntrack_tcp_timeout_last_ack = 30
net.ipv4.netfilter.ip_conntrack_tcp_timeout_close_wait = 259200
net.ipv4.netfilter.ip_conntrack_tcp_timeout_fin_wait = 120
net.ipv4.netfilter.ip_conntrack_tcp_timeout_established = 432000
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_recv = 60
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_sent = 120
net.ipv4.netfilter.ip_conntrack_max = 65496

When the problem occur it stop responding the ping request or n/w activity
i found that on restarting iptables service , the server works fine and it give output
cat /proc/net/ip_conntrack |wc -l
4421
which was 60000 before restarting iptables service


now i added the line in sysctl.conf
net.ipv4.tcp_syncookies = 1

sysctl -p

is it ok or again i will get the problem
 
Old 09-19-2007, 11:16 PM   #8
csdhiman
Member
 
Registered: Mar 2007
Posts: 47

Original Poster
Rep: Reputation: 15
The Problem Is still There
I have to restart the iptables service once in a day .
Can i disable firewall of this server . Because this server is behind a windows firewall serever ..

Or i have to update the kernel

pls suggest
 
Old 09-20-2007, 08:28 AM   #9
win32sux
Guru
 
Registered: Jul 2003
Location: Los Angeles
Distribution: Ubuntu
Posts: 9,870

Rep: Reputation: 371Reputation: 371Reputation: 371Reputation: 371
Change this:
Code:
net.ipv4.ip_conntrack_max = 65496
net.ipv4.netfilter.ip_conntrack_tcp_timeout_close_wait = 259200
net.ipv4.netfilter.ip_conntrack_tcp_timeout_established = 432000
To this:
Code:
net.ipv4.ip_conntrack_max = 98000
net.ipv4.netfilter.ip_conntrack_tcp_timeout_close_wait = 240
net.ipv4.netfilter.ip_conntrack_tcp_timeout_established = 28800
This way you slightly increase the capacity of your state table, while at the same time reducing the amount of time for established connections to timeout from 5 days (432000 seconds) to 8 hours (28800 seconds). You are also changing the amount of time connections will be kept in TIME_WAIT states to something sane. I believe these settings represent the bulk of your problem, but let us know how it goes, please.

PS: You might also wanna disable TCP SYN cookies unless you are sure you are under attack.

Last edited by win32sux; 09-20-2007 at 09:05 AM.
 
Old 02-04-2008, 05:01 AM   #10
ittec
LQ Newbie
 
Registered: Nov 2007
Posts: 15

Rep: Reputation: 0
Question The number of bytes per connection

Hi

very helpful thread. But i have a doubt about the real number of bytes used per tcp/ip connection. I did

#grep conntrack messages

and the last line output was:

Feb 3 16:35:58 dns kernel: ip_conntrack version 2.4 (8192 buckets, 65536 max) - 228 bytes per conntrack

So i understand that each tpc connection to my host uses 228 bytes of RAM actually. But how can i confirm that information? I read before another howtos that speaks of another values of bytes per connection(350 bytes f.e)

Thanks
 
Old 03-14-2008, 10:11 AM   #11
ittec
LQ Newbie
 
Registered: Nov 2007
Posts: 15

Rep: Reputation: 0
Experimenting troubles

The worst problem of this issue is that althought you increase the value of your ip_contrack_max value, this value is more and more big and it never flushes. So one day you reach the conntrack_max value. These are all values about time of open connections:

Code:
net.ipv4.netfilter.ip_conntrack_tcp_timeout_close = 30
net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait = 30
net.ipv4.netfilter.ip_conntrack_tcp_timeout_last_ack = 120
net.ipv4.netfilter.ip_conntrack_tcp_timeout_close_wait = 30
net.ipv4.netfilter.ip_conntrack_tcp_timeout_fin_wait = 60
net.ipv4.netfilter.ip_conntrack_tcp_timeout_established = 30
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_recv = 30
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_sent = 30
Is a very restrictive setup and now I need to watch and keep the performance of server. But Im not sure ....
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
ip_conntrack,table full. santhosh23 Linux - Networking 1 07-28-2007 12:36 AM
how to flush ip_conntrack table without removal of modouls ? eranzwilling Linux - Networking 1 01-22-2006 11:58 AM
ip_conntrack: table full, dropping packet. ingerul Linux - Networking 9 12-03-2004 01:46 PM
ip_conntrack table full Skunk_Face Linux - Security 1 11-01-2004 05:14 PM
ip_conntrack table full despite relatively few connections tvynr Linux - Networking 3 10-04-2004 05:03 PM


All times are GMT -5. The time now is 04:42 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration