Linux - Networking This forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game. |
| Notices |
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
Are you new to LinuxQuestions.org? Visit the following links:
Site Howto |
Site FAQ |
Sitemap |
Register Now
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
|
 |
12-23-2013, 05:40 AM
|
#1
|
|
Member
Registered: Feb 2010
Location: Amsterdam
Posts: 110
Rep: 
|
CentOS 6.5 conntrack table full and dropping packet
Hello,
I saw the morning this errors. An idea about this? Thank you.
Code:
Dec 23 08:00:06 LOC kernel: nf_conntrack: table full, dropping packet.
Dec 23 08:00:07 LOC kernel: nf_conntrack: table full, dropping packet.
Dec 23 08:00:15 LOC kernel: nf_conntrack: table full, dropping packet.
Dec 23 08:00:16 LOC kernel: nf_conntrack: table full, dropping packet.
Dec 23 08:00:16 LOC kernel: nf_conntrack: table full, dropping packet.
Dec 23 08:00:16 LOC kernel: nf_conntrack: table full, dropping packet.
Dec 23 08:00:17 LOC kernel: __ratelimit: 3 callbacks suppressed
Dec 23 08:00:30 LOC kernel: __ratelimit: 2 callbacks suppressed
Dec 23 08:00:30 LOC kernel: nf_conntrack: table full, dropping packet.
Dec 23 08:00:30 LOC kernel: nf_conntrack: table full, dropping packet.
Dec 23 08:00:37 LOC kernel: __ratelimit: 4 callbacks suppressed
Dec 23 08:00:37 LOC kernel: nf_conntrack: table full, dropping packet.
Dec 23 08:00:37 LOC kernel: nf_conntrack: table full, dropping packet.
Dec 23 08:00:37 LOC kernel: nf_conntrack: table full, dropping packet.
Dec 23 08:00:37 LOC kernel: nf_conntrack: table full, dropping packet.
Dec 23 08:00:37 LOC kernel: nf_conntrack: table full, dropping packet.
Dec 23 08:00:37 LOC kernel: nf_conntrack: table full, dropping packet.
Dec 23 08:00:43 LOC kernel: __ratelimit: 2 callbacks suppressed
Dec 23 08:00:43 LOC kernel: nf_conntrack: table full, dropping packet.
Dec 23 08:00:43 LOC kernel: nf_conntrack: table full, dropping packet.
Dec 23 08:00:46 LOC kernel: nf_conntrack: table full, dropping packet.
Dec 23 08:00:46 LOC kernel: nf_conntrack: table full, dropping packet.
Dec 23 08:00:46 LOC kernel: nf_conntrack: table full, dropping packet.
Dec 23 08:00:48 LOC kernel: __ratelimit: 2 callbacks suppressed
Dec 23 08:00:48 LOC kernel: nf_conntrack: table full, dropping packet.
Dec 23 08:00:48 LOC kernel: nf_conntrack: table full, dropping packet.
Dec 23 08:00:48 LOC kernel: nf_conntrack: table full, dropping packet.
Dec 23 08:00:50 LOC kernel: nf_conntrack: table full, dropping packet.
Dec 23 08:00:50 LOC kernel: nf_conntrack: table full, dropping packet.
Dec 23 08:00:59 LOC kernel: __ratelimit: 1 callbacks suppressed
Dec 23 08:00:59 LOC kernel: nf_conntrack: table full, dropping packet.
Dec 23 08:00:59 LOC kernel: nf_conntrack: table full, dropping packet.
Dec 23 08:01:00 LOC kernel: nf_conntrack: table full, dropping packet.
Code:
[root@LOC ~]# sysctl net.nf_conntrack_max
net.nf_conntrack_max = 65536
[root@LOC ~]#
Code:
[root@LOC ~]# uname -a
Linux LOC 2.6.32-431.1.2.0.1.el6.x86_64 #1 SMP Fri Dec 13 13:06:13 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
[root@LOC ~]#
Code:
[root@LOC ~]# egrep 'Mem|Cache|Swap' /proc/meminfo
MemTotal: 3924684 kB
MemFree: 2211696 kB
Cached: 749944 kB
SwapCached: 21952 kB
SwapTotal: 2097144 kB
SwapFree: 2021840 kB
[root@LOC ~]#
Last edited by hebeles; 12-23-2013 at 05:48 AM.
|
|
|
|
|
12-23-2013, 05:57 AM
|
#2
|
|
Senior Member
Registered: Oct 2004
Distribution: Debian Squeeze x86_64
Posts: 1,748
|
I would interpret the information given that you have more than 65356 for you conntrack table. Suprise suprise. Guess you already found that out on your own. As the number is quite high to my I say there are some stale connections still lingering around. Get the output of netstat -tulpn or ss -pan to see what connections are up.
All in all I would say that someone is DOS'ing you. Do you have syncookies enabled? Check sysctl for it. Another thing would be to tune the ttl of your connections. Also this depends what service holds all the connections. If all of the connections are legit maybe try to raise the max_nf_conntrack var of sysctl.
Forgot one. Check the rate limiting within iptables. This might have some too drastic values that fill your conntrack table.
|
|
|
|
|
12-23-2013, 06:23 AM
|
#3
|
|
Member
Registered: Feb 2010
Location: Amsterdam
Posts: 110
Original Poster
Rep:
|
My english is not good, but I understand you. Thank you for the answer. Yes, syncookies enabled.
Is there anything you can suggest here?
Code:
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
net.ipv4.icmp_echo_ignore_all=0
My iptables limit
Code:
-A INPUT -i eth3 -p udp -m udp --dport 53 -m recent --set --name DNS --rsource -j ACCEPT
-A INPUT -i eth3 -p udp -m udp --dport 53 -m recent --update --seconds 20 --hitcount 20 --name DNS --rsource -j DROP
-A INPUT -p icmp -m limit --limit 30/sec --limit-burst 200 -j ACCEPT
-A INPUT -p icmp -j DROP
|
|
|
|
|
12-23-2013, 08:16 AM
|
#4
|
|
Senior Member
Registered: Oct 2004
Distribution: Debian Squeeze x86_64
Posts: 1,748
|
Quote:
Originally Posted by hebeles
My english is not good, but I understand you. Thank you for the answer. Yes, syncookies enabled.
Is there anything you can suggest here?
Code:
net.ipv4.tcp_syncookies = 1
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0
|
Okay Syncookies are enabled.
Do you have a bridge setup on your system? Also I don't know what those sysctl settings do. But if you have no bridge you dont need to care about.
The complete list you have given where did you get it from? Do you use it in some kind of start script?
Quote:
Originally Posted by hebeles
My iptables limit
Code:
-A INPUT -i eth3 -p udp -m udp --dport 53 -m recent --set --name DNS --rsource -j ACCEPT
-A INPUT -i eth3 -p udp -m udp --dport 53 -m recent --update --seconds 20 --hitcount 20 --name DNS --rsource -j DROP
-A INPUT -p icmp -m limit --limit 30/sec --limit-burst 200 -j ACCEPT
-A INPUT -p icmp -j DROP
|
Okay 30connections per second should be more than enough for icmp. The dns could be a bit low but one resolution per second is okay.
Could be that DROP is your problem. I don't know how the kernel handles those. I know that the client just waits for an anwser from your machine. Maybe try with REJECT so the connection is ended on both sides.
Here are some sysctl settings to check. Those are taken from an Debian wheezy system and at their default value.
Code:
net.ipv4.ip_default_ttl = 64 # definitely check this one
net.ipv4.tcp_max_orphans = 262144 # this one too. Maximum of connections not associated.
net.ipv4.tcp_keepalive_time = 7200 # Check please
net.ipv4.tcp_keepalive_probes = 9
net.ipv4.tcp_keepalive_intvl = 75
net.ipv4.tcp_fin_timeout = 60 # This one too
net.netfilter.nf_conntrack_generic_timeout = 600
net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 120
net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 60
net.netfilter.nf_conntrack_tcp_timeout_established = 432000
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 60
net.netfilter.nf_conntrack_tcp_timeout_last_ack = 30
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_close = 10
net.netfilter.nf_conntrack_tcp_timeout_max_retrans = 300
net.netfilter.nf_conntrack_tcp_timeout_unacknowledged = 300
Not sure about the nf_conntrack_* ones. But might be worth to put an eye on.
Foremost to get this one solved is to get a list of connections conntrack sees. http://conntrack-tools.netfilter.org/ should help out. Else use
or
to get a list of known connections to the system.
This way we can see which type of connections are hogging conntrack. Be it http or icmp or dns. |
|
|
|
|
12-23-2013, 08:39 AM
|
#5
|
|
Member
Registered: Feb 2010
Location: Amsterdam
Posts: 110
Original Poster
Rep:
|
Yes, Syncookies enable.
I do not know about bridge but this virtual server. I share my sysctl.conf content. I not use any script.
Thank you for your further information.
I see them on logs. This attack or i ignore?
Code:
Dec 23 15:33:44 LOC kernel: IN IN=eth3 OUT= MAC=00:50:56:a5:4a:ed:00:09:0f:aa:96:8c:08:00 SRC=41.107.164.18 DST=192.168.3.6 LEN=53 TOS=0x00 PREC=0x00 TTL=107 ID=39604 PROTO=UDP SPT=20701 DPT=8009 LEN=33
Dec 23 15:33:48 LOC kernel: IN IN=eth3 OUT= MAC=00:50:56:a5:4a:ed:00:09:0f:aa:96:8c:08:00 SRC=202.58.137.56 DST=192.168.3.6 LEN=32 TOS=0x00 PREC=0x00 TTL=1 ID=279 PROTO=UDP SPT=10883 DPT=33436 LEN=12
Dec 23 15:33:52 LOC kernel: IN IN=eth3 OUT= MAC=00:50:56:a5:4a:ed:00:09:0f:aa:96:8c:08:00 SRC=202.58.137.76 DST=192.168.3.6 LEN=32 TOS=0x00 PREC=0x00 TTL=1 ID=1314 PROTO=UDP SPT=10883 DPT=33440 LEN=12
Dec 23 15:33:59 LOC kernel: IN IN=eth3 OUT= MAC=00:50:56:a5:4a:ed:00:09:0f:aa:96:8c:08:00 SRC=174.76.5.114 DST=192.168.3.8 LEN=78 TOS=0x00 PREC=0x00 TTL=39 ID=55701 DF PROTO=UDP SPT=137 DPT=137 LEN=58
Dec 23 15:33:59 LOC kernel: IN IN=eth3 OUT= MAC=00:50:56:a5:4a:ed:00:09:0f:aa:96:8c:08:00 SRC=207.28.74.1 DST=192.168.3.8 LEN=78 TOS=0x00 PREC=0x00 TTL=39 ID=49200 DF PROTO=UDP SPT=137 DPT=137 LEN=58
Dec 23 15:34:39 LOC kernel: IN IN=eth3 OUT= MAC=00:50:56:a5:4a:ed:00:09:0f:aa:96:8c:08:00 SRC=38.111.154.130 DST=192.168.3.6 LEN=78 TOS=0x00 PREC=0x00 TTL=36 ID=15730 DF PROTO=UDP SPT=137 DPT=137 LEN=58
Dec 23 15:35:02 LOC kernel: IN IN=eth3 OUT= MAC=00:50:56:a5:4a:ed:00:09:0f:aa:96:8c:08:00 SRC=91.188.117.228 DST=192.168.3.6 LEN=48 TOS=0x00 PREC=0x00 TTL=107 ID=6317 DF PROTO=TCP SPT=63619 DPT=5051 WINDOW=8192 RES=0x00 SYN URGP=0
Dec 23 15:35:05 LOC kernel: IN IN=eth3 OUT= MAC=00:50:56:a5:4a:ed:00:09:0f:aa:96:8c:08:00 SRC=91.188.117.228 DST=192.168.3.6 LEN=48 TOS=0x00 PREC=0x00 TTL=107 ID=6861 DF PROTO=TCP SPT=63619 DPT=5051 WINDOW=8192 RES=0x00 SYN URGP=0
Dec 23 15:35:07 LOC kernel: IN IN=eth3 OUT= MAC=00:50:56:a5:4a:ed:00:09:0f:aa:96:8c:08:00 SRC=96.47.146.242 DST=192.168.3.6 LEN=78 TOS=0x00 PREC=0x00 TTL=34 ID=55492 DF PROTO=UDP SPT=137 DPT=137 LEN=58
Dec 23 15:35:11 LOC kernel: IN IN=eth3 OUT= MAC=00:50:56:a5:4a:ed:00:09:0f:aa:96:8c:08:00 SRC=91.188.117.228 DST=192.168.3.6 LEN=48 TOS=0x00 PREC=0x00 TTL=107 ID=7842 DF PROTO=TCP SPT=63619 DPT=5051 WINDOW=8192 RES=0x00 SYN URGP=0
Dec 23 15:35:19 LOC kernel: IN IN=eth3 OUT= MAC=00:50:56:a5:4a:ed:00:09:0f:aa:96:8c:08:00 SRC=64.238.189.139 DST=192.168.3.6 LEN=78 TOS=0x00 PREC=0x00 TTL=32 ID=6089 DF PROTO=UDP SPT=137 DPT=137 LEN=58
|
|
|
|
|
12-23-2013, 10:01 AM
|
#6
|
|
Senior Member
Registered: Oct 2004
Distribution: Debian Squeeze x86_64
Posts: 1,748
|
The time between the logs are to far away to assume a DOS so I would not look to much into this. Also the ports are a bit funcky. Only found 137 for netbios and 5051 for ENBD in /etc/services.
Deifinitely get a list of all open connections and lets go from there.
|
|
|
|
|
12-23-2013, 10:19 AM
|
#7
|
|
Member
Registered: Feb 2010
Location: Amsterdam
Posts: 110
Original Poster
Rep:
|
Thank you. I not use this 137 port. Any seems wrong?
Code:
[root@LOC ~]# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:587 0.0.0.0:* LISTEN 1860/exim
tcp 0 0 0.0.0.0:465 0.0.0.0:* LISTEN 1860/exim
tcp 0 0 0.0.0.0:25 0.0.0.0:* LISTEN 1860/exim
tcp 0 0 0.0.0.0:5051 0.0.0.0:* LISTEN 22400/nginx
tcp 0 0 127.0.0.1:9000 0.0.0.0:* LISTEN 22423/php-fpm
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN 1790/mysqld
tcp 0 0 :::587 :::* LISTEN 1860/exim
tcp 0 0 :::465 :::* LISTEN 1860/exim
tcp 0 0 :::5654 :::* LISTEN 1511/sshd
tcp 0 0 :::25 :::* LISTEN 1860/exim
[root@LOC ~]
Code:
[root@LOC ~]# ss -lu
State Recv-Q Send-Q Local Address:Port Peer Address:Port
UNCONN 0 0 192.168.3.6:dsm-scm-target *:* *:*
[root@LOC ~]#
Last edited by hebeles; 12-23-2013 at 10:22 AM.
|
|
|
|
|
12-23-2013, 10:35 AM
|
#8
|
|
Senior Member
Registered: Oct 2004
Distribution: Debian Squeeze x86_64
Posts: 1,748
|
looks okay. I just gave wrong options for netstat. We want all the connections. Not only the listening ones. So quite on the -l option.
Please give output of |
|
|
|
|
12-23-2013, 11:22 AM
|
#9
|
|
Member
Registered: Feb 2010
Location: Amsterdam
Posts: 110
Original Poster
Rep:
|
Code:
[root@LOC ~]# ss -a
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 20 *:submission *:*
LISTEN 0 20 :::submission :::*
LISTEN 0 20 *:465 *:*
LISTEN 0 20 :::465 :::*
LISTEN 0 32 *:5652 *:*
LISTEN 0 128 :::5654 :::*
LISTEN 0 128 *:5654 *:*
LISTEN 0 20 *:smtp *:*
LISTEN 0 20 :::smtp :::*
LISTEN 0 128 *:mmcc *:*
LISTEN 0 128 127.0.0.1:cslistener *:*
LISTEN 0 50 *:mysql *:*
ESTAB 0 0 192.168.3.6:5654 8.8.22.55:outlaws
ESTAB 0 0 192.168.3.8:mmcc 8.22.160.207:23852
ESTAB 0 64 192.168.3.6:5654 8.8.22.55:5536
TIME-WAIT 0 0 127.0.0.1:cslistener 127.0.0.1:57346
[root@LOC ~]#
I hope this information helpful. Thank you |
|
|
|
All times are GMT -5. The time now is 09:04 AM.
|
|
|
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
|
|
 Latest Threads
LQ News
|
|
