LinuxQuestions.org
Latest LQ Deal: Latest LQ Deals
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Networking
User Name
Password
Linux - Networking This forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.

Notices


Reply
  Search this Thread
Old 12-03-2015, 04:09 PM   #1
postcd
Member
 
Registered: Oct 2013
Posts: 527

Rep: Reputation: Disabled
Nearly full conntrack table, 60K lines


Hello,

im having CentOS (redhat) OpenVZ VPS and i do command:

wc -l /proc/net/nf_conntrack
Quote:
62109 /proc/net/nf_conntrack
sysctl net.netfilter.nf_conntrack_count && sysctl net.nf_conntrack_max
Quote:
net.netfilter.nf_conntrack_count = 62095
net.nf_conntrack_max = 65536
tail & head on /proc/net/nf_conntrack

shows connection like this one (ESTABLISHED, ASSURED)

Quote:
ipv4 2 tcp 6 401407 ESTABLISHED src=SOMEONEELSEIP dst=MYSERVERIPHERE sport=53375 dport=80 src=MYSERVERIPHERE dst=SOMEONEELSEIP sport=80 dport=53375 [ASSURED] mark=0 secmark=0 use=2
Apache shows that many different IPs (800+) trying to connect one web directory (which is empty), the connection speed can be like 5 IPs per second.

cat /etc/sysctl.conf | grep =
Quote:
net.ipv4.ip_forward = 0
# net.ipv4.conf.default.rp_filter = 1
# net.ipv4.conf.default.accept_source_route = 0
# kernel.sysrq = 0
# kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0
# kernel.msgmnb = 65536
# kernel.msgmax = 65536
# kernel.shmmax = 68719476736
# kernel.shmall = 4294967296
i tried to add these lines into above file, but conntrack table do not decrease:
Quote:
net.netfilter.nf_conntrack_tcp_timeout_established = 600
net.netfilter.nf_conntrack_generic_timeout = 120
then i tried # sysctl -p
Quote:
net.ipv4.ip_forward = 0
net.ipv4.tcp_syncookies = 1
error: "net.bridge.bridge-nf-call-ip6tables" is an unknown key
error: "net.bridge.bridge-nf-call-iptables" is an unknown key
error: "net.bridge.bridge-nf-call-arptables" is an unknown key
error: permission denied on key 'net.netfilter.nf_conntrack_tcp_timeout_established'
error: permission denied on key 'net.netfilter.nf_conntrack_generic_timeout'
# sysctl -a | grep conn | grep time
Quote:
net.netfilter.nf_conntrack_generic_timeout = 600
net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 120
net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 60
net.netfilter.nf_conntrack_tcp_timeout_established = 432000
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 60
net.netfilter.nf_conntrack_tcp_timeout_last_ack = 30
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_close = 10
net.netfilter.nf_conntrack_tcp_timeout_max_retrans = 300
net.netfilter.nf_conntrack_tcp_timeout_unacknowledged = 300
net.netfilter.nf_conntrack_udp_timeout = 30
net.netfilter.nf_conntrack_udp_timeout_stream = 180
net.netfilter.nf_conntrack_icmp_timeout = 30
net.netfilter.nf_conntrack_events_retry_timeout = 15
I want to ask for kind advice how can anyhow secure server to prevent such high number of lines in connection tracking table? And if i can temporarily clean that table, how? How would you advise to tweak the settings? The server normal connections rate is like 50 connections per second i guess, it is webserver.

Thank you

Update:
1) this helped to temporarily reduce conntrack table size:
yum install conntrack-tools # install conntrack tools
conntrack -D -d MYSERVERIP # delete conntrack entries where destination ip is my server ip

2) And also temporarilly increasing conntrack table size limit: echo 66666 > /proc/sys/net/netfilter/nf_conntrack_max

3) Into /etc/sysctl.conf i added:
net.netfilter.nf_conntrack_tcp_timeout_established = 600
net.netfilter.nf_conntrack_generic_timeout = 120

Last edited by postcd; 12-03-2015 at 07:48 PM.
 
Old 12-03-2015, 05:25 PM   #2
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
Quote:
Originally Posted by postcd View Post
im having CentOS (redhat) OpenVZ VPS
OpenVZ, useful as it may be to some in some situations, may limit things. Be aware of its capabilities.


Quote:
Originally Posted by postcd View Post
Apache shows that many different IPs (800+) trying to connect one web directory (which is empty), the connection speed can be like 5 IPs per second.
Iptables allows for rate limiting. That would be more efficient than using any mod_throttle, mod_bw or equivalent.


Quote:
Originally Posted by postcd View Post
i tried to add these lines into above file, but conntrack table do not decrease
How about echoing the values into the actual /proc item?
Like
Code:
KEY="/proc/sys/net/netfilter/nf_conntrack_max"; VAL=$(< ${KEY}); echo $[${VAL}*2] > "${KEY}"

Quote:
Originally Posted by postcd View Post
how can anyhow secure server to prevent such high number of lines in connection tracking table?
I'd say you can't "prevent" it as you have no control over remote hosts ;-p
That said tuning it only makes sense if you get (or expect) "ip_conntrack: table full, dropping packet" in dmesg / syslog.


Quote:
Originally Posted by postcd View Post
How would you advise to tweak the settings?
Start by finding a copy of http://www.wallfire.org/misc/netfilt...track_perf.txt like http://technotes.whw1.com/component/...s-ip_conntrack and meditate on that?


Quote:
Originally Posted by postcd View Post
The server normal connections rate is like 50 connections per second i guess, it is webserver.
...also note performance isn't only sysctls, also see more generic docs like http://cdn.oreillystatic.com/en/asse...esentation.pdf
 
Old 12-03-2015, 06:45 PM   #3
postcd
Member
 
Registered: Oct 2013
Posts: 527

Original Poster
Rep: Reputation: Disabled
> How about echoing the values into the actual /proc item?
That worked to increase value. Thank you

But the attack is ongoing, there was an conntrack dump file and it shown around 1100 unique IPs in it. Please can you advise how to block kind of connections i mentioned?:

connection like this one (ESTABLISHED, ASSURED)

Quote:
ipv4 2 tcp 6 401407 ESTABLISHED src=SOMEONEELSEIP dst=MYSERVERIPHERE sport=53375 dport=80 src=MYSERVERIPHERE dst=SOMEONEELSEIP sport=80 dport=53375 [ASSURED] mark=0 secmark=0 use=2
the port numbers are variable, except it always works with port 80 on my server. Thank you

Update: this helped to temporarily reduce conntrack table size:
yum install conntrack-tools # install conntrack tools
conntrack -D -d MYSERVERIP # delete conntrack entries where destination ip is my server ip

Last edited by postcd; 12-03-2015 at 07:39 PM.
 
Old 12-06-2015, 05:00 AM   #4
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
Quote:
Originally Posted by postcd View Post
Please can you advise how to block kind of connections i mentioned?:
connection like this one (ESTABLISHED, ASSURED)
the port numbers are variable, except it always works with port 80 on my server.
Like I already said: use iptables rate limiting. So. Hunt down fitting examples on LQ (Security / Networking), configure for your situation and post for comments / corrections.


Quote:
Originally Posted by postcd View Post
Update: this helped to temporarily reduce conntrack table size:
yum install conntrack-tools # install conntrack tools
conntrack -D -d MYSERVERIP # delete conntrack entries where destination ip is my server ip
Like with most attacks tuning values up or down only makes sense if you know what the implications are. Reducing the size of the conntrack table does not prevent or mitigate any attacks: that's not what the table is for. The only thing you'll get from reduction is shorter individual entry TTL (and increased need for the kernel to do garbage collection as it tries to keep up with things) and possibly the dreaded "ip_conntrack: table full, dropping packet" in dmesg / syslog.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
CentOS 6.5 conntrack table full and dropping packet hebeles Linux - Networking 8 12-23-2013 10:22 AM
conntrack table gets full, even though the traffic is dropped by iptables yinonby Linux - Security 3 02-26-2013 11:13 AM
ip_conntrack table full csdhiman Linux - Server 10 03-14-2008 10:11 AM
ip_conntrack,table full. santhosh23 Linux - Networking 1 07-28-2007 12:36 AM
PHP Pull File Lines Into Table windisch Programming 7 11-02-2006 05:06 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Networking

All times are GMT -5. The time now is 04:01 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration