Hi,
I am having a problem with the IPv6 route cache.
No combination of changes to:
sysctl net.ipv6.route.<parameters>
improves it.
Every kernel version I have tried exhibits this problem.
I am routing thousands of IPv6 requests through a tun interface.
My problem occurs when greater-than 1024 simultanous IPv6 destination
addresses are in the cache.
The next arriving packet #1025 (UDP, ICMPv6) with a new address fails to route.
I have been able to distill this problem down to a simple configuration
that demonstrates the issue...no code just using a spare Ethernet interface
and a shell script.
To view and reproduce the problem:
1) Create a shell window and type the following:
watch -n0 -d grep NoRoute /proc/net/snmp6
2) Create another shell window and type:
watch -n0 -d wc -l /proc/net/ipv6_route
3) Create another shell window and perform the following:
- Select an interface, I picked eth1 since it is a less used interface
- Select a prefix that won't cause a problem...I'm on a private LAN
- At the shell prompt, type:
INTERFACE=eth1
sudo /sbin/ip -6 addr add 2001:8899::1/128 dev $INTERFACE
sudo /sbin/ip -6 route add 2001:8899::/32 dev $INTERFACE
- Create a shell script named looptest.sh
#!/bin/bash
#set -x
SHORTFILE=/etc/group
num=1
MAX=9999
while true; do
#usleep 20000
#nc -6 -u -w0 2001:8899::$num 1234 < /$SHORTFILE
socat -T0 -t0 stdin UDP6-datagram:[2001:8899:0:1::$num]:1234 < $SHORTFILE
STATUS=$?
if [ $STATUS != 0 ]; then
echo UDP failed with status $STATUS at $num
break;
fi
num=`expr $num + 1`
if [ $num -ge $MAX ]; then
break;
fi
done
- Run the script, after 1024 iterations it fails with:
socat[29312] E sendto(3, 0xb137230, 756, 0, AF=10 [2001:8899:0000:0001:0000:0000:0000:1024]:1234, 28): Invalid argument
UDP failed with status 1 at 1024
- While the test is running
- The number of entries in the route cache window increases
- Ip6OutNoRoutes is incremented...
- Notes:
- Adding a usleep in the loop makes it run better/longer
- dmesg | fgrep "IP route cache"
IP route cache hash table entries: 131072 (order: 8, 1048576 bytes)
- The behavior can be made much worse by changing the net.ipv6.route parameters
- For example, some values I have tried:
sudo sysctl net.ipv6.icmp.ratelimit=0
sudo sysctl -w net.ipv6.route.gc_min_interval_ms=500
sudo sysctl -w net.ipv6.route.min_adv_mss=1
sudo sysctl -w net.ipv6.route.mtu_expires=600
sudo sysctl -w net.ipv6.route.gc_elasticity=0
sudo sysctl -w net.ipv6.route.gc_interval=30
sudo sysctl -w net.ipv6.route.gc_timeout=60
sudo sysctl -w net.ipv6.route.gc_min_interval=0
sudo sysctl -w net.ipv6.route.max_size=4096
sudo sysctl -w net.ipv6.route.gc_thresh=1024
sudo sysctl -w net.ipv6.route.gc_min_interval_ms=1
sudo sysctl -w net.ipv6.route.min_adv_mss=1
sudo sysctl -w net.ipv6.route.mtu_expires=600
sudo sysctl -w net.ipv6.route.gc_elasticity=0
sudo sysctl -w net.ipv6.route.gc_interval=1
sudo sysctl -w net.ipv6.route.gc_timeout=1
sudo sysctl -w net.ipv6.route.gc_min_interval=1
sudo sysctl -w net.ipv6.route.max_size=131072
sudo sysctl -w net.ipv6.route.gc_thresh=10240
sudo sysctl -w net.ipv6.route.max_size=8388608
Maybe this is the fix...I'll test and report back:
https://git.kernel.org/cgit/linux/ke...baca867adec577
Any suggestions are appreciated.
Thanks
Kert
