Hi all,
I'm running a server with Ubuntu 16.04 (kernel 4.4.0-62-generic) with some virtual machines handling IPv6 traffic.
Apparently, every time a VM sends a ICMP6 router advertisement, I get a log in syslog of "IPV6 header not found".
I inspected the packets in the physical interface and they seem OK to me:
Code:
11:42:34.666022 02:XX:XX:XX:XX:XX > 33:33:00:00:00:01, ethertype 802.1Q (0x8100), length 158: vlan 203, p 0, ethertype 802.1Q, vlan 169, p 0, ethertype IPv6, fe80::XXXX:XXXX:XXXX > ff02::1: ICMP6, router advertisement, length 96
`....`:...........@...!w....................@...............@.!w..........@.... ........*.. ..!w................*.. ..!w............. '.
11:42:35.683194 02:XX:XX:XX:XX:XX > 33:33:00:00:00:01, ethertype 802.1Q (0x8100), length 158: vlan 203, p 0, ethertype 802.1Q, vlan 144, p 0, ethertype IPv6, fe80::XXXX:XXXX:XXXX > ff02::1: ICMP6, router advertisement, length 96
`....`:...........@...!^...................U@...............@.!^..........@.... ........*.. ..!^................*.. ..!^............. '.
Code:
sysadmin@olnmpep02318n001:~$ sudo tail -f /var/log/syslog
Aug 9 12:01:04 hostname kernel: [1638581.817138] IPv6 header not found
Aug 9 12:01:04 hostname kernel: [1638582.040507] IPv6 header not found
Aug 9 12:01:07 hostname kernel: [1638584.590434] IPv6 header not found
Aug 9 12:01:10 hostname kernel: [1638587.772732] IPv6 header not found
I am not sure if it's related to the problem, but the traffic gets Q-in-Q'ed on the way out of the server (one tag is included by the OVS the VM is attached to, and the other tag is included by a veth of VLAN type).
Anyone has faced this problem before? Any hint pointing out where I could investigate further?
Things I already did:
All the sysctl net.ipv6.conf.*.forwarding parameter is 0 and net.ipv6.conf.*.accept_ra is 0 also, so Router advertisements should not be accepted (just forwarded).
I tried to check the code printing that log, and it seems the offset is wrong or the version is not 6, but I checked the packets in Wireshark and they look fine to me...
Code:
if (*offset) {
struct ipv6hdr _ip6, *ip6;
ip6 = skb_header_pointer(skb, *offset, sizeof(_ip6), &_ip6);
if (!ip6 || (ip6->version != 6)) {
printk(KERN_ERR "IPv6 header not found\n");
return -EBADMSG;
}
start = *offset + sizeof(struct ipv6hdr);
nexthdr = ip6->nexthdr;
}
Thank you very much in advance!
Edit: New information:
I tried to investigate further the issue, tracing with systemtap the function printing the value of the params in the function call.
The stap program I used:
probe kernel.statement("*@net/ipv6/exthdrs_core.c:200") {
printf("Function call %s -> %s\n", thread_indent(1), ppfunc());
printf("%s\n", $$parms$);
printf("%s\n",$$locals$$);
printf("\nIP6->%s\n",$_ip6$$)
}
The result is the following:
Function call 1999283 vhost-38719(38729): -> ipv6_find_hdr
skb={<union>={...}, .sk=0x0, .dev=0xffff880def80c000, .cb=[...], ._skb_refdst=0, .destructor=0x0, .sp=0x0, .nfct=0x0, .nf_bridge=0x0, .len=94, .data_len=0, .mac_len=14, .hdr_len=0, .queue_mapping=11, .cloned=1, .nohdr=0, .fclone=0, .peeked=0, .head_frag=0, .xmit_more=0, .headers_start=[...], .__pkt_type_offset=[...], .pkt_type=2, .pfmemalloc=0, .ignore_df=0, .nfctinfo=0, .nf_trace=0, .ip_summed=0, .ooo_okay=0, .l4_hash=1, .sw_hash=1, .wifi_acked_valid=0, .wifi_acked=0, .no_fcs=0, .encapsulation=0, .encap_h
_ip6={.priority=8, .version=4, .flow_lbl="9�?���F�r����� ", .payload_len=34832, .nexthdr='\377', .hop_limit='\377', .saddr={.in6_u={.u6_addr8="F�r����� ", .u6_addr16=[60998, ...], .u6_addr32=[2171792966, ...]}}, .daddr={.in6_u={.u6_addr8="#�q�����", .u6_addr16=[35619, ...], .u6_addr32=[2171702051, ...]}}} ip6={.priority=?, .version=?, .flow_lbl=?, .payload_len=?, .nexthdr=?, .hop_limit=?, .saddr={.in6_u={.u6_addr8=?, .u6_addr16=[?, ...], .u6_addr32=[?, ...]}}, .daddr={.in6_u={.u6_addr8=?, .u6_addr16=[?, .
IP6->{.priority=8, .version=4, .flow_lbl="9�?���F�r����� ", .payload_len=34832, .nexthdr='\377', .hop_limit='\377', .saddr={.in6_u={.u6_addr8="F�r����� ", .u6_addr16=[60998, ...], .u6_addr32=[2171792966, ...]}}, .daddr={.in6_u={.u6_addr8="#�q�����", .u6_addr16=[35619, ...], .u6_addr32=[2171702051, ...]}}}
I also checked the dropped packets script example provided by systemtap (the one that traces kfree_skb function), and it seems there are some packages dropped:
37 packets dropped at 0xffffffff8172c1f9
10 packets dropped at 0xffffffff817dc08d
5 packets dropped at 0xffffffff8181a935
5 packets dropped at 0xffffffff8176b1fb
2 packets dropped at 0xffffffffc029627e
2 packets dropped at 0xffffffffc048f9d9
That, according to /boot/System.map-4.4.0-62-generic are:
37 by ffffffff8172bf30 t __netif_receive_skb_core
10 by ffffffff817dbf30 T ipv6_rcv
5 by ffffffff8181a8e0 t tpacket_rcv
5 by ffffffff8176b060 t ip_rcv_finish
(I can't find the other 2)
About the traces in the ipv6_find_hdr function:
* Confirms that some of the packets going through that functions are the Router Advertisements (lenght matches).
* It looks like the problem is in ipv6 header having version=4?
