persistent tcpdump "truncated-ip6

psycroptic · 05-18-2015, 01:45 AM

I have an FTP ArchLinux server connected directly to a Comcast modem. It has "dual-stack" IPv4 and IPv6 connectivity, both using Comcast's DHCP. I use it to stream music to my Verizon Samsung Galaxy S4 over LTE; the phone also has Verizon dual-stack enabled. Typically the Arch server on Comcast gets a 2001:558:6016:29::/128 v6 address, and the S4 gets a 2600::/128 address from Verizon. I have transferred many gigabytes between the server and phone over the past year or so.

Over the past 3 weeks, IPv6 connectivity between the two has slowed dramatically, while IPv4 access is normal. I notice the following in a tcpdump on the Arch server's external NIC, filtered for relevant v6 traffic:

Code:

[root@pLAN9-Server4 /]# tcpdump -pnqi external ip6 and not icmp6
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on external, link-type EN10MB (Ethernet), capture size 65535 bytes
01:26:38.973673 IP6 truncated-ip6 - 8 bytes missing!2600:977a:b10d:3da7:3022:aa50:4a0f:12b9.37882 > 2001:558:6016:29:----:----:----:----.21: tcp 8
.
.
.
01:26:51.402523 IP6 truncated-ip6 - 8 bytes missing!2600:977a:b10d:3da7:3022:aa50:4a0f:12b9.59278 > 2001:558:6016:29:----:----:----:----.1726: tcp 8
.
01:26:52.471571 IP6 truncated-ip6 - 8 bytes missing!2600:9782:b10d:3da7:3022:aa50:4a0f:12b9.59278 > 2001:558:6016:29:----:----:----:----.1726: tcp 8
.
.
.
^C
55 packets captured
55 packets received by filter
0 packets dropped by kernel

I am seeing 1 out of at least every 30-40 packets showing up as "IP6 truncated-ip6 - 8 bytes missing!". Only the packets from the phone to the server are truncated like this. I have no problem accessing the server over IPv6 from other Comcast or local AT&T U-verse locations; these errors do not occur.

Can I conclude that this is a Verizon problem? Or possibly one with my smartphone?

Ser Olmy · 05-21-2015, 07:31 PM

8 bytes just happens to be the difference between a standard Ethernet MTU (1500) and the MTU of an IPv6-in-IPv4 tunnel (1492). Now, that could just be a coincidence, but I suspect there's an MTU blackhole somewhere. If the problem is intermittent, it could be related to an IPv6 backup route.

Try traceroute'ing from both ends with increasingly bigger payloads, and see if you can spot the troublesome node/router.

psycroptic · 05-21-2015, 11:24 PM

All traceroutes from the server to the phone fail at a "telia.net" device... never heard of them. Note that the phone does not respond to ICMP6 echo-requests over LTE, from any source.

traceroute -q 1 -z 1 -F 2600:1004:b14b:907e:f86c:e255:4818:1898 1400

Code:

traceroute to 2600:1004:b14b:907e:f86c:e255:4818:1898 (2600:1004:b14b:907e:f86c:e255:4818:1898), 30 hops max, 1400 byte packets
 1  2001:558:6016:29::1 (2001:558:6016:29::1)  48.106 ms
 2  xe-3-2-0-32767-sur01.ruralhillrd.tn.nash.comcast.net (2001:558:162:50::1)  10.796 ms
 3  xe-4-1-11-0-ar03.nashville.tn.nash.comcast.net (2001:558:160:52::1)  13.109 ms
 4  he-1-1-0-4-cr02.56marietta.ga.ibone.comcast.net (2001:558:0:f6af::1)  19.968 ms
 5  he-0-12-0-0-pe04.350ecermak.il.ibone.comcast.net (2001:558:0:f8e8::2)  24.955 ms
 6  2600:804:b0f::6d (2600:804:b0f::6d)  20.588 ms
 7  *
 8  2600:805:41f::22 (2600:805:41f::22)  26.188 ms
 9  nash-b1-v6.telia.net (2001:2000:3018:80::1)  42.122 ms
10  *
11  *
12  *
13  *
14  *^C

traceroute -q 1 -z 1 -F 2600:1004:b14b:907e:f86c:e255:4818:1898 1491

Code:

traceroute to 2600:1004:b14b:907e:f86c:e255:4818:1898 (2600:1004:b14b:907e:f86c:e255:4818:1898), 30 hops max, 1491 byte packets
 1  2001:558:6016:29::1 (2001:558:6016:29::1)  30.592 ms
 2  xe-3-2-0-32767-sur01.ruralhillrd.tn.nash.comcast.net (2001:558:162:50::1)  10.390 ms
 3  xe-5-1-11-0-ar01.goodslettvll.tn.nash.comcast.net (2001:558:160:45::1)  12.232 ms
 4  he-7-14-0-0-cr01.350ecermak.il.ibone.comcast.net (2001:558:0:f76f::1)  29.757 ms
 5  he-0-12-0-0-pe04.56marietta.ga.ibone.comcast.net (2001:558:0:f61c::2)  19.568 ms
 6  2001:559::57e (2001:559::57e)  23.733 ms
 7  Loopback0.GW1.MIA19.ALTER.NET (2600:804::22)  34.352 ms
 8  2600:805:41f::22 (2600:805:41f::22)  23.953 ms
 9  nash-b1-v6.telia.net (2001:2000:3018:80::1)  44.215 ms
10  *
11  *
12  *
13  *
14  *
15  *
16  *^C

traceroute -q 1 -z 1 -F 2600:1004:b14b:907e:f86c:e255:4818:1898 1492

Code:

traceroute to 2600:1004:b14b:907e:f86c:e255:4818:1898 (2600:1004:b14b:907e:f86c:e255:4818:1898), 30 hops max, 1492 byte packets
 1  2001:558:6016:29::1 (2001:558:6016:29::1)  32.418 ms
 2  xe-3-2-0-32767-sur01.ruralhillrd.tn.nash.comcast.net (2001:558:162:50::1)  64.307 ms
 3  xe-5-1-11-0-ar01.goodslettvll.tn.nash.comcast.net (2001:558:160:45::1)  28.103 ms
 4  he-7-13-0-0-cr01.350ecermak.il.ibone.comcast.net (2001:558:0:f691::1)  25.888 ms
 5  he-0-12-0-0-pe04.350ecermak.il.ibone.comcast.net (2001:558:0:f8e8::2)  29.698 ms
 6  2600:804:b0f::6d (2600:804:b0f::6d)  19.248 ms
 7  *
 8  2600:804:80f::4a (2600:804:80f::4a)  33.734 ms
 9  nash-b1-v6.telia.net (2001:2000:3018:80::1)  33.404 ms
10  *
11  *
12  *
13  *^C

traceroute -q 1 -z 1 -F 2600:1004:b14b:907e:f86c:e255:4818:1898 1500

Code:

traceroute to 2600:1004:b14b:907e:f86c:e255:4818:1898 (2600:1004:b14b:907e:f86c:e255:4818:1898), 30 hops max, 1500 byte packets
 1  2001:558:6016:29::1 (2001:558:6016:29::1)  65.642 ms
 2  xe-3-2-0-32767-sur01.ruralhillrd.tn.nash.comcast.net (2001:558:162:50::1)  69.454 ms
 3  xe-4-1-11-0-ar03.nashville.tn.nash.comcast.net (2001:558:160:52::1)  12.041 ms
 4  he-7-15-0-0-cr01.350ecermak.il.ibone.comcast.net (2001:558:0:f790::1)  25.466 ms
 5  he-0-12-0-0-pe04.56marietta.ga.ibone.comcast.net (2001:558:0:f61c::2)  18.052 ms
 6  2600:804:b0f::6d (2600:804:b0f::6d)  24.505 ms
 7  Loopback0.GW1.MIA19.ALTER.NET (2600:804::22)  36.901 ms
 8  2600:804:80f::4a (2600:804:80f::4a)  34.679 ms
 9  nash-b1-v6.telia.net (2001:2000:3018:80::1)  49.151 ms
10  *
11  *
12  *
13  *
14  *
15  *
16  *^C

Ser Olmy · 05-22-2015, 04:00 PM

(When I said traceroute, I actually meant traceroute6, but it seems your version of traceroute handles both IPv4 and IPv6, and defaults to the latter.)

The traceroute tool uses UDP packets, not ICMP, but it seems the probes are blocked by "nash-b1-v6.telia.net" regardless of size. The trace does tell you the problem is not likely to be related to anything at the Comcast end of the connection, though.

Nevertheless, unless the Galaxy is acting as an IPv6-in-IPv4 tunnel endpoint or is connecting using a PPP/PPPoE/PPPoA client, it very unlikely that MTU issues could be caused by the phone itself. In fact, even if it should turn out that the LTE connection does use PPP, it's still the responsibility of the server end to enforce proper MTU values.

psycroptic · 05-22-2015, 11:51 PM

Here is a trace from the phone to the server. Doesn't really tell me much, and the app I found to do IPv6 traceroutes (called "IPv6 and More") doesn't give you an option to set trace packet sizes....

Code:

Traceroute from this device to 2001:558:6016:29:----:----:----:----
Max Hops : 30

[1] 2600:1004:b15b:83eb:0:35:5a7c:fa40

[2]  *

[3] 2001:4888:22:2010:208:2a1:0:1

[4] 2001:4888:22:200e:208:25::

[5] 2001:4888:22:2000:208:2a1::

[6] 2001:4888:22:2005:208:1::

[7] 2001:4888:22:2005:208:1::

[8] 2001:4888:22:1001:208:24::

[9] 2001:2000:3080:6ab::1

[10]  *

[11]  *

[12] 2001:5a0:100:700::1a

[13] 2001:558:0:f790::2

[14] 2001:558:160:52::2

[15] 2001:558:162:50::2

[16] REACHED 2001:558:6016:29:----:----:----:---- 2001:558:6016:29:----:----:----:----:

I tried lowering the server's MTU to 1400, then 1000, and the problem still exists... anything else I could do?

psycroptic · 05-24-2015, 09:59 PM

Actually, a quick google search of "ALTER.NET", a PTR shown in some of the traces, indicates many problems Verizon users have experienced with the apparently VZW-owned "ALTER.NET" organization. https://www.google.com/search?q=alte...ter.net&nfpr=1

bldewolf · 06-03-2015, 02:22 PM

Finally, someone else! I've been seeing the "truncated-ip6 - 8 bytes missing" problem for a couple months, with no solution. I'm on Time Warner Cable. I'm also using an Arch Linux box connected to the modem. What modem do you have? My problems started around the time I switched from an SB6141 to an SB6183. I haven't tried switching back to an SB6141 yet, though.

Have you found a solution? My testing traffic doesn't pass through ALTER.NET, so I can't blame them for my problems.

psycroptic · 06-03-2015, 02:47 PM

Actually I have used those exact cable modems, Im on the SB6183, but in addition to the SB6141 I have also tried a Zoom 5341J. I dont think the modem is the problem, and I have no problem connecting from other Comcast v6-enabled locations using the phone, so I think the problem is external to Comcast.

Are you also using an Android phone on Verizon?

bldewolf · 06-03-2015, 05:04 PM

I've hit the problem with every IPv6 place I've used, though I don't know of any Time Warner Cable locations to test from. I've tested from my phone on T-Mobile, a BuyVM VPS, a Linode VPS, and from my work. My VPSes go through he.net to get to TWC, but my work only passes through Internet2, so it doesn't seem like an intermediary is affecting me.

I've done packet captures using my VPS and found that the before/after isn't just a simple truncation, either. The rejected packets have changed TCP options, however ones that don't get rejected are getting passed through fine. This led me to use scapy to send packets without TCP Options, which never seemed to get mangled (though I might have just been lucky). I've also noticed it only affects the initial SYN/SYNACK of a connection (it will take a ton of retries to connect, but then you can flood traffic across that TCP connection). It makes me think there's a device doing broken connection tracking somewhere in the mix, but I have no idea where or how to find it. Or who to complain to. Or why it only seems to affect us (or me, if we're not really seeing the same issue).

psycroptic · 06-03-2015, 06:04 PM

Quote:

Originally Posted by bldewolf

I've also noticed it only affects the initial SYN/SYNACK of a connection (it will take a ton of retries to connect, but then you can flood traffic across that TCP connection).

Yes, I've seen this exact thing. In FTP transfers, it manifests as taking a very long time to start a transfer, but once it's started it transfers normally.

I will try swapping my modem, just as a last ditch type of thing....

psycroptic · 06-05-2015, 03:49 AM

A further bit of information.... i believe i have isolated the problem to just Linux connecting clients - an FTP connection from another Comcast connection to the server using Windows does not produce the truncated packets, but connecting from a Linux machine does. I will call in & swap modems tomorrow to determine if this is the cause (at this point it's starting to look a bit like it)

psycroptic · 06-05-2015, 01:55 PM

So I have switched back to the Zoom 5341J modem, and the problem has immediately rectified. It appears this may be a firmware problem with the SB6183.

bldewolf · 06-05-2015, 04:20 PM

Awesome! Thanks for testing. I've got an order in for an SB6141 that I'll use until they sort out the SB6183s, then. I switched to owning the modem when I upgraded to an SB6183, and it's annoying to own the modem but have no say in which buggy firmware it runs.

joec@home · 06-07-2015, 04:41 PM

Quote:

Originally Posted by Ser Olmy

Nevertheless, unless the Galaxy is acting as an IPv6-in-IPv4 tunnel endpoint or is connecting using a PPP/PPPoE/PPPoA client, it very unlikely that MTU issues could be caused by the phone itself. In fact, even if it should turn out that the LTE connection does use PPP, it's still the responsibility of the server end to enforce proper MTU values.

A question on this, since both endpoints are native IPv6 and no IPv4 addresses are seen in the routes, do you still think the IPv6-in-IPv4 tunnel function is being activated some where on the route?

joec@home · 06-07-2015, 04:42 PM

Additional, you might want to look for choke points on the connection in both directions using mtr instead of traceroute.