LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Networking
User Name
Password
Linux - Networking This forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.

Notices


Reply
  Search this Thread
Old 05-18-2015, 01:45 AM   #1
psycroptic
Member
 
Registered: Aug 2011
Location: USA
Distribution: ArchLinux - 3.0 kernel
Posts: 349

Rep: Reputation: Disabled
persistent tcpdump "truncated-ip6 - 8 bytes missing!" & low IPv6 performance


I have an FTP ArchLinux server connected directly to a Comcast modem. It has "dual-stack" IPv4 and IPv6 connectivity, both using Comcast's DHCP. I use it to stream music to my Verizon Samsung Galaxy S4 over LTE; the phone also has Verizon dual-stack enabled. Typically the Arch server on Comcast gets a 2001:558:6016:29::/128 v6 address, and the S4 gets a 2600::/128 address from Verizon. I have transferred many gigabytes between the server and phone over the past year or so.

Over the past 3 weeks, IPv6 connectivity between the two has slowed dramatically, while IPv4 access is normal. I notice the following in a tcpdump on the Arch server's external NIC, filtered for relevant v6 traffic:

Code:
[root@pLAN9-Server4 /]# tcpdump -pnqi external ip6 and not icmp6
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on external, link-type EN10MB (Ethernet), capture size 65535 bytes
01:26:38.973673 IP6 truncated-ip6 - 8 bytes missing!2600:977a:b10d:3da7:3022:aa50:4a0f:12b9.37882 > 2001:558:6016:29:----:----:----:----.21: tcp 8
.
.
.
01:26:51.402523 IP6 truncated-ip6 - 8 bytes missing!2600:977a:b10d:3da7:3022:aa50:4a0f:12b9.59278 > 2001:558:6016:29:----:----:----:----.1726: tcp 8
.
01:26:52.471571 IP6 truncated-ip6 - 8 bytes missing!2600:9782:b10d:3da7:3022:aa50:4a0f:12b9.59278 > 2001:558:6016:29:----:----:----:----.1726: tcp 8
.
.
.
^C
55 packets captured
55 packets received by filter
0 packets dropped by kernel
I am seeing 1 out of at least every 30-40 packets showing up as "IP6 truncated-ip6 - 8 bytes missing!". Only the packets from the phone to the server are truncated like this. I have no problem accessing the server over IPv6 from other Comcast or local AT&T U-verse locations; these errors do not occur.

Can I conclude that this is a Verizon problem? Or possibly one with my smartphone?
 
Old 05-21-2015, 07:31 PM   #2
Ser Olmy
Senior Member
 
Registered: Jan 2012
Distribution: Slackware
Posts: 3,347

Rep: Reputation: Disabled
8 bytes just happens to be the difference between a standard Ethernet MTU (1500) and the MTU of an IPv6-in-IPv4 tunnel (1492). Now, that could just be a coincidence, but I suspect there's an MTU blackhole somewhere. If the problem is intermittent, it could be related to an IPv6 backup route.

Try traceroute'ing from both ends with increasingly bigger payloads, and see if you can spot the troublesome node/router.
 
Old 05-21-2015, 11:24 PM   #3
psycroptic
Member
 
Registered: Aug 2011
Location: USA
Distribution: ArchLinux - 3.0 kernel
Posts: 349

Original Poster
Rep: Reputation: Disabled
All traceroutes from the server to the phone fail at a "telia.net" device... never heard of them. Note that the phone does not respond to ICMP6 echo-requests over LTE, from any source.

traceroute -q 1 -z 1 -F 2600:1004:b14b:907e:f86c:e255:4818:1898 1400
Code:
traceroute to 2600:1004:b14b:907e:f86c:e255:4818:1898 (2600:1004:b14b:907e:f86c:e255:4818:1898), 30 hops max, 1400 byte packets
 1  2001:558:6016:29::1 (2001:558:6016:29::1)  48.106 ms
 2  xe-3-2-0-32767-sur01.ruralhillrd.tn.nash.comcast.net (2001:558:162:50::1)  10.796 ms
 3  xe-4-1-11-0-ar03.nashville.tn.nash.comcast.net (2001:558:160:52::1)  13.109 ms
 4  he-1-1-0-4-cr02.56marietta.ga.ibone.comcast.net (2001:558:0:f6af::1)  19.968 ms
 5  he-0-12-0-0-pe04.350ecermak.il.ibone.comcast.net (2001:558:0:f8e8::2)  24.955 ms
 6  2600:804:b0f::6d (2600:804:b0f::6d)  20.588 ms
 7  *
 8  2600:805:41f::22 (2600:805:41f::22)  26.188 ms
 9  nash-b1-v6.telia.net (2001:2000:3018:80::1)  42.122 ms
10  *
11  *
12  *
13  *
14  *^C
traceroute -q 1 -z 1 -F 2600:1004:b14b:907e:f86c:e255:4818:1898 1491
Code:
traceroute to 2600:1004:b14b:907e:f86c:e255:4818:1898 (2600:1004:b14b:907e:f86c:e255:4818:1898), 30 hops max, 1491 byte packets
 1  2001:558:6016:29::1 (2001:558:6016:29::1)  30.592 ms
 2  xe-3-2-0-32767-sur01.ruralhillrd.tn.nash.comcast.net (2001:558:162:50::1)  10.390 ms
 3  xe-5-1-11-0-ar01.goodslettvll.tn.nash.comcast.net (2001:558:160:45::1)  12.232 ms
 4  he-7-14-0-0-cr01.350ecermak.il.ibone.comcast.net (2001:558:0:f76f::1)  29.757 ms
 5  he-0-12-0-0-pe04.56marietta.ga.ibone.comcast.net (2001:558:0:f61c::2)  19.568 ms
 6  2001:559::57e (2001:559::57e)  23.733 ms
 7  Loopback0.GW1.MIA19.ALTER.NET (2600:804::22)  34.352 ms
 8  2600:805:41f::22 (2600:805:41f::22)  23.953 ms
 9  nash-b1-v6.telia.net (2001:2000:3018:80::1)  44.215 ms
10  *
11  *
12  *
13  *
14  *
15  *
16  *^C
traceroute -q 1 -z 1 -F 2600:1004:b14b:907e:f86c:e255:4818:1898 1492
Code:
traceroute to 2600:1004:b14b:907e:f86c:e255:4818:1898 (2600:1004:b14b:907e:f86c:e255:4818:1898), 30 hops max, 1492 byte packets
 1  2001:558:6016:29::1 (2001:558:6016:29::1)  32.418 ms
 2  xe-3-2-0-32767-sur01.ruralhillrd.tn.nash.comcast.net (2001:558:162:50::1)  64.307 ms
 3  xe-5-1-11-0-ar01.goodslettvll.tn.nash.comcast.net (2001:558:160:45::1)  28.103 ms
 4  he-7-13-0-0-cr01.350ecermak.il.ibone.comcast.net (2001:558:0:f691::1)  25.888 ms
 5  he-0-12-0-0-pe04.350ecermak.il.ibone.comcast.net (2001:558:0:f8e8::2)  29.698 ms
 6  2600:804:b0f::6d (2600:804:b0f::6d)  19.248 ms
 7  *
 8  2600:804:80f::4a (2600:804:80f::4a)  33.734 ms
 9  nash-b1-v6.telia.net (2001:2000:3018:80::1)  33.404 ms
10  *
11  *
12  *
13  *^C
traceroute -q 1 -z 1 -F 2600:1004:b14b:907e:f86c:e255:4818:1898 1500
Code:
traceroute to 2600:1004:b14b:907e:f86c:e255:4818:1898 (2600:1004:b14b:907e:f86c:e255:4818:1898), 30 hops max, 1500 byte packets
 1  2001:558:6016:29::1 (2001:558:6016:29::1)  65.642 ms
 2  xe-3-2-0-32767-sur01.ruralhillrd.tn.nash.comcast.net (2001:558:162:50::1)  69.454 ms
 3  xe-4-1-11-0-ar03.nashville.tn.nash.comcast.net (2001:558:160:52::1)  12.041 ms
 4  he-7-15-0-0-cr01.350ecermak.il.ibone.comcast.net (2001:558:0:f790::1)  25.466 ms
 5  he-0-12-0-0-pe04.56marietta.ga.ibone.comcast.net (2001:558:0:f61c::2)  18.052 ms
 6  2600:804:b0f::6d (2600:804:b0f::6d)  24.505 ms
 7  Loopback0.GW1.MIA19.ALTER.NET (2600:804::22)  36.901 ms
 8  2600:804:80f::4a (2600:804:80f::4a)  34.679 ms
 9  nash-b1-v6.telia.net (2001:2000:3018:80::1)  49.151 ms
10  *
11  *
12  *
13  *
14  *
15  *
16  *^C
 
Old 05-22-2015, 04:00 PM   #4
Ser Olmy
Senior Member
 
Registered: Jan 2012
Distribution: Slackware
Posts: 3,347

Rep: Reputation: Disabled
(When I said traceroute, I actually meant traceroute6, but it seems your version of traceroute handles both IPv4 and IPv6, and defaults to the latter.)

The traceroute tool uses UDP packets, not ICMP, but it seems the probes are blocked by "nash-b1-v6.telia.net" regardless of size. The trace does tell you the problem is not likely to be related to anything at the Comcast end of the connection, though.

Nevertheless, unless the Galaxy is acting as an IPv6-in-IPv4 tunnel endpoint or is connecting using a PPP/PPPoE/PPPoA client, it very unlikely that MTU issues could be caused by the phone itself. In fact, even if it should turn out that the LTE connection does use PPP, it's still the responsibility of the server end to enforce proper MTU values.
 
Old 05-22-2015, 11:51 PM   #5
psycroptic
Member
 
Registered: Aug 2011
Location: USA
Distribution: ArchLinux - 3.0 kernel
Posts: 349

Original Poster
Rep: Reputation: Disabled
Here is a trace from the phone to the server. Doesn't really tell me much, and the app I found to do IPv6 traceroutes (called "IPv6 and More") doesn't give you an option to set trace packet sizes....

Code:
Traceroute from this device to 2001:558:6016:29:----:----:----:----
Max Hops : 30

[1] 2600:1004:b15b:83eb:0:35:5a7c:fa40

[2]  *

[3] 2001:4888:22:2010:208:2a1:0:1

[4] 2001:4888:22:200e:208:25::

[5] 2001:4888:22:2000:208:2a1::

[6] 2001:4888:22:2005:208:1::

[7] 2001:4888:22:2005:208:1::

[8] 2001:4888:22:1001:208:24::

[9] 2001:2000:3080:6ab::1

[10]  *

[11]  *

[12] 2001:5a0:100:700::1a

[13] 2001:558:0:f790::2

[14] 2001:558:160:52::2

[15] 2001:558:162:50::2

[16] REACHED 2001:558:6016:29:----:----:----:---- 2001:558:6016:29:----:----:----:----:
I tried lowering the server's MTU to 1400, then 1000, and the problem still exists... anything else I could do?
 
Old 05-24-2015, 09:59 PM   #6
psycroptic
Member
 
Registered: Aug 2011
Location: USA
Distribution: ArchLinux - 3.0 kernel
Posts: 349

Original Poster
Rep: Reputation: Disabled
Actually, a quick google search of "ALTER.NET", a PTR shown in some of the traces, indicates many problems Verizon users have experienced with the apparently VZW-owned "ALTER.NET" organization. https://www.google.com/search?q=alte...ter.net&nfpr=1
 
Old 06-03-2015, 02:22 PM   #7
bldewolf
LQ Newbie
 
Registered: Jun 2015
Posts: 5

Rep: Reputation: Disabled
Finally, someone else! I've been seeing the "truncated-ip6 - 8 bytes missing" problem for a couple months, with no solution. I'm on Time Warner Cable. I'm also using an Arch Linux box connected to the modem. What modem do you have? My problems started around the time I switched from an SB6141 to an SB6183. I haven't tried switching back to an SB6141 yet, though.

Have you found a solution? My testing traffic doesn't pass through ALTER.NET, so I can't blame them for my problems.
 
Old 06-03-2015, 02:47 PM   #8
psycroptic
Member
 
Registered: Aug 2011
Location: USA
Distribution: ArchLinux - 3.0 kernel
Posts: 349

Original Poster
Rep: Reputation: Disabled
Actually I have used those exact cable modems, Im on the SB6183, but in addition to the SB6141 I have also tried a Zoom 5341J. I dont think the modem is the problem, and I have no problem connecting from other Comcast v6-enabled locations using the phone, so I think the problem is external to Comcast.

Are you also using an Android phone on Verizon?
 
Old 06-03-2015, 05:04 PM   #9
bldewolf
LQ Newbie
 
Registered: Jun 2015
Posts: 5

Rep: Reputation: Disabled
I've hit the problem with every IPv6 place I've used, though I don't know of any Time Warner Cable locations to test from. I've tested from my phone on T-Mobile, a BuyVM VPS, a Linode VPS, and from my work. My VPSes go through he.net to get to TWC, but my work only passes through Internet2, so it doesn't seem like an intermediary is affecting me.

I've done packet captures using my VPS and found that the before/after isn't just a simple truncation, either. The rejected packets have changed TCP options, however ones that don't get rejected are getting passed through fine. This led me to use scapy to send packets without TCP Options, which never seemed to get mangled (though I might have just been lucky). I've also noticed it only affects the initial SYN/SYNACK of a connection (it will take a ton of retries to connect, but then you can flood traffic across that TCP connection). It makes me think there's a device doing broken connection tracking somewhere in the mix, but I have no idea where or how to find it. Or who to complain to. Or why it only seems to affect us (or me, if we're not really seeing the same issue).
 
Old 06-03-2015, 06:04 PM   #10
psycroptic
Member
 
Registered: Aug 2011
Location: USA
Distribution: ArchLinux - 3.0 kernel
Posts: 349

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by bldewolf View Post
I've also noticed it only affects the initial SYN/SYNACK of a connection (it will take a ton of retries to connect, but then you can flood traffic across that TCP connection).
Yes, I've seen this exact thing. In FTP transfers, it manifests as taking a very long time to start a transfer, but once it's started it transfers normally.

I will try swapping my modem, just as a last ditch type of thing....
 
Old 06-05-2015, 03:49 AM   #11
psycroptic
Member
 
Registered: Aug 2011
Location: USA
Distribution: ArchLinux - 3.0 kernel
Posts: 349

Original Poster
Rep: Reputation: Disabled
A further bit of information.... i believe i have isolated the problem to just Linux connecting clients - an FTP connection from another Comcast connection to the server using Windows does not produce the truncated packets, but connecting from a Linux machine does. I will call in & swap modems tomorrow to determine if this is the cause (at this point it's starting to look a bit like it)
 
Old 06-05-2015, 01:55 PM   #12
psycroptic
Member
 
Registered: Aug 2011
Location: USA
Distribution: ArchLinux - 3.0 kernel
Posts: 349

Original Poster
Rep: Reputation: Disabled
So I have switched back to the Zoom 5341J modem, and the problem has immediately rectified. It appears this may be a firmware problem with the SB6183.
 
Old 06-05-2015, 04:20 PM   #13
bldewolf
LQ Newbie
 
Registered: Jun 2015
Posts: 5

Rep: Reputation: Disabled
Awesome! Thanks for testing. I've got an order in for an SB6141 that I'll use until they sort out the SB6183s, then. I switched to owning the modem when I upgraded to an SB6183, and it's annoying to own the modem but have no say in which buggy firmware it runs.
 
Old 06-07-2015, 04:41 PM   #14
joec@home
Member
 
Registered: Sep 2009
Location: Galveston Tx
Posts: 291

Rep: Reputation: 70
Quote:
Originally Posted by Ser Olmy View Post
Nevertheless, unless the Galaxy is acting as an IPv6-in-IPv4 tunnel endpoint or is connecting using a PPP/PPPoE/PPPoA client, it very unlikely that MTU issues could be caused by the phone itself. In fact, even if it should turn out that the LTE connection does use PPP, it's still the responsibility of the server end to enforce proper MTU values.
A question on this, since both endpoints are native IPv6 and no IPv4 addresses are seen in the routes, do you still think the IPv6-in-IPv4 tunnel function is being activated some where on the route?
 
Old 06-07-2015, 04:42 PM   #15
joec@home
Member
 
Registered: Sep 2009
Location: Galveston Tx
Posts: 291

Rep: Reputation: 70
Additional, you might want to look for choke points on the connection in both directions using mtr instead of traceroute.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Using "normal", vs "low-latency" vs "real-time RT" kernels GTrax Linux - Software 7 07-10-2014 04:34 AM
"squid "excellent hardware but slow browsing and low performance dr.x Linux - Server 1 02-22-2013 02:06 AM
Centos Server Failed @ Bootup: Missing "/sbin/blkid" & "fsck" command not found beagle7 Linux - Newbie 4 08-24-2012 01:33 AM
[SOLVED] KDE - "Control Center" & "Configure Desktop" missing. Brian-M Linux - Newbie 1 04-06-2011 08:17 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Networking

All times are GMT -5. The time now is 04:27 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration