LinuxQuestions.org - How to troubleshoot intermittent, slow speeds over the internet?

- Linux - Networking (https://www.linuxquestions.org/questions/linux-networking-3/)

- - How to troubleshoot intermittent, slow speeds over the internet? (https://www.linuxquestions.org/questions/linux-networking-3/how-to-troubleshoot-intermittent-slow-speeds-over-the-internet-4175736584/)

How to troubleshoot intermittent, slow speeds over the internet?

Server A is Debian 12 and is located at my house directly plugged into an ASUS AX86U router.

Server B is Debian 12 and is located at my brother's directly plugged into an ASUS GT6 router.

I have AT&T 1 gig symmetrical and he has Comcast 1.2 down/35 up.

I recently moved my backup server to my brothers house and -sometimes- when the backup runs, we get about 45-50MB/s but MOST of the time, it's about 3MB/s. In a perfect world, we should be able to get about 90-95MB/s (maxing out my upload speed) but the internet being the internet and peering agreements sometimes being garbage, I was hoping that the 45-50MB/s would be consistent but it's not.

Even before I moved the server to his place, FTP and iperf testing would (on a great day) max out around 50MB/s but most of the time, 3MB/s.... 5MB/s..... just crummy speeds.

Since we're both residential customers, I know that trying to get AT&T or Comcast to solve this problem would be pointless but what testing can I do to 100% rule out anything within our control is causing these totally random speed fluctuations?

I believe the theoretical max speed on a 1GB ethernet is something like 943 Mb/s, 117 MB/s on a typical home network.
So 35 up is probably Mb/s and so at times I might expect peaks at 45-50 Mb/s but that is still an average of about 4 MB/s. As far as I know rsync should use the available bandwidth. Is it because rsync is only as fast as the slowest link?

Quote:

Originally Posted by michaelk (Post 6499142)

Any good at interupting iperf results?

Code:

[ ID] Interval                      Transfer            Bandwidth

[  1] 0.0000-10.5945 sec          121 MBytes          95.4 Mbits/sec



[ ID] Interval                      Transfer        Bandwidth

[  1] 0.0000-10.8670 sec          102 MBytes          78.4 Mbits/sec



[ ID] Interval                      Transfer        Bandwidth

[  1] 0.0000-20.0548 sec          1.40 GBytes          598 Mbits/sec



[ ID] Interval                      Transfer        Bandwidth

[  1] 0.0000-2.0000 sec          126 MBytes          528 Mbits/sec

[  1] 2.0000-4.0000 sec          143 MBytes          601 Mbits/sec

[  1] 4.0000-6.0000 sec          142 MBytes          596 Mbits/sec

[  1] 6.0000-8.0000 sec          142 MBytes          593 Mbits/sec

[  1] 8.0000-10.0000 sec          144 MBytes          603 Mbits/sec

[  1] 10.0000-12.0000 sec          147 MBytes          616 Mbits/sec

[  1] 12.0000-14.0000 sec          144 MBytes          605 Mbits/sec

[  1] 14.0000-16.0000 sec          142 MBytes          597 Mbits/sec

[  1] 16.0000-18.0000 sec          145 MBytes          609 Mbits/sec

[  1] 18.0000-20.0000 sec          144 MBytes          603 Mbits/sec

[  1] 0.0000-20.0727 sec          1.39 GBytes          593 Mbits/sec



[ ID] Interval                      Transfer            Bandwidth

[  1] 0.0000-10.9962 sec          95.8 MBytes          73.0 Mbits/sec



[ ID] Interval                      Transfer    Bandwidth

[  1] 0.0000-2.0000 sec          56.8 MBytes  238 Mbits/sec

[  1] 2.0000-4.0000 sec          18.0 MBytes  75.5 Mbits/sec

[  1] 4.0000-6.0000 sec          11.0 MBytes  46.1 Mbits/sec

[  1] 6.0000-8.0000 sec          8.25 MBytes  34.6 Mbits/sec

[  1] 8.0000-10.0000 sec        6.88 MBytes  28.8 Mbits/sec

[  1] 10.0000-12.0000 sec          5.50 MBytes  23.1 Mbits/sec

[  1] 12.0000-14.0000 sec          4.00 MBytes  16.8 Mbits/sec

[  1] 14.0000-16.0000 sec          8.00 MBytes  33.6 Mbits/sec

[  1] 16.0000-18.0000 sec          6.88 MBytes  28.8 Mbits/sec

[  1] 18.0000-20.0000 sec          6.75 MBytes  28.3 Mbits/sec

[  1] 20.0000-20.7718 sec          128 KBytes  1.36 Mbits/sec

[  1] 0.0000-20.7718 sec          132 MBytes  53.4 Mbits/sec



iperf3

[ ID] Interval          Transfer    Bitrate        Retr  Cwnd

[  5]  0.00-2.00  sec  14.7 MBytes  61.8 Mbits/sec    4    184 KBytes      

[  5]  2.00-4.00  sec  9.64 MBytes  40.4 Mbits/sec    0    219 KBytes      

[  5]  4.00-6.00  sec  9.62 MBytes  40.4 Mbits/sec    1    206 KBytes      

[  5]  6.00-8.00  sec  12.0 MBytes  50.3 Mbits/sec    0    230 KBytes      

[  5]  8.00-10.00  sec  10.9 MBytes  45.6 Mbits/sec    1    212 KBytes      

[  5]  10.00-12.00  sec  10.8 MBytes  45.1 Mbits/sec    1    184 KBytes      

[  5]  12.00-14.00  sec  9.56 MBytes  40.1 Mbits/sec    0    221 KBytes      

[  5]  14.00-16.00  sec  9.63 MBytes  40.4 Mbits/sec    2    143 KBytes      

[  5]  16.00-18.00  sec  8.39 MBytes  35.2 Mbits/sec    0    182 KBytes      

[  5]  18.00-20.00  sec  9.63 MBytes  40.4 Mbits/sec    1    165 KBytes      

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval          Transfer    Bitrate        Retr

[  5]  0.00-20.00  sec  105 MBytes  44.0 Mbits/sec  10            sender

[  5]  0.00-20.04  sec  102 MBytes  42.8 Mbits/sec                  receiver



iperf Done.



[ ID] Interval          Transfer            Bitrate        Retr          Cwnd

[  5]  0.00-1.00  sec  38.0 MBytes          319 Mbits/sec    6          1.46 MBytes      

[  5]  1.00-2.00  sec  17.5 MBytes          147 Mbits/sec    7            428 KBytes      

[  5]  2.00-3.00  sec  12.5 MBytes          105 Mbits/sec    0            461 KBytes      

[  5]  3.00-4.00  sec  10.0 MBytes          83.9 Mbits/sec  1            351 KBytes      

[  5]  4.00-5.00  sec  7.50 MBytes          62.9 Mbits/sec  2            192 KBytes      

[  5]  5.00-6.00  sec  5.00 MBytes          41.9 Mbits/sec  0            215 KBytes      

[  5]  6.00-7.00  sec  6.25 MBytes          52.4 Mbits/sec  2            116 KBytes      

[  5]  7.00-8.00  sec  2.50 MBytes          21.0 Mbits/sec  1          99.0 KBytes      

[  5]  8.00-9.00  sec  2.50 MBytes          21.0 Mbits/sec  0            120 KBytes      

[  5]  9.00-10.00  sec  3.75 MBytes          31.5 Mbits/sec  0            140 KBytes      

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval          Transfer    Bitrate                Retr

[  5]  0.00-10.00  sec  106 MBytes  88.5 Mbits/sec          19            sender

[  5]  0.00-10.04  sec  102 MBytes  85.4 Mbits/sec                  receiver



iperf Done.



[ ID] Interval          Transfer    Bitrate        Retr          Cwnd

[  5]  0.00-1.00  sec  45.4 MBytes  381 Mbits/sec    3          2.09 MBytes      

[  5]  1.00-2.00  sec  36.2 MBytes  304 Mbits/sec    4          786 KBytes      

[  5]  2.00-3.00  sec  22.5 MBytes  189 Mbits/sec    0            848 KBytes      

[  5]  3.00-4.00  sec  13.8 MBytes  115 Mbits/sec    3            454 KBytes      

[  5]  4.00-5.00  sec  10.0 MBytes  83.9 Mbits/sec  1            351 KBytes      

[  5]  5.00-6.00  sec  7.50 MBytes  62.9 Mbits/sec  1            272 KBytes      

[  5]  6.00-7.00  sec  7.50 MBytes  62.9 Mbits/sec  0            288 KBytes      

[  5]  7.00-8.00  sec  7.50 MBytes  62.9 Mbits/sec  1            219 KBytes      

[  5]  8.00-9.00  sec  3.75 MBytes  31.5 Mbits/sec  2            122 KBytes      

[  5]  9.00-10.00  sec  3.75 MBytes  31.5 Mbits/sec  0            141 KBytes      

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval          Transfer    Bitrate        Retr

[  5]  0.00-10.00  sec  158 MBytes  132 Mbits/sec  15            sender

[  5]  0.00-10.04  sec  155 MBytes  129 Mbits/sec                  receiver



iperf Done.



[ ID] Interval          Transfer    Bitrate        Retr          Cwnd

[  5]  0.00-1.00  sec  21.5 MBytes  180 Mbits/sec    6    652 KBytes      

[  5]  1.00-2.00  sec  18.8 MBytes  157 Mbits/sec    0    699 KBytes      

[  5]  2.00-3.00  sec  17.5 MBytes  147 Mbits/sec    4    519 KBytes      

[  5]  3.00-4.00  sec  13.8 MBytes  115 Mbits/sec    1    396 KBytes      

[  5]  4.00-5.00  sec  11.2 MBytes  94.4 Mbits/sec  0    433 KBytes      

[  5]  5.00-6.00  sec  11.2 MBytes  94.3 Mbits/sec  0    455 KBytes      

[  5]  6.00-7.00  sec  12.5 MBytes  105 Mbits/sec    0    467 KBytes      

[  5]  7.00-8.00  sec  10.0 MBytes  83.9 Mbits/sec  2    355 KBytes      

[  5]  8.00-9.00  sec  8.75 MBytes  73.4 Mbits/sec  1    276 KBytes      

[  5]  9.00-10.00  sec  7.50 MBytes  62.9 Mbits/sec  0    294 KBytes      

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval          Transfer    Bitrate        Retr

[  5]  0.00-10.00  sec  133 MBytes  111 Mbits/sec  14            sender

[  5]  0.00-10.04  sec  129 MBytes  108 Mbits/sec                  receiver



[ ID] Interval          Transfer    Bitrate        Retr  Cwnd

[  5]  0.00-1.00  sec  60.8 MBytes  510 Mbits/sec    0  3.15 MBytes      

[  5]  1.00-2.00  sec  78.8 MBytes  661 Mbits/sec    0  3.15 MBytes      

[  5]  2.00-3.00  sec  77.5 MBytes  650 Mbits/sec    0  3.15 MBytes      

[  5]  3.00-4.00  sec  77.5 MBytes  650 Mbits/sec    0  3.15 MBytes      

[  5]  4.00-5.00  sec  77.5 MBytes  650 Mbits/sec    0  3.15 MBytes      

[  5]  5.00-6.00  sec  78.8 MBytes  661 Mbits/sec    0  3.15 MBytes      

[  5]  6.00-7.00  sec  77.5 MBytes  650 Mbits/sec    0  3.15 MBytes      

[  5]  7.00-8.00  sec  76.2 MBytes  640 Mbits/sec    0  3.15 MBytes      

[  5]  8.00-9.00  sec  76.2 MBytes  640 Mbits/sec    0  3.15 MBytes      

[  5]  9.00-10.00  sec  78.8 MBytes  661 Mbits/sec    0  3.15 MBytes      

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval          Transfer    Bitrate        Retr

[  5]  0.00-10.00  sec  760 MBytes  637 Mbits/sec    0            sender

[  5]  0.00-10.04  sec  759 MBytes  634 Mbits/sec                  receiver



iperf Done.

[ ID] Interval          Transfer    Bitrate        Retr  Cwnd

[  5]  0.00-1.00  sec  59.5 MBytes  499 Mbits/sec    0  6.05 MBytes      

[  5]  1.00-2.00  sec  77.5 MBytes  650 Mbits/sec    0  6.05 MBytes      

[  5]  2.00-3.00  sec  77.5 MBytes  650 Mbits/sec    0  6.05 MBytes      

[  5]  3.00-4.00  sec  77.5 MBytes  650 Mbits/sec    0  6.05 MBytes      

[  5]  4.00-5.00  sec  77.5 MBytes  650 Mbits/sec    0  6.05 MBytes      

[  5]  5.00-6.00  sec  77.5 MBytes  650 Mbits/sec    0  6.05 MBytes      

[  5]  6.00-7.00  sec  78.8 MBytes  661 Mbits/sec    0  6.05 MBytes      

[  5]  7.00-8.00  sec  77.5 MBytes  650 Mbits/sec    0  6.05 MBytes      

[  5]  8.00-9.00  sec  77.5 MBytes  650 Mbits/sec    0  6.05 MBytes      

[  5]  9.00-10.00  sec  76.2 MBytes  640 Mbits/sec    0  6.05 MBytes      

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval          Transfer    Bitrate        Retr

[  5]  0.00-10.00  sec  757 MBytes  635 Mbits/sec    0            sender

[  5]  0.00-10.04  sec  757 MBytes  632 Mbits/sec                  receiver



iperf Done.



[ ID] Interval          Transfer    Bitrate        Retr  Cwnd

[  5]  0.00-1.00  sec  38.4 MBytes  322 Mbits/sec    0  2.87 MBytes      

[  5]  1.00-2.00  sec  75.0 MBytes  629 Mbits/sec    0  3.09 MBytes      

[  5]  2.00-3.00  sec  73.8 MBytes  619 Mbits/sec    0  3.09 MBytes      

[  5]  3.00-4.00  sec  73.8 MBytes  619 Mbits/sec    0  3.09 MBytes      

[  5]  4.00-5.00  sec  72.5 MBytes  608 Mbits/sec    0  3.09 MBytes      

[  5]  5.00-6.00  sec  75.0 MBytes  629 Mbits/sec    0  3.09 MBytes      

[  5]  6.00-7.00  sec  73.8 MBytes  619 Mbits/sec    0  3.09 MBytes      

[  5]  7.00-8.00  sec  73.8 MBytes  619 Mbits/sec    0  3.09 MBytes      

[  5]  8.00-9.00  sec  75.0 MBytes  629 Mbits/sec    0  3.09 MBytes      

[  5]  9.00-10.00  sec  73.8 MBytes  619 Mbits/sec    0  3.09 MBytes      

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval          Transfer    Bitrate        Retr

[  5]  0.00-10.00  sec  705 MBytes  591 Mbits/sec    0            sender

[  5]  0.00-10.04  sec  704 MBytes  589 Mbits/sec                  receiver



iperf Done.



[ ID] Interval          Transfer    Bitrate        Retr  Cwnd

[  5]  0.00-1.00  sec  56.5 MBytes  474 Mbits/sec    0  6.01 MBytes      

[  5]  1.00-2.00  sec  75.0 MBytes  629 Mbits/sec    0  6.01 MBytes      

[  5]  2.00-3.00  sec  73.8 MBytes  619 Mbits/sec    0  6.01 MBytes      

[  5]  3.00-4.00  sec  73.8 MBytes  619 Mbits/sec    0  6.01 MBytes      

[  5]  4.00-5.00  sec  73.8 MBytes  619 Mbits/sec    0  6.01 MBytes      

[  5]  5.00-6.00  sec  73.8 MBytes  619 Mbits/sec    0  6.01 MBytes      

[  5]  6.00-7.00  sec  73.8 MBytes  619 Mbits/sec    0  6.01 MBytes      

[  5]  7.00-8.00  sec  73.8 MBytes  619 Mbits/sec    0  6.01 MBytes      

[  5]  8.00-9.00  sec  73.8 MBytes  619 Mbits/sec    0  6.01 MBytes      

[  5]  9.00-10.00  sec  75.0 MBytes  629 Mbits/sec    0  6.01 MBytes      

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval          Transfer    Bitrate        Retr

[  5]  0.00-10.00  sec  723 MBytes  606 Mbits/sec    0            sender

[  5]  0.00-10.04  sec  722 MBytes  604 Mbits/sec                  receiver



iperf Done.



[ ID] Interval          Transfer            Bitrate        Retr          Cwnd

[  5]  0.00-1.00  sec  42.6 MBytes  358 Mbits/sec    3          1.56 MBytes      

[  5]  1.00-2.00  sec  27.5 MBytes  231 Mbits/sec    3            590 KBytes      

[  5]  2.00-3.00  sec  13.8 MBytes  115 Mbits/sec    1            452 KBytes      

[  5]  3.00-4.00  sec  10.0 MBytes  83.9 Mbits/sec    3            246 KBytes      

[  5]  4.00-5.00  sec  6.25 MBytes  52.4 Mbits/sec    1            192 KBytes      

[  5]  5.00-6.00  sec  6.25 MBytes  52.4 Mbits/sec    1            150 KBytes      

[  5]  6.00-7.00  sec  2.50 MBytes  21.0 Mbits/sec    4          86.3 KBytes      

[  5]  7.00-8.00  sec  2.50 MBytes  21.0 Mbits/sec    0            105 KBytes      

[  5]  8.00-9.00  sec  2.50 MBytes  21.0 Mbits/sec    1          90.5 KBytes      

[  5]  9.00-10.00  sec  3.75 MBytes  31.5 Mbits/sec    0            109 KBytes      

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval          Transfer    Bitrate        Retr

[  5]  0.00-10.00  sec  118 MBytes  98.7 Mbits/sec  17            sender

[  5]  0.00-10.04  sec  114 MBytes  95.2 Mbits/sec                  receiver



iperf Done.

It's likely because your next hop router is shared with many.
What technology is provided by your ISP for connectivity does matter.
Is it xDSL, FTTH, Wireless, etc?

Quote:

Originally Posted by yvesjv (Post 6499330)

It's likely because your next hop router is shared with many.
What technology is provided by your ISP for connectivity does matter.
Is it xDSL, FTTH, Wireless, etc?

FTTH.

There's also traceroute. You can check where you're going. Long ping times = heavy loading.

I once subscribed to a site which was being served by an AWS server in Dublin's Citywest Industrial Estate, not 5 miles from me. But my route to it went through trunk servers to the Excited States, through some bottleneck, and back from the Excited States to City West :banghead:

Mind you, I did find that one of it's 3 web servers gave me a more sane route, and used that for the duration of my subscription.

Quote:

Originally Posted by business_kid (Post 6499338)

Ha, yep, sometimes routing is crazy.

A traceroute to my brother didn't reveal anything horrible. The max time I saw on any hop was 30ms.

Let's do a simple test.
If you do not have "mtr" installed, install it.

https://www.bitwizard.nl/mtr/
https://www.tecmint.com/mtr-a-networ...ool-for-linux/

I've more than once used it to prove the paid for service was bad or the router was crap.
It's actually fun to use the combinations of packet sizes, intermittence and tcp or udp.
Then printscreen and timestamp your evidence.

Example:-
mtr -rnwc 100 -i 0.1 -s 1492 --tcp www.google.com

Quote:

Originally Posted by yvesjv (Post 6499527)

To my brother:

Code:

Start: 2024-05-02T20:22:16-0400

HOST: server-pc      Loss%  Snt  Last  Avg  Best  Wrst StDev

  1.|-- 10.0.0.1    81.0%    64  1032. 340.6  0.4 1032. 502.0

  2.|-- 192.168.1.254  81.0%    64    1.6 339.1  1.2 3039. 898.7

  3.|-- ???            100.0    64    0.0  0.0  0.0  0.0  0.0

  4.|-- ???            100.0    64    0.0  0.0  0.0  0.0  0.0

  5.|-- ???            100.0    64    0.0  0.0  0.0  0.0  0.0

  6.|-- 32.130.89.25  88.9%    64  1022. 1173.  15.0 3061. 1361.1

  7.|-- 192.205.32.142  0.0%    64  17.7  18.3  15.7  36.2  3.3

  8.|-- 96.110.38.153  1.6%    64  26.0  26.8  24.0  40.1  2.2

        96.110.38.157 

        96.110.38.145 

        96.110.38.149 

  9.|-- 96.110.42.162  0.0%    64  27.8  29.0  25.1 108.5  10.3

        96.110.42.174 

        96.110.42.170 

        96.110.42.166 

 10.|-- 162.151.152.10  1.6%    64  30.5  31.0  29.3  33.1  1.0

 11.|-- 96.110.214.50  0.0%    63  30.9  29.7  27.9  33.0  1.0

 12.|-- ???            100.0    63    0.0  0.0  0.0  0.0  0.0

mtr: Unexpected mtr-packet error

To Google.com:

Code:

Start: 2024-05-02T20:24:23-0400

HOST: server-pc      Loss%  Snt  Last  Avg  Best  Wrst StDev

  1.|-- 10.0.0.1    80.0%  100  7124. 2340.  0.7 7205. 3027.0

  2.|-- 192.168.1.254  82.0%  100  7106. 1876.  1.3 7303. 3003.5

  3.|-- ???            100.0  100    0.0  0.0  0.0  0.0  0.0

  4.|-- ???            100.0  100    0.0  0.0  0.0  0.0  0.0

  5.|-- ???            100.0  100    0.0  0.0  0.0  0.0  0.0

  6.|-- ???            100.0  100    0.0  0.0  0.0  0.0  0.0

  7.|-- 32.130.91.19    0.0%  100  19.8  18.0  16.0  19.8  1.1

  8.|-- 12.255.10.54    0.0%  100  1031.  29.2  15.4 1031. 101.4

        12.255.10.56  

        12.255.10.44  

  9.|-- 209.85.142.117  0.0%  100  18.6  17.9  15.8  20.6  1.2

        142.251.64.199 

 10.|-- 142.251.60.19    0.0%  100  17.2  16.7  13.7  20.0  1.4

        142.251.60.17  

 11.|-- 142.250.191.238  0.0%  100  16.0  17.0  14.4  20.8  1.4

No idea what parameters you used.

Quote:

Originally Posted by yvesjv (Post 6499683)

No idea what parameters you used.

Sorry, I used the same parameters you typed in your example.

Quote:

Originally Posted by road hazard (Post 6499741)

Sorry, I used the same parameters you typed in your example.

All good.
Try changing the packet size to default.
And also change the interval to default.
For more info try in cli: man mtr

Quote:

Originally Posted by yvesjv (Post 6499854)

All good.
Try changing the packet size to default.
And also change the interval to default.
For more info try in cli: man mtr

Is this the culprit? IP: 32.130.89.25 (see attached image... don't know how to paste it 'in-line' as part of the message.)

No idea what it is... looks like it belongs to AT&T. How would I go about getting them engaged or to even care? Or maybe I don't know how to interpret the data and what I'm seeing is normal?!

No idea about AT&T and ISPs in the US but would not be worse than here in Aus I hope.

What I like to do when performing these tests is to transfer a huge file at the same time.
Take for example the FreeBSD amd64 iso, it's over 4G.
https://www.freebsd.org/where/

Then perform tests with mtr while transferring direct from the hdd of a local laptop to another hdd on the remote server.
The tests are done with tcp and also udp.
Don't forget the time stamping for every test.

Once the evidence is gathered, then escalate to your ISP(s) Customer Service teams.
They should have SLAs in place for paying business customers.

I'm curious: Are you and your brother in separate towns?

Quote:

Originally Posted by yvesjv (Post 6499890)

We live in the same state and are about 2 hours away from each other.

Since we're both just average home users, it will be difficult/almost impossible to get an issue like this brought to the attention of somebody at my ISP with the power to fix it. From my limited knowledge, it appears that whatever that hop is, there is massive packet loss happening there.