[SOLVED] Our expensive WAN Ethernet doesn't play well with Linux / UNIX

Skaperen · 11-24-2011, 11:24 AM

Quote:

Originally Posted by acid_kewpie

why could duplex not be an issue? mismatched settings can cripple network connections.

one things here is that as i understand it, you're still association switches, cabling, wan and internet in one this. what are your transfer rates like between linux boxes on the same switch etc?

Yes, they can. But it didn't seem to be the OP's issue.

Testing between local machines on the same switch would be a good test for reference.

ethoms · 11-24-2011, 12:34 PM

Quote:

Originally Posted by acid_kewpie

why could duplex not be an issue? mismatched settings can cripple network connections.

one things here is that as i understand it, you're still association switches, cabling, wan and internet in one this. what are your transfer rates like between linux boxes on the same switch etc?

Between Solaris 10 box and FreeBSD box (through the linksys 10/100/1000 switch):
With 100BaseTx full duplex:
scp blah@blah:/blah/blah .
100MB.file 100% 100MB 9.1MB/s 00:11

With 1000baseTx full duplex:
100MB.file 100% 100MB 50.0MB/s 00:02

So, no problem with the NICs or the switch.

@Skaperen:
The switch is unmanaged, I really don't want any thing fancy like VLANs, I just want to split the connection to use some of my 16 public static IP addresses. Even without any switchgear or router etc there is a problem. I will start with tcpdump from your suggestions, I'm really out of my depth at this point BTW. Gotta learn some time I suppose.

ethoms · 11-24-2011, 12:45 PM

Well, this is strange. i turned off PMTU discovery, then set my MTU to 300.

Now look at my route get:

#route get heanet.dl.sourceforge.net
route to: heanet.dl.sourceforge.net
destination: default
mask: default
gateway: xxx.xxx.xxx.113
interface: re1
flags: <UP,GATEWAY,DONE,STATIC>
recvpipe sendpipe ssthresh rtt,msec mtu weight expire
0 0 0 0 300 1 0

THEN, I turn it back to 500 and 1500 but the mtu in my route get stays at 300. I will go down but not up. Again out of my depth here.

ethoms · 11-24-2011, 01:05 PM

Quote:

Originally Posted by Skaperen

I agree at this point that link speed (you aren't even getting near 10 mbps, much less 100 or 1000), MTU, or duplex, is not the issue.

Use tcpdump and watch the traffic. Is it smoothly slow, or does it come in bursts and pause? Are there too many or too few ARP queries/answers? Do they all get answered? If there are anomalies in ARP, are they happening for upstream sending queries to find MACs of your hosts, or are they happening for your hosts finding the MAC of the upstream gateway. Are the MACs for the gateway consistent or do they change?

Can you configure the HP Procurve 1400-8G switch to do VLANs (all ports untagged), with two VLANs, where VLAN #1 connects to the upstream modem, and VLAN #2 is all other ports. Enable routing between these two VLANs (hopefully this feature is present in this switch). If there is an ARP issue, maybe this will change things around.

Why would the tech's Windows laptop work? Maybe it has the upstream gateway statically configured in the ARP table. Maybe it's using a different gateway IP address (yes, this can work when the fibre connection is really bridged instead of routed ... our remote office happens to be connected this way, although they give us a gateway outside of our block of 8, so we can use all 8).

If this still doesn't figure it out, a tool that can insert transparently in an ethernet cable and show the traffic would be useful. I built a small Slackware box with 3 ethernet ports to do this, where the first is a normal networked port, and the two added ports (conveniently on a dual NIC card) are bridged. Then I do tcpdump on the bridge and see what's happening. It has helped diagnose a few problems in the network.

"If there is an ARP issue, maybe this will change things around". That sounds interesting. I think next thing I will do is test with a freshly installed Windows XP laptop. That would rule out whether the ISP engineer had any customisations like ARP table entries. I will report back with the results. On my bike to the office soon, fortunately its only 10 mins ride away.

ethoms · 11-26-2011, 03:25 PM

OK, I got it solved. My slow download issue was attributed to data link speed and duplex. The fibre optic modem is very fussy about what hardware it talks to, auto-negotiation is hit and miss. Even when I bought a managed switch (by Cisco) and forced a port to 100baseTx full duplex, it wouldn't talk to it at all, the modem would simply take down it's ethernet port in protest. Two different windows laptops, one would get 10-20KBps, the other 150-200KBps, both were set to auto detect speed and duplex. So, the engineers laptop was a hit, my Linux laptop was a miss.

Now knowing that the modem only talks 100KBps I bought a Cisco managed switch and forced all ports to 100BaseTx FD. As mentioned above, couldn't get it talk to the modem. So I decided to force 100BaseTx FD on all my servers NICs in the OS. On FreeBSD server, it worked, the linksys switch would deliver full download throughput ~160KBps, but Solaris would still only deliver 10-15KBps. Is this normal? Are these fibre optic modems normally so fussy with hardware compatibility?

So then I decided to use the router that was supplied my connection, but it only has one port in, one port out. Well, I had to learn about routing. I thought a router was for transferring one subnet (a WAN IP block) to another subnet (a LAN). I didn't realize that both subnets (one on each ETH port) can be part of the same public subnet. In my case, one subnet on 255.255.255.254 on ETH0, will actually overlap the subnet 255.255.255.240 on ETH1. The gateway on my servers is still the same.

Now I can download at full speed with my server NICs running at 1000BaseTX full duplex, so server to server data links are maximized. And my email test with a 4MB attachment gets delivered successfully. Yeehaaa!

It turns out that the modem needs to talk to a layer 3 router and doesn't like being split by a layer 2 switch. But not all layer 3 devices are supported.

A big thanks to Shaperen and acid_kewpie for all your help!

ethoms · 11-26-2011, 03:30 PM

I forgot to mention: I had to bridge the two subnets/interfaces.

acid_kewpie · 11-26-2011, 03:46 PM

Excuse me while I do 20 laps of honour...

Skaperen · 11-27-2011, 07:29 PM

Quote:

Originally Posted by ethoms

I forgot to mention: I had to bridge the two subnets/interfaces.

Then it's not routing. But that makes it work and is more flexible.

ethoms · 03-23-2014, 10:40 AM

I know it's an old post now, but I felt I should recap and thank those that helped me.

Thanks acid_kewpie and Skaperen for helping me out.

Well, I've been running nicely for a long while now. The HP router in bridging mode didn't work out, it kept hanging (or at least routing stopped until reboot) about once or twice a week. Anyway, the way I solved it fully was to just put everything on one server, in my case FreeBSD 8.3. I use jails quite a bit for virtualization, each jail having it own IP address. The one other server is on a LAN and I use firewall (ipfilter) to port forward to it. And it's super stable for a long time now.

So, the lesson learnt is that my ISP has a backend (probably Cisco) that doesn't like layer 2. I guess it only wants to talk to one MAC address. That's OK for those that like using expensive Cisco routers as a switch. If I had to have several physcial servers I'd consider pfSense to bridge the internet facing NIC. Or for Linux only advocates, perhaps Endian or IP Fire or Vyatta.