Advice on traffic shaping/control/routing?
 

Hey guys,

In the past, my ISP (a local DSL provider in the US) simply provided their customers with an ADSL box that more or less was a bridge. The connected machine would get my real world-routable IP, and I had that connection going into my Linux box on one Ethernet interface; my Linux box then did routing, NAT, port mapping, etc. to another Ethernet interface connected to a switch for my LAN. In the past everything worked just fine and I've used that setup for nearly 7 years on this ISP.

Now however, my ISP has changed their setup and since they've done this I've been having net performance issues horribly. The new setup involves a Linux-based ADSL router that does it own NAT and provides my Linux box a 172.16.200.x IP, which I was able to force to a static 172.16.200.2. I inquired about making this new box use a bridge mode like the old box did, and was told it wasn't an option because "our infrastructure doesn't support that", whatever that means. (Sometimes I think ISP's throw out canned jibberish just to shut people up; a friend of mine has not yet had his box replaced as they're rolling them out slowly, and his machine still gets the real world IP, so clearly it's not an "infrastructure" problem.)

The ADSL router's advanced routing configuration is locked by my ISP and thus I'm unable to do any port mapping, custom firewalling and other fancy routing that I do on my Linux box on it; thus, I have to deal with double-NAT so I can still do that routing as desired. I simply told the ADSL router to treat 172.16.200.2 as its DMZ host.

Anyway, the real problem. For the most part, this works. Incoming connections still get directed through my Linux box to the correct LAN machine and all that. The only problem is since then, some serious performance issues have been cropping up. The most significant is that when one download is in progress, that is able to run at full link speed, the connection is entirely saturated and any other requests to any other services on the Web are impossible because the download has claimed all of the available downstream bandwidth. It is severe enough that if a single download is running, and I try to start a second, assuming the second download even starts prior to timing out, I can expect 1-2KB/sec on that transfer until the first one is done; when the first one completes, the second one will suddenly speed up to link speed.

Additionally, at random times, connections seem to stall without reason, download speeds temporarily plummet and then just as quickly return to normal (from high-grade servers like Microsoft's, which never exhibited that behavior on the old ADSL box), sites that clearly can push more bits are topping out at slow (~50KB/sec) speeds, and then restarting the download fixes it... and so on.

I share my LAN with a roommate and thus we are constantly fighting for link usage. If one of us starts downloading, the other has no net access effectively until the download completes. It's extremely annoying.

I'm looking for some advice to see if I can do anything in my own Linux box to try to enable some fair packet queueing to help mitigate these problems. Although, as I have no access to anything on the router (I only know it's Linux-based from some of the status outputs in its Web interface, which clearly include reports from GNU "uptime" and "free") I have no idea what sorts of queueing disciplines the router is using. Of course, trying to get real technical help from any ISP is an exercise in patience, as you must wade through layers and layers of "newbie help" and even then you often can't get to a tech who really understands your situation, because some other tech will just cut you off with "we don't support that" just to get you out of the way.

To summarize:

Thanks in advance for any help, and if any more info is needed please let me know.

fm

Can you tell us, how does your and another computer connect to internet now?

And how is called what ADSL router? Is it switch/router?

They connect via my Linux box. Here's a better diagram that should illustrate it better:

So to summarize, the DSL router has two interfaces: one with my real world-routable IP, one staticly assigned to 172.16.200.1. The 172 link is connected to an Ethernet card in my Linux box and my Linux box has 172.16.200.2 on that interface. A second interface in my linux box has 192.168.1.1 and runs DHCP to dish out 192.168.1.x IPs to my machines.

Here is the output I get when I access my router:

As you can see this is clearly Linux. It looks like I do have access to an "adsl" command that looks like this:

Although I'm not familiar enough with DSL technical details to understand if that command would help me. Plus of course if I mess anything up, my ISP will do nothing to help me except charge me a service fee and drop in a new router.

The following are all the commands available to me via telnet:

The following is available to me via the Web interface:

The modem is Comtrend branded.

Advice?

fm

Ok, thanks, but now you really do not need to connect everything trough your linux computer. If ADSL router has dhcp, use it. So you can connect everything to switch and switch to ADSL.
Switch will divide traffic between consumers.

You can try, it will be easy to change all back.

The reason I still use my linux box for routing is because, as I do not have access to the deep routing control in the router, I am unable to do some of the port mapping, packet shaping and so on that I currently do with my linux box. For example, mapping an external port to a port on an internal LAN machine. I also already do use tc to control some traffic rates to clients - sometimes per client (e.g. a machine dedicated to an FTP site is restricted to 32K/sec up, but no other machine is blocked) and so on. Additionally, I run an internal nameserver. Sure, I could configure each and every individual private machine with a static IP, but that's seriously just a pain and I'm used to running dhcpd on my linux box and letting it assign static IPs to certain MACs and so on.

So I still need the power of my Linux box in between my LAN and the network.

fm

If you want to map external port to some internal IP, you HAVE to do it on ADSL router.
If it has NAT and act like a router.


Does you TC rules affect on local connections, or on transit only?


Does ADSL modem have DHCP and NAT inside. Get manual and find out it.
Write an email to ISP, let them explain how it works and what it can do.

Correct. This is why I do not use the router's features. I want to map ports to internal IPs, however the ADSL router (which, yes, does provide DHCP and NAT to the internal interface) does not allow me to do this.

Regarding my ISP, I spoke with someone I know well at my ISP and was told the following: that the "official policy" bans any open services (as most ISPs do), however, they have never taken any steps to scan ports, block ports, etc. unless they receive complaints from other users or external entities. I have run services on my host for all 8 years of my service with this ISP, and they actually know that, but as I'm responsible with things I run (i.e. they're only run for myself/I use appropriate bandwidth throttling/etc) they have never taken issue.

However, their tech staff decided to install these new routers, and as it is in policy that servers can't be run, they're basically not going to help you do so. While I can set a DMZ host (which is what I did, I simply pointed all inbound traffic to my Linux box's internal IP) and still run my services without issue (and again, this guy knows that but it's again an issue of as long as it isn't bothering anyone). The problem is, as I said, that my ISP will never "help" you run services, i.e. allow you to do port mapping and traffic shaping on inbound connections in their new routers.

Getting back to the original issue of this post, the issue of running services/mapping ports/etc. is not a problem at all, as simply setting my Linux box as DMZ host has solved that. The issue, if you recall, is that at some point, packet prioritization is failing to ensure fair packet queueing to the LAN, such so that one computer can jeopardize the entire bandwidth by simply downloading a file. I am assuming that there has to be something I can do in my own Linux box using tc or similar to enforce fair packet dequeueing from the internet interface to the LAN interface.

Right now I use tc to limit packets outbound to the Internet (as I stated above, limiting bandwidth for services I run so as not to be unfair to the ISP/its other customers).

fm

Take a look at wondershaper. It's a shell script that uses tc to control bandwith. Install it on your "Gateway" right after the modem.
Herese a link that describes the setup of whondershaper

http://ubuntuforums.org/showthread.php?t=25911

That means, that ADSL modem acts like bridge, it simply sends everything to inside.

As I understand you have fixed UP/DOWN link speed, which should be stated in the contract. And you can use it fully as you like - UP link and DOWN link.

To find the answer to your problem, you need to implement graphical tool for measure up/down link speed. Why, it is very important, because, there are many causes that can give the loss connectivity. Some of them: tc (very good) shapes only speed, it prevent one IP uses all bandwidth, but it allow to make connections. Some time cheap ADSL modem simply can't handle many connection at the same time. Another, outbound connections can "eat" all bandwidth, not one consumer, but all available. In that case new connection simply won't start.
Another - your ADSL modem might be weak and can't handle too many connection (remember that you have to count all of them in/out bound).
Another - your ISP cheating and do not provide full bandwidth.

To sum up, you need a graphical tool to measure IN and OUT bound speed, and when you can find the answer what is going on.

nimnull22, please don't take this the wrong way but I do not think you're fully understanding the problem at hand. I am not a newbie at networking and have a lot of IT experience, but not as much with routing and advanced traffic shaping, which is what I'm studying right now to see if I can solve this problem.

In the past, I had an ADSL modem that was indeed a raw bridge. What this meant, was that it simply forwarded traffic via the DSL interface to my ISP. My Linux box would receive its real world-routable IP on the WAN interface.

The new router, however, is no longer a bridge. Instead, it's a NAT router. This means that it itself grabs the real world-routable IP, and passes my Linux box an internal NAT IP (172.16.200.2). What I asked my ISP, and what they denied due to their silly excuses, is to convert this DSL box into a bridge, so it simply passes data through like you described. Therefore, I'm stuck with two layers of NAT. And I think that is the source of the problem.

In the past, when the ADSL box was simply a bridge, I had no issues with multiple connections. Linux tended to implement fair queueing automatically. Meaning, if I and my roommate were both downloading, we both got approximately half of my link speed. If only one of us was downloading, we could utilize all the speed, but as soon as the other started downloading, the first person to be downloading would see their speed drop to half the allocated link speed.

On the new system, if one person starts downloading, the other person cannot do anything and experiences massive packet loss. This means that no longer are packets being fairly distributed, and one user can "steal" all the available bandwidth utilization and cause all other connections to time out.

I'm thinking what I may need to do is one of the following:

1. Implement fair packet dequeueing on the WAN interface on my Linux box for outbound traffic. This would hopefully ensure that each machine gets to have equal chances to "acknowledge" its receipt of packets, and hopefully that will straighten the rest out
2. I may instead need to implement fair packet dequeueing on the LAN interface, setup one class so that actual traffic from my Linux box itself to the LAN is not interfered with, then setup another to ensure that the maximal downlink from the net is never 100% utilized, which would hopefully solve the problem in a similar fashion.

The only remaining issue is that we don't know what that router is doing. It may be doing some sort of "automatic ACK" or its own weird forms of packet shaping, and as I said I have no access to learn exactly what it is doing. This is the biggest interference to me figuring out a solution to this.

zhjim, I will look at your link. As I've been really diving into tc lately trying to really understand it, hopefully that will be helpful to review a script and implement it.


fm

Ok, I see you know what to do. Good luck.

I believe that in the above quote somewhere, you hit the nail right on the head -- traffic shaping is what you need, and some TOS (Type of Service) packet header mangling couldn't hurt either. TOS mangling (done using iptables mangle table) alters the "priority" value of certain packets, like ACKs, so that they are pushed through to the other end faster. In and of themselves, those packets don't change the traffic speed directly, but they do make the end-end linkage more efficient/responsive.

The "trouble" with tc (besides the brutal lack of good documentation) when used to do queueing discipline to share bandwidth amongst machines on a single connection, is that you really need to know fairly accurately, what sort of average maximum reliable speed you can expect from the internet connection. Without knowing this, one of two things happen: 1) you make the discipline too tight, and get slow speeds all around, or 2) your disciplines add up to a total Maximum potential traffic speed that is greater than that provided by the ISP or which your connection can handle, and you're back to square one with one or two transfers saturating the network and the other machines starving.

I found it took a lot of patience and tiny tweaks, and careful observation of the results of those tweaks, to get qdiscs to work effectively.

NOTE: When I did this, I was setting up my tc qdiscs on my Linux firewall box with 3 NICs (well, two NICs and a modem), which masqueraded a SLOOOOW dial-up connection for sharing between two desktop machines. The ridiculous slowness of the dial-up connection made for very tricky balancing of traffic between the two machines, because realistically, it was near impossible to divide up 3 kb/sec between two machines :/
With your DSL connection, you should have a lot more leeway either way, so it shouldn't be so finicky.

I initially had the same trouble as you: one transfer saturated the connection, and nobody else could do anything; connections timed out all over the place, etc..
What I did was divide the bandwidth potential in half first, between the two desktop machines, with the condition that each desktop could have a MAX of the FULL bandwidth if the other desktop machine was using none. Then, I subdivided each desktop machines allotment in halves again, one half for http and one half for ftp traffic; this allowed me to be downloading something via ftp, but still able to browse the internet via http. Like I said, it was a slow connection, and as you can imagine, this was brutally slow again, but it worked!
AND, it eliminated that effect of transfers going fast-fast-fast and then suddenly dropping to nothing, then going fast fast fast and dropping to nothing again, over and over. (That's caused by the ISP having cached the transfer for you, and trying to stuff it over the line to you faster than you can receive it, which causes a hiccup while your end tells the ISP to HOLD ON A MINUTE, I'm Full!! After the hiccup, it begins again.)

The ISP equipment is semi-intelligent in that it can learn what is the maximum speed you can handle, and adjust traffic accordingly; the trouble is, the limit it determines is usually too high, causing the hiccups at regular intervals. By using qdiscs, the idea is to limit MAX traffic rate to *just below* the maximum possible bandwidth. If you can get it just right, the ISP will learn to send the traffic at *just* the right rate, just under the max, and the transfer will flow along at a steady rate the whole time, and the hiccups will not happen.

Now, I'm not a networking guru, so I really have little idea what exactly your SWITCH does, other than act a bit like a router in that it allows a number of machines to share one NIC. The difference between a router and a switch, for the sake of this conversation, I really don't quite know.
That said, if I were to make a direct comparison between your setup and my dial-up connection, I would be putting the tc qdisks on the LINUX BOX 192.168.1.1, and at that location, it would be able to discipline traffic. BUT: I haven't read lately about tc, but when I set mine up, I queued based on NIC name on the firewall box, not by IP of the machine connected to that NIC (IIRC), so since you're using a switch there, and each machine on the switch is effectively using the same NIC on the WAN side of the switch, I guess you'd want to qdisc by IP address, not by NIC. If tc can do this, great! But I don't know either way if it can or cannot.

So, sorry for the long windedness! I hope you got something out of this, and if you would like to have a look at my fairly simple tc script (which I haven't even looked at since getting wireless broadband and tossing the dial-up) I will dig it out and post it for you. I may have to boot up the firewall machine to find it, as I don't know if I have the qdisc script on this desktop machine or not.

PS - IIRC again, I built both HTB and CBQ disciplining modules for my kernel, but for the tc stuff I did, I think I used CBQ method, because the documentation was better, and I could find a few snippets of code on the net to learn from. (Don't quote me on this!)

PPS - Forgot to mention: you wrote, "..we don't know what that router is doing.." -- and yes, that's true, but since we don't know, and can't change it anyhow, may as well not worry about it. FWIW, most routers of the home-user variety offer options for some TOS functionality these days, but I can't say how good or bad or otherwise their TOS ability is vs. iptables TOS ability. I would guess though, that the ISP-provided router also has some form of TOS ability, but like you discovered, trying to get meaningful, knowledgeable information from ISP technicians about something like this, is like pulling your own teeth out with rusty tweezers.

Anyhow, best of luck!!

Sasha