[SOLVED] extremely slow upload to the internet

EDIT: Marked this thread as solved since it was (partially) fixed (see my last post).

I have a fresh debian wheezy installation (x64) on a machine in my home network. The network is protected by a smoothwall hardware firewall. Within the network everything is fine, normal up and download speeds. However, outside the network (so connections across the internet) my upload is very slow (20 kbps), my download speed is normal.

I have other linux machines in my home network who can easily reach 1mbps upload so I do not think it is related to the hardware firewall I'm using.

Is it possible that it is routing related? (my download seems unaffected)

EDIT: if I use scp to retrieve a big file from this computer across the internet it is also slow (even though the connection is initiated from the outside). does this mean that it can not be routing related? (if I understand it correctly, there is no more routing involved since the connection isn't initiated from the "slow" machine?)

Some extra info:

uname -r:
3.2.0-4-amd64

lspci | grep Gigabit
02:00.0 Ethernet controller: Intel Corporation 82545EM Gigabit Ethernet Controller (Copper) (rev 01)

ifconfig:
eth0 Link encap:Ethernet HWaddr 00:0c:29:77:33:66
inet addr:192.168.100.150 Bcast:192.168.100.255 Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:fe77:3366/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:413 errors:0 dropped:0 overruns:0 frame:0
TX packets:304 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:52056 (50.8 KiB) TX bytes:210986 (206.0 KiB)

route:
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default smoothwall 0.0.0.0 UG 0 0 0 eth0
192.168.100.0 * 255.255.255.0 U 0 0 0 eth0

If you need any other information, just let me know.

Thanks in advance

No idea where to start so maybe start by showing us, from both a machine that works OK and one that does not, the output of: ifconfig, route -n, iptables-save, 'tc -s qdisc show', 'tcptraceroute www.google.com', 'time curl -v www.google.com/ncr' and for good measure something like a wget of a kernel source tar ball? Repeat for your fresh Debian Wheezy installation after putting it in the Smoothwalls DMZ and from the Smoothwall itself as far as it allows those or equivalent commands?

Slow machine ifconfig:

eth0 Link encap:Ethernet HWaddr 00:0c:29:77:33:66
inet addr:192.168.100.150 Bcast:192.168.100.255 Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:fe77:3366/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:39354 errors:0 dropped:0 overruns:0 frame:0
TX packets:37706 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:5907448 (5.6 MiB) TX bytes:59635364 (56.8 MiB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)


--------
normal machine ifconfig:

eth0 Link encap:Ethernet HWaddr 40:61:86:07:8d:52
inet addr:192.168.100.200 Bcast:192.168.100.255 Mask:255.255.255.0
inet6 addr: fe80::4261:86ff:fe07:8d52/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:48598100 errors:0 dropped:0 overruns:0 frame:0
TX packets:56877315 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:25776481722 (24.0 GiB) TX bytes:57469051115 (53.5 GiB)
Interrupt:10 Base address:0x6000

eth1 Link encap:Ethernet HWaddr 40:61:86:07:8d:53
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
Interrupt:5 Base address:0x8000

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:514228703 errors:0 dropped:0 overruns:0 frame:0
TX packets:514228703 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:29517407557 (27.4 GiB) TX bytes:29517407557 (27.4 GiB)

--------------------------------------------------------------------------
route -n is for both the slow and normal machine identical:

Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.100.1 0.0.0.0 UG 0 0 0 eth0
192.168.100.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0

--------------

iptables-save is empty for both slow and normal machine

---------------
"tc -s qdisc show" on slow machine:

qdisc mq 0: dev eth0 root
Sent 59646198 bytes 43595 pkt (dropped 0, overlimits 0 requeues 1)
backlog 0b 0p requeues 1



"tc -s qdisc show on normal machine:

qdisc pfifo_fast 0: dev eth0 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Sent 57469290663 bytes 56878620 pkt (dropped 0, overlimits 0 requeues 250862)
backlog 0b 0p requeues 250862
qdisc pfifo_fast 0: dev eth1 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0

----------------------------------------------------------------

tcptraceroute www.google.com
gives identical routes equally fast on both machines.

----------------------------------------------------------------
the time curl -v www.google.com/ncr command returns in about 0.1 sec on both the slow and normal pc.

-------------------------------------------------

wgetting an iso file from debian.org goes at full speed on both machines (around 4.5 megabytes per sec)
so download speed is normal

scp-ing a large file to a machine outside of the network goes around 1 megabyte per sec on the normal machine.
on the slow machine uploading goes at 15-20 kb per sec.


--------------------------------------------------

I'll try to put the slow machine in the smoothwall dmz tomorrow.

Any MAC starting with 00:0c:29: means VMware, so your slow machine seems to be a virtual device. Is its host or your Smoothwall configured for any QoS other bandwidth arbitration?

Yes, the slow machine is indeed a vm on an ESX host.
Neither the ESX host nor the smoothwall is configured for QoS.
There are other unaffected machines running on the ESX host, even machines in the same (behind the smoothwall) network.

Going to try and put the slow machine in a DMZ now

I did something even better, instead of putting it in a DMZ I put it outside the network and gave it its own external ip (have a block of 16 ips available here).

The problem seems to have disappeared now that it is outside of the network. uploading at a steady 1 megabyte per sec.

did the set of commands listed earlier again, route, tcptraceroute, tc -s qdisc show, etc..
Again it was similar to the normal machines.

I guess that for some reason the smoothwall is the problem.
Since I am not experiencing any problems with other machines behind the smoothwall I do not really know how to proceed.
The logs of the smoothwall which I can access through the webinterface give no indication of the "slow" machine being treated differently than the other machines in that network.

EDIT:
I just realized another possibility:
both the slow machine and the smoothwall are hosted on the ESX host. I have other fast machines on that host but they are not in the smoothwall network. I have fast non-virtual machines which use the smoothwall network, but the slow machine is the only virtual machine behind the smoothwall.
This means that It is also possible that something is going wrong with the virtual network connection between the slow machine and the smoothwall.

So after almost completely reconfiguring the smoothwall the problem still persisted.
I played around a bit with the network settings of the ESX host, put the slow machine on a separate NIC, no success.
until I changed the NIC speed from "auto negotiate" to "100 mbit full duplex". (auto negotiate found 1000mbit before). Now my upload is running steady at 350-400 kilobytes per sec. It is not the full 1 megabyte per sec which it should be but it is a big improvement over the 20 kb I got before.

If someone has some idea on why this seams to be working I'm very interrested. It might hold the key to getting the full 1mbps. If not, I'll probably settle for the lower upload speed for now and mark the topic as solved. 350-400 is usable whereas 20 was not.