Interesting problem, see if there is similar difference in speed susing different applications (ftp, scp, etc) on different external hosts.
When you mentioned the two times 43 and 384 seconds, for what size of download was that?
scp (secure (ssh) copy) is nice in that it has a progress meter to see if the download is constant or not.
Always check /var/log/messages as always and perhaps dmesg to see if anything strange with ethernet driver on boot up.
If still no clues on the problem then you can use Ethereal
to sniff the packets between your linux box and the host downloading from.
You can also run Ethereal on windows and get comparison on the patterns of packets being sent between your local machine and the remote host.
You should be able to get the timings and packet data using Ethereal.
I wouldn't mind helping out how to run ethereal or read the logging output.