This just started.. system's been running fine for a year.

Part of the application process is to ftp images to a LAN server. This normally takes nearly no time, since the images are on the order of 10-15K.

On one workstation, it's still more or less instant, but the other one takes 15-20 seconds. Trying to troubleshoot it, here's what I see: pings to the server are running about 0.1ms, pretty steady, on both systems; on the good one, a manual ftp (or ssh, or scp) gets a nearly immediate response (ftp asks for user, other two want password), but on the "bad" system, all three of those take 15-20 seconds before a response is received.

I thought of hardware problems, but would the pings be fast if bad network cable, or card, or switch?

Any ideas?

The pings could be fast because they're small amounts of data and perhaps your NIC is having an issue with large packets. I'd suggest sniffing the interface with ethereal and looking for signs of problems (i.e. excessive numbers of TCP retransmits). You can also looks at the ifconfig output and see if there are a lot of packet errors detected. if you perchance have a managed switch, you might be able to get some of this data from it too. Try moving whhich port on your hub/switch the machine is connected to, in case the problem is a bad hub/switch port.

Yes - I looked at the ifconfig stats on both sides, and there are zero errors shown. The "bad" workstation did have iptables running for some reason (these are behind a pretty stout firewall, and none of the internal boxes have their firewalls running) but I killed that and it doesn't seem to have made the slightest difference.

We did change the cables/switch ports, also with no changes, so it's just about got to be a software issue of some sort... even if it's just bad or corrupted data being fed to something.

I'm thinking it's not specifically related to the ftp, since ssh and scp demonstrate the same behavior, i.e. not even responding for 15-20 seconds... as though they were trying (successfully, finally) to punch through a firewall.

I'm still looking through logs for some kind of clue...

Thanks for the response, I appreciate it!

Are you connecting by IP address or by hostname? It could be a DNS lookup failure (on either end).
Run tcpdump in a second shell on the affected machine and see what's happening on the wire when you use ftp.

ip's.

The problem's got to be on the "bad" workstation, because I can ftp to the LAN server from my office (200 miles away) with immediate response... and the same is true from every other computer I've tried - I ssh to another computer and ftp from there to that LAN server, no problems, no slowth.

Tell me a bit about tcpdump, please? I'm thinking what I'll be doing is opening two ssh sessions on the "bad" w/s, and then start ftp on one and tcpdump on the other. No doubt tcpdump has a way to capture its output to a file, then we look at the output and see... what? I'm thinking we also want to do the same thing on the "good" w/s and compare the outputs? And, yes, as soon as I'm done responding here I'm going to go RTF tcpdump M, don'cha know...

I suggest you do this from the keyboard of the 'bad' workstation and look at the tcpdump output live - ssh adds extra network traffic to the mix that will need to be filtered out later. Open two sessions (preferably in a GUI so you can see both concurrently). In one session start tcpdump -i eth0. In the other session start a command line ftp connection to the server. Watch what happens in the tcpdump session: does the workstation send out the ftp connection request immediately... does the server reply immediately?

Alternatively you could do this using ssh and dump tcpdump's output to a file for analysis. Grep the file for lines containing the workstation and server IP addresses. You'll be looking for the same thing - does the workstation send out the ftp connection request immediately... does the server reply immediately?

RESOLUTION:

It turned out to be DNS addresses.

The client's ISP had changed his service a few months (I think maybe 5 or 6) earlier, from a 206. to 12. and the DNS server changed as well. However, as it turned out, the old DNS servers continued to respond until a couple of weeks ago, when this problem manifested.

This is somewhat weird, because both workstations had "old" DNS servers listed, but only one workstation showed the problem.

Actually, it's doubly weird, because the ftp (as well as scp and ssh) connection was - as I said before - by IP address.

Setting the correct DNS address(es) in /etc/resolv.conf cured the problems.

I think perhaps the two workstations had installed slightly different versions of RH9. At the client's central office, where they build the systems, I found a set of CDs labeled RH9 which they got from their hardware supplier, and also a set of CDs with RH9 that were included with the book Red Hat 9 Bible... so I have a suspicion (not yet confirmed) that at least the ftp's were slightly different.

The real question in my mind is why on earth the "bad" ftp (and scp, ssh, and possibly others) would bother to attempt connection with a DNS server when it was given an IP address??? Is it perhaps a setup issue?

I have run into the same problem before.
It appears that a lot of programs will do reverse lookup on an IP to look for the host-name, if the DNS information is not correct this will make it hang until it times out and doesn't do it. I am not sure where this is configured, most likely in the programs conf file.

Another solution would have been to add a host entry for the machine you were trying to get to in your /etc/hosts file.

I assume from this that you've tried it and it worked, without a DNS server available - that is , the reverse lookup?

Do these ill-behaved programs look in hosts before they look in resolv.conf? I hadn't thought about the possibilities of reverse lookups in those two...

I'll have to give that a try when I can do it without disrupting business...

yes /etc/hosts is queried first and then /etc/resolv.conf
I don't think is ill behavior in the application part (maybe unexpected) you just need accurate information for name resolution.

Tomayto, tomahto. My pronunciation is "ill behavior".

When I give a program an IP address, I don't expect it to use a DNS to find out the associated name, unless finding the name is its job (or part of it, like logging with names).

I noticed that ntpupdate -u 10.10.10.10 did NOT go off to find the names.

If it's a conf matter, that's all well and good, but if the DEFAULT behavior is to do it, well, then... I have to say it's ill-behaved. And if that's not the default, but the distro's default installation includes conf that tells a program like ftp to do name lookup unless told not to... then it's ill-behaved on the part of whoever put it in the installation set.

It could well have been explicitly advertised as such and I missed it. Whatever.

All that said, I still think it's a Bad Thing for ftp to lookup an IP address as a matter of course.

I'll take a look at the "good" workstation after they close and see if it's a different version of ftp or the same ftp but different configurations.

Red Hat 9 - hmmm... what's the .conf for ftp? Would another parameter in a different configuration affect ftp? Lessee...