60 NICs? No i have 60+ clients connected to the server. The server has 1 NIC. Yes, there are a couple switches that cover this office, but I don't have access to them.

Iozone run for client side NFS only show latency plateaus when the filesize exceeds the buffer cache, but that is normal from what I read.

I haven't tried this other than the large computational tasks running on various machines. Those tasks to slow down the network in general. I have not tried pure network stress. That is something I'll look into.

This is why I initially suspected the video drivers, but swapping out drivers or changing the cards from Nvidia to ATI didn't solve the problem. I agree that it could be the fact that POS Dells are our only purchasing option.

I claim complete ignorance here.

Thanks for all the suggestions btw.

Long time ago, I had extensive trouble with network traffic failing for some seconds, then recovering. After intensive troubleshooting I found out that all machines affected were connected to one special switch (actually it was a HUB at this time). The HUB was pretty hot, I replaced it, no more problems thereafter ...

You should gather more information about your freeze-problem.

Edit:
When main memory gets short, its content is sourced out to harddisk with a big penalty in speed. This is good for big computational tasks with high memory usage - e.g. simulations. Better to get a result slowly than to break computations that ran for days.

When I see this list:
there is nothing included that swap could be any good for.

So do a "swapon -s" and check whether the server is using a noticeable amount of swap. Some kb are usual.

Filename Type Size Used Priority
/dev/hda3 partition 16386292 148 -1


I also ran iperf. Not sure if that is great tool for networkd performance, but here's the results from the server side with client connects:
Server and client IP's have been changed to protect the innocent.
server=111.11.111.11
client= 222.22.222.22

------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 4] local 111.11.111.11 port 5001 connected with 222.22.222.22 port 53814
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.1 sec 111 MBytes 91.8 Mbits/sec
[ 5] local 111.11.111.11 port 5001 connected with 222.22.222.22 port 53815
[ 5] 0.0-10.1 sec 110 MBytes 90.9 Mbits/sec
------------------------------------------------------------
Client connecting to 222.22.222.22, TCP port 5001
TCP window size: 165 KByte (default)
------------------------------------------------------------
[ 5] local 111.11.111.11 port 34319 connected with 222.22.222.22 port 5001
[ 5] 0.0-10.0 sec 108 MBytes 90.4 Mbits/sec
[ 4] local 111.11.111.11 port 5001 connected with 222.22.222.22 port 53816
------------------------------------------------------------
Client connecting to 222.22.222.22, TCP port 5001
TCP window size: 40.3 KByte (default)
------------------------------------------------------------
[ 6] local 111.11.111.11 port 34320 connected with 222.22.222.22 port 5001
[ 6] 0.0-10.0 sec 96.6 MBytes 80.9 Mbits/sec
[ 4] 0.0-10.0 sec 92.6 MBytes 77.4 Mbits/sec

perfectly okay - supposed you did this on the server under load :)

First of all, your primary problem still is not network traffic, but freezes, right?

For the beginning, write a simple script that writes the actual time to a _local_ file, every 500 milliseconds, and execute it on a client affected by the freezes. After that review if really there is a timestamp inside the file all 500 ms. This will tell you whether the whole client freezes or just the gui.
To be completely sure, write the file to ramdisk and run the script dispatched.

Second, iperf does produce artificial network traffic and measures the result. It's good to see healthy results, but so this does not help us either in finding what is wrong.

You need something to record and evaluate your real world traffic, not artificial traffic.

Wireshark is not made to do this ... it is made to analyze decent connections and packet contents.

ntop is the tool of choice, but it doesn't install for you.

"darkstat" is said to be a good replacement for ntop.

Some console tools you could try are: "EthStatus", "IPTraf", "Nolad", "iftop", "MRTG"

Watch for either peaks in traffic, or "black holes" in traffic, on either the whole network or a single affected client, on a resolution down beyond seconds, not minutes or hours.

btw, what about swap usage on the clients? 4 to 8 GB should be more than enough, except there is some memory hog running. Once had all memory and then swap running full due to a buggy KdeIOSlave ... caused lags, freezes and random OOM-kills. And just had KDE Dolphin fill up my tmp directory, causing "disk full" errors ... results on NFS-homes could be interesting ...

This shows timestamps close to every 500ms. It drifts a bit, but no major freezing.

I did get ntop installed, but I'm not sure what I should be looking for. So much info, but nothing smaking me in the face as odd.

Something I did notice that is odd. If I start top and just drag a terminal box around in the GUI, my CPU usage for both Xorg and firefox jumps to 20% each. Seems excessive for just dragging a stupid box?

As long as you did this test on a client that suffered from freezes during the test, it shows that there are no system freezes for seconds. So the most obvious part suffering from the freezes is the gui, so far.

Another thing you could check is the hard disk subsystem. Short freezes on the disk controller may lead to gui freezes. But, again, emacs shouldn't suffer from this (except there is swap usage). Look for a tool to measure disk I/O throughput in realtime.

And, again, have a look at the swap usage on the clients.

uhm ... I'm not an ntop-wizard, either. And a detailed tutorial possibly is to much for this forum, beside the fact that there surely already are some tutorials out there.

When running ntop on the server, there are plenty of connections you could monitor. Maybe it is easier to install ntop on a client and let it monitor the traffic ... make sure it probes the traffic on a resolution beyond one second and when there are freezes, look if you see something suspicious in the ntop data.

Uhm ... depends ... redrawing of the screen should be the job of X, and X is not known for it's speed and elegance ...

Firefox is a resource hog. This is especially true when not using the Adobe Flash plugin but the open source one (gnash I think) and not using Sun Java but the openjdk Java.

Looks like you should check whether these freezes are not that kind of random, but directly connected to firefox usage or slow X performance.

btw, keep in mind the thing about storing a firefox cache on NFS.