I just returned from a 2 week vacation, did my usual apt-get upgrades on my server (deb sid w/ kernel 2.6.8), and I noticed that when I tried to get to my website it was really slow, and eventually it would not connect at all. Back at home I checked the server and it seemed that after a reboot everything was fine again, but if left on about 5 hours or so, ping levels to the router would get to about 1000ms, and if left on overnight it would eventually be disconnected completely. Right now I've found that I can do an ifdown/ifup and get it back to normal again, but it will just start the same cycle. I am running an apache 1.3 server, SSH, and Samba on it. Anyone have any ideas why my server's net connection slows down to a crawl and then just stops completely? THANKS!

-Ryan

How is your server connected to the outside world? By that I mean is it via a LAN or directly? Is it via ethernet or USB? Is it cable, ADSL or something else? What brand is/are the ethernet card/modem/whatever else?

I have to say, at first glance this sounds a little like a memory leak.

That was my first guess too, and using 'top' I can see that apache has quite a few more processes than usual (7-8) and they are each using about 8% of my memory. I'm usually left with about 10mb (of 128mb) free, but this has never been a problem in the past. I could try killing apache and seeing if I still get disconnected after time. The server connects to the WAN through a D-Link router along with 3 other desktops. The ethernet card in the server is a davicom 10/100 using tulip drivers. Thanks for the help!

Does this figure (memory usage) increase steadily over time? Mind you, I was more thinking about memory leaks in the drivers rather than apache. The tulip driver works pretty well, though, and has been around for some time.

Could it also be that your webserver is being flooded with traffic that is saturating its upload (or download)? Are the other three computers behind that same router affected in any way?

The apache memory usage increases from about 4% to about 8% over time and each of 8 or so apache processes does this in sequence. I've never watched this closely though, so this could very well be normal for apache.

Also, I don't have a very popular site so I really doubt is getting too much traffic, but I can test this by simply turning off the Virtual Server ports on my router. The other computers (2 windows desktops and 1 linux desktop) seem to be operating normally, but I haven't checked the speed of their network connections. They certainly have not yet completely dropped the connection, but I will do some ping testing to see if they are still running at full speed while the other one gets the slow pings. Thank you so much for the help with this!

A memory leak would rise steadly and without bound. It may still be one but it doesn't sound likely.

Could you also check for any changes in the output of netstat -p over time?

I will run that periodically and see how it looks. Right now this is what I get:

Is there any software that can monitor and log all network activity so that I can analyze it to see when and why the network connection starts to degrade?

I've narrowed my problem down to the connection between the Debian server and my router. No other machines on my net have slow connections or high pings. I've also written a script (which I can post if anyone desires) that will check every 5 minutes to make sure a ping to my router will go through, and if it doesn't it will do an 'ifdown etho', 'ifup eth0' to get the connection back. So now I've got a bandaid, but I'd like to find the root cause of this problem. Is there a monitoring utility I can use to just montior the ethernet connection between my server and my router? Thanks for the help!

I've been following this & I just re-read the above, it occurs to me: Did you have the problem before the apt-get upgrades?

If not, then the Q becomes "What changed?"

And raising an issue on my mind for a while, how do you do an "apt-get downgrade"? (I know the command doesn't exist, but is there an easy way to reverse an upgrade gone bad?)

No, my server had been working just fine before my trip AFAIK, but after the upgrade something went wrong. I have to also add though that it was also moved from one household to another (I moved into my new home when I returned from my trip). This also means I went from a D-Link DI-624+ to a DI-524 (almost identical except no 'Xtreme G'). Since there were quite a few variables since the problem started, I wanted to try and locate the problem with monitoring tools and pinpoint what is currently causing the errors. Perhaps apt has a log of what upgrades were done at that time that I could look through. Then maybe I could go through the individual programs that were upgraded and switch each one back (if that's possible).

It's just really frustrating since it was working so well before. I never even had it hooked up to any I/O, just put it in a closet with power and a LAN connection, and used SSH to do stuff on it when I needed to. Just for the record, the things I run on it are: Apache 1.3, mySQL, squid, samba, exim4, sshd, and cron. Nothing too far from a base Debian system.

Would [t]ethereal help? I have used it primarily to show how "chatty" "Winders" boxen are on a LAN. It seems to be able to make massive log files rapidly. But it does have filtering capability that I am not familiar with.

Also, another custom script to gather & log "top' results &/or ping times.

Do you have the option to go back to the DI-624+, even temporarily, just in case it's a hardware related problem?

Have you ever had trouble with/after apt-get upgrades before?

Have you done any additional apt-get upgrades?

Do have the time/HD space luxury of building a temporary replacement server? Say using Sarge, Etch, or something like what you had just before the trouble started?

Which is more important to you now, fixing the problem or understanding it? In terms of the Alexander reference, the sword or the marlin spike?

I actually just recently started using tethereal to do some monitoring, but I'm not getting real great info. I usually use the '-z conv,tcp' option so that I can get a summary of all the activity at the end of the scan, but the summary doesn't really tell me anything out of the ordinary (at least that I can see).
I could try looking up some type of top or ping logging scripts, but I think first I'd like to make sure I am getting all the info I can out of tethereal because it seems like that should easily be what I need.
Unfortunately, the DI-624 is now my Mom's router, and though I think I could get it eventually, I'm not sure how soon it would be. But eventually I'd definately like to go that route. I debated on picking up a WRT54gs to switch it out with (since I wanted one anyway to put linux on) and see if that helps. I still may try that.
I've never had any upgrade problems before, my little server has been pretty stable since I installed it. It's an old Celeron 300a BookPC with 128mb RAM (soon to be 512). I mainly just use it to ssh in from work and use the squid proxy through an ssh tunnel for all my work web surfing, and the web server is mainly for my family and friends to look at pictures, no more than a dozen people a day (I think).
Now that I have a working script that will retsore my connection automatically, my main importance is understanding the problem, even though it gets really annoyingly slow right before it dies! I think my next steps will be to 1) do more research on tethereal and see what other kinds of data I can obtain from it, 2) switch back to the DI-624 and see if I get the same errors, and 3) build another identical server on a spare machine and see what happens (hopefully it won't come to that). Thank again for all your time spent with this!

One other thing, can you check your /var/log/syslog and your /var/log/messages for any unusual entries. Specifically vj decompression errors.

Are you logging the results in any way? Esp. when it has to restart eth0. That way you might be able to find a pattern & do a proactive restart every n-1 minutes.

BTW, thanks for answering each of the Q's suggestions in my last post.

There is nothing good in either syslog or messages except for the log of when my script has to restart eth0. It seems like I get this message once every day or so. It seems to me that sometimes the server recovers from the laginess somehow. So it'll start getting slower and slower, but then sometimes instead of dying it will go back to normal speed. But then it just slows down again in the same way. It's too bad I can't log the speed of the connection over time, that might provide some good data. Is there any way to do that? Maybe during the pinging in my script I can record the averages and log it each time. Since it runs every 5 minutes that may give me a good indication. So any clue how I could do this?