TCP Crawler network not working after many TCP requests
 

Hi,

I have a crawler on my VPS VMWare Server (also tried it on my dedicated server). After many requests about 200.000/6h my network is gone. I use lynx to get any website outside of local network and any site is not responding - also tried to obtain IP address instead of host name - no results.

I can get every local site. For example when I am on 190.24.2.1 I can obtain a page which is placed on 190.24.2.5 but obtaining 191.54.23.34 is not possible.

I tried many things such as changing TCP timeouts to less, restarting virtual machine and no results. After few hours - 4-5 hours I get network working. It is not working as fast as it should but it is ok.

I am wondering where is the problem. Have I got to configure linux? Am I missing something? Maybe some router stack is blocking my ip? (Other crawlers put on same VPS are working properly, but after 6 hours they also crashes)

I also notices that I can get like 30.000 web sites per hour (some with errors, some with no response, some with timeout surely) but this number decreases hour by hour and finally reaches zero.

Can you help me and suggest where may ba a problem?

My system: Ubuntu 10.04 LTS Server, 1GHz, 512 MB RAM, 5 MB/s.

Sounds like the network is throttling the crawlers (a good thing IMHO). Check with the network admin.