LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Software (http://www.linuxquestions.org/questions/linux-software-2/)
-   -   How to avoid performance impact during web spidering with lighttpd server ? (http://www.linuxquestions.org/questions/linux-software-2/how-to-avoid-performance-impact-during-web-spidering-with-lighttpd-server-4175460922/)

5883 05-06-2013 02:57 PM

How to avoid performance impact during web spidering with lighttpd server ?
 
Hi everyone,

i'm using lighttpd as web server in our linux board.
Normally everything is ok, cpu usage is low.

Now if someone use cyber security tool like rapid7 to scan the website (the site for the linux board), i can see cpu usage jumps to 90%.

I can add authentication to the web pages, that solves the problem.

But some customers don't like password protection for the webpage,
any suggestions ?

Thanks !

unSpawn 05-06-2013 03:27 PM

Iptables connection rate limiting?

5883 05-07-2013 10:01 AM

that could hurt the performance.

the thing is, say i have a page called
http://mydomain/webpage1.pl

if the software scan starts, there're ~10 of webpage1.pl if you see from "ps",
that makes the cpu 90%.

wondering how to limit the number of instances of webpage1.pl.

Quote:

Originally Posted by unSpawn (Post 4946132)
Iptables connection rate limiting?


unSpawn 05-07-2013 01:45 PM

Quote:

Originally Posted by 5883 (Post 4946640)
that could hurt the performance.

Unless you set the threshold ludicrously low that simply ain't the case. Here's how: figure out what your average requests per minute is and what you get on the busiest time. Create a "-m recent" rule for the first and create a rule that triggers a log line if the "recent" limit is exceeded. Use 'ab' or any other benchmark tool and test response time. Post the rule and the test results. Watch your logs for any limit violations.


Quote:

Originally Posted by 5883 (Post 4946640)
the thing is

The thing is the network layer is the foundation for anything else. It does not make sense to deploy mod_bw, mod_evasive, mod_security, mod_cband, mod_throttle, mod_bwshare, bw_mod, mod_bandwidth, mod_ratelimit or any other application level modules when you have not implemented access controls at the network layer.

5883 05-08-2013 10:45 AM

can you please see if it's my web client perl code, or it's the lighttpd config ? it's related to web spider test.

from the scan tool, it's log shows these:

2013-05-08T14:16:31 [INFO] [Thread: SPIDER::do-spider-injection-tests] [Site: My product 10.50.181.255] Preparing spider query-based Blind SQL injection tests

2013-05-08T14:16:31 [INFO] [Thread: SPIDER::do-spider-injection-tests] [Site: My product 10.50.181.255] Preparing spider form-based SQL Injection tests

2013-05-08T14:16:31 [INFO] [Thread: SPIDER::do-spider-injection-tests] [Site: My product 10.50.181.255] Preparing spider form-based Blind SQL injection tests

2013-05-08T14:16:31 [INFO] [Thread: SQL Injection] [Site: My product 10.50.181.255] [10.50.181.255:80] Performing spider query param SQL injection tests

2013-05-08T14:16:31 [INFO] [Thread: SQL Injection] [Site: My product 10.50.181.255] [10.50.181.255:80] Performing spider form-based SQL injection tests

2013-05-08T14:17:06 [INFO] [Thread: SQL Injection] [Site: My product 10.50.181.255] [10.50.181.255:80] [PostInjectionResponseProvider] Exception while trying to get response for payload r%277%22nx: Peer closed connection before first line could be read

2013-05-08T14:17:25 [INFO] [Thread: SQL Injection] [Site: My product 10.50.181.255] [10.50.181.255:80] [PostInjectionResponseProvider] Exception while trying to get response for payload r%277%22nx: Peer closed connection before first line could be read

2013-05-08T14:17:57 [INFO] [Thread: SQL Injection] [Site: My product 10.50.181.255] [10.50.181.255:80] [PostInjectionResponseProvider] Exception while trying to get response for payload r%277%22nx: Peer closed connection before first line could be read





Quote:

Originally Posted by unSpawn (Post 4946750)
Unless you set the threshold ludicrously low that simply ain't the case. Here's how: figure out what your average requests per minute is and what you get on the busiest time. Create a "-m recent" rule for the first and create a rule that triggers a log line if the "recent" limit is exceeded. Use 'ab' or any other benchmark tool and test response time. Post the rule and the test results. Watch your logs for any limit violations.



The thing is the network layer is the foundation for anything else. It does not make sense to deploy mod_bw, mod_evasive, mod_security, mod_cband, mod_throttle, mod_bwshare, bw_mod, mod_bandwidth, mod_ratelimit or any other application level modules when you have not implemented access controls at the network layer.


unSpawn 05-09-2013 12:58 AM

Quote:

Originally Posted by 5883 (Post 4947286)
can you please see if it's my web client perl code, or it's the lighttpd config ? it's related to web spider test.

I'm sorry but the output of that tool has absolutely no relation to your firewall rule set (which I strongly recommend you address first), Perl code or lighttpd configuration.

5883 05-09-2013 09:53 AM

i made the max_connection_per_ip = 1 in the lighttpd, and the problem goes away.

Hopefully that's it.

Quote:

Originally Posted by unSpawn (Post 4947639)
I'm sorry but the output of that tool has absolutely no relation to your firewall rule set (which I strongly recommend you address first), Perl code or lighttpd configuration.



All times are GMT -5. The time now is 05:04 AM.