LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices

Reply
 
Search this Thread
Old 05-06-2013, 02:57 PM   #1
5883
Member
 
Registered: Aug 2004
Posts: 170

Rep: Reputation: 0
How to avoid performance impact during web spidering with lighttpd server ?


Hi everyone,

i'm using lighttpd as web server in our linux board.
Normally everything is ok, cpu usage is low.

Now if someone use cyber security tool like rapid7 to scan the website (the site for the linux board), i can see cpu usage jumps to 90%.

I can add authentication to the web pages, that solves the problem.

But some customers don't like password protection for the webpage,
any suggestions ?

Thanks !
 
Old 05-06-2013, 03:27 PM   #2
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,140
Blog Entries: 54

Rep: Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791
Iptables connection rate limiting?
 
Old 05-07-2013, 10:01 AM   #3
5883
Member
 
Registered: Aug 2004
Posts: 170

Original Poster
Rep: Reputation: 0
that could hurt the performance.

the thing is, say i have a page called
http://mydomain/webpage1.pl

if the software scan starts, there're ~10 of webpage1.pl if you see from "ps",
that makes the cpu 90%.

wondering how to limit the number of instances of webpage1.pl.

Quote:
Originally Posted by unSpawn View Post
Iptables connection rate limiting?
 
Old 05-07-2013, 01:45 PM   #4
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,140
Blog Entries: 54

Rep: Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791
Quote:
Originally Posted by 5883 View Post
that could hurt the performance.
Unless you set the threshold ludicrously low that simply ain't the case. Here's how: figure out what your average requests per minute is and what you get on the busiest time. Create a "-m recent" rule for the first and create a rule that triggers a log line if the "recent" limit is exceeded. Use 'ab' or any other benchmark tool and test response time. Post the rule and the test results. Watch your logs for any limit violations.


Quote:
Originally Posted by 5883 View Post
the thing is
The thing is the network layer is the foundation for anything else. It does not make sense to deploy mod_bw, mod_evasive, mod_security, mod_cband, mod_throttle, mod_bwshare, bw_mod, mod_bandwidth, mod_ratelimit or any other application level modules when you have not implemented access controls at the network layer.

Last edited by unSpawn; 05-07-2013 at 01:47 PM.
 
Old 05-08-2013, 10:45 AM   #5
5883
Member
 
Registered: Aug 2004
Posts: 170

Original Poster
Rep: Reputation: 0
can you please see if it's my web client perl code, or it's the lighttpd config ? it's related to web spider test.

from the scan tool, it's log shows these:

2013-05-08T14:16:31 [INFO] [Thread: SPIDER::do-spider-injection-tests] [Site: My product 10.50.181.255] Preparing spider query-based Blind SQL injection tests

2013-05-08T14:16:31 [INFO] [Thread: SPIDER::do-spider-injection-tests] [Site: My product 10.50.181.255] Preparing spider form-based SQL Injection tests

2013-05-08T14:16:31 [INFO] [Thread: SPIDER::do-spider-injection-tests] [Site: My product 10.50.181.255] Preparing spider form-based Blind SQL injection tests

2013-05-08T14:16:31 [INFO] [Thread: SQL Injection] [Site: My product 10.50.181.255] [10.50.181.255:80] Performing spider query param SQL injection tests

2013-05-08T14:16:31 [INFO] [Thread: SQL Injection] [Site: My product 10.50.181.255] [10.50.181.255:80] Performing spider form-based SQL injection tests

2013-05-08T14:17:06 [INFO] [Thread: SQL Injection] [Site: My product 10.50.181.255] [10.50.181.255:80] [PostInjectionResponseProvider] Exception while trying to get response for payload r%277%22nx: Peer closed connection before first line could be read

2013-05-08T14:17:25 [INFO] [Thread: SQL Injection] [Site: My product 10.50.181.255] [10.50.181.255:80] [PostInjectionResponseProvider] Exception while trying to get response for payload r%277%22nx: Peer closed connection before first line could be read

2013-05-08T14:17:57 [INFO] [Thread: SQL Injection] [Site: My product 10.50.181.255] [10.50.181.255:80] [PostInjectionResponseProvider] Exception while trying to get response for payload r%277%22nx: Peer closed connection before first line could be read





Quote:
Originally Posted by unSpawn View Post
Unless you set the threshold ludicrously low that simply ain't the case. Here's how: figure out what your average requests per minute is and what you get on the busiest time. Create a "-m recent" rule for the first and create a rule that triggers a log line if the "recent" limit is exceeded. Use 'ab' or any other benchmark tool and test response time. Post the rule and the test results. Watch your logs for any limit violations.



The thing is the network layer is the foundation for anything else. It does not make sense to deploy mod_bw, mod_evasive, mod_security, mod_cband, mod_throttle, mod_bwshare, bw_mod, mod_bandwidth, mod_ratelimit or any other application level modules when you have not implemented access controls at the network layer.
 
Old 05-09-2013, 12:58 AM   #6
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,140
Blog Entries: 54

Rep: Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791
Quote:
Originally Posted by 5883 View Post
can you please see if it's my web client perl code, or it's the lighttpd config ? it's related to web spider test.
I'm sorry but the output of that tool has absolutely no relation to your firewall rule set (which I strongly recommend you address first), Perl code or lighttpd configuration.
 
Old 05-09-2013, 09:53 AM   #7
5883
Member
 
Registered: Aug 2004
Posts: 170

Original Poster
Rep: Reputation: 0
i made the max_connection_per_ip = 1 in the lighttpd, and the problem goes away.

Hopefully that's it.

Quote:
Originally Posted by unSpawn View Post
I'm sorry but the output of that tool has absolutely no relation to your firewall rule set (which I strongly recommend you address first), Perl code or lighttpd configuration.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: Installing Joomla 1.5.6 On A Lighttpd Web Server (Debian Etch) LXer Syndicated Linux News 0 09-16-2008 03:30 PM
LXer: Installing Drupal 6.4 On A Lighttpd Web Server (Debian Etch) LXer Syndicated Linux News 0 09-14-2008 10:41 AM
LXer: How To Speed up Lighttpd / Apache Web Server Access and Downloads LXer Syndicated Linux News 0 04-27-2008 12:30 AM
LXer: Howto Setup a Lighttpd Chroot Jail Web Server under Debian Linux LXer Syndicated Linux News 0 05-06-2007 06:31 PM
How to make Lighttpd Web Server to use /etc/shadow file vikasumit Linux - Software 4 05-06-2006 12:51 AM


All times are GMT -5. The time now is 01:25 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration