Hi, all

I meet a serious problem, there is only one domain, only one website on my LAMP, but from past night, I got 10G+ access.log, the content is like the following:

203.152.222.81 - - [09/Jun/2011:12:33:39 +0100] "GET http://i.yimg.jp/images/commerce/js/.../jquery.min.js HTTP/1.1" 200 84362 "http://locoplace.jp/t000001869" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1)"
203.152.222.81.static.zoot.jp - - [09/Jun/2011:12:33:41 +0100] "GET http://m.tabelog.com/osaka/A2702/A270202/27055983/ HTTP/1.1" 302 129 "http://r.tabelog.com/osaka/A2702/A270202/27055983/" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1)"
sol-fttb.57.175.118.46.sovam.net.ua - - [09/Jun/2011:12:33:39 +0100] "GET http://www.google.com/search?as_q=gu...ut=ie&filter=0 HTTP/1.0" 302 367 "http://www.google.com/search?as_q=gulfport+biloxi+casinos+inurl:/down.htm&num=100&hl=en&output=ie&filter=0" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7) Gecko/20041122 Firefox/0.5.6+"
static.6.99.4.46.clients.your-server.de - - [09/Jun/2011:12:33:41 +0100] "GET http://img.informe.com/img/thumbs/s/...net_medium.jpg HTTP/1.0" 200 11023 "-" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/533.2 (KHTML, like Gecko) Chrome/6.0"
alagny-251-1-58-161.w90-61.abo.wanadoo.fr - - [09/Jun/2011:12:33:42 +0100] "GET http://www.mon06.com/FH.php HTTP/1.1" 200 590 "http://www.mon06.com/BAN.php" "Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; DigExt)"
ip-77-95-17-117.hosting.undns.ru - - [09/Jun/2011:12:33:42 +0100] "POST http://working.u0f.ru/check.php HTTP/1.1" 200 500 "http://7a68443f5c/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
221.4.87.117.broad.xz.js.dynamic.163data.com.cn - - [09/Jun/2011:12:33:41 +0100] "GET http://cofen.net HTTP/1.1" 301 - "http://www.whitehouse.net" "Mozilla/4.0 (compatible; MSIE 6.0; Windows 5.1)"
61.160.221.232 - - [09/Jun/2011:12:33:42 +0100] "GET http://www.sport.co.uk/public/javascripts/suckerfish.js HTTP/1.0" 200 469 "http://www.sport.co.uk" "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; Alexa Toolbar)"
98.126.55.218.static.krypt.com - - [09/Jun/2011:12:33:42 +0100] "GET http://www.google.com/intl/en_ALL/images/logo.gif HTTP/1.0" 200 8558 "http://mail.google.com/mail/?shva=1#inbox/122g5dg42r78d731" "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Alexa Toolbar)"



Is anybody help me to find what's the problem?

The problem is you're running a web site on a public IP. There's nothing you can do to prevent scans like this. Everyone gets them.

Assuming that the real problem you're asking about is the big log file (you didn't mention any other effects of this "attack"), then you can turn off Apache logging in httpd.conf. Or (though I'm not sure of this) maybe a program like fail2ban can be configured to notice things like this and set iptables to drop further connections from offending IPs.

1 members found this post helpful.

You can use an appliance, or VM such as FortiWEB to block scans like these. Also, an IPS/IDS will catch something like this and block access from the source IP in the router/firewall.

Hope this helps.

1 members found this post helpful.

Thanks very much, but I do not think it's a common problem, it takes me 300+ connections per mins, I think maybe i need to something applications to ban them.

Regards, it's useful information.

Block their IP addresses, as they happen, on the firewall(hardware, or iptables) or router.

Um, are you using this as some sort of a proxy? All of those log entries are successful connections, and I have a hard time believing your hosting all of those sites.

1 members found this post helpful.

That and the user agents are all different. If this isn't a proxy log, I would be suspicious of some form of cross-site scripting or linking.

I have disabled all mod_proxy for apache, so it can not be a proxy server for apache 2, isn't it?

If it is cross-site script, how can I find out it? disable all virtual host?

I find that it's hard to use fail2ban, because the connections are not regular, the user agent and remote address are all different.

I think it might be a DDOS or other attack of some type.

Shouldn't there be a "/" character between all those GET's and those odd URL's ?

Remember reading about something fishy when that was missing that in requests, but
don't recall exactly what it was.

If that is the case, fail2ban might be able to block it.

A filter something like this:

failregex = <HOST>.*(GET|POST).*$

ignoreregex = <HOST>.*(GET|POST) \/

Thanks, the problem is : it's not really IP address in access_log, you see, 203.152.222.81.static.zoot.jp, I do not think fail2ban should solve these type of ip address.

If you really are experiencing a DDOS situation, which I don't see that there is evidence that you are, then it would mean that you have angered the wrong group and attracted their attention. A DDOS requires a significant amount of resources, such as a large bot net, as well as centralized control. You would also be seeing symptoms that are a lot more severe than a log full of garbage entries. Additionally, I believe you would be seeing an overload of attempts to access YOUR resources, not connect GET strings from your Apache's access log. Please confirm, these connections are originating with YOUR web server, not from traffic from your web browser?

Your original post mentions that you are running one website and one host. In your reply to me, you mention disable ALL virtual hosts. Please clarify, as I consider multiple virtual hosts to be multiple web sites and this impacts how you need to proceed.

One thing that this made me think of is: are you perchance running any sort of advertisement, or rss news feed on your website that would cause it to go to different places to pull in content? Also, to be impacted by XSS, you would need to be accepting input from users and then redisplaying this content on your site. What type of site are you running and does it have active content from your users, e.g. a forum or comment section?

Sorry for the uncleared information, I have never met DDOS before, it's the first time. Let me describe more about the 'attack', and what steps I have tried.

First, at the first day, I find the traffic of my LAMP is very very heavy. There are two virtual hosts on my Apache, nothing for serving, only two index.php in web root, the content is simple text: 'this is domain1' and 'this is domain2'.

It is impossible to be having too much traffic, 9MB per second from #iftop. So I start to track the issue.

1. I tried to upgrade sshd, httpd.
2. I tried to check the status of all my configuration.

I am sure nobody change my httpd.conf, sshd.conf, my index.php in web root.

After that, I remove one of the virtual host, including delete index.php and web root folder (/var/www/domain1), issue didn't solved.

Then, OK, let me remove all virtual hosts, issue didn't solved.

Check the logs again, dummy-domain-log is normal, nothing special, only the default access_log contains the entities I mentioned before.

Then, I start to remove everything about httpd, then install a fresh httpd with last version. Issue didn't solved.

About proxy? I start to remove everything about mod_proxy and mod_ftp_proxy,mod_http_proxy

Issue didn't solved.

Is it about apache? OK, I change the port from 80 to 8080, issue solved, #netstat -ant, no so many connections.

Next, I shut down httpd server, start up a very simple wsgi server return 'hello' to all requests, not luck, many connections destroyed my wsgi server.

At last, I do not know what to do next.

That's all what I met and what I did, I have no experience about it. Sorry if I submit too many unclear information, I hope I describe myself clearly.

Thanks.

Something isn't making complete sense. In your original post, what log file did you post from (file name and path please)? If that was your Apache access.log file it indicates that something is amiss as it shows connections from various user agents going to sites that are not part of your domain. In my opinion, the nature and origin of these connections is something that needs to be identified conclusively. From what I can see in the logs and from what you are telling me (not a proxy, and a simple web page with no content being served up), as much as I am loathe to say it, I would be looking very closely for a possible compromise as something appears to be using your system to connect to these locations using various browser agents.

Netstat will show you connection attempts. I would suggest, as root, running netsat -pane and also lsof -pwn and see if you can tie the connections back to an active process running on your system. You will see all the connection, inbound and outbound. If you run a web browser, you will get lots of connections. On the inbound, much of this is just noise. It generally does not indicate that you are under any form of attack. If you are getting so many connections that your web server fails to function, that is a problem and you will need to deal with it. This is the impression I am getting from your last post.

One of the first things I would recommend is the application fail2ban which will put a temporary block on an IP that attempts to access invalid site locations. This is generally enough to stop the scripts and bots. There is a chance that you have inherited an IP address that was home to a game server or something that attracts a lot of attention and you are not set up to handle it. Fail2ban will help. Next, you can look to see if these are coming from a particular IP or set of IPs and block those, or even block the ISP. If this stops it, you know that it was a dumb script running in the background. If it moves are re-appears you know that there is more intelligence behind the activity and that you are being targeted for something. If that does not work, I would start looking at more advanced methods, but developing the correct action, will depend on actual, factual, log based information. For example, it will be necessary to determine if you are experiencing a sync flood, a slow loris, etc. Therefore, if fail2ban does not solve your problem I would suggest starting a new thread in the security forum on the subject and reference this thread with a link.

Looking at that log snippet you posted, the first hit there from 203.152.222.81 is a valid ip, but then the rest of them have that extra junk
appeneded on them... Very odd.

It looks to me like the format for a DNS lookup.