LinuxQuestions.org
View the Most Wanted LQ Wiki articles.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices



Reply
 
Search this Thread
Old 07-17-2012, 07:02 PM   #1
Zippy1970
Member
 
Registered: Sep 2007
Posts: 118

Rep: Reputation: 17

A few hours ago, I received a Munin alert that traffic had exceeded a threshold I had set. When I checked my munin page on my webserver I saw something very strange. I saw the exact same traffic passing over all IP addresses on my webserver. The box handles 8 IP addresses and for each IP address, munin showed the exact same outgoing traffic. Here are two IP addresses that normally have no traffic at all:

http://i515.photobucket.com/albums/t...1970/munin.gif

As you can see, the traffic is exactly the same. If I look at my apache logs, I don't see anything strange going on. When I do a tcpdump, I see a lot of outgoing UDP traffic like this:

Code:
01:44:17.436809 IP host86-128-99-49.range86-128.btcentralplus.com.13673 > my.domain.com.domain:  57113+ [1au] TXT? webtoolkitz.ru. (43)
01:44:17.436811 IP my.domain.com.domain > host86-128-99-49.range86-128.btcentralplus.com.13673  57113 17/2/2 TXT[|domain]
(I replaced my real domain for "my.domain.com")

I downloaded and installed the latest version of rkhunter on my system and that found nothing.

I also see a lot of this with tcpdump:

Code:
01:27:26.016121 IP my.domain.com.www > 194-28-157-94.zenprotection.com.58400: S 4071749720:4071749720(0) ack 1 win 5840 <mss 1460>
01:27:26.062299 IP 194-28-157-94.zenprotection.com.58400 > my.domain.com.www: S 0:0(0) win 8192
01:27:26.062310 IP my.domain.com.www > 194-28-157-94.zenprotection.com.58400: S 4071749720:4071749720(0) ack 1 win 5840 <mss 1460>
01:27:26.062312 IP 194-28-157-94.zenprotection.com.58400 > my.domain.com.www: S 0:0(0) win 8192
01:27:26.062315 IP my.domain.com.www > 194-28-157-94.zenprotection.com.58400: S 4071749720:4071749720(0) ack 1 win 5840 <mss 1460>
01:27:26.062395 IP 194-28-157-94.zenprotection.com.58400 > my.domain.com.www: S 0:0(0) win 8192
01:27:26.062399 IP my.domain.com.www > 194-28-157-94.zenprotection.com.58400: S 4071749720:4071749720(0) ack 1 win 5840 <mss 1460>
01:27:26.121356 IP 194-28-157-94.zenprotection.com.58400 > my.domain.com.www: S 0:0(0) win 8192
01:27:26.121361 IP my.domain.com.www > 194-28-157-94.zenprotection.com.58400: S 4071749720:4071749720(0) ack 1 win 5840 <mss 1460>
01:27:26.155947 IP 194-28-157-94.zenprotection.com.58400 > my.domain.com.www: S 0:0(0) win 8192
Edit: It's mostly the request from the first window above that generate the bulk of the traffic. Those are apparently DNS requests.

So what's going on? Or how can I find out what's going on?
Attached Images
File Type: gif munin.gif (56.2 KB, 12 views)

Last edited by Zippy1970; 07-18-2012 at 05:27 AM.
 
Old 07-18-2012, 04:54 AM   #2
Noway2
Senior Member
 
Registered: Jul 2007
Distribution: Ubuntu 10.10, Slackware 64-current
Posts: 2,124

Rep: Reputation: 776Reputation: 776Reputation: 776Reputation: 776Reputation: 776Reputation: 776Reputation: 776
Looks to me like either a vulnerability scanner or DOS script. Do you perchance run a public DNS server?
 
Old 07-18-2012, 06:53 AM   #3
Zippy1970
Member
 
Registered: Sep 2007
Posts: 118

Original Poster
Rep: Reputation: 17
I'm running Bind (9) if that's what you mean.
 
Old 07-18-2012, 11:29 AM   #4
Noway2
Senior Member
 
Registered: Jul 2007
Distribution: Ubuntu 10.10, Slackware 64-current
Posts: 2,124

Rep: Reputation: 776Reputation: 776Reputation: 776Reputation: 776Reputation: 776Reputation: 776Reputation: 776
Yes, but how are you running it? Do you make it accessible to the world or is it local only? If it is accessible, is it "open" or designed to only resolve your public facing servers? For example, I too run Bind and it is the authoritative name server for my domains and if you query it for one of them on their public facing address, it will give you the proper address. If you instead query for something like google.com, it will refuse to answer that for you.

There are two things that I think you need to consider and review. One is whether or not your DNS is configured as it should be. BTW, don't run an open DNS. Two, if you are getting hit with excessive queries finding a proper counter measure to discourage the practice OR switching to a different DNS such as your registrar's which may be better equipped to handle this sort of thing.
 
Old 07-18-2012, 11:45 AM   #5
Zippy1970
Member
 
Registered: Sep 2007
Posts: 118

Original Poster
Rep: Reputation: 17
Quote:
Originally Posted by Noway2 View Post
Yes, but how are you running it? Do you make it accessible to the world or is it local only?
I think it's local only. I do know recursion is disabled and that it's setup to answer on random ports.

Quote:
There are two things that I think you need to consider and review. One is whether or not your DNS is configured as it should be.
I really am not very well known with bind so I can't be sure it's configured as it should be. Here are my config files:

/etc/bind/named.conf:
Code:
zone "255.in-addr.arpa" {
        type master;
        file "/etc/bind/db.255";
};

include "/etc/bind/named.conf.local";
/etc/bind/named/conf.local:
(empty)

/etc/bind/named.conf.options:
Code:
acl recurseallow { 1.2.3.4; 127.0.0.1; };

options {
        directory "/var/cache/bind";

        // If there is a firewall between you and nameservers you want
        // to talk to, you might need to uncomment the query-source
        // directive below.  Previous versions of BIND always asked
        // questions using port 53, but BIND 8.1 and later use an unprivileged
        // port by default.

        // query-source address * port 53;

        // If your ISP provided one or more IP addresses for stable
        // nameservers, you probably want to use them as forwarders.
        // Uncomment the following block, and insert the addresses replacing
        // the all-0's placeholder.

        // forwarders {
        //      0.0.0.0;
        // };

        auth-nxdomain no;    # conform to RFC1035
        listen-on-v6 { any; };

        allow-recursion { recurseallow; };recursion yes;

};
 
Old 07-18-2012, 01:50 PM   #6
Noway2
Senior Member
 
Registered: Jul 2007
Distribution: Ubuntu 10.10, Slackware 64-current
Posts: 2,124

Rep: Reputation: 776Reputation: 776Reputation: 776Reputation: 776Reputation: 776Reputation: 776Reputation: 776
I am not a Bind expert, more of a medium level user and perhaps one of the experts (e.g. Bathory) will chime in, but I am pretty sure that:
Quote:
allow-recursion { recurseallow; };recursion yes;
is allowing public access to your DNS. If your DNS is public and recursive, it is only a matter of time before it will be abused, which seems to be your problem. Listening on non standard or random ports will not stop the problems as a port scan will quickly reveal the service. It only stops the lowest of the low who are too lazy or stupid to look for it elsewhere.

I would either configure an ACL to allow recursion ONLY from your private LAN side, and / or have it listen on port 53 and block this traffic from outside of your network.
For example:
Code:
acl mynet { 192.168.0.0/24; };

options {
        listen-on port 53 { 127.0.0.1; 192.168.0.xx; };
        listen-on-v6 port 53 { ::1; };
        allow-query     { any; };
        recursion yes;
        allow-recursion { mynet; };
};
 
Old 07-18-2012, 02:50 PM   #7
Zippy1970
Member
 
Registered: Sep 2007
Posts: 118

Original Poster
Rep: Reputation: 17
Quote:
Originally Posted by Noway2 View Post
I am not a Bind expert, more of a medium level user and perhaps one of the experts (e.g. Bathory) will chime in, but I am pretty sure that: is allowing public access to your DNS.
I'm no bind expert either but I'm pretty sure what you did is pretty much the same as what I did.
 
Old 07-19-2012, 02:50 AM   #8
subfolder
LQ Newbie
 
Registered: Jul 2012
Posts: 4

Rep: Reputation: Disabled
This is your standard bill-of-fare DNS amplification attack. The attack is directed at the site whose information is being requested, not you (directly) - but they're using your DNS (and many others) as a means of attacking the target.

As you've probably guessed, the requesting IP addresses are forged. Your DNS requests the record from the target. The cumulative effect of all those requests from all the venue DNS servers is designed to overwhelm the target DNS. (Secondarily, the real version of the forged address receives the non-authoritative response from your DNS.)

If you look up ways to deal with amplification attacks, you'll find an answer to suit you. The answers I've found involve turning off recursion, which I cannot implement, since I require recursion. Alternatively, I have done two things:
First, I set up a fake entry for the target address, as if I'm authoritative for it. This way I'm no longer helping the attack on the primary target.
Second, I wrote a quick-n-dirty perl script that reads a tail of the DNS log, every second, then adds the forged IP to the firewall (ipfw for me using FreeBSD - iptables for you Linux folks.) Of course, my DNS logging includes a "category queries" and "category security" logs, so I have the full info of the requests, and I'm not stuck having to do a tcpdump when I _happen_ to notice a traffic spike.

Now all I'll have to do is clean up the firewall list whenever the attack is over. In my experience, these attacks are short-lived; a few days to a few weeks, at most.


Next, I'll cut-and-paste a security e-mail I got from one of my hosts on how to preclude recursion, which they say solves the problem.
 
Old 07-19-2012, 02:55 AM   #9
subfolder
LQ Newbie
 
Registered: Jul 2012
Posts: 4

Rep: Reputation: Disabled
This is from the e-mail I got from hivelocity/noc4hosts whom is one of the companies I have the misfortune of being a customer of. (Therefore, the IP's mentioned are specific to their service - you'll have to tailor to your own services. You may wish to contact your upstream provider for assistance and details.)
-=-=-=-

1. Linux Servers running Bind you need to modify file /etc/named.conf. (Please back up the file before making any changes)

Notice the first line is setup for 127.0.0.1. This will allow the local Linux machine to query locally if the linux
machine has nameserver 127.0.0.1 in there to be able to query locally. The example here adds additional Hivelocity IP
blocks which allows these blocks to use your DNS for recursive and caching lookups. You could edit these lines and put
only your required or preferred subnets to lock down your DNS even further.

options {
recursion yes;
allow-recursion { 127.0.0.1/32; 199.193.112.0/21;
91.209.57.0/24; 38.112.194.0/24;
199.119.100.0/22; 216.109.224.0/20;
199.167.144.0/21; 68.233.224.0/19;
96.31.64.0/19; 74.50.96.0/19;
206.51.232.0/21; 206.51.224.0/21;
69.46.0.0/19; 66.96.80.0/20; };
allow-query-cache { 127.0.0.1/32; 199.193.112.0/21;
91.209.57.0/24; 38.112.194.0/24;
199.119.100.0/22; 216.109.224.0/20;
199.167.144.0/21; 68.233.224.0/19;
96.31.64.0/19; 74.50.96.0/19;
206.51.232.0/21; 206.51.224.0/21;
69.46.0.0/19; 66.96.80.0/20; };
};

After the change you need to restart Bind with command "service named restart" or "/etc/init.d/named restart"
 
Old 07-19-2012, 04:55 AM   #10
Noway2
Senior Member
 
Registered: Jul 2007
Distribution: Ubuntu 10.10, Slackware 64-current
Posts: 2,124

Rep: Reputation: 776Reputation: 776Reputation: 776Reputation: 776Reputation: 776Reputation: 776Reputation: 776
I missed the "acl recurseallow { 1.2.3.4; 127.0.0.1; };" line. Like I said, no expert!
It looks like subfolder is thinking along the same lines, except they add the allow-query-cache block. Rereading these posts a few times brought me to two questions:
1 - are you sure that the DNS response, i.e. the part where TCP dump says: TXT[|domain] that it isn't sending back an error message like REFUSED rather than a DNS record. This would still be data in an amplification attack that the recipient would either have to block or process.
2 - If it is not an error message, is your network set up so that data arriving on your public IP 1.2.3.4 appears to your DNS as coming from your IP and hence is allowed? One of the differences between your ACL and mine is that I don't allow recursion on my public IP (or even on the DMZ IP that it translates to). Instead, I specifically allow queries on it, with the allow query any, but not recursion.
 
Old 07-19-2012, 06:01 AM   #11
subfolder
LQ Newbie
 
Registered: Jul 2012
Posts: 4

Rep: Reputation: Disabled
For the third time, the configuration stub I cited (above) was a direct cut-n-paste from an email from my provider. I cannot verify its veracity. I did not implement it, nor can I, since my DNS's must be unilaterally recursive. Since it's directed at Linux users (I use FreeBSD), I suspect it's likely it may be a usable example in this forum.
 
Old 07-19-2012, 04:44 PM   #12
Noway2
Senior Member
 
Registered: Jul 2007
Distribution: Ubuntu 10.10, Slackware 64-current
Posts: 2,124

Rep: Reputation: 776Reputation: 776Reputation: 776Reputation: 776Reputation: 776Reputation: 776Reputation: 776
At subfolder: I understand that it is a cut and paste with IP addresses designated for another system. What I am saying is that if I am understanding it correctly, your suggesting the same thing that we are: to use an ACL to stipulate only which IP addresses are allowed to perform recursive queries. My follow up post from this morning was to double check that 1) the existing ACLS may in fact be working and the resulting traffic is a REFUSED message. If this is the case, there is little more that can be done, other than to use a tactic to limit the number of refusals that will be sent before the server simply ignores the query, b) that the fact that the public IP address is listed in the ACL isn't somehow invalidating the ACL.
 
Old 07-20-2012, 09:07 AM   #13
bathory
Guru
 
Registered: Jun 2004
Location: Piraeus
Distribution: Slackware
Posts: 10,975

Rep: Reputation: 1343Reputation: 1343Reputation: 1343Reputation: 1343Reputation: 1343Reputation: 1343Reputation: 1343Reputation: 1343Reputation: 1343Reputation: 1343
Hi,

According to your configuration your dns is not an open resolver. You can omit the "recursion yes;" as the "allow-recursion..." supersedes it.
Anyway, from the logs in your 1st post looks like host86-128-99-49.range86-128.btcentralplus.com is using your server as a resolver. Is this host part of your LAN? I guess it is because the recursion ACL.
So if you get much traffic from it, you should check if it's compromised somehow and sends out spam or something like that, that make extensive use of dns.

Regards
 
Old 07-21-2012, 06:26 AM   #14
Zippy1970
Member
 
Registered: Sep 2007
Posts: 118

Original Poster
Rep: Reputation: 17
Quote:
Originally Posted by bathory View Post
Hi,

According to your configuration your dns is not an open resolver.
It was.

When I first noticed the traffic, I checked if bind was setup to disallow recursion from outside my network. It wasn't. So I added the lines for disallowing recursion.

At first I thought it made no difference because I still saw the same lines in my log files. What I didn't notice however, was that data traffic dropped to almost nothing because those recursive lookups were denied. So in fact disallowing recursion was the solution (even though I didn't notice it straight away).

Anyway, the attack stopped as abruptly as it started after a day or so. Thank you all for your help. It helped me understand a little bit more how bind works.
 
Old 07-23-2012, 12:50 PM   #15
subfolder
LQ Newbie
 
Registered: Jul 2012
Posts: 4

Rep: Reputation: Disabled
Apparently, default logging settings of bind, in Linux, don't give information that can be easily discerned for viewing the details of an attack. I rely on the logs to see the data commonly associated with the more common attacks. Also, I have a quick-n-dirty perl-script that continually monitors the logs to automatedly add a firewall entry, as they are found. Additionally, for a server with multiple IP's, I have a separate bind process for each of the IP's, each with their own configuration and logging.

Here's the relevant snippet of one of them - you may find useful;

......

listen-on { 111.222.33.44; };
};

logging{
channel dns_log {
file "/var/log/dns.111.222.33.44.log" versions unlimited size 10m;
severity debug 99;
print-severity yes;
print-time yes;
print-category yes;
};
channel dns_loh {
file "/var/log/dns.111.222.33.44.loh" versions unlimited size 10m;
severity debug 99;
print-severity yes;
print-time yes;
print-category yes;
};
channel dns_loi {
file "/var/log/dns.111.222.33.44.loi" versions 99 size 10m;
severity debug 99;
print-severity yes;
print-time yes;
print-category yes;
};
category queries {
dns_log;
};
category security {
dns_loh;
};
category default {
dns_loi;
};
};

........

"category queries" produces a log with entires like this;

23-Jul-2012 16:37:23.147 queries: info: client 176.227.198.146#24983: query: isc.org IN ANY +E
23-Jul-2012 16:37:23.199 queries: info: client 176.227.198.146#16996: query: isc.org IN ANY +E
23-Jul-2012 16:37:23.350 queries: info: client 176.227.198.146#16326: query: isc.org IN ANY +E
23-Jul-2012 16:37:23.600 queries: info: client 176.227.198.146#24983: query: isc.org IN ANY +E
23-Jul-2012 16:37:23.670 queries: info: client 176.227.198.146#16996: query: isc.org IN ANY +E
23-Jul-2012 16:37:23.851 queries: info: client 176.227.198.146#16326: query: isc.org IN ANY +E
23-Jul-2012 16:37:23.986 queries: info: client 176.227.198.146#24983: query: isc.org IN ANY +E
23-Jul-2012 16:37:24.426 queries: info: client 176.227.198.146#24983: query: isc.org IN ANY +E
23-Jul-2012 16:37:24.625 queries: info: client 176.227.198.146#16996: query: isc.org IN ANY +E
23-Jul-2012 16:37:24.999 queries: info: client 176.227.198.146#16996: query: isc.org IN ANY +E
23-Jul-2012 16:37:25.078 queries: info: client 176.227.198.146#16996: query: isc.org IN ANY +E

"category security" produces a log that looks like this;

23-Jul-2012 16:37:02.373 security: debug 3: client 176.227.198.146#16996: request is not signed
23-Jul-2012 16:37:02.373 security: debug 3: client 176.227.198.146#16996: recursion available
23-Jul-2012 16:37:02.373 security: debug 3: client 176.227.198.146#16996: query (cache) 'isc.org/ANY/IN' approved
23-Jul-2012 16:37:02.908 security: debug 3: client 176.227.198.146#16996: request is not signed
23-Jul-2012 16:37:02.908 security: debug 3: client 176.227.198.146#16996: recursion available
23-Jul-2012 16:37:02.908 security: debug 3: client 176.227.198.146#16996: query (cache) 'isc.org/ANY/IN' approved
23-Jul-2012 16:37:02.932 security: debug 3: client 176.227.198.146#16326: request is not signed
23-Jul-2012 16:37:02.932 security: debug 3: client 176.227.198.146#16326: recursion available
23-Jul-2012 16:37:02.932 security: debug 3: client 176.227.198.146#16326: query (cache) 'isc.org/ANY/IN' approved
23-Jul-2012 16:37:03.030 security: debug 3: client 176.227.198.146#24983: request is not signed
23-Jul-2012 16:37:03.030 security: debug 3: client 176.227.198.146#24983: recursion available
23-Jul-2012 16:37:03.030 security: debug 3: client 176.227.198.146#24983: query (cache) 'isc.org/ANY/IN' approved
23-Jul-2012 16:37:03.416 security: debug 3: client 176.227.198.146#16996: request is not signed
23-Jul-2012 16:37:03.417 security: debug 3: client 176.227.198.146#16996: recursion available
23-Jul-2012 16:37:03.417 security: debug 3: client 176.227.198.146#16996: query (cache) 'isc.org/ANY/IN' approved
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Route traffic to certain IP addresses out a certain network interface cati Linux - Networking 2 04-13-2012 09:51 AM
Block all incoming mail traffic except certain addresses jennyzon Linux - Networking 1 01-13-2009 07:35 AM
Binding 2 NICs (MAC addresses) to 2 IP Addresses in same Subnet RedHat EL4.0 skhira Linux - Networking 13 02-24-2008 09:16 PM
Binding 2 NICs (MAC addresses) to 2 IP Addresses in same Subnet RedHat EL4.0 skhira Linux - Networking 1 02-09-2008 08:17 AM
mechanics of mapping process memory addresses to physical addresses on amd64 Tischbein Linux - Kernel 2 02-01-2007 09:09 PM


All times are GMT -5. The time now is 05:29 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration