LinuxQuestions.org
LinuxAnswers - the LQ Linux tutorial section.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices

Reply
 
Search this Thread
Old 05-04-2010, 09:25 AM   #1
svenito
LQ Newbie
 
Registered: May 2010
Posts: 10

Rep: Reputation: 0
Question Unusual amount of incoming traffic on eth0


Hi,

I have a VPS Ubuntu jaunty server. Recently I have noticed a huge spike in incoming data as can be seen on the monthly mrtg graph here
http://unlogic.co.uk/share/localhost_eth0-month.png

monitoring eth0 with bmon confirms that there's generally around 200-500 packets coming in per second and 20-400kbps RX at different times.
vnstat shows that yesterday I had 10.77GB alone - this is not a heavy traffic server, as the 15MB out shows.

Looking at 'netstat -an' shows no unusual or persistent connections, syslog shows the usual iptable rules but nothing that could generate so much traffic. Apache access logs also don't show anything unusual.

I'm having a hard time tracking down what might be causing all this incoming data and other people have also had a look on the system without finding any possible cause.

The server runs apache2, mailman, postfix, courier, sshd, smtpd, and mysql, I've pasted the iptables rules below, just in case I missed something there.

I'm basically at a loss on how to track down the cause of the incoming data and would like any input or suggestions as to what I should do next. I've tried iftop but it won't display anything for some reason.

Code:
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
DROP       tcp  --  anywhere             anywhere            tcp dpt:ssh state NEW recent: UPDATE seconds: 60 hit_count: 8 TTL-Match name: SSH side: source 
           tcp  --  anywhere             anywhere            tcp dpt:ssh state NEW recent: SET name: SSH side: source 
DROP       all  --  118.67.190.37        anywhere            
DROP       all  --  ctx1.macleoddixon.com  anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:ssh 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:www 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:8008 
ACCEPT     tcp  --  anywhere             anywhere            tcp flags:ACK/ACK 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:https 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:smtp 
LOG        all  --  anywhere             anywhere            LOG level debug prefix `IPTABLES-IN Default Drop: ' 
DROP       all  --  anywhere             anywhere            

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
REJECT     all  --  anywhere             anywhere            reject-with icmp-port-unreachable 

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
 
Old 05-04-2010, 10:02 AM   #2
smoker
Senior Member
 
Registered: Oct 2004
Distribution: Fedora Core 4, 12, 13, 14, 15, 17
Posts: 2,279

Rep: Reputation: 248Reputation: 248Reputation: 248
Who else is on the server ? It's quite possible to generate large incoming traffic if somebody is using wget or ftp to transfer files.
Automatic updates could pull that kind of bandwidth too.
You say "as the 15MB out shows" but I don't see 15MB anywhere.

Are you losing GB of space on the disks ? If not it could be MRTG that's not reporting correctly or reporting on the wrong thing.

Last edited by smoker; 05-04-2010 at 10:10 AM.
 
Old 05-04-2010, 10:03 AM   #3
svenito
LQ Newbie
 
Registered: May 2010
Posts: 10

Original Poster
Rep: Reputation: 0
No one. Just me.
 
Old 05-04-2010, 10:18 AM   #4
smoker
Senior Member
 
Registered: Oct 2004
Distribution: Fedora Core 4, 12, 13, 14, 15, 17
Posts: 2,279

Rep: Reputation: 248Reputation: 248Reputation: 248
Do you own unlogic.co.uk ?
If not, speak to them.

Oh, I see you do.

Well if you allow comments and membership, it is quite possible that a bot has taken control of your web site and is up to no good.

But until you check if you have disk space going missing, and where it is going missing, there isn't much to be said.

Last edited by smoker; 05-04-2010 at 10:21 AM.
 
Old 05-04-2010, 12:03 PM   #5
svenito
LQ Newbie
 
Registered: May 2010
Posts: 10

Original Poster
Rep: Reputation: 0
Yeah, I own the server and all domains on it.
Disk space history shows no increase in usage that isn't expected, and especially not to that extent (10GB a day I'd notice)
mrtg also plots diskspace usage and there's no increase in storage usage that coincides with the increase in incoming traffic.
 
Old 05-04-2010, 12:15 PM   #6
svenito
LQ Newbie
 
Registered: May 2010
Posts: 10

Original Poster
Rep: Reputation: 0
I've also tried stopping apache, mailman, postfix, courier, and mysql - basically every service except ssh and am still getting a constant 40kbps in, which can go up to 400kbps at times\

I assume I shouldn't expect a ssh session to transmit that much.
 
Old 05-04-2010, 12:51 PM   #7
smoker
Senior Member
 
Registered: Oct 2004
Distribution: Fedora Core 4, 12, 13, 14, 15, 17
Posts: 2,279

Rep: Reputation: 248Reputation: 248Reputation: 248
I would suggest that MRTG or vnstat is not showing the true picture. Are you positive that the vnstat figure was for a day and not cumulative for a different period ?
 
Old 05-04-2010, 12:55 PM   #8
svenito
LQ Newbie
 
Registered: May 2010
Posts: 10

Original Poster
Rep: Reputation: 0
this is the output of vnstat which I just ran now - I ran dumpcap, but I don't really know enough to decipher the output with tcpdump and to see what, if anything, is out of place.

mrtg has always been accurate up to a certain point, when incoming traffic spiked
Code:
Database updated: Tue May  4 16:35:39 2010

        eth0 [disabled]

           received:      35.90 GB (99.9%)
        transmitted:      43.40 MB (0.1%)
              total:      35.94 GB

                        rx     |     tx     |  total
        -----------------------+------------+-----------
        yesterday     10.77 GB |   15.77 MB |   10.79 GB
            today      5.03 GB |    6.37 MB |    5.04 GB
        -----------------------+------------+-----------
        estimated      6.75 GB |       8 MB |    6.76 GB

Last edited by svenito; 05-04-2010 at 12:55 PM. Reason: missed off some info
 
Old 05-04-2010, 01:11 PM   #9
smoker
Senior Member
 
Registered: Oct 2004
Distribution: Fedora Core 4, 12, 13, 14, 15, 17
Posts: 2,279

Rep: Reputation: 248Reputation: 248Reputation: 248
Enable the interface and run tethereal from an ssh session.

man tethereal for the options, but -N n would show network addresses of the connections.

You should soon see what's going in and out. You can write to a file if you want with
Code:
tethereal -N n > filename
 
Old 05-04-2010, 02:05 PM   #10
svenito
LQ Newbie
 
Registered: May 2010
Posts: 10

Original Poster
Rep: Reputation: 0
Thanks for your help so far. Unfortunately when running tethereal I get this output

0.000002 -> Ethernet [Malformed Packet]
0.000002 -> Ethernet [Malformed Packet]
0.000002 -> Ethernet [Malformed Packet]
0.000002 -> Ethernet [Malformed Packet]
0.000002 -> Ethernet [Malformed Packet]
0.000002 -> Ethernet [Malformed Packet]
0.000002 -> Ethernet [Malformed Packet]
0.000002 -> Ethernet [Malformed Packet]
0.000002 -> Ethernet [Malformed Packet]
0.000002 -> Ethernet [Malformed Packet]

Could be related to the errors I get when running snort. Which, IIRC, is something related to a 64bit kernel
 
Old 05-04-2010, 02:36 PM   #11
smoker
Senior Member
 
Registered: Oct 2004
Distribution: Fedora Core 4, 12, 13, 14, 15, 17
Posts: 2,279

Rep: Reputation: 248Reputation: 248Reputation: 248
It could also confuse vnstat.
You need to get that sorted really.
 
Old 05-04-2010, 03:07 PM   #12
svenito
LQ Newbie
 
Registered: May 2010
Posts: 10

Original Poster
Rep: Reputation: 0
I'll look into it and post my findings. Thanks smoker. Appreciate your help
 
Old 05-05-2010, 09:17 AM   #13
svenito
LQ Newbie
 
Registered: May 2010
Posts: 10

Original Poster
Rep: Reputation: 0
Hi,

I ran dumpcap and I get some interesting output. A lot of blocks like this
Code:
16:22:26.1273068390 a4:ca:00:00:00:00 (oui Unknown) > 01:06:00:19:d1:a4 (oui Unknown) Null Information, send seq 0, rcv seq 0, Flags [Command], length 4294967282
16:22:26.1273068390 00:00:00:00:00:00 (oui Ethernet) IP > 00:00:00:00:00:00 (oui Ethernet) Unknown DSAP 0x0c Information, send seq 86, rcv seq 0, Flags [Response], length 4294967282
16:22:26.1273068390 b5:a0:4b:a8:48:98 (oui Unknown) > 2d:96:f7:1e:6a:4f (oui Unknown), ethertype Unknown (0xd01d), length 0: 
        0x0000:  a646 070e dd45 7984 5ee9 77a6 e9c4 4415  .F...Ey.^.w...D.
        0x0010:  bfdd 5134 5674 667a 3b4b 5f60 4a06 01a7  ..Q4Vtfz;K_`J...
        0x0020:  f385 279f 68c0 4180 741a 52c6 9839 7a38  ..'.h.A.t.R..9z8
        0x0030:  a735 acc7 0000 0000 0000 0000 0000 0000  .5..............
        0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0050:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0060:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0070:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0080:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0090:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00a0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00b0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00c0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00d0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00e0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00f0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0100:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0110:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0120:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0130:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0140:  0000 0000 0000 0000 0000 0000 0000 0000  ................
And the zeros go on for pages. Some of it looks like spam too
Code:
16:22:26.1273068724 20:69:6e:20:74:68 (oui Unknown) > 6c:69:73:68:65:64 (oui Unknown), ethertype Unknown (0x
6520), length 0: 
        0x0000:  4d6f 7262 6964 6974 7920 616e 6420 4d6f  Morbidity.and.Mo
        0x0010:  7274 616c 6974 7920 5765 656b 6c79 2052  rtality.Weekly.R
        0x0020:  6570 6f72 742c 2070 7562 6c69 7368 6564  eport,.published
        0x0030:  2062 7920 0000 0000 0000 0000 0000 0000  .by.............
.
.
.
.
I've test my mail server for relay via abuse.net and it reports it to be secure, so I am guessing it's incoming spam
Other blocks with a lot of empty data start with HTML markup, but not with HTML that would appear on my server, like urls from the bbc.co.uk and dooley.boards etc
Code:
16:22:26.1273068725 22:55:41:2d:39:37 (oui Unknown) > 61:63:6b:65:72:28 (oui Unknown), ethertype Unknown (0x
3033), length 0: 
        0x0000:  3034 372d 3122 293b 0d0a 0909 0909 7061  047-1");......pa
        0x0010:  6765 5472 6163 6b65 722e 5f74 7261 636b  geTracker._track
        0x0020:  5061 6765 7669 6577 2829 3b0d 0a09 0909  Pageview();.....
        0x0030:  097d 2063 3e3c 212d 2d5b 6966 206c 7465  .}.c><!--[if.lte
        0x0040:  2049 4520 365d 3e3c 7461 626c 653e 3c74  .IE.6]><table><t
        0x0050:  723e 3c74 643e 3c21 5b65 6e64 6966 5d2d  r><td><![endif]-
        0x0060:  2d3e 3c75 6c3e 3c6c 693e 3c61 2068 7265  -><ul><li><a.hre
        0x0070:  663d 222f 646f 6f6c 6579 2d62 6f61 7264  f="/dooley-board
        0x0080:  732d 696e 632d 3138 3234 636f 2d6d 622d  s-inc-1824co-mb-
        0x0090:  3137 2d78 2d32 332d 6f61 6b2d 6672 616d  17-x-23-oak-fram
        0x00a0:  652d 636f 6d62 696e 6174 696f 6e2d 626f  e-combination-bo
I can share out the full dump file if needed.
 
Old 05-05-2010, 10:06 AM   #14
smoker
Senior Member
 
Registered: Oct 2004
Distribution: Fedora Core 4, 12, 13, 14, 15, 17
Posts: 2,279

Rep: Reputation: 248Reputation: 248Reputation: 248
Run ls -l /var/log and see how big the maillog files are. If they are huge, then incoming spam is a good probability.
You might find some ip addresses worth blocking too. The stuff with html in might be html email. I can't see how incoming traffic could be html based otherwise. Unless you have some kind of aggregator running on your site (php ?)

running tail -f /var/log/maillog might show up the culprits too.
 
1 members found this post helpful.
Old 05-05-2010, 01:10 PM   #15
svenito
LQ Newbie
 
Registered: May 2010
Posts: 10

Original Poster
Rep: Reputation: 0
Thanks smoker.
I'll keep an eye on the mail log and block any repeat offenders

I also found out that the company running the servers are having issues with their bandwidth monitoring software which may also be the cause of the unusual high traffic.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Incoming traffic prioritize dorian33 Linux - Networking 8 10-26-2008 05:44 AM
monitoring the incoming traffic narendra.pant Linux - Networking 1 08-11-2006 04:21 PM
generate large amount of traffic data Mr_C Linux - Networking 3 03-09-2006 11:38 PM
Allow Incoming Traffic clarence1720 Mandriva 15 12-06-2004 11:26 PM
IP tables / squid incoming traffic xilace Linux - Software 5 10-25-2004 01:38 PM


All times are GMT -5. The time now is 05:45 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration