This is an x-post from the General Forum, because I believe this has wound up to be a network issue.

A few days ago I noticed that roughly every minute, one of my servers runs choppy, and the CPU usage jumps up to 19-20% for about 20 seconds. My telnet or ssh connection becomes choppy, and often is dropped as a result.

Today I think I have recognized this to be network problem. I used atop to view network traffic alongside CPU info. It is an extended version of top, and refreshes the screen every 10 seconds by default. I noticed that normally, when CPU is near-idle, I get roughly 100 Packets In (RX) per atop refresh period. This is probably mostly traffic generated from my ssh connection. However, during these CPU spikes I described, I get roughly 29,000 RX Packets per refresh period. My outbound packets (TX) are stable, and hover around 1-2 per refresh.

Despite an uptime of just about 15 hours, ifconfig tells me I have over 106,105,362 RX packets and 31,000 TX packets. I have hardly run anything on this machine since reboot. I run a netstat and there is only one tcp connection open during all this, my own ssh session.

Any thoughts as to what might be happening??? Any way to tell what these packets are or where they are coming from? Is there a way to stop them??

Thanks in advance,
-B

Load up iptraf and take a look at the connections hitting your machine.

you can find it here :

http://freshmeat.net/search/?q=iptraf&section=projects

It's not as comprehensive as say, ethereal, but it's also a whole lot easier to set up.

With that running you should be able to see connections hitting your external interface, and where they come from.

(I too have used atop, and it's a nice program )

Slick.

Thanks for the advice, I have installed IPTraf and it works great, however I am having trouble making heads or tails of the data I am collecting.

I am getting MANY of these messages (IP's masked to protect the innocent):
UDP (404 bytes) from 6x.9x.2xx.2xx:1321 to 2xx.7x.6x.1xx:1434 on eth0
UDP (404 bytes) from 6x.9x.2xx.2xx:1321 to 2xx.1xx.2xx.6x:1434 on eth0
UDP (404 bytes) from 6x.9x.2xx.2xx:1321 to 2xx.9x.6x.1xx:1434 on eth0
UDP (404 bytes) from 6x.9x.2xx.2xx:1321 to 2xx.2xx.1xx.8x:1434 on eth0

The from address is always the same, but the to address changes. None of them are mine. I have tried to ping these systems, all of which result no replies.

Not quite sure what's going on, except UDP/1434 is an ms-sql port, and could be related to that latest virus.

I'm wondering why I'd be receiving these at all, when I'm not the TO or FROM.... local misconfiguration in network setup maybe? Is this normal?

any ideas? thanks in advance....
-B

You are most certainly getting hit by the slammer worm, yes.

If I recall correctly it doesn't spoof it's addresses, but it *does* send out packets in both a sequential IP address hit, and also various random patterns.

So.. the address it's being sent to *should* be yours, and the address it's being sent from *should* be the infected machine. You may very well have problems contacting the affected machine as it's more than likely too busy spewing out packets to listen to your ping.

Quite why the destination address isn't yours is a mystery, but you might have a misconfigured network config there somewhere, yes.

It's relatively easy to fix, mind - simply block UDP 1434

You can do this with an iptables command similar to the one below :

iptables -A INPUT -p udp --dport 1434 -j DROP

Stick it somewhere up the top of your iptables script. Note, however, that it will disable use of SQL - if you're running it.

[edit]
Ahh.. it occurs to me that you might be on a shared hub. In which case you'll see traffic destined for any machines also on that hub, or indeed uplinked from that hub.
So, perhaps not a configuration issue, and more 'how it works'..
[/edit]

Slick.

Slick,
A few more things...

First off, your thorough responses and insight are greatly appreciated. People like you make this board a great resource. Your ideas all makes sense based on what I know of the ATTBI network (which isn't much...)

At first, I didn't think this was related to the other thread I started entitled "Help reading TCPDUMP output". However, after digging further with TCPDUMP, I believe these network spikes are NOT the port 1434 UDP traffic as we previously thought, but instead the empty RESET tcp packets. Here's why:

I ran a tcpdump -c 1000 during an idle (non-spiking) time, and it collected 1000 packets in about 40 seconds. They all look, from what I can tell, "normal", including a fair amount of 1434 UDP traffic. I ran the same tcpdump during a spike and it collected 1000 packets in less than a half-second.

Upon reviewing the tcpdump output from the full duration of a spike (which lasted roughly 55 seconds), I received roughly 3,000 of these RESET packets per second. They are all coming from the same client2.attbi.com address we were discussing in the other thread, and all destined for 10.10.10.10:6667.

You say that these packets are not destined for me, and that makes sense, but they seem to be completely destroying my network bandwidth and using up some CPU as well. As I stated before, my network gets choppy, the CPU jumps up to 20% and often my remote connections get dropped. ( ) This happens roughly every 2-3 minutes, but that interval seems to vary.

<SIDENOTE>
Just for kicks, I tried ping 10.10.10.10 and got:

PING 10.10.10.10 (10.10.10.10) from 6x.9x.2xx.1xx : 56(84) bytes of data.
From 2x.1xx.8.3x: Time to live exceeded
From 2x.1xx.8.3x: Time to live exceeded
...

I also tried pinging the source address and got no response.
</SIDENOTE>

Is there any way to prevent this traffic? How could it possibly be legitimate? Should I call AT&T, and if so, what could they do?

Thanks again for all your help,
-B

FYI , For some reason, IPTraf was not seeing these packets. It could be because they were RESET, and therefore no connection was established, but I don't know.

UDP *is* connectionless ... that's why a pest like slammer
can be so active :) ... no waiting for acknowledgements,
just blasting them out.

Cheers,
Tink

First :

Thanks for the back-pat...

Anyway.. As Tinkster says, UDP is connectionless. And slammer uses this to great effect. What you're seeing below is definately slammer, although the traffic quoted in your other thread is not.

I'll write here about the questions asked in this thread.

10.<anything> is a private network address. In a similar manner to the more common 192.168.xxx.xxx. As such it's not *supposed* to be floating around on the internet. In fact, internet routers, if set up properly, should be set to drop private IP packets if they see them. However, a large number don't, and this is how you end up with spoofed addresses being used in Denial of Service attacks.

That said, AT&T are probably using 10.x addresses for the cable modems (which you should never see, but more on that later). So, it wouldn't be altogether suprising to see a 10.x address on the network. Due to it's 'shared' nature, you *will* get packets sent to you but not specifically destined for you, and there's nothing you, personally, can do about it.
What you *can* do is to block anything from a 10.x address when it hits your network card. You shouldn't be recieving anything in that subnet anyway...

You can do that with a line like the following in your iptables rules :

$IPTABLES -A INPUT -i $EXTIF -s 10.0.0.0/8 -j DROP

where $IPTABLES = /sbin/iptables
$EXTIF is your external interface
$INTIF is your internal interface

I would suggest also blocking all the *other* private ip addresses (which again, shouldn't be hitting your internet interface)

$IPTABLES -A INPUT -i $EXTIF -s 172.16.0.0/12 -j DROP # RFC1918 Private
$IPTABLES -A INPUT -i $EXTIF -s 0.0.0.0/8 -j DROP # Broadcast
$IPTABLES -A INPUT -i $EXTIF -s 127.0.0.0/8 -j DROP # Loopback
$IPTABLES -A INPUT -i $EXTIF -s 192.168.0.0/16 -j DROP #
$IPTABLES -A INPUT -i $EXTIF -s 192.0.2.0/24 -j DROP # TEST-NET
$IPTABLES -A INPUT -i $EXTIF -s 169.254.0.0/16 -j DROP # Unconfigured DHCP
$IPTABLES -A INPUT -i $EXTIF -s 224.0.0.0/4 -j DROP # Class D / Multicast
$IPTABLES -A INPUT -i $EXTIF -s 240.0.0.0/5 -j DROP # Class E / Reserved
$IPTABLES -A INPUT -i $EXTIF -s 255.255.255.255 -j DROP # Broadcast

Note on the above though - you may have some issues with traceroute if you block 10.x addresses and one of the hops replies from a 10.x address. It's rather a large blanket rule which can have unforseen side-effects.

That rule should stop your machine from becoming unresponsive, as packets don't have to traverse your firewall rules (put the above rules at the *top* of your list) to figure out if it should accept/drop the packet 3000 times a second. It can hit the first DROP and you're done with it.

Prevention - see above.
It's not legitimate - your neighbour has a 'misconfigured machine'. It could be intentional, it could be viral, or it could be user error..

I really doubt you'll get a positive response from AT&T. In my experience, if you start talking about packet-level stuff to a level 1 tech they'll simply go 'huh?'
You might have better luck if you press your case and get to (eventually) a tier 3 support person who generally knows what a UDP packet is, that you shouldn't be seeing 10.x traffic, and why.

I've had *dismal* experiences with my customer support though (comcast), and don't hold out any more hope for yours...

As for what they could do - they can cut off the link to your errant neighbour, either until (s)he fixes it, or depending on how malicious the intent, altogether.

Last :


The reason for that is that IPTraf doesn't put your network card into promiscuous mode by default, and tcpdump does.

Promiscuous mode means that a network card will listen to *everything* that so much as waves in it's direction. Any traffic not meant for you which gets sent to it (by nature of the shared line you enjoy with your neighbours) it'll show. Ordinarily network cards only listen to things sent specifically to them (by MAC address) and nothing more (Okay.. broadcast stuff too, for the finicky). Obviously, if it's not for you, then you don't *want* to be listening to it - it's just more work for your card / computer to do.

You can tell IPTraf to 'go promiscious' in one of the menus
--> configure --> Force Promiscuous mode.

It'll then show you all the stuff that's not destined for your machine (your 10.x stuff) aswell as stuff that is.

*pant*

Slick.

Slick,
Although your idea makes sense, unfortunately the addition of the 10.0.0.0/8 -j DROP rule did not solve my problem. I agree with the "dismal" service experience on the part of my internet provider, they did not really know what a "packet" was. I contacted every department (including Abuse) in order to alert them to this issue, but I am still getting the incredible amount of traffic that is ruining my bandwidth.
It was elevated to the "Server" level, meaning normal technicians cannot help, and they need to call in the real techies. Unfortunately it looks like I'll have to wait until they do something about this before my bandwidth is saved.

thanks again for all the insight... pat, pat.

-BDC