Debian/Asterisk box eth0 going promisc odd times, rootkit scanned clean...

JakeX · 03-29-2006, 12:54 PM

I've got a box running:

Quote:

Linux version 2.6.6-1-686 (herbert@gondolin) (gcc version 3.3.3 (Debian 20040401))

And the latest rls of Asterisk 2.1 (i believe, version doesn't matter in this instance).

Anyways, I find that the box randomly (say every 1-3 days) has eth0 go into promisc mode:

Quote:

Mar 29 12:50:55 debian kernel: NET: Registered protocol family 17
Mar 29 12:50:55 debian kernel: device eth0 entered promiscuous mode
Mar 29 12:51:55 debian kernel: device eth0 left promiscuous mode
Mar 29 12:55:13 debian kernel: device eth0 entered promiscuous mode
Mar 29 12:57:52 debian kernel: device eth0 left promiscuous mode

Today is ANOTHER one of those days. I have ran chkrootkit (latest rls in the past 3 days). I've ran rkhunter (right after this promisc mode happened ~1pm).

TCPDUMP isn't being used on this system, no other users have access to this system. Its a firewalled Asterisk box.

iptables --list:

Quote:

Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all -- debian anywhere
ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED
ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:2222
ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:8888
ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:sip
ACCEPT udp -- anywhere anywhere udp dpt:sip
ACCEPT udp -- anywhere anywhere udp dpts:10000:20000
REJECT all -- anywhere anywhere reject-with icmp-port-unreachable

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Port 2222 ==> sshd
Port 8888 ==> apache2
sip ports ==> asterisk
10000:20000 > media ports for asterisk (take from voip-info.org for asterisk firewall).

Now I've done a lot of reading up on what could be causing eth0 to go into promisc mode, here is my ifconfig -a:

Quote:

eth0 Link encap:Ethernet HWaddr 00:0D:61:XX:XX:XX
inet addr:XXX.XXX.XXX.XXX Bcast:XXX.255.255.255 Mask:255.255.255.224
inet6 addr: XXXX::XXX:61ff:fe30:23bc/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:803485 errors:0 dropped:0 overruns:0 frame:0
TX packets:67659 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:57304673 (54.6 MiB) TX bytes:7833665 (7.4 MiB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

sit0 Link encap:IPv6-in-IPv4
NOARP MTU:1480 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

ip's XXX'd out

Overview of what happens:
eth0 goes promisc at random times, then service returns to normal (as far as asterisk users notice). This machine is test level for the voicemail app. eth0 going into promisc 'cuts off' any connection to the asterisk for like a good 15 min, then it 'magically' works again.

ifpromisc (from chkrootkit) shows no promisc mode. I see it in the log tho. It happens every couple days, I'm new to linux (about 6 months usage.. but hazy in many things). And any probs I run into I rack my brains over (many I've been able to figure out). This one I can't seem to figure out, all posts or google searches scream 'you've been hacked!' But I run the scans, iptables running in place. I've ran nessus scans that only find the apache & sshd ports :|

Scratching my head here.. if anyone could help out, and please let me know the commands to do something if you do. Just in case I am not familiar with it. Appreciate any help with this matter.. thanks.

unSpawn · 03-29-2006, 07:05 PM

ifpromisc (from chkrootkit) shows no promisc mode.
Long story short that's because of how apps request devices to use promiscuous mode: (SIOCGIFFLAGS ioctl vs setsockopt, IFF_PROMISC vs MR_PACKET_PROMISC. The CRT maintainer believes deprecated ioctl check in ifpromisc and alike are the only "useful" method, hence the dection prob: simply use /sbin/ip (iproute2 package) to detect promisc mode properly. I've patched CRT for ages for that now, goes somewhat like this:

Code:

# Tools
-TOOLS="aliens asp bindshell lkm rexedcs sniffer w55808 wted scalper slapper z2 chkutmp"
+TOOLS="aliens asp bindshell lkm rexedcs sniffer w55808 wted scalper slapper z2 chkutmp promisctest"
 
 ######################################################################
+# custom functions
+
+promisctest () { 
+ip="/sbin/ip"
+	if [ ! "$SYSTEM" = "Linux" ]; then
+		printf "%snot tested: non-Linux system.\n"
+		return ${NOT_TESTED}
+	elif [ ! "${VERSION:2:1}" -ge "4" ]; then
+		printf "%snot tested: unsupported kernel version.\n"
+		return ${NOT_TESTED}
+	elif [ ! -x ${ip} ]; then
+		printf "%snot tested: could not exec ${ip}.\n"
+		return ${NOT_TESTED}
+	fi
+	printf "%s\n"
+	${ip} link show | ${egrep} "^[0-9]" | while read DEVF; do
+	DEVF=( ${DEVF} ); let DEVFLEN="${#DEVF[0]}+${#DEVF[1]}+${#DEVF[2]}"
+	if [ "${#DEVF[2]}" -le "4" -o "${DEVFLEN}" -le "9" ]; then
+		printf "%snot tested: device ${DEVF[1]} has only device flags: ${DEVF[2]}.\n"
+	else	
+		printf "%s${DEVF[@]}" | ${egrep} -qe "PROMISC"
+		case "$?" in
+		1)	if [ "${EXPERT}" = "t" ]; then
+				printf "%s${DEVF[1]} has device flags: ${DEVF[2]}\n"
+			else
+				printf "%s${DEVF[1]}\tis not promisc\n"
+			fi;;
+		0)
+			if [ "${EXPERT}" = "t" ]; then
+				printf "%s${DEVF[1]} has device flags: ${DEVF[2]}\n"
+			else
+				printf "%s${DEVF[1]}\tIS PROMISC\n"
+			fi;;
+		esac
+	fi
+	done
+}
+
+######################################################################
 # tools functions
 
 #

This shellscript snippet shows you who has requested membership using setsockopt() though it could be off a wee bit:

Code:

sepSpace() { tr -s " " | sed -e "s/^ //g" -e "s/ / /g" | cut -d " " -f "$@"; }
inodes=$(grep /proc/net/packet -ve ^sk|sepSpace 9); inodes=$(echo $inodes|tr " " "|")
pids=$(lsof | egrep "($inodes)" | sepSpace 2); for p in $pids; do stat -c %N /proc/"$p"/exe; done

Kinda run like "tail log | grep promisc | execute snippet", something like that or cronjob it per 15 minutes or other relevant interval. Kinda crude method results matter. Whatever binaries it shows, just checksum (dunno? debsum?) it. And I'm pretty sure it won't show "weird" stuff: post anything you think is relevant.

JakeX · 03-30-2006, 10:55 AM

thanks unSpawn, but I'm a little confused by your answer. I don't understand the CRT patch thing, or what to do with that. Sorry its all a little beyond my level of linux comprehension currently.

I understand the last part regarding the shellscript, will that log the result somewhere for me? This machine is remote, so i ssh into it, and would need to run the command in the bg and have it log to file so I can check in on it later.

/var/log/messages shows when the adapter goes promisc (at least it is logged there). Goes promisc for roughly 1-2 min then back to normal. Its random, but not frequent (like i said about every 1-3 days avg).

so I should do the following ?

Quote:

tail -f /var/log/message | grep "promisc" | /home/jakub/promisc.check.sh

Where promisc.check.sh --> shellscript you submitted
So from that I understand, tail the log, if it gives you "promisc" then run the script? should I also output that like so:

Quote:

tail -f /var/log/message | grep "promisc" | /home/jakub/promisc.check.sh > /home/jakub/promisc.log

Sorry as I said, I'm new to it all :P

Thanks

unSpawn · 03-30-2006, 02:48 PM

but I'm a little confused by your answer.
OK. Sorry for the confusion, then.

Just add this line in /etc/crontab:

Code:

*/15 * * * * root /sbin/ip link show | grep -qi promisc && /usr/local/sbin/promisc.check.sh 2>&1>> /var/log/promisc.check.log

This will run /sbin/ip as root, check if the device is in promiscuous mode, execute promisc.check.sh and log to /var/log/promisc.check.log. Make sure promisc.check.sh is anywhere in root's path (as root: echo $PATH) and is executable (chmod 0700 promisc.check.sh) by root. Run it for say 24 hours. Should be enough.

JakeX · 03-31-2006, 09:57 AM

Hmm unSpawn could I simply run the following if bg mode:

Code:

tail -f /var/log/messages | grep -qi promisc && /home/jakub/promisc.check.sh 2>&1>> /var/log/promisc.check.log &

I don't have the /sbin/ip package installed, and am wondering if maybe I could get away from it by tail'ing the messages log (since it gets logged when the interface goes into promisc). Then run the script. I honestly don't know whats happening on this box, as I've just noticed that 'loss of voicemail connectivity' is closely associated with the eth0 device going promisc. Maybe its a driver module issue? Since I've spoken to a couple people very proficient with linux os's & they tell me going into promisc mode shouldn't interrupt service for the asterisk. Hmm

I have dual 100 network adapters in this 1U box, the one in use (only one is plugged up currently, I should get the 2nd up) is:

Quote:

debian:/home/jakub# modinfo e100
filename: /lib/modules/2.6.6-1-686/kernel/drivers/net/e100.ko
description: Intel(R) PRO/100 Network Driver
author: Copyright(c) 1999-2004 Intel Corporation
license: GPL
vermagic: 2.6.6-1-686 preempt 686 gcc-3.3
depends: mii
alias: pci:v00008086d00001029sv*sd*bc02sc00i*
alias: pci:v00008086d00001030sv*sd*bc02sc00i*
alias: pci:v00008086d00001031sv*sd*bc02sc00i*
alias: pci:v00008086d00001032sv*sd*bc02sc00i*
alias: pci:v00008086d00001033sv*sd*bc02sc00i*
alias: pci:v00008086d00001034sv*sd*bc02sc00i*
alias: pci:v00008086d00001038sv*sd*bc02sc00i*
alias: pci:v00008086d00001039sv*sd*bc02sc00i*
alias: pci:v00008086d0000103Asv*sd*bc02sc00i*
alias: pci:v00008086d0000103Bsv*sd*bc02sc00i*
alias: pci:v00008086d0000103Csv*sd*bc02sc00i*
alias: pci:v00008086d0000103Dsv*sd*bc02sc00i*
alias: pci:v00008086d0000103Esv*sd*bc02sc00i*
alias: pci:v00008086d00001050sv*sd*bc02sc00i*
alias: pci:v00008086d00001051sv*sd*bc02sc00i*
alias: pci:v00008086d00001052sv*sd*bc02sc00i*
alias: pci:v00008086d00001053sv*sd*bc02sc00i*
alias: pci:v00008086d00001054sv*sd*bc02sc00i*
alias: pci:v00008086d00001055sv*sd*bc02sc00i*
alias: pci:v00008086d00001064sv*sd*bc02sc00i*
alias: pci:v00008086d00001065sv*sd*bc02sc00i*
alias: pci:v00008086d00001066sv*sd*bc02sc00i*
alias: pci:v00008086d00001067sv*sd*bc02sc00i*
alias: pci:v00008086d00001068sv*sd*bc02sc00i*
alias: pci:v00008086d00001069sv*sd*bc02sc00i*
alias: pci:v00008086d0000106Asv*sd*bc02sc00i*
alias: pci:v00008086d0000106Bsv*sd*bc02sc00i*
alias: pci:v00008086d00001059sv*sd*bc02sc00i*
alias: pci:v00008086d00001209sv*sd*bc02sc00i*
alias: pci:v00008086d00001229sv*sd*bc02sc00i*
alias: pci:v00008086d00002449sv*sd*bc02sc00i*
alias: pci:v00008086d00002459sv*sd*bc02sc00i*
alias: pci:v00008086d0000245Dsv*sd*bc02sc00i*
parm: debug:Debug level (0=none,...,16=all)

But thats just some additional info.. trying to be as helpfull as possible, appreciate all the advice, even if some is a lil over my head. Guess its just part of the learning curve and I do go out and research it when I don't get it. So it helps :)

unSpawn · 04-01-2006, 02:35 PM

tail -f /var/log/messages | grep -qi promisc && /home/jakub/promisc.check.sh 2>&1>> /var/log/promisc.check.log &
Uhm, no. This construction will only run once. Looking back at your log excerpt the interval for showing promiscuous mode isn't that big, so a temporary cronjob running it at a 5 min interval for say 2 hrs max should do it, right?

Could you also check the contents of any crontabs while you're at it and make a list of what apps use libpcap?:

Code:

find /bin /sbin /usr/bin /usr/sbin /usr/local/bin /usr/local/sbin /opt -type f | while read f; do 
[ -x "$f" ] && ldd "${f}" 2>/dev/null|grep -q libpcap && echo "${f}"; done

To give you an idea of apps that use libpcap here's some list: dsniff, ethereal, ettercap, firewalk, fragroute, nessusd, p0f, ppp, pppd, snort, tcpflow, tcpick, tcpkill. I've left out tools that don't apply like Wifi scanners.

Maybe its a driver module issue?
Could be. I've read there are ethernet device drivers that won't even show promiscuous mode when they're bound to more than one interface. In any case I haven't heard of modules setting promisc mode all by itself and for no apparent reason.

Since I've spoken to a couple people very proficient with linux os's & they tell me going into promisc mode shouldn't interrupt service for the asterisk.
Entering or exiting promisc mode should not interfere with the working of a working device.

JakeX · 04-03-2006, 02:31 PM

I ran the Code you provided last and it returned the following:

Quote:

/usr/bin/tethereal
/usr/bin/ethereal
/usr/sbin/pppd

I don't have the 'ip' package so I am not able to run the 'ip link show' crontab portion. I'm using debian? which package should I get for this?

Could it have something to do possibly with having this machine initially setup on an ADSL pppoe connection (when it was in office for testing on its own ADSL connection). It is now located in a coho with a 10mbit setup. I'm a little skeptical at running apt-get upgrade on it, in case I get some failure.

What worries me is that the device goes promisc and at that point if I run a tethereal trace on port 5060 I get no sip requests. But 10-15 min later it works perfectly.

Maybe when I run the crontab with the ip package that will shine some more light on this problem :|

edit: unSpawn, again I appreciate the time to help out

I added the iproute package & started the crontab. I'll post when I have something..

unSpawn · 04-04-2006, 10:40 AM

I ran the code you provided and it returned: /usr/bin/tethereal, /usr/bin/ethereal, /usr/sbin/pppd
All cool. Can't see what Pppd would have to do but it may behave similar to dhclient/hdcpcd?.. Nothing to worry here.

Could it have something to do possibly with having this machine initially setup on an ADSL pppoe connection (when it was in office for testing on its own ADSL connection).
Nice find. Question is is the ppoe still active somewhere? If you don't need it I'd remove it.

I added the iproute package & started the crontab. I'll post when I have something..
OK. I'll wait.

JakeX · 04-04-2006, 11:09 AM

Well I went ahead and removed the ppp packages:

Quote:

apt-get remove --purge ppp

which got rid of:

Quote:

Reading package lists... Done
Building dependency tree... Done
The following packages will be REMOVED:
ppp* pppconfig* pppoeconf*

I didn't have a ppp0 device initialized nor did I have any ppp initialization errors in the logs. I remember I didn't set ppp0 to start on startup, was always done manually, but maybe this will help with something.

Currently its been going on 4 days no 'promisc' mode in the logs. Hmm only time will tell..

JakeX · 04-06-2006, 01:41 PM

Hmmm well maybe something I've done has 'fixed' this lil issue, its going strong now since March 29th as the last day it showed 'promisc' in the logs.

I've removed ppp packages via apt-get. No outage issues with voicemail server. Maybe it was the ppp

Eh oh well I'll keep monitoring

JakeX · 04-11-2006, 01:40 PM

Hmm crap.. it happened again, after about 13 days of uptime without any 'going promisc' problems.
It went promisc again and interrupted the voicemail app (Asterisk)

Code:

Apr 11 14:28:57 debian kernel: device eth0 entered promiscuous mode
Apr 11 14:29:09 debian kernel: device eth0 left promiscuous mode

The cronjob didn't catch it in time (as you can see it was roughly 12 sec interval that time).
Hmm unSpawn.. any other ideas? I'm thinking some script to run till it goes promisc and then at that moment run the scan. Because the crontab that I ran (every 4 min) didn't catch the interval.

I've found that a reboot is the only way to get service back quick (at least to get Asterisk working again). Stopping the service and/or restarting has no effect.