View Single Post
Old 11-13-2011, 09:57 AM  
Registered: May 2001
Posts: 27,140
Blog Entries: 54

Rep: Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791
Originally Posted by Eotnak View Post
this is where I had a problem. I tried looking up what command could display current running processes relating to eth0 and couldn't find the command, then I thought about ports but the ports being opened are random at times and sequential at others.
'lsof -Pwlni' displays process and network data. 'netstat antupe [interval]' does the same but continuously. Problem is catching rare occasions takes time and if you don't know what to look for best parallelize and best do that using only available tools as to minimize disturbing the file system.

Since you run Fedora you should have access to the audit daemon. While aimed at keeping a check on all things MAC we can also use it to trace system calls. (Like Syscalltrack which worked well with Linux 2.4 kernels but not with 2.6 ones.) Why not use strace? Well, if you don't know what to look for then there's no process to attach to. Besides auditctl loads rules and look at thinks from a kernel perspective (just like Netfilter does) instead of relying on userland tools (there's a reason why binaries like netstat got replaced by old school rootkits). What audit will deliver is the time in epoch, the used system call, any arguments to the application including any address it connects with, the process name and process Id and owner.

First prep audit by cleaning out all rules and setting defaults, then load up some rules. Below can be saved and run as shell script:
# Clear out rules
auditctl -D
# Increase the buffers
auditctl -b 10240
# For me this keeps it from crashing under heavy load
auditctl -r 0
# Track setXid binaries:
find /tmp /var/tmp /bin /sbin /usr /opt /home -type f -perm -04000 -o -perm -02000 -printf "auditctl -a always,exit -F path=%p -F perm=x -F auid\!=4294967295 -k BIN_seXuid\n"|/bin/sh
# Track whatever common tools Apache may access:
HTTPUID=$(getent passwd apache|awk -F':' '{print $3}')
for BINARY in $(which perl ruby java php whoami GET wget curl elinks links lynx lwp-download lwp-mirror lwp-request lwp-rget 2>/dev/null); do 
auditctl -a always,exit -F path=${BINARY} -F perm=x -F auid=${HTTPUID} -k HTTPD_problem
# Track available shells:
( cat /etc/shells; rpm -q -g "System Environment/Shells" --qf="%{NAME}\n"|while read NAME; do rpm -ql $NAME; done )\
|sort -u|awk '/bin\// {print "auditctl -a always,exit -F path="$1" -F perm=x -F auid\\!=4294967295 -k SHell"}'|/bin/sh
# Track socket calls. Sure you could do '-a exit,always -S socketcall'.
awk '/^#define.SYS_/ {print "-a entry,always -F arch=b32 -S socketcall -F a0="$3" -k "$2}' \
/usr/src/kernels/$(uname -r)-$(uname -m)/include/linux/net.h | while read LINE; do auditctl $LINE; done
# Check rule loading:
auditctl -l
- To clear out the rules quickly run 'auditctl -D' or to restore original rules 'service auditd restart'.
- Note "-F auid\!=4294967295" is for IA-32 and not 64-bit systems.
- On 64-bit systems replace "-F arch=b32" with "-F arch=b64".
- On 64-bit systems socket calls are NOT multiplexed: check net.h or use 'ausyscall --dump' instead.
- Depending on system use (best perform other auditing before or after) audit may log a lot. If you run Rsyslogd then set num_logs=2, (iterations) max_log_file=10 (size in megs) and max_log_file_action=rotate in /etc/audit/auditd.conf (and restart) and see Centralizing the audit log for remote logging and do check if rsyslogd watches the right file descriptor on rotating audit.log.

* Once you've got sufficient logging you can use 'ausearch' to find the socketcall system call (nr 102 on IA-32) and narrow down to a start time in MM/DD/YYYY format ("today" also works), check out a suspicious process name, let's say "curl", and interpret data:
ausearch -sc 102 -ts yesterday -x curl -i

Netfilter allows you, with exceptions on SMP systems, to filter traffic by UID using the "owner" module. While it doesn't provide as much nfo as audit it helps you to quickly confirm which user is generating traffic and drop it. For example to filter for all initial egress traffic from all local users who do not have a shell like /sbin/nologin, /sbin/halt, /sbin/shutdown or /bin/false you should first clear all rules ('service iptables stop') and then load the rules like this (these are SMP safe):
service iptables stop
# Set INPUT policy
/sbin/iptables --policy INPUT DROP
egrep -v "n/(nologin|false|shut|halt)" /etc/passwd \
| awk -F':' '{print "-A OUTPUT -o eth0 -m state --state NEW -m owner --uid-owner " $3" -j LOG --log-prefix \"OUT_"$3" \""}' \
| /bin/bash
# Drop all egress traffic after logging:
/sbin/iptables -A OUTPUT -o eth0 -j DROP

Originally Posted by Eotnak View Post
I had 4 virtual hosts running on this machine, all port 80 traffic redirected by zone edit to port 5204. Also, the web sites were on a separate disk mounted as /wwwmnt. Log files were separated to their own files. The log files for these sites are empty but the (archive?) files are full. Would any of this affect the logwatch report?
Generally speaking more data means more clues.
Click here to see the post LQ members have rated as the most helpful post in this thread.