LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices


Reply
  Search this Thread
Old 10-24-2011, 08:54 PM   #1
Eotnak
LQ Newbie
 
Registered: Oct 2011
Posts: 6

Rep: Reputation: Disabled
Apache server trying outgoing connections to unknown IPs on ports 80 and 53


OK, so I've been learning my way through Fedora trying to progress to LFS and FreeBSD. I have a Fedora 14 machine running Apache 2.2.17, and about 2 days ago, I came across the server and saw a black screen blazing through text so fast I couldn't read it. I didn't know if it was a crash or I'd been compromised, so I pulled the ethernet cable and hit the reset button. Upon restarting I was greeted with:

Inodes that were part of a corrupted orphan linked list found

/dev/mapper/vg_192-lv_root: UNEXPECTED INCONSISTENCY; RUN fsck manually (i.e., without -a or -p options)

I ran fsck and selected y for about 15 to 30 error fixes. I would have written them down but I'd been planning on rebuilding this box. Now, however, I'd like to know the cause of this problem, so I've been googling access log messages, syslog messages, etc. Just now I stuck in some external firewall (router) rules that only allow traffic between the apache box and 3 IP addresses that I use (home and work) I found the following (edited) in the firewall log:

Code:
Oct 24 19:58:07 2011 TCP 192.168.1.xx:60664->204.141.87.16:80 on ixp0 [repeated 6 times, last time on Oct 24 19:59:40 2011] 
Oct 24 19:54:57 2011 TCP 192.168.1.xx:47455->204.141.87.11:80 on ixp0 [repeated 6 times, last time on Oct 24 19:56:31 2011] 
Oct 24 19:54:06 2011 TCP 192.168.1.xx:47454->204.141.87.11:80 on ixp0 [repeated 5 times, last time on Oct 24 19:54:52 2011] 
Oct 24 19:50:57 2011 TCP 192.168.1.xx:60661->204.141.87.16:80 on ixp0 [repeated 6 times, last time on Oct 24 19:52:30 2011] 
Oct 24 19:48:57 2011 TCP 192.168.1.xx:60660->204.141.87.16:80 on ixp0 [repeated 6 times, last time on Oct 24 19:50:30 2011] 
Oct 24 19:47:57 2011 TCP 192.168.1.xx:60659->204.141.87.16:80 on ixp0 [repeated 5 times, last time on Oct 24 19:48:42 2011] 
Oct 24 19:47:27 2011 TCP 192.168.1.xx:60658->204.141.87.16:80 on ixp0 [repeated 4 times, last time on Oct 24 19:47:48 2011] 
Oct 24 19:45:47 2011 TCP 192.168.1.xx:60657->204.141.87.16:80 on ixp0 [repeated 6 times, last time on Oct 24 19:47:20 2011]
they were blocked and logged. It's a US company...don't know why my server is attempting to contact these IPs?

some other info. Server had been up and running with phpbb 3.0.9 (no registered users other than myself) for about 4 months. I looked over the inactive users list and filtered all of the IP blocks of those users (Russia and Ukraine) in the firewall about 2 nights before this happened. Running vsftp 2.3.4 with 2 users, one user's home directory root of one of the virual hosts, and the other was /home/user. Both could log in locally and move up outside their home directories. I was in the middle of figuring out how to lock down the one user (vhost home dir) and was going to remove FTP access for the other but I forgot to.

I know I was littered with security holes, and plan on addressing them before the new one goes online. I still have this one running as described, but don't plan on unleashing it. I would like to find out if it has been broken into or not before I start over. Anyone have any idea where to check, or why it's trying to connect to those 2 ip addresses?

thank you for your time and for sharing your priceless knowledge

Some more firewall security log:

Code:
Oct 24 21:28:55 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:52906->128.63.2.53:53 on ixp0
Oct 24 21:28:56 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:50508->192.228.79.201:53 on ixp0
Oct 24 21:28:56 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:29322->192.203.230.10:53 on ixp0
Oct 24 21:28:56 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:44569->192.203.230.10:53 on ixp0
Oct 24 21:28:56 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:5220->192.228.79.201:53 on ixp0
Oct 24 21:28:56 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:14340->128.8.10.90:53 on ixp0
Oct 24 21:28:56 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:44225->192.112.36.4:53 on ixp0
Oct 24 21:28:56 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:63708->192.36.148.17:53 on ixp0
Oct 24 21:28:57 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:37912->192.203.230.10:53 on ixp0
Oct 24 21:28:57 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:25084->192.112.36.4:53 on ixp0
Oct 24 21:28:57 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:8359->192.112.36.4:53 on ixp0
Oct 24 21:28:57 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:32749->192.112.36.4:53 on ixp0
Oct 24 21:28:57 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:5118->192.203.230.10:53 on ixp0
Oct 24 21:28:57 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:39603->199.7.83.42:53 on ixp0
Oct 24 21:28:58 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:18159->202.12.27.33:53 on ixp0
Oct 24 21:28:58 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:14484->128.8.10.90:53 on ixp0
Oct 24 21:28:58 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:2248->193.0.14.129:53 on ixp0
Oct 24 21:28:58 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:64654->193.0.14.129:53 on ixp0
Oct 24 21:28:58 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:51058->193.0.14.129:53 on ixp0
Oct 24 21:28:58 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:8535->193.0.14.129:53 on ixp0
Oct 24 21:28:58 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:60558->193.0.14.129:53 on ixp0
Oct 24 21:28:58 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:38959->193.0.14.129:53 on ixp0
Oct 24 21:28:59 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:28654->192.36.148.17:53 on ixp0
Oct 24 21:28:59 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:54992->192.5.5.241:53 on ixp0
Oct 24 21:28:59 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:62956->198.41.0.4:53 on ixp0
Oct 24 21:28:59 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:41303->128.63.2.53:53 on ixp0
Oct 24 21:28:59 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:31276->192.5.5.241:53 on ixp0
Oct 24 21:28:59 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:14432->199.7.83.42:53 on ixp0
Oct 24 21:28:59 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:3762->192.5.5.241:53 on ixp0
Oct 24 21:29:00 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:63866->128.63.2.53:53 on ixp0
Oct 24 21:29:00 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:1180->192.33.4.12:53 on ixp0
Oct 24 21:29:01 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:25832->128.63.2.53:53 on ixp0
Oct 24 21:29:01 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:62628->199.7.83.42:53 on ixp0
Oct 24 21:29:01 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:34998->192.36.148.17:53 on ixp0
Oct 24 21:29:01 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:53695->198.41.0.4:53 on ixp0
Oct 24 21:29:01 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:49673->192.58.128.30:53 on ixp0
Oct 24 21:29:01 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:8749->202.12.27.33:53 on ixp0
Oct 24 21:29:01 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:28551->128.8.10.90:53 on ixp0
Oct 24 21:29:02 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:21114->192.203.230.10:53 on ixp0
Oct 24 21:29:02 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:29879->193.0.14.129:53 on ixp0
Oct 24 21:29:02 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:4627->192.5.5.241:53 on ixp0
Oct 24 21:29:02 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:31509->192.58.128.30:53 on ixp0
Oct 24 21:29:03 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:49220->202.12.27.33:53 on ixp0
Oct 24 21:29:03 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:3657->192.33.4.12:53 on ixp0
Oct 24 21:29:03 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:44276->192.58.128.30:53 on ixp0
Oct 24 21:29:03 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:14984->192.5.5.241:53 on ixp0
Oct 24 21:29:03 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:16620->192.228.79.201:53 on ixp0
Oct 24 21:29:04 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:43240->192.58.128.30:53 on ixp0
Oct 24 21:29:04 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:48719->202.12.27.33:53 on ixp0
Oct 24 21:29:04 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:52690->198.41.0.4:53 on ixp0
Oct 24 21:29:04 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:7414->128.8.10.90:53 on ixp0
Oct 24 21:29:04 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:16525->192.228.79.201:53 on ixp0
Oct 24 21:29:04 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:4261->199.7.83.42:53 on ixp0
Oct 24 21:29:05 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:27184->192.203.230.10:53 on ixp0
Oct 24 21:29:05 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:44511->192.112.36.4:53 on ixp0
Oct 24 21:29:06 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:4327->199.7.83.42:53 on ixp0
Oct 24 21:29:08 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:16148->193.0.14.129:53 on ixp0
Oct 24 21:29:09 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:40769->128.63.2.53:53 on ixp0
Oct 24 21:29:11 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:38067->202.12.27.33:53 on ixp0
Oct 24 21:29:12 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:64440->192.5.5.241:53 on ixp0
Oct 24 21:29:14 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:52151->192.58.128.30:53 on ixp0
Oct 24 21:29:15 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:6116->198.41.0.4:53 on ixp0
Oct 24 21:29:17 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:9749->193.0.14.129:53 on ixp0
Oct 24 21:29:19 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:12706->192.58.128.30:53 on ixp0
Oct 24 21:29:21 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:65001->192.5.5.241:53 on ixp0
Oct 24 21:29:23 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:30482->128.63.2.53:53 on ixp0
Oct 24 21:29:25 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:40425->202.12.27.33:53 on ixp0
Oct 24 21:29:27 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:14780->198.41.0.4:53 on ixp0
Oct 24 21:29:58 2011 Outbound Traffic Blocked - Advanced Filter Rule TCP 192.168.1.xx:47461->204.141.87.11:80 on ixp0

Last edited by unSpawn; 10-25-2011 at 03:26 AM. Reason: //Add BB code blocks
 
Click here to see the post LQ members have rated as the most helpful post in this thread.
Old 10-25-2011, 03:57 AM   #2
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594
Quote:
Originally Posted by Eotnak View Post
(..) Now, however, I'd like to know the cause of this problem, so I've been googling access log messages, syslog messages, etc.
Did you let Logwatch check all your logs? IMHO that's the easiest way to weed out crud and find leads. And if you are researching specific patterns are there any you should share apart from the ones below?


Quote:
Originally Posted by Eotnak View Post
Just now I stuck in some external firewall (router) rules that only allow traffic between the apache box and 3 IP addresses that I use (home and work) I found the following (edited) in the firewall log: (..) they were blocked and logged. It's a US company...don't know why my server is attempting to contact these IPs?
Can't say w/o combing over logs (if any). But I can say that's why hitting the reset button is bad. It makes you lose volatile data (users, processes, network connections) that could have been helpful.


Quote:
Originally Posted by Eotnak View Post
I know I was littered with security holes, and plan on addressing them before the new one goes online. I still have this one running as described, but don't plan on unleashing it. I would like to find out if it has been broken into or not before I start over. Anyone have any idea where to check, or why it's trying to connect to those 2 ip addresses?
Well, you may intend to do so now but let's emphasize proper host hardening needs to be done before exposing a machine to hostile 'nets. Fedora, SANS, CIS and OWASP provide enough documentation and testing to ensure even a user-OS like Fedora (why not Centos instead?) is relatively well-hardened. As for post-incident auditing I'd start with log auditing using Logwatch, verifying package contents and the CERT Intruder Detection Checklist. These should provide you with leads you can post about here.

Note that you having had to fsck the machine lead to changes in the file system so if there's any file recovery to do best keep the file system(s) as much read-only as possible. If using a Live CD like Helix 2008R1 (MD5 hash 93a285bfa8ab93d664d508e5b12446d3) doesn't allow you to access your LVM then at least use a Fedora-based (installer?) CD that does.
 
2 members found this post helpful.
Old 10-25-2011, 09:09 AM   #3
Eotnak
LQ Newbie
 
Registered: Oct 2011
Posts: 6

Original Poster
Rep: Reputation: Disabled
thank you very much for the info! CentOS: I've glanced over it a number of times, but never gave any consideration because I'd been using Fedora since Core 4. I'm entirely self taught by reading a few books, and hundreds of tutorials and posts from great sites like this, and there's no one I know that I can talk *nix with. However, I have looked into CentOS after reading your post and it obviously makes more sense for my purposes on this machine.

I will run logwatch and post the output tonight. I will trim it down if I get a chance, otherwise I might print out the whole thing? As well as any oddities I find verifying packages and going through the CERT checklist. (sidenote: my wife just went back to work and I have a 4 month old daughter to take care of)

I do have very minimal file recovery to do, however for the most part, this was just like a "lets see what happens if I go online" machine. I got carried away when I actually did get it to work...forgot that it was just a test. Your advice on hardening is well noted, and I will start with a CentOS build, and read as much security info as I can get my hands on BEFORE going online this time. Which is fine because I need to flash my router anyway, so I was planning on a chance of downtime.

Again thank you so much for the info, unSpawn. Right now I feel like a house builder who was given a nail gun to replace his hammer...for FREE!
 
Old 10-25-2011, 02:03 PM   #4
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594
Quote:
Originally Posted by Eotnak View Post
thank you very much for the info! CentOS: I've glanced over it a number of times, but never gave any consideration because I'd been using Fedora since Core 4. I'm entirely self taught by reading a few books, and hundreds of tutorials and posts from great sites like this, and there's no one I know that I can talk *nix with. However, I have looked into CentOS after reading your post and it obviously makes more sense for my purposes on this machine.
Most of us here are self-taught. Some of us work or have worked in IT or related fields. What we share is a passion for all things Linux and trying to pass on what we've learned. So: welcome.


Quote:
Originally Posted by Eotnak View Post
I will run logwatch and post the output tonight. I will trim it down if I get a chance, otherwise I might print out the whole thing? As well as any oddities I find verifying packages and going through the CERT checklist.
Sure. Post as much as you think you need to. Usually the more the better. If you check a range like 'logwatch.pl --numeric --detail 5 --service all --range All --archives --print', see if you can attach the report as plain text file if it's too big to post in a reply. If it's still too big and you can't split it ('man split') shoot me an email and we'll discuss handing it off.


Quote:
Originally Posted by Eotnak View Post
Your advice on hardening is well noted, and I will start with a CentOS build, and read as much security info as I can get my hands on BEFORE going online this time.
We've handled quite a lot of incidents in this forum over the years and in a way more detailed way than I've seen in any other general Linux forum. So searching this forums threads you'll find good hardening nfo (sometimes encapsulated in a whole story like this one) or see the LQ FAQ: Security references.
After reading some basic docs feel free to create a separate thread about hardening if you think you'd benefit from it.

Last edited by unSpawn; 10-25-2011 at 02:07 PM. Reason: //Quoting and stuff
 
Old 10-25-2011, 04:07 PM   #5
Noway2
Senior Member
 
Registered: Jul 2007
Distribution: Gentoo
Posts: 2,125

Rep: Reputation: 781Reputation: 781Reputation: 781Reputation: 781Reputation: 781Reputation: 781Reputation: 781
I agree with unSpawn, start with logwatch to see what that dredges up. On the surface, the entries you are referring to look like web page and DNS queries, things that would appear quite normal. If you go to the web site, however, you get a strange, but simple page that says that URL "/" is not valid with a reference number. The IP does not have a reverse loopup, but is part of NTT and I've never heard of them, but that doesn't mean much other than one, this is not a valid web site that you would be going to, and two this looks like a person's machine. With the second case, this may be a form of phone home. I would look very carefully for any indication of what process, or user may be or have been responsible for these connections as they seem to be designed to look normal to someone analyzing web browsing logs, which I doubt they are. unSpawn also referenced the CERT check list. Be sure to pay close attention to the part about hidden files, especially in a location like /tmp.
 
Old 10-25-2011, 09:52 PM   #6
Eotnak
LQ Newbie
 
Registered: Oct 2011
Posts: 6

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by unSpawn View Post
If you check a range like 'logwatch.pl --numeric --detail 5 --service all --range All --archives --print', see if you can attach the report as plain text file if it's too big to post in a reply.
I tried
Quote:
logwatch --numeric --detail 5 --service all --range All --archives --print
and thought that it hung up the machine. After about 25 minutes, I canceled and tried to output to a file
Quote:
logwatch --numeric --detail 5 --service all --range All --archives --save logwatchoutput
and walked away. I figure it took over an hour, but it finished with a 150 MB executable text file. Is this file useful, or do I have to copy/paste screen output via --print? (currently processing again)
 
Old 10-25-2011, 11:58 PM   #7
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594
Quote:
Originally Posted by Eotnak View Post
it finished with a 150 MB executable text file. Is this file useful, or do I have to copy/paste screen output via --print?
The report might somewhere show a section where repeated logged lines have small variations that don't impact security like only the change of a PID. If that's the case, cut out the second to second last line. If unsure see what the file compresses to with bzip2, after all it's plain text so the compression rate should be good. Note LQ's attachment size is max 256.0 KB.
 
Old 10-26-2011, 01:17 PM   #8
Eotnak
LQ Newbie
 
Registered: Oct 2011
Posts: 6

Original Poster
Rep: Reputation: Disabled
tried to attach as logwatchoutput_edited.gz but the board says no on that extension, so I renamed to logwatchoutput_edited.gz.txt.

again this is an executable text file, so I hope that's not an issue.

removed 2347436 lines from file which brought it down around 450k then gzipped it. The lines I removed were disk errors similar to a bunch that I left in the file. There is a line that I wrote where they were removed. I'll get to copy/paste/split into a plain text file in case this isn't sufficient.
Attached Files
File Type: txt logwatchoutput_edited.gz.txt (44.3 KB, 10 views)
 
Old 10-26-2011, 03:27 PM   #9
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594
Thanks for the log. I saw you ran "--service all --range All" but have logs actually been rotated? I'd expect that "--archives" would result in way much more nfo wrt Selinux but especially httpd and right now there's about zilch clues...
 
Old 10-26-2011, 04:09 PM   #10
Noway2
Senior Member
 
Registered: Jul 2007
Distribution: Gentoo
Posts: 2,125

Rep: Reputation: 781Reputation: 781Reputation: 781Reputation: 781Reputation: 781Reputation: 781Reputation: 781
I concur that the log file doesn't show much. Are these attempts till occurring and can you see the process tree and file path for the PID that is trying to establish these connections?
 
Old 10-28-2011, 08:39 AM   #11
Eotnak
LQ Newbie
 
Registered: Oct 2011
Posts: 6

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by unSpawn View Post
Thanks for the log. I saw you ran "--service all --range All" but have logs actually been rotated? I'd expect that "--archives" would result in way much more nfo wrt Selinux but especially httpd and right now there's about zilch clues...
I typed "logwatch --numeric --detail 5 --service all --range All --archives --save logwatchoutput" and then edited and zipped the file. So should I have used "--archives" in place of "--service all --range All" ? Am I using the switches improperly?

Quote:
Originally Posted by Noway2 View Post
I concur that the log file doesn't show much. Are these attempts till occurring and can you see the process tree and file path for the PID that is trying to establish these connections?
Yes, the attempts are still occurring when I plug in the ethernet cable, I don't know what happens if unplugged because the log I was viewing was on the external firewall. I won't be able to check on the PIDs until Monday I think. So stay tuned I guess, I definitely want to see something here, pretty fun playing detective.
 
Old 10-30-2011, 09:38 AM   #12
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594
Quote:
Originally Posted by Eotnak View Post
Am I using the switches improperly?
No, you ran 'logwatch --numeric --detail 5 --service all --range All --archives --save logwatchoutput' which was good. I was just hoping you had more, I mean 4 months worth of logging should definitely result in more, especially when it comes to publicly accessible services like HTTP, FTP and SSH.
 
Old 11-12-2011, 08:58 AM   #13
Eotnak
LQ Newbie
 
Registered: Oct 2011
Posts: 6

Original Poster
Rep: Reputation: Disabled
thanks again to unSpawn and Noway2

Quote:
Originally Posted by Noway2
I concur that the log file doesn't show much. Are these attempts till occurring and can you see the process tree and file path for the PID that is trying to establish these connections?
this is where I had a problem. I tried looking up what command could display current running processes relating to eth0 and couldn't find the command, then I thought about ports but the ports being opened are random at times and sequential at others. Here's where I need a little noob-nudge...can you give me a hint on what search terms I should use?

Quote:
Originally Posted by unSpawn
No, you ran 'logwatch --numeric --detail 5 --service all --range All --archives --save logwatchoutput' which was good. I was just hoping you had more, I mean 4 months worth of logging should definitely result in more, especially when it comes to publicly accessible services like HTTP, FTP and SSH.
I had 4 virtual hosts running on this machine, all port 80 traffic redirected by zone edit to port 5204. Also, the web sites were on a separate disk mounted as /wwwmnt. Log files were seperated to their own files. The log files for these sites are empty but the (archive?) files are full. Would any of this affect the logwatch report?

I will start going through the CERT Intruder Detection Checklist and report...
 
Old 11-13-2011, 09:57 AM   #14
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594
Quote:
Originally Posted by Eotnak View Post
this is where I had a problem. I tried looking up what command could display current running processes relating to eth0 and couldn't find the command, then I thought about ports but the ports being opened are random at times and sequential at others.
'lsof -Pwlni' displays process and network data. 'netstat antupe [interval]' does the same but continuously. Problem is catching rare occasions takes time and if you don't know what to look for best parallelize and best do that using only available tools as to minimize disturbing the file system.

Since you run Fedora you should have access to the audit daemon. While aimed at keeping a check on all things MAC we can also use it to trace system calls. (Like Syscalltrack which worked well with Linux 2.4 kernels but not with 2.6 ones.) Why not use strace? Well, if you don't know what to look for then there's no process to attach to. Besides auditctl loads rules and look at thinks from a kernel perspective (just like Netfilter does) instead of relying on userland tools (there's a reason why binaries like netstat got replaced by old school rootkits). What audit will deliver is the time in epoch, the used system call, any arguments to the application including any address it connects with, the process name and process Id and owner.

First prep audit by cleaning out all rules and setting defaults, then load up some rules. Below can be saved and run as shell script:
Code:
# Clear out rules
auditctl -D
# Increase the buffers
auditctl -b 10240
# For me this keeps it from crashing under heavy load
auditctl -r 0
# Track setXid binaries:
find /tmp /var/tmp /bin /sbin /usr /opt /home -type f -perm -04000 -o -perm -02000 -printf "auditctl -a always,exit -F path=%p -F perm=x -F auid\!=4294967295 -k BIN_seXuid\n"|/bin/sh
# Track whatever common tools Apache may access:
HTTPUID=$(getent passwd apache|awk -F':' '{print $3}')
for BINARY in $(which perl ruby java php whoami GET wget curl elinks links lynx lwp-download lwp-mirror lwp-request lwp-rget 2>/dev/null); do 
auditctl -a always,exit -F path=${BINARY} -F perm=x -F auid=${HTTPUID} -k HTTPD_problem
done
# Track available shells:
( cat /etc/shells; rpm -q -g "System Environment/Shells" --qf="%{NAME}\n"|while read NAME; do rpm -ql $NAME; done )\
|sort -u|awk '/bin\// {print "auditctl -a always,exit -F path="$1" -F perm=x -F auid\\!=4294967295 -k SHell"}'|/bin/sh
# Track socket calls. Sure you could do '-a exit,always -S socketcall'.
awk '/^#define.SYS_/ {print "-a entry,always -F arch=b32 -S socketcall -F a0="$3" -k "$2}' \
/usr/src/kernels/$(uname -r)-$(uname -m)/include/linux/net.h | while read LINE; do auditctl $LINE; done
# Check rule loading:
auditctl -l
- To clear out the rules quickly run 'auditctl -D' or to restore original rules 'service auditd restart'.
- Note "-F auid\!=4294967295" is for IA-32 and not 64-bit systems.
- On 64-bit systems replace "-F arch=b32" with "-F arch=b64".
- On 64-bit systems socket calls are NOT multiplexed: check net.h or use 'ausyscall --dump' instead.
- Depending on system use (best perform other auditing before or after) audit may log a lot. If you run Rsyslogd then set num_logs=2, (iterations) max_log_file=10 (size in megs) and max_log_file_action=rotate in /etc/audit/auditd.conf (and restart) and see Centralizing the audit log for remote logging and do check if rsyslogd watches the right file descriptor on rotating audit.log.

* Once you've got sufficient logging you can use 'ausearch' to find the socketcall system call (nr 102 on IA-32) and narrow down to a start time in MM/DD/YYYY format ("today" also works), check out a suspicious process name, let's say "curl", and interpret data:
Code:
ausearch -sc 102 -ts yesterday -x curl -i

Netfilter allows you, with exceptions on SMP systems, to filter traffic by UID using the "owner" module. While it doesn't provide as much nfo as audit it helps you to quickly confirm which user is generating traffic and drop it. For example to filter for all initial egress traffic from all local users who do not have a shell like /sbin/nologin, /sbin/halt, /sbin/shutdown or /bin/false you should first clear all rules ('service iptables stop') and then load the rules like this (these are SMP safe):
Code:
service iptables stop
# Set INPUT policy
/sbin/iptables --policy INPUT DROP
egrep -v "n/(nologin|false|shut|halt)" /etc/passwd \
| awk -F':' '{print "-A OUTPUT -o eth0 -m state --state NEW -m owner --uid-owner " $3" -j LOG --log-prefix \"OUT_"$3" \""}' \
| /bin/bash
# Drop all egress traffic after logging:
/sbin/iptables -A OUTPUT -o eth0 -j DROP


Quote:
Originally Posted by Eotnak View Post
I had 4 virtual hosts running on this machine, all port 80 traffic redirected by zone edit to port 5204. Also, the web sites were on a separate disk mounted as /wwwmnt. Log files were separated to their own files. The log files for these sites are empty but the (archive?) files are full. Would any of this affect the logwatch report?
Generally speaking more data means more clues.
 
Old 11-13-2011, 11:31 AM   #15
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594
Quote:
Originally Posted by Eotnak View Post
I will start going through the CERT Intruder Detection Checklist and report...
Some notes regarding that:

* Running a Live CD like KNOPPIX or Helix 2008R1 (MD5 hash 93a285bfa8ab93d664d508e5b12446d3) or a Linux installation disk in rescue mode and using only those binaries helps ensure malware that hides data via means of kernel, libraries or binaries stays dead.
1. Best copy logs off of the system and process them (on a virtual machine) on a known safe workstation in a different location. Quickest way IMHO is to grab all of /var.
2. On Linux you can combine it and run 'find / -perm -4000 -perm -2000'. A more elaborate example like 'find / -type f -a \( -perm -04000 -o -perm -02000 \) -printf "%T@ %A@ %C@ %U %G %m %l %s %y %Z \"%p\"\n" > /dev/shm/setxid.log 2>&1;' also logs ownership, permissions and MAC times.
3. If Samhain or Aide (or even tripwire) was in use run it, else with RPM you can run 'rpm -Vva|grep -v '^\.\{8\}'. For both goes: if unsure use a backup of the database.
4. Sniffers means any application that 'grep -qm1 libpcap /path/tobinary' (or like libpcap) but also a trojaned sshd. The latter should stand out due to failing 'rpm -V' or finding an OpenSSH tarball or source files (often compiled locally) or finding collateral strings or files like /usr/bin/ssh2d, /ssh2d, backdoor.h, backdoor_active, magic_pass_active, /usr/include/gpm2.h, /dev/saux, /usr/bin/take/pid, /usr/sbin/sshd3, /usr/lib/dmis/dmisd, /lib/secure/libhij.so, /usr/sbin/sshd3, /tmp/pass_ssh.log, /usr/include/gpm2.h, /etc/ssh/.sshd_auth, /usr/lib/.sshd.h, /var/run/.defunct, /etc/httpd/run/.defunct, /usr/lib/libutil1.2.1.2.so, /bin/ceva, /sbin/syslogd%, /usr/include/shup.h, /etc/rpm/sshdOLD, /etc/rpm/sshOLD, /usr/share/passwd.h, /usr/include/netda.h. Also 'rpm -q --dump' may provide a list of known files, meaning everything else is up for inspection. Any files that are part of an application that was downloaded as tarball (forum software, web log, etc, etc) can be checked against the contents of the original archive (speedup: 'man sha1deep').
5. Apart from what's found in /var/spool a crontab may be loaded if the user is not denied access (/etc/cron.deny). So if the cracker can control say Apache to load a crontab then it'll be done with something like 'crontab -u httpd /writable_dir/file' which attempts may turn up in log analysis.
6. For /etc/inetd.conf substitute /etc/xinet* .
7 and on: for this you could run Rootkit Hunter as long as the CDROM comes with version RKH 1.3.8..
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
VPS Setting up multiple outgoing ips Smasher03 Linux - Server 2 06-11-2011 10:25 AM
How to rotate outgoing ips on squid testingpepe Linux - Server 8 08-25-2010 01:01 AM
Getting things straight: Apache, SSL, Multiple External IPs / Internal IPs robin.com.au Linux - Server 21 10-13-2007 11:39 PM
Using mail client with unknown outgoing server allelopath Linux - Networking 1 06-05-2006 06:02 PM
My Ubuntu 5.10 server don't want outgoing connections salah1 Linux - Networking 0 05-14-2006 04:58 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Security

All times are GMT -5. The time now is 04:05 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration