LinuxQuestions.org
Did you know LQ has a Linux Hardware Compatibility List?
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Virtualization and Cloud
User Name
Password
Linux - Virtualization and Cloud This forum is for the discussion of all topics relating to Linux Virtualization and Linux Cloud platforms. Xen, KVM, OpenVZ, VirtualBox, VMware, Linux-VServer and all other Linux Virtualization platforms are welcome. OpenStack, CloudStack, ownCloud, Cloud Foundry, Eucalyptus, Nimbus, OpenNebula and all other Linux Cloud platforms are welcome. Note that questions relating solely to non-Linux OS's should be asked in the General forum.

Notices



Reply
 
Search this Thread
Old 02-26-2013, 08:18 PM   #16
mbvpixies78
Member
 
Registered: Nov 2007
Location: IL
Distribution: CentOS 6 & Fedora 17
Posts: 179
Blog Entries: 3

Original Poster
Rep: Reputation: 15

I've increased time between log rotation... that way if I'm busy or something comes on slow, I'll have plenty of data to study.
How do I send a copy of logs via email for safe handling elsewhere?

Quote:
Originally Posted by unSpawn View Post
*The benefit of running a capable SAR is that you will be able to determine the effect of some of the tuning you do. Don't change configuration, sysctls or turn other knobs because some (misinformed, outdated) web log says so: to measure is to know and IMHO that's the only reasonable approach.
I'd never heard of a SAR until you mentioned it, so I'm schooling myself on that-- what I should have done in the first place.


Quote:
I detect professional deformation here ;-p
I guess I never get tired of stating the obvious.

Quote:
Anyway, if you run a publicly accessible (web) server you're bound to find evidence of vulnerability scanners. Apart from the fact your web server will mostly send 4nn-type return codes back anyway, hardening the server and web stack should be a given (also see mod_security, Netfilter rate limiting, fail2ban).
I have fail2ban, but it doesn't notice the most common exploit attempts I see in my logs-- php and 'myphpadmin' access attempts, so maybe I need to tweak it or add something else.

mod_security I'd heard briefly about but need to learn to implement... sounds like I have plenty to do.

Netfilter rate limiting is something I know nothing about, but I'm assuming it's limiting traffic through-put via certain protocols and/or ports?

Would it be worthwhile to set up an IDS (snort)?

Last edited by mbvpixies78; 02-26-2013 at 08:24 PM.
 
Old 02-26-2013, 09:28 PM   #17
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,688
Blog Entries: 54

Rep: Reputation: 2955Reputation: 2955Reputation: 2955Reputation: 2955Reputation: 2955Reputation: 2955Reputation: 2955Reputation: 2955Reputation: 2955Reputation: 2955Reputation: 2955
Quote:
Originally Posted by mbvpixies78 View Post
How do I send a copy of logs via email for safe handling elsewhere?
Don't use email: let your stock syslog daemon handle that. See the basic example at http://www.rsyslog.com/doc/rsyslog_conf_examples.html and search the page for "remote".


Quote:
Originally Posted by mbvpixies78 View Post
I'd never heard of a SAR until you mentioned it, so I'm schooling myself on that-- what I should have done in the first place.
Don't kick yourself for it. Just check what these tools can do for you, deploy the one you need and be happy they're provided free of cost.


Quote:
Originally Posted by mbvpixies78 View Post
I guess I never get tired of stating the obvious.
What?


Quote:
Originally Posted by mbvpixies78 View Post
I have fail2ban, but it doesn't notice the most common exploit attempts I see in my logs-- php and 'myphpadmin' access attempts, so maybe I need to tweak it or add something else. mod_security I'd heard briefly about but need to learn to implement... sounds like I have plenty to do.
Either install mod_security, yes, and have fail2ban check its messages (http://www.fail2ban.org/wiki/index.php/Mod_Security) or enhance fail2bans http regexes (http://www.fail2ban.org/wiki/index.php/Apache).


Quote:
Originally Posted by mbvpixies78 View Post
Netfilter rate limiting is something I know nothing about, but I'm assuming it's limiting traffic through-put via certain protocols and/or ports?
Yes, the easiest example would be limiting ("-m limit --limit 30/s") the amount of new connections ("-m state --state NEW") your web server ("-p tcp --dport 80") should handle. The "fun" (not Spock's definition of fun) is you can combine several modules. For example you can limit connections and apply that rule to say any IP addresses in a /16 range, meaning limits won't "punish" or affect other ranges. The most important benefit of using the firewall is that you don't have to rely on any application level solutions: safer, more efficient.


Quote:
Originally Posted by mbvpixies78 View Post
Would it be worthwhile to set up an IDS (snort)?
That kind of depends. If your VM guest only allows TCP/80 traffic and only for mirroring then what it exposes will be very limited: no web-based management panels or interpreter-based applications remote hosts can muck with. This means its "attack surface" is limited to abuse of or flaws in the web server (configuration!) itself and the networking aspects of the Linux kernel. That in turn means the tuned rule set you deploy may be limited to filter for relevant risks like web server-centric flaws, any attempts to inject shell code and say known hostile addresses (all provided the guest is properly hardened first obviously). Try running Snort that way and then decide.
 
Old 03-02-2013, 08:52 PM   #18
mbvpixies78
Member
 
Registered: Nov 2007
Location: IL
Distribution: CentOS 6 & Fedora 17
Posts: 179
Blog Entries: 3

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by unSpawn View Post
Don't use email: let your stock syslog daemon handle that. See the basic example at http://www.rsyslog.com/doc/rsyslog_conf_examples.html and search the page for "remote".
Don't kick yourself for it. Just check what these tools can do for you, deploy the one you need and be happy they're provided free of cost.
Ok, I believe I've set up rsyslog correctly.. will find out soon enough.

Quote:
What?
Pointing out that there are likely issues with primary care-givers in such people is stating the obvious.

Quote:
Either install mod_security, yes, and have fail2ban check its messages (http://www.fail2ban.org/wiki/index.php/Mod_Security) or enhance fail2bans http regexes (http://www.fail2ban.org/wiki/index.php/Apache).
Before I do that I need to see that I've got monit configured properly. After install, every time the server boots I get email notifications (and log entries stating) httpd failed to start, then monit tries to start it 5 times and disables monit monitoring of httpd. I'm not sure what I set up wrong there, because httpd is running all the while, but monit thinks it's not.


Quote:
Yes, the easiest example would be limiting ("-m limit --limit 30/s") the amount of new connections ("-m state --state NEW") your web server ("-p tcp --dport 80") should handle. The "fun" (not Spock's definition of fun) is you can combine several modules. For example you can limit connections and apply that rule to say any IP addresses in a /16 range, meaning limits won't "punish" or affect other ranges. The most important benefit of using the firewall is that you don't have to rely on any application level solutions: safer, more efficient.
Ok, I have some things similar to this already in the mirror's firewall. I will probably post the firewall in a separate thread to see if anyone sees any potential problems in it since it's the first one I've put together-- and that with internet sources.


Quote:
That kind of depends. If your VM guest only allows TCP/80 traffic and only for mirroring then what it exposes will be very limited: no web-based management panels or interpreter-based applications remote hosts can muck with. This means its "attack surface" is limited to abuse of or flaws in the web server (configuration!) itself and the networking aspects of the Linux kernel. That in turn means the tuned rule set you deploy may be limited to filter for relevant risks like web server-centric flaws, any attempts to inject shell code and say known hostile addresses (all provided the guest is properly hardened first obviously). Try running Snort that way and then decide.
Ok, will do, and may look into setting up a honeypot (like honeypotd) to make that more worthwhile, at some point.

In the mean time I'm still waiting to see if it fails again-- hasn't yet since updating VirtualBox manually.

The ssh log (/var/log/secure) for the host looks like the following (logged every minute):
Quote:
Feb 25 16:35:58 [host-name] sshd[25184]: Connection closed by 127.0.0.1
Feb 25 16:36:58 [host-name] sshd[25195]: Connection closed by 127.0.0.1
Feb 25 16:37:58 [host-name] sshd[25205]: Connection closed by 127.0.0.1
Feb 25 16:38:58 [host-name] sshd[25213]: Connection closed by 127.0.0.1
What sort of problem might this indicate?
**EDIT:**
*ding* <-- the light-bulb went on. This indicates Monit querying sshd once every minute. My initial monit setup was wrong for httpd, but apparently it worked (a little too) well for sshd.

Last edited by mbvpixies78; 03-02-2013 at 11:23 PM.
 
Old 03-02-2013, 10:17 PM   #19
mbvpixies78
Member
 
Registered: Nov 2007
Location: IL
Distribution: CentOS 6 & Fedora 17
Posts: 179
Blog Entries: 3

Original Poster
Rep: Reputation: 15
I looked at my host's VirtualBox logs and also searched for information on the bug mentioned (re: ext 4, host i/o cache not enabled.

Excerpt of host's Virtualbox log:
Quote:
00:00:00.701071 OS type: 'RedHat'
00:00:00.710472 File system of '/home/admin/VirtualBox VMs/Apache Mirror 2/Snapshots' (snapshots) is ext4
00:00:00.710491 File system of '/home/admin/VirtualBox VMs/Apache Mirror 3/Apache Mirror.vdi' is ext4
00:00:00.720391 Console: VM runtime error: fatal=false, errorID=Ext4PartitionDetected message="The host I/O cache for at least one controller is disabled and the medium '/home/admin/VirtualBox VMs/Apache Mirror 3/Apache Mirror.vdi' for this VM is located on an ext4 partition. There is a known Linux kernel bug which can lead to the corruption of the virtual disk image under these conditions.
00:00:00.720398 Either enable the host I/O cache permanently in the VM settings or put the disk image and the snapshot folder onto a different file system.
I did as suggested and the warning disappeared. Also read the following on the Oracle forums (https://forums.virtualbox.org/viewto...?f=7&t=41683):
Quote:
Linux kernels older than 2.6.36 have a known bug which can lead to disk corruption if certain kernel API (direct I/O are used to access the file). So just enable the host cache for these controllers and you will not see this message again. Or upgrade your host kernel to 2.6.36.
My kernel is 2.6.32, however.

Last edited by mbvpixies78; 03-02-2013 at 10:21 PM.
 
Old 03-02-2013, 10:48 PM   #20
mbvpixies78
Member
 
Registered: Nov 2007
Location: IL
Distribution: CentOS 6 & Fedora 17
Posts: 179
Blog Entries: 3

Original Poster
Rep: Reputation: 15
I installed monit via yum (rpmforge), but that didn't work-- so I removed the yum package and reinstalled manually per http://blog.hostonnet.com/installing...-centos-server.

Apparently monit has a lot of ways it can monitor/restart:
Quote:
check process apache with pidfile /usr/local/apache/logs/httpd.pid
group apache
start program = "/etc/init.d/httpd start"
stop program = "/etc/init.d/httpd stop"
if failed host XXX port 80 protocol http
and request "/monit/token" then restart
if cpu is greater than 60% for 2 cycles then alert
if cpu 80% for 5 cycles then restart
if totalmem 500 MB for 5 cycles then restart
if children 250 then restart
if loadavg(5min) greater than 10 for 8 cycles then stop
if 3 restarts within 5 cycles then timeout

Last edited by mbvpixies78; 03-02-2013 at 11:53 PM.
 
Old 03-03-2013, 10:23 AM   #21
mbvpixies78
Member
 
Registered: Nov 2007
Location: IL
Distribution: CentOS 6 & Fedora 17
Posts: 179
Blog Entries: 3

Original Poster
Rep: Reputation: 15
This morning I found the vm in an aborted state again, so the VirtualBox update was not the problem. I set logging for the host and the vm to high a few days ago.

Here is an excerpt from the host's log sent out about 3 o'clock this morning:

Which other logs should I post?


Quote:
--------------------- iptables firewall Begin ------------------------


Listed by source hosts:
Dropped 488 packets on interface eth0
From 10.42.43.1 - 24 packets
To 255.255.255.255 - 24 packets
Service: bootpc (udp/68) (IP DROP SPOOF A: ) - 24 packets

[ ... ]

--------------------- Kernel Begin ------------------------


WARNING: Segmentation Faults in these executables
VBoxTestOGL : 1 Time(s)

1 Time(s): device eth0 entered promiscuous mode
2 Time(s): device eth0 left promiscuous mode
2 Time(s): r8169 0000:02:00.0: eth0: link down
2 Time(s): r8169 0000:02:00.0: eth0: link up
1 Time(s): vboxnetflt: dropped 0 out of 338569 packets
1 Time(s): vboxnetflt: dropped 0 out of 6471597 packets

Last edited by mbvpixies78; 03-03-2013 at 10:24 AM.
 
Old 03-19-2013, 08:34 PM   #22
mbvpixies78
Member
 
Registered: Nov 2007
Location: IL
Distribution: CentOS 6 & Fedora 17
Posts: 179
Blog Entries: 3

Original Poster
Rep: Reputation: 15
Please let me know if I should instead start a new thread on this...

My VM will no longer boot ('kernel panic'), and this suggests a bad (virtual?) hard drive. I cloned the VM a while back and so was able to simply boot the clone in place of the mucked up original.

The warning I used to get when booting VirtualBox about risk of corruption-- I'm wondering if that's the cause of this, or perhaps just a coincidence. Either way, I enabled host I/O caching as suggested and was unable to find any further information about the bug or any other clues in the logs, other than a recommendation to update the kernel.

My apache mirror's kernel is currently 2.6.32 and the recommendation to fix the VirtualBox bug issue is to update to at least 2.6.36. My concern, having never done a manual kernel update on a live server before, is what will this do in terms of causing other problems? Sure it's just a volunteer gig and I can always go back to a clone if I mess this one up, but I'd rather not have to spend a ton of free (literal and figurative) time fixing other issues that crop up because of a kernel update that's not an official part of RHEL/CentOS.

The same concerns arise with going from CentOS 6.3 to 6.4. My question is-- assuming this might fix any problems pertaining to virtual disk corruption and/or a VirtualBox bug and/or any future kernel panics-- should I upgrade first to 6.4 and then the kernel, or vice versa, or does it not matter at all how I go about this in terms of prevention of headaches? Is there a smart way to upgrade the kernel that will mean less likelihood that anything else breaks in the process of updates not officially sanctioned by Red Hat/CentOS community?

(If the kernel update does not fix the VirtualBox bug, what then? My understanding is that KVM is still pretty unreliable for my purposes. Should I then switch to a free version of VMWare?)

Thanks much in advance.

Last edited by mbvpixies78; 03-19-2013 at 08:39 PM.
 
Old 09-05-2013, 08:10 PM   #23
mbvpixies78
Member
 
Registered: Nov 2007
Location: IL
Distribution: CentOS 6 & Fedora 17
Posts: 179
Blog Entries: 3

Original Poster
Rep: Reputation: 15
Update:

It was a bad hard drive, I suspect. Will confirm with a little more time observing the server's behavior.

Basically, the stability got worse and worse, even getting to a point where the host OS was no longer accessible remotely, then I saw the following (rough notes follow) when trying to reinstall fresh OS (figured it was worth a try):

/tmp/program.log:
ext2fs_mkdir: Attempt to read block from filesystem resulted in short read while creating root dir

SCSI Mod ePage Offset:
response length too short
resp_len=47
offset=50
bd_len=46

>> terminate command early due to bad response to IEC mode page
log sense failed, IE page [scsi response fails sanity test]




So with that, I replaced the hard drive (luckily had a 2-day-old backup of my vm's on an old external hdd,) and since then it's been pretty smooth sailing.

Still trying to get monit to work properly though... on vm, it freaks out every time the vm boots, saying httpd isn't running and refuses to start, even though httpd is running fine. Need to fine-tune some things, clearly.

Learning a lot while studying for LPIC-1! Any suggestions for how to change careers and get into IT would be greatly appreciated! Hoping to obtain an MSCIS, but not guaranteed this option for some time. Certs? (CCNA, LPIC and RHCSA?)
 
Old 09-07-2013, 10:14 AM   #24
mbvpixies78
Member
 
Registered: Nov 2007
Location: IL
Distribution: CentOS 6 & Fedora 17
Posts: 179
Blog Entries: 3

Original Poster
Rep: Reputation: 15
No issues whatsoever since replacement of hard drive.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
In the Virtualbox forums, how do I change email address? Advice Pro General 2 07-20-2012 10:27 AM
Can Apache send error messages (crit, warn & such) by email each time they get logged yogi799 Linux - Software 2 11-24-2009 04:08 PM
LXer: Microsoft Takes a Dump to the European Commission LXer Syndicated Linux News 0 01-27-2006 12:01 PM
Webpages/Email takes ages to load after period of inactivity Robstro Linux - Networking 1 05-26-2004 02:31 AM
How to forcely dump the history of user commands to the admin dump file. mcp_achindra Linux - Security 1 03-19-2004 01:04 PM


All times are GMT -5. The time now is 11:45 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration