LinuxQuestions.org
Visit Jeremy's Blog.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices


Reply
  Search this Thread
Old 09-05-2015, 11:35 PM   #1
throwaway_questions
LQ Newbie
 
Registered: Sep 2015
Posts: 4

Rep: Reputation: Disabled
I Got Hacked. Lessons Learned (long)


I got hacked with a wordpress XSS a couple of days ago and didn't know it. My ISP notified me and my host was briefly, a node in a botnet. The thread is a "lessons learned" thread so maybe other can learn from my dumb mistakes.

I'm a sysadmin by trade and have been running a couple of different servers on the same host at home for maybe 10 years at this point without a second thought. I should know better but did not take the time to do it right.

If my shortcuts below don't make sense, then let me know I'll define them better.
1. Harden PHP.ini.
I'm a Debian user from way back and assumed the php.ini was tight. It's not particularly tight. Google "php.ini hardening" and the results are mostly the same information.

2.Set documentRoot to a readonly partion.
I did it by making a disk image and mounting it via loop.
Code:
dd if=/dev/zero of=disk.img bs=1k count=2048
nano /etc/modprobe.d/loop.conf option loop max_part=2
modprobe -r loop
modprobe loop -v
losetup /dev/loop0 /path/to/disk.img
fdisk /dev/loop0 (you should know the rest)
mount example: mount /dev/loop0p1 /mnt/disk.img -t ext4 -Onodev,nosuid,noexec,ro
There's more to do here as you need to script the disk image getting mounted on reboot.
Ok, the readonly partition is ready.
3. Repoint/reconfigure Apache to use the read only partition.

4. Check Files and Apache config
Files in documentRoot are never owned by the apache user.
Limit .htaccess. Hopefully forbid it in the Apache config.

If you need a writable directory, that needs to be separate and some apache configs are needed to tighten up the writing.
Code:
(Options None, SetHandler none),
no .htaccesss allowed (AllowOverride None)
PHP.ini write configs needs similar consideration

Giving Apache write access has to be thought about carefully and definitely separated from the document root.

MONITORING
You need to install something to check that files don't get written to your http service directory. Debian has a package called tiger that's tripwire and some other apps glued together. Tripwire is very nice and worth the time to learn. You need warnings emailed to you!
Clamscan!

NO EXCUSES
I needed to track wordpress releases and deploy them in a timely manner.

Last edited by throwaway_questions; 09-06-2015 at 09:56 AM.
 
Old 09-06-2015, 03:42 AM   #2
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
Quote:
Originally Posted by throwaway_questions View Post
I got hacked with a wordpress XSS a couple of days ago and didn't know it. My ISP notified me and my host was briefly, a node in a botnet. The thread is a "lessons learned" thread so maybe other can learn from my dumb mistakes. I'm a sysadmin by trade and have been running a couple of different servers on the same host at home for maybe 10 years at this point without a second thought. I should know better but did not take the time to do it right.
Thanks for posting frankly about the mistakes you made. Especially the "did not take the time to do it right" (plus the common "I don't have the knowledge to" / "I did not know I had to") is something a lot of servers suffer from in terms of hardening, being maintained, audited and kept up to date.


Quote:
Originally Posted by throwaway_questions View Post
1. Harden PHP.ini.
I'm sorry but that's not the first thing to address. For years Linux distributions have offered extensive user, admin and security documentation for any new and seasoned Linux user alike to become familiar with. Next to that SANS ISC, OWASP, Cisecurity and others have offered HOWTOs, benchmarks and advice to not only change peoples mindset about security but be practically able to audit security. Because security is a continuous process of auditing and adjusting. On top of that applications offer security guidelines themselves and WordPress is no different. Its Codex offers quite usable advice on hardening WordPress any WP user should read (regularly) and act on. Like the Codex points out you should start with server hardening as that's the foundation to enable applications on. If the platform itself doesn't do user and network ACLs, file system and traffic rate limiting / monitoring / scrubbing, auditing and reporting then that is something to correct first. (It probably should not hurt to ask specific questions in the LQ Linux Security forum should you need more guidance.)
 
1 members found this post helpful.
Old 09-06-2015, 04:18 AM   #3
ugjka
Member
 
Registered: May 2015
Location: Latvia
Distribution: Arch, Centos
Posts: 368
Blog Entries: 5

Rep: Reputation: 264Reputation: 264Reputation: 264
For those running Centos and alike

http://stopdisablingselinux.com/
 
Old 09-06-2015, 04:42 AM   #4
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
Quote:
Originally Posted by ugjka View Post
For those running Centos and alike
http://stopdisablingselinux.com/
Funny, correct, but unfortunately still only one aspect of what needs addressing IMNSHO.
 
Old 09-06-2015, 08:21 AM   #5
Habitual
LQ Veteran
 
Registered: Jan 2011
Location: Abingdon, VA
Distribution: Catalina
Posts: 9,374
Blog Entries: 37

Rep: Reputation: Disabled
Personally, I find it is a very humbling experience when it happens.
When it does, time to sharpen the tools in our utility-belts!
 
Old 09-06-2015, 09:45 AM   #6
throwaway_questions
LQ Newbie
 
Registered: Sep 2015
Posts: 4

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by unSpawn View Post
Funny, correct, but unfortunately still only one aspect of what needs addressing IMNSHO.
UnSpawn, if only it were so easy. Again, no excuses, but I assumed Debian was delivering a tight php.ini and I was wrong. It's not the Debian maintainer's fault either. As someone that admins selinux machines, that too can be poorly configured.

If you are relatively new to Apache configs, then that becomes it's own learning curve where most people stop at "I got it working."

I agree with you in practice and principal, but, the reality is so much less than that.

Last edited by throwaway_questions; 09-06-2015 at 09:53 AM.
 
Old 09-06-2015, 09:28 PM   #7
sundialsvcs
LQ Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 10,659
Blog Entries: 4

Rep: Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941
The "I got hacked" lesson that I learned was: "don't use Plesk." Don't use the "convenient" mechanisms provided by a VPS provider. Start with a bare-Linux, change its root-password within seconds, and then build it by hand.

I, too, got careless once, and the system was hacked by penetrating the Plesk management-software, which I had been lazy enough to use. Once.
 
1 members found this post helpful.
Old 09-06-2015, 10:24 PM   #8
astrogeek
Moderator
 
Registered: Oct 2008
Distribution: Slackware [64]-X.{0|1|2|37|-current} ::12<=X<=15, FreeBSD_12{.0|.1}
Posts: 6,264
Blog Entries: 24

Rep: Reputation: 4195Reputation: 4195Reputation: 4195Reputation: 4195Reputation: 4195Reputation: 4195Reputation: 4195Reputation: 4195Reputation: 4195Reputation: 4195Reputation: 4195
Quote:
Originally Posted by sundialsvcs View Post
...Don't use the "convenient" mechanisms provided by a VPS provider. Start with a bare-Linux, change its root-password within seconds, and then build it by hand.
Absolutely! Build it by hand, manage it by secure shell, write your own firewall rules - expose NOTHING that you do not know how to install, configure, monitor and control - or don't run on a VPS!
 
Old 09-09-2015, 06:02 PM   #9
Ryanms3030
Member
 
Registered: Jan 2014
Distribution: Debian, Mint, CentOS, Ubuntu
Posts: 261

Rep: Reputation: Disabled
My friend's wordpress site got hacked by ISIS or some other terrorists last year. I had a public facing server for a while and I admittedly don't know enough about security. My main security was running fail2ban on Debian. I didn't get hacked in the few months the server was on line (that I know of) but the logs certainly showed constant attempts to log into SSH like 10 times every second. I'm just starting to get very interested in learning about network security and every day I learn about some little thing that I should be doing on my home network that I didn't know about and I'm sure 99 of the people with internet and a router don't know. It's a crazy world on the web
 
Old 09-24-2015, 05:43 PM   #10
Rinndalir
Member
 
Registered: Sep 2015
Posts: 733

Rep: Reputation: Disabled
Thanks for sharing the details.

Couple of different things occurred to me after reading this. First was the realization that the plumber's house always has broken plumbing or the electrician's house always needs a rewiring. In this case the sysadmin's server always needs more attention. The plumber got done with a whole day of plumbing so why would they want to do more at home.

The other one that I thought of was that realization when you're watching a movie and the entire premise of the movie relies on one thing happening. Putting gas in the car or locking the screen door or making a copy of a document. That one little thing gets done and the movie's over.

Anyway what I am saying is that if you were hosting your stuff at a good ISP you wouldn't have to worry about any of that. I know, I know hosting your own is ideal for a lot of reasons. But the good ISPs live and breath protecting their network and you.
 
Old 09-25-2015, 02:30 AM   #11
dlugasx
Member
 
Registered: Dec 2008
Location: Germany/Poland
Distribution: CentOS / Debian / Solaris / RedHat
Posts: 266

Rep: Reputation: 19
I would never install Apache(crap) with an application like Wordpress. If You would like to properly protect You environment, think about Nginx and Naxsi module.


Quote:
Originally Posted by throwaway_questions View Post
I got hacked with a wordpress XSS a couple of days ago and didn't know it. My ISP notified me and my host was briefly, a node in a botnet. The thread is a "lessons learned" thread so maybe other can learn from my dumb mistakes.

I'm a sysadmin by trade and have been running a couple of different servers on the same host at home for maybe 10 years at this point without a second thought. I should know better but did not take the time to do it right.

If my shortcuts below don't make sense, then let me know I'll define them better.
1. Harden PHP.ini.
I'm a Debian user from way back and assumed the php.ini was tight. It's not particularly tight. Google "php.ini hardening" and the results are mostly the same information.

2.Set documentRoot to a readonly partion.
I did it by making a disk image and mounting it via loop.
Code:
dd if=/dev/zero of=disk.img bs=1k count=2048
nano /etc/modprobe.d/loop.conf option loop max_part=2
modprobe -r loop
modprobe loop -v
losetup /dev/loop0 /path/to/disk.img
fdisk /dev/loop0 (you should know the rest)
mount example: mount /dev/loop0p1 /mnt/disk.img -t ext4 -Onodev,nosuid,noexec,ro
There's more to do here as you need to script the disk image getting mounted on reboot.
Ok, the readonly partition is ready.
3. Repoint/reconfigure Apache to use the read only partition.

4. Check Files and Apache config
Files in documentRoot are never owned by the apache user.
Limit .htaccess. Hopefully forbid it in the Apache config.

If you need a writable directory, that needs to be separate and some apache configs are needed to tighten up the writing.
Code:
(Options None, SetHandler none),
no .htaccesss allowed (AllowOverride None)
PHP.ini write configs needs similar consideration

Giving Apache write access has to be thought about carefully and definitely separated from the document root.

MONITORING
You need to install something to check that files don't get written to your http service directory. Debian has a package called tiger that's tripwire and some other apps glued together. Tripwire is very nice and worth the time to learn. You need warnings emailed to you!
Clamscan!

NO EXCUSES
I needed to track wordpress releases and deploy them in a timely manner.
 
Old 09-25-2015, 03:21 AM   #12
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
Quote:
Originally Posted by dlugasx View Post
I would never install Apache(crap) (..)
Words like "crap" only convey an opinion, not technically sound reasons why. So instead please describe why and in detail. *And while I wasn't aware of the WAF called "Naxsi" (thanks for that) I doubt you'll be "properly protecting an environment" without requiring modification of web server config, headers, loaded modules, etc, etc...
 
Old 09-25-2015, 04:24 AM   #13
ericson007
Member
 
Registered: Sep 2004
Location: Japan
Distribution: CentOS 7.1
Posts: 735

Rep: Reputation: 154Reputation: 154
Well with a name like naxsi, does it keep out all intruders or just certain minorities. I am confident apache is not as crap as you state otherwise nginx would have taken the lead ages ago and would have been the default choice. Wonder why big comercial distros and smaller ones still rely on crap...
 
Old 09-25-2015, 04:37 AM   #14
dlugasx
Member
 
Registered: Dec 2008
Location: Germany/Poland
Distribution: CentOS / Debian / Solaris / RedHat
Posts: 266

Rep: Reputation: 19
Quote:
Originally Posted by unSpawn View Post
Words like "crap" only convey an opinion, not technically sound reasons why. So instead please describe why and in detail. *And while I wasn't aware of the WAF called "Naxsi" (thanks for that) I doubt you'll be "properly protecting an environment" without requiring modification of web server config, headers, loaded modules, etc, etc...
You have right, maybe I was too ... saying "crap". Anyhow this is my opinion about Apache and its architecture which compare to nginx is old and not efficient. A specially in the security layer. At the moment I dont see any reason to put Apache in front of the internet (Even with mod_security). Apache should stay behind something more secure and more flexible to manage incoming traffic. The best solution is in my opinion F5 or Nginx/Naxsi or some kind of caching server (varnish). Another thing is software... Wordpress with 1000 modules written with focus on the functionality and not security.

Ofcourse You need to modify Your system. I didnt say that You shouldnt do it. If You would like to build something secure I suggest to compile complete environment and customize it. First thing which I`m usually doing, I`m preparing Nginx. After compilation, server header, response, and complete HTTP communication looks like an Apache or some other server. Exploiting starting to be much harder if You dont know what You exactly attacking. Naxsi... basic naxsi rules block in my opinion 90% of the attacks. Its quite efficient system much more practical than mod_security where You really needs to know, what needs to be blocked.

Its quite long topic... To wordpress I suggest to use what wordpress.org using - nginx.


Good luck with hardening Apache .
 
Old 09-25-2015, 04:39 AM   #15
dlugasx
Member
 
Registered: Dec 2008
Location: Germany/Poland
Distribution: CentOS / Debian / Solaris / RedHat
Posts: 266

Rep: Reputation: 19
Quote:
Originally Posted by ericson007 View Post
Well with a name like naxsi, does it keep out all intruders or just certain minorities. I am confident apache is not as crap as you state otherwise nginx would have taken the lead ages ago and would have been the default choice. Wonder why big comercial distros and smaller ones still rely on crap...
Nginx has one big problem... doesnt support Java applications. That`s why is not so high.
But look on the PHP environment... here You cannot compare Apache with nginx .
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: Microsoft's Lessons Learned From Linux LXer Syndicated Linux News 0 03-29-2012 08:03 PM
LXer: Lessons Learned, Again LXer Syndicated Linux News 0 08-09-2008 02:10 PM
LXer: Lessons Learned? One Can Only Hope LXer Syndicated Linux News 0 03-09-2008 10:11 PM
LXer: Lessons learned from OSS in retail LXer Syndicated Linux News 0 11-04-2006 07:21 AM
LXer: Lessons learned from the NSF TeraGrid LXer Syndicated Linux News 0 04-28-2006 12:21 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Security

All times are GMT -5. The time now is 11:36 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration