Hi All,

I have a worrying situation that my rkhunter has started to pickup lately :

Warning: Account 'wale1' is root equivalent (UID = 0)

There is a user callled wale that has somehow been added to the passwd file and given himself root privileges. A last command on that user shows that he was logged onto the system for over 7 hours last night.

wale1 pts/1 normanje.co.za Mon Feb 9 21:28 - 04:52 (07:23)

Any advice on how to start investigating how this user has been able to gain root privileges would be very much appreciated.

Regards,

Mike

The user is not "gaining" root access - that user IS root - UID=0!

From what you said, this is not the first access, and if they were in ther for 7 hours overnight - even if that was the first time, which seems doubtful - then you must assume they have thoroughly corrupted the machine!

You must really just take it offline - immediately - and consider it dead. Begin an autopsy afterward, but change the drive or reformat and reinstall from scratch before you return it to use.

If you need help with the autopsy you will need to provide a good bit more info as well, distro, version, what it is used for, is it behind a router, on a LAN, firewalled, ssh configuration, etc... etc...

Good luck!

And welcome to LQ!

1 members found this post helpful.

Hi AstroGeek,

Thanks for the speedy reply and the welcome as well as the stern message. I am a week in, in a new position and am getting familiar with this company's systems. My background is not security so I've been thrown in the deep end so to speak. This particular machine is a web server hosting multiple sites and being managed by something called ISPCONFIG (http://www.ispconfig.org/page/home.html) so taking it offline in a hurry is going to be difficult and I will have to relay as much info up the chain as possible. Below is a grep for this user in the secure logs :

Feb 8 06:27:00 web01-c14-dc1 sshd[21652]: pam_unix(sshd:session): session closed for user wale
Feb 8 19:53:14 web01-c14-dc1 sshd[23631]: Accepted password for wale from 58.97.146.130 port 25430 ssh2
Feb 8 19:53:14 web01-c14-dc1 sshd[23631]: pam_unix(sshd:session): session opened for user wale by (uid=0)
Feb 8 20:15:11 web01-c14-dc1 sshd[23631]: pam_unix(sshd:session): session closed for user wale
Feb 9 13:20:42 web01-c14-dc1 su: pam_unix(su:session): session opened for user wale by mikeh(uid=0)
Feb 9 13:21:23 web01-c14-dc1 su: pam_unix(su:session): session closed for user wale
Feb 9 21:22:30 web01-c14-dc1 sshd[10278]: Invalid user wale from 31.200.240.33
Feb 9 21:22:30 web01-c14-dc1 sshd[10279]: input_userauth_request: invalid user wale
Feb 9 21:22:33 web01-c14-dc1 sshd[10278]: pam_succeed_if(sshd:auth): error retrieving information about user wale
Feb 9 21:22:35 web01-c14-dc1 sshd[10278]: Failed password for invalid user wale from 31.200.240.33 port 45174 ssh2
Feb 9 21:22:39 web01-c14-dc1 sshd[10278]: pam_succeed_if(sshd:auth): error retrieving information about user wale
Feb 9 21:22:41 web01-c14-dc1 sshd[10278]: Failed password for invalid user wale from 31.200.240.33 port 45174 ssh2
Feb 9 21:23:14 web01-c14-dc1 useradd[10435]: failed adding user 'wale', data deleted
Feb 9 21:23:46 web01-c14-dc1 useradd[10532]: new user: name=wale, UID=0, GID=0, home=/home/wale, shell=/bin/bash
Feb 9 21:23:56 web01-c14-dc1 passwd: pam_unix(passwd:chauthtok): password changed for wale
Feb 9 21:24:39 web01-c14-dc1 sshd[10671]: pam_access(sshd:account): access denied for user `wale' from `virt2030.unelink.net'
Feb 9 21:24:39 web01-c14-dc1 sshd[10671]: Failed password for wale from 31.200.240.33 port 45504 ssh2
Feb 9 21:24:39 web01-c14-dc1 sshd[10679]: fatal: Access denied for user wale by PAM account configuration
Feb 9 21:25:13 web01-c14-dc1 sshd[10836]: pam_access(sshd:account): access denied for user `wale' from `normanje.co.za'
Feb 9 21:25:13 web01-c14-dc1 sshd[10837]: fatal: Access denied for user wale by PAM account configuration
Feb 9 21:25:13 web01-c14-dc1 sshd[10836]: Failed password for wale from 41.78.6.163 port 56408 ssh2
Feb 9 21:27:23 web01-c14-dc1 sshd[11250]: pam_access(sshd:account): access denied for user `wale' from `normanje.co.za'
Feb 9 21:27:23 web01-c14-dc1 sshd[11252]: fatal: Access denied for user wale by PAM account configuration
Feb 9 21:27:23 web01-c14-dc1 sshd[11250]: Failed password for wale from 41.78.6.163 port 56433 ssh2
Feb 9 21:27:35 web01-c14-dc1 useradd[11354]: new group: name=walex, GID=650
Feb 9 21:27:35 web01-c14-dc1 useradd[11354]: new user: name=walex, UID=650, GID=650, home=/home/walex, shell=/bin/bash
Feb 9 21:27:53 web01-c14-dc1 useradd[11433]: new group: name=wale1, GID=651
Feb 9 21:27:53 web01-c14-dc1 useradd[11433]: new user: name=wale1, UID=0, GID=651, home=/home/wale1, shell=/bin/bash
Feb 9 21:28:04 web01-c14-dc1 passwd: pam_unix(passwd:chauthtok): password changed for wale1
Feb 9 21:28:19 web01-c14-dc1 sshd[11480]: Accepted password for wale1 from 41.78.6.163 port 56442 ssh2
Feb 9 21:28:19 web01-c14-dc1 sshd[11480]: pam_unix(sshd:session): session opened for user wale1 by (uid=0)
Feb 10 04:52:03 web01-c14-dc1 sshd[11480]: pam_unix(sshd:session): session closed for user wale1
Feb 10 06:39:29 web01-c14-dc1 userdel[19873]: delete user 'walex'
Feb 10 06:39:29 web01-c14-dc1 userdel[19873]: removed group 'walex' owned by 'walex'

The stuff in bold, I'm assuming is him trying to access as a user that was removed and then suddenly just adding another user, walex, with root privileges. The box still seems to be functioning as expected but I realize that shouldn't make me complacent. I will take your advice about autopsy.

Anything else you can add?

Sorry to be so late getting back in here.

I am not a PAM user so maybe someone else can comment on that aspect of things.

But looking at the timestamps and sequence of events it appears to me that creation and subsequent deletion of new group and user, and setting of password were not done by a human - too fast I think. So there is something running on that box that either automatically or on demand does that for the intruder.

It is also possible/likely if this is on a local network with other machines, some or all of them are also compromised. So I would say if this is a business environment you need to take a hard fast survey of the whole environment.

And however painful it may be, the ONLY way to end access via that machine will be to discinnect it from the network.

As noticed all bets are off already as this user "wale" already has root access, so what s/he does with it is of no consequence (the bold part) as apparently this machine allows access to SSH users with uid=0, which is NOT a SSH Best Practice.

So focus on mitigating risks first which means isolating this machine from the network:
- The best way would be to simply drop traffic at the switch or edge router or bring the Ethernet interfaces on the machine down.
- Easiest way would be to configure another (known secure!!!) machine on the network to use the IP address of this server and just put up an inert "under construction" sign and nothing more (meaning not exposing any other services, no web-based control panels, no external SSH access and the latter definitely not as root!).
- The most basic way would be to just stop all but the rudimentary services (I mean SSH access) and raise the firewall to deny any traffic from outside your network.

When operating in a commerical environment there's four things (at least) to consider as you mitigate things:
- Check if you have company procedures in place outlining what to do in case of a breach of security. This dictates the steps (compliance-wise if any) you will have to take.
- Inventory what kind of web sites or applications paying clients have running, your SLAs and what leeway your contracts allow you (litigation-wise ;-p) as this dictates how you'll have to manage damage control, communication, migration etc, etc.
- Ensure you have complete, recent backups (and set them all aside marked as "tainted" until proven otherwise).
- Immediately inventory all adjacent or related systems. If the breach is isolated that's bad but if it's spread across your network(s) or affects systems elsewhere you have to act even faster informing others.
Note the above should not take hours or even days. To work efficiently you should work as a team if possible: one person in charge keeping an overview and directing investigations, one person handling comms and one or more persons performing the investigations.

Note where I say "what s/he does with it is of no consequence" that only means you have to widen your focus and not stare blindly at one particular /var/log/secure excerpt. In these days of less-than-stellar-maintained and easy-to-breach systems and with the constant barrage of data exfiltration and Id theft news a security incident is bad but how fast and how well you respond will tell customers if they can (still) trust you regardless.


As for the post-mortem, as there's no known point of origin, you'll have to mark all systems as "untrusted" until proven "clean" by having investigated them properly: argueing something is safe just because (insert seemingly valid reason here) is not proper conduct in the face of a breach of security.
- Ensure you have one workstation that is known safe.
- Push / pull in the /var/log directories and start making detailed reports of log file entries with Logwatch.
- On each system:
-- Note the OS, release and update details (if any),
-- Note if it has it been hardened properly and if it has it been updated regularly.
-- Run a system integrity check with whatever means its package management allows you.
-- Save listings of running processes and open files: 'lsof -Pwln;'
-- When running RHEL or derivatives also check package integrity ('rpm -qVva 2>&1|grep -v "^\.\{8\}";')
-- Check for files in /boot, /etc, /home, /tmp and other common directories that are not part of any package, are owned by root or a daemon or unprivileged user, have a setuid or setgid bit set or look otherwise out of place by name or modification time.
-- Note which services the machine provides,
** and add anything to what you report here if it seems out of place, related or unrelated.

Please stay with this thread until completion, please respond as quickly and as detailed as possible, please ask questions if unsure and most of all: good luck!

1 members found this post helpful.

Firstly, I would like to thank everyone that has replied so far and I appreciate the stern tone.

Just to repeat, I am a week in, in a this new position coming from a much larger and securer environment where I focused on continuous deployment and automation. My security knowledge is minimal but that is no excuse. So I'm in the deep end, getting to grips with a new environment, procedures and personalities.

I will take everything being said here with the utmost seriousness.

Thanks again.

If you have a security office - call them in ASAP.

They will likely have the authority to take the system offline immediately - or they will isolate it for the FBI. Doing anything else can/will contaminate any evidence they will need IF they determine to investigate.

Also record (as keep a notebook) of everything you do, with time stamps. And I mean EVERYTHING (phone calls, meetings, discussions...) and any thing done to the system, and who directed it.

1 members found this post helpful.

Hi All,

Thanks for everyone's contribution. Please see below report.

Nature of breach

Over the last couple of days, a user has been added on "victim".xxxx.xxxx with UID 0 or root user privileges. The user names have varied from wale, walex, wale1 and feofeo. Records of this user being added can be found in the /var/log/secure log file :

(SECURE LOG FILE FEBRUARY 10th)

Feb 10 23:18:49 "victim" useradd[23604]: new group: name=feofeo, GID=652
Feb 10 23:18:49 "victim" useradd[23604]: new user: name=feofeo, UID=0, GID=652, home=/home/feofeo, shell=/bin/bash
Feb 10 23:18:59 "victim" passwd: pam_unix(passwd:chauthtok): password changed for feofeo
Feb 10 23:19:14 "victim" sshd[23684]: Accepted password for feofeo from 31.200.240.33 port 45055 ssh2
Feb 10 23:19:14 "victim" sshd[23684]: pam_unix(sshd:session): session opened for user feofeo by (uid=0)
Feb 11 05:47:29 "victim" sshd[23684]: pam_unix(sshd:session): session closed for user feofeo

(RKHUNER REPORT 11th FEBRUARY 04:34am)

Warning: Account 'wale1' is root equivalent (UID = 0)
Warning: Account 'feofeo' is root equivalent (UID = 0)

Although no major disruptions in service have been experienced, it is disconcerting that an unauthorised user has privileges with the potential to incur massive disruptions. It is suspected that this breach is a remote code execution carried out on demand or automatically by the intruder via a vulnerable service on the breached machine.

Investigations

Initially it was suspected that the ISPconfig was the vulnerable service. ISPconfig is running version 3.0.5.3 and should be on the latest version 3.0.5.04p5. There is a documented breach http://www.exploit-db.com/exploits/34241/ but after further investigations it was clarified that this breach is only susceptible to a correctly authenticated admin user on ISPconfig. See below correspondence from ISPconfig forum :

The method in that post describes that a correctly authenticated server administrator (aka root user) is able to create a new shell user from within a webbased controlpanel where he can choose the name on its own or not. You can do that in other panels like webmin as well, the root / admin user in webmin can create others users and so could the ispconfig admin. But as some people thought that the ability to create shell users with administrative privileges within a webbased controlpanel is an exploit, we limited that function in july so that you cant do that in ispconfig anymore. Second: If you installed any ispconfig update since end of july, you could not have added that root user while logged in as administrator as the function has been removed.

The admin username on ISPconfig has been changed and it is recommended that an updated version of ISPconfig is installed on both master and slave machines.

Upon further investigations it was found that a security alert involving the glibc library was announced on 27th of January 2015 and had in fact been vulnerable since it’s first release on November 10th 2000 and was only fixed May 21 2013 with the release of glibc-2.17 and glibc-2.18. The risk of this vulnerability : “There is a remote code execution risk due to this vulnerability. An attacker who exploits this issue can gain complete control of the compromised system.”

Affected systems are running versions of glibc older than 2.12-1.149.el6_6.5

http://www.cyberciti.biz/faq/cve-201...os-rhel-linux/

https://www.youtube.com/watch?v=zHRRLsZtWAA


The affected machine, "victim".xxxxx.xxxx, has the following versions of glibc :

glibc-common-2.12-1.132.el6_5.2.x86_64
glibc-2.12-1.132.el6_5.2.x86_64
glibc-devel-2.12-1.132.el6_5.2.x86_64
glibc-headers-2.12-1.132.el6_5.2.x86_64

This makes it vulnerable to this exploit. All other web machines are up to date with the recommended versions of glibc. The ISPconfig machine is also on the older version and it is recommended that it too be updated.

Recommended Actions

• Patch all affected machines with the recommended update patches and reboot.
• Update ISPconfig to the latest version
• Upgrade "victim".xxxx.xxxx to CentOS 6.6 as it is currently on 6.2
• Upgrade ispconfig1.xxxxx.xxxx to CentOS 6.6 as it is currently on 6.2

Immediate Actions

Patch "victim".xxxx.xxxx with the recommended version of glibc and reboot.

This has been my crash course introduction to security issues 101. Hopefully this issue is mitigated now. I will post any further updates if anything else arises. Thanks again for everyone's contribution.

//BTW: please note I removed your hosts designation to avoid exposure and whatever may follow from that.


No. While outages are resons for concern the fact that you have not experienced service disruptions is a blessing but no more than that. More importantly shrugging repeated and unauthorised root access off as just "disconcerting" is plain wrong.


While as investigator you're free to entertain any working hypotheses you would like I must emphasize your investigation should back up your statement with evindence, meaning: network device and local log file entries, file system entities, login records, event time line. If this is a guess then this is not good enough.


Changing the ISPconfig admin username is not good enough unless you know (evidence) this is the incursion vector. It should be shut down or at least be isolated, be made available solely over HTTPS and restricting access with a white list (or better: listen on localhost only and require SSH tunnel and pubkey access).

The glibc "GHOST" vuln may be bad but 0) it's about deprecated functionality, 1) there's no known public exploits (as far as I know), 2) it would require an additional exploit to say gain root.

While it is good to notice that your other web machines are up to date with respect to glibc this does not mean they should not be investigated thoroughly:
- In the period 2010-09 to 2015-01 clean-mx.de registered 7 phishing and 3 malware activities, the majority of which occurring in 2014-12.
- Google Safe Browsing tested 55 sites within your AS Number in the past 3 months, finding 2 sites that are serving malicious software and 1 site that appears to function as an intermediary for the infection of 1 site elsewhere.
- CentOS 6.2 was release June 24th 2012 and ISPConfig 3.0.5.3 was released August 9th 2013. I therefore assert there's been no (or a criminally negligable amount of) maintenance done, meaning the company failed to focus properly on system performance, integrity and security.

So let the following three lines sink in:

You had a breach of security involving repeated, unauthorized root access.
In such a situation slipshod approaches like "patching" or "fixing" are wrong.
Please fix things the right way

OK. I know. World of Hurt. But let's face it, if you don't do things properly now there is no way in Hell you will later.


Ergo:
- Stop unauthorized access (white list and firewall and enforce SSH pubkeys),
- Investigate (more, further, with a wider scope) if the source of the breach was not found,
- Stop or isolate exposed services,
- Commission new machines that are first brought up to date and hardened before being exposed,
- Inform users,
- Migrate users enforcing they have the latest versions of about everything,
- Review and enforce proper procedures for systems maintenance, incident handling et cetera.


Ask questions if you need to (BTW you're invited to contact me by email if you would like me to go over evidence together) but please, please make certain this root compromise gets treated with priority and in the only proper way.

Thanks Uspawn for another stern reply. Yes, I have to agree - patching is not the solution to this issue. I am having to deal with resistance from the chain of command to get this machine offline and autopsied. I will keep on it. Thanks for your offer of email correspondence.

I'm sorry, it's not that I try to be difficult about these things, it's more that I care and try to be as efficient and clear as possible. After all you can use all the help you can get dealing with root compromises...


Understandable. Not the technical side of things but I'm accustomed to dealing with this to. So if they have difficult questions, if you need ammo have trouble presenting your arguments or anything else just ask / discuss here if you feel like it.