Linux - SecurityThis forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
1.) If used by others please notify other users the system was compromised and that they should change credentials (SSH keys, pass phrases, passwords) used to access that system and systems they access from the compromised machine.
2.) Run command #1 (see bottom of post) and then mitigate damage by disabling unnecessary services like webserver, any webservers on other ports included in web-based management panels, database, FTP, etc, etc.
*If you don't have Out of Band access (DRAC, IPMI, other type of remote console) you will need to keep SSH access.
**If unsure at least stop unnecessary services and limit access (using the firewall or where applicable) /etc/hosts.allow) to your (management) IP address or IP range.
3.) Only after stabilising prepare backups. Mark this and any previous backups as unfit for reuse, they'll only serve as reference.
* Any talk of repartitioning, reformatting and re-installing is off limits during the investigation. We'll get to that.
4.) Now lets build an understanding of the situation. Questions:
- What is the location (home, office, colocation, could, other) of the machine,
- does audit data (/var/log/audit, /var/log/secure or equivalent, AIDE, Samhian or even tripwire logs) exist?,
- does auth data (per service log files, /var/log/secure, /var/log/btmp*, /var/log/wtmp*) exist and does it go back to before command #0 (see bottom of post),
- do system and daemon logs exist and also go back to before a perceived date of compromise,
- which services the machine or machines provide (including web-based management panels, statistics, web log, forum, shopping cart, plugins, themes, addons and other software if any),
- was the software was kept up to date,
- have there been earlier breaches or anomalies,
- have you found setuid root files in /tmp or any other directory the ftp user, web server user or unprivileged users can write to,
- have you seen any "odd" commands in user shell histories,
- also if users connect via SSH using passwords or FTP do check, or have them virus scan, their systems.
5.) Post back results from the actions performed as per the CERT Intruder Detection Checklist. Then get your system and daemon logs and audit data off of the compromised server and onto a workstation you can trust. Run those log files through Logwatch (see command #2).
6.) Please compress the resulting logs from the commands I asked you to run. Contact me to discuss dropping them off.
*I know, that's a lot to answer but the more detailed you respond the better we start investigating.
Please ask specific questions before performing if necessary and please reply verbosely.
Please stay with the thread (subscribe?) until completion and reply as soon as possible when replies are posted.
Code:
# 0.: command to show modification dates. Select the earliest one. Your system logs should go back to before that date.
stat /usr/sbin/sshd /usr/bin/w /usr/bin/who|grep ^Mod|sort -k2'
# 1.: commands to fetch volatile data before killing processes and closing off services:
( /bin/ps axfwwwe -opid,ppid,gid,uid,cmd 2>&1; /usr/sbin/lsof -Pwln 2>&1; /bin/ls -al /var/spool/cron 2>&1; /bin/netstat -anTpe 2>&1; /usr/bin/lastlog 2>&1; /usr/bin/last -wai 2>&1; /usr/bin/who -a 2>&1; ) | tee /tmp/log0.txt
# 2.: Logwatch usage, using the "--detail High --service All --range All --archives --numeric" args.
# (With perl-Date-Manip installed a range can also be expressed like "--range 'between 2012/11/26 and 2012/12/01'": see --range Help).
# If you can't find Logwatch prefix its whole path, should be something like "usr/share/logwatch/scripts/".
logwatch.pl --numeric --detail High --service All --range All --archives --print 2>&1 | tee /tmp/log1.txt
# 3.: Find MAC times:
find / -printf "%T@ %A@ %C@ %u %g %m %y \"%p\"\n" 2>&1 | tee /tmp/log2.txt
So, after some investigation, I found some strange things. An unknown line in root crontab:
Code:
* * * * * /.../lib/update >/dev/null 2>&1
This /.../lib/ contains scripts and othoer files of an IRC proxy called "miau".
The Change time of sshd and who is the same day as this /.../ directory was created.
Only /usr/sbin/sshd and /aquota.user has the immutable flag in the filesystem. As far as I can tell, there are no suspicious root setuid files:
/var/log/secure is empty, /var/log/{u,b,w}tmp exist and has up-to-date modification times.
I also found a key in root's authorized_keys which was not familiar to any users who have access to the server. But that could be a key which we used when we moved to this server from our previous one this year.
The auth.log for the day Jun 13 has unfortunately been purged already. The earliest day of auth.log.4.gz is Jun 22.
# strings -an1 /.../lib/update
#!/bin/sh
if test -r /.../lib/pid; then
pid=$(cat /.../lib/pid)
if $(kill -CHLD $pid >/dev/null 2>&1)
then
exit 0
fi
fi
cd /.../lib
./r &>/dev/null
Code:
# file /aquota.user; strings -an1 /aquota.user
/aquota.user: data
:
:
\p
'
!
d
`
e
"
&
h
|
!
i
W
j
k
`
'
pQ
'
m
'
'
Z
n
v
@
'
:
o
%
P
'
0
p
'
l'
Y
p
'
+=
P~
'
1
50
p
'
@
'
0
'
x
`
'
ZJ
'
GO
t
'
Ms
'
1
D
'
'
b
!'
;
"'
-
O
#'
0
$'
Q
`
%'
&'
r
0:
And here is the "CRON" process, which does the dirty thing. This one is running since the date of the modifications in the system:
I tried to find this CRON with capitals with find, but no match. It seems the ps output was manipulated with this "psf -- Process Stack Faker" which is also in the /.../lib/ directory. I've sent you the content of the whole dir by mail.
- This host is a webserver. It runs unknown software of which we don't know if it was kept up to date. It does run Plesk.
- 383 files have MD5 hash mismatches, among which files you'd expect to change like Plesk php.ini but als files that should not like /usr/sbin/sshd.
- Additionally there's at least one setuid root file I don't recognize as such: /usr/include/arpa/suid.
- A potentially foreign SSH pubkey in root's authorized_keys (root shouldn't be accessed over the 'net anyway).
- Directories and files were created on July 13th.
- /var/log/secure is empty but there's no 'stat' data given.
- System logs like auth.log are insufficiently long rotated to allow for investigation.
As not all questions were answered this leaves insufficient clues to answer critical questions, and I'll cut this short by suggesting you change all credentials, mark backups as tainted, set up a new host, properly harden it, do not load data that wasn't inspected thoroughly and start anew.
- This host is a webserver. It runs unknown software of which we don't know if it was kept up to date. It does run Plesk.
- 383 files have MD5 hash mismatches, among which files you'd expect to change like Plesk php.ini but als files that should not like /usr/sbin/sshd.
- Additionally there's at least one setuid root file I don't recognize as such: /usr/include/arpa/suid.
- A potentially foreign SSH pubkey in root's authorized_keys (root shouldn't be accessed over the 'net anyway).
- Directories and files were created on July 13th.
- /var/log/secure is empty but there's no 'stat' data given.
- System logs like auth.log are insufficiently long rotated to allow for investigation.
As not all questions were answered this leaves insufficient clues to answer critical questions, and I'll cut this short by suggesting you change all credentials, mark backups as tainted, set up a new host, properly harden it, do not load data that wasn't inspected thoroughly and start anew.
I could also get log0.txt (netstat gave error beacause of an unknown parameter "T") and log2.txt. (However these outputs have some sensitive data too, imho of my currently vulnerable host.) Logwatch was not installed, I did it now, but it does not know "--print" parameter, and without it, it gives no output, only running and does not give output. (Even it's default output is stdout.)
I could also get log0.txt (netstat gave error beacause of an unknown parameter "T")
Then you could have asked right there and then. In the "old" netstat I use as reference "-T" stands for "do not trim long lines".
Quote:
Originally Posted by agocs
and log2.txt. (However these outputs have some sensitive data too, imho of my currently vulnerable host.)
All I can say is any information you share is treated carefully and you alone decide to obfuscate details and share data (or not).
Quote:
Originally Posted by agocs
Logwatch was not installed, I did it now, but it does not know "--print" parameter, and without it, it gives no output, only running and does not give output. (Even it's default output is stdout.)
With all due respect I did not say "install Logwatch", I said: "Then get your system and daemon logs and audit data off of the compromised server and onto a workstation you can trust. Run those log files through Logwatch (see command #2)."
Apart from that please go over the questions again. You may find you haven't answered all and you may find you haven't replied as verbose as possible. For example you said "this is a web server" while I asked "which services the machine or machines provide (including web-based management panels, statistics, web log, forum, shopping cart, plugins, themes, addons and other software if any), was the software was kept up to date, have there been earlier breaches or anomalies,".
Also please indicate if you're working on setting up a new machine and if the victimized one is properly isolated by now.
The problem is, that I am not administering this server officially, I was only trying to help a friend to find out what happened. (I am not even a pro Linux administrator nor the owner is.) I also have two of my sites on this server so I only know anything about those. (They're Wordpress blogs.) I don't know what type of other sites are hosted on this (~ 20 sites), what services, shopping carts, plugins, etc. do they provide. This also means, that I do not have much time to spend on this investigation unfortunately, because I have to do my job too. I told your advice to the owner of the server, but right now, there's no resource to set up a new server from scratch, as it would take about 2 weeks with all the tasks. So, at least for now, we have to move on with this server. This server hosts only non-profit sites, and because of this, no professional support could be afforded. I know, it sounds scary, but that what we have now: limited resource (time and knowledge) to run this host.
The packages are updated only once or twice a year. The owner wants to do it now again, because it is timely now.
I am grateful for your help, but it can be seen that it's quite difficult to do this investigation because of our circumstances (limited resources, server services can't be shutted down, not being able to move to new one, etc.)
So right now, until there is time to set up a new server, we need to focus on patching this one as much as we can, and try to be alert for any alien activities, do any precautionary measure if possible to detect them.
Plus, both the owner and me are going on vacation now for a week. So we're trying to survive now until we're back.
Then, if you can give us any further tips/instructions to try to harden this host as we can, or about anything else, I'd be grateful. If you don't want to put any further effort to this intrusion, I totally understand. (This is like cooking a soup from water and and salt only...)
The packages are updated only once or twice a year. The owner wants to do it now again, because it is timely now.
Right now it appears this machine was compromised about forty days ago. It's not some "simple" breach confined to one single web site but a root compromise. So in my book "timely" isn't a word one should use in a case like this when throwing up hedges, making excuses and showing signs of procrastination.
Quote:
Originally Posted by agocs
This server hosts only non-profit sites, and because of this, no professional support could be afforded.
That simply is a misconception.
To start with, and regardless of where this server is hosted, somebody pays real money for the location, security (if any), electricity, lines, equipment and maintenance. Then setting up a server, installing, configuring and maintaining the OS requires knowledge, time and effort. Which equals money. Supporting clients with their applications, questions and troubleshooting requires knowledge, time and effort as well. Which equals money as well. Now running a non-profit may require all sorts of registration like with the Chamber of Commerce, the Tax Office, etc, etc and there'll be legal requirements like keeping the books. Which requires knowledge, time and effort as well. Which equals money too. So unless you're blithely living under a Rock you know damn well there is no such thing as "free money".
Secondly, and no matter what kind of business one is in, a web site is an efficient way to share information, attract potential customers and build a business. No matter what goods or services a business is selling, the basis for this all is building a trust relationship with clients. Do I need to explain what a lack of trust does for ones business and brand image? What's worse, do you have any idea what it takes to try and regain trust?
Third customers trusted this machines owner to take responsibility for hosting these businesses web sites and keep them from harm. If s/he judges her or his level of professionalism not to be up to par then s/he shouldn't have taken on the job IMHO. Now apart from combating hedges and misconceptions in the end there's only so much I can do to instil a sense of urgency. Remember Linux may be free to use but using it isn't free of responsibilities. Ultimately this machines owner has to take responsibility.
Quote:
Originally Posted by agocs
(..) I am not administering this server officially, (..) (I am not even a pro Linux administrator nor the owner is.) I also have two of my sites on this server so I only know anything about those. (They're Wordpress blogs.) I don't know what type of other sites are hosted (..) I do not have much time to spend on this investigation (..) right now, there's no resource to set up a new server from scratch, as it would take about 2 weeks with all the tasks. (..) it's quite difficult to do this investigation because of our circumstances (limited resources, server services can't be shutted down, not being able to move to new one, etc.) (..) So, at least for now, we have to move on with this server. (..) what we have now: limited resource (time and knowledge) to run this host. (..) So right now, until there is time to set up a new server, we need to focus on patching this one as much as we can, and try to be alert for any alien activities, do any precautionary measure if possible to detect them. (..) Plus, both the owner and me are going on vacation now for a week. (..)
You may say you're not administering this server officially but you're the one that alerted us to this case more than two weeks ago. Plus you clearly already have root access so there's no excuse not to find out. Plus I think I've outlined what you should do quite well so it's not like you have to figure out all things and all by yourself. And provided you are actually willing to put in the work a migration could have been done over the weekend.
In short: we're talking about an ongoing root compromise, so this server can not be kept in use and patched. Willingly allowing unsuspecting customers to have their credentials sniffed and data robbed is nothing short of criminal negligence. To drive the point home: wow would you feel if your Bank told you it sniffed and sold your data for the past year?
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
