Linux - SecurityThis forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
Could you explain this last bit in a bit more depth? Are you saying that the logging command you found isn't actually logging? That might be a good thing, but it still doesn't explain how it got there and unless you do a bit more investigating, you don't know if other compromises have happened.
Hi Hangdog,
Do you want me to upload all the output screen here. Its quite long dowh.. while there too many line to be censored due it containing server ip. thank you.
Click here to see the post LQ members have rated as the most helpful post in this thread.
Also, if you didn't delete /dev/httpd which is the suspected ssh logger, you can use it's timestamps to perhaps help narrow down in the logs where to start looking?
nomb
Hi nomb,
Im not sure how to check the timestamp of the /dev/httpd. can you help me here? thank you in advance.
Im not sure how to check the timestamp of the /dev/httpd. can you help me here? thank you in advance
Try running stat /dev/httpd and see what that puts out (and please post it). That should give you some dates that you can use as entry points for searching your logs.
Quote:
Do you want me to upload all the output screen here. Its quite long dowh.. while there too many line to be censored due it containing server ip. thank you
Let me check into where would be a good place to store them. If you want to email them to me as attachments, that would be fine as well. In the spirit of full disclosure, I'd like to share these with a couple other people from this forum who have skills well beyond mine in analyzing this sort of data. If you're not comfortable with that, please let me know. You can email me by clicking on my name in the upper left.
By the way, unSpawn pointed me in a direction that suggests this has the potential to be a pretty substantial exploit. If you read through this thread, there is definitely the potential for there to be a trojaned SSH server installed. You'll probably want to run rpm -Vv on the openssh-server package to see if it is the official package or if it has been modified. If it has been modified, you need to isolate that machine pronto.
Thx for the reply Giaamy,
Im doing a forensic task now. The sever was 1st invade by mySQL injection. It was running joomla and now i've remove the joomla main folder. Is there any forensic step is recon for this?
If they hacked joomla, that would only give them user access to your server, the same user as apache runs as, whether that is the username of the owner of that hosting account or nobody.
They would then have to escalate their privilege to root, for example through an out of date kernel, to be able to take over the server, write to /dev, etc.
Hi Hangdog,
Do you want me to upload all the output screen here. Its quite long dowh.. while there too many line to be censored due it containing server ip. thank you.
Use vi (or sed) to find all instances of your server's IP and replace it with 'xxx.xxx.xxx.xxx'.
Ex:
sed -i 's/1.2.3.4/xxx.xxx.xxx.xxx/g' /home/biotones/logs/log1.txt
Try running stat /dev/httpd and see what that puts out (and please post it). That should give you some dates that you can use as entry points for searching your logs.
If you read through this thread, there is definitely the potential for there to be a trojaned SSH server installed. You'll probably want to run rpm -Vv on the openssh-server package to see if it is the official package or if it has been modified. If it has been modified, you need to isolate that machine pronto.
Code:
# rpm -Vv openssh-server
........ c /etc/pam.d/sshd
........ /etc/rc.d/init.d/sshd
........ /etc/ssh
S.5....T c /etc/ssh/sshd_config
........ /usr/libexec/openssh/sftp-server
S.5....T /usr/sbin/sshd
........ d /usr/share/man/man5/sshd_config.5.gz
........ d /usr/share/man/man8/sftp-server.8.gz
........ d /usr/share/man/man8/sshd.8.gz
........ /var/empty/sshd
........ /var/empty/sshd/etc
- you log in over SSH as root. You should not do that. Ever.
- 'md5sum /usr/sbin/sshd' will not work on prelinked systems: compare 'rpm -q --dump openssh-server |grep n/sshd' with 'prelink --verify --md5 /usr/sbin/sshd'.
* rpm -Vv shows your sshd binary and config changed. Did you inspect sshd_config for changes?
* /dev/httpd is owned by root user and group. This means GAME OVER as no unprivileged or system user should be able to write into /dev/. If catting the file shows logins and passwords then this means game over twice as much.
* you stated you cut off Joomla but you still serve databases to (adjacent?) systems. Root compromise of this system holds risks for adjacent servers as well. You must broaden the scope of your investigation to include those machines.
Before we continue I'd like to dwell on critical business infrastructure for second because you may object and likely for the wrong reasons. Running GNU/Linux is all about performance, protecting assets and providing services in a continuous, stable and secure way. Like you trust Linux so your customers trust you. Isolating a (perceived) compromised machine holds a business risk but likewise your customers finding out the machine got compromised and their data (or login/passwd combos) got stolen holds a risk too. It is my humble opinion that if you let this situation persist the damage due to losing customer trust (easily lost, difficult to regain), customer lawsuits and damage to adjacent systems will be higher than off-lining the machine. (If you do not have a spare, trustworthy machine to reroute services to then you have a hiatus in your business continuity planning.) Also please realize that this problem does not only concern you: if this machine was or is being used as a springboard to systems owned by others then you may run the risk of litigation there as well. In short please understand that there are no compelling, valid reasons to keep a root-compromised machine accessible and in operational use.
Until you can confirm there is no root compromise I recommend you perform tasks in this order:
- shut down all services (mysql, sendmail, lighttpd) that are not required for system operation of this machine (leaving SSH),
- disable all user accounts that are not required for system operation of this machine,
- change all root and user passwords on this and adjacent machines you provide services to and vice versa,
- raise the firewall on this machine to only allow SSH traffic from and to your management IP (range),
- make a backup for investigative purposes only of this machine and store off-site (we might want to do log correlation later on).
- create a new unprivileged user account, set a strong password, enable SSH pubkey auth only and set up Sudo so it can perform tasks requiring root rights,
- forcefully re-install all OpenSSH components including dependencies like libraries.
If you performed these steps then at this point you should end up with an isolated environment that you can work in in a less unsafe way. Since you have indicated a SQL injection you should be able to correlate logs (maybe using Logwatch to point out leads) to find out how the attacker got in. Next to that performing the steps from the CERT doc could provide valuable information. Note that at this point there should be no talk of "restoring" because you are not able to verify the integrity of backups (if you actually made them).
If the above approach does not suit you please let us know what you intend to do and please provide clear, complete, detailed information at all times.
* If you would like to share information, reporting or logs beyond the capacity of LQ you are invited to contact me by email.
Last edited by unSpawn; 09-09-2010 at 04:54 PM.
Reason: //More *is* more.
- you log in over SSH as root. You should not do that. Ever.
- 'md5sum /usr/sbin/sshd' will not work on prelinked systems: compare 'rpm -q --dump openssh-server |grep n/sshd' with 'prelink --verify --md5 /usr/sbin/sshd'.
* rpm -Vv shows your sshd binary and config changed. Did you inspect sshd_config for changes?
* /dev/httpd is owned by root user and group. This means GAME OVER as no unprivileged or system user should be able to write into /dev/. If catting the file shows logins and passwords then this means game over twice as much.
* you stated you cut off Joomla but you still serve databases to (adjacent?) systems. Root compromise of this system holds risks for adjacent servers as well. You must broaden the scope of your investigation to include those machines.
Regarding this root allowed in ssh is really my fault. The server was long installed by another colleague which i believe have forgotten on hardening the machine, however done is done and we learn from mistake. The other web server running on Windows server. btw the server is under a firewall therefore no ssh will be allowed in unless it is from my organization IP.
Quote:
Before we continue I'd like to dwell on critical business infrastructure for second because you may object and likely for the wrong reasons. Running GNU/Linux is all about performance, protecting assets and providing services in a continuous, stable and secure way. Like you trust Linux so your customers trust you. Isolating a (perceived) compromised machine holds a business risk but likewise your customers finding out the machine got compromised and their data (or login/passwd combos) got stolen holds a risk too. It is my humble opinion that if you let this situation persist the damage due to losing customer trust (easily lost, difficult to regain), customer lawsuits and damage to adjacent systems will be higher than off-lining the machine. (If you do not have a spare, trustworthy machine to reroute services to then you have a hiatus in your business continuity planning.) Also please realize that this problem does not only concern you: if this machine was or is being used as a springboard to systems owned by others then you may run the risk of litigation there as well. In short please understand that there are no compelling, valid reasons to keep a root-compromised machine accessible and in operational use.
This is just part of my organization video feeding server. There is no other party involved in this matter.
Quote:
Until you can confirm there is no root compromise I recommend you perform tasks in this order:
- shut down all services (mysql, sendmail, lighttpd) that are not required for system operation of this machine (leaving SSH),
- disable all user accounts that are not required for system operation of this machine,
- change all root and user passwords on this and adjacent machines you provide services to and vice versa,
- raise the firewall on this machine to only allow SSH traffic from and to your management IP (range),
- make a backup for investigative purposes only of this machine and store off-site (we might want to do log correlation later on).
- create a new unprivileged user account, set a strong password, enable SSH pubkey auth only and set up Sudo so it can perform tasks requiring root rights,
- forcefully re-install all OpenSSH components including dependencies like libraries.
If you performed these steps then at this point you should end up with an isolated environment that you can work in in a less unsafe way. Since you have indicated a SQL injection you should be able to correlate logs (maybe using Logwatch to point out leads) to find out how the attacker got in. Next to that performing the steps from the CERT doc could provide valuable information. Note that at this point there should be no talk of "restoring" because you are not able to verify the integrity of backups (if you actually made them).
Ill do it ASAP
Quote:
If the above approach does not suit you please let us know what you intend to do and please provide clear, complete, detailed information at all times.
* If you would like to share information, reporting or logs beyond the capacity of LQ you are invited to contact me by email.
This is just part of my organization video feeding server. There is no other party involved in this matter.
I'm confused here. You stated the below in response to someone stating to take the machine off the network:
Quote:
Originally Posted by biotones
Dude thx 4 the reply, how ever this is indeed a critical web hosting server.
unSpawn responded to that with the following:
Quote:
Originally Posted by unSpawn
Before we continue I'd like to dwell on critical business infrastructure for second because you may object and likely for the wrong reasons. Running GNU/Linux is all about performance, protecting assets and providing services in a continuous, stable and secure way. Like you trust Linux so your customers trust you. Isolating a (perceived) compromised machine holds a business risk but likewise your customers finding out the machine got compromised and their data (or login/passwd combos) got stolen holds a risk too. It is my humble opinion that if you let this situation persist the damage due to losing customer trust (easily lost, difficult to regain), customer lawsuits and damage to adjacent systems will be higher than off-lining the machine. (If you do not have a spare, trustworthy machine to reroute services to then you have a hiatus in your business continuity planning.) Also please realize that this problem does not only concern you: if this machine was or is being used as a springboard to systems owned by others then you may run the risk of litigation there as well. In short please understand that there are no compelling, valid reasons to keep a root-compromised machine accessible and in operational use.
It appears that you're now saying that the system isn't critical, or maybe that it's less critical than initially thought.
To be honest, this is a bit upsetting. There are several threads every week that state a system has been compromised and that assistance is needed to resolve the issue. When advise is given, one of the first responses to almost every single thread is that the system is critical and it can't be removed from the network. You see, its a catch-22 type thing...if you're not willing to follow that advise, then you're probably not in as dire a situation as the thread subject line and post content suggests (which usually reads, "system compromised...help!!!!").
As unSpawn mentioned, hiding the issue from your customers will seriously cause problems later...trust is a HARD thing to earn with customers, and much harder to regain if lost. Again, there's also the issue of your compromised machine attacking other machines. IMO, it is very hard to argue the fact that a system is critical. If it were that critical, you'd probably have already had the proper security layers in place (and wouldn't be accessing a machine via SSH as root).
I'm betting that this machine is still online. That creates an ethical problem, IMO.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
i=40;33:so=00;35:bd=40;33;01:cd=40;33;01
r=01;05;37;41:mi= 01;05;37;41:ex=00;32:*.cmd=00;32:*.exe=00;32:*.com=00;32:*.btm=00;32:*.bat=00;32:*.sh=00;32:*.csh=00 ;32:*.tar=00;31:*.tgz=00;31:*.arj=00;31:*.taz=00;31:*.lzh=00;31:*.zip=00;31:*.z=00;31:*.Z=00;31:*.gz =00;31:*.bz2=00;31:*.bz=00;31:*.tz=00;31:*.rpm=00;31:*.cpio=00;31:*.jpg=00;35:*.gif=00;35:*.bmp=00;3 5:*.xbm=00;35:*.xpm=00;35:*.png=00;35:*.tif=00;35: MAIL=/var/spool/mail/root PATH=/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/root/bin INPUTRC=/etc/inputrc PWD=/root LANG=en_US.UTF-8 SHLVL=1 HOME=/root LOGNAME=root SSH_CONNECTION=::ffff:ip 10113 ::ffff:ip 22 LESSOPEN=|/usr/bin/lesspipe.sh %s G_BROKEN_FILENAMES=1 _=/bin/ps
