LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Security (https://www.linuxquestions.org/questions/linux-security-4/)
-   -   plagued by malware (https://www.linuxquestions.org/questions/linux-security-4/plagued-by-malware-4175576011/)

skyred5 03-28-2016 01:29 AM

plagued by malware
 
Hello everyone, first post on this forum!

I am a new user of linux and apparently, due to work requirements I am required to manage a linux DB server. However there is a major problem with it right now. It seems that the DB server has been infected by some kind of malware/virus/bot, that's making constant and ALOT of outgoing transmissions particularly to China. The name of this thing is called "dafa2016". I tried to google it up, but I'm not getting much info on it.

I do have linux malware detect but it does not detect anything. I have since blocked all outgoing connections from the particular server on my firewall to prevent it from causing slow network.

I had some basic help from a colleague to trace the directory of this malware and to remove it, but it is regenerating itself. Seriously need help on this.

If more information is required, please do let me know. I'll try my best to provide.

ButterflyMelissa 03-28-2016 02:01 AM

Hi,
Welcome to the forum! And...with a great one too...
Okay, some basics. Got rkhunter installed? Let's start here.
You do know Linux has regular security updates. Whan last did you update the system? Is it a rolling release? Not optimal for a server. So, if the one you have now is EOL...get ready to camp out on the office...for a weekend.
Is the server for the office only? Or on the net? Reason is: if it's for the office only, you have no pressure, well...less pressure...
If it is EOL, a backup of the data, and a re-install and I think that'll do it...
What brand (distro) do you use?
If it regenerates itself, there can be two reasons: either you did not yet find the root of the thing, or there is a rootkit and the stuff is re-installed remotely...
I round off my questions with some links, one on malware removal, on detecting and one extract from a forum...
We're all here for you, so good luck...
Melissa

skyred5 03-28-2016 03:41 AM

Quote:

Originally Posted by ButterflyMelissa (Post 5522305)
Hi,
Welcome to the forum! And...with a great one too...
Okay, some basics. Got rkhunter installed? Let's start here.
You do know Linux has regular security updates. Whan last did you update the system? Is it a rolling release? Not optimal for a server. So, if the one you have now is EOL...get ready to camp out on the office...for a weekend.
Is the server for the office only? Or on the net? Reason is: if it's for the office only, you have no pressure, well...less pressure...
If it is EOL, a backup of the data, and a re-install and I think that'll do it...
What brand (distro) do you use?
If it regenerates itself, there can be two reasons: either you did not yet find the root of the thing, or there is a rootkit and the stuff is re-installed remotely...
I round off my questions with some links, one on malware removal, on detecting and one extract from a forum...
We're all here for you, so good luck...
Melissa

Hello! Thanks for the reply!

No, I don't have this rkhunter installed. This linux system is actually for a managed services vendor(Sciencelogic). The version was recently updated last November. The server is for the office. I'll take a look at the links that you provided!

ButterflyMelissa 03-28-2016 04:21 AM

Quote:

The version was recently updated last November
Ehm, I hate to put a damper on your enthousiasm, but...that is an eternity...
When did the malware come to light? Possibly...AFTER the update...
My systems update on a weekly base...
I'd go for an install of rkhunter...
It's for the office, that is good, annoying about the malware, but...good...
So, what distro was that again?
Try to paste/replicate/type the output of this:
Code:

uname -a
...you...enter that in the console...
Melissa

skyred5 03-28-2016 04:56 AM

Quote:

Originally Posted by ButterflyMelissa (Post 5522347)
Ehm, I hate to put a damper on your enthousiasm, but...that is an eternity...
When did the malware come to light? Possibly...AFTER the update...
My systems update on a weekly base...
I'd go for an install of rkhunter...
It's for the office, that is good, annoying about the malware, but...good...
So, what distro was that again?
Try to paste/replicate/type the output of this:
Code:

uname -a
...you...enter that in the console...
Melissa

This is the output result.

Linux newio_sldb01 2.6.18-406.el5 #1 SMP Tue Jun 2 17:25:57 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux

ButterflyMelissa 03-28-2016 05:08 AM

Emm, okay, a few names:
debian
ubuntu
fedora
any of these ring a bell?
Melissa

skyred5 03-28-2016 05:12 AM

Quote:

Originally Posted by ButterflyMelissa (Post 5522364)
Emm, okay, a few names:
debian
ubuntu
fedora
any of these ring a bell?
Melissa

It's ubuntu. I'm sure of that.

unSpawn 03-28-2016 05:36 AM

Quote:

Originally Posted by skyred5 (Post 5522290)
Hello everyone, first post on this forum!

Welcome to LQ, sorry to see the cause isn't positive.


Quote:

Originally Posted by skyred5 (Post 5522290)
I have since blocked all outgoing connections from the particular server on my firewall to prevent it from causing slow network. I had some basic help from a colleague to trace the directory of this malware and to remove it, but it is regenerating itself.

This first reflex is fine, but deleting foreign objects without properly investigating first is completely wrong. (Blame your colleague for apparently now knowing how to handle incidents either.)


Quote:

Originally Posted by skyred5
2.6.18-406.el5

This identifies it as a Red Hat Enterprise Linux 5 machine or equivalent. That kernel is from June 2nd last year. The current kernel, as far as I know, is 2.6.18-409.el5 of March 16th, meaning this system likely wasn't kept up to date at all. I don't "blame" you for that, from your initial post it's clear you're a (reluctant?) new Linux user, but do realize that not keeping systems up to date combined with lax security and auditing makes it low hanging fruit for those with malicious intent.

Please become root and post full output (see "/tmp/.log.txt") of
Code:

( \ps axfwwwe 2>&1; lsof -Pwln 2>&1; netstat -antupe 2>&1; lastlog 2>&1; last -wai30 2>&1; cat /var/spool/cron/* 2>&1; find /etc/ -type f -print0  2>&1|xargs -0 -iX rpm -qf 'X' 2>&1 ) | tee /tmp/.log.txt
Simultaneously search your backup archives and mark them as "suspicious" so they won't inadvertedly be used to "restore" a system you do not know it's state of. Also notify your departments incident response team (if any), seniors or higher-ups a system was compromised.

skyred5 03-28-2016 05:45 AM

1 Attachment(s)
Quote:

Originally Posted by unSpawn (Post 5522372)
Welcome to LQ, sorry to see the cause isn't positive.



This first reflex is fine, but deleting foreign objects without properly investigating first is completely wrong. (Blame your colleague for apparently now knowing how to handle incidents either.)



This identifies it as a Red Hat Enterprise Linux 5 machine or equivalent. That kernel is from June 2nd last year. The current kernel, as far as I know, is 2.6.18-409.el5 of March 16th, meaning this system likely wasn't kept up to date at all. I don't "blame" you for that, from your initial post it's clear you're a (reluctant?) new Linux user, but do realize that not keeping systems up to date combined with lax security and auditing makes it low hanging fruit for those with malicious intent.

Please become root and post full output (see "/tmp/.log.txt") of
Code:

( \ps axfwwwe 2>&1; lsof -Pwln 2>&1; netstat -antupe 2>&1; lastlog 2>&1; last -wai30 2>&1; cat /var/spool/cron/* 2>&1; find /etc/ -type f -print0  2>&1|xargs -0 -iX rpm -qf 'X' 2>&1 ) | tee /tmp/.log.txt
Simultaneously search your backup archives and mark them as "suspicious" so they won't inadvertedly be used to "restore" a system you do not know it's state of. Also notify your departments incident response team (if any), seniors or higher-ups a system was compromised.

It's a freaking long post, I'll put it in a notepad for convenience.

ilesterg 03-28-2016 05:59 AM

You sure have a lot of active root logins, and I can see 2 IPs logged in from Singapore. Assuming those 2 logins are legitimate SSH session, it is actually a good practice to DISABLE root logins via SSH. In case those 2 logins are NOT legitimate, you should panic now. Then kill those sessions.

Code:

root    pts/0        Mon Mar 28 10:39  still logged in    10.10.8.104
root    pts/0        Mon Mar 28 09:55 - 10:00  (00:05)    10.10.8.104
root    pts/0        Mon Mar 28 09:37 - 09:42  (00:05)    10.10.8.104
root    pts/0        Mon Mar 28 05:48 - 06:30  (00:41)    10.10.8.104
root    pts/0        Mon Mar 28 04:47 - 05:01  (00:14)    10.10.8.104
root    pts/0        Mon Mar 28 04:34 - 04:45  (00:10)    10.10.8.104
root    pts/1        Mon Mar 28 04:23 - 06:19  (01:55)    10.10.8.104
root    pts/0        Mon Mar 28 03:59 - 04:34  (00:35)    10.10.8.104
root    pts/0        Fri Mar 25 16:33 - 16:40  (00:07)    220.255.180.160
root    pts/0        Wed Mar 23 08:44 - 08:59  (00:15)    10.10.8.104
root    pts/0        Wed Mar 23 08:17 - 08:44  (00:26)    10.10.8.104
root    pts/0        Wed Mar 23 08:12 - 08:14  (00:01)    10.10.8.104
root    pts/0        Wed Mar 23 07:52 - 08:04  (00:12)    10.10.8.104
root    pts/1        Wed Mar 23 07:36 - 07:51  (00:15)    10.10.8.104
root    pts/1        Wed Mar 23 07:25 - 07:31  (00:05)    10.10.8.104
root    pts/1        Wed Mar 23 07:24 - 07:25  (00:00)    10.10.8.104
root    pts/0        Wed Mar 23 06:24 - 07:48  (01:23)    10.10.8.104
root    pts/1        Wed Mar 23 03:54 - 04:00  (00:05)    10.10.8.104
root    pts/0        Wed Mar 23 03:49 - 04:00  (00:10)    10.10.8.104
root    pts/0        Wed Mar 23 03:20 - 03:29  (00:09)    10.10.8.104
root    pts/0        Tue Mar 22 07:34 - 08:57  (01:23)    10.10.10.66
root    pts/0        Tue Mar 22 07:12 - 07:18  (00:06)    10.10.10.66
root    pts/0        Tue Mar 22 07:06 - 07:12  (00:06)    10.10.10.66
root    pts/0        Tue Mar 22 06:25 - 06:33  (00:07)    10.10.10.66
root    pts/0        Fri Mar 18 09:46 - 09:52  (00:05)    10.10.8.104
root    pts/0        Fri Mar 18 08:48 - 08:56  (00:07)    10.10.8.104
root    pts/0        Thu Mar 17 03:05 - 04:05  (01:00)    10.10.8.104
root    pts/0        Thu Mar 17 02:12 - 03:05  (00:52)    10.10.8.104
root    pts/0        Wed Mar 16 07:16 - 08:06  (00:49)    10.10.8.104
root    pts/0        Tue Mar 15 13:35 - 14:30  (00:55)    220.255.180.160


unSpawn 03-28-2016 06:30 AM

Quote:

Originally Posted by skyred5 (Post 5522377)
It's a freaking long post, I'll put it in a notepad for convenience.

You missed posting requested '\ps axfwwwe;' output. What's more you either didn't run 'lsof -Pwln;' as root or its output was doctored as I'm missing crucial nfo.

Couple of things:
- Obviously root should not ever be allowed to log in over SSH,
- There's also some "phonehome" user accounts,
- There is a process running as root with PID 27256
- There's the "/etc/rc.d/init.d/DbSecuritySpt" indicative of ValdikSS/billgates-botnet-tracker (see for example https://www.linuxquestions.org/quest...1/).

In short: root compromise, ergo isolate the machine from the network now.

skyred5 03-28-2016 06:30 AM

Quote:

Originally Posted by ilesterg (Post 5522379)
You sure have a lot of active root logins, and I can see 2 IPs logged in from Singapore. Assuming those 2 logins are legitimate SSH session, it is actually a good practice to DISABLE root logins via SSH. In case those 2 logins are NOT legitimate, you should panic now. Then kill those sessions.

Code:

root    pts/0        Mon Mar 28 10:39  still logged in    10.10.8.104
root    pts/0        Mon Mar 28 09:55 - 10:00  (00:05)    10.10.8.104
root    pts/0        Mon Mar 28 09:37 - 09:42  (00:05)    10.10.8.104
root    pts/0        Mon Mar 28 05:48 - 06:30  (00:41)    10.10.8.104
root    pts/0        Mon Mar 28 04:47 - 05:01  (00:14)    10.10.8.104
root    pts/0        Mon Mar 28 04:34 - 04:45  (00:10)    10.10.8.104
root    pts/1        Mon Mar 28 04:23 - 06:19  (01:55)    10.10.8.104
root    pts/0        Mon Mar 28 03:59 - 04:34  (00:35)    10.10.8.104
root    pts/0        Fri Mar 25 16:33 - 16:40  (00:07)    220.255.180.160
root    pts/0        Wed Mar 23 08:44 - 08:59  (00:15)    10.10.8.104
root    pts/0        Wed Mar 23 08:17 - 08:44  (00:26)    10.10.8.104
root    pts/0        Wed Mar 23 08:12 - 08:14  (00:01)    10.10.8.104
root    pts/0        Wed Mar 23 07:52 - 08:04  (00:12)    10.10.8.104
root    pts/1        Wed Mar 23 07:36 - 07:51  (00:15)    10.10.8.104
root    pts/1        Wed Mar 23 07:25 - 07:31  (00:05)    10.10.8.104
root    pts/1        Wed Mar 23 07:24 - 07:25  (00:00)    10.10.8.104
root    pts/0        Wed Mar 23 06:24 - 07:48  (01:23)    10.10.8.104
root    pts/1        Wed Mar 23 03:54 - 04:00  (00:05)    10.10.8.104
root    pts/0        Wed Mar 23 03:49 - 04:00  (00:10)    10.10.8.104
root    pts/0        Wed Mar 23 03:20 - 03:29  (00:09)    10.10.8.104
root    pts/0        Tue Mar 22 07:34 - 08:57  (01:23)    10.10.10.66
root    pts/0        Tue Mar 22 07:12 - 07:18  (00:06)    10.10.10.66
root    pts/0        Tue Mar 22 07:06 - 07:12  (00:06)    10.10.10.66
root    pts/0        Tue Mar 22 06:25 - 06:33  (00:07)    10.10.10.66
root    pts/0        Fri Mar 18 09:46 - 09:52  (00:05)    10.10.8.104
root    pts/0        Fri Mar 18 08:48 - 08:56  (00:07)    10.10.8.104
root    pts/0        Thu Mar 17 03:05 - 04:05  (01:00)    10.10.8.104
root    pts/0        Thu Mar 17 02:12 - 03:05  (00:52)    10.10.8.104
root    pts/0        Wed Mar 16 07:16 - 08:06  (00:49)    10.10.8.104
root    pts/0        Tue Mar 15 13:35 - 14:30  (00:55)    220.255.180.160


The 10.10.8.x and 10.10.10.x are legit ip address... I would need to check on the 220.x.x.x ip address. *panic mode ON*

ButterflyMelissa 03-28-2016 06:44 AM

Quote:

*panic mode ON*
Dont panic, absorb, silence your mind, become one with the battlefield...
If I may sugest, in light of this:
Quote:

You sure have a lot of active root logins, and I can see 2 IPs logged in from Singapore
...to find out how to remove rootkits too...just me (not that educated but extremely interested) venting some advice...
Next, indeed, do disallow root access over SSH...because:
Quote:

Obviously root should not ever be allowed to log in over SSH
Melissa
Edit - for this suggestion I stand open for critique as it is somewhat far fetched...open passwd (usualy found in /etc) and check if any account other than root has root rights...
Root is identified by group 0 (as I was told) in the image, entry 3

unSpawn 03-28-2016 07:13 AM

Quote:

Originally Posted by ButterflyMelissa (Post 5522396)
...to find out how to remove rootkits too...

No. No. No: anything foreign injected into a system operating as root user shall not be "removed" or otherwise "cleaned up". That is not a security best practice.

Security is a continuous process requiring a layered approach. Any software in the web stack that is not updated, stale, unsupported or outright vulnerable is low hanging fruit. Each OS comes with installation, admin and security documentation and documentation of software in the web stack often has a specific security section. It's just that people often don't read much, don't care enough or outright neglect admin best practices. This is what you get.

Here the only approach is to set up a new server, harden it properly and then migrate data but only after thorough inspection.

ButterflyMelissa 03-28-2016 07:48 AM

Quote:

No. No. No: anything foreign injected into a system operating as root user shall not be "removed" or otherwise "cleaned up". That is not a security best practice.
Lesson learned... :)
@ OP - please dont take my input a fool proof :)
Melissa
(looking on, with intense interest)


All times are GMT -5. The time now is 12:57 AM.