LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices


Reply
  Search this Thread
Old 03-28-2016, 01:29 AM   #1
skyred5
LQ Newbie
 
Registered: Mar 2016
Posts: 14

Rep: Reputation: Disabled
Unhappy plagued by malware


Hello everyone, first post on this forum!

I am a new user of linux and apparently, due to work requirements I am required to manage a linux DB server. However there is a major problem with it right now. It seems that the DB server has been infected by some kind of malware/virus/bot, that's making constant and ALOT of outgoing transmissions particularly to China. The name of this thing is called "dafa2016". I tried to google it up, but I'm not getting much info on it.

I do have linux malware detect but it does not detect anything. I have since blocked all outgoing connections from the particular server on my firewall to prevent it from causing slow network.

I had some basic help from a colleague to trace the directory of this malware and to remove it, but it is regenerating itself. Seriously need help on this.

If more information is required, please do let me know. I'll try my best to provide.
 
Old 03-28-2016, 02:01 AM   #2
ButterflyMelissa
Senior Member
 
Registered: Nov 2007
Location: Somewhere on my hard drive...
Distribution: Manjaro
Posts: 2,766
Blog Entries: 23

Rep: Reputation: 411Reputation: 411Reputation: 411Reputation: 411Reputation: 411
Hi,
Welcome to the forum! And...with a great one too...
Okay, some basics. Got rkhunter installed? Let's start here.
You do know Linux has regular security updates. Whan last did you update the system? Is it a rolling release? Not optimal for a server. So, if the one you have now is EOL...get ready to camp out on the office...for a weekend.
Is the server for the office only? Or on the net? Reason is: if it's for the office only, you have no pressure, well...less pressure...
If it is EOL, a backup of the data, and a re-install and I think that'll do it...
What brand (distro) do you use?
If it regenerates itself, there can be two reasons: either you did not yet find the root of the thing, or there is a rootkit and the stuff is re-installed remotely...
I round off my questions with some links, one on malware removal, on detecting and one extract from a forum...
We're all here for you, so good luck...
Melissa
 
Old 03-28-2016, 03:41 AM   #3
skyred5
LQ Newbie
 
Registered: Mar 2016
Posts: 14

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by ButterflyMelissa View Post
Hi,
Welcome to the forum! And...with a great one too...
Okay, some basics. Got rkhunter installed? Let's start here.
You do know Linux has regular security updates. Whan last did you update the system? Is it a rolling release? Not optimal for a server. So, if the one you have now is EOL...get ready to camp out on the office...for a weekend.
Is the server for the office only? Or on the net? Reason is: if it's for the office only, you have no pressure, well...less pressure...
If it is EOL, a backup of the data, and a re-install and I think that'll do it...
What brand (distro) do you use?
If it regenerates itself, there can be two reasons: either you did not yet find the root of the thing, or there is a rootkit and the stuff is re-installed remotely...
I round off my questions with some links, one on malware removal, on detecting and one extract from a forum...
We're all here for you, so good luck...
Melissa
Hello! Thanks for the reply!

No, I don't have this rkhunter installed. This linux system is actually for a managed services vendor(Sciencelogic). The version was recently updated last November. The server is for the office. I'll take a look at the links that you provided!
 
Old 03-28-2016, 04:21 AM   #4
ButterflyMelissa
Senior Member
 
Registered: Nov 2007
Location: Somewhere on my hard drive...
Distribution: Manjaro
Posts: 2,766
Blog Entries: 23

Rep: Reputation: 411Reputation: 411Reputation: 411Reputation: 411Reputation: 411
Quote:
The version was recently updated last November
Ehm, I hate to put a damper on your enthousiasm, but...that is an eternity...
When did the malware come to light? Possibly...AFTER the update...
My systems update on a weekly base...
I'd go for an install of rkhunter...
It's for the office, that is good, annoying about the malware, but...good...
So, what distro was that again?
Try to paste/replicate/type the output of this:
Code:
uname -a
...you...enter that in the console...
Melissa

Last edited by ButterflyMelissa; 03-28-2016 at 04:23 AM.
 
Old 03-28-2016, 04:56 AM   #5
skyred5
LQ Newbie
 
Registered: Mar 2016
Posts: 14

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by ButterflyMelissa View Post
Ehm, I hate to put a damper on your enthousiasm, but...that is an eternity...
When did the malware come to light? Possibly...AFTER the update...
My systems update on a weekly base...
I'd go for an install of rkhunter...
It's for the office, that is good, annoying about the malware, but...good...
So, what distro was that again?
Try to paste/replicate/type the output of this:
Code:
uname -a
...you...enter that in the console...
Melissa
This is the output result.

Linux newio_sldb01 2.6.18-406.el5 #1 SMP Tue Jun 2 17:25:57 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
 
Old 03-28-2016, 05:08 AM   #6
ButterflyMelissa
Senior Member
 
Registered: Nov 2007
Location: Somewhere on my hard drive...
Distribution: Manjaro
Posts: 2,766
Blog Entries: 23

Rep: Reputation: 411Reputation: 411Reputation: 411Reputation: 411Reputation: 411
Emm, okay, a few names:
debian
ubuntu
fedora
any of these ring a bell?
Melissa
 
Old 03-28-2016, 05:12 AM   #7
skyred5
LQ Newbie
 
Registered: Mar 2016
Posts: 14

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by ButterflyMelissa View Post
Emm, okay, a few names:
debian
ubuntu
fedora
any of these ring a bell?
Melissa
It's ubuntu. I'm sure of that.
 
Old 03-28-2016, 05:36 AM   #8
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
Quote:
Originally Posted by skyred5 View Post
Hello everyone, first post on this forum!
Welcome to LQ, sorry to see the cause isn't positive.


Quote:
Originally Posted by skyred5 View Post
I have since blocked all outgoing connections from the particular server on my firewall to prevent it from causing slow network. I had some basic help from a colleague to trace the directory of this malware and to remove it, but it is regenerating itself.
This first reflex is fine, but deleting foreign objects without properly investigating first is completely wrong. (Blame your colleague for apparently now knowing how to handle incidents either.)


Quote:
Originally Posted by skyred5
2.6.18-406.el5
This identifies it as a Red Hat Enterprise Linux 5 machine or equivalent. That kernel is from June 2nd last year. The current kernel, as far as I know, is 2.6.18-409.el5 of March 16th, meaning this system likely wasn't kept up to date at all. I don't "blame" you for that, from your initial post it's clear you're a (reluctant?) new Linux user, but do realize that not keeping systems up to date combined with lax security and auditing makes it low hanging fruit for those with malicious intent.

Please become root and post full output (see "/tmp/.log.txt") of
Code:
( \ps axfwwwe 2>&1; lsof -Pwln 2>&1; netstat -antupe 2>&1; lastlog 2>&1; last -wai30 2>&1; cat /var/spool/cron/* 2>&1; find /etc/ -type f -print0  2>&1|xargs -0 -iX rpm -qf 'X' 2>&1 ) | tee /tmp/.log.txt
Simultaneously search your backup archives and mark them as "suspicious" so they won't inadvertedly be used to "restore" a system you do not know it's state of. Also notify your departments incident response team (if any), seniors or higher-ups a system was compromised.

Last edited by unSpawn; 03-28-2016 at 05:37 AM.
 
1 members found this post helpful.
Old 03-28-2016, 05:45 AM   #9
skyred5
LQ Newbie
 
Registered: Mar 2016
Posts: 14

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by unSpawn View Post
Welcome to LQ, sorry to see the cause isn't positive.



This first reflex is fine, but deleting foreign objects without properly investigating first is completely wrong. (Blame your colleague for apparently now knowing how to handle incidents either.)



This identifies it as a Red Hat Enterprise Linux 5 machine or equivalent. That kernel is from June 2nd last year. The current kernel, as far as I know, is 2.6.18-409.el5 of March 16th, meaning this system likely wasn't kept up to date at all. I don't "blame" you for that, from your initial post it's clear you're a (reluctant?) new Linux user, but do realize that not keeping systems up to date combined with lax security and auditing makes it low hanging fruit for those with malicious intent.

Please become root and post full output (see "/tmp/.log.txt") of
Code:
( \ps axfwwwe 2>&1; lsof -Pwln 2>&1; netstat -antupe 2>&1; lastlog 2>&1; last -wai30 2>&1; cat /var/spool/cron/* 2>&1; find /etc/ -type f -print0  2>&1|xargs -0 -iX rpm -qf 'X' 2>&1 ) | tee /tmp/.log.txt
Simultaneously search your backup archives and mark them as "suspicious" so they won't inadvertedly be used to "restore" a system you do not know it's state of. Also notify your departments incident response team (if any), seniors or higher-ups a system was compromised.
It's a freaking long post, I'll put it in a notepad for convenience.
Attached Files
File Type: txt linux DB server.txt (94.2 KB, 25 views)
 
Old 03-28-2016, 05:59 AM   #10
ilesterg
Member
 
Registered: Jul 2012
Location: München
Distribution: Debian, CentOS/RHEL
Posts: 587

Rep: Reputation: 72
You sure have a lot of active root logins, and I can see 2 IPs logged in from Singapore. Assuming those 2 logins are legitimate SSH session, it is actually a good practice to DISABLE root logins via SSH. In case those 2 logins are NOT legitimate, you should panic now. Then kill those sessions.

Code:
root     pts/0        Mon Mar 28 10:39   still logged in    10.10.8.104
root     pts/0        Mon Mar 28 09:55 - 10:00  (00:05)     10.10.8.104
root     pts/0        Mon Mar 28 09:37 - 09:42  (00:05)     10.10.8.104
root     pts/0        Mon Mar 28 05:48 - 06:30  (00:41)     10.10.8.104
root     pts/0        Mon Mar 28 04:47 - 05:01  (00:14)     10.10.8.104
root     pts/0        Mon Mar 28 04:34 - 04:45  (00:10)     10.10.8.104
root     pts/1        Mon Mar 28 04:23 - 06:19  (01:55)     10.10.8.104
root     pts/0        Mon Mar 28 03:59 - 04:34  (00:35)     10.10.8.104
root     pts/0        Fri Mar 25 16:33 - 16:40  (00:07)     220.255.180.160
root     pts/0        Wed Mar 23 08:44 - 08:59  (00:15)     10.10.8.104
root     pts/0        Wed Mar 23 08:17 - 08:44  (00:26)     10.10.8.104
root     pts/0        Wed Mar 23 08:12 - 08:14  (00:01)     10.10.8.104
root     pts/0        Wed Mar 23 07:52 - 08:04  (00:12)     10.10.8.104
root     pts/1        Wed Mar 23 07:36 - 07:51  (00:15)     10.10.8.104
root     pts/1        Wed Mar 23 07:25 - 07:31  (00:05)     10.10.8.104
root     pts/1        Wed Mar 23 07:24 - 07:25  (00:00)     10.10.8.104
root     pts/0        Wed Mar 23 06:24 - 07:48  (01:23)     10.10.8.104
root     pts/1        Wed Mar 23 03:54 - 04:00  (00:05)     10.10.8.104
root     pts/0        Wed Mar 23 03:49 - 04:00  (00:10)     10.10.8.104
root     pts/0        Wed Mar 23 03:20 - 03:29  (00:09)     10.10.8.104
root     pts/0        Tue Mar 22 07:34 - 08:57  (01:23)     10.10.10.66
root     pts/0        Tue Mar 22 07:12 - 07:18  (00:06)     10.10.10.66
root     pts/0        Tue Mar 22 07:06 - 07:12  (00:06)     10.10.10.66
root     pts/0        Tue Mar 22 06:25 - 06:33  (00:07)     10.10.10.66
root     pts/0        Fri Mar 18 09:46 - 09:52  (00:05)     10.10.8.104
root     pts/0        Fri Mar 18 08:48 - 08:56  (00:07)     10.10.8.104
root     pts/0        Thu Mar 17 03:05 - 04:05  (01:00)     10.10.8.104
root     pts/0        Thu Mar 17 02:12 - 03:05  (00:52)     10.10.8.104
root     pts/0        Wed Mar 16 07:16 - 08:06  (00:49)     10.10.8.104
root     pts/0        Tue Mar 15 13:35 - 14:30  (00:55)     220.255.180.160
 
Old 03-28-2016, 06:30 AM   #11
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
Quote:
Originally Posted by skyred5 View Post
It's a freaking long post, I'll put it in a notepad for convenience.
You missed posting requested '\ps axfwwwe;' output. What's more you either didn't run 'lsof -Pwln;' as root or its output was doctored as I'm missing crucial nfo.

Couple of things:
- Obviously root should not ever be allowed to log in over SSH,
- There's also some "phonehome" user accounts,
- There is a process running as root with PID 27256
- There's the "/etc/rc.d/init.d/DbSecuritySpt" indicative of ValdikSS/billgates-botnet-tracker (see for example https://www.linuxquestions.org/quest...1/#post5167596).

In short: root compromise, ergo isolate the machine from the network now.
 
Old 03-28-2016, 06:30 AM   #12
skyred5
LQ Newbie
 
Registered: Mar 2016
Posts: 14

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by ilesterg View Post
You sure have a lot of active root logins, and I can see 2 IPs logged in from Singapore. Assuming those 2 logins are legitimate SSH session, it is actually a good practice to DISABLE root logins via SSH. In case those 2 logins are NOT legitimate, you should panic now. Then kill those sessions.

Code:
root     pts/0        Mon Mar 28 10:39   still logged in    10.10.8.104
root     pts/0        Mon Mar 28 09:55 - 10:00  (00:05)     10.10.8.104
root     pts/0        Mon Mar 28 09:37 - 09:42  (00:05)     10.10.8.104
root     pts/0        Mon Mar 28 05:48 - 06:30  (00:41)     10.10.8.104
root     pts/0        Mon Mar 28 04:47 - 05:01  (00:14)     10.10.8.104
root     pts/0        Mon Mar 28 04:34 - 04:45  (00:10)     10.10.8.104
root     pts/1        Mon Mar 28 04:23 - 06:19  (01:55)     10.10.8.104
root     pts/0        Mon Mar 28 03:59 - 04:34  (00:35)     10.10.8.104
root     pts/0        Fri Mar 25 16:33 - 16:40  (00:07)     220.255.180.160
root     pts/0        Wed Mar 23 08:44 - 08:59  (00:15)     10.10.8.104
root     pts/0        Wed Mar 23 08:17 - 08:44  (00:26)     10.10.8.104
root     pts/0        Wed Mar 23 08:12 - 08:14  (00:01)     10.10.8.104
root     pts/0        Wed Mar 23 07:52 - 08:04  (00:12)     10.10.8.104
root     pts/1        Wed Mar 23 07:36 - 07:51  (00:15)     10.10.8.104
root     pts/1        Wed Mar 23 07:25 - 07:31  (00:05)     10.10.8.104
root     pts/1        Wed Mar 23 07:24 - 07:25  (00:00)     10.10.8.104
root     pts/0        Wed Mar 23 06:24 - 07:48  (01:23)     10.10.8.104
root     pts/1        Wed Mar 23 03:54 - 04:00  (00:05)     10.10.8.104
root     pts/0        Wed Mar 23 03:49 - 04:00  (00:10)     10.10.8.104
root     pts/0        Wed Mar 23 03:20 - 03:29  (00:09)     10.10.8.104
root     pts/0        Tue Mar 22 07:34 - 08:57  (01:23)     10.10.10.66
root     pts/0        Tue Mar 22 07:12 - 07:18  (00:06)     10.10.10.66
root     pts/0        Tue Mar 22 07:06 - 07:12  (00:06)     10.10.10.66
root     pts/0        Tue Mar 22 06:25 - 06:33  (00:07)     10.10.10.66
root     pts/0        Fri Mar 18 09:46 - 09:52  (00:05)     10.10.8.104
root     pts/0        Fri Mar 18 08:48 - 08:56  (00:07)     10.10.8.104
root     pts/0        Thu Mar 17 03:05 - 04:05  (01:00)     10.10.8.104
root     pts/0        Thu Mar 17 02:12 - 03:05  (00:52)     10.10.8.104
root     pts/0        Wed Mar 16 07:16 - 08:06  (00:49)     10.10.8.104
root     pts/0        Tue Mar 15 13:35 - 14:30  (00:55)     220.255.180.160
The 10.10.8.x and 10.10.10.x are legit ip address... I would need to check on the 220.x.x.x ip address. *panic mode ON*
 
Old 03-28-2016, 06:44 AM   #13
ButterflyMelissa
Senior Member
 
Registered: Nov 2007
Location: Somewhere on my hard drive...
Distribution: Manjaro
Posts: 2,766
Blog Entries: 23

Rep: Reputation: 411Reputation: 411Reputation: 411Reputation: 411Reputation: 411
Quote:
*panic mode ON*
Dont panic, absorb, silence your mind, become one with the battlefield...
If I may sugest, in light of this:
Quote:
You sure have a lot of active root logins, and I can see 2 IPs logged in from Singapore
...to find out how to remove rootkits too...just me (not that educated but extremely interested) venting some advice...
Next, indeed, do disallow root access over SSH...because:
Quote:
Obviously root should not ever be allowed to log in over SSH
Melissa
Edit - for this suggestion I stand open for critique as it is somewhat far fetched...open passwd (usualy found in /etc) and check if any account other than root has root rights...
Root is identified by group 0 (as I was told) in the image, entry 3

Last edited by ButterflyMelissa; 03-28-2016 at 06:56 AM.
 
Old 03-28-2016, 07:13 AM   #14
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
Quote:
Originally Posted by ButterflyMelissa View Post
...to find out how to remove rootkits too...
No. No. No: anything foreign injected into a system operating as root user shall not be "removed" or otherwise "cleaned up". That is not a security best practice.

Security is a continuous process requiring a layered approach. Any software in the web stack that is not updated, stale, unsupported or outright vulnerable is low hanging fruit. Each OS comes with installation, admin and security documentation and documentation of software in the web stack often has a specific security section. It's just that people often don't read much, don't care enough or outright neglect admin best practices. This is what you get.

Here the only approach is to set up a new server, harden it properly and then migrate data but only after thorough inspection.
 
1 members found this post helpful.
Old 03-28-2016, 07:48 AM   #15
ButterflyMelissa
Senior Member
 
Registered: Nov 2007
Location: Somewhere on my hard drive...
Distribution: Manjaro
Posts: 2,766
Blog Entries: 23

Rep: Reputation: 411Reputation: 411Reputation: 411Reputation: 411Reputation: 411
Quote:
No. No. No: anything foreign injected into a system operating as root user shall not be "removed" or otherwise "cleaned up". That is not a security best practice.
Lesson learned...
@ OP - please dont take my input a fool proof
Melissa
(looking on, with intense interest)
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
centos 5 machine plagued with 'mount error 13 = Permission denied' tonj Linux - Software 22 03-28-2015 05:32 AM
LXer: Delicious irony: iPhone 5S and iOS 7 plagued by Blue Screen of Death LXer Syndicated Linux News 0 10-12-2013 05:11 PM
[SOLVED] May have contracted malware. Yes, malware. Firefox on Ubuntu Fiesty. Seeking a fix drachenchen Linux - Security 22 08-17-2008 01:05 PM
May have contracted malware. Yes, malware. Firefox on Ubuntu Fiesty. Seeking a fix drachenchen Linux - Security 1 06-12-2008 05:10 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Security

All times are GMT -5. The time now is 08:48 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration