LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 10-10-2020, 06:07 PM   #1
Ketmen
LQ Newbie
 
Registered: Oct 2020
Posts: 5

Rep: Reputation: Disabled
Where to look for trojan file in ubuntu infected server?


Problem description: Noticed high CPU, htop shows 4 of 8 my cpu cores are 100% loaded. Htop not showing corect %load. htop/top were not showing PIDs which were running on 4 100% loaded cores.
Thanks to perf i identified pid which is occupying my 4 cores.
So, someone was hijacking my cpu, most probably for curency mining. Changed root passsword with 40 simbols, but intruder get again in my server. After some investigation found that intruder is deleting traces of his work in my server.
Found that he is saving my complex pass in /var/tmp/pam.log and probably sending the password via e-mail to intruder.
After I protected ssh port he was unable to get in.
I am novice with Linux and my questions for experts are:
1. Can you recognize what type of trojin I am dealing with?
2. The server is quarantined for learning purpose. Could you help me to find the file where the e-mail of the intruder is?
3. I am assuming he is doing keystroke logging using some Linux commands or scripts. Any idea where to look for those scripts?
Thank you
 
Old 10-10-2020, 06:49 PM   #2
berndbausch
LQ Addict
 
Registered: Nov 2013
Location: Tokyo
Distribution: Mostly Ubuntu and Centos
Posts: 6,316

Rep: Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002
I have no answers to your questions, but on Ubuntu, sshd logs accesses to /var/log/auth.log by default. Worth checking there.

Check the files under /etc/pam.d for /var/tmp/pam.log.

If you suspect email, check your mail system's logs. It would, however, be very unusual that a trojan uses a method that leaves traces as obvious as email.

Of course, log files can be tampered with. An intrusion detection system can help you identify illegitimate accesses, but also suspicious outgoing network traffic. You can also use auditing (see, e.g., https://www.theurbanpenguin.com/inst...-ubuntu-18-04/) to find out which process creates and writes to /var/tmp/pam.log.

Your description of the (h)top reports is strange. If you can re-create the case, I would like to see evidence that the load is incorrect and that no running processes are shown. Which process used your CPUs?
 
Old 10-10-2020, 07:58 PM   #3
wpeckham
LQ Guru
 
Registered: Apr 2010
Location: Continental USA
Distribution: Debian, Ubuntu, RedHat, DSL, Puppy, CentOS, Knoppix, Mint-DE, Sparky, VSIDO, tinycore, Q4OS,Manjaro
Posts: 5,512

Rep: Reputation: 2657Reputation: 2657Reputation: 2657Reputation: 2657Reputation: 2657Reputation: 2657Reputation: 2657Reputation: 2657Reputation: 2657Reputation: 2657Reputation: 2657
In all my years in the business I have seen some malware exploits, but only ONE actual manned breakin that was somewhat successful. (Admin was smart financial guy, but not security trained and got tired of entering long passwords, so reset to use one that was short and easy to remember. Guess which account allowed the breaking!)

Once I isolated and recovered the system for examination, I discovered rootkits, reporting and keylogging malware, zombie worms, and several other tricks installed. We had a talk, and agreed that that system would require a complete replacement rebuild to recover. (Luckily, he DID always take the backups I had recommended on a regular basis. HE lost a couple of days, but no data).

Same in this case. If you are SURE your system was compromised, go to your backups, clean and rebuild your system, and restore the data you need. Never trust a system recovered in place or retaining any files from the suspected breakin event. It might have "sleepers" embedded somewhere.

Last edited by wpeckham; 10-10-2020 at 07:59 PM.
 
Old 10-10-2020, 09:40 PM   #4
berndbausch
LQ Addict
 
Registered: Nov 2013
Location: Tokyo
Distribution: Mostly Ubuntu and Centos
Posts: 6,316

Rep: Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002
Quote:
Originally Posted by wpeckham View Post
If you are SURE your system was compromised
I doubt that OP has done due diligence here. However, the server doesn't seem to do anything useful right now except that it is used for learning purposes.

I do wonder how OP achieved this:
Quote:
found that intruder is deleting traces of his work
in spite of this self-description:
Quote:
I am novice with Linux
In short, I question whether this system has really been under attack.
 
1 members found this post helpful.
Old 10-10-2020, 09:43 PM   #5
jefro
Moderator
 
Registered: Mar 2008
Posts: 21,939

Rep: Reputation: 3619Reputation: 3619Reputation: 3619Reputation: 3619Reputation: 3619Reputation: 3619Reputation: 3619Reputation: 3619Reputation: 3619Reputation: 3619Reputation: 3619
I'd just make a clean install and be sure to secure it better.
 
Old 10-10-2020, 10:05 PM   #6
Ketmen
LQ Newbie
 
Registered: Oct 2020
Posts: 5

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by berndbausch View Post
I doubt that OP has done due diligence here. However, the server doesn't seem to do anything useful right now except that it is used for learning purposes.

I do wonder how OP achieved this:

in spite of this self-description:

In short, I question whether this system has really been under attack.
Well, obviously I am at wrong site. You are questioning my skills instead trying to help???
And obviously I am questioning your status "guru". Your questions, in your first post are like you are novice, because each novice first will check point you mentioned.
Novice means that Linux is very complex to say I am a guru. Even on Google you can find information about things I am talking about. For your info : https://www.trendmicro.com/vinfo/us/...it-for-stealth.
And yeas if you have just some db on server, yeah the easiest way is to reinstall it. But if the server is part of significant process then you have to find some other solutions.
Unless someone really points to the problem, not to me, I will stop discussion.
Take care
 
Old 10-11-2020, 12:27 AM   #7
berndbausch
LQ Addict
 
Registered: Nov 2013
Location: Tokyo
Distribution: Mostly Ubuntu and Centos
Posts: 6,316

Rep: Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002
Your description is insufficient for pointing to any problems. I asked questions to clarify what's going on.

You have to describe what makes you think the server is infected and how you come to the conclusion that the attacker deletes traces, and which traces; which process uses the CPUs; whether /var/tmp/pam.log is used by any of the PAM config files in /etc/pam.d; what you find in /var/log/auth.log and the log files of the MTA that is installed on your server.

Since you seem to be hurt by my remark: I am not questioning your skills. If you say that you are a Linux novice, you are telling the world that you consider your Linux skills inferior. My Guru status is not something I have control over; anybody with 5000 posts in Linuxquestions.org becomes a "Guru" automatically. Joke about it as you like; I find it funny, too. Before 5000, I was a "Senior Member".
 
1 members found this post helpful.
Old 10-11-2020, 03:11 AM   #8
ondoho
LQ Addict
 
Registered: Dec 2013
Posts: 19,872
Blog Entries: 12

Rep: Reputation: 6053Reputation: 6053Reputation: 6053Reputation: 6053Reputation: 6053Reputation: 6053Reputation: 6053Reputation: 6053Reputation: 6053Reputation: 6053Reputation: 6053
Quote:
Originally Posted by Ketmen View Post
Well, obviously I am at wrong site. You are questioning my skills instead trying to help???
And obviously I am questioning your status "guru". Your questions, in your first post are like you are novice, because each novice first will check point you mentioned.
Novice means that Linux is very complex to say I am a guru. Even on Google you can find information about things I am talking about. For your info : https://www.trendmicro.com/vinfo/us/...it-for-stealth.
And yeas if you have just some db on server, yeah the easiest way is to reinstall it. But if the server is part of significant process then you have to find some other solutions.
Unless someone really points to the problem, not to me, I will stop discussion.
Take care
Whenever something like this is posted, and NONE of the additional information requested was provided, I think it's safe to say that the original request wasn't serious anyhow.
Let's just hope this one keeps their promise to not come back.
 
3 members found this post helpful.
Old 10-11-2020, 08:48 AM   #9
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 26,553

Rep: Reputation: 7946Reputation: 7946Reputation: 7946Reputation: 7946Reputation: 7946Reputation: 7946Reputation: 7946Reputation: 7946Reputation: 7946Reputation: 7946Reputation: 7946
Quote:
Originally Posted by Ketmen View Post
Well, obviously I am at wrong site. You are questioning my skills instead trying to help???
And obviously I am questioning your status "guru". Your questions, in your first post are like you are novice, because each novice first will check point you mentioned.
Novice means that Linux is very complex to say I am a guru. Even on Google you can find information about things I am talking about. For your info : https://www.trendmicro.com/vinfo/us/...it-for-stealth.
And yeas if you have just some db on server, yeah the easiest way is to reinstall it. But if the server is part of significant process then you have to find some other solutions.
Unless someone really points to the problem, not to me, I will stop discussion.
Take care
If you can't provide any proof, what do you think we'll be able to help with? You claim to be a novice...yet somehow seem to know about how to check the process table, and identify a PID....but don't find the program name too? Then say your server is a part of 'significant' process so you can't reload it....but also claim you quarantined the server. And if you're a novice, how and why did you get the job of managing many servers?

May want to check the MANY other threads about essentially the same thing. Also from novices.
 
Old 10-11-2020, 09:35 AM   #10
dugan
LQ Guru
 
Registered: Nov 2003
Location: Canada
Distribution: distro hopper
Posts: 11,200

Rep: Reputation: 5307Reputation: 5307Reputation: 5307Reputation: 5307Reputation: 5307Reputation: 5307Reputation: 5307Reputation: 5307Reputation: 5307Reputation: 5307Reputation: 5307
There are programs like rkhunter that you can look into.
 
Old 10-11-2020, 10:01 AM   #11
pan64
LQ Addict
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 21,692

Rep: Reputation: 7274Reputation: 7274Reputation: 7274Reputation: 7274Reputation: 7274Reputation: 7274Reputation: 7274Reputation: 7274Reputation: 7274Reputation: 7274Reputation: 7274
Quote:
Originally Posted by Ketmen View Post
Unless someone really points to the problem, not to me, I will stop discussion.
That is just funny. You need some help and you refused to communicate to the people wanted to help you. From my side it is ok, but as far as I see this is a serious problem.

If you really need help you need to give us more details, like:
1. how did you know that intruder is deleting traces of his work? (I'm really interested).
2. also I would like to see how did you check your CPU is hijacked.
These are very important informations if you wish to identify the intruder or trojan.
 
1 members found this post helpful.
Old 10-11-2020, 11:19 AM   #12
Ketmen
LQ Newbie
 
Registered: Oct 2020
Posts: 5

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by pan64 View Post
That is just funny. You need some help and you refused to communicate to the people wanted to help you. From my side it is ok, but as far as I see this is a serious problem.

If you really need help you need to give us more details, like:
1. how did you know that intruder is deleting traces of his work? (I'm really interested).
2. also I would like to see how did you check your CPU is hijacked.
These are very important informations if you wish to identify the intruder or trojan.
1. When i tried to login with putty a new popup screen appeared about new ssh key, which means somehow my old key was deleted. When I went to /var/log/auth.log noticed that the logging started a few hours ago. After isolating PID which is causing high cpu, with ps -efw PID to find when the PID started, noticed that the time is matching the auth logging started. Found my root pass in plain text, no encryption, in /var/tmp/pam.log whis telling me that intruder seems to be recording my pass as I am typing it, most probably keystroke recording.
2. Noticed that my server is slow, when open htop noticed that 4 cores are loaded each 100%, but there were no PIDs showing such load. htop was showing just about 30% load. No way that you can find those pid using regular Linux commands, check microtreend arical, the link I postedd above. I used perf record and in the report noticed directories and apps that actually do not exist on my server and soo on...
Well, I broke my promise because you are asking direct questions about issue for the learning purpose. And yes I am novice with ubuntu, but I work on other similar things so it is easy for me to learn, I just need ubuntu sintax, but I am short for tips and tricks.
I hope my answers helped you. From someone's post you see his level of knowledge/experience.Have no time to talk about me just about issues. From the reactions on my post I have concluded that my issue is for top Linux guys.
Take care
 
Old 10-11-2020, 11:48 AM   #13
Ketmen
LQ Newbie
 
Registered: Oct 2020
Posts: 5

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by dugan View Post
There are programs like rkhunter that you can look into.
Thanks for the tip. Will do research on rkhunter first.
 
Old 10-11-2020, 12:19 PM   #14
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 26,553

Rep: Reputation: 7946Reputation: 7946Reputation: 7946Reputation: 7946Reputation: 7946Reputation: 7946Reputation: 7946Reputation: 7946Reputation: 7946Reputation: 7946Reputation: 7946
Quote:
Originally Posted by Ketmen View Post
1. When i tried to login with putty a new popup screen appeared about new ssh key, which means somehow my old key was deleted. When I went to /var/log/auth.log noticed that the logging started a few hours ago. After isolating PID which is causing high cpu, with ps -efw PID to find when the PID started, noticed that the time is matching the auth logging started. Found my root pass in plain text, no encryption, in /var/tmp/pam.log whis telling me that intruder seems to be recording my pass as I am typing it, most probably keystroke recording.
No, it's probably because you got a new IP address from your DHCP server. And amazing that a 'hacker' would leave all that for a new user to easily find
Quote:
2. Noticed that my server is slow, when open htop noticed that 4 cores are loaded each 100%, but there were no PIDs showing such load. htop was showing just about 30% load. No way that you can find those pid using regular Linux commands, check microtreend arical, the link I postedd above. I used perf record and in the report noticed directories and apps that actually do not exist on my server and soo on...
Except you said you DID get a PID earlier.
Quote:
Well, I broke my promise because you are asking direct questions about issue for the learning purpose. And yes I am novice with ubuntu, but I work on other similar things so it is easy for me to learn, I just need ubuntu sintax, but I am short for tips and tricks.
I hope my answers helped you. From someone's post you see his level of knowledge/experience.Have no time to talk about me just about issues. From the reactions on my post I have concluded that my issue is for top Linux guys.
Take care
Right... except the commands for Ubuntu are the same as other linux systems. Probably best to seek help, definitely
 
Old 10-11-2020, 12:27 PM   #15
sgosnell
Senior Member
 
Registered: Jan 2008
Location: Baja Oklahoma
Distribution: Debian Stable and Unstable
Posts: 1,943

Rep: Reputation: 542Reputation: 542Reputation: 542Reputation: 542Reputation: 542Reputation: 542
Quote:
when open htop noticed that 4 cores are loaded each 100%, but there were no PIDs showing such load. htop was showing just about 30% load.
Which is it? 100% or 30%? You are contradicting yourself.
 
1 members found this post helpful.
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Will my dvd/cd get infected if I put it into an infected computer? kangaroobop4 Linux - Newbie 16 09-30-2020 03:38 PM
RAT infected files (remote access tool - remote access trojan) - corrupt? jettjett Linux - Newbie 16 03-20-2018 10:07 PM
LXer: Powerful, highly stealthy Linux trojan may have infected victims for years LXer Syndicated Linux News 0 12-09-2014 11:31 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 08:12 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration