Where to look for trojan file in ubuntu infected server?
 

Problem description: Noticed high CPU, htop shows 4 of 8 my cpu cores are 100% loaded. Htop not showing corect %load. htop/top were not showing PIDs which were running on 4 100% loaded cores.
Thanks to perf i identified pid which is occupying my 4 cores.
So, someone was hijacking my cpu, most probably for curency mining. Changed root passsword with 40 simbols, but intruder get again in my server. After some investigation found that intruder is deleting traces of his work in my server.
Found that he is saving my complex pass in /var/tmp/pam.log and probably sending the password via e-mail to intruder.
After I protected ssh port he was unable to get in.
I am novice with Linux and my questions for experts are:
1. Can you recognize what type of trojin I am dealing with?
2. The server is quarantined for learning purpose. Could you help me to find the file where the e-mail of the intruder is?
3. I am assuming he is doing keystroke logging using some Linux commands or scripts. Any idea where to look for those scripts?
Thank you

I have no answers to your questions, but on Ubuntu, sshd logs accesses to /var/log/auth.log by default. Worth checking there.

Check the files under /etc/pam.d for /var/tmp/pam.log.

If you suspect email, check your mail system's logs. It would, however, be very unusual that a trojan uses a method that leaves traces as obvious as email.

Of course, log files can be tampered with. An intrusion detection system can help you identify illegitimate accesses, but also suspicious outgoing network traffic. You can also use auditing (see, e.g., https://www.theurbanpenguin.com/inst...-ubuntu-18-04/) to find out which process creates and writes to /var/tmp/pam.log.

Your description of the (h)top reports is strange. If you can re-create the case, I would like to see evidence that the load is incorrect and that no running processes are shown. Which process used your CPUs?

In all my years in the business I have seen some malware exploits, but only ONE actual manned breakin that was somewhat successful. (Admin was smart financial guy, but not security trained and got tired of entering long passwords, so reset to use one that was short and easy to remember. Guess which account allowed the breaking!)

Once I isolated and recovered the system for examination, I discovered rootkits, reporting and keylogging malware, zombie worms, and several other tricks installed. We had a talk, and agreed that that system would require a complete replacement rebuild to recover. (Luckily, he DID always take the backups I had recommended on a regular basis. HE lost a couple of days, but no data).

Same in this case. If you are SURE your system was compromised, go to your backups, clean and rebuild your system, and restore the data you need. Never trust a system recovered in place or retaining any files from the suspected breakin event. It might have "sleepers" embedded somewhere.

I doubt that OP has done due diligence here. However, the server doesn't seem to do anything useful right now except that it is used for learning purposes.

I do wonder how OP achieved this:
in spite of this self-description:
In short, I question whether this system has really been under attack.

I'd just make a clean install and be sure to secure it better.

Well, obviously I am at wrong site. You are questioning my skills instead trying to help???
And obviously I am questioning your status "guru". Your questions, in your first post are like you are novice, because each novice first will check point you mentioned.
Novice means that Linux is very complex to say I am a guru. Even on Google you can find information about things I am talking about. For your info : https://www.trendmicro.com/vinfo/us/...it-for-stealth.
And yeas if you have just some db on server, yeah the easiest way is to reinstall it. But if the server is part of significant process then you have to find some other solutions.
Unless someone really points to the problem, not to me, I will stop discussion.
Take care

Your description is insufficient for pointing to any problems. I asked questions to clarify what's going on.

You have to describe what makes you think the server is infected and how you come to the conclusion that the attacker deletes traces, and which traces; which process uses the CPUs; whether /var/tmp/pam.log is used by any of the PAM config files in /etc/pam.d; what you find in /var/log/auth.log and the log files of the MTA that is installed on your server.

Since you seem to be hurt by my remark: I am not questioning your skills. If you say that you are a Linux novice, you are telling the world that you consider your Linux skills inferior. My Guru status is not something I have control over; anybody with 5000 posts in Linuxquestions.org becomes a "Guru" automatically. Joke about it as you like; I find it funny, too. Before 5000, I was a "Senior Member".

Whenever something like this is posted, and NONE of the additional information requested was provided, I think it's safe to say that the original request wasn't serious anyhow.
Let's just hope this one keeps their promise to not come back.

If you can't provide any proof, what do you think we'll be able to help with? You claim to be a novice...yet somehow seem to know about how to check the process table, and identify a PID....but don't find the program name too? Then say your server is a part of 'significant' process so you can't reload it....but also claim you quarantined the server. And if you're a novice, how and why did you get the job of managing many servers?

May want to check the MANY other threads about essentially the same thing. Also from novices.

There are programs like rkhunter that you can look into.

That is just funny. You need some help and you refused to communicate to the people wanted to help you. From my side it is ok, but as far as I see this is a serious problem.

If you really need help you need to give us more details, like:
1. how did you know that intruder is deleting traces of his work? (I'm really interested).
2. also I would like to see how did you check your CPU is hijacked.
These are very important informations if you wish to identify the intruder or trojan.

1. When i tried to login with putty a new popup screen appeared about new ssh key, which means somehow my old key was deleted. When I went to /var/log/auth.log noticed that the logging started a few hours ago. After isolating PID which is causing high cpu, with ps -efw PID to find when the PID started, noticed that the time is matching the auth logging started. Found my root pass in plain text, no encryption, in /var/tmp/pam.log whis telling me that intruder seems to be recording my pass as I am typing it, most probably keystroke recording.
2. Noticed that my server is slow, when open htop noticed that 4 cores are loaded each 100%, but there were no PIDs showing such load. htop was showing just about 30% load. No way that you can find those pid using regular Linux commands, check microtreend arical, the link I postedd above. I used perf record and in the report noticed directories and apps that actually do not exist on my server and soo on...
Well, I broke my promise because you are asking direct questions about issue for the learning purpose. And yes I am novice with ubuntu, but I work on other similar things so it is easy for me to learn, I just need ubuntu sintax, but I am short for tips and tricks.
I hope my answers helped you. From someone's post you see his level of knowledge/experience.Have no time to talk about me just about issues. From the reactions on my post I have concluded that my issue is for top Linux guys.
Take care

Thanks for the tip. Will do research on rkhunter first.

No, it's probably because you got a new IP address from your DHCP server. And amazing that a 'hacker' would leave all that for a new user to easily find
Except you said you DID get a PID earlier.
Right... except the commands for Ubuntu are the same as other linux systems. Probably best to seek help, definitely

Which is it? 100% or 30%? You are contradicting yourself.