Being a paranoid (due to lack of knowledge in this field) geek I have already wiped the drive and re-installed the system in question. So unfortunately I will not be able to "test" or "look" at that systems log files anymore. Sorry for that. However the answer to my question(s) is/are good knowledge so I am still going to ask.

Scenario:

Web dev server:
Static IP on DMZ to the Internet. Root and two user accounts (user1 & user2). Https, http & ssh allowed. SELinux and firewall on. All updates applied.

Workstation:
user3 ssh into Web dev server to work on web dev files in user1 or user2 home directories
su to root to copy files from user1 or user2 to /var/www/html/ for testing before upload to hosting service.


One day I noticed two files (web pages) that were not the updated version they should have been. Thinking an update may have some how accidentally reverted them I just copied them back from the back up and continued working. A couple days later I ssh'd in again to update the same two files and again found them as the original. I felt this was out of place as there were no updates to the system or any other changes that I knew of. SELinux did not have any alerts or warnings for me, so I figured things were good for the most part. I checked last and did not see any odd logins. I checked lastb and found a CRAP load of ssh attempts for accounts not on the system. One attempt lasted 12 hours trying hundreds of names human and system names alike. So a ran lastb |grep user1 and found no entries. I ran lastb |grep user2 and found no entries. ROOT, root, r00t & R00T all had multiple failed attempts.

Finally the question!
Did they get in my system and I just don't know it?
Were they the ones who changed my files?
User1 & user2 did not have failed ssh attempts so were the failed entries deleted or did they not even try them since there was no lastb entry for them?
What else could have caused the files to be reverted to the originals?

It's my belief that if they are trying to ssh in and keep failing to guess the right user account and password combo then in theory they did not get in. Would that be correct?

This may sound a bit harsh but what you believe, think, worry about, think you think or guess does not matter as you wiped all "evidence". Now some members may be in for the thrill guessing games and wild speculation bring but I prefer to base any diagnosis on cold hard facts. The only consolation you have is you're in the presence of new as well as seasoned Linux users who all fell for the "let's wipe everything and hope it'll be cool anyway" reflex.

Fist of all, no it is not a bit hash, maybe not helpful! But not hash. After all on a Linux QUESTION site not a single one of my questions was answered! If this seems a bit hash delete my account.

“what you believe, think, worry about, think you think or guess does not matter “ ~unSpawn
Actually it does! What we call “cold hard facts” comes from what we “believe” to be true about something. It is our belief, suspicions & knowledge of what we know to be true that lead to computer security in the first place. After all if no one ever questioned or believed that the system was slowing down do to mass out bound connections we never would have figured out that viruses spread machine to machine. Get my point?

“I prefer to base any diagnosis on cold hard facts.” ~unSpawn
FACT!: Internet routable IP address (first mistake)
FACT!: mass amounts of failed login attempts with invalid usernames in /var/log/btmp
FACT!: actual user account names not listed in the /var/log/btmp
FACT!: files reverting to original state of creation and not maintaining updated status.
FACT!: system up to date
FACT!: firewall and Selinux on
FACT!: only open ports https, http & ssh
FACT!: no SELinux alerts
FACT!: to copy files from a user account to /var/www/html/* one must have elevated access or more than just user permissions

“The only consolation you have is you're in the presence of new as well as seasoned Linux users” ~unSpawn
For some one who does not like games, you seem to be promoting it by basing your opinion with out “cold hard facts” of the situation. I realize your upset that the evidence was destroyed and no further research can be done ( “unfortunately I will not be able to "test" or "look" at that systems log files anymore. Sorry for that.” ~W@M). However my situation was not one of “users who all fell for the "let's wipe everything and hope it'll be cool anyway" reflex.”~unSpawn, it was my only dev box (second mistake) and I was under a deadline to produce the pages that kept getting reverted. @55uming the worst of a compromised system I understand most compromised systems result in wiping the box. True usually after review of evidence. 30 minutes up and running again vs 3 days with no answers, yes I will destroy the evidence for that kind turn around.

I expecte seasoned Linux users would have some input on what to look for in the future or suggestions on what files to look at or something other than the useless comment you made that does not even reflect any form of moderation!

“Now some members may be in for the thrill guessing games and wild speculation bring”~unSpawn
I am not here for a guessing game. Yeah, okay I could have rephrased my questions. So, let me do that now.

“Did they get in my system and I just don't know it? “
How can you tell if some one is maintaining access to your box without a current active connection?

“Were they the ones who changed my files? “
How can you tell if a user account has changed a file if you do not recall the dates of creation or modification?

“User1 & user2 did not have failed ssh attempts so were the failed entries deleted or did they not even try them since there was no lastb entry for them? “
If a cracker found an active user account and is trying to use it for access would it show in the btmp log as a failed attempt?
If that failed attempt log was deleted wouldn't that mean they have access so that they could alter the log?

“What else could have caused the files to be reverted to the originals?”
Does ssh have caching? Could it have not saved the changes to the two files I worked on last?
FACT! Not Mentioned earlier: Only the last two files worked on were reverted to the original. All other files keep updated changes.

“It's my belief that if they are trying to ssh in and keep failing to guess the right user account and password combo then in theory they did not get in. Would that be correct?”
How can you tell if a cracker used a valid user account to gain access to your system?

Simply writing FACT! all over a reply does not promote incomplete generic details (from memory) to any kind of actionable response. It is merely an exercise now that you covered their tracks.

I read here... that unSpawn is one of the developers of rkhunter. I for one, choose to listen to him when he speaks. He knows his stuff.

It's a poor workman who blames his tools.

Thanks for the vote of confidence but I'm not infallible. Regardless of who writes what it's always good to check and double check.


No I don't. I'll give you credit for trying though.


Someone stole your pinata.


No.


Linux doesn't come with all-encompassing logging out of the box so generally speaking that leaves one at a minimum with correlating login records, shell history, log file entries.


No, UNIX doesn't have "creation time". The "c" stands for "change" meaning ownership, access permissions, etc. If DAC doesn't prohibit it then it'll be log correlation plus MAC times.


Generally speaking, and if configured correctly, services and the login process log failed login attempts to btmp and log files.


No, not necessarily. Output could be misinterpreted or the file could have been logrotated.


At a minimum log correlation. Again.

I thought I just "did" double-check? I didn't want to invade your privacy with a frivilous PM, so I just stuck it out there.

You're just below "walks on water" in my book.

Your skill is evident, even without an rkhunter 'tag'

You may want to change the "su to root" just to install files. I would suggest using sudo with a specific utility/script to copy the files to apache (and don't use the root account - instead have the script run as the apache user.

sudo can then be configured to log all legitimate updates, and you won't have to worry about a root login. And the script (since it is running as the apache user) cannot write anywhere that apache cannot write - and that protects the rest of the system.

Alternatively, you can set the ownership of the files to be someone other than apache, with apache only having group access. This gives additional protection because a breakin working through apache will only receive the apache credentials, and again, only have write access to files apache could write to in the first place; and not to the files being installed.

As to the your ultimate question of whether or not they got in, I can think of two reasons why an intruder would have stopped probing your SSH system: either they gave up, or they made a successful hit. By looking at lastlog and another log, such as secure (or wherever logins are recorded in your system), one can get a pretty good idea of whether or not they have been successful. Without this information, I know of no way to determine if they got in or not. The files reverting to an earlier revision is suspicious, but doesn't necessarily mean an intruder. In fact, this would be a rather unusual activity for your typical intruder. Chances are there is a more mundane answer.

Going forward, I would suggest a combination of things to improve your security posture:
1 - do not allow password SSH and use key based authentication.
2 - discourage brute force SSh attempts through use of a utility like fail2ban
3 - Since you mentioned the server being in a DMZ, requiring VPN access to even get to the SSH port would be a wise move.
4 - use logwatch to get daily email reports, which will include who logged in, from where, and any privilege escalations

1 members found this post helpful.

Take a deeeeep breath, W@m, and understand that, between Unspawn and Noway2 and the other "regulars" on this forum, your situation is in the best and most-knowledgeable hands possible. Anyway, you wiped the system so the evidence is gone. (Too bad you didn't remove that hard-drive completely and install a brand new one...) But as for how to keep whatever-happened from happening again, my cordial advice is: "ssshhhhh! Stop! It's gonna be OK! Now, Listen to these people!" If they don't know, it ain't worth knowin'.