LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices

Reply
 
Search this Thread
Old 12-07-2012, 05:06 PM   #1
w@m
LQ Newbie
 
Registered: Jun 2007
Location: Bay Area
Distribution: RedHat, Fedora and CentOS. Since 2004
Posts: 10

Rep: Reputation: 0
Question n00b wtmp / Security Breach question


Being a paranoid (due to lack of knowledge in this field) geek I have already wiped the drive and re-installed the system in question. So unfortunately I will not be able to "test" or "look" at that systems log files anymore. Sorry for that. However the answer to my question(s) is/are good knowledge so I am still going to ask.

Scenario:

Web dev server:
Static IP on DMZ to the Internet. Root and two user accounts (user1 & user2). Https, http & ssh allowed. SELinux and firewall on. All updates applied.

Workstation:
user3 ssh into Web dev server to work on web dev files in user1 or user2 home directories
su to root to copy files from user1 or user2 to /var/www/html/ for testing before upload to hosting service.


One day I noticed two files (web pages) that were not the updated version they should have been. Thinking an update may have some how accidentally reverted them I just copied them back from the back up and continued working. A couple days later I ssh'd in again to update the same two files and again found them as the original. I felt this was out of place as there were no updates to the system or any other changes that I knew of. SELinux did not have any alerts or warnings for me, so I figured things were good for the most part. I checked last and did not see any odd logins. I checked lastb and found a CRAP load of ssh attempts for accounts not on the system. One attempt lasted 12 hours trying hundreds of names human and system names alike. So a ran lastb |grep user1 and found no entries. I ran lastb |grep user2 and found no entries. ROOT, root, r00t & R00T all had multiple failed attempts.

Finally the question!
Did they get in my system and I just don't know it?
Were they the ones who changed my files?
User1 & user2 did not have failed ssh attempts so were the failed entries deleted or did they not even try them since there was no lastb entry for them?
What else could have caused the files to be reverted to the originals?

It's my belief that if they are trying to ssh in and keep failing to guess the right user account and password combo then in theory they did not get in. Would that be correct?
 
Old 12-07-2012, 09:47 PM   #2
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,666
Blog Entries: 54

Rep: Reputation: 2953Reputation: 2953Reputation: 2953Reputation: 2953Reputation: 2953Reputation: 2953Reputation: 2953Reputation: 2953Reputation: 2953Reputation: 2953Reputation: 2953
Quote:
Originally Posted by w@m View Post
(..) I have already wiped the drive (..) Did they get in my system (..) It's my belief that (..)
This may sound a bit harsh but what you believe, think, worry about, think you think or guess does not matter as you wiped all "evidence". Now some members may be in for the thrill guessing games and wild speculation bring but I prefer to base any diagnosis on cold hard facts. The only consolation you have is you're in the presence of new as well as seasoned Linux users who all fell for the "let's wipe everything and hope it'll be cool anyway" reflex.
 
Old 12-09-2012, 02:27 PM   #3
w@m
LQ Newbie
 
Registered: Jun 2007
Location: Bay Area
Distribution: RedHat, Fedora and CentOS. Since 2004
Posts: 10

Original Poster
Rep: Reputation: 0
Thumbs down

Quote:
Originally Posted by unSpawn View Post
This may sound a bit harsh but what you believe, think, worry about, think you think or guess does not matter as you wiped all "evidence". Now some members may be in for the thrill guessing games and wild speculation bring but I prefer to base any diagnosis on cold hard facts. The only consolation you have is you're in the presence of new as well as seasoned Linux users who all fell for the "let's wipe everything and hope it'll be cool anyway" reflex.
Fist of all, no it is not a bit hash, maybe not helpful! But not hash. After all on a Linux QUESTION site not a single one of my questions was answered! If this seems a bit hash delete my account.

“what you believe, think, worry about, think you think or guess does not matter “ ~unSpawn
Actually it does! What we call “cold hard facts” comes from what we “believe” to be true about something. It is our belief, suspicions & knowledge of what we know to be true that lead to computer security in the first place. After all if no one ever questioned or believed that the system was slowing down do to mass out bound connections we never would have figured out that viruses spread machine to machine. Get my point?

“I prefer to base any diagnosis on cold hard facts.” ~unSpawn
FACT!: Internet routable IP address (first mistake)
FACT!: mass amounts of failed login attempts with invalid usernames in /var/log/btmp
FACT!: actual user account names not listed in the /var/log/btmp
FACT!: files reverting to original state of creation and not maintaining updated status.
FACT!: system up to date
FACT!: firewall and Selinux on
FACT!: only open ports https, http & ssh
FACT!: no SELinux alerts
FACT!: to copy files from a user account to /var/www/html/* one must have elevated access or more than just user permissions

“The only consolation you have is you're in the presence of new as well as seasoned Linux users” ~unSpawn
For some one who does not like games, you seem to be promoting it by basing your opinion with out “cold hard facts” of the situation. I realize your upset that the evidence was destroyed and no further research can be done ( “unfortunately I will not be able to "test" or "look" at that systems log files anymore. Sorry for that.” ~W@M). However my situation was not one of “users who all fell for the "let's wipe everything and hope it'll be cool anyway" reflex.”~unSpawn, it was my only dev box (second mistake) and I was under a deadline to produce the pages that kept getting reverted. @55uming the worst of a compromised system I understand most compromised systems result in wiping the box. True usually after review of evidence. 30 minutes up and running again vs 3 days with no answers, yes I will destroy the evidence for that kind turn around.

I expecte seasoned Linux users would have some input on what to look for in the future or suggestions on what files to look at or something other than the useless comment you made that does not even reflect any form of moderation!

“Now some members may be in for the thrill guessing games and wild speculation bring”~unSpawn
I am not here for a guessing game. Yeah, okay I could have rephrased my questions. So, let me do that now.

“Did they get in my system and I just don't know it? “
How can you tell if some one is maintaining access to your box without a current active connection?

“Were they the ones who changed my files? “
How can you tell if a user account has changed a file if you do not recall the dates of creation or modification?

“User1 & user2 did not have failed ssh attempts so were the failed entries deleted or did they not even try them since there was no lastb entry for them? “
If a cracker found an active user account and is trying to use it for access would it show in the btmp log as a failed attempt?
If that failed attempt log was deleted wouldn't that mean they have access so that they could alter the log?

“What else could have caused the files to be reverted to the originals?”
Does ssh have caching? Could it have not saved the changes to the two files I worked on last?
FACT! Not Mentioned earlier: Only the last two files worked on were reverted to the original. All other files keep updated changes.

“It's my belief that if they are trying to ssh in and keep failing to guess the right user account and password combo then in theory they did not get in. Would that be correct?”
How can you tell if a cracker used a valid user account to gain access to your system?
 
Old 12-09-2012, 05:05 PM   #4
Habitual
Senior Member
 
Registered: Jan 2011
Distribution: Undecided
Posts: 3,618
Blog Entries: 1

Rep: Reputation: Disabled
Quote:
...“I prefer to base any diagnosis on cold hard facts.” ~unSpawn
...
FACT!: to copy files from a user account to /var/www/html/* one must have elevated access or more than just user permissions.
Simply writing FACT! all over a reply does not promote incomplete generic details (from memory) to any kind of actionable response. It is merely an exercise now that you covered their tracks.

I read here... that unSpawn is one of the developers of rkhunter. I for one, choose to listen to him when he speaks. He knows his stuff.

It's a poor workman who blames his tools.
 
Old 12-09-2012, 08:56 PM   #5
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,666
Blog Entries: 54

Rep: Reputation: 2953Reputation: 2953Reputation: 2953Reputation: 2953Reputation: 2953Reputation: 2953Reputation: 2953Reputation: 2953Reputation: 2953Reputation: 2953Reputation: 2953
Quote:
Originally Posted by Habitual View Post
He knows his stuff.
Thanks for the vote of confidence but I'm not infallible. Regardless of who writes what it's always good to check and double check.


Quote:
Originally Posted by w@m View Post
Get my point?
No I don't. I'll give you credit for trying though.


Quote:
Originally Posted by w@m View Post
or something other than the useless comment you made that does not even reflect any form of moderation!
Someone stole your pinata.


Quote:
Originally Posted by w@m View Post
Does ssh have caching?
No.


Quote:
Originally Posted by w@m View Post
How can you tell if some one is maintaining access to your box without a current active connection?
Linux doesn't come with all-encompassing logging out of the box so generally speaking that leaves one at a minimum with correlating login records, shell history, log file entries.


Quote:
Originally Posted by w@m View Post
How can you tell if a user account has changed a file if you do not recall the dates of creation or modification?
No, UNIX doesn't have "creation time". The "c" stands for "change" meaning ownership, access permissions, etc. If DAC doesn't prohibit it then it'll be log correlation plus MAC times.


Quote:
Originally Posted by w@m View Post
If a cracker found an active user account and is trying to use it for access would it show in the btmp log as a failed attempt?
Generally speaking, and if configured correctly, services and the login process log failed login attempts to btmp and log files.


Quote:
Originally Posted by w@m View Post
If that failed attempt log was deleted wouldn't that mean they have access so that they could alter the log?
No, not necessarily. Output could be misinterpreted or the file could have been logrotated.


Quote:
Originally Posted by w@m View Post
How can you tell if a cracker used a valid user account to gain access to your system?
At a minimum log correlation. Again.
 
Old 12-10-2012, 11:32 AM   #6
Habitual
Senior Member
 
Registered: Jan 2011
Distribution: Undecided
Posts: 3,618
Blog Entries: 1

Rep: Reputation: Disabled
Quote:
Originally Posted by unSpawn View Post
...Regardless of who writes what it's always good to check and double check....
I thought I just "did" double-check? I didn't want to invade your privacy with a frivilous PM, so I just stuck it out there.

Quote:
Originally Posted by unSpawn View Post
...I'm not infallible....
You're just below "walks on water" in my book.

Your skill is evident, even without an rkhunter 'tag'
 
Old 12-11-2012, 12:43 PM   #7
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 2,325

Rep: Reputation: 593Reputation: 593Reputation: 593Reputation: 593Reputation: 593Reputation: 593
You may want to change the "su to root" just to install files. I would suggest using sudo with a specific utility/script to copy the files to apache (and don't use the root account - instead have the script run as the apache user.

sudo can then be configured to log all legitimate updates, and you won't have to worry about a root login. And the script (since it is running as the apache user) cannot write anywhere that apache cannot write - and that protects the rest of the system.

Alternatively, you can set the ownership of the files to be someone other than apache, with apache only having group access. This gives additional protection because a breakin working through apache will only receive the apache credentials, and again, only have write access to files apache could write to in the first place; and not to the files being installed.

Last edited by jpollard; 12-11-2012 at 12:45 PM.
 
Old 12-12-2012, 09:22 AM   #8
Noway2
Senior Member
 
Registered: Jul 2007
Distribution: Ubuntu 10.10, Slackware 64-current
Posts: 2,124

Rep: Reputation: 776Reputation: 776Reputation: 776Reputation: 776Reputation: 776Reputation: 776Reputation: 776
As to the your ultimate question of whether or not they got in, I can think of two reasons why an intruder would have stopped probing your SSH system: either they gave up, or they made a successful hit. By looking at lastlog and another log, such as secure (or wherever logins are recorded in your system), one can get a pretty good idea of whether or not they have been successful. Without this information, I know of no way to determine if they got in or not. The files reverting to an earlier revision is suspicious, but doesn't necessarily mean an intruder. In fact, this would be a rather unusual activity for your typical intruder. Chances are there is a more mundane answer.

Going forward, I would suggest a combination of things to improve your security posture:
1 - do not allow password SSH and use key based authentication.
2 - discourage brute force SSh attempts through use of a utility like fail2ban
3 - Since you mentioned the server being in a DMZ, requiring VPN access to even get to the SSH port would be a wise move.
4 - use logwatch to get daily email reports, which will include who logged in, from where, and any privilege escalations
 
1 members found this post helpful.
Old 12-12-2012, 06:44 PM   #9
sundialsvcs
Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 5,452

Rep: Reputation: 1172Reputation: 1172Reputation: 1172Reputation: 1172Reputation: 1172Reputation: 1172Reputation: 1172Reputation: 1172Reputation: 1172
Take a deeeeep breath, W@m, and understand that, between Unspawn and Noway2 and the other "regulars" on this forum, your situation is in the best and most-knowledgeable hands possible. Anyway, you wiped the system so the evidence is gone. (Too bad you didn't remove that hard-drive completely and install a brand new one...) But as for how to keep whatever-happened from happening again, my cordial advice is: "ssshhhhh! Stop! It's gonna be OK! Now, Listen to these people!" If they don't know, it ain't worth knowin'.

Last edited by sundialsvcs; 12-12-2012 at 06:45 PM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Yet another thread about a security breach Fredde87 Linux - Security 19 10-16-2009 09:12 AM
[SOLVED] possible security breach johnh10000 Linux - Security 18 10-13-2009 12:23 PM
[Security Questions] Last Login, how good is this feature for security breach info? t3gah Linux - Security 2 06-14-2005 02:02 AM
Security breach? lhoff Linux - Security 5 02-15-2002 02:33 AM


All times are GMT -5. The time now is 03:01 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration