How do I know I've been breached?
 

I searched through the invalid ssh login thread but I could not find anything about how to tell if these dictionary attacks have been working against my server. I have looked into the logs, but I am pretty new to this so I don't think I can tell what is "normal" server behavior and what is not. Is there a sure-fire way to tell if someone has broken into your server? This is a personal server I am using for development so the good news is I only have one user (me) and the root account, and I have changed both of those passwords and the user name. Here is a sample of the error I see in the log:
"***sshd[8552]: Invalid user maruyama from 193.15.55.4"

Also, I try to report what I can on these to their host, but it is a real pain to trace them back to the right person.

Any help would be greatly appreciated!

Yes, comparing old checksums of files with current ones. That's 100% sure if you trust the hashing methods.
Several integrity systems are available in Linux.

Otherwise, theorically, noway to be sure.

I agree; if a person is able to break into your system, then that person usually knows it's not wise to leave traces behind, so probably most of the "traces" that you would hope to find are cleaned up by the time you start looking for them, if your system happened to be breached, so probably the only method that is half way sure are checksums; that means, though, that you can't store the checksums anywhere where they are accessible from the server (or any other place on the network), because if somebody got to know where they are, they could be re-created to try to fool you.

There is a lot of software made to ease your task of making sure nothing is altered; tripwire is one of them. The thumb rule is to never trust, but always think your system is not secure. If somebody was interested enough in breaking into the system that person could do it, because there is (and probably is never going to be) no way to make (for example) a server bullet-proof. All you can do is hope you notice what's going on before it's "too late", pull the server off the network and start fixing it. Take a look at tripwire for example and start searching around for software for your needs.

If there was a simple method to see, by reading logfiles, if somebody is really trying or has succeeded in breaching a system, then it would already be automated and wouldn't happen anymore.

The first thing that comes to mind is to take a look at two logs on the server.

/var/log/secure
/var/log/messages

You will be able to see attempts from user to log into the server and failing. Also run the last command it will tell you who logged in, from what IP and when. Compare the two if you see an unknown IP in last's output look in the users directory and look for the IP in the logs and see if you can see when they broke in and to who's account.

One of the things you should have done before putting this on the net is installed a file integrity checker like Aide, Samhain or Tripwire. If a file being monitored is changed, those programs will let you know about it.

I'd say check your system logs, but you shouldn't trust them totally, as rootkits usually wipe log traces of intrusions. Rootkits also replace certain tools with hacked versions that are designed to hide activity that would be logged, so that if you use a hacked version of, say, lsof, you won't see what you need to see to investigate a compromise. In this case, it is probably best to use investigative tools from a trusted source (the tools would best be run from a CD-R).

I've no real faith in HIDS, but my experience with them aren't the best. If your machine is hacked, NOTHING on that machine is trustworthy, event a HIDS. TCPDump goes a LONG way, as it is a NIDS...it will see anything going over the wire, such as backdoor traffic.

I've two NIDS placed on my network. One monitors internal traffic and the other monitors traffic before my LAN. One other monitors my public colo server, which dumps its logs to my home machine's IDS server. I can access traffic logs of all three servers from the IDS server. The thing with NIDS is that they are rule-based...if you don't do periodical updates, you're not going to be using the latest/greatest rules.

Is there a sure-fire way to tell if someone has broken into your server?
Basically you follow the steps as outlined for any breach of security we've handled over the years in this forum. Maybe check out a few threads to get the idea. Run your audit tools (from Live CD if in doubt) and follow this guideline: Intruder Detection Checklist (CERT): http://www.cert.org/tech_tips/intrud...checklist.html.


Also, I try to report what I can on these to their host, but it is a real pain to trace them back to the right person.
A laudable approach but a Sysiphus task still. If you still choose to report, best report to the abuse@ address. Notice a lot of ISP's dump those reports due to whatever reasons valid or not. Automated reporting like Dshield and Mywatchmen are more favourable since they have a global scope which benefits everyone.


If you determined it's a true breach of security post your findings here, if not head over to the LQ FAQ: Security references and see if you should change or add to your boxen's security posture.

Some Thoughts
 

An old buddy of mine posted this on a now-gone website:

Assuming your system uses /etc/profile.d to store scripts to be run on login, do the following:

vi /etc/profile.d/loginreport.sh
#and add the command
mail -s "$USER Login: $HOST $SSH_CONNECTION" user@domain.com < /dev/null > /dev/null
#and change permissions on the newly created file
chmod 755 /etc/profile.d/loginreport.sh

This will send out an email to the specified adddress (user@domain.com - preferrably not an address on the local machine) anytime an ssh connection is successful.

To protect from the dictionary attacks, I recommend APF/BFD which will auto-block IP's afer a specified number of unsuccessful attempts. Use the following links to get started:

http://www.dedicated-resources.com/g...Firewall).html
http://www.dedicated-resources.com/g...etection).html


Deion "Mule" Christopher

This will send out an email to the specified adddress (user@domain.com - preferrably not an address on the local machine) anytime an ssh connection is successful.
What this does is send out an email any time /etc/profile.d/loginreport.sh is run: the script doesn't check for type of login. Other than that if it's run it'll be too late since the intruder already logged in, so beefing up on Firewall, host and SSH security should be done before implementing any "logging" like this. Most SSH "blockers" where mentioned ages ago in http://www.linuxquestions.org/questi...d.php?t=340366.

"Other than that if it's run it'll be too late since the intruder already logged in, so beefing up on Firewall, host and SSH security should be done before implementing any "logging" like this. Most SSH "blockers" where mentioned ages ago in http://www.linuxquestions.org/questi...d.php?t=340366."

Your quite right. That is why I also posted the links for APF and BFD in my reply. I should have included a link for rkhunter as well though.

I'm not sure I'm following your comment unSpawn: The originator asked, "Is there a sure-fire way to tell if someone has broken into your server?" to which I suggested the mail command above.

Good point though - once you receive the e-mail about an ssh connection, it is too late. But you will at least have an IP to use as a lead...

Deion "Mule" Christopher

I'm not sure I'm following your comment unSpawn
Sorry, which comment are you referring to I should clarify?


But you will at least have an IP to use as a lead.
For anyone using a box they own that's helpful. For those using one or more intermediaries, not really: the trail ends there.

geoffmat, the suggestions in this thread are good, and in particular I concur with the HIDS (I like aide, personally) recommendation. Be sure to keep your HIDS db on an external device / server.

Also important: This may be a good time to reevaluate the sshd service you're providing. Who are you providing sshd access to? Can it be restricted to certain networks / users? Does it really need to be open to the world?

a. Do not allow root login for ssh. Modify your /etc/ssh/sshd_config file:
b. Close any unnecessary ports.
c. Watch who is logged on your computer (man who).
d. Watch any connection with state "ESTABLISHED" and user "0" (root).
e. Remember: Trust is a weakness.

Might I add to what Road map suggested?

If you want to move to the paranoia range of things, set up ssh to allow "a single, normal user in the wheel group" to ssh in ONLY from a specific machine inside your internal network. Then set up that chosen machine for remote access (XP use RDP/VNC | Linux use ssh/vnc). This will produce two security hurdles to overcome to access your box...

Deion "Mule" Christopher