[SOLVED] Not really sure what is going on, possibly compromised?
Linux - SecurityThis forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
Not really sure what is going on, possibly compromised?
Ok, long story short, I created an SSH server to interface with my intranetwork, I was an idiot and made the password pretty simple, a couple weeks after booting up the SSH server I decided to browse my logs and there were thousands of failed password attempts in my auth.log file. After some research I setup some basic IPtables rules and the failed password attempts almost completely went away.
-N SSHATTACK
-A SSHATTACK -j LOG --log-prefix "Possible SSH Attack! " --log-level 7
-A SSHATTACK -j DROP
-A INPUT -i wlan0 -p tcp -m state --dport 22 --state NEW -m recent --set
-A INPUT -i wlan0 -p tcp -m state --dport 22 --state NEW -m recent --update --seconds 120 --hitcount 2 -j SSHATTACK
So this worked, at least it seemed, I set up the rules to block incoming traffic into port 22 and only allow for one password attempt before blocking the IP address. Problem is, I'm noticing some strange activity in the syslog file, I'm still getting hundreds of warnings from individual IP addresses... How is this possible???
I've got at least 200 "Possible SSH Attack!" log warnings from this single IP address, with hundreds more from a small list of other IP addresses as posted above. Here's whats got so confused, there are no correlating "Failed password" attempts in the auth.log file. The IP addresses in the syslog file that are attempting hundreds of passwords every minute don't even show up in the auth.log whatsoever.
My thoughts are as follows...
Someone has access to my system and the syslog is logging command inputs as SSH traffic from an IP address that should be blocked, therefor its not logged as a failed password attempt in auth.log.
---OR---
This is something totally benign and I have nothing to worry about but I've got a lot to learn about how linux works
Either way I'm confused, and any help would be appreciated.
I vote for confused Let me recap what I think you said:
1. You observed many failed login attempts on port 22
2. You added iptable rules to a. Log the attempts and b. DROP the connection.
3. You no longer see failed login attempts (because you've dropped them!)
4. You see many log entries about the dropped connections
In short, you dropped the connections and configured to log the drops. You no longer see connection attempts, but you now see the log entries you configured.
Did I get that right?
It looks to me like your iptables rule is doing what you wanted configured it to do.
Ok, long story short, I created an SSH server to interface with my intranetwork, I was an idiot and made the password pretty simple, a couple weeks after booting up the SSH server I decided to browse my logs and there were thousands of failed password attempts in my auth.log file. After some research I setup some basic IPtables rules and the failed password attempts almost completely went away.
Code:
-N SSHATTACK
-A SSHATTACK -j LOG --log-prefix "Possible SSH Attack! " --log-level 7
-A SSHATTACK -j DROP
-A INPUT -i wlan0 -p tcp -m state --dport 22 --state NEW -m recent --set
-A INPUT -i wlan0 -p tcp -m state --dport 22 --state NEW -m recent --update --seconds 120 --hitcount 2 -j SSHATTACK
So this worked, at least it seemed, I set up the rules to block incoming traffic into port 22 and only allow for one password attempt before blocking the IP address. Problem is, I'm noticing some strange activity in the syslog file, I'm still getting hundreds of warnings from individual IP addresses... How is this possible???
I've got at least 200 "Possible SSH Attack!" log warnings from this single IP address, with hundreds more from a small list of other IP addresses as posted above. Here's whats got so confused, there are no correlating "Failed password" attempts in the auth.log file. The IP addresses in the syslog file that are attempting hundreds of passwords every minute don't even show up in the auth.log whatsoever.
My thoughts are as follows...
Someone has access to my system and the syslog is logging command inputs as SSH traffic from an IP address that should be blocked, therefor its not logged as a failed password attempt in auth.log.
---OR---
This is something totally benign and I have nothing to worry about but I've got a lot to learn about how linux works
Since your box was exposed, it's probably on a list-of-things-to-break-in-to, and they keep banging on it. If you are dropping that IP address, that's why you're not seeing any further "failed password" attempts.
If you're new to this, I'd suggest a few things to do immediately:
Change SSH to be a different port, something other than 22
DO NOT expose SSH to the outside world, ever. Set up a VPN (which isn't as daunting as you'd think), and use it, and only SSH over that VPN connection.
If you're using a firewall appliance, get it checked immediately too.
If you're more comfortable with Windows/Mac, and are using those, get THOSE systems between your Linux system and the Internet. While Linux is more capable, in the time it takes you to get things locked down, damage may be done. Address the problems first with something you know.
If you're not comfortable with doing such things, hire someone. Seriously...security isn't something to take lightly, and someone who knows what they're doing can save you far more than you pay them.
Thank you both for the quick responses, If I'm understanding @scasey correctly, I have blocked them properly, but they are still attempting to initiate an SSH connection with no password log, only an attempted connection log from a known blocked IP address?
I honestly feel like a fool, I totally didn't think that system would log connection attempts, I thought it was only logging password attempts, so when I didn't see any correlating password attempts I became confused.
And thanks for the suggestions @TB0ne, I have done a bit of reading on a few of your suggestions, but I hadn't thought to run it through a VPN, thank you for the words of wisdom my friend, and I can only keep learning.
I honestly feel like a fool, I totally didn't think that system would log connection attempts, I thought it was only logging password attempts, so when I didn't see any correlating password attempts I became confused.
Please don't feel foolish. Things like that are easy to get confused about*
To be sure you understand: It's only logging the connection attempts because you told it to:
Code:
-N SSHATTACK
-A SSHATTACK -j LOG --log-prefix "Possible SSH Attack! " --log-level 7
-A SSHATTACK -j DROP
If you removed the bolded line, then those attempts would no longer be logged...
[Caveat: I think...I really don't have my head around iptables at all...don't make any changes based on what I say, given that what you have seems to be working.]
*A story:
My mail customers were being swamped (maybe 10 per day for 3 or 4 customers) by solicitation emails from political candidates. Reporting the UCE had no effect, so we blocked the sending IP addresses. The resulting log entries about the blocked delivery attempts jumped to around 100 per hour(!)...and still no response from the sending company. That caused excessive mail log rollover, masking the real activity.
We finally moved the blocks to the firewall...no more entries in the mail logs. We didn't care about the increase in the firewall logging.
Thanks again, now that I take a step back I see that the INPUT rules are simply detecting attempted SSH connections regardless of password attempt, and it jumps to the SSHATTACK rule once the hitcount hits 2 in a single minute, so a persistent client attempting an SSH connection over and over regardless of timeout would continue to generate SSHATTACK logs indefinitely.
This has all been a learning experience for me, I'm currently studying for my CCNA certification, and the deeper I go with hands on Cisco the more I need to work with Linux, so its a constant tug of war between the two. As an old guy attempting to learn a new trade I find myself lost in a vast ocean of never ending knowledge and it gets rather intimidating sometimes. I suppose now I've got to learn more about the Cisco Firewall I haven't done much with since I got it
I want to add a bit of "Security" info here, for a different "view" of the problem.
PART 1
I once did some welfare-involved work at a Police Station, along with a few other people. As a "volunteer" effort we were setting up a small room at the Station with a dozen or so networked PCs of the NON-Linux kind.
One Volunteer had his own IT sales company and had some deeper knowledge about IT - and hacking. He wanted to show us his "hacking" prowess. He was at one end of the room using his variety of hacking tools, targetting them one-by-one at a PC at the other end of the room. The sequence went something like this...
"Here's one" he'd say. (Click)
Person at the other PC says, "Nup; it was blocked."
"What about this one?" (Click).
"Blocked it!"
And so on.
It took about 5 or 6 different "tools" until he got onto the PC's hard-drive and was looking around - and the other person could not detect it.
He had several dozen different hack tools - perhaps a hundred.
[Funny part here is, he told everyone to use a particular type of cleaning tool to wipe their PCs at home. I had only recently tested cleaning tools and demonstrated to him that the one he recommended DID NOT work as it described. I think it shocked him.]
PART 2
I had a similar experience to you, teeps.
My modem was being constantly hit from Internet by attacks of various kinds - each one flagged to me by email. I believed that I had my modem protection set up correctly to block them all. (However I had that other hacking demonstration always on my mind too.)
On 3 or 4 occasions I had spoken about the hacking attempts to my ISP provider. Their help-desk was being handled from OUTSIDE my country. (And it's probably not any of the main countries you might immediately think of.) Every time I mentioned it to them I was reassured there was no issue with it.
On one occasion of discussion with them I seemed to be getting a particular kind of "oblique" answer to my questions, which sounded like I was getting the run-a-round. So I used the words "...report it to the Federal Police, and let them sort it out...".
I never received another email about a hack attempt.
PART 3
Now, I know, those (true) stories are not "Linux" and you've marked your post "solved". So, points?
1. You may not be able to see "it" but it might be there anyway.
2. Be MORE careful.
3. Dig deeper maybe.
4. Be alert.
5. Listen to others about security. (I mean, listen to the "what" in what they are saying.)
6. Forums like this are, generally, a good place to understand stuff, and find out about the "new" stuff.
While not directly "Linux" I hope this has been a useful addition to the theme, for you and others.
i understand that the discussion here is not directly about ssh, but...
i'm surprised that nobody mentioned ssh keys, instead of passwords?
i have that set up on my server, and after i made sure it works, disabled password login completely.
surely that would be much better in the long run?
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.