Our server was just trashed a second time by a hacker, even though we're running the APF firewall and BFD ... I've been browsing the logs, but haven't found evidence of how he got in, or don't know what to look for. But, I've noticed that just prior to a brute force attack, the IP addr that the attack comes from is usually logged earlier (anywhere from a few minutes to several hours before the actual attack commences). The loggings always look like these (cut-n-pasted from the /var/log/secure file). My thoughts are - these IP addrs are trying to do something with SSH, and they're not providing an ID string ... is that adequate criterea to block their IP on the spot? This pre-emptive blocking could go a long ways toward thwarting these SOB's. But, are there legitimate contact attempts that would look like this (and thus put one in a position of not being able to block an IP simply because there was SSH-related contact with no ID string)?
thanks,
js



Sep 16 10:37:00 plesk sshd[5596]: Did not receive identification string from ::ffff:202.95.228.201
Sep 16 10:37:00 plesk sshd[5602]: Did not receive identification string from ::ffff:202.95.228.201
Sep 16 10:37:00 plesk sshd[5601]: Did not receive identification string from ::ffff:202.95.228.201
Sep 16 11:44:35 plesk sshd[7496]: Did not receive identification string from ::ffff:65.111.168.85
Sep 16 11:44:36 plesk sshd[7497]: Did not receive identification string from ::ffff:65.111.168.85
Sep 16 11:44:36 plesk sshd[7498]: Did not receive identification string from ::ffff:65.111.168.85
Sep 16 12:16:22 plesk sshd[8804]: Did not receive identification string from ::ffff:201.217.17.150
Sep 16 12:16:23 plesk sshd[8805]: Did not receive identification string from ::ffff:201.217.17.150
Sep 16 12:16:23 plesk sshd[8806]: Did not receive identification string from ::ffff:201.217.17.150
Sep 16 14:02:16 plesk sshd[12947]: Did not receive identification string from ::ffff:72.232.1.130
Sep 16 14:02:16 plesk sshd[12948]: Did not receive identification string from ::ffff:72.232.1.130
Sep 16 14:02:16 plesk sshd[12949]: Did not receive identification string from ::ffff:72.232.1.130
Sep 16 16:29:45 plesk sshd[18669]: Did not receive identification string from ::ffff:64.146.186.2
Sep 16 16:29:45 plesk sshd[18670]: Did not receive identification string from ::ffff:64.146.186.2
Sep 16 16:29:48 plesk sshd[18671]: Did not receive identification string from ::ffff:64.146.186.2
Sep 16 16:44:28 plesk sshd[22946]: Did not receive identification string from ::ffff:220.227.174.66
Sep 16 16:44:29 plesk sshd[22947]: Did not receive identification string from ::ffff:220.227.174.66
Sep 16 16:44:29 plesk sshd[22948]: Did not receive identification string from ::ffff:220.227.174.66
Sep 16 17:15:58 plesk sshd[7952]: Did not receive identification string from ::ffff:201.217.17.150
Sep 16 17:15:59 plesk sshd[7953]: Did not receive identification string from ::ffff:201.217.17.150
Sep 16 17:15:59 plesk sshd[7954]: Did not receive identification string from ::ffff:201.217.17.150
Sep 16 18:01:06 plesk sshd[8762]: Did not receive identification string from ::ffff:65.98.11.18
Sep 16 18:01:06 plesk sshd[8763]: Did not receive identification string from ::ffff:65.98.11.18
Sep 16 18:01:06 plesk sshd[8764]: Did not receive identification string from ::ffff:65.98.11.18

welcome to LQ!!!
it is my understanding that the first thing BOTH parties establishing an SSH v2 connection must do is provide an ID string... this is AFAIK the only *proper* way to establish a normal SSH v2 connection... if a connection attempt is made without providing an ID string, i'd suggest you treat it like a scan immediately... it's typical behaviour for an ssh scan trying to determine the protocol and software versions you are running... just my ...

Until a proper security expert turns up, your first job should be to isolate the server - remove it from the network and from the internet connection. Once that is done, you can start your forensics. Just remember, this is now an untrustworthy server.

Unspawn has put together a list of references, I would suggest you read through them while you await more solid answers: https://www.linuxquestions.org/quest...ad.php?t=45261

he did mention pre-emptive blocking, so i think he's referring to his _new_ install... the way i understood it was, like, he's trying to take steps so that the new box doesn't get cracked again... sorry that my post wasn't solid enough...

I saw the premptive blocking comment, but he also said his server was trashed. Which is why I said that it should be considered untrusted and removed from any network.

And as to "solidity", I'm pretty much just saying what Unspawn told me to say the last time I gave poor (or at least not full) advice to someone. If the server is trashed or if it is considered in anyway breached, the first thing to do would be to take it off the network and hopefully contain the damage.

i agree... it's just that for some reason i got the impression that that had already been done... but you're absolutely right - with security incidents like this there are some things that should be stated anyhow - no matter what... hopefully digimon will post back and fill-us-in on his/her status... i hope all is well... =/

@ digimon: i'm curious about your blocker script... did you make it yourself??

Hi - thanks for the responses. and thanks for the welcome

There's a bit of a story associated with this - myself and a business partner lease a server from EV1servers in Houston. I'm a chip designer - he's a commercial (wedding) photographer. Neither of us have sys-admin level experience/knowledge (but we're learning!). Our first experience was last Jan - malicious mischief in which all index pages were replace with a black "hacked from beyond" page. I think that was an SSH-1 exploit. We were violating every don't-do in the book and had it coming. This second attack came as a bit of a surprise - we were staying (mostly) on top of software updates, had the APF firewall running and the BFD brute-force-detect cron script going, disabled SSH-1 and direct root login, etc. I woke up Friday morning to find that I couldn't download email, couldn't http to my sites, and then found that I could not FTP or SSH in either. I immediately got on the phone with EV1, asking for help. I didn't get any until 16 hours later - around midnight. In the mean time, I'm wondering if my server is spamming the world, attacking other servers, etc? I was pretty frustrated with EV1 - I had not access of any sort and did not get timely help from them. At midnight, I chose to order a priority restore - basically they just drop in a new drive with a ready-2-go Linux/Apache/etc image on it, and slave the old drive so we can recover whatever is recoverable, and maybe analyze it to figure out what was going on. In this case, it appeared to basically be another malicious mischief attack in which many files had been deleted, but sort of randomly. So, for any given domain, you didn't know what was still okay and what wasn't. Since I'm not a guru, I chose to go the route of hard labor - rebuild each website from the ground up and either restoring from the few backups we had or asking customers to re-upload their sites. But, bottom line - I took the hacked system off-line and rebuilt from the ground-up.

I've spent the last couple of days studying the basics of security and observing activity via the system logfiles - even started spreadsheet in which I'm logging 'first encounter' (usually the SSH contact without the identification string) and attacks. I've only been at this for 24-ish hours now, and I've seen about a dozen no-ID string incidents. Because of the short time frame, my data is not statistically significant ... But what I've seen is that about 60 to 70% of the no-ID string warnings come back as brute-force attacks, anywhere from within several minutes to several hours. And, I've trace-routed each no-ID string encounter, and all of them are IP's that have no connection to my site or business - i.e., aliens.

After I figured out how APF and BFD work, I realized that it would be easy to conjur up a custom script to tighten security - i.e., a simple script could easily watch the log files and almost immediately block the no-ID-string IP's. And, that's where I'm headed, but my first step was to seek out expert opinion on whether I'd cause other problems if I implemented such a security policy. And, it sounds like I probably wouldn't, or if there were an occasional glitch, it may be tolerable, considering the increased level of security it would buy.

But, to qualify this, I still have no idea of how we were hacked this last time around, and I'm guessing that it will take lots more studying or some $'s to find out. I may just accept not finding out, studying and implementing better security based on available documentation as you've mentioned, and maybe implementing this script to block IP's immediately upon the no-ID-string encounters.

- js

My primary job is securing networks and I've also done quite a bit of website hosting in my time.

Why do you have SSH open on the usual port? Change it to something else to avoid automated SSH attacks. This will immediately reduce the amount of attacks you recieve but please notice that this will NOT secure your machine.

Yes, a pre-emptive IP blocking technique would be quite good too... http://www.pettingers.org/code/sshblack.html or similar - there are hundreds of these to choose from on www.freshmeat.net and similar website, pre-built to make sure that you don't block IP's that are doing "innocent" things. I used to use one and it got about 20-30 different IP's a day on a popular server - it won't stop them, they will just choose a different IP but without it you will see hundreds of attempts at logins (all with different variations on default usernames and passwords) from each attack.

If you're not sure that the compromises were done over SSH, don't immediately rule everything else out. Most webhosts see brute-force SSH attempts even when they have no accounts that can log in via SSH. In a secure configuration, no amount of brute-force guessing will compromise any account.

My first suspect would be your Apache setup with it's scripting languages - make sure they are RELIGIOUSLY kept up to date and that any scripts you use are thoroughly checked before you allow them - even things like PHP bulletin boards can be dangerous if incorrectly programmed (see the recent Debian break-ins which were done via an out-of-date script on the Apache setup on the same machine).

If your customers are not using them, block SSH logins to all but your IP address. Set up public-key logins and disallow password logins altogether (the best way without a doubt). If your customers are allowed to login via SSH and they choose a weak password and you've not got every piece of up-to-date software (including kernels), they can try all sorts of local attacks over SSH.

But the most important things are a) backups - buy the backup package from your host and then downtime will be in minutes, not hours or days. b) software updates - everything. Anything that you don't use and therefore don't want to update everything single time it changes, remove it from the server. Update your kernels. Update your apache, PHP, openssl, everything you can find. c) audit PHP etc. scripts and everything that runs under your website. d) passwords - if you can, get rid of them altogether (public key logins). If not, make sure they are secure (impose quite strict password control). If you can, block anyone but you getting to any sort of login whatsoever.

@win32sux/XavierP: sorry that my post wasn't solid enough. / If the server is trashed or if it is considered in anyway breached
I realise for all of us it's quite easy to regurgitate information and recite any mantra's we know, and to less-frequent visitors answering questions is a game of wedging in the first reply, but to us it ain't, or shouldn't be. And that means we have to make sure we have all the info we need. The first question we should ask ourselves is did the OP did include all nfo necessary for us to be able to answer in a methodical and detailed way? Of course you should add the boilerplate statements when in doubt, but the OP should fill in the gaps so we don't have to interprete any.
I am glad to see you both handled that well.


@digimon
(myself and a business partner lease a server from EV1servers
OT note but I als keep an eye on the EV1 security forum. Haven't seen you post there...)

Also since Ledow got in 10 minutes before me I'll try and not repeat things but try to add to that instead.


Why do you have SSH open on the usual port? Change it to something else to avoid automated SSH attacks.
I agree moving the SSH port does *nothing* to enhance security and therefore just serves to waste time. Secure SSH properly first, then look at things to play with.


are there legitimate contact attempts that would look like this
Definately not and win32sux is right about that. You can mimick this yourself by pointing nc, a browser or even telnet to port 22. If you don't/can't send the expected handshake sshd will discard the request and it shows up as such in the logs. About the only time it's legitimate is when you use remote monitoring, but then obviously the remote IP would be the same and you would recognise the IP.
In addition to Ledows remarks about securing SSH:
- Regardless of what you use SSH for you should tighten access by not allowing root account logins (use an unprivileged account to log in, use sudo for repetetive management tasks and only su to root if needed) and not allowing SSH protocol version 1 (if you need it you will know it).
- You can configure about any PAM service to deny certain accounts login and that goes for ssh too.
- If you use SSH solely for management purposes it would be helpful if you could shield it from probing by only allowing access to it (in tcp_wrappers and the firewall) from your management IP addresses or ranges.
- If you need to restrict access due to bruteforcing you don't need to scour Freshmeat for it. Have a look at this thread http://www.linuxquestions.org/questi...d.php?t=340366 which lists more than a few methods for blocking scans. While (re)inventing the wheel can be a good thing thse tools are more or less tried by many which means less chances of errors or flaws, more support, etc etc.
- Finally it could come in handy to add a backup ssh entry to Xinetd (if you run it) on a separate port in case sshd seems down. Obviously this doesn't handle all malfunctions but could come in handy. Xinetd is a good choice because it can (and should) be configured with IP address restrictions.


Of course this is pointless if the box isn't hardened properly.
I would be interested in helping you assess in what ways the box is hardened enough now. That starts with a list of (publicly) accessable software, services and versions as well (see examples below), plus if possible a list of things you already implemented. Maybe you already used a checklist like CERT's http://www.cert.org/tech_tips/unix_s...cklist2.0.html and SANS' http://www.sans.org/top20/ (for more see the LQ FAQ: Security references, post #1 "Basics, important sites, HOWTO's, handbooks, hardening, tips" see "Checklists" and "Securing").


I would also be interested in helping you find out what the point of entry was. I think the best way to start would be to make inventory of what software, services (and versions) where available at the time, especially things running on top of Apache I mark as "vulnerable by default" meaning any PHP-based SW like Fora, CMSes, Blogs and such. If your old HD is available as /mnt/olddisk (meaning old RPM database is available and has full path "/mnt/olddisk/var/lib/rpm") you could generate the list automagically like this:
or if you want to manually, add this temporary function to your running Bash shell:
* of course you could take the versions from /mnt/olddisk/var/log/rpmpkgs but that implies the cronjob refreshing it was run periodically and unedited.