[SOLVED] server has been hacked. any ideas on how to find/remove hacker/s

Originally Posted by timsoft most routers that are used by people who have my servers do not have full firewall control, only basic nat/port forwarding options, and some DOS flood blocking options. Maybe if they were running openwrt or something they would allow more on the router firewall control, but I just use "off the shelve" economical routers.

I have a Mikrotik router with RouterOS. It does everything you could ever want and more, and the routers are very economical. Just in case you wanted a recommendation.

I had a look at a mikrotik router, as a customer who had 5G LOS BOB broadband had one from his isp, but its implementation of iptables via its web interface on mikrotik os appears a bit different, and the os appears closed source, and relies on cloud for licence management. Because my own internet is satellite based, with high latency (think 750ms ping times), and packet conglomeration for "tcp accelleration", I tend to avoid anything cloud. (my data allowance is limited, and expensive (by uk terrestial broadband standards), which was how the one hack was bad news, it ate up my monthly allowance in 6days!, and top-up data would have cost me $260 US equivelant).
I quick google search of mikrotik routers came up with
https://www.zdnet.com/article/thousa...-user-traffic/
which wasn't exactly inspiring, even though they are probably one of the most versatile, outside of openwrt based routers.
thanks for the recommendation though.

Originally Posted by timsoft I had a look at a mikrotik router, as a customer who had 5G LOS BOB broadband had one from his isp, but its implementation of iptables via its web interface on mikrotik os appears a bit different, and the os appears closed source, and relies on cloud for licence management. Because my own internet is satellite based, with high latency (think 750ms ping times), and packet conglomeration for "tcp accelleration", I tend to avoid anything cloud. (my data allowance is limited, and expensive (by uk terrestial broadband standards), which was how the one hack was bad news, it ate up my monthly allowance in 6days!, and top-up data would have cost me $260 US equivelant). I quick google search of mikrotik routers came up with https://www.zdnet.com/article/thousa...-user-traffic/ which wasn't exactly inspiring, even though they are probably one of the most versatile, outside of openwrt based routers. thanks for the recommendation though.

I don't know anything about how the licensing works, as my router came with the software already. And yes, there have been some security issues lately, but apparently they only affected users who did not keep them updated, from what I read. I can understand the desire for something open source, though.

there were some good comments from astrogeek as well re iptables. As I don't do servers very often, I find myself having to relearn/redo so much each time, but in this case, it was more critical that I stopped the hack, then could spend more time working out ways to improve security above denyhosts which i used anyway.
Unfortunately samba has changed a fair bit over the years, as has vsftp, apache and ssh, and even the default config for something like dovecot breaks things between updates, which means relearning things, as documentation becomes obsolete. I am more of an IT generalist, than targeting a particular branch, like network security. That is one of the reasons I like slackware so much, it doesn't change a lot over time (that and the helpful people on linuxquestions, and slackbuilds.org mailing list).
I did create a script for my samba server which firewalls a client if it is infected with ransomware, to protect the server's files; so I have done a bit of iptables stuff before, it's just that I get so rusty, with often years between having to be familiar with it.

I have been waiting for 15.0 to come out, so I could update all the servers, but just could not wait any more for some of them, which were getting too old. Now there are two stable v5.x kernels, hopefully one will become a long-term support version, and maybe we will get a new version of slackware. The added nvme boot support, and support for newer hardware (cpu/apu etc) will be good, as "current" is too much of a moving target for me to use. Anyhow, I'll get back to polishing my firewall rules, and changing all the passwords.
It would be nice if sshd could be configured to allow password login for any user for local lan login, but key login only for remote (wan) login. Then I would have the convenience for local logins and the security for remote logins.

Originally Posted by timsoft That is one of the reasons I like slackware so much, it doesn't change a lot over time (that and the helpful people on linuxquestions, and slackbuilds.org mailing list).

One thing I worry about with slackware though is that it is (or at least used to be) insecure by default. By default it starts (or used to start) sshd, and by default it accepts password login. So Joe User sets up his new slackware, enters a weak password thinking it doesn't matter because he is the only user, and then he is subject to dictionary attacks from the internet. (Break-in will be within minutes if he, or some other user on the system, chooses a password of "password".)

What is likely to save him is that his IPv4 router does NAT. That will no longer necessarily be true once IPv6 becomes more prevalent.

Originally Posted by chrisVV One thing I worry about with slackware though is that it is (or at least used to be) insecure by default. By default it starts (or used to start) sshd, and by default it accepts password login. So Joe User sets up his new slackware, enters a weak password thinking it doesn't matter because he is the only user, and then he is subject to dictionary attacks from the internet. (Break-in will be within minutes if he, or some other user on the system, chooses a password of "password".) What is likely to save him is that his IPv4 router does NAT. That will no longer necessarily be true once IPv6 becomes more prevalent.

Slackware is not for the faint-hearted. I do not know whether this is good or not. But, you have to learn how to use it. I am still an amateur, but I've been using Slackware since 2004. And I am happy. I work with SLES 12, but Slackware is much, much easier. Probably because of the years in love.

One thing I worry about with slackware though is that it is (or at least used to be) insecure by default.

While this might be true, I think there is a simple "Occam's Razor" explanation. I don't recall Pat sharing his thoughts on his philosophy about the initial install, but I think there is a clue on the first reboot after installing. Pat does not force user's to create a non-root account. Many people would scream about such a shortcoming. Well, no big deal. Create a non-root account. Intro to Computers 101.

I think Pat's slant toward all of this is the user not only gets a full-featured system, but the defaults are designed as teaching aides.

Some people might consider this design damn cool. Those who are not computer savvy probably brown their pants. No bragging intended. Just a statement of facts.

I well remember my first efforts with Slackware. I thought WTF. Now, many years later, I find Pat's design to be Goldilocks -- just right. I have to add my own sweat equity, but by golly, I get the operating system I want and not what some egotistical upstream developer thinks I want or thinks I need. I'm not a member of the lowest common denominator with respect to using computers. I have half a clue. I suspect Pat prefers to treat users the same way. Spoon feed a person or teach a person how to fish. Yadda yadda.

I also suspect there is a laziness factor involved. Possibly Pat's attitude is that he sees no reason to cuddle users. Provide the basics and let the dice roll. If users can't figure out how to do some basic computer tasks then perhaps he has designed a sneaky way to weed out those users. Weeding out such users saves him a ton of support time.

One of these days I need to road trip through Minnesota. Buy the guy a beer and stick a Ben Franklin or two in his shirt pocket. And if he'll let me, give him a good bear hug or at least a good firm hardy handshake. The guy is just too damned smart. A legend in his own time.

rkhunter and lynis are added to my tools. I did find /dev/.eudev hidden folder on the server, which was flagged by rkhunter but being 13.1, maybe it is supposed to be there.

rkhunter is a cranky sucker. Crude. I use rkhunter on our servers at work. More than few times I have considered removing the app. So many exceptions that need to be defined. But I leave the package installed....

if I could figure out how to implement rate limiting on remote login's to my web app, that would protect the http side of things as well

I have an Asus RT-AC66U router. I never trusted the vendor firmware with all of the usual phone home and indirect data mining and tracking. I installed DD-WRT, which is not a robust firmware by any means. Nonetheless, here are the rate limiting firewall rules I use:

# Rate limit brute force attacks on forwarded ports
iptables -t nat -I PREROUTING -p tcp -d $(nvram get wan_ipaddr) --dport 1234 -j DNAT --to 192.168.1.10:22
iptables -I FORWARD -p tcp -d 192.168.1.10 --dport 22 -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -I FORWARD 2 -p tcp -d 192.168.1.10 --dport 22 -m state --state NEW -m limit --limit 3/min --limit-burst 3 -j ACCEPT
iptables -I FORWARD 3 -p tcp -d 192.168.1.10 --dport 22 -j logreject

Numbers changed to protect my ego. I am guessing you can adapt this to a normal Linux distro. In my use case, port 1234 is a port-forwarding to port 22 on my LAN server.

Caveat emptor: I am not an iptables guru.

As far as the ssh stuff I've now got a good selection of things to try out

Just disable password logins and use keys. Or, use both if you have a recent version of SSH. When using both, if the key exchange fails then the SSH server never prompts the user for a password. Arguable a simple form of 2FA.

As a personal practice, I always passphrase-protect my private keys, regardless of whatever policy is used upstream.

Originally Posted by OutSiderBR Slackware is not for the faint-hearted.

Neither is driving a bus or hiking through the forest. But people still do them.