Security of a small server

Didier Spaier · 12-06-2019, 03:53 PM

I have set up a Linode VPS as web and email server (daily backup provided). This is my first server on a not shared hosting plan so I have a lot to learn.

The web server uses Apache + PHP to serve static pages including a blog built with Pelican (purely static so far) and also a Dokuwiki wiki. I have a Letsencrypt certificate (used acme.sh to get it).

The email server uses Postfix + Dovecot + MySQL with DKIM and is only used to receive the emails on behalf of the slint.fr domain (Postfix doesn't act as a relay).

I ssh to the VPS using a SSH key (PasswordAuthentication no, PermitRootLogin no in sshd_config).

I have in /etc/rc.d/rc/local this code snippet:

Code:

if [ -e /etc/iptables/iptables.rules ]; then
    iptables-restore < /etc/iptables/iptables.rules
fi

the rules for the filter table are:

Code:

:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [59:11784]
:TCP - [0:0]
:UDP - [0:0]
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -m conntrack --ctstate INVALID -j DROP
-A INPUT -p icmp -m icmp --icmp-type 8 -m conntrack --ctstate NEW -j ACCEPT
-A INPUT -p udp -m conntrack --ctstate NEW -j UDP
-A INPUT -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j TCP
-A INPUT -j REJECT --reject-with icmp-proto-unreachable
-A TCP -p tcp -m tcp --dport 22 -j ACCEPT
-A TCP -p tcp -m tcp --dport 80 -j ACCEPT
-A TCP -p tcp -m tcp --dport 443 -j ACCEPT
-A TCP -p tcp -m tcp --dport 25 -j ACCEPT
-A TCP -p tcp -m tcp --dport 465 -j ACCEPT
-A TCP -p tcp -m tcp --dport 487 -j ACCEPT
-A TCP -p tcp -m tcp --dport 993 -j ACCEPT
-A TCP -p tcp -m tcp --dport 995 -j ACCEPT

Executable daemon managers:

Code:

rc.dovecot
rc.httpd
rc.inet1
rc.inet2
rc.inetd
rc.keymap
rc.local
rc.local_shutdown
rc.loop
rc.mcelog
rc.messagebus
rc.mysqld
rc.opendkim
rc.postfix
rc.sshd
rc.syslog
rc.sysvinit
rc.udev
rc.ulogd

Two questions:

What is wrong/missing in these settings?
I would like to avoid rootkits, but am puzzled: there are a lot software available for that @SBo but I have no idea which would be the best in my case. A search with "rootkit" as argument gives:
Code:
```
chkrootkit
rkhunter
samhain
tiger
unhide
```
Which one(s) should I use?

Thanks in advance for any clue, have a good day,

NetrixTardis · 12-06-2019, 07:11 PM

install fail2ban. configure it to watch your active servi ea such as sshd, postfix dovecot. I'm not sure if it's beneficial for webserver. On a personal note, since my vps is for my use only, and I have no reason not to do, I block all of APNIC. I go through their site, find a list of all their netblocks, and drop all their traffic. since you are hosting skint.fr, this may not be a good idea but something to consider.

astrogeek · 12-07-2019, 01:19 AM

Quote:

Originally Posted by Didier Spaier

I have set up a Linode VPS as web and email server (daily backup provided). This is my first server on a not shared hosting plan so I have a lot to learn.

Way to go! Now prepare for baptism by fire, but try to keep it simple, hope for the best, and you'll be fine!

Quote:

Originally Posted by Didier Spaier

Two questions:[*]What is wrong/missing in these settings?[*]I would like to avoid rootkits, but am puzzled: there are a lot software available for that @SBo but I have no idea which would be the best in my case.
...
Thanks in advance for any clue, have a good day,

I would say that important things are missing from your iptables rules, such as any rate or connection limits, differentiation of traffic other than by destination port (file downloads vs web pages, for example), and a few strategic logging rules to know what it is actually doing.

That said, writing and understanding your own rule sets is definitely better than using a syntax simplifying front end such as UFW, in my opinion. If you don't understand the rules syntax and the problems they solve, you cannot effectively use a simplified syntax. If you do understand, then you don't want or need an alternate syntax. Work through the iptables-extensions man page using a local target to explore and validate your rules sets before taking them live, you'll quickly figure out what you actually need.

I would also suggest becoming familiar with tc if you have not already been exposed. With it you can very easily place important limits on incoming traffic and fully shape your outgoing traffic. For example, you will need to simply ban abusive traffic at times, for which you will frequently see long lists of iptables rules used.

But if you combine ipset with tc you can very efficiently manage and ban all traffic from IPs or whole networks before it enters the netfilter stack, like this:

Code:

Create a banned set...

ipset create banned hash:net 1024 counters

Add an ingress filter to match entries in the set against source IP...

tc qdisc add dev eth0 ingress
tc filter add dev eth0 parent ffff: protocol ip prio 10 basic match ipset(banned src) action drop

Now add abusers to the set, manually or with an iptables rule for adaptive blocking....

ipset add banned 1.2.3.0/24

(Note: You may need to escape the parens in above tc filter... ipset$...$ to hide them from the shell)

You will never see them again unless you look at the set counter or filter stats:

Code:

ipset list banned
...
...or...

# tc -s filter ls dev eth0 parent ffff:
filter protocol ip pref 10 basic
filter protocol ip pref 10 basic handle 0x1
  ipset(banned src)

        action order 1: gact action drop
         random type none pass val 0
         index 1 ref 1 bind 1 installed 32838 sec used 297 sec
        Action statistics:
        Sent 6234 bytes 118 pkt (dropped 118, overlimits 0 requeues 0)
        backlog 0b 0p requeues 0

Much simpler and easy to manage - and best of all, you don't have to see it again in your iptables rules or your logs!

With similarly simple tc filters and use of hierarchical qdiscs and classes on egress, you can assign different bandwith and priority to different types of traffic (downloads vs web pages or mail, guaranteed ssh bandwidth under load, etc.), and simplify your iptables rules in the process. This helps to keep your Slackware Linode standing up straight under stress!

I would also recommend learning to use ModSecurity with your Apache server. It is easy to build and install, a bit more complex to decide which rules to use. In the end, you will find that security does not result from which programs you install, it comes from learning to use whatever tools are available and from gaining a full understanding of each problem that presents itself, and responding based on that understanding.

Hope this helps!

Turbocapitalist · 12-07-2019, 01:31 AM

Quote:

Originally Posted by Didier Spaier

I ssh to the VPS using a SSL key (PasswordAuthentication no, PermitRootLogin no in ssh_config).

I'll assume those are typos in the post and not where you made the settings, but just in case, that should be sshd_config instead of ssh_config, and SSH key instead of SSL.

Otherwise, congratulations on the leap forward.

Didier Spaier · 12-07-2019, 03:16 AM

@astrogeek: thanks for the valuable advice, that I will (try to) follow. And (I should have mentioned that already) thanks a lot for your series of articles Creating a Virtual Mail Server with Postfix, Dovecot and MySQL that helped me a lot to have the email server up and running.

For the records I also found since a few other tutorials from: Linode, DigitalOcean and Gerardo Zamudio (this one also based on Slackware, interesting indeed though I may not need all tools he mentions yet) but followed yours.

@Turbocapitalist: fixed, thank.

teoberi · 12-07-2019, 03:29 AM

At question 1 the answer is more complicated!
Your firewall is a bit thin and a good firewall is not an easy task. I started about 10 years ago with an example of a firewall found on the net and that I have improved for my needs. @astrogeek referred to "any rate or connection limits" ie: limit for TCP-SYN-Flood detection, Ping-Flood-Detection, SYN Cookies settings, etc.
In Apache do you redirect traffic from 80 -> 443 (e.g. mod_rewrite)? Apache also has a few configuration options to apply (e.g. ServerSignature Off; ServerTokens Prod; TraceEnable Off)
Php also has security options (e.g. expose_php).
I see that you do not use SPF and possibly DMARC.
SSH can also be configured to improve security (changed the default port, deactivated IPv6 if you do not use it, restricted access using the "AllowUsers" directive or even firewall rules).
I don't see any content filter (e.g. Amavisd) and no antispam system (e.g. Spamassassin or Rspamd) or antivirus (e.g. Sophos antivirus for Linux + SAVDI, Clamav).
For question 2 I recommend you Lynis-Security auditing tool for Linux
https://cisofy.com/lynis/
and a system scan with Tenable Nessus
https://www.tenable.com/products/nessus
with which you can detect and correct a lot of security problems.
For rootkit detection I use:
1. chkrootkit (updated by developers once a year);
2. rkhunter (without too many updates).
Both chkrootkit and rkhunter are extremely easy to compile from sources and a cron task performs the scan once a week.

Didier Spaier · 12-07-2019, 06:47 AM

OK, now reading Linux Packet Filtering and iptables so I can understand the basics.

As an aside, about getting a Let's Encrypt certificate I'd like to thank Eric Hameleers for this article on his blog. Although I ended up using acme.sh instead as easier to setup for a beginner like me than dehydrated, I value the time spent by Eric to provide clear and detailed explanations.

ricky_cardo · 12-07-2019, 08:46 AM

Here are the tc options I used to limit port 80 just for reference and example:
- I put this /etc/rc.d/rc.tc_upload_limit_p80 (chmod +x )

Code:

##CONNECTION HARD LIMIT 110.23 Mbps DL and 10.07 Mbps UL for my place## Not scientific just speed test
##mbps = 1024 kbps = 1024 * 1024 bps => byte/s
##mbit = 1024 kbit => kilo bit/s.
##mb = 1024 kb = 1024 * 1024 b => byte
##mbit = 1024 kbit => kilo bit.
##Internally, the number is stored in bps and b.
##But when tc prints the rate, it uses following :
##1Mbit = 1024 Kbit = 1024 * 1024 bps => byte/s


#File with some rules to attempt to slow down the upload of service on port 22 to prevent loss of interactivity.
#delete the qdisc so we can try from the beginning
tc qdisc del dev eth0 root
#add primary qdisc
tc qdisc add dev eth0 root handle 1:0 htb default 2
# add primary class slower card is 100-100
tc class add dev eth0 parent 1:0 classid 1:1 htb rate 1000mbit ceil 1000mbit
#add 2 classes inside the primary class
tc class add dev eth0 parent 1:1 classid 1:2 htb rate 990mbit ceil 1000mbit
tc class add dev eth0 parent 1:1 classid 1:3 htb rate 492kbit ceil 492kbit
#tell which algorithm the classes use
tc qdisc add dev eth0 parent 1:2 sfq
tc qdisc add dev eth0 parent 1:3 sfq


tc filter add dev eth0 parent 1:0 protocol ip u32 match ip sport 80 0xffff flowid 1:3
tc filter add dev eth0 parent 1:0 protocol ip u32 match ip dport 80 0xffff flowid 1:3

then the old add to /etc/rc.d/rc.local

Code:

## set some transmission control rules:
if [ -x /etc/rc.d/rc.tc_upload_limit_p80 ]; then
      /etc/rc.d/rc.tc_upload_limit_p80
fi

uteck · 12-07-2019, 09:02 AM

The TIger package will do a lot of the security work for you. Here is the package descrition from apt, but should work the same on Slack.

Quote:

TIGER, or the 'tiger' scripts, is a set of Bourne shell scripts, C programs
and data files which are used to perform a security audit of different
operating systems. The tools can be both run altogether once to generate an
audit report of the system and they can also be run periodically to
provide information on changes to the system's security once a
security baseline has been defined. Consequently, they can be used
also as a host intrusion detection mechanism.
.
The tools rely on specialised external security tools such as John the Ripper,
Chkroot and integrity check tools (like Tripwire, Integrit or Aide) for some
of the tasks. The periodic review mechanism relies on the use of the cron task
scheduler and an email delivery system.
.
TIGER has one primary goal: report ways the system's security can be
compromised.

Skaendo · 12-07-2019, 10:49 AM

On my little home server (and personal computers), I use IPSet and a script called ipset-blacklist by trick77 in my firewall. It's very small (one script and one config), very powerful, very easy to setup and very fast. It blocks known malicious IP's that are published in blacklists and can use a local blacklist as well. You can also use it to block entire countries, single addresses, or blocks of addresses.

My blacklist currently has over 79500 entries, that includes single addresses, blocks and entire countries.
So I am effectively blocking well over 100,000 addresses.

The only downside is I think that it only supports IPv4?

teoberi · 12-07-2019, 11:12 AM

Quote:

Originally Posted by Didier Spaier
As an aside, about getting a Let's Encrypt certificate I'd like to thank Eric Hameleers for this article on his blog. Although I ended up using acme.sh instead as easier to setup for a beginner like me than dehydrated, I value the time spent by Eric to provide clear and detailed explanations.

I still prefer the old method of doing things manually (so I know what I'm doing).
https://gethttpsforfree.com/

astrogeek · 12-07-2019, 10:34 PM

You are most welcome, and thanks for the kind comments about my Slackdocs article (which has long been in need of an update!), I am pleased that you found it useful!

I would also suggest changing your port scan trap rule here...

Quote:

Originally Posted by Didier Spaier

Code:

-A INPUT -j REJECT --reject-with icmp-proto-unreachable

... into a simple DROP, and replace it with one which can identify and discard the majority of port scans much earlier in the stack, as follows...

In a sane world governed by mutual respect, politely notifying the sender of a request the reason it could not be fulfilled would be the right thing to do. But the reality of the internet is Mad Max.

If you send any response, even one recommended by the protocols, to a TCP --syn or INVALID packet (a packet not actually known to be from the sender it claims to be from) not destined for an open port, your server will subsequently be used to generate nuisance or wannabe DDoS traffic aimed at others. If the reply you send is bigger than the request used to generate it, you provide the originator with a traffic amplifier as well.

The originators send out requests aimed at servers already identified as responding to probe traffic (i.e. yours), but they spoof the IP of their actual intended target as the source IP. Your server, and a few thousand others all helpfully reply to the target with an ICMP message or TCP Reset - at the same time... worst case is that the target does not understand what has happend and decides to retaliate - against you.

The only polite and safe thing we can do under those circumstances is to keep quiet - just DROP probes sent to unopened ports.

I generally try to drop as much probe traffic as possible, as early as possible, by dropping any TCP --syn packets bound for an unopened port in the raw:PREROUTING table, something like this (where ${OPEN_PORTS) is a list of the ports we are actually listening on, $INET is the internet interface):

Code:

-t raw -A PREROUTING -i ${INET} -p tcp -m tcp --syn -m multiport !--dports ${OPEN_PORTS} -j DROP

Then of course, drop INVALID tcp packets as early as possible in the INPUT chain:

Code:

-t filter -A INPUT -p tcp -m conntrac --ctstate INVALID -j DROP

This all has the added benefit that you can now log anything which reaches the end of your INPUT chain as an unexpected or unhandled event instead of a torrent of probes. That is, if your preceeding rules are as comprehensive as they should be, then nothing should ever reach the end of your INPUT chain, and when something does show up there it needs attention. This turns your wastebasket into an important troubleshooting tool and moves the probe traffic mostly out of sight.

Another member mentioned SYN-Floods, which you will be subject to among other problems and should address. You can look it up easily enough for the full details, but basically it is an attack in which your server is flooded with TCP SYN packets which never respond to the ACK/SYN you send in reply. If using connection tracking, as most are, then your server must keep a record of the incomplete handshake while awaiting the ACK, which never comes. This can quickly exceed the number of connections your system can keep track of, after which it begins to drop additional incoming SYNs... and stops responding to legitimate traffic.

The modern kernel is not as susceptible to this as older ones were, but you still need to be sure that the basic protections are in place. You need to enable the kernels protection mechanism, tcp_syncookies, and limit the number of times it will retry to obtain a response (another exploitable amplifier):

Code:

echo 1 > /proc/sys/net/ipv4/tcp_syncookies
echo 3 > /proc/sys/net/ipv4/tcp_synack_retries

The best course in my opinion, is to use the iptables SYNPROXY mechanism to handle the TCP handshake out of band, so that by the time a packet is ACCEPTed by your rules, it has completed the handshake and you know it is a two-way conversation. A topic I must defer to a later post...

Hope this helps!

syg00 · 12-08-2019, 04:42 AM

Both the "P" in TCP/IP means "protocol" - that to me means someone planned things. Like security maybe.
Guess I was wrong - what a bloody shambles.

Sorry for the interruption; back to your normal transmission ...

Alien Bob · 12-08-2019, 04:49 AM

Quote:

Originally Posted by syg00

Both the "P" in TCP/IP means "protocol" - that to me means someone planned things. Like security maybe.
Guess I was wrong - what a bloody shambles.

Sorry for the interruption; back to your normal transmission ...

With comments like this I can also easily write up to 18k posts without contributing anything of value. Way to go.

syg00 · 12-08-2019, 05:28 AM

I wasn't meaning to demean anyone here contributions, but was (apparently poorly) ranting re my frustration on understanding the needless complexity that networks add to our lives. I always feel like an outsider looking in.