[SOLVED] iptables DNAT stopped working

mfoley · 11-25-2017, 10:12 PM

I've been working on this for a few days with no luck. I need some help. For literally years I've routed incoming eth0 requests for port 1912 to port 3389 on a LAN workstation. A couple of days ago it simply stopped working. I did do a restore on the server so I must have done something to mess this up. But I can't figure out what. I've compared before and after restore on the rc.firewall script and they are identical. Here are my iptables rules:

Code:

# Generated by iptables-save v1.6.0 on Sat Nov 25 22:59:04 2017
*raw
:PREROUTING ACCEPT [691:99829]
:OUTPUT ACCEPT [593:106315]
COMMIT
# Completed on Sat Nov 25 22:59:04 2017
# Generated by iptables-save v1.6.0 on Sat Nov 25 22:59:04 2017
*mangle
:PREROUTING ACCEPT [691:99829]
:INPUT ACCEPT [663:93364]
:FORWARD ACCEPT [4:202]
:OUTPUT ACCEPT [593:106315]
:POSTROUTING ACCEPT [597:106517]
COMMIT
# Completed on Sat Nov 25 22:59:04 2017
# Generated by iptables-save v1.6.0 on Sat Nov 25 22:59:04 2017
*nat
:PREROUTING ACCEPT [130:15069]
:INPUT ACCEPT [101:8456]
:OUTPUT ACCEPT [29:2005]
:POSTROUTING ACCEPT [15:997]
-A PREROUTING -i eth0 -p tcp -m tcp --dport 587 -j REDIRECT --to-ports 25
-A PREROUTING -i eth0 -p tcp -m tcp --dport 1912 -j DNAT --to-destination 192.168.0.58:3389
-A POSTROUTING -o eth0 -j MASQUERADE
COMMIT
# Completed on Sat Nov 25 22:59:04 2017
# Generated by iptables-save v1.6.0 on Sat Nov 25 22:59:04 2017
*filter
:INPUT DROP [5:350]
:FORWARD ACCEPT [2:96]
:OUTPUT ACCEPT [583:105043]
:bad_people - [0:0]
:checkcount - [0:0]
:logdrop - [0:0]
-A INPUT -i eth0 -j bad_people
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -i eth1 -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A INPUT -i eth1 -p udp -j ACCEPT
-A INPUT -s 192.168.0.0/24 -p icmp -m icmp --icmp-type 8 -m state --state NEW,RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 22 --tcp-flags FIN,SYN,RST,ACK SYN -j checkcount
-A INPUT -i eth0 -p tcp -m tcp --dport 22 --tcp-flags FIN,SYN,RST,ACK SYN -m limit --limit 1/sec --limit-burst 3 -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m multiport --dports 25,80,143,443,587,993 -j ACCEPT
-A FORWARD -i eth1 -j ACCEPT
-A FORWARD -i eth0 -j bad_people
-A checkcount -m recent --set --name DEFAULT --mask 255.255.255.255 --rsource
-A checkcount -m recent --rcheck --hitcount 12 --name DEFAULT --mask 255.255.255.255 --rsource -j logdrop
-A checkcount -j RETURN
-A logdrop -j LOG --log-prefix "SSH Break-in attempt " --log-level 6
-A logdrop -j DROP
COMMIT
# Completed on Sat Nov 25 22:59:04 2017

Note the 1st DNAT rule. That doesn't appear to be working. If I monitor port 1912 using tcpdump, I can see the remote requests hitting the server but the attempt times out after 2 minutes. If I try the destination port from the server it connects:

Code:

> telnet 192.168.0.58 3389
Trying 192.168.0.58...
Connected to 192.168.0.58.
Escape character is '^]'.

Any idea why the remote attempt to port 1912 is not working?

lazydog · 11-26-2017, 02:09 AM

One thing that jumps out at me is

Code:

-A FORWARD -i eth0 -j bad_people

But you don't have anything listed for this chain. What used to be in this chain?

When you say you are comparing the scripts are you comparing what used to work before you did the restore to the restored rules or are you comparing the restored rules to what you have in backup?

mfoley · 11-26-2017, 10:57 AM

Quote:

Originally Posted by lazydog

One thing that jumps out at me is

Code:

-A FORWARD -i eth0 -j bad_people

But you don't have anything listed for this chain. What used to be in this chain?

I do not have anything listed for that chain to keep clutter down in this posting. If I listed all members of that chain there would be an additional 970 lines in my listing! The purpose of that chain is for a security script to dynamically block bad actors. It is described in this thread: https://www.linuxquestions.org/quest...120/page3.html. Yes, I have confirmed that the IP I'm trying to connect from is not in the iptables DROP list. I am able to connect to the target server remotely from port 22.

Quote:

When you say you are comparing the scripts are you comparing what used to work before you did the restore to the restored rules or are you comparing the restored rules to what you have in backup?

Well, those sound like the same thing to me. Yes, the script when it was working before, which is what have in my backup, compared to the script running now. They are the same which means the script is unchanged since at least September and everything was running fine until I did a restore two days ago. I restored from a backup made that same day (2 days ago). The reason was that I did a backup of the system before trying an upgrade to a package, which didn't work, so I restored that freshly made backup to remove the changes made by the package upgrade. The "restore" event may be coincidence. Maybe something was broken well before that and it took the reboot after the restore to reset whatever.

Perhaps the script itself might be clearer than the iptables-save output:

Code:

usr/sbin/iptables -P INPUT ACCEPT
/usr/sbin/iptables -P FORWARD ACCEPT
/usr/sbin/iptables -P OUTPUT ACCEPT
/usr/sbin/iptables -t nat -P PREROUTING ACCEPT
/usr/sbin/iptables -t nat -P POSTROUTING ACCEPT
/usr/sbin/iptables -t nat -P OUTPUT ACCEPT
/usr/sbin/iptables -t mangle -P PREROUTING ACCEPT
/usr/sbin/iptables -t mangle -P INPUT ACCEPT
/usr/sbin/iptables -t mangle -P FORWARD ACCEPT
/usr/sbin/iptables -t mangle -P OUTPUT ACCEPT
/usr/sbin/iptables -t mangle -P POSTROUTING ACCEPT
/usr/sbin/iptables -t raw -P PREROUTING ACCEPT
/usr/sbin/iptables -t raw -P OUTPUT ACCEPT
/usr/sbin/iptables -F
/usr/sbin/iptables -F -t nat
/usr/sbin/iptables -F -t mangle
/usr/sbin/iptables -F -t raw
/usr/sbin/iptables -X
/usr/sbin/iptables -X -t nat
/usr/sbin/iptables -X -t mangle
/usr/sbin/iptables -X -t raw
/usr/sbin/iptables -P INPUT DROP
/usr/sbin/iptables -N bad_people

/usr/sbin/iptables --table nat --append POSTROUTING --out-interface eth0 -j MASQUERADE
/usr/sbin/iptables --append FORWARD --in-interface eth1 -j ACCEPT
/usr/sbin/iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 587 -j REDIRECT --to-port 25
/usr/sbin/iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 1912 -j DNAT --to-destination 192.168.0.58:3389

/usr/sbin/iptables -A INPUT -i eth0 -j bad_people
/usr/sbin/iptables -A FORWARD -i eth0 -j bad_people

/usr/sbin/iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
/usr/sbin/iptables -A INPUT -i lo -j ACCEPT
/usr/sbin/iptables -A INPUT -i eth1 -p tcp --syn -j ACCEPT
/usr/sbin/iptables -A INPUT -i eth1 -p udp -j ACCEPT

/usr/sbin/iptables -N logdrop
/usr/sbin/iptables -A logdrop -j LOG --log-level 6 --log-prefix SSH Break-in attempt 
/usr/sbin/iptables -A logdrop -j DROP
/usr/sbin/iptables -N checkcount
/usr/sbin/iptables -A checkcount -m recent --set
/usr/sbin/iptables -A checkcount -m recent --rcheck --hitcount 12 -j logdrop
/usr/sbin/iptables -A checkcount -j RETURN

/usr/sbin/iptables -A INPUT -p icmp --icmp-type 8 -s 192.168.0.0/24 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT

/usr/sbin/iptables -A INPUT -p tcp --syn --dport 22 -i eth0 -j checkcount
/usr/sbin/iptables -A INPUT -p tcp --syn -m limit --limit 1/s --limit-burst 3 -i eth0 --dport 22 -j ACCEPT
/usr/sbin/iptables -A INPUT -i eth0 -p tcp --syn -m multiport --dports 25,80,143,443,587,993 -j ACCEPT

Thanks for your help -- I'm locked out of remote desktop access until I can get this sorted out. I'll keep trying things on my end.

mfoley · 11-27-2017, 12:48 PM

I've tried restoring a firewall script from last March. That script definitely had been running for many month, possibly years. Same problem. It wasn't firewall settings.

I did find the problem. The gateway was normally set to the same server as the iptables/DNAT machine. Over the weekend, I changed the gateway to be an external router (though I forgot I did that). All workstations and other servers were able to use the new gateway setting. Everything worked but this DNAT. Not sure why. I've put the gateway back the way it was and the DNAT stuff is now working.

However, I'll be doing more experimentation with the alternate gateway this evening and this week, so this issue may crop up again. I do want the alternate gateway and I do want the DNAT to work! I'LL BE BACK! (maybe)

lazydog · 11-27-2017, 05:24 PM

Quote:

Originally Posted by mfoley

I do not have anything listed for that chain to keep clutter down in this posting. If I listed all members of that chain there would be an additional 970 lines in my listing!

Wow, have you thought about using or switching to ipset for this? Would clean up a lot of lines in your rules and would be a lot faster.

Quote:

Originally Posted by mfoley

I did find the problem. The gateway was normally set to the same server as the iptables/DNAT machine. Over the weekend, I changed the gateway to be an external router (though I forgot I did that). All workstations and other servers were able to use the new gateway setting. Everything worked but this DNAT. Not sure why. I've put the gateway back the way it was and the DNAT stuff is now working.

Glad to hear you figured this out.

mfoley · 11-27-2017, 09:48 PM

Quote:

Originally Posted by lazydog

Wow, have you thought about using or switching to ipset for this? Would clean up a lot of lines in your rules and would be a lot faster.

I've never heard of ipset. I'll check it out. For more complete disclosure, 795 of those DROPs in the bad_people chain are geoBlocks on IPs in China, Russia and the Ukraine. Only 178 of them are actual bad actors otherwise. These get reset at the beginning of each month so the list is close to its max size as of these postings. Speed doesn't seem to be an issue. It's a rather powerful server (4GHz Intel i7, 16G memory) and it doesn't have much to do with its time other than handling mail.

Quote:

Glad to hear you figured this out.

Well, I found the culprit for this problem, but I haven't yet solved the real problem of why the DNAT doesn't work if the gateway is not the same machine as the iptables DNATs. I'm pursing the gateway question in this thread: https://www.linuxquestions.org/quest...es-4175618196/, in which I see you are a participant. This DNAT issue seemed like a side-distraction, so I didn't want to clutter that thread with this issue -- not being sure they were related. I close this one out for now and continue posting it the other thread.

Thanks for your feedback!