Problem with port forwarding (NAT) on FC 5
 

As what I want to do is basic, it should be somewhere in this forum, but unfortunately I was unable to find it. So I apologize beforehand.
First of all, let me introduce myself. I am a physics professor who is in charge of our departmental network. This happened because I like to do it and there was no one else to do it. For this reason, although I have been using computers before Linux even existed, I consider myself a beginner. I also a complete newbie on forums.
I have recently updated a server that was running RedHat 9 to FC 5. Among other things I did NAT 9 using IPTABLES). I configured it the same way as before but now it is unstable: it stops working, but if I restart the network service it works again. The only information that I saw and though would be useful was this type of message in the log but I don’t what they mean.
Jul 19 17:38:14 newton kernel: IN=eth1 OUT=eth1 SRC=200.221.2.45 DST=192.168.30.14 LEN=84 TOS=0x00 PREC=0x00 TTL=53 ID=36897 PROTO=ICMP TYPE=0 CODE=0 ID=14598 SEQ=40792
newton is the server name. eth0 is the external interface and eth1 is the internal.
Any help would be greatly appreciated.

That is info about a packet that a rule in netfilter (iptables) said to log. I believe icmp type=0 is an echo reply.

I don't know what this means.

Could you give more informatioon.? What do you mean it stops working?

---

If they are not too long perhaps you could post your firewall rules. If you set them up using a script, posting that would be great. Otherwise you can save the output of iptables-save to a file:

iptables-save > filename

Thank you very much for your response.
Here is a non ICMP example of a message:
Jul 21 10:01:27 newton kernel: IN=eth1 OUT=eth1 SRC=203.178.137.175 DST=192.168.30.11 LEN=64 TOS=0x00 PREC=0x00 TTL=47 ID=42991 DF PROTO=TCP SPT=80 DPT=56152 WINDOW=65535 RES=0x00 ACK SYN URGP=0

If I understand this log correctly it is saying that the packet came in through eth1. As the source is an internet IP it should have come in through eth0.
I am sorry about this typo:
I meant

Among other things it did NAT (using IPTABLES).

By "stop working" I mean NAT no longer works and therefore the internet is no longer accessible from the LAN. To find out when the problem happens, I ping a computer on the university computer center every minute and when this type of log appear (lots of them)the ping stops getting a response.
The script I use is below. I didn't write it. I found it on the internet some year ago. I have omitted the comments and blank lines.

IPTABLES=/sbin/iptables
DEPMOD=/sbin/depmod
MODPROBE=/sbin/modprobe
EXTIF="eth0"
INTIF="eth1"
$IPTABLES -F
$IPTABLES -t nat -F
$DEPMOD -a
$MODPROBE ip_tables
$MODPROBE ip_conntrack
$MODPROBE ip_conntrack_ftp
$MODPROBE ip_conntrack_irc
$MODPROBE iptable_nat
$MODPROBE ip_nat_ftp
$IPTABLES -P INPUT ACCEPT
$IPTABLES -F INPUT
$IPTABLES -P OUTPUT ACCEPT
$IPTABLES -F OUTPUT
$IPTABLES -P FORWARD DROP
$IPTABLES -F FORWARD
$IPTABLES -t nat -F
$IPTABLES -A FORWARD -i $EXTIF -o $INTIF -m state --state ESTABLISHED,RELATED -j ACCEPT
$IPTABLES -A FORWARD -i $INTIF -o $EXTIF -j ACCEPT
$IPTABLES -A FORWARD -j LOG
$IPTABLES -t nat -A POSTROUTING -s 192.168.30.0/24 -o eth0 --j SNAT --to 200.17.xxx.xxx

You understand correctly. For our purposes, the most interesting info in the log entries are probably

• IN=
• OUT=
• SRC=
• DST=

which are the input and output interfaces, and soure and destination IP addresses. What struck me as really strange with both packets you've listed is that the input and output interface is the same. While I suppose that is possible, TMK it is quite unusual.

You may need to help me with whether the IP addresses make sense. I gather your LAN is 192.168.30.0/24. Are 200.221.2.45 and 203.178.137.175 internet addresses? TYK, do they make sense? Perhaps the first address (the echo reply) was your university computer center?

This may sound like a strange question, but is it possible that another computer on your LAN has an internet connection? Possibly through a dialup modem? (I am just trying to make sense of a strange situation.)

Your firewall logs packets that get dropped by the FORWARD chain, and nothing else, so this makes sense. Another useful piece of information is whether you can successfully ping your computer center from this computer. Also check whether the computers on your LAN can ping this computer and vice versa.

-----

So this is strange. I'd like to get some more info from you.

The ifconfig command w/o options will list all active interfaces and route -n lists the routing table. Would you run these and redirect their outputs to files? (These commands can be run by normal users, but they may not be in a normal user's PATH. If you have trouble, just run them as root.) Do this both while the system is still running normally and after it stops working (to different file names!) and see if there is any change. The ifconfig command lists some statistics (among other things) that I would expect to change, but that is the only difference I would expect.

Also run the following command (you probably want to copy and paste it -- it may look strange to you but it is correct) as root before and after your problem occurs, redirecting the results to (different) filenames (instead of junk as I show).

Compare these two files with the diff command to see if there are any differences. The iptables-save command lists the firewall rules and the sed command removes comments and packet/byte counts that are expected to change. Again, I would expect no difference.

-----

This is (I hope) a little off topic for this thread, but I noticed that (unless you are behind a firewall, "router," or NAT device) you are allowing unrestricted Internet (or whatever network eth0 is attached to) access to this box. Below I have listed your script with some changes/additions (in red) that will restrict this access some.

A proper treatment would be more involved. I believe the default installation of FC installs a firewall. It might be instructive for you to look at it. Depending upon how you have implimented your script, this might still be avaible on your machine after you have booted but before you run your script. Your script flushes all existing firewall rules. There are also tools on the Internet you can dowload to help you build firewalls.

You also might want to look at the iptables man page to understand what these commands do.

EDIT: I made a horrible mistake in the script on my sample line for allowing ssh. The line referencing --dport 22 should not contain the options "-m state --state ESTABLISHED,RELATED"

Thank you very much for your reply.
I also think it is very odd that the packets are coming in through the wrong interface. All logs entries in this regard have this characteristic: the input and output interfaces are the same.
Regarding the IPs 200.221.2.45, it is from www.uol.com.br, a well known ISP in Brazil that I was pinging as a test. But I have no idea why someone would want to reach 203.178.137.175. The IPs from our university are all 200.17.xxx.xxx.
Although theoretically possible it would difficult to do and wouldn’t make any sense. I doubt very much that anyone is doing that. By the other hand, there is a fact regarding the physical structure of our network that I did not mention and that may be relevant: the cables from both interfaces in this server go to the same switch. This switch is also shared by other servers and LAN computers.
Yes, it works both ways in all cases.
There is another odd things I noticed. The route command doesn’t show the loopback (lo). I though I had made some wrong configuration, but when I installed FC 5 in another machine and only made the installation time configuration it was the same.
That is exactly what happened.
The other diff commands produced no output.
Thank you very much for the suggestions on the firewall. There is a GUI interface to setup iptables rules called firestarter or something like it. Do you recommend it? This machine provides mail, http services for the internet and also pop3 and IMAP for the intranet. Will the rules you suggested still allow these services?

this is not good... in fact, it's very bad... =(

i do not... i think you are already doing this the best way - with an iptables script...

no, but he provided an example of how you would add rules for these services...

if i may, allow me to tweak the script even further... this script provides better flushing/deleting of the chains, and it has rules for the four services you mentioned...


EDIT: you should also try activating rp_filter in the kernel, so that your box does source validation by reversed path... just append a line like this to the script:or you could use your "/etc/sysctl.conf" for that instead, if you want (i assume that's where you are activating your forwarding from, right?)...

in any case, you really should physically separate your networks if possible... having both cards go to the same switch is not only a huge security risk, it also might create weird network issues (ahem), and it kind of defeats the entire purpose of having a dual-zone firewall...

Yes, I suspect this is the source of the problem. I am glad win32sux chimed in here because this is beginning to get to the edge of my knowledge/competence. His suggestion about serparating the networks, if possible, is a good one. If it is not possible, the only options I know of are to try to figure out/correct why the switch is routing the I-net packets to eth1, or to change the script to where it doesn't matter.

Win32sux has considerably cleaned up the script (thanks win32sux) and corrected my omission of providing for the loopback interface. He also added the lines for the services you are running and changed the FORWARDing rule so that it no longer checks interfaces. Depending on how connection tracking works, this alone may solve your problem. (win32sux or anybody else -- do you know if the state module requires a match on the interface? I.e. can a packet leave one interface and the response come back in a different interface, and still match?) If not, the only other solution I know at the firewall level is to omit that rule entirely, and let everything through (ugh!).

The only problem I have with this last script relates to the interface/switch problem. As long as this situation persists, the interface is not a reliable way to determine where a packet comes from. So perhaps it would be better to replace "-i $EXTIF" with "-s "EXTIF_IP" and "-i $INTIF" with "-s $INTIF_IP." Not entirely satisfactory, but it would work around the current problem.

Up until about a week ago I thought the loopback entry in the routing table was needed. I have now found out it is not -- at least not under all circumstances. Can somebody with more knowledge clarify this?

These differences in ifconfig are the statistics I mentioned. The essential stuff did not change. When diff produces no output it means the files are identical.

I have no experience with firestarter so I can't recommend one way or the other. The firewalls I have used have been from scripts similar to yours. They have evolved considerably over time, but my original firewall was based on the book Building Linux And Open BSD Firewalls by Wes Sonnenreich & Tom Yates. This is a decent book but it is old enough to be based on ipchains instead of iptables. While these are similar, there are definite differences, particularly in SNAT/MASQUERADE, the FORWARD chain, referencing interfaces, and logging. Ipchains also doesn't provide stateful inspection (--state ).

Even if you go with your own script as win32sux suggests, where firestarter might be useful is to show you how to do certain things. In other words, you could play with it on a test system and study the firewalls it produces. One of the things Sonnenreich & Yates emphasized, which I agree with, is to get maximum benefit from a firewall, the administrator needs to understand what the firewall is doing and why.

In our scripts, win32sux and I both omitted things like checking for reserved addresses that shouldn't be coming in from the Internet. I would imagine (certainly hope) that firestarter would address these sort of things. But I don't know that you can reliably do a lot of this as long as packets are coming in on the wrong interface.

but packets in the FORWARD and INPUT chains won't ever have those IPs as their source addresses, so the packets wouldn't match those rules...

matches like "-s $INTIF_IP" and "-s $EXTIF_IP" would only work in things like the OUTPUT chain (which is already set to ACCEPT so they would be pointless)...

perhaps you meant "-d $INTIF_IP" and "-d $EXTIF_IP" instead?? either way, i think the rp_filter should be taking care of that already...

Oops. What I actually meant was replacing "-i $EXTIF" with "! -s $INTIF_NET" and replacing "-i $INTIF" with "-s $INTIF_NET" The packets on the INPUT and FORWARD chains should still have their LAN source addresses, even if in the second case they end up being SNATed. And the packets from the I-net should always retain their source address.

As I read your script a little closer, I believe the only times you use -i $INTIF" you do so in conjunction with "-s $INTIF_NET," so my suggestion on these would just amount to dropping the "-i $INTIF" And I think the only time you use "-i $EXTIF" is in conjunction with the web and SMTP services. He may actually want these available from the LAN also. In which case just drop the "-i $EXTIF" altogether. If not, replacing it with "! -s $INTIF_NET" would be more reliable while he has problems with the I-net coming in on the wrong interface.

It's been a while since I've thought about rp_filter. I'll read up on it.

In any case, thanks for the script cleanup and the rest of your post.

no need to replace it - you can use both matches simultaneously... the only reason i didn't do that was because rp_filter takes care of that, but doing it won't hurt... example:
no need for that, it's best to use both... take a close look at the script and you'll see that's precisely what i did... the packet will only match if it's coming into $INTIF and has a source address in the $INTIF_NET subnet...

dropping it would make the rule less effective...

well, i based the script on what he said exactly (nothing more, nothing less)... i quote:i respectfully disagree... IMHO specifying the interface which the rule is intended for is a good idea... there is no reason at all why packets which come into the LAN interface with Internet source IPs should be sent to ACCEPT... on the other hand, having the inverted source match (-s ! $INTIF_NET) on "-i $EXTIF" rules would be okay, as shown in my --dport 80 example above...

no problem... i really enjoy having iptables discussions...

Thanks a lot to win32sux and blackhole54 for your extremely useful comments.
One thing still puzzles me. I have another server that uses RedHat 9 and the same iptables script and NAT works here. Any ideas why?
I went to check on the cable connection to the switches and I found out that the internal and external interfaces are in fact connected to different switches, but they are interconnected. There is no immediate solution for situation but, as we are buying another switch, I believe it will be sorted out soon.
Regarding sysctl.conf, it contains, among other lines,
Is that OK?
As soon and I run win32sux's script, NAT stopped working and even restarting the network would make it work again. I had to restart iptables. Lots of lines like the ones below appeared to the log.
Jul 25 17:43:49 newton kernel: INPUT DROP: IN=eth1 OUT= MAC=xx:xx:xx:xx:xx:xx:xx:01:01:c7:25:63:xx:xx SRC=192.168.30.118 DST=192.168.30.15 LEN=48 TOS=0x00 PREC=0x00 TTL=128 ID=31890 DF PROTO=TCP SPT=2159 DPT=8080 WINDOW=65535 RES=0x00 SYN URGP=0
Jul 25 17:43:50 newton kernel: INPUT DROP: IN=eth1 OUT= MAC=xx:xx:xx:xx:xx:xx:xx:50:2c:06:94:c7:xx:xx SRC=192.168.30.11 DST=192.168.30.15 LEN=128 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=600 DPT=610 LEN=108
Any suggestions?

about the NAT not working with my script... are there any FORWARD DROPs in the logfile?? did you double-check that you have all the variables properly set?? if the firewall is filtering something in the FORWARD chain it should show-up in the logfile...

Did you remember to change the line

to reflect your actual IP address? If you have and it still doesn't work, you should look for log entries containing "FOWARD DROP: "

These reflect the increased security of your firewall. In these cases it blocked packets sent to it at ports tcp/8080 and udp/610. If your log is getting swamped by dropped packet entries, you can limit the number of logged packets by appending something like

to the the command(s) with -j LOG in it(them). Of course, you will then lose information about the packets that were blocked but not logged. See the iptables man page for more info.

I really don't know what's going on, so I'll just ramble a little in case there is anything worthwhile in my thoughts. ;) My first inclination was to blame an upstream switch or router which seems to be routing packets to the wrong interface, and to think it had nothing to do with your server. But the fact that you can reliably (temporarily) get things working again by issuing a network restart would seem to indicate it is not that simple. As does the fact that apparently doing exactly the same thing on RH9 doesn't cause the problem.

The only thing I can think of is to investigate the ARP (address resolution protocol) traffic on your server and see if you can figure out anything from it. This really isn't a solid plan. I am just introducing you to another tool that may allow you to accidently stumble onto the answer. ("Chance favors the prepared mind." --Louis Pasteur) Packets are sent to ethernet cards using the cards' MAC address. ARP is the means by which one computer discovers the MAC address of the computer/interface associated with a particular IP address. I'll illustrate with an example that will introduce you to the tcpdump command. (I've obfuscated the MAC addresses with x's.)

This was run on the machine with address 192.168.1.5. I issued the tcpdump as you can see, telling it to display arp messages numerically. In another window I then pinged 192.168.1.3 so that both computers would have to discover the other's MAC address. The first message is192.168.1.5 asking for 192.168.1.3's address and the second message is the reply. The third message 192.168.1.3 asking for 192.168.1.5's address and the fourth message is the reply.

Note that a computer will only remember a MAC address for a short period of time before it again has to poll.

This was a quick experiment so that I could introduce you to tcpdump to show you how this works. I had a couple problems I didn't understand. Dropping the -n causes tcpdump to attempt to show you host names instead of addresses. The option "-i eth0" causes it to only list that one interface. However, when I tried either of these, I only got the first two lines and not the second two. I don't know why.

Thats all that I can think of. If you want, play with this and see what you can figure out. All I can do is wish you good luck and hope you get your new equipment soon.

EDIT: The ifconfig command will show you the MAC address of both of your ethernet cards.

I am sorry about saying that NAT was not working. I was not quite correct. I run your script again and found out that from a computer in the internal network: a ping to a computer in the computer center stopped working; I can no longer make a ssh to this server (it neither accepts nor rejects it, there is simply no response); I cannot open its home page. By the other hand, using a direct connection I can open an outside page on the browser and the ssh session I had continued to work.
There were no FORWARD DROPs in the logfile.
EXTIF_IP= was properly set.