Inscrutable Ubuntu as a *simple* gateway

UserLane · 02-14-2009, 06:06 PM

I've got 10+ years experience running multiple FreeBSD servers, and 25+ years with unix/xenix/venix, so I'm familiar with setting up isc-dhcpd and using unix as a router, but I'm stumped by Ubuntu!

I've installed Ubuntu (Intrepid) on a server that I use (for 5 years) to route TCP/IP packets from network 172.16.1.x to 192.168.0.x, and on to the public internet. I've installed dhcpd3 on this server and copied my original working dhcpd.conf to /etc/dhcp3/dhcpd.conf. DHCPd starts on boot and does not complain, but does not respond to DHCP requests. He claims to be listening on eth1, but I think he's either lying or maybe has a different concept of what eth1 is ....

I've configured eth0 (the external nic) in /etc/network/interfaces using dhcp (from my upstream server) as follows:

auto eth0
iface eth0 inet dhcp

He gets his ip in the range from 192.168.0.1 to 192.168.0.200 and can access the public internet without problem when booted into Ubuntu *or* his native FreeBSD.

I've configured eth1 (the internal nic) in /etc/network/interfaces staticly, as follows:

auto eth1
iface eth1 inet static
address 172.16.1.200
network 172.16.1.0
netmask 255.255.255.0
broadcast 172.16.1.255

But there are two problems:

1) When I attach a computer to a hub on the internal (eth1) network and try to configure it using DHCP, it fails (i.e. does NOT get a DHCPOFFER). But if I reboot the server into Freebsd (again, using the identical dhcpd.conf file) the connected computer can get an IP address and can route to the public internet.

2) If I configure the attached computer using a static IP (e.g. 172.16.1.201) that computer can PING the gateway (172.16.1.200) AND it can ping the "external" (eth0) nic on the server, but can NOT see the public internet or the upstream server!

Does Ubuntu require routed or gated to accomplish this seemingly trivial feat? I could not find either in the package manager, and can't find a reasonably clear reason why this is failing.

Thanks for your attention - I get that this has been asked before, about a bazillion times, but I don't seem to find a clear answer in the forums.

lane
~ trying to wrap my head around this linux "fad"

alan_ri · 02-14-2009, 06:49 PM

I was just looking at this thread; http://www.linuxquestions.org/questi...u-8.10-704691/
It might be just what you need.

UserLane · 02-14-2009, 07:31 PM

Quote:

Originally Posted by alan_ri

I was just looking at this thread; http://www.linuxquestions.org/questi...u-8.10-704691/
It might be just what you need.

The server can still see both his internal and external nics:

From ifconfig -a, I see this:

Code:

eth0      Link encap:Ethernet  HWaddr 00:13:72:14:c1:92  
          inet addr:192.168.0.200  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fe80::213:72ff:fe14:c192/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:25316 errors:0 dropped:0 overruns:0 frame:0
          TX packets:18198 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:27558961 (27.5 MB)  TX bytes:2413091 (2.4 MB)
          Interrupt:17 

eth1      Link encap:Ethernet  HWaddr 00:a0:cc:34:3a:82  
          inet addr:172.16.1.200  Bcast:172.16.1.255  Mask:255.255.255.0
          inet6 addr: fe80::2a0:ccff:fe34:3a82/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:197 errors:493 dropped:0 overruns:0 frame:0
          TX packets:44 errors:42 dropped:0 overruns:0 carrier:40
          collisions:0 txqueuelen:1000 
          RX bytes:20993 (20.9 KB)  TX bytes:6062 (6.0 KB)
          Interrupt:16 Base address:0xdc00 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host

And he can ping the computer connected on his internal nic hub, when the other computer is configured with a static IP.

I had thought this was maybe an IPV6 issue, but now I'm not at all sure.

For now I'm using static assignments on the computers connected to the hub on the internal nic, and can deal with isc-dhcpd later. But the more pressing issue is that the server doesn't want to route from his internal nic to the public internet unless I boot him into FreeBSD.

I really appreciate your reply.

lane

chort · 02-14-2009, 07:41 PM

Did you actually enable IP forwarding?

How are you testing from the internal box to see if it can "see" the Internet (be detailed, there are subtle differences that matter).

UserLane · 02-14-2009, 08:02 PM

Quote:

Originally Posted by chort

Did you actually enable IP forwarding?

How are you testing from the internal box to see if it can "see" the Internet (be detailed, there are subtle differences that matter).

I've been checking out various sysctl variables that are familiar to me.

My understanding is that I can "echo 1>> path/to/sysctl" to turn these on immediately, and then update /etc/sysctl.conf to preserve them across boots. Please let me know if that is an incorrect assumption.

Anyway, I'm looking at these:

/proc/sys/net/ipv4/conf/eth1/forwarding
/proc/sys/net/ipv4/conf/eth0/forwarding

/proc/sys/net/ipv6/conf/eth0/forwarding
/proc/sys/net/ipv6/conf/eth1/forwarding

and

/proc/sys/net/ipv4/ip_forward

All of which are already set to "1"

I'm not sure what I did to make that happen (in FreeBSD it occurs on boot when more than one nic is detected) but this is the way they appear from the last boot, and still the internal computer can only ping as far as the external nic on the gateway, but no further. He *can* ping the upstream server and even access the public internet when I boot into FreeBSD, so the only difference is Ubuntu in the middle.

I thought maybe Ubuntu was rejecting ICMP by default (which it may be doing) but the fact is that I can't get a regular TCP packet through from the internal nic to the external nic using http, ftp, or telnet.

Thanks for your input.

lane

UserLane · 02-14-2009, 08:12 PM

I'm trying to be critical of my own posts, in order to resolve this without too much head scratching, and I realize that I hadn't even shown the route table, yet, which, of course, is about as critical as "is the power on?"

This is the route table when booted under Ubuntu:

Destination Gateway Genmask Flags Metric Ref Use Iface
172.16.1.0 * 255.255.255.0 U 0 0 0 eth1
192.168.0.0 * 255.255.255.0 U 0 0 0 eth0
link-local * 255.255.0.0 U 1000 0 0 eth0
default 192.168.0.1 0.0.0.0 UG 100 0 0 eth0

The asterisks under Gateway look peculiar, but I think that's just because I am more familiar with how BSD does things. Other than that, including the sysctl variables, and the fact that this same computer booted in place under BSD does what I want ... well ... it makes me scratch my head!

chort · 02-14-2009, 08:35 PM

How are you trying to connect to upstream hosts, though? This is why I said be specific. Are you using the IP, or the hostnames? If you're trying to connect to everything by it's DNS name it is entirely possible that DNS resolution is failing, but that you can actually pass traffic.

If you've been using hostnames to test, switch to IPs. Try to ping 4.2.2.1 from your internal machine. At the same time, run a tcpdump on eth0 and see if the traffic is actually going out. Paste the results.

UserLane · 02-14-2009, 09:19 PM

Quote:

Originally Posted by chort

How are you trying to connect to upstream hosts, though? This is why I said be specific. Are you using the IP, or the hostnames? If you're trying to connect to everything by it's DNS name it is entirely possible that DNS resolution is failing, but that you can actually pass traffic.

If you've been using hostnames to test, switch to IPs. Try to ping 4.2.2.1 from your internal machine. At the same time, run a tcpdump on eth0 and see if the traffic is actually going out. Paste the results.

Sorry for the confusion, and thank you *so* much for sticking with me on this ...

My Upstream server, the one that provides dhcp and dns to the Ubuntu middle-man, is at 192.168.0.1. I've been addressing him only by his IP address, never by his name.

And, since I use him for dns, I only ask him to resolve external names. For instance, I will confirm that /etc/resolv has "nameserver 192.168.0.1" (and only that) and then do "nslookup www.yahoo.com" or whatever. But ... fact is that I haven't gotten that far because I have a webserver on 192.168.0.1 and my test (when I boot in FreeBSD) is "lwp-request http://192.168.0.1" -- which succeeds ... but then fails when I boot the middle server into Ubuntu. I get "no route to host" when I try to configure the remote computer (my laptop) with dhcp, and "timed out" when I configure it with a static ip.

Thanks for clarifying. Here is the tcpdump -i eth1 from the middle computer (Ubuntu), aka lholcombe-desktop, which is the computer that I am trying to get to act as a gateway.

In spite of the insistance in the dump that the requesting computer (lane-laptop) is at 172.16.1.32, he is actually staticly configured at 172.16.1.201 and can ping lholcombe-desktop, who is the gateway, with internal nic (eth1) at 172.168.1.200:

/var/lib/dhcp3/dhcpd.leases *does* show lane-laptop at 172.16.1.32, but he is for 100% sure got an ip (and only one ip) of 172.16.1.201!

OMG! This is so inscrutable!

Anyway, from the laptop, staticly configured at 172.16.1.201, I did:

"lwp-request http://192.168.0.1"

and here is the result:

Code:

root@lholcombe-desktop:~# tcpdump -i eth1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 96 bytes
21:01:58.198357 IP lane-laptop.local.netbios-ns > 172.16.1.255.netbios-ns: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
21:01:58.406326 IP lholcombe-desktop.local.mdns > 224.0.0.251.mdns: 0 PTR (QM)? 255.1.16.172.in-addr.arpa. (43)
21:01:59.414344 IP lholcombe-desktop.local.mdns > 224.0.0.251.mdns: 0 PTR (QM)? 255.1.16.172.in-addr.arpa. (43)
21:02:00.478322 IP lane-laptop.local.59263 > 192.168.0.1.www: S 479857764:479857764(0) win 5840 <mss 1460,sackOK,timestamp 277376 0,nop,wscale 6>
21:02:01.422341 IP lholcombe-desktop.local.mdns > 224.0.0.251.mdns: 0 PTR (QM)? 255.1.16.172.in-addr.arpa. (43)
21:02:03.478104 IP lane-laptop.local.59263 > 192.168.0.1.www: S 479857764:479857764(0) win 5840 <mss 1460,sackOK,timestamp 278126 0,nop,wscale 6>
21:02:03.518319 IP lholcombe-desktop.local.mdns > 224.0.0.251.mdns: 0 PTR (QM)? 201.1.16.172.in-addr.arpa. (43)
21:02:03.518618 IP lane-laptop.local.mdns > 224.0.0.251.mdns: 0*- [0q] 1/0/0 (Cache flush) PTR[|domain]
21:02:03.646342 IP lholcombe-desktop.local.mdns > 224.0.0.251.mdns: 0 PTR (QM)? 251.0.0.224.in-addr.arpa. (42)
21:02:04.650335 IP lholcombe-desktop.local.mdns > 224.0.0.251.mdns: 0 PTR (QM)? 251.0.0.224.in-addr.arpa. (42)
21:02:06.654357 IP lholcombe-desktop.local.mdns > 224.0.0.251.mdns: 0 PTR (QM)? 251.0.0.224.in-addr.arpa. (42)
21:02:08.750318 IP lholcombe-desktop.local.mdns > 224.0.0.251.mdns: 0 PTR (QM)? 200.1.16.172.in-addr.arpa. (43)
21:02:08.750436 IP lholcombe-desktop.local.mdns > 224.0.0.251.mdns: 0*- [0q] 1/0/0 (Cache flush) PTR[|domain]
21:02:08.878345 IP lholcombe-desktop.local.mdns > 224.0.0.251.mdns: 0 PTR (QM)? 1.0.168.192.in-addr.arpa. (42)
21:02:09.198076 IP lane-laptop.local.netbios-dgm > 172.16.1.255.netbios-dgm: NBT UDP PACKET(138)
21:02:09.198121 IP lane-laptop.local.netbios-dgm > 172.16.1.255.netbios-dgm: NBT UDP PACKET(138)
21:02:09.478941 IP lane-laptop.local.59263 > 192.168.0.1.www: S 479857764:479857764(0) win 5840 <mss 1460,sackOK,timestamp 279626 0,nop,wscale 6>
21:02:09.882336 IP lholcombe-desktop.local.mdns > 224.0.0.251.mdns: 0 PTR (QM)? 1.0.168.192.in-addr.arpa. (42)
21:02:11.198005 IP lane-laptop.local.netbios-dgm > 172.16.1.255.netbios-dgm: NBT UDP PACKET(138)
21:02:11.886339 IP lholcombe-desktop.local.mdns > 224.0.0.251.mdns: 0 PTR (QM)? 1.0.168.192.in-addr.arpa. (42)
21:02:13.197942 IP lane-laptop.local.netbios-dgm > 172.16.1.255.netbios-dgm: NBT UDP PACKET(138)
21:02:14.477800 arp who-has lholcombe-desktop.local tell lane-laptop.local
21:02:14.477835 arp reply lholcombe-desktop.local is-at 00:a0:cc:34:3a:82 (oui Unknown)
21:02:15.197896 IP lane-laptop.local.netbios-dgm > 172.16.1.255.netbios-dgm: NBT UDP PACKET(138)
21:02:17.197843 IP lane-laptop.local.netbios-dgm > 172.16.1.255.netbios-dgm: NBT UDP PACKET(138)
21:02:17.197862 IP lane-laptop.local.netbios-ns > 172.16.1.255.netbios-ns: NBT UDP PACKET(137): REGISTRATION; REQUEST; BROADCAST
21:02:19.198046 IP lane-laptop.local.netbios-ns > 172.16.1.255.netbios-ns: NBT UDP PACKET(137): REGISTRATION; REQUEST; BROADCAST
21:02:19.198114 IP lane-laptop.local.netbios-ns > 172.16.1.255.netbios-ns: NBT UDP PACKET(137): REGISTRATION; REQUEST; BROADCAST
21:02:21.197695 IP lane-laptop.local.netbios-ns > 172.16.1.255.netbios-ns: NBT UDP PACKET(137): REGISTRATION; REQUEST; BROADCAST
21:02:21.197792 IP lane-laptop.local.netbios-ns > 172.16.1.255.netbios-ns: NBT UDP PACKET(137): REGISTRATION; REQUEST; BROADCAST
21:02:21.478628 IP lane-laptop.local.59263 > 192.168.0.1.www: S 479857764:479857764(0) win 5840 <mss 1460,sackOK,timestamp 282626 0,nop,wscale 6>
21:02:23.197657 IP lane-laptop.local.netbios-ns > 172.16.1.255.netbios-ns: NBT UDP PACKET(137): REGISTRATION; REQUEST; BROADCAST
21:02:23.197729 IP lane-laptop.local.netbios-ns > 172.16.1.255.netbios-ns: NBT UDP PACKET(137): REGISTRATION; REQUEST; BROADCAST
21:02:25.197593 IP lane-laptop.local.netbios-ns > 172.16.1.255.netbios-ns: NBT UDP PACKET(137): REGISTRATION; REQUEST; BROADCAST
21:02:25.197714 IP lane-laptop.local.netbios-dgm > 172.16.1.255.netbios-dgm: NBT UDP PACKET(138)
21:02:25.197934 IP lane-laptop.local.netbios-dgm > 172.16.1.255.netbios-dgm: NBT UDP PACKET(138)
21:02:25.197945 IP lane-laptop.local.netbios-dgm > 172.16.1.255.netbios-dgm: NBT UDP PACKET(138)
21:02:25.201516 arp who-has 172.16.1.32 tell lane-laptop.local
21:02:25.406358 IP lholcombe-desktop.local.mdns > 224.0.0.251.mdns: 0 PTR (QM)? 32.1.16.172.in-addr.arpa. (42)
21:02:26.201494 arp who-has 172.16.1.32 tell lane-laptop.local
21:02:26.414334 IP lholcombe-desktop.local.mdns > 224.0.0.251.mdns: 0 PTR (QM)? 32.1.16.172.in-addr.arpa. (42)
21:02:27.201470 arp who-has 172.16.1.32 tell lane-laptop.local
21:02:28.422348 IP lholcombe-desktop.local.mdns > 224.0.0.251.mdns: 0 PTR (QM)? 32.1.16.172.in-addr.arpa. (42)
21:02:31.201366 arp who-has 172.16.1.32 tell lane-laptop.local
21:02:32.201338 arp who-has 172.16.1.32 tell lane-laptop.local
21:02:33.201309 arp who-has 172.16.1.32 tell lane-laptop.local
21:02:36.201230 arp who-has 172.16.1.32 tell lane-laptop.local
21:02:37.201206 arp who-has 172.16.1.32 tell lane-laptop.local
21:02:38.201177 arp who-has 172.16.1.32 tell lane-laptop.local
21:02:41.201100 arp who-has 172.16.1.32 tell lane-laptop.local
21:02:42.202077 arp who-has 172.16.1.32 tell lane-laptop.local
21:02:43.201045 arp who-has 172.16.1.32 tell lane-laptop.local
21:02:45.477991 IP lane-laptop.local.59263 > 192.168.0.1.www: S 479857764:479857764(0) win 5840 <mss 1460,sackOK,timestamp 288626 0,nop,wscale 6>
21:03:12.616334 IP lane-laptop.local.mdns > 224.0.0.251.mdns: 0 PTR (QM)? _pgpkey-hkp._tcp.local. (40)
^C
54 packets captured
54 packets received by filter
0 packets dropped by kernel

UGH! I checked to make sure ad-hoc dns updates is NOT on, so I just can't figure why the middle guy is remembering lane-laptop at .32, when his request comes in on .201!

I'm going to completely remove isc-dhcpd from the Ubuntu middle man, reboot everybody, go smoke something, wipe out /var/lib/dhcp3, reconfigure the laptop to a new hostname and static address ... and then try again.

chort · 02-14-2009, 09:44 PM

It's easier to debug networking problems with tcpdump's name resolution turned off. Also, it's more interesting to see what's on the external side of the router (eth0) to see if the packets are going out. We know they're coming in, since your laptop can ping both interfaces on the router.

# tcpdump -lnpt -i eth0

Come to think of it, does the machine at 192.168.0.1 have a static route to 172.16.1.0/24? If not, it won't know where to send return traffic to.

UserLane · 02-14-2009, 10:06 PM

Quote:

Originally Posted by chort

It's easier to debug networking problems with tcpdump's name resolution turned off. Also, it's more interesting to see what's on the external side of the router (eth0) to see if the packets are going out. We know they're coming in, since your laptop can ping both interfaces on the router.

# tcpdump -lnpt -i eth0

Come to think of it, does the machine at 192.168.0.1 have a static route to 172.16.1.0/24? If not, it won't know where to send return traffic to.

But I'm giving up.

I did like I said, and removed isc-dhcpd from the middle-man, and then reconfigured both the middle man and the laptop with new static ip addresses, on the assumption that there was some sort of erroneous configuration information being preserved across boots.

But even after rebooting the whole network I still get the same indecipherable results: The laptop can "see" the middle server, and vice-versa, while the middle server can see the laptop and the internet. But the laptop can't see the upstream server *or* the internet. That'd be great, if that was the way I wanted it, but it's not ... some might call it a "feature," but I'm gonna just call it a bug.

So I'm just going to go back to what works: BSD.

Ubuntu sure is a pretty desktop, but he doesn't seem ready yet to handle network traffic. Maybe it's just intrepid ... maybe hardy can do it ... but I just don't have the luxury of tinkering with this any more.

Thanks, again, for your help, and your general contributions to the community.

BTW: 192.196.0.1 serves dhcp to the middle server (on 192.168.0.x) *and* has a static route to his internal network at 172.16.1.0/24. They've known one another for about eight years

g'night

lane

chort · 02-15-2009, 01:09 AM

I'll let you in on a not-so-secret: I actually don't like Linux very much (I much prefer OpenBSD and OS X). Despite that, I think you're giving up a bit quickly. Linux can certainly act as a simple packet-forwarding router (and serve DHCP leases, and answer DNS requests, etc). The capability is built-in to the kernel, so it's not like a certain distribution would lack this.

The packet capture you showed from the inside interface was showing TCP SYN packets going out, but no responses coming back. That's either because the box wasn't forwarding them to the next hop, or because the next hop didn't have a route back. If we would have seen a tcpdump from eth0 we could have discovered whether or not it was actually forwarding the packets, but oh well...

alan_ri · 02-15-2009, 09:04 AM

...and what I was trying to say in my first post here is that maybe a simple kernel upgrade or patch could resolve the problem because the bug/s have been reported and AFAIK all of them have been fixed.I'm talking about Intrepid.

UserLane · 02-15-2009, 01:10 PM

Quote:

Originally Posted by alan_ri

...and what I was trying to say in my first post here is that maybe a simple kernel upgrade or patch could resolve the problem because the bug/s have been reported and AFAIK all of them have been fixed.I'm talking about Intrepid.

Well, when you are right, you are right.

And you are right - I gave up too soon (but maybe that was for the best)...

I shut everyone down last night, and with fresh eyes, I went over it all again this morning.

I did the tcpdump on eth0, and did not see anything for a moment .... and then I saw a couple of RipV2 packets from 192.168.0.1 ... and then it hit me: I was *not* using static routes, instead "routed" was, for years, advertising routes between 192.168.0.x and 172.16.1.x. So I added the static route on 192.168.0.1 to 172.16.1.0/24.

Then, for good measure, I installed ssh on the Ubuntu gateway and verified that "ping" still worked from the laptop to the gateway using static routes (which they did), and I was able to ssh from the laptop to the gateway - but was still unable to get "through" the gateway.

So I checked the sysctl variables, one more time, and found that they'd all be reset to "0". So I changed net.ipv4.ip_forward to "1" in /etc/sysctl.conf, rebooted, and voila! I could get through the gateway and on to the internet!

My apologies to you, linus torvalds, and the entire community for giving up too soon.

Thanks for bearing with me on that. I'll try to "pay it forward," as I get up to speed.

lane

chort · 02-15-2009, 01:54 PM

Glad to see you got it working.