Help answer threads with 0 replies.
Go Back > Blogs > Musings on technology, philosophy, and life in the corporate world
User Name


Hi. I'm a Unix Administrator, mathematics enthusiast, and amateur philosopher. This is where I rant about that which upsets me, laugh about that which amuses me, and jabber about that which holds my interest most: Unix.
Rate this Entry

Lessons Learned, Applied

Posted 05-29-2014 at 03:25 AM by rocket357
Updated 05-29-2014 at 10:16 AM by rocket357

I decided to take the move to Seattle as an opportunity to completely rebuild my home network and harden/update/reinforce particular areas I felt needed attention in my last configuration. A fresh install of OpenBSD-5.5 ensured that nothing from the old configuration remained (partially because I upgraded to a larger CF card, and partially because I just felt like cleaning house completely).

First off, I had a single port that was trunked to the switch, with multiple vlans running across it. I felt this was the most likely area to bottleneck during heavy internet usage across multiple vlans, so I trunked it across two interfaces (my firewall only has 4 interfaces, and I really wanted to use 4, but I have multiple ISPs as well, so I couldn't use any more (for now). Ahh well. I have a few machines in mind for when I go big).

Ok, let's configure some stuff. First, the trunk itself. My firewall has two PCI nics (re0 and rl0) and two built-in nics (fxp0 and fxp1). I've chosen to interface (haha) the built-in nics to the internal network. Once they're configured to come up on boot, let's configure the trunk:

echo "up" > /etc/hostname.fxp0
echo "up" > /etc/hostname.fxp1
echo "trunkproto lacp trunkport fxp0 trunkport fxp1" > hostname.trunk0
sh /etc/netstart
You may ask, why not setup an ip address on the trunk? Well, I'd like to have the ips split by vlan, so I'm going to setup sub-interfaces to trunk0 (vlan1100, vlan1101, etc...) to serve as actual gateways for the various vlans. Let's do that now.

echo "inet NONE vlan 1100 vlandev trunk0 description management" > /etc/hostname.vlan1100
echo "inet NONE vlan 1101 vlandev trunk0 description internal" > /etc/hostname.vlan1101
echo "inet NONE vlan 2000 vlandev trunk0 description jonathon_workstation" > /etc/hostname.vlan2000
echo "inet NONE vlan 2001 vlandev trunk0 description jody_desktop" > /etc/hostname.vlan2001
echo "inet NONE vlan 2002 vlandev trunk0 description jayda_desktop" > /etc/hostname.vlan2002
echo "inet NONE vlan 2003 vlandev trunk0 description zach_desktop" > /etc/hostname.vlan2003
sh /etc/netstart
Yes, the 2000+ vlans are all /30's. I'm paranoid. Sue me.

Ok, that's enough for OpenBSD on the internal side (for now). On to the switch!

I won't lie, it's been a while since I: 1) purchased a switch, and 2) used CatOS. Those two statements should go hand in hand in your mind =)

I had to lookup how to do the CatOS side (again). Here's the basic idea:

show channelprotocol
set channelprotocol lacp 2
set port lacp-channel 2/13-14
These two ports are what I've assigned to the internal network. I have 1-12 configured for the external stuff, 13-24 configured for internal services, 25-36 configured for desktops and other end-user machines, and 37-48 configured as my "playground". Enough talk! More typing!

set port lacp-channel 2/13-14 mode active
set trunk 2/13 desirable dot1q
Alrighty, the lacp trunk should be up. I don't have any way to test as of yet (ping -I, for instance, would send internally on the firewall). Must continue on!

I set a few "public" vlans up on the switch, namely 102, 103, and 104 (ISP1, ISP2, and pfsync for when I throw a failover firewall into the mix in the future). These are setup as vlan 102 = 2/1-3 (2/4 is mirror for ISP1 to go to a snort box...not configured yet), vlan 103 = 2/5-7 (2/8 is mirror for ISP2 to go to the same snort box in the future), and vlan 104 is for pfsync (again, all planning ahead...something I didn't really do last time I designed a home network).

Just before I left San Antonio, I bought a Clear modem from a friend. I called their customer service and got it hooked up. Played around with it for a bit and noted the quirks about Clear service (not many, surprisingly, but I haven't played with it much as of yet...mostly just the lack of bridge mode). I plugged the clear modem into one of the open ports on the firewall and configured it for dhcp:

echo "dhcp" > /etc/hostname.re0
sh /etc/netstart
Noted that it was working as planned, then I ran a few dumb speed tests:

So far, so good. Now I need to setup the other ISP. Hrmmm...wife is still busy working. Nevertheless, I shall carry on. The DSL modem has multiple ports, so I swipe one and plug it into rl0.

echo "dhcp" > /etc/hostname.rl0
sh /etc/netstart
Many of you are going "Oh for the love are you DOING?!" right now. The big issue is that the dhclient instances will conflict and setup routing in strange ways. But I failed to note that and continued on to...

sysctl net.inet.ip.forwarding=1
sysctl net.inet.ip.multipath=1
Edit /etc/sysctl.conf to uncomment the appropriate lines for the above controls. Now I can define multiple default gateways that the OpenBSD firewall will use in an equal cost multipath configuration. This is where I hit a bit of unknown, as I've never tried this with two routes configured via **dhcp**. I'm not entirely certain this is going to do what I expect...I may have to figure out how to hard-code addresses on the modems (neither of which support bridge mode...dang!).

Oh, and before I forget...

pkg_add ladvd && echo "pkg_scripts=ladvd" >> /etc/rc.conf.local && /etc/rc.d/ladvd start
I'm a sucker for things being organized, what can I say? I like the interface descriptions:

2/2    fw.j3z.local              rl0           OpenBSD 5.5 i386
2/6    fw.j3z.local              re0           OpenBSD 5.5 i386
2/13   fw.j3z.local              fxp0          OpenBSD 5.5 i386
2/14   fw.j3z.local              fxp1          OpenBSD 5.5 i386
Once I start adding in additional BSD and Linux machines, the CDP output will be tremendously helpful (on both the switch AND the BSD or Linux machines).

At this point, I *should* be able to ping -I $ext1_ip and ping -I $ext2_ip, but alas, my config is broken. I sometimes can ping from re0, sometimes from rl0, but never from them at the same time. Double checking routes shows that I have multiple default gateways per multipath, and I deleted /etc/mygate, so that shouldn't be causing issues. And yet, I cannot use both ISPs simultaneously.

This, as they say, is a showstopper. I'll need to get this resolved before I can migrate everything behind the firewall (I literally have an 8 port dumb switch plugged into the DSL modem and everyone is running off of that...sigh).

A bit of googling later, I noticed that route priority needs to be explicitly set. AAARRRGGGHHH!!!

Alrighty, set both uplinks to priority 8. Now to test again. Still failing. Another bit of googling later, I see that others have had to explicitly disable routes in dhclient.conf. Ahh, that makes sense... Another round of testing, and it appear (sorta) to work. I say sorta, because the way multipath works, it round robins connections to each ISP, so if I hit one time, it may use ISP1, a second time may use ISP2, but at any given point will only be reachable via *one* of the ISPs.

This poses an issue. I'd like to configure ifstated to do interface state monitoring, so I can dynamically add/remove routes and firewall rules as ISPs have their usual issues. The end goal, afterall, is to have my traffic routed over ISP1, and my wife's traffic routed over ISP2, but in the event of a failure, we'll share the up ISP. I'm really I just need to put the finishing touches on the configuration.

First, I need something to test that can be reached directly over each ISP link...a test that can assure me that traffic is indeed crossing the link and coming back. To accomplish this, I need two IPs out on the internet that can only be reached via one of the ISPs each. Two static routes later, I have what I need:
route add
route add
Now whenever I ping, it always uses ISP1, and when I ping, it always uses ISP2. Now I can have a little fun with ifstated.conf. I snagged a decent config from here to jumpstart my ifstated configuration. The only real edit I performed was to ping from ISP1 and from ISP2.

After testing the ifstated.conf config and giving it my blessing, I decided to hack together a quick pf config that would prove that this configuration would accomplish what I want. Here's the outline:

anchor "jody"
load anchor jody from "/etc/netconf/jody-rules"
anchor "jonathon"
load anchor jonathon from "/etc/netconf/jonathon-rules"
Add a few rdr-to rules in /etc/netconf/$person-rules, and we're all set. Now all traffic from my machine will default to ISP1, all of my wife's traffic will default to ISP2, and everyone else will get something in between.

At this point I need to edit my ifstated ruleset to remove the anchor rules when an ISP drops (defaulting everyone to the ISP that is still up), but that's enough for today.

I need sleep now.
Posted in Uncategorized
Views 260 Comments 0
« Prev     Main     Next »
Total Comments 0




All times are GMT -5. The time now is 07:43 AM.

Main Menu
Write for LQ is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration