Hi. I'm jon.404, a Unix/Linux/Database/Openstack/Kubernetes Administrator, AWS/GCP/Azure Engineer, mathematics enthusiast, and amateur philosopher. This is where I rant about that which upsets me, laugh about that which amuses me, and jabber about that which holds my interest most: *nix.
Lessons Learned, Applied
I decided to take the move to Seattle as an opportunity to completely rebuild my home network and harden/update/reinforce particular areas I felt needed attention in my last configuration. A fresh install of OpenBSD-5.5 ensured that nothing from the old configuration remained (partially because I upgraded to a larger CF card, and partially because I just felt like cleaning house completely).
First off, I had a single port that was trunked to the switch, with multiple vlans running across it. I felt this was the most likely area to bottleneck during heavy internet usage across multiple vlans, so I trunked it across two interfaces (my firewall only has 4 interfaces, and I really wanted to use 4, but I have multiple ISPs as well, so I couldn't use any more (for now). Ahh well. I have a few machines in mind for when I go big).
Ok, let's configure some stuff. First, the trunk itself. My firewall has two PCI nics (re0 and rl0) and two built-in nics (fxp0 and fxp1). I've chosen to interface (haha) the built-in nics to the internal network. Once they're configured to come up on boot, let's configure the trunk:
You may ask, why not setup an ip address on the trunk? Well, I'd like to have the ips split by vlan, so I'm going to setup sub-interfaces to trunk0 (vlan1100, vlan1101, etc...) to serve as actual gateways for the various vlans. Let's do that now.
Yes, the 2000+ vlans are all /30's. I'm paranoid. Sue me.
Ok, that's enough for OpenBSD on the internal side (for now). On to the switch!
I won't lie, it's been a while since I: 1) purchased a switch, and 2) used CatOS. Those two statements should go hand in hand in your mind =)
I had to lookup how to do the CatOS side (again). Here's the basic idea:
These two ports are what I've assigned to the internal network. I have 1-12 configured for the external stuff, 13-24 configured for internal services, 25-36 configured for desktops and other end-user machines, and 37-48 configured as my "playground". Enough talk! More typing!
Alrighty, the lacp trunk should be up. I don't have any way to test as of yet (ping -I 10.0.0.1 10.1.0.1, for instance, would send internally on the firewall). Must continue on!
I set a few "public" vlans up on the switch, namely 102, 103, and 104 (ISP1, ISP2, and pfsync for when I throw a failover firewall into the mix in the future). These are setup as vlan 102 = 2/1-3 (2/4 is mirror for ISP1 to go to a snort box...not configured yet), vlan 103 = 2/5-7 (2/8 is mirror for ISP2 to go to the same snort box in the future), and vlan 104 is for pfsync (again, all planning ahead...something I didn't really do last time I designed a home network).
Just before I left San Antonio, I bought a Clear modem from a friend. I called their customer service and got it hooked up. Played around with it for a bit and noted the quirks about Clear service (not many, surprisingly, but I haven't played with it much as of yet...mostly just the lack of bridge mode). I plugged the clear modem into one of the open ports on the firewall and configured it for dhcp:
Noted that it was working as planned, then I ran a few dumb speed tests:
So far, so good. Now I need to setup the other ISP. Hrmmm...wife is still busy working. Nevertheless, I shall carry on. The DSL modem has multiple ports, so I swipe one and plug it into rl0.
Many of you are going "Oh for the love of...wtf are you DOING?!" right now. The big issue is that the dhclient instances will conflict and setup routing in strange ways. But I failed to note that and continued on to...
Edit /etc/sysctl.conf to uncomment the appropriate lines for the above controls. Now I can define multiple default gateways that the OpenBSD firewall will use in an equal cost multipath configuration. This is where I hit a bit of unknown, as I've never tried this with two routes configured via **dhcp**. I'm not entirely certain this is going to do what I expect...I may have to figure out how to hard-code addresses on the modems (neither of which support bridge mode...dang!).
Oh, and before I forget...
I'm a sucker for things being organized, what can I say? I like the interface descriptions:
Once I start adding in additional BSD and Linux machines, the CDP output will be tremendously helpful (on both the switch AND the BSD or Linux machines).
At this point, I *should* be able to ping -I $ext1_ip 8.8.8.8 and ping -I $ext2_ip 8.8.8.8, but alas, my config is broken. I sometimes can ping from re0, sometimes from rl0, but never from them at the same time. Double checking routes shows that I have multiple default gateways per multipath, and I deleted /etc/mygate, so that shouldn't be causing issues. And yet, I cannot use both ISPs simultaneously.
This, as they say, is a showstopper. I'll need to get this resolved before I can migrate everything behind the firewall (I literally have an 8 port dumb switch plugged into the DSL modem and everyone is running off of that...sigh).
A bit of googling later, I noticed that route priority needs to be explicitly set. AAARRRGGGHHH!!!
Alrighty, set both uplinks to priority 8. Now to test again. Still failing. Another bit of googling later, I see that others have had to explicitly disable routes in dhclient.conf. Ahh, that makes sense... Another round of testing, and it appear (sorta) to work. I say sorta, because the way multipath works, it round robins connections to each ISP, so if I hit www.linkedin.com one time, it may use ISP1, a second time may use ISP2, but at any given point www.linkedin.com will only be reachable via *one* of the ISPs.
This poses an issue. I'd like to configure ifstated to do interface state monitoring, so I can dynamically add/remove routes and firewall rules as ISPs have their usual issues. The end goal, afterall, is to have my traffic routed over ISP1, and my wife's traffic routed over ISP2, but in the event of a failure, we'll share the up ISP. I'm really close...now I just need to put the finishing touches on the configuration.
First, I need something to test that can be reached directly over each ISP link...a test that can assure me that traffic is indeed crossing the link and coming back. To accomplish this, I need two IPs out on the internet that can only be reached via one of the ISPs each. Two static routes later, I have what I need:
Now whenever I ping 8.8.8.8, it always uses ISP1, and when I ping 8.8.4.4, it always uses ISP2. Now I can have a little fun with ifstated.conf. I snagged a decent config from here to jumpstart my ifstated configuration. The only real edit I performed was to ping 8.8.8.8 from ISP1 and 8.8.4.4 from ISP2.
After testing the ifstated.conf config and giving it my blessing, I decided to hack together a quick pf config that would prove that this configuration would accomplish what I want. Here's the outline:
Add a few rdr-to rules in /etc/netconf/$person-rules, and we're all set. Now all traffic from my machine will default to ISP1, all of my wife's traffic will default to ISP2, and everyone else will get something in between.
At this point I need to edit my ifstated ruleset to remove the anchor rules when an ISP drops (defaulting everyone to the ISP that is still up), but that's enough for today.
I need sleep now.
First off, I had a single port that was trunked to the switch, with multiple vlans running across it. I felt this was the most likely area to bottleneck during heavy internet usage across multiple vlans, so I trunked it across two interfaces (my firewall only has 4 interfaces, and I really wanted to use 4, but I have multiple ISPs as well, so I couldn't use any more (for now). Ahh well. I have a few machines in mind for when I go big).
Ok, let's configure some stuff. First, the trunk itself. My firewall has two PCI nics (re0 and rl0) and two built-in nics (fxp0 and fxp1). I've chosen to interface (haha) the built-in nics to the internal network. Once they're configured to come up on boot, let's configure the trunk:
Code:
echo "up" > /etc/hostname.fxp0 echo "up" > /etc/hostname.fxp1 echo "trunkproto lacp trunkport fxp0 trunkport fxp1" > hostname.trunk0 sh /etc/netstart
Code:
echo "inet 10.0.0.1 255.255.0.0 NONE vlan 1100 vlandev trunk0 description management" > /etc/hostname.vlan1100 echo "inet 10.1.0.1 255.255.0.0 NONE vlan 1101 vlandev trunk0 description internal" > /etc/hostname.vlan1101 echo "inet 10.2.0.1 255.255.255.252 NONE vlan 2000 vlandev trunk0 description jonathon_workstation" > /etc/hostname.vlan2000 echo "inet 10.2.0.5 255.255.255.252 NONE vlan 2001 vlandev trunk0 description jody_desktop" > /etc/hostname.vlan2001 echo "inet 10.2.0.9 255.255.255.252 NONE vlan 2002 vlandev trunk0 description jayda_desktop" > /etc/hostname.vlan2002 echo "inet 10.2.0.13 255.255.255.252 NONE vlan 2003 vlandev trunk0 description zach_desktop" > /etc/hostname.vlan2003 sh /etc/netstart
Ok, that's enough for OpenBSD on the internal side (for now). On to the switch!
I won't lie, it's been a while since I: 1) purchased a switch, and 2) used CatOS. Those two statements should go hand in hand in your mind =)
I had to lookup how to do the CatOS side (again). Here's the basic idea:
Code:
show channelprotocol set channelprotocol lacp 2 set port lacp-channel 2/13-14
Code:
set port lacp-channel 2/13-14 mode active set trunk 2/13 desirable dot1q
I set a few "public" vlans up on the switch, namely 102, 103, and 104 (ISP1, ISP2, and pfsync for when I throw a failover firewall into the mix in the future). These are setup as vlan 102 = 2/1-3 (2/4 is mirror for ISP1 to go to a snort box...not configured yet), vlan 103 = 2/5-7 (2/8 is mirror for ISP2 to go to the same snort box in the future), and vlan 104 is for pfsync (again, all planning ahead...something I didn't really do last time I designed a home network).
Just before I left San Antonio, I bought a Clear modem from a friend. I called their customer service and got it hooked up. Played around with it for a bit and noted the quirks about Clear service (not many, surprisingly, but I haven't played with it much as of yet...mostly just the lack of bridge mode). I plugged the clear modem into one of the open ports on the firewall and configured it for dhcp:
Code:
echo "dhcp" > /etc/hostname.re0 sh /etc/netstart ping www.google.com
Code:
lynx http://mirrors.syringanetworks.net/pub/OpenBSD/5.5/i386/base55.tgz
Code:
echo "dhcp" > /etc/hostname.rl0 sh /etc/netstart ping www.google.com
Code:
sysctl net.inet.ip.forwarding=1 sysctl net.inet.ip.multipath=1
Oh, and before I forget...
Code:
pkg_add ladvd && echo "pkg_scripts=ladvd" >> /etc/rc.conf.local && /etc/rc.d/ladvd start
Code:
2/2 fw.j3z.local rl0 OpenBSD 5.5 i386 2/6 fw.j3z.local re0 OpenBSD 5.5 i386 2/13 fw.j3z.local fxp0 OpenBSD 5.5 i386 2/14 fw.j3z.local fxp1 OpenBSD 5.5 i386
At this point, I *should* be able to ping -I $ext1_ip 8.8.8.8 and ping -I $ext2_ip 8.8.8.8, but alas, my config is broken. I sometimes can ping from re0, sometimes from rl0, but never from them at the same time. Double checking routes shows that I have multiple default gateways per multipath, and I deleted /etc/mygate, so that shouldn't be causing issues. And yet, I cannot use both ISPs simultaneously.
This, as they say, is a showstopper. I'll need to get this resolved before I can migrate everything behind the firewall (I literally have an 8 port dumb switch plugged into the DSL modem and everyone is running off of that...sigh).
A bit of googling later, I noticed that route priority needs to be explicitly set. AAARRRGGGHHH!!!
Alrighty, set both uplinks to priority 8. Now to test again. Still failing. Another bit of googling later, I see that others have had to explicitly disable routes in dhclient.conf. Ahh, that makes sense... Another round of testing, and it appear (sorta) to work. I say sorta, because the way multipath works, it round robins connections to each ISP, so if I hit www.linkedin.com one time, it may use ISP1, a second time may use ISP2, but at any given point www.linkedin.com will only be reachable via *one* of the ISPs.
This poses an issue. I'd like to configure ifstated to do interface state monitoring, so I can dynamically add/remove routes and firewall rules as ISPs have their usual issues. The end goal, afterall, is to have my traffic routed over ISP1, and my wife's traffic routed over ISP2, but in the event of a failure, we'll share the up ISP. I'm really close...now I just need to put the finishing touches on the configuration.
First, I need something to test that can be reached directly over each ISP link...a test that can assure me that traffic is indeed crossing the link and coming back. To accomplish this, I need two IPs out on the internet that can only be reached via one of the ISPs each. Two static routes later, I have what I need:
Code:
route add 8.8.8.8 192.168.0.1 route add 8.8.4.4 192.168.15.1
After testing the ifstated.conf config and giving it my blessing, I decided to hack together a quick pf config that would prove that this configuration would accomplish what I want. Here's the outline:
Code:
anchor "jody" load anchor jody from "/etc/netconf/jody-rules" anchor "jonathon" load anchor jonathon from "/etc/netconf/jonathon-rules"
At this point I need to edit my ifstated ruleset to remove the anchor rules when an ISP drops (defaulting everyone to the ISP that is still up), but that's enough for today.
I need sleep now.
Total Comments 0
