Hello Everybody,

this is the first I am dealing with this sort of virtual networking and virtualization

On an Arch linux Host I am running QEMU, KVM and virt-manager.

I have:

- a VM to host and run an instance of pfSense.
- this VM is connected via a MACVTAP device to enp13s0

I want:

- connect more VMs to this pfSense Instance, such that they are plugged into a second port.

Let me try to draw a picture:

Here are the things that I am stuck with:

1. What sort of Interface do I need to create for the second NIC of the psSense VM?
2. How should that be connected to the Bridge

In previous Setups I was able to connect multiple VMs to a bridge, such those got an IP from my router, but now they should get the IP from the DHCP running within the pfSense VM. Basically I want to put all VMs (execept the pfSense one) into a network that governed locally.

The virt-manager Interface allows to choose between:
- default Network
- bridge mode
- macvtap device
for a network controller, or directly edit the xml configuration. Are the Options provided by the interface enough to do what I would like to, or do I need to edit the xml?

Thanks for help!

From what I understand, you want to run one VM with pfSense which handles networking for VM-1 and VM-2, where VM-1 and VM-2 are isolated from host network connections. To do so, you need two virtual bridge interfaces, and an equivalent number of TUN/TAP interfaces as there are connections (think of TUN as the wire, and TAP as the pair of RJ-45 plugs at the ends of the wire.)
The first virtual bridge (br0) sits between the host network adapter and the pfSense VM NIC-0 with a TUN/TAP connection connecting to br0. This bridge (br0) ensures that the host and the pfSense VM get individual IP addresses from the external router. While you can connect directly, it should disconnect networking to the host though I am not sure about this; I haven't tested such a scenario. You can do away with this bridge if the host has external network access.

The second virtual bridge (br1) connects pfSense VM NIC 1 and other VMs (VM-1 and VM-2) as you have illustrated. Each of these VMs need their own TUN/TAP connections.

Furthermore, the bridge is the "router", so you running DHCP service on pfSense VM won't forward them over the virtual bridge (br1). You need to run DHCP service on the host for br1 to assign IP addresses to VM-1 and VM-2. IMO, if you're using VMs (not containers); it is best to assign IP addresses to TUN/TAP connections, and use static IP assignment for the VMs.

1 members found this post helpful.

Thanks for your reply. You are right 98% right with what you wrote.

That's the key information I have been digging into that for a long time and the setup of the interfaces is as described. Those tap devices are created by kvm/qemu and all I needed to add was one bridge and one macvtap device for the pfsense instance. BUT I couldn't make dhcp work, such that pfsense is serving them. Assigning a static ip to the bridge and adding a route did the trick.

But one reason why I just want to create this setup is to get ips from the virtual router (pfsense vm).

What would it take to make that happen?

The last reply of @mc.prtk made me think if I am on the wrong track. On my way of trail and error I ended up with a setup that is nearly completely as described by her/him (?).

As a result of my findings here, it's like that: KVM or QEMU or LIBVIRT - I do not know which exact componet it does - is creating the tap devices needed to connect to the bridges, so basically nothing to do for the user.

Instead of a bridge to connect one (or more) VM(s) to a network one can use macvtap interfaces - they basically have the same effect as a bridge but are told to be more efficient/modern. I do not have the expertise to judge that, I am happy that @mc.prtk pushed me a lot on my way.

As mentioned in my last post and lined out by @mc.prtk the bridge connection the virtual router VM (pfsense) and its clients might not forward dhcp traffic. And that is indeed one of my main intents here. So I tried to find some way around this, but I couldn't find anything saying that linux bridges wont forward dhcp traffic. Since they operate on Layer 2 it would require them to look into the packets to block it - but that is what they do not - at least as far as I can understand it.

So I stumbled over this document: wiki.linuxfoundation.org/networking/bridge. At the very bottom there is a paragraph about dhcp:

I guess I need to fix some configuration of the bridge (br0) and than dhcp should work?! That's at least what I am going for today.

Got it up ad running.

The Bridge does it, but it needs to be configured correctly, otherwise those problems arise:

1. forward-delay and mentioned above. For a newly started VM 30 Seconds are too long to wait for an IP, so the interface doesn't come up. Pinging and all the debugging stuff doesn't work as well in that time. Setting it to zero does fix it.

2. auto-connect / auto-route when the bridge comes up, NetworkManager sets it as the first default route, causing packets to enter an infinite loop. Host -> Pfsense -> Host setting ipv4.nev-default to yes stops that.

3. using a static IP for br0 on the host. Without it didn't work well, sometimes it did sometimes not. I guess that is because the bridge has to exists prior to the pfsense instance, which is dhcp - hen egg problem.

Bonus. Since pfsense is doing the local routing, the pfsense instance itself can be connected to the internet via libvirts default connection which uses NAT and that does work over WLAN as well.