I have a particular problem and wonder if anyone could give me some advice on how to solve this properly. Basically I need to setup a virtual network over multiple libvirt servers.

Simple example scenario:
3 libvirt hosts. Lets go with Debian 10, libvirt and KVM.
They are connected over ANY kind of network. Let's say Ethernet.
But it could even be a mixed network, made of libvirt hosts that are deployed on different sites, could be IPv6 or InfiniBand... .
On each of these 3 libvirt hosts there is one VM deployed. So in total 3 VMs.
Now I want to create a virtual IPv4 network between these 3 VMs.
Lets call it "Office network".
We could even add an internal virtual router or switch in form of a VM.
But lets just say these 3 VMs are supposed to be directly connect to each other.



There are certain requirements for this Virtual network:
The network runs over the libvirt host network but is isolated from it.
Meaning that if I would look at the traffic coming from e.g. VM1 to VM2,
I would not see any traffic related to the libvirts hosts network.
Ideally there would be no hint to that there is a virtual network switch in place.

Some more caveats:
- Can't be a commercial solutions.
- The solution needs to be scalable. Say 1000 nodes/servers.
- I would prefer not to use solutions like OpenStack.
- Can't use VLAN because I need more than 4095 networks.
- Keep it as simple as possible. And with simple I mean elegant, not reinventing the wheel. Unix philosophy.

So what is your advice and experience?
Routing? SDN? VxLAN? vSwitch? Any other ready made software?

Your terminology is not very precise. By "libvirt host", I assume you mean a hypervisor?

Implementing multiple virtual networks across multiple hypervisors is not only pretty straightforward, it's what every IaaS provider does. The networks share the same physical infrastructure and are segregated using VLAN tagging.Look up 802.1ad, also known as "QinQ".

Use an overlay network (a.k.a. tunneling), e.g. the abovementioned VXLAN, or Geneve or GRE. OpenvSwitch can be used to implement it, but Linuxbridges work as well, at least for VXLAN. OpenStack does it this way, but you don't need OpenStack to create your own overlay.