[SOLVED] QEMU/KVM with NAT and IPTABLES -> no outgoing traffic

MagTheRose · 10-25-2018, 05:48 PM

Hi,

I have a little root-server with qemu/kvm and nat vms.

For some reason (I guess missing iptables rules), there's only incoming traffic to the guests, but no outgoing traffic. I.e., I can log in via SSH directly to a vm, but there's no way to download some package via wget (of course, apt-proxy with apt-cacher on the host is working for installation).

Also, with traceroute, all tcp/udp-traffic seems to stop on the host.
Interestingly enaugh, ICMP is working on the vms.

I tried several howtos on the internet, (e.g., iptables -t nat -A POSTROUTING -s 172.16.0.0/28 -j MASQUERADE) but nothing seems to work properly.

Any Ideas?
Which additional information is useful? iptables -t nat -L?

berndbausch · 10-25-2018, 07:53 PM

Quote:

Originally Posted by MagTheRose

For some reason (I guess missing iptables rules), there's only incoming traffic to the guests, but no outgoing traffic. I.e., I can log in via SSH directly to a vm, but there's no way to download some package via wget (of course, apt-proxy with apt-cacher on the host is working for installation).

Do the VMs run a server that listens on port 80? Is port 80 open on the VMs? Can port 80 be blocked somewhere along the path between the wget command and the VM? Did you try to download via other ports (to test other ports, you can use netcat to set up a dummy server)?

Quote:

Also, with traceroute, all tcp/udp-traffic seems to stop on the host.
Interestingly enaugh, ICMP is working on the vms.

I don't think I understand this. Can you elaborate: What do you do and what do you observe to come to these conclusions?

Quote:

Which additional information is useful? iptables -t nat -L?

Which commands you used to set up the network, create the VMs and connect the VMs to the network. Which commands you used to test access to the VMs, and where you ran those commands (on the host or elsewhere).

mralk3 · 10-25-2018, 11:45 PM

Enable IP forwarding within the kernel. Edit /etc/sysctl.conf and add:

Code:

net.ipv4.ip_forward = 1

Then run as root:

Code:

sysctl -p

The vms should now have access to the internet assuming you do not have any firewall rules blocking access.

MagTheRose · 10-26-2018, 01:48 AM

Quote:

Originally Posted by mralk3

Enable IP forwarding within the kernel. Edit /etc/sysctl.conf and add:

Code:

net.ipv4.ip_forward = 1

That was done, since then a few reboots took place, so I guess it shoud work:

Code:

root@server1:~# cat /etc/sysctl.conf 
net.ipv4.ip_forward=1
dev.raid.speed_limit_min = 20000
dev.raid.speed_limit_max = 100000

Quote:

Originally Posted by berndbausch

Do the VMs run a server that listens on port 80? Is port 80 open on the VMs? Can port 80 be blocked somewhere along the path between the wget command and the VM? Did you try to download via other ports (to test other ports, you can use netcat to set up a dummy server)?

Yes, one VM is running Apache. I only tried download via 80/443. I'll give the netcat-solution a try.

Quote:

Originally Posted by berndbausch

I don't think I understand this. Can you elaborate: What do you do and what do you observe to come to these conclusions?

Traceroute to www.heise.de on Port 80 stops after one hop. Ping www.heise.de returns the Pings with about 35ms.

Quote:

Originally Posted by berndbausch

Which commands you used to set up the network, create the VMs and connect the VMs to the network. Which commands you used to test access to the VMs, and where you ran those commands (on the host or elsewhere).

For configuration I use virt-manager (debian package) on my notebook via SSH.
Access is tested from home (elsewhere).
Additional iptables rules for the port forwarding are setup via Shell-script on the server after starting the VMs, e.g.:

Code:

virsh start webserver
iptables -I FORWARD -o virbr0 -d  172.16.0.3 -j ACCEPT
iptables -t nat -I PREROUTING -p tcp --dport 80 -j DNAT --to 172.16.0.3:80
iptables -t nat -I PREROUTING -p tcp --dport 443 -j DNAT --to 172.16.0.3:443

berndbausch · 10-26-2018, 10:59 AM

Can I summarize your problem?

You have a server with VMs. One of the VMs runs a web server. You can access the VM via ssh from your PC at home, but trying to download files with wget fails.

From your last post, I deduce that you don't access the VM's IP address to download the files. Rather, you access the host server's IP address, and have port forwarding rules in place that forward your HTTP requests to the VM. Are you doing the same with SSH packets?

From the VM, you can ping addresses on the internet, but when you run traceroute on the VM it doesn't list any routers beyond the VM's host.

Is that correct?

My guess is that something is wrong with your netfilter configuration. I would use tcpdump on the VM host to see what happens with http packets. I would also add iptables LOG rules to confirm that packets are processed as expected. Or find out at which point they are not processed as expected.

MagTheRose · 10-26-2018, 04:48 PM

@berndbausch

Yep, that's correct.
Of course, SSH port forwarding is done exactly the same way as http(s), except that it's not 22 on the source side, since 22 is used by the host, obviously.
Unfortunately, I only have one public IP which makes NAT neccessary.

Furthermore, I'm pretty sure that's a problem with iptables, too. But that's exactly my problem; On all previous servers I could use bridged networking with kvm, so iptables is quite new for me.
I guess some rules are set by kvm for the guests, while I set up some rules for the portfording – maybe there is some kind of problem mixing them?!?
Or there ist some rule missing either by the kvm hooks or by my rules?

berndbausch · 10-26-2018, 05:50 PM

Quote:

Originally Posted by MagTheRose

I guess some rules are set by kvm for the guests, while I set up some rules for the portfording – maybe there is some kind of problem mixing them?!?

Yes, libvirt does create its own rules, depending on how you connect the VM. I agree that you should investigate there.

MagTheRose · 10-28-2018, 12:59 PM

Quote:

Originally Posted by berndbausch

Yes, libvirt does create its own rules, depending on how you connect the VM. I agree that you should investigate there.

What is the best way to do that?

berndbausch · 10-28-2018, 07:26 PM

Quote:

Originally Posted by MagTheRose

What is the best way to do that?

Look at all netfilter rules. I think the easiest way to dump them all out is the iptables-save script (if it exists on your system).

If the number of rules is overwhelming, look at those that involve virbr0 and other bridges managed by libvirt, and those that perform SNAT and DNAT.

For example, this is what I find on a Centos system with a default libvirt and KVM installation:

Code:

# iptables-save >/tmp/ipt
# grep virbr0 /tmp/ipt
-A POSTROUTING -o virbr0 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill
-A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
-A FORWARD -d 192.168.122.0/24 -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 192.168.122.0/24 -i virbr0 -j ACCEPT
-A FORWARD -i virbr0 -o virbr0 -j ACCEPT
-A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable
-A OUTPUT -o virbr0 -p udp -m udp --dport 68 -j ACCEPT

Then try to understand how your own tweaking interferes with these rules.

Disclaimer: I try hard to look smart with netfilter rules, but I am not necessarily an expert in this matter.

MagTheRose · 10-29-2018, 11:04 AM

Quote:

Originally Posted by berndbausch

Look at all netfilter rules. I think the easiest way to dump them all out is the iptables-save script (if it exists on your system).

It exists - for anonymity I replaced the real public IP by 99.99.99.99, which shouldn't make a difference in the sense of the rules.

Code:

# Generated by iptables-save v1.6.0 on Mon Oct 29 15:05:19 2018
*mangle
:PREROUTING ACCEPT [3729220:2986947132]
:INPUT ACCEPT [138556:7499245]
:FORWARD ACCEPT [3590661:2979447707]
:OUTPUT ACCEPT [115636:6627109]
:POSTROUTING ACCEPT [3706282:2986073984]
-A POSTROUTING -o virbr0 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill
COMMIT
# Completed on Mon Oct 29 14:05:19 2018
# Generated by iptables-save v1.6.0 on Mon Oct 29 15:05:19 2018
*nat
:PREROUTING ACCEPT [110090:6055630]
:INPUT ACCEPT [108240:5921558]
:OUTPUT ACCEPT [4859:361869]
:POSTROUTING ACCEPT [51048:2660858]
-A PREROUTING -p tcp -m tcp --dport 443 -j DNAT --to-destination 172.16.0.3:443
-A PREROUTING -p tcp -m tcp --dport 80 -j DNAT --to-destination 172.16.0.3:80
-A PREROUTING -p tcp -m tcp --dport 65533 -j DNAT --to-destination 172.16.0.3:22
-A PREROUTING -p tcp -m tcp --dport 65532 -j DNAT --to-destination 172.16.0.2:22
-A PREROUTING -p tcp -m tcp --dport 443 -j DNAT --to-destination 172.16.0.3:443
-A PREROUTING -p tcp -m tcp --dport 80 -j DNAT --to-destination 172.16.0.3:80
-A POSTROUTING -s 172.16.0.0/28 -d 224.0.0.0/24 -j RETURN
-A POSTROUTING -s 172.16.0.0/28 -d 255.255.255.255/32 -j RETURN
-A POSTROUTING -s 172.16.0.0/28 ! -d 172.16.0.0/28 -p tcp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 172.16.0.0/28 ! -d 172.16.0.0/28 -p udp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 172.16.0.0/28 ! -d 172.16.0.0/28 -j MASQUERADE
-A POSTROUTING -s 172.16.0.0/28 ! -d 172.16.0.0/28 -p tcp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 172.16.0.0/28 ! -d 172.16.0.0/28 -p udp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 172.16.0.0/28 ! -d 172.16.0.0/28 -j MASQUERADE
-A POSTROUTING -s 172.16.0.0/28 -o enp3s0 -j SNAT --to-source 99.99.99.99
-A POSTROUTING -s 172.16.0.0/28 ! -d 172.16.0.0/28 -j MASQUERADE
-A POSTROUTING -s 99.99.99.99/32 ! -d 172.16.0.0/28 -j MASQUERADE
COMMIT
# Completed on Mon Oct 29 14:05:19 2018
# Generated by iptables-save v1.6.0 on Mon Oct 29 15:05:19 2018
*filter
:INPUT ACCEPT [135000:6731923]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [113197:5930200]
-A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
-A INPUT -s 172.16.0.0/28 -i virbr0 -j ACCEPT
-A INPUT -s 172.16.0.0/28 -i virbr0 -j ACCEPT
-A FORWARD -d 172.16.0.3/32 -o virbr0 -j ACCEPT
-A FORWARD -d 172.16.0.2/32 -o virbr0 -j ACCEPT
-A FORWARD -d 172.16.0.0/28 -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 172.16.0.0/28 -i virbr0 -j ACCEPT
-A FORWARD -i virbr0 -o virbr0 -j ACCEPT
-A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -d 172.16.0.3/32 -o virbr0 -j ACCEPT
-A FORWARD -d 172.16.0.2/32 -o virbr0 -j ACCEPT
-A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 172.16.0.0/28 -i virbr0 -j ACCEPT
-A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 172.16.0.0/28 -i virbr0 -j ACCEPT
-A OUTPUT -o virbr0 -p udp -m udp --dport 68 -j ACCEPT
COMMIT
# Completed on Mon Oct 29 14:05:19 2018

Quote:

Originally Posted by berndbausch

Then try to understand how your own tweaking interferes with these rules.

Hm, first interesting point is, that the forwarding rules for http(s) seems to appear twice?!

Quote:

Originally Posted by berndbausch

Disclaimer: I try hard to look smart with netfilter rules, but I am not necessarily an expert in this matter.

Me neither ☺
Don't be afraid – this is just my "learning" machine, there's nothing really productive on the server (except some personal stuff, which I'd like to have running). Furthermore, I have backups of all the stuff, so if we kill the network somehow, I can simply reboot and restore everything ;-)

MagTheRose · 11-03-2018, 06:47 PM

Got it,

Code:

iptables -t nat -I PREROUTING -p tcp -d 99.99.99.99 --dport 80 -j DNAT --to 172.16.0.3:80
iptables -t nat -I PREROUTING -p tcp -d 99.99.99.99 --dport 443 -j DNAT --to 172.16.0.3:443

seems to work...