Problem with configuring Xen Bridge Networking

l2g · 04-28-2010, 01:53 AM

** Posted in Virtualization but perhaps belong here ***

Hi,
It's been a while since I last posted in LQ.. feels great to be back!

I have installed a Xen with 2 VM's inside. They are all under the same subnet.
The Xen machine can see the outside network but both machines in it cannot.
How do I create the bridge correctly (to xenbr0 i guess) so it fixes this?

Here's what I tried:
===========
It's RHEL5.3

1. '/etc/sysconfig/network-scripts/ifcfg-eth0' is ok on both VM's
2. iptables service is disabled, system-config-network shows IP,DNS, GW are correct.
3. On the VM's the route -n shows the gw ip though I cannot ping it.
4. checked hosts.allow, hosts.
5. In '/etc/xen/xend-config.sxp' unmarked:
(network-script network-bridge)
(vif-script vif-bridge)
6. in xen/<images_location>/ I modified vm.cfg so vif = [ 'ip=10.2.0.54' ] for one of my virtual machines.
7. I cannot however find '/etc/network/interfaces', can anyone advise if actually in my case it's the
ifcfg-eth0?

7. I basically followed the Bridged Networking scenario in Xen Networking: http://wiki.kartbuilding.net/index.php/Xen_Networking but to no avail yet.

I'll be happy to post any info, let me know.

Ifconfig on the Xen Parent:
-----------
[root@XEN_PARENT]# ifconfig -a
eth0 Link encap:Ethernet HWaddr 00:50:56:8B:3A:E4
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:130021 errors:0 dropped:0 overruns:0 frame:0
TX packets:75097 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:10315149 (9.8 MiB) TX bytes:12038346 (11.4 MiB)
Interrupt:17 Base address:0x2000

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:127 errors:0 dropped:0 overruns:0 frame:0
TX packets:127 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:18162 (17.7 KiB) TX bytes:18162 (17.7 KiB)

vif1.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:83 errors:0 dropped:0 overruns:0 frame:0
TX packets:73422 errors:0 dropped:266 overruns:0 carrier:0
collisions:0 txqueuelen:32
RX bytes:9868 (9.6 KiB) TX bytes:6378453 (6.0 MiB)

vif3.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:350 errors:0 dropped:0 overruns:0 frame:0
TX packets:44129 errors:0 dropped:341 overruns:0 carrier:0
collisions:0 txqueuelen:32
RX bytes:27754 (27.1 KiB) TX bytes:3818085 (3.6 MiB)

xenbr0 Link encap:Ethernet HWaddr 00:50:56:8B:3A:E4
inet addr:10.2.0.52 Bcast:0.0.0.0 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:130303 errors:0 dropped:0 overruns:0 frame:0
TX packets:74569 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:8526828 (8.1 MiB) TX bytes:11982125 (11.4 MiB)

-----------------------------------

Any help will be greatly appreciated.

Thanks,
Liron

paulsm4 · 04-28-2010, 07:29 PM

Hi -

This seems to be a duplicate post.

Did you try the suggestions I made in the previous post? What were the results? What's the current problem?

Thanx in advance .. PSM

PS:
If you are still having problems, be sure to specify:
a) what distro and version are the VMs?
b) are you using network manager, or the "raw" configuration files (they could be conflicting)?

l2g · 04-29-2010, 03:26 AM

Hi Paul,
Can you say how do I test for network issues between the xenbr0 bridge and the Physical network card?
Apparently both eth0 and xenbr0 have the same MAC address. and with Xen-3.0, there is no need for Peth0 anymore..

I can see that 'arping' to the VMs through xenbr0 bridge goes to eth0 interface on the VM, because it shows the correct MAC address for eth0:

# arping -b -c 1 -I xenbr0 10.2.0.55
ARPING 10.2.0.55 from 10.2.0.52 xenbr0
Unicast reply from 10.2.0.55 [00:16:3E:07:03:49] 1.557ms
Sent 1 probes (1 broadcast(s))
Received 1 response(s)

Now on 10.2.0.55 I do 'ifconfig eth0' and get the same MAC: 00:16:3E:07:03:49

I tried to #echo "1" > /proc/sys/net/ipv4/ip_forward on Domain-0 to see if it would help, alas nothing!
Also tried to add different routes in my routing table on the VM, it would still not ping anywhere but Domain-0.

I posted before my network-bridge script status, ip route list, brctl show, ifconfig

As for the current status:
1. Domain-0(Xen) has ip 10.2.0.52/24 and can reach the gw on 10.2.0.254.
2. Ping from Domain-0 to the VM's (10.2.0.54/24 and 10.2.0.55/24) and back works.
3. eth0 is confirmed as Xen Virtual Ethernet ( I can see it in system-config-network).
4. route -n shows my default gw (0.0.0.0 10.2.0.254 0.0.0.0 eth0) but I *cannot* reach it.
5. ifconfig on Dom-0 shows xenbr0, eth0, vif2.0, lo ok.

I'd really appreciate any help here.

Liron.

l2g · 04-29-2010, 03:28 AM

Just to add:

Xen-3.0(Domain-0 10.2.0.52/24) RHEL5.3 both VM's (10.2.0.54/10.2.0.55) all under subnet 255.255.255.0

Thanks.

l2g · 04-30-2010, 05:04 AM

***** Updating --- can you think of what could be causing this issue? *****
Thanks!

#tcpdump -nvvi xenbr0 | grep -i 10.2.0.55
---------------------------------------------
11:19:41.548806 arp who-has 10.2.0.254 tell 10.2.0.55
11:19:41.549059 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto: ICMP (1), length: 84) 10.2.0.55 > 10.2.0.254: ICMP echo request, id 19463, seq 19, length 64

#tcpdump -nvvi vif1.0 host 10.2.0.55
------------------------------------
11:21:23.403063 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto: ICMP (1), length: 84) 10.2.0.55 > 10.2.0.254: ICMP echo request, id 22279, seq 7, length 64

#tcpdump -vv -n -c 1000 | grep -i 10.2.0.254
--------------------------------------------
11:22:46.256389 arp who-has 10.2.0.254 tell 10.2.0.55
11:22:46.256662 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto: ICMP (1), length: 84) 10.2.0.55 > 10.2.0.254: ICMP echo request, id 23047, seq 8, length 64

# iptables -L -vv -n
--------------------
Chain INPUT (policy ACCEPT 10869 packets, 769K bytes)
pkts bytes target prot opt in out source destination
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:8003

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED PHYSDEV match --physdev-out vif1.0
0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif1.0 udp spt:68 dpt:67
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED PHYSDEV match --physdev-out vif1.0
0 0 ACCEPT all -- * * 10.2.0.55 0.0.0.0/0 PHYSDEV match --physdev-in vif1.0

Chain OUTPUT (policy ACCEPT 8294 packets, 2149K bytes)
pkts bytes target prot opt in out source destination

----------------------------------------------

As in http://wiki.xensource.com/xenwiki/Xe...a03fa0fea0ed8b

I tried:

#echo "1" > /proc/sys/net/ipv4/ip_forward

I also tried:

iptables -A FORWARD -m physdev --physdev-in eth0 --physdev-out '!' eth0 -j ACCEPT
iptables -A FORWARD -m physdev --physdev-out eth0 --physdev-in '!' eth0 -j ACCEPT

Still Nothing!

----------------------------------------------

# cat /etc/xen/xend-config.sxp | grep -v "^#" | grep "[a-z]"
-----------------------------------------------------------
(logfile /var/log/xen/xend.log)
(loglevel DEBUG)
(network-script network-bridge)
(vif-script vif-bridge)
(dom0-min-mem 196)
(enable-dom0-ballooning yes)
(dom0-cpus 0)
(vnc-listen '0.0.0.0')
(vncpasswd '')
(xend-domains-lock-path /opt/ovs-agent-2.3/utils/dlm.py)

-----------------------------------------------

# brctl show
bridge name bridge id STP enabled interfaces
xenbr0 8000.0050568b3ae4 no vif1.0
eth0
--------------------------------------------------------------------

# xm list
Name ID Mem VCPUs State Time(s)
Domain-0 0 564 2 r----- 67.2
SiebelApp 1 1024 1 -b---- 32.6

---------------------------------------------------------------------

#xm network-list 1
-------------------
Idx BE MAC Addr. handle state evt-ch tx-/rx-ring-ref BE-path
0 0 00:16:3e:07:03:49 0 4 10 815 /816 /local/domain/0/backend/vif/1/0

--------------------

# xm network-list 1 -l
(0
((mac 00:16:3e:07:03:49)
(handle 0)
(protocol x86_32-abi)
(backend-id 0)
(state 4)
(backend /local/domain/0/backend/vif/1/0)
(tx-ring-ref 815)
(rx-ring-ref 816)
(event-channel 10)
(request-rx-copy 1)
(feature-rx-notify 1)
(feature-sg 1)
(feature-gso-tcpv4 1)
)
)

-------------------------------