LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Virtualization and Cloud
User Name
Password
Linux - Virtualization and Cloud This forum is for the discussion of all topics relating to Linux Virtualization and Linux Cloud platforms. Xen, KVM, OpenVZ, VirtualBox, VMware, Linux-VServer and all other Linux Virtualization platforms are welcome. OpenStack, CloudStack, ownCloud, Cloud Foundry, Eucalyptus, Nimbus, OpenNebula and all other Linux Cloud platforms are welcome. Note that questions relating solely to non-Linux OS's should be asked in the General forum.

Notices


Reply
  Search this Thread
Old 06-03-2020, 06:15 AM   #1
czezz
Member
 
Registered: Nov 2004
Distribution: Slackware/Solaris
Posts: 924

Rep: Reputation: 43
[Openstack/Devstack] Cannot reach Internet via Floating IP


Hi all,
This is my devstack lab installed on VirtualBox.
Code:
stack@vlab:~$ cat /etc/devstack-version
DevStack Version: stein
Change: 669b0c5e4eba1a721dc02650e000fb0ac6bfd11b Merge "Remove crusty old python 3 package version logic" into stable/stein 2019-04-26 09:32:45 +0000
OS Version: Ubuntu 16.04 xenial
I have created instance (vm: heat_server), with floating IP/port hoping it will have access to internet.
Code:
stack@vlab:~$ openstack server list
+--------------------------------------+-------------+---------+------------------------------------+--------------------------+---------+
| ID                                   | Name        | Status  | Networks                           | Image                    | Flavor  |
+--------------------------------------+-------------+---------+------------------------------------+--------------------------+---------+
| 6f8fbc70-dfc5-4840-ba18-cf7869863af6 | heat_server | ACTIVE  | heat_network=10.1.1.4, 172.24.4.10 | cirros-0.4.0-x86_64-disk | m1.tiny |
| 0eb31f9d-4524-4382-9868-6bb1df1d84fd | testvm2     | SHUTOFF | fxnet=10.2.0.5                     | cirros-0.4.0-x86_64-disk | m1.tiny |
| 4766f91f-b1ec-44f6-96c4-6d1164ba754e | testvm1     | SHUTOFF | fxnet=10.2.0.11                    | cirros-0.4.0-x86_64-disk | m1.tiny |
+--------------------------------------+-------------+---------+------------------------------------+--------------------------+---------+
Unfortunately it does not reach Internet.
I did a small investigation:
1. I checked for network "public" - OK
2. I checked for subnet "public-subnet" - OK
3. I checked for router "public" and I see - NONE

Questions:
1. Could that be a reason of not reaching internet from my instance (missing public router)?
2. Is this true that connectivity to internet via Floating IP is in fact NATed by Neutron via internal network (in my case 10.1.1.x) ?
Code:
openstack floating ip list
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+
| ID                                   | Floating IP Address | Fixed IP Address | Port                                 | Floating Network                     | Project                          |
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+
| 54cd81a9-771c-475f-94db-d69fe92bc512 | 172.24.4.10         | 10.1.1.4         | e6681d9c-8456-4cf6-ae07-71abce21a823 | e0ef1d2b-1d8e-42e0-9b3a-ecbe784f24e1 | a5e6e3dbad9f4f4b8810fe0a1246e1f7 |
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+


Listings:

stack@vlab:~$ openstack network show public
+---------------------------+----------------------------------------------------------------------------+
| Field                     | Value                                                                      |
+---------------------------+----------------------------------------------------------------------------+
| admin_state_up            | UP                                                                         |
| availability_zone_hints   |                                                                            |
| availability_zones        | nova                                                                       |
| created_at                | 2019-05-07T13:47:00Z                                                       |
| description               |                                                                            |
| dns_domain                | None                                                                       |
| id                        | e0ef1d2b-1d8e-42e0-9b3a-ecbe784f24e1                                       |
| ipv4_address_scope        | None                                                                       |
| ipv6_address_scope        | None                                                                       |
| is_default                | True                                                                       |
| is_vlan_transparent       | None                                                                       |
| location                  | None                                                                       |
| mtu                       | 1500                                                                       |
| name                      | public                                                                     |
| port_security_enabled     | True                                                                       |
| project_id                | 7e31368f4f9a4b31a91257394592fef2                                           |
| provider:network_type     | flat                                                                       |
| provider:physical_network | public                                                                     |
| provider:segmentation_id  | None                                                                       |
| qos_policy_id             | None                                                                       |
| revision_number           | 3                                                                          |
| router:external           | External                                                                   |
| segments                  | None                                                                       |
| shared                    | False                                                                      |
| status                    | ACTIVE                                                                     |
| subnets                   | 51bd9836-acf8-4369-8f4c-8466307694a7, fec1ef4c-d70d-43d8-8c6e-8ea03325f52f |
| tags                      |                                                                            |
| updated_at                | 2019-05-07T13:47:52Z                                                       |
+---------------------------+----------------------------------------------------------------------------+


stack@vlab:~$ openstack subnet show public-subnet
+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| allocation_pools  | 172.24.4.2-172.24.4.254              |
| cidr              | 172.24.4.0/24                        |
| created_at        | 2019-05-07T13:47:14Z                 |
| description       |                                      |
| dns_nameservers   |                                      |
| enable_dhcp       | False                                |
| gateway_ip        | 172.24.4.1                           |
| host_routes       |                                      |
| id                | fec1ef4c-d70d-43d8-8c6e-8ea03325f52f |
| ip_version        | 4                                    |
| ipv6_address_mode | None                                 |
| ipv6_ra_mode      | None                                 |
| location          | None                                 |
| name              | public-subnet                        |
| network_id        | e0ef1d2b-1d8e-42e0-9b3a-ecbe784f24e1 |
| project_id        | 7e31368f4f9a4b31a91257394592fef2     |
| revision_number   | 0                                    |
| segment_id        | None                                 |
| service_types     |                                      |
| subnetpool_id     | None                                 |
| tags              |                                      |
| updated_at        | 2019-05-07T13:47:14Z                 |
+-------------------+--------------------------------------+

stack@vlab:~$ openstack router list
+--------------------------------------+-------------+--------+-------+-------------+-------+----------------------------------+
| ID                                   | Name        | Status | State | Distributed | HA    | Project                          |
+--------------------------------------+-------------+--------+-------+-------------+-------+----------------------------------+
| 1bb436b3-fe08-4b38-8542-30587a87bc40 | heat_router | ACTIVE | UP    | False       | False | a5e6e3dbad9f4f4b8810fe0a1246e1f7 |
| a71b95db-d12e-46de-85b5-14d4368186be | router1     | ACTIVE | UP    | False       | False | a5e6e3dbad9f4f4b8810fe0a1246e1f7 |
| beba3591-e0f2-4f03-a70a-b2022c9aca01 | fxrouter    | ACTIVE | UP    | False       | False | a5e6e3dbad9f4f4b8810fe0a1246e1f7 |
+--------------------------------------+-------------+--------+-------+-------------+-------+----------------------------------+
 
Old 06-04-2020, 12:37 AM   #2
berndbausch
LQ Addict
 
Registered: Nov 2013
Location: Tokyo
Distribution: Mostly Ubuntu and Centos
Posts: 6,316

Rep: Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002
Devstack creates a "fake" external network which is not connected to the external world at all, except if you deliberately connect it. For instructions, see https://docs.openstack.org/devstack/...essible-guests.
 
Old 06-04-2020, 05:17 PM   #3
czezz
Member
 
Registered: Nov 2004
Distribution: Slackware/Solaris
Posts: 924

Original Poster
Rep: Reputation: 43
I made it finally to work.
But now my problem is that after instance reboot, I cannot access it via Floating IP
I can still get in via internal network (via qrouter).

Last edited by czezz; 06-04-2020 at 05:21 PM.
 
Old 06-04-2020, 05:52 PM   #4
berndbausch
LQ Addict
 
Registered: Nov 2013
Location: Tokyo
Distribution: Mostly Ubuntu and Centos
Posts: 6,316

Rep: Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002
If you want assistance, you need to describe your setup.
 
Old 06-05-2020, 02:58 AM   #5
czezz
Member
 
Registered: Nov 2004
Distribution: Slackware/Solaris
Posts: 924

Original Poster
Rep: Reputation: 43
Hi Berndbausch,
Thank you for reply.

I did fresh installation of Devstack (from Master branch - DevStack Version: victoria)
In local.conf I have specified:
Code:
HOST_IP=192.168.56.109 - this is eth1/enp0s8 and refers to Host-Only interface in my VirtualBox 
FLAT_INTERFACE=enp0s8 
FLOATING_RANGE=192.168.56.224/27 - for this installation I assigned floating IPs on the same class as HOST_IP (its not clear yet to me whether it should be the same class or should be different. The point is that by adding Floating IP to the instance I could access it from Devstack itself - and that is/was OK for the moment).
I have deployed stack using this example https://docs.ukcloud.com/articles/op...-template.html , with 2 changes:
1. Removed volume (for some reason it gives errors and blocks deployment - to troubleshoot later)
2. Replace network "Internet" with "Public"



Creation went quite smooth:
Code:
stack@vlab:~$ openstack stack list
+--------------------------------------+-------------+-----------------+----------------------+--------------+
| ID                                   | Stack Name  | Stack Status    | Creation Time        | Updated Time |
+--------------------------------------+-------------+-----------------+----------------------+--------------+
| 8404831a-fd3b-46b6-8001-55484df072fd | basic-stack | CREATE_COMPLETE | 2020-06-04T20:42:31Z | None         |
+--------------------------------------+-------------+-----------------+----------------------+--------------+

stack@vlab:~$ heat resource-list basic-stack
WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead
+-----------------------+-------------------------------------------------------------------------------------+------------------------------------+-----------------+----------------------+
| resource_name         | physical_resource_id                                                                | resource_type                      | resource_status | updated_time         |
+-----------------------+-------------------------------------------------------------------------------------+------------------------------------+-----------------+----------------------+
| heat_network          | cd78103f-e606-40a7-8399-b1e99651757f                                                | OS::Neutron::Net                   | CREATE_COMPLETE | 2020-06-04T20:42:31Z |
| heat_network_subnet   | d64acd35-b70b-4dd8-9a51-34a70aaf9f45                                                | OS::Neutron::Subnet                | CREATE_COMPLETE | 2020-06-04T20:42:31Z |
| heat_router           | 7563b1f4-2635-412b-8053-dda164a5b279                                                | OS::Neutron::Router                | CREATE_COMPLETE | 2020-06-04T20:42:31Z |
| heat_router_interface | 7563b1f4-2635-412b-8053-dda164a5b279:subnet_id=d64acd35-b70b-4dd8-9a51-34a70aaf9f45 | OS::Neutron::RouterInterface       | CREATE_COMPLETE | 2020-06-04T20:42:31Z |
| heat_server           | 95fce8b8-9d4d-4ccd-a17b-cb367537e504                                                | OS::Nova::Server                   | CREATE_COMPLETE | 2020-06-04T20:42:31Z |
| heat_server_ip_assoc  | 1                                                                                   | OS::Neutron::FloatingIPAssociation | CREATE_COMPLETE | 2020-06-04T20:42:31Z |
| heat_server_port      | cc3eef0e-7f01-4c5b-841e-ade976ae0dbc                                                | OS::Neutron::Port                  | CREATE_COMPLETE | 2020-06-04T20:42:31Z |
| heat_server_public_ip | 882e79f8-f983-430c-b3e3-3758fa3158e7                                                | OS::Neutron::FloatingIP            | CREATE_COMPLETE | 2020-06-04T20:42:31Z |
+-----------------------+-------------------------------------------------------------------------------------+------------------------------------+-----------------+----------------------+
After adding 2 rules to the "default" security group:
Code:
openstack security group rule create default --protocol tcp --dst-port 22:22 --remote-ip 0.0.0.0/0
openstack security group rule create --protocol icmp default
I could access instance/vm heat_server via 192.168.56.227 from Devstack.
After reboot of the instance and later also all Devstack Vbox VM, I have no more access via 192.168.56.227.
I can still access it via: sudo ip netns exec qrouter-7563b1f4-2635-412b-8053-dda164a5b279 ssh cirros@10.1.1.9
Code:
stack@vlab:~$ openstack server list
+--------------------------------------+-------------+---------+---------------------------------------------------------+--------------------------+---------+
| ID                                   | Name        | Status  | Networks                                                | Image                    | Flavor  |
+--------------------------------------+-------------+---------+---------------------------------------------------------+--------------------------+---------+
| 95fce8b8-9d4d-4ccd-a17b-cb367537e504 | heat_server | ACTIVE  | heat_network=10.1.1.9, 192.168.56.227                   | cirros-0.5.1-x86_64-disk | m1.tiny |
| 1935c01d-7451-439d-9335-a0c2690e145a | vm2         | SHUTOFF | private=10.0.0.21, fd07:a88f:8dd3:0:f816:3eff:fee1:9f91 | cirros-0.5.1-x86_64-disk | m1.tiny |
+--------------------------------------+-------------+---------+---------------------------------------------------------+--------------------------+---------+
I dont know where to look now / where to start troubleshooting.

Last edited by czezz; 06-05-2020 at 03:06 AM.
 
Old 06-05-2020, 04:10 AM   #6
berndbausch
LQ Addict
 
Registered: Nov 2013
Location: Tokyo
Distribution: Mostly Ubuntu and Centos
Posts: 6,316

Rep: Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002
The IP address of external bridge br-ex is not persistent. When you reboot, you lose it. Same for the physical volumes used by Cinder and the filesystem used by Swift.

I use this script to recreate them (only tested on Ussuri and a few earlier versions, and it uses the standard IP address):
Code:
$ cat restore-after-reboot
sudo ip a add 172.24.4.1/24 dev br-ex
sudo ip link set dev br-ex up

sudo mount -t xfs -o loop,noatime,nodiratime,logbufs=8 /opt/stack/data/swift/drives/images/swift.img /opt/stack/data/swift/drives/sdb1

if ! losetup | grep -q volumes-default-backing-file
then
    sudo losetup -f --show --direct-io=on /opt/stack/data/stack-volumes-default-backing-file
else
    echo stack-volumes-default-backing-file exists already
fi

if ! losetup | grep -q volumes-lvmdriver-1-backing-file
then
    sudo losetup -f --show --direct-io=on /opt/stack/data/stack-volumes-lvmdriver-1-backing-file
else
    echo stack-volumes-lvmdriver-1-backing-file exists already
fi

sudo systemctl restart devstack@c-vol
 
1 members found this post helpful.
Old 06-05-2020, 04:42 AM   #7
czezz
Member
 
Registered: Nov 2004
Distribution: Slackware/Solaris
Posts: 924

Original Poster
Rep: Reputation: 43
YES!
Thank you very much for this information.

Floating IP:
I was not aware that br-ex IP is not persistent.
Adding IP and setting int up solves the problem.


Volume:
I didnt test it yet. Hopefully later today in the evening.
Will let know about the results here.

Update: 2020.06.15
Volume tested and it works with solution provided by Berndbausch

Last edited by czezz; 06-15-2020 at 10:02 AM.
 
Old 03-05-2024, 03:23 AM   #8
willlq
LQ Newbie
 
Registered: Mar 2024
Posts: 1

Rep: Reputation: 0
same here

this can help
```
sudo ip addr flush ens33
sudo ovs-vsctl add-port br-ex ens33
sudo ip addr add 192.168.72.33/24 dev br-ex

```
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Openstack vs Devstack - DIFFERENCE fawaz25 Linux - Virtualization and Cloud 2 04-17-2019 10:36 AM
LXer: Essentials of OpenStack Administration Part 6: Installing DevStack (Lab) LXer Syndicated Linux News 0 01-12-2017 03:51 AM
LXer: Setup Nova-Docker driver && Openstack Kilo on Ubuntu 14.04 in devstack environment recoverable LXer Syndicated Linux News 0 04-28-2015 10:32 PM
LXer: DevStack Ceph, OpenStack Paris Summit voting, and more LXer Syndicated Linux News 0 08-04-2014 07:32 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Virtualization and Cloud

All times are GMT -5. The time now is 12:57 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration