LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 02-07-2019, 08:09 PM   #1
qskousen
LQ Newbie
 
Registered: Feb 2019
Posts: 7

Rep: Reputation: Disabled
Potential DHCP bug? Ubuntu 18.04 server


[Sorry if this is not the right place to put this, I am new to these forums and a better place was not immediately apparent.]

I am using Ubuntu 18.04 server:

Code:
$ cat /proc/version
Linux version 4.15.0-45-generic (buildd@lgw01-amd64-031) (gcc version 7.3.0 (Ubuntu 7.3.0-16ubuntu3)) #48-Ubuntu SMP Tue Jan 29 16:28:13 UTC 2019
in a LXC (LXD) container. The container was newly created, ~5 hours old, when I was suddenly disconnected from my SSH session, originating from the container host. The container network is using a network bridge (not the default LXD bridge) created with netplan:

Code:
network:
    ethernets:
        ens18:
            dhcp4: false
    bridges:
        br0:
            interfaces: [ens18]
            dhcp4: true
    version: 2
DHCP server is a PFSense box.

The reason for the disconnection became apparent when, after some digging, it appeared the container had requested a new IP from DHCP roughly 4 hours previous. PFSense DHCP logs:

Code:
Feb 7 20:58:52 	dhcpd 		DHCPOFFER on 10.0.55.25 to 00:16:3e:0c:d0:d0 (mailhog) via em1
Feb 7 20:58:52 	dhcpd 		DHCPREQUEST for 10.0.55.25 (10.0.55.1) from 00:16:3e:0c:d0:d0 (mailhog) via em1
Feb 7 20:58:52 	dhcpd 		DHCPACK on 10.0.55.25 to 00:16:3e:0c:d0:d0 (mailhog) via em1
Feb 7 21:03:08 	dhcpd 		DHCPDISCOVER from 00:16:3e:0c:d0:d0 (mailhog) via em1
Feb 7 21:03:10 	dhcpd 		DHCPOFFER on 10.0.55.26 to 00:16:3e:0c:d0:d0 (mailhog) via em1
Feb 7 21:03:10 	dhcpd 		DHCPREQUEST for 10.0.55.26 (10.0.55.1) from 00:16:3e:0c:d0:d0 (mailhog) via em1
Feb 7 21:03:10 	dhcpd 		DHCPACK on 10.0.55.26 to 00:16:3e:0c:d0:d0 (mailhog) via em1
Feb 7 22:03:10 	dhcpd 		DHCPREQUEST for 10.0.55.26 from 00:16:3e:0c:d0:d0 (mailhog) via em1
Feb 7 22:03:10 	dhcpd 		DHCPACK on 10.0.55.26 to 00:16:3e:0c:d0:d0 (mailhog) via em1
Feb 7 23:03:11 	dhcpd 		DHCPREQUEST for 10.0.55.26 from 00:16:3e:0c:d0:d0 (mailhog) via em1
Feb 7 23:03:11 	dhcpd 		DHCPACK on 10.0.55.26 to 00:16:3e:0c:d0:d0 (mailhog) via em1
Feb 8 00:03:11 	dhcpd 		DHCPREQUEST for 10.0.55.26 from 00:16:3e:0c:d0:d0 (mailhog) via em1
Feb 8 00:03:11 	dhcpd 		DHCPACK on 10.0.55.26 to 00:16:3e:0c:d0:d0 (mailhog) via em1
The second DHCP request, at 21:03:08, occurred right after I had issued an apt upgrade:

Code:
Start-Date: 2019-02-07  21:02:58
Commandline: apt upgrade
Requested-By: mbs (1001)
Upgrade: libcom-err2:amd64 (1.44.1-1ubuntu1, 1.44.1-1ubuntu1.1), libcurl4:amd64 (7.58.0-2ubuntu3.5, 7.58.0-2ubuntu3.6), libsystemd0:amd64 (237-3ubuntu10.11, 237-3ubuntu10.12), libkmod2:amd64 (24-1ubuntu3.1, 24-1ubuntu3.2), snapd:amd64 (2.34.2+18.04, 2.37.1.1+18.04), e2fsprogs:amd64 (1.44.1-1ubuntu1, 1.44.1-1ubuntu1.1), python-apt-common:amd64 (1.6.3, 1.6.3ubuntu1), openssh-sftp-server:amd64 (1:7.6p1-4ubuntu0.1, 1:7.6p1-4ubuntu0.2), udev:amd64 (237-3ubuntu10.11, 237-3ubuntu10.12), kmod:amd64 (24-1ubuntu3.1, 24-1ubuntu3.2), libudev1:amd64 (237-3ubuntu10.11, 237-3ubuntu10.12), libss2:amd64 (1.44.1-1ubuntu1, 1.44.1-1ubuntu1.1), libext2fs2:amd64 (1.44.1-1ubuntu1, 1.44.1-1ubuntu1.1), systemd-sysv:amd64 (237-3ubuntu10.11, 237-3ubuntu10.12), libpam-systemd:amd64 (237-3ubuntu10.11, 237-3ubuntu10.12), libdrm2:amd64 (2.4.91-2, 2.4.95-1~18.04.1), systemd:amd64 (237-3ubuntu10.11, 237-3ubuntu10.12), openssh-server:amd64 (1:7.6p1-4ubuntu0.1, 1:7.6p1-4ubuntu0.2), openssh-client:amd64 (1:7.6p1-4ubuntu0.1, 1:7.6p1-4ubuntu0.2), libnss-systemd:amd64 (237-3ubuntu10.11, 237-3ubuntu10.12), curl:amd64 (7.58.0-2ubuntu3.5, 7.58.0-2ubuntu3.6), libcurl3-gnutls:amd64 (7.58.0-2ubuntu3.5, 7.58.0-2ubuntu3.6), python3-apt:amd64 (1.6.3, 1.6.3ubuntu1), libdrm-common:amd64 (2.4.91-2, 2.4.95-1~18.04.1)
End-Date: 2019-02-07  21:03:37
Here is some (hopefully relevant) journalctl entries from the period, with some snipped for brevity:

Code:
Feb 07 21:02:42 mailhog sudo[701]: mbs : TTY=pts/0 ; PWD=/home/mbs ; USER=root ; COMMAND=/usr/bin/apt upgrade
Feb 07 21:02:42 mailhog sudo[701]: pam_unix(sudo:session): session opened for user root by mbs(uid=0)
Feb 07 21:03:02 mailhog dbus-daemon[198]: [system] Reloaded configuration
Feb 07 21:03:02 mailhog dbus-daemon[198]: [system] Reloaded configuration
Feb 07 21:03:02 mailhog dbus-daemon[198]: [system] Reloaded configuration
Feb 07 21:03:02 mailhog dbus-daemon[198]: [system] Reloaded configuration
Feb 07 21:03:02 mailhog dbus-daemon[198]: [system] Reloaded configuration
Feb 07 21:03:02 mailhog dbus-daemon[198]: [system] Reloaded configuration
Feb 07 21:03:02 mailhog dbus-daemon[198]: [system] Reloaded configuration
Feb 07 21:03:02 mailhog dbus-daemon[198]: [system] Reloaded configuration
Feb 07 21:03:02 mailhog dbus-daemon[198]: [system] Reloaded configuration
Feb 07 21:03:02 mailhog dbus-daemon[198]: [system] Reloaded configuration
Feb 07 21:03:02 mailhog dbus-daemon[198]: [system] Reloaded configuration
Feb 07 21:03:03 mailhog dbus-daemon[198]: [system] Reloaded configuration
Feb 07 21:03:03 mailhog dbus-daemon[198]: [system] Reloaded configuration
Feb 07 21:03:03 mailhog dbus-daemon[198]: [system] Reloaded configuration
Feb 07 21:03:04 mailhog systemd[1]: Reloading.
Feb 07 21:03:04 mailhog systemd[1]: system.slice: Failed to reset devices.list: Operation not permitted
Feb 07 21:03:04 mailhog systemd[1]: lxd.socket: Failed to reset devices.list: Operation not permitted
Feb 07 21:03:04 mailhog systemd[1]: systemd-networkd-wait-online.service: Failed to reset devices.list: Operation not permitted
... several pages of various 'Operation not permitted' which I believe is normal in an unprivileged container , then a lot of service starting/stopping. Then the network manager reports it successfully got an IP (this is the new IP):

Code:
Feb 07 21:03:10 mailhog systemd-networkd[1128]: eth0: DHCPv4 address 10.0.55.26/24 via 10.0.55.1
Feb 07 21:03:10 mailhog systemd-networkd[1128]: eth0: Configured
Feb 07 21:03:10 mailhog systemd-networkd-wait-online[1130]: managing: eth0
Feb 07 21:03:10 mailhog systemd-networkd-wait-online[1130]: ignoring: lo
Feb 07 21:03:10 mailhog systemd[1]: Started Wait for Network to be Configured.
-- Subject: Unit systemd-networkd-wait-online.service has finished start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit systemd-networkd-wait-online.service has finished starting up.
-- 
-- The start-up result is RESULT.
After reconnecting to the container via SSH on the new IP, I discovered I could not ping anything:

Code:
$ ping 10.0.55.1
connect: Invalid argument
After looking at the routes, it became apparent why it was failing:

Code:
$ ip route
default via 10.0.55.1 dev eth0 proto dhcp src 10.0.55.25 metric 100 
default via 10.0.55.1 dev eth0 proto dhcp src 10.0.55.26 metric 100 
10.0.55.0/24 dev eth0 proto kernel scope link src 10.0.55.26 
10.0.55.1 dev eth0 proto dhcp scope link src 10.0.55.25 metric 100 
10.0.55.1 dev eth0 proto dhcp scope link src 10.0.55.26 metric 100
The routes for the old IP had never been cleared, so it was trying to access the network on an IP that it no longer had.

To make things more confusing, I was later able to ping anything EXCEPT the PFSense box, without having changed anything.

What could have caused the double-IP requisition from the container? Why would PFSense offer a second IP instead of the same one, to the same MAC address? Why were the routes not cleared when the IP was changed?

Sorry if this post was information overload, but I do not know what could have caused this, so I don't know what might be relevant. If I have not included any information that may be relevant, please let me know and I will supply it if I can.
 
Old 02-23-2019, 03:43 PM   #2
tehfishman
LQ Newbie
 
Registered: Feb 2019
Posts: 1

Rep: Reputation: Disabled
I'm experiencing the same issue with Ubuntu Server 18.04 in LXC with PFSense as a network gateway - containers will get a new IP address from DHCP and will not remove the old route when the lease expires. The only part of my setup that appears to be different from yours is that I'm running this in proxmox with openvswitch.

I have noticed that, across my containers, they all seem to have this issue when the unattended-upgrades systemd timer runs (ships by default) and systemd does a reload. I suspect that for me, a short-term workaround may be to just disable the unattended upgrades so that I can have some stability, and reboot my containers whenever I install patches.

I've added my systemd-networkd logs below, since all the other logs in my setup reflect pretty much the same thing. In my case, you can see it happens multiple times. I feel like maybe this is a combination of bugs like what you alluded to at the end of your post - PFSense DHCP shouldn't be issuing a lease for a new IP on a known MAC, and (IMO) systemd-networkd shouldn't be requesting a new lease.

Code:
user@syncthing:~$ sudo journalctl -u systemd-networkd -b 0
-- Logs begin at Mon 2018-07-02 04:24:18 UTC, end at Sat 2019-02-23 20:18:30 UTC. --
Feb 04 20:11:35 syncthing systemd[1]: systemd-networkd.service: Failed to reset devices.list: Operation not permitted
Feb 04 20:11:35 syncthing systemd[1]: Starting Network Service...
Feb 04 20:11:35 syncthing systemd-networkd[131]: eth0: IPv6 successfully enabled
Feb 04 20:11:35 syncthing systemd-networkd[131]: Enumeration completed
Feb 04 20:11:35 syncthing systemd[1]: Started Network Service.
Feb 04 20:11:35 syncthing systemd-networkd[131]: eth0: Gained carrier
Feb 04 20:11:36 syncthing systemd-networkd[131]: eth0: DHCPv4 address 192.168.2.152/24 via 192.168.2.1
Feb 04 20:11:37 syncthing systemd-networkd[131]: eth0: Gained IPv6LL
Feb 04 20:11:37 syncthing systemd-networkd[131]: eth0: Configured
Feb 08 06:03:00 syncthing systemd[1]: systemd-networkd.service: Failed to reset devices.list: Operation not permitted
Feb 08 06:03:01 syncthing systemd[1]: systemd-networkd.service: Failed to reset devices.list: Operation not permitted
Feb 08 06:03:03 syncthing systemd[1]: systemd-networkd.service: Failed to reset devices.list: Operation not permitted
Feb 08 06:03:03 syncthing systemd[1]: systemd-networkd.service: Failed to reset devices.list: Operation not permitted
Feb 19 06:46:02 syncthing systemd[1]: systemd-networkd.service: Failed to reset devices.list: Operation not permitted
Feb 19 06:46:04 syncthing systemd[1]: systemd-networkd.service: Failed to reset devices.list: Operation not permitted
Feb 19 06:46:04 syncthing systemd[1]: systemd-networkd.service: Failed to reset devices.list: Operation not permitted
Feb 19 06:46:05 syncthing systemd[1]: systemd-networkd.service: Failed to reset devices.list: Operation not permitted
Feb 19 06:46:05 syncthing systemd[1]: systemd-networkd.service: Failed to reset devices.list: Operation not permitted
Feb 19 06:46:15 syncthing systemd[1]: systemd-networkd.service: Failed to reset devices.list: Operation not permitted
Feb 19 06:46:15 syncthing systemd[1]: Stopping Network Service...
Feb 19 06:46:15 syncthing systemd[1]: Stopped Network Service.
Feb 19 06:46:15 syncthing systemd[1]: systemd-networkd.service: Failed to reset devices.list: Operation not permitted
Feb 19 06:46:15 syncthing systemd[1]: Starting Network Service...
Feb 19 06:46:15 syncthing systemd-networkd[15403]: eth0: Gained IPv6LL
Feb 19 06:46:15 syncthing systemd-networkd[15403]: Enumeration completed
Feb 19 06:46:15 syncthing systemd[1]: Started Network Service.
Feb 19 06:46:17 syncthing systemd-networkd[15403]: eth0: DHCPv4 address 192.168.2.165/24 via 192.168.2.1
Feb 19 06:46:17 syncthing systemd-networkd[15403]: eth0: Configured
Feb 23 18:17:20 syncthing systemd[1]: Stopping Network Service...
Feb 23 18:17:20 syncthing systemd[1]: Stopped Network Service.
Feb 23 18:17:20 syncthing systemd[1]: systemd-networkd.service: Failed to reset devices.list: Operation not permitted
Feb 23 18:17:20 syncthing systemd[1]: Starting Network Service...
Feb 23 18:17:20 syncthing systemd-networkd[20314]: eth0: Gained IPv6LL
Feb 23 18:17:20 syncthing systemd-networkd[20314]: Enumeration completed
Feb 23 18:17:20 syncthing systemd[1]: Started Network Service.
Feb 23 18:17:22 syncthing systemd-networkd[20314]: eth0: DHCPv4 address 192.168.2.168/24 via 192.168.2.1
Feb 23 18:17:22 syncthing systemd-networkd[20314]: eth0: Configured
Hopefully I've done this right - I'm new around here and found this post while googling.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Potential Exploit? Potential Backdoor? Strange code in '/usr/android/adb' Package: android-tools-adb slicktrail Linux - Security 1 12-05-2016 06:05 AM
[SOLVED] Potential bug in NetworkManager.Slackbuild on -current (patches not applied) K4rolis Slackware 8 01-17-2016 02:13 PM
Potential save password bug in slackbuilds.org remmina script TL_CLD Slackware 1 03-01-2012 05:07 PM
Learning Bash, potential bug or total misunderstanding regarding globbing. crispyleif Linux - Newbie 16 02-07-2009 03:32 PM
dhcp bug or bug in my head belda Linux - Networking 2 01-26-2008 09:46 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 12:04 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration