eth0 losing connection from dhcp server

abefroman · 09-18-2017, 06:08 PM

FreePBX (CentOS 7) is crashing every 15-60 minutes on AWS EC2 instance.

Here's what I have in /var/log/messages (the sigint is where I rebooted it):

Code:

Sep 16 09:21:11 freepbx dnsmasq[599]: reading /etc/resolv.conf
Sep 16 09:21:11 freepbx dnsmasq[599]: using nameserver 172.31.0.2#53
Sep 16 09:21:11 freepbx avahi-daemon[600]: Withdrawing address record for 172.31.20.124 on eth0.
Sep 16 09:21:11 freepbx avahi-daemon[600]: Leaving mDNS multicast group on interface eth0.IPv4 with address 172.31.20.124.
Sep 16 09:21:11 freepbx avahi-daemon[600]: Interface eth0.IPv4 no longer relevant for mDNS.
Sep 16 09:30:18 freepbx auditd[577]: Audit daemon rotating log files
Sep 16 09:39:18 freepbx systemd-logind: Removed session 57.
Sep 16 11:24:44 freepbx systemd: Received SIGINT.

It seems like its shutting down eth0, I don't have any web or ssh access when it does that. However if I get an instance screenshot it shows its at the login prompt, so it seems like the server is fine, it just doesn't have network connectivity any more.

I ran:

Code:

sar -u 2 5000

and notice there was a CPU spike, but only to 38%, right before I lost connection.

I contacted FreePBX and they blamed the problem on AWS, I contacted AWS and as you can guess they blamed the problem on FreePBX.

Thanks in advance.

Update: An AWS engineer responded as follows:

Quote:

As the issue was with the private IP being released (at-least appears in messages), we checked the DHCP server of the instance to see if the DHCP client of the instance is sending the DHCPRequest or not. Based on the investigation and logs, I can confirm that the DHCP client of the instance ##### is requesting the private IP 172.xxx.xxx.xxx of the instance via DHCPRequest and get the renewal successfully via DHCPAck. Having said that, it appears that the IP address is not being removed, it just that avahi detected that the IP address has been removed, and is changing its internal state in response and we need to look further.

jefro · 09-19-2017, 04:30 PM

Set up wireshark to view the handshake for yourself?

Set up new instance with exact same OS or different OS to see if same happens?

abefroman · 09-19-2017, 06:07 PM

Quote:

Originally Posted by jefro

Set up wireshark to view the handshake for yourself?

Set up new instance with exact same OS or different OS to see if same happens?

Jefro, it seems your right, its NOT making a DHCP request, and going offline when ever the DHCP lease expires.

And on a new instance it works fine UNTIL I set the time zone.

Code:

ln -s /usr/share/zoneinfo/America/Chicago /etc/localtime

And its odd, because when the server boots up, it has UTC, even with that symlink.

Code:

[root@voip6 ~]# date
Tue Sep 19 21:43:33 CDT 2017
[root@voip6 ~]# date
Tue Sep 19 16:43:35 CDT 2017
[root@voip6 ~]# tail -f /var/log/messages 
Sep 19 21:43:12 voip6 dhclient[1334]: XMT: Solicit on eth0, interval 2110ms.
Sep 19 21:43:14 voip6 dhclient[1334]: XMT: Solicit on eth0, interval 4180ms.
Sep 19 21:43:18 voip6 dhclient[1334]: XMT: Solicit on eth0, interval 8450ms.
Sep 19 21:43:26 voip6 dhclient[1334]: XMT: Solicit on eth0, interval 17380ms.
Sep 19 21:43:30 voip6 systemd-logind: New session 1 of user root.
Sep 19 21:43:30 voip6 su: (to asterisk) root on pts/1
Sep 19 16:43:33 voip6 chronyd[626]: Selected source 173.49.198.27
Sep 19 16:43:33 voip6 chronyd[626]: System clock wrong by -18000.000537 seconds, adjustment started
Sep 19 16:43:33 voip6 chronyd[626]: System clock was stepped by -18000.000537 seconds
Sep 19 16:43:33 voip6 systemd: Time has been changed

Could it be its not making a new DHCP request because it doesn't think its time yet, due to the timezone change?

jefro · 09-19-2017, 10:18 PM

https://linux.die.net/man/1/dhcpdump may help. Think it can output to log file.

TZ could do that if some other things flakey. Wonder if you ran a script that did both the time zone and followed with a lease renew.

abefroman · 09-19-2017, 10:38 PM

Code:

[root@voip6 ~]# date
Tue Sep 19 21:43:33 CDT 2017
[root@voip6 ~]# date
Tue Sep 19 16:43:35 CDT 2017

This is as its booting up (I just happened to do it quickly), some how its pulling the UTC time, but putting CDT for the time zone. I have rdate set to run by a cron at midnight, however its running when the server reboots (because crond is restarting), but after it gets the first dhcp lease. rdate then resets the clock back 5 hours, but then it loses network connectivity before renewing the dhcp lease and needs to be restart.

So the question is why is it putting the current UTC time with the CDT timezone?

lazydog · 09-20-2017, 01:58 PM

I do not believe this is the issue because the DHCP server only hands out DHCP addresses it doesn't control network access. At some point the server will ask the client or the client will inform the server that the lease is still needed. This usually happens long before the lease has expired. Maybe this isn't happening or the request isn't getting to the system.

jefro · 09-21-2017, 03:41 PM

Some ideas maybe.

https://serverfault.com/questions/54...nd-showing-utc

https://unix.stackexchange.com/quest...te-and-hwclock

linux2017 · 10-20-2017, 10:29 AM

abefroman, I'm having this exact issue. I'm running the FreePBX distro (Sangoma 7). with FreePBX 14.0.1.14 and Asterisk 14.6.2-1.sng7, inside an Amazon EC2 Instance via an AMI image i uploaded.

Did you resolve this?

It only happens when I change the timezone from UTC to New_York. (server is rebooted after changing timezone). Doesn't matter if I change timezone from command line or via the FreePBX GUI the system loses connectivity within 15 to 60 minutes until I reboot.

Problem occurs with this configuration:

[root@freepbx ~]# ls -l /etc/localtime
lrwxrwxrwx 1 root root 36 Oct 19 15:31 /etc/localtime -> /usr/share/zoneinfo/America/New_York
[root@freepbx ~]#

Problem does not occur with this configuration:

[root@freepbx ~]#
[root@freepbx]# ls -la /etc/localtime
lrwxrwxrwx. 1 root root 25 Oct 12 18:35 /etc/localtime -> ../usr/share/zoneinfo/UTC
[root@freepbx]#