Linux froze and got back working 5 days later, no reboot.

Hello my friends,

I've come across the weirdest of glitches.

I'm running a simple weather monitor (just temperature) on a Raspbeery Pi 3B (Raspbian Linux 4.14.98-v7+). It has a 3G modem that sends out simple packets to my server at home.

On june 5th I lost remote access to the device. Since it is located somewhere difficult to physically access, I couldn't manually reboot it.

To my surprise, on June 10th it got back to life. It has not registered a reboot! Here's the weird excerpt from the log at /var/log/messages:

Jun 5 17:54:28 raspberrypi NetworkManager[345]: <info> [1559768068.0169] policy: set 'VIVO ZAP' (wwan0) as default for IPv4 routing and DNS
Jun 5 17:54:28 raspberrypi NetworkManager[345]: <info> [1559768068.0174] dns-mgr: Writing DNS information to /sbin/resolvconf
Jun 5 17:54:28 raspberrypi NetworkManager[345]: <info> [1559768068.1295] device (ttyUSB2): Activation: successful, device activated.
Jun 10 23:14:33 raspberrypi NetworkManager[345]: <info> [1560219273.9397] (mon0): using nl80211 for WiFi device control
Jun 10 23:14:33 raspberrypi NetworkManager[345]: <info> [1560219273.9622] devices added (path: /sys/devices/platform/soc/3f300000.mmc/mmc_host/mmc1/mmc1:0001/mmc1:0001:1/net/mon0, iface: mon0)
Jun 10 23:14:33 raspberrypi NetworkManager[345]: <info> [1560219273.9623] device added (path: /sys/devices/platform/soc/3f300000.mmc/mmc_host/mmc1/mmc1:0001/mmc1:0001:1/net/mon0, iface: mon0): no ifupdown configuration found.

It's as if the operating system froze for some reason and suddenly got back working 5 days later! Absolutely no reboot registered.

I have absolutely no other log.

Does anyone know what could have happened there?

Welcome to LQ!

I would not interpret 5 days of missing logs as "froze".

The last activity before the gap looks like a change to the DNS resolver and activation of a USB/tty device. The first activity after the gap is restoration of the wifi interface.

I am not much knowledgable about Pi's, but my first impression is that the device was accessed locally (to itself) and possibly used for another purpose during that time, with the logs then cleared and the device restored, although that is a very poorly informed guess.

Yeah, I was wondering how much hardening had been done on it too ...

Originally Posted by astrogeek Welcome to LQ! I would not interpret 5 days of missing logs as "froze". The last activity before the gap looks like a change to the DNS resolver and activation of a USB/tty device. The first activity after the gap is restoration of the wifi interface. I am not much knowledgable about Pi's, but my first impression is that the device was accessed locally (to itself) and possibly used for another purpose during that time, with the logs then cleared and the device restored, although that is a very poorly informed guess.

Hello, thanks for the reply.

No one has physical access to the device locally. It is installed 100 feet high and locked inside a case.

My thoughts were along different lines…

Weird hardware issues don't generally get logged. For instance, think of the havoc a single drop of water could provide bridging the legs of something. You're probably familiar with the unlogged delays trying to mount an unreadable cdrom. All you need is a bit of hardware retrying for a low level and getting a high one. Dampness is a great cause of 'gremlins.' And water evaporates, leaving no trace.

Back in the day when I did Electronic hardware repairs, there used to be 'sealants,' insulated lacquers for spraying on PCBs. You might find them locally. Anything stuck out should be well sealed - IP67, I think.