[SOLVED] Raspberry pi 2 runs fine for a month and then starts freezing more and more

yveske · 05-10-2020, 09:16 PM

I'm running Raspbian Buster Lite, headless on a raspberry pi 2 and this problem actually started around the end of 2019. Every time this start happening I do a fresh install and I'm basically good for a month and then the problem starts again, it starts with freezing once a day but very quickly it’s up to multiple times a day (5-6 times). With freezing I mean I can’t do anything to reach it and my only option is a hard reset.

I included syslog (log is from moment it freezes but seems like logs from starting up) and debug log but have all other logs if needed.
Thanks in advance.

pan64 · 05-11-2020, 02:56 AM

do you really want to run all those things on the rpi2? (mysql, mono, transmission, dhcpd, samba, ...)
I would guess it is out of memory (but would be nice to give more details).

fatmac · 05-11-2020, 05:04 AM

SD cards were originally created as storage media, constant reading & writing will wear them out.

If you're not already doing so, boot from SD card, but run your system from an external HDD or SSD.
(& giving it a reasonable allowance of swap space will probably help too).

yveske · 05-11-2020, 06:03 AM

Quote:

Originally Posted by fatmac

SD cards were originally created as storage media, constant reading & writing will wear them out.

If you're not already doing so, boot from SD card, but run your system from an external HDD or SSD.
(& giving it a reasonable allowance of swap space will probably help too).

I have been trying this before but wasn't a solution as well. Also, the SD card should be fine, I have been using two new cards since my first ones were worn out. And if I test them they are good. I think pan64 is right and it's time to upgrade, think the things I run on it (especially mono) are getting a bit too much.

yveske · 05-11-2020, 06:10 AM

Quote:

Originally Posted by pan64

do you really want to run all those things on the rpi2? (mysql, mono, transmission, dhcpd, samba, ...)
I would guess it is out of memory (but would be nice to give more details).

It actually have been running fine this way for years, must be 5 years now. But you are probably right, it's probably time to upgrade. Those programs, especially mono, aren't getting any lighter.

I did use a cronjob to check my memory every 5 minutes and the free memory seems low

Code:

              total        used        free      shared  buff/cache   available
Mem:            926         370          37           8         518         492
Swap:            99           7          92
              total        used        free      shared  buff/cache   available
Mem:            926         370          36           8         518         491
Swap:            99           7          92
              total        used        free      shared  buff/cache   available
Mem:            926         370          37           8         518         491
Swap:            99           7          92
              total        used        free      shared  buff/cache   available
Mem:            926         370          37           8         518         492
Swap:            99           7          92
              total        used        free      shared  buff/cache   available
Mem:            926         370          37           8         518         491
Swap:            99           7          92
              total        used        free      shared  buff/cache   available
Mem:            926          40         758           6         127         830
Swap:            99           0          99
              total        used        free      shared  buff/cache   available
Mem:            926         256         378           7         290         611
Swap:            99           0          99
              total        used        free      shared  buff/cache   available
Mem:            926         274         359           7         291         593
Swap:            99           0          99

Last three are from after rebooting.

pan64 · 05-11-2020, 06:31 AM

I would switch off swap and also check if there was a disk full somewhere...
What OS is it? How is it partitioned?

syg00 · 05-11-2020, 06:46 AM

What benefit is there in disabling swap ?. Especially if re-booting 5 or 6 times a day.
There has to be some logs somewhere. Unless something doesn't want to be seen. When you re-install, do you use the same passwords and security settings ?.

yveske · 05-11-2020, 06:52 AM

Quote:

Originally Posted by pan64

I would switch off swap and also check if there was a disk full somewhere...
What OS is it? How is it partitioned?

Switched off swap and have two partitions on the SD card, a boot and a root partition. They are both not half full. I'm running Raspbian Buster Lite from the raspberry pi website.

yveske · 05-11-2020, 06:58 AM

Quote:

Originally Posted by syg00

What benefit is there in disabling swap ?. Especially if re-booting 5 or 6 times a day.
There has to be some logs somewhere. Unless something doesn't want to be seen. When you re-install, do you use the same passwords and security settings ?.

Got nothing to hide, what logs could come in handy? I do usually generate a new password when I do a fresh install, root login disabled.

syg00 · 05-11-2020, 07:02 AM

No, I wasn't aspersing you - I was wondering if you had been hacked by something like a bitcoin miner say. Some of the good ones will attempt to cover their presence, although removal of logs is pretty drastic.
My thought was if you used the same passwords you might be (more) subject to being re-attacked.

I've never used Debian - does it forward the systemd journal to rsyslog ?.

yveske · 05-11-2020, 07:20 AM

Quote:

Originally Posted by syg00

No, I wasn't aspersing you - I was wondering if you had been hacked by something like a bitcoin miner say. Some of the good ones will attempt to cover their presence, although removal of logs is pretty drastic.
My thought was if you used the same passwords you might be (more) subject to being re-attacked.

Haha, ok I see. Wouldn't this be a bit over the top for the bit of power they could get from my device?

Quote:

Originally Posted by syg00

I've never used Debian - does it forward the systemd journal to rsyslog ?.

It isn't but I enabled ForwardToSyslog=yes in /etc/systemd/journald.conf

yveske · 05-11-2020, 02:59 PM

Quote:

Originally Posted by syg00

I've never used Debian - does it forward the systemd journal to rsyslog ?.

These are the logs of my latest freeze, now they include the systemd journal.
Turning the swap memory to zero didn't work.

ondoho · 05-12-2020, 12:18 AM

Quote:

Originally Posted by yveske

Wouldn't this be a bit over the top for the bit of power they could get from my device?

"They" is probably an automated script somewhere that just tries to get in wherever possible. It probably doesn't even know how capable your machine is and doesn't care if it occasionally kills a host.

But I think it more likely that your raspi2 is a bit overloaded with mysql, mono, transmission, dhcpd, samba...

yveske · 05-12-2020, 02:20 AM

Quote:

Originally Posted by ondoho

"They" is probably an automated script somewhere that just tries to get in wherever possible. It probably doesn't even know how capable your machine is and doesn't care if it occasionally kills a host.

Good point

Quote:

Originally Posted by ondoho

But I think it more likely that your raspi2 is a bit overloaded with mysql, mono, transmission, dhcpd, samba...

Yes, was afraid of that. Well, I shouldn't complain, five years running 24/7 for €40 seems a good deal. Guess it's finally time to update to a raspi4

Thanks everyone for their time.

pan64 · 05-12-2020, 02:44 AM

you can still use pi2 (actually I run pihole on it).