Orange Pi and kernel 4.19.1 (system non responsive after 5am each day)

ricky_cardo · 11-27-2018, 07:08 PM

Thanks for suggestions ! and assistance!
One bug crushed (of my own making)

Problem 1: I have long used the internal emmc for root and SATA for storage (mostly backups of other systems on lan)
The problem was I started running out of space...logs grow and tmp (should have put tmp and var on the sata partitions)
as a bandaid I made links:

Code:

lrwxrwxrwx   1 root root     10 Sep 29 16:22 tmp -> /boxee/tmp/
drwxr-xr-x  16 root root   4096 Mar  9  2002 usr/
lrwxrwxrwx   1 root root     10 Sep 29 11:32 var -> /boxee/var/
lrwxrwxrwx   1 root root     18 Nov 27 19:01 SlackBuilds -> /boxee/SlackBuilds/

I forgot to add the exec fstab option....
got it now

Code:

#device          mount point      file system options                             dump  pass
UUID=2c9acaf9-e7f1-4e49-b3fb-d00ed950a414 /boxee ext4 rw,auto,async,users,exec    0     1

This was my permissions issue... (with Slackbuilds)
--remains to be seen if it was causing other issues as well (since I had /var/log in there it's possible logrote or syslog was causing lockups)
actually decided KISS

Code:

UUID=2c9acaf9-e7f1-4e49-b3fb-d00ed950a414 /boxee ext4 defaults    0     1

** don't know why this is kernel specific but we'll see.

ricky_cardo · 11-27-2018, 08:47 PM

Still locking up with 4.19 series kernel, suspect related to disk activity.
--going to revert to kernel in extra 4.17

-I added the suggested ntp commands to cron.hourly so it is possible I'll regain access in 1hr. If the date switch to 1977 can be fixed by cron.
(the lockup seems to be sshd hangs and local serial connection is non-responsive.)((but from logs system seems alive and set to 1977, from inspection of previous hangs))

Scenario that caused hang: I was backing up my SATA connected drive to a usb connected drive. (uptime was maybe 30 minutes)
using this command

Code:

 
mount /dev/sdb1 /mnt/tmp
cd /mnt/tmp ;/usr/bin/rsync -Pavv --delete /boxee .

after 3 minutes remote ssh session froze, successive connection hangs with no response, not even a timeout.

(was backing up to re-arrange partition table on SATA, make room for SWAP, /var/ and /tmp, so I can get them off the emmc correctly)

ricky_cardo · 11-27-2018, 11:03 PM

ntpdate script did not restore, so powercycled and rolled back to 4.17.19 in extra.

(feeling strongly heavy disk and ethernet activity caused CPU stall)
here is syslog from last one:

Code:

Nov 27 21:35:12 orange kernel: [ 4225.497093] [<c03019f8>] (__irq_svc) from [<c0493104>] (move_expired_inodes+0x88/0x1b4)
Nov 27 21:35:12 orange kernel: [ 4225.497096] [<c0493104>] (move_expired_inodes) from [<c049361c>] (queue_io+0x6c/0x124)
Nov 27 21:35:12 orange kernel: [ 4225.497099] [<c049361c>] (queue_io) from [<c0494b64>] (wb_writeback+0x158/0x310)
Nov 27 21:35:12 orange kernel: [ 4225.497101] [<c0494b64>] (wb_writeback) from [<c04954cc>] (wb_workfn+0x290/0x494)
Nov 27 21:35:12 orange kernel: [ 4225.497104] [<c04954cc>] (wb_workfn) from [<c03537b0>] (process_one_work+0x228/0x408)
Nov 27 21:35:12 orange kernel: [ 4225.497107] [<c03537b0>] (process_one_work) from [<c035444c>] (worker_thread+0x2b4/0x428)
Nov 27 21:35:12 orange kernel: [ 4225.497110] [<c035444c>] (worker_thread) from [<c03587c4>] (kthread+0x130/0x148)
Nov 27 21:35:12 orange kernel: [ 4225.497113] [<c03587c4>] (kthread) from [<c03010d8>] (ret_from_fork+0x14/0x3c)
Nov 27 21:35:12 orange kernel: [ 4225.497115] Exception stack(0xe9a39fb0 to 0xe9a39ff8)
Nov 27 21:35:12 orange kernel: [ 4225.497118] 9fa0:                                     00000000 00000000 00000000 00000000
Nov 27 21:35:12 orange kernel: [ 4225.497121] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Nov 27 21:35:12 orange kernel: [ 4225.497123] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000
Nov 27 21:35:12 orange kernel: [ 4225.497752] rcu: rcu_sched kthread starved for 734441 jiffies! g156301 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=0
Nov 27 21:35:12 orange kernel: [ 4225.497755] rcu: RCU grace-period kthread stack dump:
Nov 27 21:35:12 orange kernel: [ 4225.497781] [<c08b8318>] (__schedule) from [<c08b84b0>] (schedule+0x7c/0x98)
Nov 27 21:35:12 orange kernel: [ 4225.497793] [<c08b84b0>] (schedule) from [<c08bb84c>] (schedule_timeout+0x2f8/0x380)
Nov 27 21:35:12 orange kernel: [ 4225.497808] [<c08bb84c>] (schedule_timeout) from [<c0398170>] (rcu_gp_kthread+0x518/0x8c8)
Nov 27 21:35:12 orange kernel: [ 4225.497822] [<c0398170>] (rcu_gp_kthread) from [<c03587c4>] (kthread+0x130/0x148)
Nov 27 21:35:12 orange kernel: [ 4225.497833] [<c03587c4>] (kthread) from [<c03010d8>] (ret_from_fork+0x14/0x3c)
Nov 27 21:35:12 orange kernel: [ 4225.497837] Exception stack(0xee0f7fb0 to 0xee0f7ff8)
Nov 27 21:35:12 orange kernel: [ 4225.497844] 7fa0:                                     00000000 00000000 00000000 00000000
Nov 27 21:35:12 orange kernel: [ 4225.497851] 7fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Nov 27 21:35:12 orange kernel: [ 4225.497857] 7fe0: 00000000 00000000 00000000 00000000 00000013 00000000
Dec 12 15:12:34 orange kernel: [ 4815.510411] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:

Going to stay on 4.17.19 for now. Re-enable backup script (rdiff-backup) see if it goes back to stable.

--I see some talk about inode -- not ruling out some HD issue, just interesting that newer kernel causes me issues.
--going to slow this down do a couple days of backups on older kernel, backup the backup with rsync and go from there. (maybe once that's done re-arange partitions to make SWAP, and TMP mount point on the SATA drive and free up space on EMMC.) ((then I'll clean up the sloppy simlinks)).

interndan · 11-29-2018, 12:51 PM

Sorry that didn't work. As I said, I've been following this thread with an eye to getting Slackwarearm back on my Orange Pi PC, so I did a search and came across a thread on one of the Debian forums. It seems one of their upgrades had the same or at least similar effect and settting up ntp time was one of there suggested fixes. I don't know if it worked for them either, but their issue seemed also to affect Raspberry Pi with the same distro.

On my Raspberry, I have installed a rtc, which involes disabeling fake-hwclock. I don't know if it is the same for the orangepi, as I haven't gotten that far yet, but it also lets settime update the hw-clock. While My rasberry isn't affected because the new kernel hasn't been put on it, I wonder if that would bypass the issue?

drmozes · 11-29-2018, 05:07 PM

Quote:

Originally Posted by interndan

Sorry that didn't work. As I said, I've been following this thread with an eye to getting Slackwarearm back on my Orange Pi PC, so I did a search and came across a thread on one of the Debian forums. It seems one of their upgrades had the same or at least similar effect and settting up ntp time was one of there suggested fixes. I don't know if it worked for them either, but their issue seemed also to affect Raspberry Pi with the same distro.

On my Raspberry, I have installed a rtc, which involes disabeling fake-hwclock. I don't know if it is the same for the orangepi, as I haven't gotten that far yet, but it also lets settime update the hw-clock. While My rasberry isn't affected because the new kernel hasn't been put on it, I wonder if that would bypass the issue?

The OrangePI doesn't have an RTC unless you add one to the GPIO pins, so syncing its time to an NTP server is a standard thing.
This is a different issue from that - if you sync with NTP (if it even works - it didn't when I first played with 4.19), it'll still be totally broken.

The Raspberry Pi does not use the Slackware ARM Kernel.

I have two Orange Pis that work fine with 4.19, which are currently building the packages without any issue. As a test, I reverted the GPU frequency governor to "performance" and put the Kernel on to my Banana Pi and tried to build the Kernel on it (just to give it something to do), and it hung shortly after unpacking the source archive, where as it was up for 5 days with the governor set to "on demand" (As it currently is in -current's kernel). I'll put the previous build back on it later and see what happens. I think I'll be leaving the governor as "on demand" for the foreseeable future.

drmozes · 11-29-2018, 05:12 PM

It also occurred to me that the last time I tried 4.19 was when the machines were in a slightly warmer environment, where as now, it doesn't exceed much over 12 degrees C. I'm wondering whether they were getting too hot. Have you tried pointing a desk fan at it ?
My Orange Pi's also have some of the heat sinks on a few of the chips.

interndan · 11-29-2018, 06:18 PM

I was aware the Orange Pi didn't have a rtc. I thought that it was using fake-hwclock, thus my efforts. But if changing the GPU frequency to on demand resolves the problem, that obviously isn't the case. (or at least not he problem) So I'll continue to quietly follow along here while I try to fill in some of those large and rather shocking holes in my basic knowledge of the inner workings of Linux. I've been happily running Slackware on all my systems so many years now that I had no idea really they existed, at least to the extent I've discovered after I tried getting Slackware back on my Orange Pi by trying to follow your instructions.

ricky_cardo · 11-29-2018, 11:24 PM

Could be temperature related. Definitely rock solid on 4.17.19, (no hangs) (can do intensive disk writes,etc.)

The Model I have is The Orange pi plus 2 (same as orange pie plus, with more ram memory)
- no heat sinks, though I do have a fan blowing. (going to keep playing around).
(my spidey sense telling me drive write related) when running 4.19.x
--(seems to hang consistently when I use slackpkg upgrade kernel (source) ((so long as I grab the headers, kernel and modules first recovery is a reboot away))
--suspect it is the many writes to mmc (it actually hangs in the delete phase)
--It also might hang on intensive writes to esata (which is really usb, I believe I read)
--both cases it seems heavy like IO operations to disk.
--but if I do little it still hangs after ~24 hrs

Question: Is it a shared bus for NIC and DiskIO maybe something there?

I do have a bananaPiPro also and it is running fine on 4.19.5

--The interesting thing to is after the hang the device reverts to 1977, regardless if I am running ntpd or cron a manual /usr/sbin/ntpdate 192.168.1.243 (hourly)
--which means the device must loose ability to ntp when it is in a hung state.

--To answer the thermal question is there a command like on the raspberry pi to check temp?

Code:

root@slackberrypi ~ $ /opt/vc/bin/vcgencmd measure_temp                     
temp=30.4'C

be nicer to see it in IMPERIAL Temp

(we like fahrenheit here)

NOTE: needed to add these 2 links

Code:

ln -s /opt/vc/lib/libvchiq_arm.so /lib/libvchiq_arm.so
ln -s /opt/vc/lib/libvcos.so /lib/libvcos.so

On orangepi (no results)
find /sys/ -name "temp"
find /sys/ -name "thermal"

drmozes · 11-30-2018, 01:54 AM

Quote:

Originally Posted by interndan

I've been happily running Slackware on all my systems so many years now that I had no idea really they existed, at least to the extent I've discovered after I tried getting Slackware back on my Orange Pi by trying to follow your instructions.

So does it work then? you followed the instructions and what happened?
As I said, I have two Orange Pi's both installed using the instructions I wrote (as the instructions were written based on what I did to make it work first time around) and they work fine.
So presumably with ricky's it's an issue with the board itself, or the PSU or something like that, which the newer Kernel has caused to become manifest due to the way it's regulating the board. I don't know, but I'm interested to see what someone else's OrangePi does.

interndan · 11-30-2018, 08:42 AM

Actually that was when I discovered the gaping holes in my background knowledge. I was attempting to follow your instructions, and found none of them working as described. I was getting absolutely no response from the orange pi. None of the screen commands seemed to do anything, and my system kept refusing access to /export after I was sure I had changed ownership on it. When I launched the python server, it returned a 0.0.0.0 ip address. I'm sure that is wrong. and I couldn't get the tftp server to work.

I had Slackware arm on it once, using some images provided by another gentleman with some unofficial patches to get HDMI working. But I was trying to resolve an overscan issue with the little tv I use for a monitor on my work cart and broke the system. I got distracted and put it away for a while. When I returned I found none of the previous links worked and I had lost a hard drive in my desktop, so the files I had downloaded were gone.

Now that winter has inclined me more to indoor activities, I thought I would rebuild it using official packages and instructions. So I bought myself the appropriate cable and a usb hard drive and started in. So far with not much luck. Meanwhile I pulled out my Raspberry and have been toying with it while I try to fill those gaps.

drmozes · 11-30-2018, 09:44 AM

Quote:

Originally Posted by interndan

I was getting absolutely no response from the orange pi.

From what? the serial cable? Some times this is because the TX and RX are swapped - try swapping them -- but make sure you leave the PWR as it is!
Also it could be that the cable is just of poor quality. I've had several ones that would either splatter key presses in to the console (dangerous!) or corrupt the output to the console.
These are the most recent purchases, which seem to be holding up so far.
https://www.amazon.co.uk/Raspberry-P.../dp/B01N4X3BJB

interndan · 11-30-2018, 10:23 AM

Ok, thanks for the pointer. I'll see if it is available here in the US. This is the one I currently have:
https://www.amazon.com/gp/product/B0...?ie=UTF8&psc=1

Meanwhile if you have a pointer to where I can study up on the use of screen and python server I would appreciate that as well. I don't expect you to hold my hand here, but I'm really weak on both of those as well.

interndan · 11-30-2018, 10:32 AM

Ok , never mind on screen I just got my search inquiry right and found what should get me started.

drmozes · 11-30-2018, 10:53 AM

Quote:

Originally Posted by interndan

Ok, thanks for the pointer. I'll see if it is available here in the US. This is the one I currently have:
https://www.amazon.com/gp/product/B0...?ie=UTF8&psc=1

They'll be fine - it's a standard cable.

Quote:

Meanwhile if you have a pointer to where I can study up on the use of screen and python server I would appreciate that as well. I don't expect you to hold my hand here, but I'm really weak on both of those as well.

You don't need to - these tools aren't required to use Slackware on ARM. screen is just being used as a terminal emulator - you could use any other, but screen is easiest to document because the configuration of which serial port and the serial options can be supplied at the command line. screen itself is a great tool though - I've used it almost every day for over twenty years. All of my mail, IRC, whatever else, is under several screen sessions. If you use console based applications on remote Unix servers, screen allows you to leave the processes running and disconnect from the server. When you login again, you can re-open screen and continue as if you never disconnected.

The python server is simply opening an HTTP server that allows you to install over HTTP.
I found that one can easily open an HTTP server within the PWD, so I documented how to do that so people had an easy way to install (rather than installing and configuring an FTP server or Apache), if NFS wasn't available.

interndan · 11-30-2018, 12:15 PM

I see. I think however that I'll spend a couple of days perusing the manuals on the use of screen and the python server anyway. One, I think it will stand me in good stead down the road, and also I suspect I'd have almost as much trouble regardless of the tools I use. I have very little (almost none) experience in remotely accessing other systems/servers. At least from the command line.