LinuxQuestions.org - Odroid-C4/HC4 S905X3 (aarch64)

- slarm64 (https://www.linuxquestions.org/questions/slarm64-132/)

- - Odroid-C4/HC4 S905X3 (aarch64) (https://www.linuxquestions.org/questions/slarm64-132/odroid-c4-hc4-s905x3-aarch64-4175705616/)

Odroid-C4/HC4 S905X3 (aarch64)

previous discussion Odroid-C4

The Odroid Team has provided a board for testing.

fixed work legacy kernel 4.9.295

Code:

U-Boot 2021.10-meson-sm1 (Jan 03 2022 - 04:14:46 +0200) odroid-c4/hc4

Code:

Linux odroid-c4 4.9.295 #1 SMP PREEMPT Fri Dec 31 21:50:18 EET 2021 aarch64 GNU/Linux

slarm64-current-aarch64-server-odroid_c4-4.9.295-build-20220103.img.zst
slarm64-current-aarch64-server-odroid_c4-4.9.295-build-20220103.img.zst.sha256

Good work on the c4/hc4. I reloaded my hc4 this week and notice I see IRQ errors (irq37). Under the new Kernel
- there is a hang/slowdown for couple minutes then recovers (from ~37sec to 214sec in this sample)

Just wanted to pass this along and I think a fix might be to add irqpoll to extraargs in uEnv.txt (I usually add the fsck options too as the power sometimes blinks.)

cat /boot/uEnv.txt

Code:

verbosity=4

console=both

extraargs=irqpoll fsck.mode=force fsck.repair=yes

ethaddr=00:50:43:84:fb:2f

fdtfile=meson-sm1-odroid-hc4.dtb

This is what I was seeing in dmesg when the odroid-hc4 was sluggish / non responsive

Code:

rcu: INFO: rcu_preempt self-detected stall on CPU

[  84.254659] rcu: 0-....: (14960 ticks this GP) idle=ba1/1/0x4000000000000002 softirq=1735/1735 fqs=6597

[  84.255291] (t=15000 jiffies g=893 q=745)

[  84.255551] Task dump for CPU 0:

[  84.255709] task:kworker/0:1H    state:R  running task    stack:    0 pid: 1180 ppid:    2 flags:0x0000000a

[  84.256381] Workqueue:  0x0 (mmc_complete)

[  84.256913] Call trace:

[  84.257048]  dump_backtrace+0x0/0x1d0

[  84.257488]  show_stack+0x18/0x70

[  84.257871]  sched_show_task+0x14c/0x170

[  84.258173]  dump_cpu_task+0x44/0x54

[  84.258509]  rcu_dump_cpu_stacks+0xe8/0x12c

[  84.258857]  rcu_sched_clock_irq+0xa24/0xd40

[  84.259220]  update_process_times+0x9c/0xec

[  84.259571]  tick_sched_handle+0x34/0x60

[  84.259969]  tick_sched_timer+0x4c/0xa4

[  84.260355]  __hrtimer_run_queues+0x138/0x1e0

[  84.260711]  hrtimer_interrupt+0xe8/0x244

[  84.261064]  arch_timer_handler_phys+0x38/0x50

[  84.261429]  handle_percpu_devid_irq+0x84/0x130

[  84.261751]  handle_domain_irq+0x60/0x90

[  84.262112]  gic_handle_irq+0x4c/0xd0

[  84.262474]  call_on_irq_stack+0x2c/0x54

[  84.262786]  do_interrupt_handler+0x54/0x60

[  84.263122]  el1_interrupt+0x30/0x80

[  84.263486]  el1h_64_irq_handler+0x18/0x24

[  84.263869]  el1h_64_irq+0x78/0x7c

[  84.264149]  finish_task_switch.isra.0+0x9c/0x270

[  84.264450]  __schedule+0x2e4/0x8a0

[  84.264731]  preempt_schedule_irq+0x44/0x130

[  84.265026]  el1_interrupt+0x60/0x80

[  84.265383]  el1h_64_irq_handler+0x18/0x24

[  84.265760]  el1h_64_irq+0x78/0x7c

[  84.266034]  schedule+0x60/0x104

[  84.266305]  worker_thread+0x1a8/0x470

[  84.266627]  kthread+0x150/0x160

[  84.266896]  ret_from_fork+0x10/0x20

[  213.542830] irq 37: nobody cared (try booting with the "irqpoll" option)

[  213.543163] CPU: 0 PID: 1781 Comm: irq/37-0.0:00 Tainted: G        C        5.15.12 #1

[  213.543572] Hardware name: Hardkernel ODROID-HC4 (DT)

[  213.543759] Call trace:

[  213.543892]  dump_backtrace+0x0/0x1d0

[  213.544362]  show_stack+0x18/0x70

[  213.544745]  dump_stack_lvl+0x68/0x84

[  213.545149]  dump_stack+0x18/0x34

[  213.545517]  __report_bad_irq+0x4c/0xdc

[  213.545857]  note_interrupt+0x32c/0x3ec

[  213.546255]  handle_irq_event+0xd0/0x140

[  213.546615]  handle_fasteoi_irq+0xa4/0x1f4

[  213.546916]  handle_domain_irq+0x60/0x90

[  213.547264]  gic_handle_irq+0x4c/0xd0

[  213.547621]  call_on_irq_stack+0x2c/0x54

[  213.547929]  do_interrupt_handler+0x54/0x60

[  213.548263]  el1_interrupt+0x30/0x80

[  213.548623]  el1h_64_irq_handler+0x18/0x24

[  213.549000]  el1h_64_irq+0x78/0x7c

[  213.549272]  _raw_spin_unlock_irq+0x14/0x50

[  213.549611]  irq_thread_fn+0x60/0x9c

[  213.549969]  irq_thread+0x158/0x290

[  213.550322]  kthread+0x150/0x160

[  213.550588]  ret_from_fork+0x10/0x20

[  213.550894] handlers:

[  213.551011] [<0000000029cf190a>] irq_default_primary_handler threaded [<0000000017490907>] phy_interrupt

[  213.551771] Disabling IRQ #37

UPDATE
irqpoll - no help

Thanks for the info, I'll take a look at Odroid-C4.

Seems a ok under 4.9.295 so it does not seem to be hardware related.
- reproducible for me under 5.15.12 (and IRQ 37 is Ethernet 10/100/1000) if that is helpful...

I'm testing on the hc4 btw

Thanks, it's helpful.

less important but also noticeable, under the newer kernel (init 6; reboot; shutdown -r 1) all yield power down without restarting,
- I spun up (slarm64-current-aarch64-base-odroid_c4-5.10.5-build-20210107.img.zst)
-sata is recognized (hc4 model) ((seems to not see sata natively on 4.9.295))
-reboot works

Kernel slowdown / hangs seem fixed, but dmesg still complaining about IRQ#37 on the "hc4"
SATA - working well
- reboot seems to need physical unplug/replug

(which is to say:
init 6
reboot
shutdown -r 1

all leave the hc4 in a frozen (but seemly stopped) state that needs a unplug/replug.

dmesg:

Code:

[  60.425074] irq 37: nobody cared (try booting with the "irqpoll" option)

[  60.425090] CPU: 0 PID: 1786 Comm: irq/37-0.0:00 Tainted: G        C        5.16.2 #1

[  60.425096] Hardware name: Hardkernel ODROID-HC4 (DT)

[  60.425100] Call trace:

[  60.425102]  dump_backtrace+0x0/0x1d0

[  60.425115]  show_stack+0x18/0x70

[  60.425120]  dump_stack_lvl+0x68/0x84

[  60.425127]  dump_stack+0x18/0x34

[  60.425131]  __report_bad_irq+0x4c/0xdc

[  60.425139]  note_interrupt+0x32c/0x3ec

[  60.425145]  handle_irq_event+0xd0/0x140

[  60.425151]  handle_fasteoi_irq+0xa4/0x1f4

[  60.425155]  generic_handle_domain_irq+0x3c/0x60

[  60.425160]  gic_handle_irq+0x44/0xc4

[  60.425165]  call_on_irq_stack+0x2c/0x54

[  60.425169]  do_interrupt_handler+0x80/0x94

[  60.425174]  el1_interrupt+0x34/0x84

[  60.425178]  el1h_64_irq_handler+0x18/0x24

[  60.425183]  el1h_64_irq+0x78/0x7c

[  60.425186]  wake_threads_waitq+0x0/0x60

[  60.425191]  kthread+0x178/0x184

[  60.425196]  ret_from_fork+0x10/0x20

[  60.425200] handlers:

[  60.425202] [<00000000c2ce8197>] irq_default_primary_handler threaded [<00000000b8e3c45e>] phy_interrupt

[  60.425218] Disabling IRQ #37

So far this version still seems to behave the best:

Code:

slarm64-current-aarch64-base-odroid_c4-5.10.5-build-20210107.img.zst

Could not find any issue...

I'm also using two different HC4s here is my feedback:

5.15.12 from -current seems to work fine, but you have to give it a minute for it to disable IRQ 37 as I'm also seeing the

Code:

[ 60.425074] irq 37: nobody cared (try booting with the "irqpoll" option)

error. My system reports that irq 37 is:

Code:

meson-gpio-irqchip 26 Level 0.0:00

When booted with irqpoll, my system burns a bit of CPU polling the IRQ as the load average hovers around 1

Code:

# cat /proc/interrupts

          CPU0      CPU1      CPU2      CPU3

 37:  87817875          0          0          0  meson-gpio-irqchip  26 Level    0.0:00

When I go back to default then the load average drops and everything seems fine after the irq is disabled.

I went ahead and loaded up 5.16.2 and it appears to act exactly the same, it's slow until the IRQ is disabled, then seems to work okay.

schu

here is a similar problem, but the interrupt is different.

Quote:

Originally Posted by sndwvs (Post 6328511)

here is a similar problem, but the interrupt is different.

Thanks for that. I rebuilt 5.16.2 from your source package with that patch and I no longer see the IRQ issue and the machine runs nicely. Can you add that to your list of patches for building the next kernels?

Also, I updated the mirrors.aptalaska.net to have the slarm64-15.0 build. It's what I can do to help.

Quote:

Originally Posted by akschu (Post 6330629)

I spoke too soon. The patch causes the kernel to run well and not disable the IRQ, but it no longer allows me to reboot the machine without power cycle.

Disabling the IRQ is less of a problem for me than not being able to remotely reboot.

Quote:

Originally Posted by akschu (Post 6330629)

Hi Akschu,
thanks for testing and maintaining the repository. added patch.