Solve suspend issues in certain Macbooks

litelinux · 01-03-2023, 03:47 PM

UPDATE The issue came back after a kernel upgrade. Guess I need more debugging...

After some extensive troubleshooting I finally solved my long-standing suspend problem, so I'm keeping this here for reference (and for people who are in the same situation).

The problem goes as follows: starting from kernel version 5.15, my Macbook Pro (model A1398) wouldn't wake up from suspend. Running tools like Intel's pm-graph didn't yield anything interesting. So I switched the lid close action to hibernate and ignored the problem.

A few days ago I picked up the problem again and looked at various logs. dmesg didn't contain anything other than a jump in time between log entries, but the syslog had something like this:

Code:

Jan  4 05:04:51 dankstar kernel: thunderbolt 0000:07:00.0: device link creation from 0000:06:00.0 failed
Jan  4 05:04:52 dankstar kernel: ACPI Warning: SystemIO range 0x0000000000000840-0x000000000000084F conflicts with OpRegion 0x0000000000000800-0x0000000000000863 (\x5cGPIO) (20220331/utaddress-204)
Jan  4 05:04:52 dankstar kernel: ACPI Warning: SystemIO range 0x0000000000000830-0x000000000000083F conflicts with OpRegion 0x0000000000000800-0x0000000000000863 (\x5cGPIO) (20220331/utaddress-204)
Jan  4 05:04:52 dankstar kernel: ACPI Warning: SystemIO range 0x0000000000000800-0x000000000000082F conflicts with OpRegion 0x0000000000000800-0x0000000000000863 (\x5cGPIO) (20220331/utaddress-204)
Jan  4 05:04:52 dankstar kernel: ACPI Warning: SystemIO range 0x0000000000000800-0x000000000000082F conflicts with OpRegion 0x0000000000000810-0x0000000000000813 (\x5cIO_D) (20220331/utaddress-204)
Jan  4 05:04:52 dankstar kernel: ACPI Warning: SystemIO range 0x0000000000000800-0x000000000000082F conflicts with OpRegion 0x0000000000000800-0x000000000000080F (\x5cIO_T) (20220331/utaddress-204)
Jan  4 05:04:52 dankstar kernel: lpc_ich: Resource conflict(s) found affecting gpio_ich
Jan  4 05:04:52 dankstar kernel: wl: loading out-of-tree module taints kernel.
Jan  4 05:04:52 dankstar kernel: wl: module license 'MIXED/Proprietary' taints kernel.
Jan  4 05:04:52 dankstar kernel: Disabling lock debugging due to kernel taint
Jan  4 05:04:52 dankstar kernel: eth0: Broadcom BCM43a0 802.11 Hybrid Wireless Controller 6.30.223.271 (r587334)
Jan  4 05:04:52 dankstar kernel: 
Jan  4 05:04:52 dankstar kernel: applesmc applesmc.768: hwmon_device_register() is deprecated. Please convert the driver to use hwmon_device_register_with_info().
Jan  4 05:04:52 dankstar udevd[765]:  Unable to EVIOCGABS device "/dev/input/event10"
Jan  4 05:04:52 dankstar last message buffered 3 times
Jan  4 05:04:55 dankstar kernel: FAT-fs (sda1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
Jan  4 05:04:58 dankstar udevd[760]:  Unable to EVIOCGABS device "/dev/input/event10"
Jan  4 05:04:58 dankstar last message buffered 3 times
Jan  4 05:05:04 dankstar udevd[751]:  specified group 'adbusers' unknown
Jan  4 05:05:05 dankstar bluetoothd[1685]: profiles/audio/vcp.c:vcp_init() D-Bus experimental not enabled
Jan  4 05:05:05 dankstar bluetoothd[1685]: src/plugin.c:plugin_init() Failed to init vcp plugin
Jan  4 05:05:05 dankstar bluetoothd[1685]: profiles/audio/mcp.c:mcp_init() D-Bus experimental not enabled
Jan  4 05:05:05 dankstar bluetoothd[1685]: src/plugin.c:plugin_init() Failed to init mcp plugin
Jan  4 05:05:05 dankstar bluetoothd[1685]: profiles/audio/bap.c:bap_init() D-Bus experimental not enabled
Jan  4 05:05:05 dankstar bluetoothd[1685]: src/plugin.c:plugin_init() Failed to init bap plugin
Jan  4 05:05:05 dankstar bluetoothd[1685]: Failed to set privacy: Rejected (0x0b)
Jan  4 05:05:05 dankstar dnsmasq[1975]: no servers found in /etc/resolv.conf, will retry
Jan  4 05:05:06 dankstar kernel: L1TF CPU bug present and SMT on, data leak possible. See CVE-2018-3646 and https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/l1tf.html for details.
Jan  4 05:05:06 dankstar kernel: ahci 0000:04:00.0: VPD access failed.  This is likely a firmware bug on this device.  Contact the card vendor for a firmware update

Then there were no further messages until the next force restart. So I guessed that it was a thunderbolt problem. While searching for more info I found this blog entry, where someone in the comments say that the thunderbolt module needs to be disabled before suspend, and reactivated after resume. (UPDATE Bluetooth seemed to cause issues too, so I added it to the script)

Therefore the solution is to:

Add to /lib/elogind/system-sleep (or /lib64/elogind/system-sleep depending on your architecture) a file named 50-suspend-fix (could be any name):

Code:

#!/bin/sh
case "${1-}" in
    'pre')
        exec /etc/rc.d/rc.bluetooth stop
        exec modprobe -r thunderbolt
        ;;

    'post')
        exec /etc/rc.d/rc.bluetooth start
        exec modprobe thunderbolt
        ;;

    *)
        exit 64
        ;;
esac

Make sure the file is executable (chmod +x 50-suspend-fix).

Then you can reboot (or not?) and test if it works. At least it worked for me

Cheers,
ltlnx

jloco · 01-04-2023, 03:13 AM

I put set this up on a macbookpro9,2 (A1298?) after installing the latest updates in -current and rebooted. I have a "Suspend" option and I can't say I've ever used it before, but it does suspend and resume now... so if it was broken... thanks

litelinux · 01-09-2023, 05:00 AM

The suspend problem came back after a kernel upgrade, so I suspect this is not the issue. Sorry for the noise; guess I need further debugging...

rizitis · 01-09-2023, 05:34 AM

what is the output of?

Code:

cat /proc/acpi/wakeup

keefaz · 01-09-2023, 05:59 AM

Not sure if the exec command is executed following a previous exec one

litelinux · 01-09-2023, 01:07 PM

@keefaz Thanks for the tip! Unfortunately even with exec disabled, suspend didn't work as expected and still takes about 10 mins to resume (suspend?). I built a debug kernel (also 6.1.4, with sources in /usr/src) with the following debug options enabled: (they were near the ftrace line so I enabled them all, the pluses are ones asked at compile time)

Code:

 ACPI_DEBUG n -> y
 ACPI_DEBUGGER n -> y
 BLK_DEV_IO_TRACE n -> y
 BRANCH_PROFILE_NONE y -> n
 FTRACE_RECORD_RECURSION n -> y
 FTRACE_SORT_STARTUP_TEST n -> y
 FTRACE_STARTUP_TEST n -> y
 FTRACE_SYSCALLS n -> y
 FUNCTION_PROFILER n -> y
 HIST_TRIGGERS n -> y
 HWLAT_TRACER n -> y
 IRQSOFF_TRACER n -> y
 MMIOTRACE n -> y
 OSNOISE_TRACER n -> y
 PM_DEBUG n -> y
 PREEMPTIRQ_DELAY_TEST n -> m
 PREEMPT_TRACER n -> y
 PROFILE_ANNOTATED_BRANCHES n -> y
 RING_BUFFER_STARTUP_TEST n -> y
 RING_BUFFER_VALIDATE_TIME_DELTAS n -> y
 SCHED_TRACER n -> y
 STACK_TRACER n -> y
 SYNTH_EVENT_GEN_TEST n -> y
 TIMERLAT_TRACER n -> y
 TRACEPOINT_BENCHMARK n -> y
 TRACER_SNAPSHOT n -> y
 TRACE_EVENT_INJECT n -> y
+ACPI_DEBUGGER_USER m
+BRANCH_TRACER y
+DPM_WATCHDOG y
+DPM_WATCHDOG_TIMEOUT 120
+EVENT_TRACE_STARTUP_TEST y
+EVENT_TRACE_TEST_SYSCALLS y
+FTRACE_RECORD_RECURSION_SIZE 128
+FTRACE_SELFTEST y
+HIST_TRIGGERS_DEBUG y
+MMIOTRACE_TEST m
+PM_ADVANCED_DEBUG y
+PM_SLEEP_DEBUG y
+PM_TEST_SUSPEND y
+PM_TRACE y
+PM_TRACE_RTC y
+PREEMPTIRQ_TRACEPOINTS y
+RING_BUFFER_ALLOW_SWAP y
+RING_BUFFER_RECORD_RECURSION y
+RV_MON_WIP n
+TRACER_MAX_TRACE y
+TRACER_SNAPSHOT_PER_CPU_SWAP y
+TRACE_BRANCH_PROFILING y
+TRACE_IRQFLAGS y
+TRACE_IRQFLAGS_NMI y
+TRACE_PREEMPT_TOGGLE y
+TRACING_BRANCHES y
+TRACING_MAP y

Interestingly suspend worked properly with this kernel. I ran both kernels with Intel's pm-graph profiling tool (results attached below). So... maybe it was something I enabled in the kernel config? I really can't be certain at this point

keefaz · 01-09-2023, 01:44 PM

Maybe do a diff between config, from the debug kernel graph it seems there is a huge delay in CPU boot, you say suspend / waking work with this kernel?
Please post output of cat /proc/acpi/wakeup suggested by rizitis
Maybe try adding kernel boot parameter " noapic ", works with some laptops, got info from a quick search

litelinux · 01-09-2023, 02:27 PM

/proc/acpi/wakeup:

Code:

Device  S-state   Status   Sysfs node
P0P2      S3    *enabled   pci:0000:00:01.0
PEG1      S3    *disabled
EC        S4    *disabled  platform:PNP0C09:00
GMUX      S3    *disabled  pnp:00:03
HDEF      S3    *disabled  pci:0000:00:1b.0
RP03      S3    *enabled   pci:0000:00:1c.2
ARPT      S4    *disabled  pci:0000:02:00.0
RP04      S3    *enabled   pci:0000:00:1c.3
RP05      S3    *enabled   pci:0000:00:1c.4
XHC1      S3    *enabled   pci:0000:00:14.0
ADP1      S4    *disabled  platform:ACPI0003:00
LID0      S4    *enabled   platform:PNP0C0D:00

According to the output of pm-graph, there is a large discrepancy in suspend time between the regular kernel (231187 ms) and the debug one (2196 ms); but the real time difference is like 2 minutes (on regular kernel) according to /var/log/messages:

Code:

Jan  9 11:09:07 dankstar NetworkManager[1632]: <info>  [1673233747.4591] dhcp4 (eth0): canceled DHCP transaction, DHCP client pid 2751
Jan  9 11:09:07 dankstar NetworkManager[1632]: <info>  [1673233747.4592] dhcp4 (eth0): activation: beginning transaction (timeout in 45 seconds)
Jan  9 11:09:07 dankstar NetworkManager[1632]: <info>  [1673233747.4592] dhcp4 (eth0): state changed no lease
Jan  9 11:09:07 dankstar NetworkManager[1632]: <info>  [1673233747.4606] device (eth0): set-hw-addr: set MAC address to A2:98:6A:AE:06:54 (scanning)
Jan  9 11:09:07 dankstar NetworkManager[1632]: <info>  [1673233747.4975] device (eth0): supplicant interface state: completed -> disconnected
Jan  9 11:09:07 dankstar NetworkManager[1632]: <info>  [1673233747.4981] device (eth0): state change: disconnected -> unmanaged (reason 'sleeping', sys-iface-state: 'managed')
Jan  9 11:09:07 dankstar NetworkManager[1632]: <info>  [1673233747.4989] device (eth0): set-hw-addr: reset MAC address to 60:F8:1D:B0:FE:E4 (unmanage)
Jan  9 11:09:07 dankstar kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Jan  9 11:09:07 dankstar kernel: ACPI: bus type thunderbolt unregistered
Jan  9 11:09:07 dankstar elogind-daemon[1603]: Suspending system...                                                                                                                          
Jan  9 11:09:07 dankstar kernel: PM: suspend entry (deep)
Jan  9 11:12:05 dankstar syslogd[1398]: syslogd v2.4.4: restart.
Jan  9 11:11:54 dankstar kernel: Linux version 5.19.17 (root@z-mp.slackware.lan) (gcc (GCC) 12.2.0, GNU ld version 2.39-slack151) #1 SMP PREEMPT_DYNAMIC Mon Oct 24 13:00:29 CDT 2022
Jan  9 11:11:54 dankstar kernel: Command line: BOOT_IMAGE=dev000:\x5cEFI\x5cSlackware\x5cvmlinuz-generic-5.19.17  root=/dev/sda2 resume=/dev/sda4 vga=normal ro ro
Jan  9 11:11:54 dankstar kernel: x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
Jan  9 11:11:54 dankstar kernel: x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
Jan  9 11:11:54 dankstar kernel: x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
Jan  9 11:11:54 dankstar kernel: x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
Jan  9 11:11:54 dankstar kernel: x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
Jan  9 11:11:54 dankstar kernel: signal: max sigframe size: 1776
Jan  9 11:11:54 dankstar kernel: BIOS-provided physical RAM map:
Jan  9 11:11:54 dankstar kernel: BIOS-e820: [mem 0x0000000000000000-0x0000000000057fff] usable
Jan  9 11:11:54 dankstar kernel: BIOS-e820: [mem 0x0000000000058000-0x0000000000058fff] reserved
Jan  9 11:11:54 dankstar kernel: BIOS-e820: [mem 0x0000000000059000-0x000000000008efff] usable
Jan  9 11:11:54 dankstar kernel: BIOS-e820: [mem 0x000000000008f000-0x000000000008ffff] reserved
Jan  9 11:11:54 dankstar kernel: BIOS-e820: [mem 0x0000000000090000-0x000000000009ffff] usable
Jan  9 11:11:54 dankstar kernel: BIOS-e820: [mem 0x00000000000a0000-0x00000000000bffff] reserved
Jan  9 11:11:54 dankstar kernel: BIOS-e820: [mem 0x0000000000100000-0x000000007ad0ffff] usable
Jan  9 11:11:54 dankstar kernel: BIOS-e820: [mem 0x000000007ad10000-0x000000007ad52fff] ACPI NVS
Jan  9 11:11:54 dankstar kernel: BIOS-e820: [mem 0x000000007ad53000-0x000000007ad5efff] usable
Jan  9 11:11:54 dankstar kernel: BIOS-e820: [mem 0x000000007ad5f000-0x000000007ad8efff] ACPI data
Jan  9 11:11:54 dankstar kernel: BIOS-e820: [mem 0x000000007ad8f000-0x000000007ae38fff] usable
Jan  9 11:11:54 dankstar kernel: BIOS-e820: [mem 0x000000007ae39000-0x000000007ae8efff] reserved
Jan  9 11:11:54 dankstar kernel: BIOS-e820: [mem 0x000000007ae8f000-0x000000007aed1fff] usable
Jan  9 11:11:54 dankstar kernel: BIOS-e820: [mem 0x000000007aed2000-0x000000007aefefff] unusable
Jan  9 11:11:54 dankstar kernel: BIOS-e820: [mem 0x000000007aeff000-0x000000007af82fff] usable
Jan  9 11:11:54 dankstar kernel: BIOS-e820: [mem 0x000000007af83000-0x000000007afeffff] reserved
Jan  9 11:11:54 dankstar kernel: BIOS-e820: [mem 0x000000007aff0000-0x000000007affffff] usable
Jan  9 11:11:54 dankstar kernel: BIOS-e820: [mem 0x000000007b000000-0x000000007f9fffff] reserved
Jan  9 11:11:54 dankstar kernel: BIOS-e820: [mem 0x00000000e00f8000-0x00000000e00f8fff] reserved

Will try noapic. BTW the code I posted on the last post is the config diff, generated with scripts/diffconfig from the source folder.

rizitis · 01-09-2023, 03:21 PM

If it's working now with new kernel stay as you are. imacs and suspend is big problem in linux you never know what is happening exactly. If you disable LIDO what happened? Give a try... You can disable everything from there, its permanently after reboot everything return as it was... Only XHC1is for usb divices and you will loose keybord mouse etc but power button will work so no problem.

litelinux · 03-06-2023, 12:23 AM

Bumping this thread as resorting to s2idle with

Code:

echo "s2idle" > /sys/power/mem_sleep

works. Everything else, except switching to kernels from other distros, doesn't work.

LuckyCyborg · 03-06-2023, 01:01 AM

Quote:

Originally Posted by litelinux

Therefore the solution is to:

Add to /lib/elogind/system-sleep (or /lib64/elogind/system-sleep depending on your architecture) a file named 50-suspend-fix (could be any name):

Code:

#!/bin/sh
case "${1-}" in
    'pre')
        exec /etc/rc.d/rc.bluetooth stop
        exec modprobe -r thunderbolt
        ;;

    'post')
        exec /etc/rc.d/rc.bluetooth start
        exec modprobe thunderbolt
        ;;

    *)
        exit 64
        ;;
esac

Make sure the file is executable (chmod +x 50-suspend-fix).

This script is wrong.

Code:

exec /etc/rc.d/rc.bluetooth stop
exec modprobe -r thunderbolt

Doing "exec" will make the child program to take over, so the second line "modprobe -r" will never be executed.

I for one I will use something like this:

Code:

/bin/sh /etc/rc.d/rc.bluetooth stop
/sbin/modprobe -r thunderbolt