LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Software (https://www.linuxquestions.org/questions/linux-software-2/)
-   -   Kernel Panic "echo 0 > /proc/sys/kernel" (https://www.linuxquestions.org/questions/linux-software-2/kernel-panic-echo-0-proc-sys-kernel-4175629199/)

tuxthegreat 05-07-2018 11:32 AM

Kernel Panic "echo 0 > /proc/sys/kernel"
 
I've been getting an unusual panic during the restart cycle, also during normal operation, it says the following :
Code:

"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Now I know about the sysctl.conf tweak
Code:

vi /etc/sysctl.conf
and I added two lines at the bottom
Code:

vm.dirty_background_ratio = 5
vm.dirty_ratio = 10

but it still persists. Here is the output when it occurs
Code:

May  7 17:21:59 odroid kernel: [54375.164266] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May  7 17:21:59 odroid kernel: [54375.164271] kworker/3:0    D    0 18892      2 0x80000000
May  7 17:21:59 odroid kernel: [54375.164298] Workqueue: events rtl_work_func_t [r8152]
May  7 17:21:59 odroid kernel: [54375.164301] Call Trace:
May  7 17:21:59 odroid kernel: [54375.164313]  __schedule+0x3d6/0x8b0
May  7 17:21:59 odroid kernel: [54375.164319]  schedule+0x36/0x80
May  7 17:21:59 odroid kernel: [54375.164325]  rpm_resume+0x17e/0x6d0
May  7 17:21:59 odroid kernel: [54375.164332]  ? wait_woken+0x80/0x80
May  7 17:21:59 odroid kernel: [54375.164337]  rpm_resume+0x2d5/0x6d0
May  7 17:21:59 odroid kernel: [54375.164343]  __pm_runtime_resume+0x4e/0x80
May  7 17:21:59 odroid kernel: [54375.164352]  usb_autopm_get_interface+0x22/0x60
May  7 17:21:59 odroid kernel: [54375.164359]  rtl_work_func_t+0x70/0x421 [r8152]
May  7 17:21:59 odroid kernel: [54375.164364]  ? __schedule+0x3de/0x8b0
May  7 17:21:59 odroid kernel: [54375.164372]  process_one_work+0x15b/0x410
May  7 17:21:59 odroid kernel: [54375.164379]  worker_thread+0x4b/0x460
May  7 17:21:59 odroid kernel: [54375.164385]  kthread+0x10c/0x140
May  7 17:21:59 odroid kernel: [54375.164390]  ? process_one_work+0x410/0x410
May  7 17:21:59 odroid kernel: [54375.164395]  ? kthread_create_on_node+0x70/0x70
May  7 17:21:59 odroid kernel: [54375.164400]  ret_from_fork+0x35/0x40
May  7 17:24:00 odroid kernel: [54495.996257] INFO: task kworker/3:0:18892 blocked for more than 120 seconds.
May  7 17:24:00 odroid kernel: [54495.996272]      Not tainted 4.13.0-39-generic #44~16.04.1-Ubuntu
May  7 17:24:00 odroid kernel: [54495.996278] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May  7 17:24:00 odroid kernel: [54495.996285] kworker/3:0    D    0 18892      2 0x80000000
May  7 17:24:00 odroid kernel: [54495.996314] Workqueue: events rtl_work_func_t [r8152]
May  7 17:24:00 odroid kernel: [54495.996319] Call Trace:
May  7 17:24:00 odroid kernel: [54495.996338]  __schedule+0x3d6/0x8b0
May  7 17:24:00 odroid kernel: [54495.996348]  schedule+0x36/0x80
May  7 17:24:00 odroid kernel: [54495.996357]  rpm_resume+0x17e/0x6d0
May  7 17:24:00 odroid kernel: [54495.996368]  ? wait_woken+0x80/0x80
May  7 17:24:00 odroid kernel: [54495.996375]  rpm_resume+0x2d5/0x6d0
May  7 17:24:00 odroid kernel: [54495.996385]  __pm_runtime_resume+0x4e/0x80
May  7 17:24:00 odroid kernel: [54495.996399]  usb_autopm_get_interface+0x22/0x60
May  7 17:24:00 odroid kernel: [54495.996411]  rtl_work_func_t+0x70/0x421 [r8152]
May  7 17:24:00 odroid kernel: [54495.996419]  ? __schedule+0x3de/0x8b0
May  7 17:24:00 odroid kernel: [54495.996431]  process_one_work+0x15b/0x410
May  7 17:24:00 odroid kernel: [54495.996442]  worker_thread+0x4b/0x460
May  7 17:24:00 odroid kernel: [54495.996452]  kthread+0x10c/0x140
May  7 17:24:00 odroid kernel: [54495.996461]  ? process_one_work+0x410/0x410
May  7 17:24:00 odroid kernel: [54495.996469]  ? kthread_create_on_node+0x70/0x70
May  7 17:24:00 odroid kernel: [54495.996476]  ret_from_fork+0x35/0x40
May  7 17:26:01 odroid kernel: [54616.827773] INFO: task kworker/3:0:18892 blocked for more than 120 seconds.
May  7 17:26:01 odroid kernel: [54616.827789]      Not tainted 4.13.0-39-generic #44~16.04.1-Ubuntu
May  7 17:26:01 odroid kernel: [54616.827794] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May  7 17:26:01 odroid kernel: [54616.827802] kworker/3:0    D    0 18892      2 0x80000000
May  7 17:26:01 odroid kernel: [54616.827830] Workqueue: events rtl_work_func_t [r8152]
May  7 17:26:01 odroid kernel: [54616.827836] Call Trace:
May  7 17:26:01 odroid kernel: [54616.827854]  __schedule+0x3d6/0x8b0
May  7 17:26:01 odroid kernel: [54616.827865]  schedule+0x36/0x80
May  7 17:26:01 odroid kernel: [54616.827874]  rpm_resume+0x17e/0x6d0
May  7 17:26:01 odroid kernel: [54616.827884]  ? wait_woken+0x80/0x80
May  7 17:26:01 odroid kernel: [54616.827892]  rpm_resume+0x2d5/0x6d0
May  7 17:26:01 odroid kernel: [54616.827901]  __pm_runtime_resume+0x4e/0x80
May  7 17:26:01 odroid kernel: [54616.827915]  usb_autopm_get_interface+0x22/0x60
May  7 17:26:01 odroid kernel: [54616.827927]  rtl_work_func_t+0x70/0x421 [r8152]
May  7 17:26:01 odroid kernel: [54616.827936]  ? __schedule+0x3de/0x8b0
May  7 17:26:01 odroid kernel: [54616.827948]  process_one_work+0x15b/0x410
May  7 17:26:01 odroid kernel: [54616.827958]  worker_thread+0x4b/0x460
May  7 17:26:01 odroid kernel: [54616.827968]  kthread+0x10c/0x140
May  7 17:26:01 odroid kernel: [54616.827977]  ? process_one_work+0x410/0x410
May  7 17:26:01 odroid kernel: [54616.827985]  ? kthread_create_on_node+0x70/0x70
May  7 17:26:01 odroid kernel: [54616.827992]  ret_from_fork+0x35/0x40

I know this doesn't give much info if any at all, but I think I narrowed it down to either my exfat mounts, which I have a lot of, or my USB Hubs, however I nipped that issue by blacklisting the UAS and using the generic USB that comes with Ubuntu by default. I got the instructions for this tweak here https://unix.stackexchange.com/quest...river-problem? It may be an I/O overload issue, however USB3.1 type C should be able to handle a couple of external hard drives no?

I also wanted to add on the way this kernel panic happens, it just doesn't suddenly lock-up and only remedy is cold reboot via power button, first the network goes down, I notice this because my ssh session times out, then I am unable to open/write/read to any hard drive / ssd. Then everything locks up except the mouse and I am able to hover over the power icon and restart / shutdown, the restart doesn't work as mentioned, the shutdown sometimes works, sometimes doesn't. And I am unable to dmesg or syslog the event as those two seem to fail as well.

RandomTroll 05-07-2018 01:35 PM

The log entries report on a problem with a Realtek r8152 network card.

tuxthegreat 05-07-2018 01:57 PM

Quote:

Originally Posted by RandomTroll (Post 5851743)
The log entries report on a problem with a Realtek r8152 network card.

Any idea as to what could be done ? Ah I see it now
Quote:

Workqueue: events rtl_work_func_t [r8152]
Is that network card also responsible for wifi or does wifi have it's own card.

RandomTroll 05-07-2018 05:14 PM

I just looked at Realtek's site. The only 8152 they mention is an Ethernet (=wired) card. You have a separate card for wireless if you have wireless. You can turn off or not-load the 8152 if you don't need it. If it isn't working you should disable or not-load it in any case.
Code:

lsusb
will tell you what USB devices you have;
Code:

lspci
will tell you what built-in devices you have.

AwesomeMachine 05-07-2018 05:59 PM

I think the the realtek 8152 is a USB-ethernet adapter chip.

tuxthegreat 05-07-2018 06:06 PM

Quote:

Originally Posted by RandomTroll (Post 5851819)
I just looked at Realtek's site. The only 8152 they mention is an Ethernet (=wired) card. You have a separate card for wireless if you have wireless. You can turn off or not-load the 8152 if you don't need it. If it isn't working you should disable or not-load it in any case.
Code:

lsusb
will tell you what USB devices you have;
Code:

lspci
will tell you what built-in devices you have.

If I disable the semi-working 8152 ethernet I will be left with no internet as my wifi isn't working, I turned it off some time ago and I can't figure out how to get it back. I have a separate thread in networking about my lack of wifi here https://www.linuxquestions.org/quest...er-4175629211/ So you see I can't disable my only means of connectivity.

tuxthegreat 05-08-2018 07:58 AM

An update, I got my wifi going, and I left it on wifi when I went to sleep, I woke up to find my system non-responsive again, again with the Workqueue: events rtl_work_func_t [r8152] kernel panic,
Code:

May  8 08:33:37 odroid kernel: [22354.813321] kworker/2:3    D    0 11174      2 0x80000000
May  8 08:33:37 odroid kernel: [22354.813348] Workqueue: events rtl_work_func_t [r8152]
May  8 08:33:37 odroid kernel: [22354.813353] Call Trace:
May  8 08:33:37 odroid kernel: [22354.813372]  __schedule+0x3d6/0x8b0
May  8 08:33:37 odroid kernel: [22354.813382]  schedule+0x36/0x80
May  8 08:33:37 odroid kernel: [22354.813391]  rpm_resume+0x17e/0x6d0
May  8 08:33:37 odroid kernel: [22354.813402]  ? load_balance+0x1b5/0x9c0
May  8 08:33:37 odroid kernel: [22354.813410]  ? wait_woken+0x80/0x80
May  8 08:33:37 odroid kernel: [22354.813417]  rpm_resume+0x2d5/0x6d0
May  8 08:33:37 odroid kernel: [22354.813427]  __pm_runtime_resume+0x4e/0x80
May  8 08:33:37 odroid kernel: [22354.813440]  usb_autopm_get_interface+0x22/0x60
May  8 08:33:37 odroid kernel: [22354.813454]  rtl_work_func_t+0x70/0x421 [r8152]
May  8 08:33:37 odroid kernel: [22354.813465]  ? __schedule+0x3de/0x8b0
May  8 08:33:37 odroid kernel: [22354.813479]  process_one_work+0x15b/0x410
May  8 08:33:37 odroid kernel: [22354.813493]  worker_thread+0x4b/0x460
May  8 08:33:37 odroid kernel: [22354.813522]  kthread+0x10c/0x140
May  8 08:33:37 odroid kernel: [22354.813534]  ? process_one_work+0x410/0x410
May  8 08:33:37 odroid kernel: [22354.813542]  ? kthread_create_on_node+0x70/0x70
May  8 08:33:37 odroid kernel: [22354.813551]  ret_from_fork+0x35/0x40
May  8 08:35:37 odroid kernel: [22475.644661] INFO: task kworker/2:3:11174 blocked for more than 120 seconds.
May  8 08:35:37 odroid kernel: [22475.644675]      Not tainted 4.13.0-39-generic #44~16.04.1-Ubuntu
May  8 08:35:37 odroid kernel: [22475.644679] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message

How do I go about blacklisting the r8152 ethernet module and using wifi only? Also after the reboot my wifi modules aren't there anymore, what can I do to activate them permanently? The command to load the wifi module is modprobe iwlwifi
Also I forgot to add one important detail, I upgraded my connection to 250/100 it used to be 60/10 , could this be the cause for my ethernet kernel panic? This is exactly when this problem started.

RandomTroll 05-08-2018 12:51 PM

Quote:

Originally Posted by tuxthegreat (Post 5852039)
How do I go about blacklisting the r8152 ethernet module and using wifi only?

Code:

man modprobe
explains blacklisting; /etc/modprobe.d/blacklist.conf


Quote:

Originally Posted by tuxthegreat (Post 5852039)
after the reboot my wifi modules aren't there anymore, what can I do to activate them permanently?

/etc/rc.d/rc.modules

Quote:

Originally Posted by tuxthegreat (Post 5852039)
I upgraded my connection to 250/100 it used to be 60/10 , could this be the cause for my ethernet kernel panic?

I would hope not: hardware should choose to run at a safe speed.

Quote:

Originally Posted by tuxthegreat (Post 5852039)
This is exactly when this problem started.

Possibly the higher speed uncovered the chip's shortcomings.

tuxthegreat 05-08-2018 07:38 PM

As it turns out the iwlwifi was listed in the /etc/modprobe.d/blacklist.conf. When I was getting rid of wifi capabilities I must of put it in there and forgot about it, I now have a .txt file with any modifications / updates and so on I do in case I have to go back on something I did a few months ago and revert my action. So I # the iwlwifi mod in the blacklist.conf and I rebooted and wifi was available, I guess I can use the same process to blacklist the r8152 driver, but something caught my eye in my current lsmod
Code:

lsmod
Module                  Size  Used by


usbnet                45056  1 cdc_ether
cdc_ether              16384  0
r8152                  57344  0
mii                    16384  2 usbnet,r8152
asus_wireless          16384  0
mac80211              782336  1 iwlmvm
iwlwifi              249856  1 iwlmvm
cfg80211              614400  3 iwlmvm,iwlwifi,mac80211

If I am reading this lsmod correctly, the mii module is using the usbnet and the r8152, and usbnet is using cdc_ether. Which begs the question, which of these modules do I have to blacklist to go wifi only, even though that will cut my speed drastically but id rather have lower speed than a daily headache from a kernel panic. Now I think I know why the usbnet is connected with the r8152, you see my laptop didn't come with a standard RJ45 Ethernet port, instead it came with a USB3.1 Type-c port to which I can connect a USB adapter with a RJ45 port.
By the way /etc/rc.d/rc.modules doesn't exist on this installation
Code:

cd /etc/rc.d
bash: cd: /etc/rc.d: No such file or directory

And
Code:

nano /etc/rc.d/rc.modules
gives me a blank file.

RandomTroll 05-08-2018 09:02 PM

Quote:

Originally Posted by tuxthegreat (Post 5852283)
I now have a .txt file with any modifications / updates and so on I do in case I have to go back on something I did a few months ago and revert my action.

I keep a log of my actions; I also archive all my LinuxQuestions messages for reference.

Quote:

Originally Posted by tuxthegreat (Post 5852283)
which of these modules do I have to blacklist to go wifi only

Only r8152: that's the driver necessary to use the chip. Modules separate functions. mii is useful to other chips: you don't want to blacklist it.

Quote:

Originally Posted by tuxthegreat (Post 5852283)
my laptop didn't come with a standard RJ45 Ethernet port, instead it came with a USB3.1 Type-c port to which I can connect a USB adapter with a RJ45 port.

Then you can just remove it.

Quote:

Originally Posted by tuxthegreat (Post 5852283)
By the way /etc/rc.d/rc.modules doesn't exist on this installation

You have a different distribution. Ask on your distribution's forum. It'll probably load automatically all the time now that you've removed it from the blacklist so you may not need to bother.

There may be a way to throttle the r8152's bandwidth.

tuxthegreat 05-09-2018 01:56 AM

I am sort of stuck with the adapter, it not only has 1 RJ45 ethernet port but it also contains 3 USB 3.0 ports which I use, this model http://2click.fen.pl/files/image/Newsletter_Y-30951.jpg I don't think if I blacklist the r8152 module the USB part of the hub will stop to function. That's a good idea that you posted *throttling the r8152* what if I throttled the program that uses that much bandwidth, could that be a solution ? Or do we need to throttle the r8152 at the source ?
Quote:

You have a different distribution. Ask on your distribution's forum. It'll probably load automatically all the time now that you've removed it from the blacklist so you may not need to bother.
You are correct, it does load automatically now that I removed it.

RandomTroll 05-09-2018 09:39 AM

Quote:

Originally Posted by tuxthegreat (Post 5852347)
I don't think if I blacklist the r8152 module the USB part of the hub will stop to function.

They're separate functions.

Quote:

Originally Posted by tuxthegreat (Post 5852347)
what if I throttled the program that uses that much bandwidth, could that be a solution ?

If that's possible - I'd be surprised...

Quote:

Originally Posted by tuxthegreat (Post 5852347)
Or do we need to throttle the r8152 at the source ?

If you can't throttle the program.

in /usr/src/linux-4.16.8/drivers/net/usb/r8152.c I see
Code:

#define RTL8152_MAX_TX 4
#define RTL8152_MAX_RX 10

You could lower those and build your own module. Better yet you could make them parameters so you could figure out the maximum speed by just unloading it then reloading it with different numbers.

tuxthegreat 05-09-2018 05:17 PM

Quote:

Originally Posted by RandomTroll (Post 5852476)
They're separate functions.


If that's possible - I'd be surprised...


If you can't throttle the program.

in /usr/src/linux-4.16.8/drivers/net/usb/r8152.c I see
Code:

#define RTL8152_MAX_TX 4
#define RTL8152_MAX_RX 10

You could lower those and build your own module. Better yet you could make them parameters so you could figure out the maximum speed by just unloading it then reloading it with different numbers.

I do not have this dir/conf file /usr/src/linux-4.16.8/drivers/net/usb/r8152.c Instead in my /usr/src I see the following :
Code:

ls -a
.  linux-headers-4.13.0-38          linux-headers-4.13.0-39
..  linux-headers-4.13.0-38-generic  linux-headers-4.13.0-39-generic

I am currently on the -39 revision, in my /usr/src/linux-headers-4.13.0-39-generic/drivers/net/usb I see 2 files :
Code:

/usr/src/linux-headers-4.13.0-39-generic/drivers/net/usb$ ls
Kconfig  Makefile

When I cat Kconfig it indeed contains a quite a bit of info regarding different modules available, here is a snippet for my r8152 driver
Code:

config USB_RTL8152
        tristate "Realtek RTL8152/RTL8153 Based USB Ethernet Adapters"
        select MII
        help
          This option adds support for Realtek RTL8152 based USB 2.0
          10/100 Ethernet adapters and RTL8153 based USB 3.0 10/100/1000
          Ethernet adapters.

          To compile this driver as a module, choose M here: the
          module will be called r8152.

I don't see a r8152.c with the 2 config items you suggested,
Code:

#define RTL8152_MAX_TX 4
#define RTL8152_MAX_RX 10

I was thinking if I am going to go any further with compiling modules, kernel configs maybe I should boot into the -38 revision as a test install, and once I get something working, move it over to the -39.

And update, with wifi enabled and ethernet enabled I still got a kernel panic, with both r8152 and all the wifi modules listed in the call trace, it always happens every 18 hours it seems. I am going to blacklist the r8152 and see if that helps.

RandomTroll 05-09-2018 10:48 PM

Quote:

Originally Posted by tuxthegreat (Post 5852618)
I do not have this dir/conf file /usr/src/linux-4.16.8/drivers/net/usb/r8152.c

You have to use your kernel version.

Quote:

Originally Posted by tuxthegreat (Post 5852618)
I don't see a r8152.c with the 2 config items you suggested

If you haven't installed the kernel source you won't. That's a package too.

Quote:

Originally Posted by tuxthegreat (Post 5852618)
with wifi enabled and ethernet enabled I still got a kernel panic, with both r8152 and all the wifi modules listed in the call trace

The wifi could be bycatch or the problem is elsewhere in the network stack and catches out both of them.

I used Skype when it first came out. It runs at 48K, not alterable. My hardware wouldn't support audio at that speed, garbled every conversation. I re-wrote the driver to make the maximum sampling rate a parameter. I could unload it then reload it with, say, 5K as the maximum sampling rate (as written it returned 0 - no maximum), which worked. POTS is 2.3K, so this handled phone conversations, all I wanted.

tuxthegreat 08-02-2018 03:31 AM

I may have finally found a solution to my problem. I am going to post it here in case anyone else comes across this issue. I noticed that the panics always happened when I left my laptop on for extended periods of time without any user interaction so I started came to the conclusion that perhaps the usb was getting deadlocked in runtime suspend and causing a panic. So I did the following: First I prepared some info that I will need later on,
Code:

cd /sys/bus/pci/devices/ and then you ls (list)
This shows all your pci devices. You also need to know which pci device is the correct one so you can do
Code:

lspci
I disabled usb auto-pm
Code:

# echo on | tee /sys/bus/usb/devices/*/power/control
Then I
Code:

# echo on > /sys/bus/pci/devices/<bus_name>/power/control
Now from the lspci list you will know which the <bus_name> you need for this to work. To verify the status of the powermanagement you can use powertop to see if it's listed as Bad. I added those two commands to my crontab and we shall see if it works.


All times are GMT -5. The time now is 05:24 AM.