Linux Mint 17.3 System Hanging

bradonomics · 10-07-2016, 10:53 PM

I switched to Linux ~2 years ago. Mostly it's been great. Recently my system has started hanging, maybe once-or-twice per week. The CPU and RAM usage isn't maxed (maybe 50%), but the system becomes unresponsive to the point of forcing a hard reboot.

I'm not sure how to troubleshoot this. Are there logs or something I can dig through?

System Info:
Operating System: Linux Mint 17.3 (64-bit)
Cinnamon Desktop: v2.8.8
Linux Kernel: 4.4.0-21-generic
Processor: Intel Core i7-4700MQ CPU @ 2.40GHz x 4
Memory: 7.5 GiB
Hard Drive: 953.3 GB
Graphics Card: NVIDIA GK107M [GeForce GT 750M]

mostlyharmless · 10-08-2016, 04:37 PM

There's journalctl. See https://www.digitalocean.com/communi...e-systemd-logs

You can also look at top and ps. Sometimes, alas, system updates (you do regular updates?) make a previously smooth running system run dorkily....just like with Windows and the Apple OS

frankbell · 10-08-2016, 08:40 PM

Is there any pattern as to the programs you are running when this happens?

bradonomics · 10-08-2016, 09:23 PM

I can't say for sure which programs are open when it's happening. I spend my days as a web developer so I'd guess 90+% of the time I've got the following open when the system hangs:

Chrome
Atom
File Browser (nemo)
Terminal

It looks like the journald displays logs concerning systemd. The Linux Mint 17 series doesn't use systemd by default. At least I don't think it does. Here's my dpkg output:

Code:

$ dpkg -l systemd
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-=================================
un  systemd        <none>       <none>       (no description available)

As for updates, yes, I update regularly.

frankbell · 10-08-2016, 09:50 PM

Quote:

I can't say for sure which programs are open when it's happening. I spend my days as a web developer so I'd guess 90+% of the time I've got the following open when the system hangs:

It seems clear that you aren't choking it with applications. I understand that Chrome runs each tab sandboxed in some way, but that's a misty water-colored memory of something I read once.

Mint 17 stores its logs in /var/log, the old-fashioned way. I would start with mintsystem.log, syslog, and Xorg.0.log, but, frankly, if the system is freezing, it might not have the opportunity to log what's causing the freeze. It's the best place to start looking, though.

Update:

I had a late night thought: Perhaps you could run top or a similar command in a window. If the computer is freezing, it probably shouldn't be minimized, because you would lose access to it--perhaps it could be spread out in a wide window of very low height across the bottom of the screen. When the computer freezes, that might give you an idea of what process was using the most resources at the time of the freeze.

I know it sounds kind of loony, but I figured it was worth sharing.

onebuck · 10-09-2016, 09:56 AM

Moved: This thread is more suitable in <Linux Mint> and has been moved accordingly to help your thread/question get the exposure it deserves.

Jjanel · 10-09-2016, 11:01 PM

Can you still ping it? (maybe run the `top` loggedin from another system)
Look into sysrq (echo "1" [string, not binary!] to enable it-all)

mazinoz · 10-11-2016, 06:35 PM

For security reasons alone, I'd like to suggest you do a backup of data and /etc, list of programs and then zero the master boot record and install a newer version of LM like 18. Because

http://blog.linuxmint.com/?p=2994

TxLonghorn · 10-12-2016, 07:19 AM

Quote:

Originally Posted by mazinoz

For security reasons alone, I'd like to suggest you do a backup of data and /etc, list of programs and then zero the master boot record and install a newer version of LM like 18. Because
http://blog.linuxmint.com/?p=2994

The February event when the website was hacked is ancient history, and not relevant to the current Mint 17.
In answer to your question about systemd. Mint 17.x does use a few systemd elements.

mazinoz · 10-12-2016, 07:01 PM

"The February event when the website was hacked is ancient history, and not relevant to the current Mint 17."

They say they are using 17.3, it is possible they have been using the hacked version.

Using a live CD is a good suggestion.

A fresh install with latest programs and updates may just fix a bug that is hard to identify, and you know you are using a clean system.

bradonomics · 10-28-2016, 11:34 PM

The hung state returned today and per the suggestion above, here is my "top" output:

Code:

top - 11:31:13 up  3:00,  2 users,  load average: 1.27, 1.88, 2.02
Tasks: 257 total,   2 running, 253 sleeping,   0 stopped,   2 zombie
%Cpu(s):  0.7 us, 12.9 sy,  0.0 ni, 86.0 id,  0.3 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:   7895144 total,  3621720 used,  4273424 free,   431516 buffers
KiB Swap:  8104956 total,        0 used,  8104956 free.  1660068 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                           
 6397 root      20   0       0      0      0 R  99.8  0.0 115:44.96 kworker/0:3                                                                       
 2600 brad      20   0 1830972 411652  92848 S   1.7  5.2   4:05.85 cinnamon                                                                          
 1261 root      20   0  343108  10624   8876 S   1.3  0.1   1:07.41 NetworkManager                                                                    
 1635 root      20   0  332628 120068  50264 S   1.0  1.5   3:02.92 Xorg                                                                              
  948 syslog    20   0  255844   5252   2548 S   0.7  0.1   1:21.96 rsyslogd                                                                          
    7 root      20   0       0      0      0 S   0.3  0.0   0:05.57 rcu_sched                                                                         
10557 root      20   0       0      0      0 S   0.3  0.0   0:02.84 kworker/u16:0                                                                     
13062 brad      20   0 1000864 217408 112752 S   0.3  2.8   0:08.29 chrome                                                                            
13508 root      20   0       0      0      0 S   0.3  0.0   0:00.04 kworker/1:1                                                                       
13513 brad      20   0  603840  27680  21376 S   0.3  0.4   0:00.13 gnome-terminal                                                                    
    1 root      20   0   33916   4544   2792 S   0.0  0.1   0:01.09 init                                                                              
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kthreadd                                                                          
    3 root      20   0       0      0      0 S   0.0  0.0   0:05.49 ksoftirqd/0                                                                       
    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H                                                                      
    8 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh                                                                            
    9 root      rt   0       0      0      0 S   0.0  0.0   0:00.00 migration/0                                                                       
   10 root      rt   0       0      0      0 S   0.0  0.0   0:04.28 watchdog/0                                                                        
   11 root      rt   0       0      0      0 S   0.0  0.0   0:00.01 watchdog/1                                                                        
   12 root      rt   0       0      0      0 S   0.0  0.0   0:00.01 migration/1                                                                       
   13 root      20   0       0      0      0 S   0.0  0.0   0:00.01 ksoftirqd/1                                                                       
   15 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/1:0H                                                                      
   16 root      rt   0       0      0      0 S   0.0  0.0   0:00.01 watchdog/2                                                                        
   17 root      rt   0       0      0      0 S   0.0  0.0   0:00.01 migration/2                                                                       
   18 root      20   0       0      0      0 S   0.0  0.0   0:00.08 ksoftirqd/2                                                                       
   20 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/2:0H                                                                      
   21 root      rt   0       0      0      0 S   0.0  0.0   0:00.01 watchdog/3                                                                        
   22 root      rt   0       0      0      0 S   0.0  0.0   0:00.00 migration/3

The "kworker/0:3" process seems to be the culprit. I've done a bit of searching but didn't find anything definitive on what to do about this process.

Jjanel · 10-29-2016, 11:24 PM

Cool! Is this `top` display on a *remote* computer (logging-in to freezing Mint)?
Did this freeze? Can you still `ping` the Mint system from 'this' system? (What is 'this system'?)
Did you play with sysrq (on the freezing/frozen Mint system)?

Can you [sort of] watch for `top` showing like this kworker using a lot of TIME?
(to try to catch it, before the system freezes) Try: sudo lsof -p <its PID>

Here's my web-search for this (it returns tons of stuff that I know NOthing about!):
kworker how to determine|"find out"|tell what it is doing|running?

TxLonghorn · 10-30-2016, 05:36 AM

kworker (the k is for kernel, not KDE) is the likely culprit.
Here is a helpful discussion of kworker →
http://askubuntu.com/questions/33640...much-cpu#52299
The first thing I would try is to switch to the latest (supported) kernel available.
Open the Update Manager > View menu > Linux kernels.
Go to the bottom of the list and install the 4.4.0-45 kernel. If that doesn't work you can try the 4.5 kernel, which works for me in my 17.3 - but it is not LTS. (I am not sure if the 4.5 has been fixed for the Mad Cow or Crazy Cow or whatever that Cow bug was recently - but the 4.4.0-45 kernel does include the bug fix.)

EDIT: It was named "Dirty Cow"...

Quote:

Oct 21, 2016
"Ubuntu users urged to patch their systems immediately"

Canonical urged all users to patch their systems immediately by installing linux-image-4.8.0-26 (4.8.0-26.28) for Ubuntu 16.10, linux-image-4.4.0-45 (4.4.0-45.66) for Ubuntu 16.04 LTS, linux-image-3.13.0-100 (3.13.0-100.147) for Ubuntu 14.04 LTS, and linux-image-3.2.0-113 (3.2.0-113.155) for Ubuntu 12.04 LTS, as well as linux-image-4.4.0-1029-raspi2 (4.4.0-1029.36) for Ubuntu 16.04 LTS for Raspberry Pi 2.

http://news.softpedia.com/news/canon...#ixzz4OZJ8b4do