[SOLVED] Strange recurring disk space problem and failure to boot to console prompt

Miranden · 09-23-2013, 04:02 PM

I am having a weird problem that just started a few days ago on Slackware64-current. I always boot to run level 3 and start X with "startx," but over the past few days I have had errors starting X because of (I thought) running out of space on my root partition. This is what my partitions look like:

Code:

bash-4.2# fdisk -l

Disk /dev/sda: 124.0 GB, 124000000000 bytes
255 heads, 63 sectors/track, 15075 cylinders, total 242187500 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x5dd495e2

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1            2048      526335      262144   83  Linux
/dev/sda2          526336   242187499   120830582    5  Extended
/dev/sda5          528384    67581951    33526784   83  Linux
/dev/sda6        67584000   233314303    82865152   83  Linux
/dev/sda7       233316352   242187499     4435574   82  Linux swap

So last week I had this error starting X which said "failure to activate keymap" and "failure to activate core devices." I read this thread with the indentical error and thought I had run out of space. So, I resized my 21 GB root partition to about 26 GB, and everything worked fine.

With the next updates to -current, however, it happened again. I thought, wow, Grub 2 must take up a lot of space (

) and so yesterday I gave my root partition another 6 GB, bringing it up to 32 GB.

Then today I rebooted for some reason, and I got an error at the console right after the "darkstar" prompt showed up. It said something about no space being left on disk (I don't remember exactly what it was, except it said it five or six times all in a row), but the prompt was gone, so I could not enter anything or do anything except reboot. Same thing happened again.

So I booted with a GParted live cd and looked at what it told me. It showed all my partitions correctly sized, but it said I had 16 GB free on /. None of my partitions were full at all.

Then, thinking something was very screwy, I tried booting up Slackware again. It booted fine and gave me my prompt. So I tried "df," and this is what it gives me:

Code:

bash-4.2# df
Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/sda5       32869232 32852620         0 100% /
/dev/sda1         245679    18047    210429   8% /boot
/dev/sda6       81432296  7675208  69597456  10% /home
tmpfs            1960688        0   1960688   0% /dev/shm

This is directly contrary to what the GParted live cd told me, and it also makes no sense because I did not install 6 GB of software since yesterday. I didn't even update again. What the heck is going on here?

Oh, and also, I can't log into X as anything other than root now either, which is what happened when it ran out of space before.

Any help appreciated!

Didier Spaier · 09-23-2013, 04:44 PM

It would certainly help to know which subdirectories of / are filling it up. Please post back output of following command:

Code:

du --exclude=/home -achd 1 /

Miranden · 09-23-2013, 04:47 PM

Quote:

Originally Posted by Didier Spaier

It would certainly help to know which subdirectories of / are filling it up. Please post back output of following command:

Code:

du --exclude=/home -achd 1 /

Thank you. That is:

Code:

bash-4.2# du --exclude=/home -achd 1 /
121M	/root
0	/sys
16K	/lost+found
36K	/tmp
828K	/run
3.9G	/opt
17M	/etc
0	/dev
du: cannot access '/proc/1236/task/1236/fd/4': No such file or directory
du: cannot access '/proc/1236/task/1236/fdinfo/4': No such file or directory
du: cannot access '/proc/1236/fd/4': No such file or directory
du: cannot access '/proc/1236/fdinfo/4': No such file or directory
0	/proc
23M	/lib64
200M	/lib
16M	/boot
18M	/sbin
4.0K	/srv
40K	/mnt
68K	/media
20G	/var
7.7G	/usr
13M	/bin
32G	/
32G	total
bash-4.2#

Didier Spaier · 09-23-2013, 05:08 PM

20G in /var that's way too much. Here I have less than 1G.

So please look into /var, maybe some logfile is becoming *very* heavy. Do this (as root):

Code:

du  -achd 1 /var

Then refine looking into the heaviest directory. If it's /var/log for instance (only a guess :-) you would write:

Code:

du  -achd 1 /var/log

.

Miranden · 09-23-2013, 05:21 PM

It was a good guess.

Here it is:

Code:

bash-4.2# du -achd 1 /var
646M	/var/cache
0	/var/mail
16K	/var/state
108M	/var/tmp
8.9M	/var/www
200K	/var/run
12K	/var/lock
508K	/var/spool
24K	/var/db
19G	/var/log
28K	/var/yp
161M	/var/lib
4.0K	/var/empty
0	/var/adm
0	/var/rwho
144K	/var/man
8.0K	/var/games
24K	/var/named
20G	/var
20G	total
bash-4.2#

then

Code:

4.0K	/var/log/sa
164K	/var/log/secure.1
11M	/var/log/sbopkg
4.0K	/var/log/httpd
0	/var/log/btmp
8.0K	/var/log/pm-suspend.log
0	/var/log/spooler
4.0K	/var/log/samba
204K	/var/log/wpa_supplicant.log.1
8.6M	/var/log/removed_scripts
0	/var/log/maillog.1
48K	/var/log/wpa_supplicant.log
4.0K	/var/log/iptraf
1.4M	/var/log/wtmp.1
4.0K	/var/log/cron
8.0K	/var/log/faillog
2.0M	/var/log/debug.1
624K	/var/log/messages
12K	/var/log/hp
4.0K	/var/log/uucp
4.0K	/var/log/nfsd
23M	/var/log/packages
24K	/var/log/cron.1
64K	/var/log/dmesg
28K	/var/log/Xorg.0.log
514M	/var/log/syslog.1
32K	/var/log/Xorg.0.log.old
19G	/var/log
19G	total
bash-4.2#

But am I reading it wrong? It says there are 19 G total, but I do not see what file is making it that amount. What it has listed here under /var/log does not add up to 19 G.

Edit: Oops. I had to pipe it through "less," because the top was getting cut off for some reason. Here is that output:

Code:

du -achd 1 /var/log | less
12K     /var/log/secure
18G     /var/log/syslog
8.0K    /var/log/lastlog
4.0K    /var/log/cups
240K    /var/log/debug
8.6M    /var/log/messages.1
8.0K    /var/log/btmp.1
0       /var/log/maillog
0       /var/log/spooler.1
5.4M    /var/log/scripts
40M     /var/log/removed_packages
4.0K    /var/log/pm-powersave.log
88K     /var/log/setup
104K    /var/log/ConsoleKit
176K    /var/log/wtmp
4.0K    /var/log/sa
164K    /var/log/secure.1
11M     /var/log/sbopkg
4.0K    /var/log/httpd
0       /var/log/btmp
8.0K    /var/log/pm-suspend.log
0       /var/log/spooler
4.0K    /var/log/samba
204K    /var/log/wpa_supplicant.log.1
8.6M    /var/log/removed_scripts
0       /var/log/maillog.1
48K     /var/log/wpa_supplicant.log
4.0K    /var/log/iptraf
1.4M    /var/log/wtmp.1
4.0K    /var/log/cron
8.0K    /var/log/faillog
2.0M    /var/log/debug.1
624K    /var/log/messages
12K     /var/log/hp
4.0K    /var/log/uucp
4.0K    /var/log/nfsd
23M     /var/log/packages
24K     /var/log/cron.1
64K     /var/log/dmesg
28K     /var/log/Xorg.0.log
514M    /var/log/syslog.1
32K     /var/log/Xorg.0.log.old
19G     /var/log
19G     total

Didier Spaier · 09-23-2013, 05:35 PM

Looks weird, yes.

try:

Code:

find /var/log -size +5M /var/log/ -ls

You can refine the size, see "man find".

EDIT After having seen the full output of "du -achd 1 /var/log | less", nothing is weird (but the size of /var/log/syslog, as dr.s pointed out :-)

dr.s · 09-23-2013, 05:42 PM

Quote:

Originally Posted by Miranden

...

Code:

du -achd 1 /var/log | less
...
18G     /var/log/syslog
...
19G     /var/log
19G     total

Have a look at syslog. I had a recent issue with kernel 3.10.5 (solved in later kernels) that caused syslog to be flooded with mei/mei-me module errors.

Kallaste · 09-23-2013, 06:02 PM

Sorry, error

Miranden · 09-23-2013, 06:13 PM

Well I cannot boot again now because I ran out of space again. I deleted everything in /tmp thinking that would be enough, but I guess whatever is writing in syslog is writing fast. I will have to boot with a live cd to fix it. When I do, will it be okay to copy my 18 G /var/log/syslog file to a flash drive and then delete it? It will get recreated, won't it?

When I look at the file to see why it is getting so large, what should I be looking for? Just some error that keeps repeating over and over?

Thanks again!

TobiSGD · 09-23-2013, 06:32 PM

If you have a log file that fills up so fast (18GB is a whole lot of text) you shouldn't have to run the system to see the culprit, just have a look at the existing syslog. Keep in mid that processing a 18GB textfile can take some time, so maybe you start with looking at the last 100 lines using the tail command.

dr.s · 09-23-2013, 07:55 PM

Quote:

Originally Posted by Miranden

Well I cannot boot again now because I ran out of space again. I deleted everything in /tmp thinking that would be enough, but I guess whatever is writing in syslog is writing fast. I will have to boot with a live cd to fix it. When I do, will it be okay to copy my 18 G /var/log/syslog file to a flash drive and then delete it? It will get recreated, won't it?

When I look at the file to see why it is getting so large, what should I be looking for? Just some error that keeps repeating over and over?

Thanks again!

Look for repeating errors or patterns, something has to stand out in a file that grows so quickly to 18GB, once you determine the issue or save a copy elsewhere, you can delete it.

NoStressHQ · 09-23-2013, 09:07 PM

Quote:

Originally Posted by Miranden

[...] I will have to boot with a live cd to fix it. [...]

Just boot with Slackware Install CD/DVD... Then mount your disk "somewhere" (/mnt ? ) and then you'd be able to access the file as /mnt/var/log/syslog...

Code:

 
# less /mnt/var/log/syslog

or

Code:

 
# tail /mnt/var/log/syslog

You should be able to see the culprit then... Or trace back to it.

Good luck

Garry.

Miranden · 09-24-2013, 08:22 AM

It seems I have 4 mei_me errors repeating over and over:

Code:

Sep 23 18:31:18 darkstar kernel: [ 3711.809406] mei_me 0000:00:16.0: unexpected reset: dev_state = RESETTING
Sep 23 18:31:18 darkstar kernel: [ 3711.809424] mei_me 0000:00:16.0: reset: unexpected enumeration response hbm.
Sep 23 18:31:18 darkstar kernel: [ 3711.809427] mei_me 0000:00:16.0: unexpected reset: dev_state = RESETTING
Sep 23 18:31:18 darkstar kernel: [ 3711.809444] mei_me 0000:00:16.0: reset: wrong host start response
Sep 23 18:31:18 darkstar kernel: [ 3711.809448] mei_me 0000:00:16.0: unexpected reset: dev_state = RESETTING
Sep 23 18:31:18 darkstar kernel: [ 3711.809462] mei_me 0000:00:16.0: reset: unexpected enumeration response hbm.
Sep 23 18:31:18 darkstar kernel: [ 3711.809465] mei_me 0000:00:16.0: unexpected reset: dev_state = RESETTING
Sep 23 18:31:18 darkstar kernel: [ 3711.809478] mei_me 0000:00:16.0: reset: wrong host start response

It is the same bug reported here: https://bugs.archlinux.org/task/36288. Apparently it is occuring for these people when they come out of suspend. But I tried monitoring syslog with "tail -f /var/log/syslog" both before and after resuming from suspend, and for some reason now it is not happening at all for me. I'm betting it will start again, though. I have hundreds (or thousands) of lines from yesterday, and I haven't changed anything since then.

So what should I do about this? Is it safe to blacklist the mei_me module, or should I change to a different kernel? I am wondering why some of the people on the page I linked say the bug was fixed for them after 3.10.5, and for others it continued. Is there maybe something wrong with the way I am suspending my system? All I do for that is put

Code:

#!/bin/sh
# Default acpi script that takes an entry for all actions

IFS=${IFS}/
set $@

case "$1" in
  button)
    case "$2" in
      power) /sbin/init 0
         ;;
      lid)
         logger "Suspending to RAM"
         echo mem > /sys/power/state
         ;;
      *) logger "ACPI action $2 is not defined"
         ;;
    esac
    ;;
  *)
    logger "ACPI group $1 / action $2 is not defined"
    ;;
esac

in my /etc/acpi/acpi_handler.sh. Is this the correct way to do it?

Thanks!

ruario · 09-24-2013, 09:04 AM

I would blacklist it if I was you, others here with that issue certainly have

Quote:

Originally Posted by AlleyTrotter

Same problem here!
I finally got it fixed by blacklisting module 'mei-me'

Code:

john@linux:~$ cat /etc/modprobe.d/BLACKLIST-mei.conf 
# Do not load the kernel mei modules, since they interfere with S3
blacklist mei-me

Can't take credit as I got the recommendation from fearless leader PV
Of course the configuration of the kernel must include

Code:

CONFIG_INTEL_MEI=m
CONFIG_INTEL_MEI_ME=m

built as modules so's we can blacklist them.
Hope that helps
John

Sorry! Should have included that my problem also include a flood of the lines below in syslog. However the last line of my and your kernel dump are identical.

Code:

Aug  3 09:57:55 linux kernel: [ 1172.798841] mei_me 0000:00:16.0: unexpected reset: dev_state = RESETTING
Aug  3 09:57:55 linux kernel: [ 1172.799750] mei_me 0000:00:16.0: reset: wrong host start response

Miranden · 09-24-2013, 09:41 AM

Hmm, well if Pat says to blacklist it, I guess that's all we need to say about it.

I'm still wondering about one other thing though. Can someone please explain to me about how these bug reports work? What I mean is, why is the bug report I linked before on the Arch site? If it is a kernel issue, why is it being fixed by the Arch devs? If I wanted to tell someone that I still had the issue, who would I tell (of course I see that it says on the above arch link that it is fixed in 3.11.1, but I don't know if that is just for Arch or for the kernel itself, which is one of the reasons I am asking this question.)