[SOLVED] No space left on device, but no space used in directories

tioui · 05-04-2020, 07:26 PM

Hello everyone.

I have a strange problem on my server. When I looked at my discs usage, I get this:

Code:

# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            1.9G     0  1.9G   0% /dev
tmpfs           386M  5.6M  381M   2% /run
/dev/sda1        20G   19G     0 100% /
tmpfs           1.9G  8.0K  1.9G   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/sdb1       196G  103G   84G  56% /var

So, as you can see, the root partition is used at 100% with over 19Gib used. So I use ncdu to scan the disc to see where the files are. I got this:

Code:

    102.8 GiB [##########] /var
    2.7 GiB [          ] /usr
  422.1 MiB [          ] /lib
  101.0 MiB [          ] /root
   60.9 MiB [          ] /boot
   43.2 MiB [          ] /opt
    8.3 MiB [          ] /bin
    7.3 MiB [          ] /etc
    6.0 MiB [          ] /sbin
    5.5 MiB [          ] /run
  104.0 KiB [          ] /home
   60.0 KiB [          ] /tmp
    8.0 KiB [          ] /media
    8.0 KiB [          ] /mnt
    4.0 KiB [          ] /dev
    4.0 KiB [          ] /lib64
e   4.0 KiB [          ] /srv
e   4.0 KiB [          ] /lost+found
.   0.0   B [          ] /proc
    0.0   B [          ] /sys
@   0.0   B [          ]  initrd.img
@   0.0   B [          ]  initrd.img.old
@   0.0   B [          ]  vmlinuz
@   0.0   B [          ]  vmlinuz.old

Note that the var directory is on another disc. It seems that there is only like 3GiB used on the system. I tried rebooting the server and doing a "fsck -f" on the partition. I have looked on google and the two reasons this could happen does not seems to fit here. The first is some processes could be reserving deleted files. But since I reboot the system, they should have been deleted for good (well I presume). The other reason is the file system could be missing inodes. But when I do a "df -i", I seems to have plenty of inodes left:

Code:

# df -i
Filesystem       Inodes  IUsed    IFree IUse% Mounted on
udev             491108    323   490785    1% /dev
tmpfs            494071    464   493607    1% /run
/dev/sda1       1280000 234484  1045516   19% /
tmpfs            494071      3   494068    1% /dev/shm
tmpfs            494071      3   494068    1% /run/lock
tmpfs            494071     15   494056    1% /sys/fs/cgroup
/dev/sdb1      13107200 275017 12832183    3% /var

So, I am out of idea. The next thing I can do is reinstall, but it is quite a pain. If someone can help me so that I don't have to reinstall the server, I would be very grateful.

Thanks.

Louis M

scasey · 05-04-2020, 07:54 PM

My guess is log files...
What’s in /var/log. ?

tioui · 05-04-2020, 07:58 PM

Quote:

Originally Posted by scasey

My guess is log files...
What’s in /var/log. ?

Since /var is on another disk, every log in /var/log should be in another disk too.

ehartman · 05-04-2020, 08:03 PM

Quote:

Originally Posted by tioui

Since /var is on another disk, every log in /var/log should be in another disk too.

That is correct, and it's the root (/) partition that is full.
My guess would be /tmp and/or temporary files without a filename that haven't been marked "for remove" yes. Did you do a fsck on the / partition to find any "lost+found" files?

tioui · 05-04-2020, 08:10 PM

Quote:

Originally Posted by ehartman

That is correct, and it's the root (/) partition that is full.
My guess would be /tmp and/or temporary files without a filename that haven't been marked "for remove" yes. Did you do a fsck on the / partition to find any "lost+found" files?

I have indeed use a "fsck -f" and it does not find anything. The lost_found directory is empty.

File that does not have a filename? That is interesting. I did not know that those existed. Is it possible to list every file that don't have a filename.

ehartman · 05-04-2020, 08:20 PM

Quote:

Originally Posted by tioui

Is it possible to list every file that don't have a filename.

As they do not have a filename, no. They're a pure I-node with data attached, so can only be listed by scanning all I-nodes and eliminating those that are found somewhere in one of the directories. Normally they should have been "marked for delete" too.
The fsck run should have found them and assigned some sort of name, to put them into the lost+found directory (ext? file systems only, I don't know which type of fs your / partition is).

scasey · 05-04-2020, 08:24 PM

Quote:

Originally Posted by tioui

Since /var is on another disk, every log in /var/log should be in another disk too.

Oops! My bad. Missed that. Sorry.

tioui · 05-04-2020, 08:46 PM

Quote:

Originally Posted by ehartman

I don't know which type of fs your / partition is.

It is a Ext4 filesystem.

syg00 · 05-05-2020, 12:24 AM

You could do a bind mount to look at the underlying disk, but easier from a liveCD. I'm guessing when you created the new /var you didn't delete all the files on the old /var on the root that can no longer be seen because of the mount.

cbing · 05-05-2020, 01:06 AM

Hi!

Try with lsof to view possible deleted/unlinked files.

lsof | grep deleted or lsof +L1

descendant_command · 05-05-2020, 01:41 AM

Quote:

Originally Posted by syg00

You could do a bind mount to look at the underlying disk, but easier from a liveCD. I'm guessing when you created the new /var you didn't delete all the files on the old /var on the root that can no longer be seen because of the mount.

^^ This.

ehartman · 05-05-2020, 03:00 AM

Quote:

Originally Posted by tioui

It is a Ext4 filesystem.

As others have already remarked:
could you, in single user level, unmount all other filesystems and then look if any of the mount directories are NON-empty?
The files in those directories are non-visible when the other fs is mounted over them, but still take up space.

pan64 · 05-05-2020, 03:17 AM

Quote:

Originally Posted by ehartman

As others have already remarked:
could you, in single user level, unmount all other filesystems and then look if any of the mount directories are NON-empty?
The files in those directories are non-visible when the other fs is mounted over them, but still take up space.

No, that is not a good idea. Much better to do a bind mount, see post #9.
https://unix.stackexchange.com/quest...s-a-bind-mount

ehartman · 05-05-2020, 05:33 AM

Quote:

Originally Posted by pan64

No, that is not a good idea. Much better to do a bind mount, see post #9.

It could have worked on the OP's system, as both /usr and /home are in the / fs.
So only /var must be UNmounted and that's not really needed in single user level.
OK: you must be able to login as root in that single user system.

I forgot if he was using one of the ubuntu's or another silly distro that doesn't let you do so.

tioui · 05-05-2020, 07:57 AM

Quote:

Originally Posted by syg00

I'm guessing when you created the new /var you didn't delete all the files on the old /var on the root that can no longer be seen because of the mount.

Wow. Ok. Now I feel bad. You are right. There was old stuff in /var that was using 16Gib of space. I did not check in liveCD because this is a live server with services used by users.

Quote:

Originally Posted by pan64

Much better to do a bind mount.

This is a good trick. I will keep it in mind. Thank you.

And thanks to everyone to have help me. You guys are awesome.