[SOLVED] Debian disk full

fviganeux · 07-23-2022, 02:05 PM

Hi there!
I am having trouble finding answers to this question for I get quite a large ammount of results...
Thing is I get a disk full in Debian, and when I try -from root directory- something like:

Code:

du -shx . | sort -rh | head

I get this:

Code:

 58G     .

Previously I tried:

Code:

du -h --max-depth=1

Results:

Code:

8.0K    ./media
1.4G    ./lib
182M    ./boot
4.0K    ./mnt
4.0K    ./lib64
16K     ./opt
0       ./dev
15G     ./usr
176K    ./root
du: cannot access './proc/2225/task/2225/fd/4': No such file or directory
du: cannot access './proc/2225/task/2225/fdinfo/4': No such file or directory
du: cannot access './proc/2225/fd/3': No such file or directory
du: cannot access './proc/2225/fdinfo/3': No such file or directory
0       ./proc
40K     ./tmp
0       ./sys
13M     ./etc
8.3M    ./bin
89M     ./run
4.0K    ./srv
5.4M    ./sbin
959M    ./home
53G     ./var
16K     ./lost+found
70G     .

Thanks a lot!
Regards!

jailbait · 07-23-2022, 02:47 PM

What are you trying to find out with your command? You say the results are weird. What results are you looking for?

michaelk · 07-23-2022, 03:33 PM

The -x option skips different filesystems so different results would be expected. Which filesystem is full? Post the output of the command

df -h

sundialsvcs · 07-23-2022, 04:13 PM

In any case, let us hope that your system uses LVM = Logical Volume Management, in which case the solution to your problem should be very easy and straightforward to solve.

fviganeux · 07-23-2022, 04:44 PM

Oh, don't they look weird? Then, it's me the one that doesn't know better.
What draws my attention, for I do not know what it means, is the:

Code:

 70GB   .

And, here we have the output of the suggested command:

Code:

Filesystem      Size  Used Avail Use% Mounted on
udev            3.9G     0  3.9G   0% /dev
tmpfs           799M   89M  710M  12% /run
/dev/sda2        58G   58G     0 100% /
tmpfs           4.0G     0  4.0G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           4.0G     0  4.0G   0% /sys/fs/cgroup
overlay          58G   58G     0 100% /var/lib/docker/overlay2/33037fa94724c02f9cd8a91c893f27299ec7b80ad791cf11ce9f0355dc2b50cd/merged
shm              64M     0   64M   0% /var/lib/docker/containers/d0722cf0ed5d18cf50ebd2f65cc83e6167c2adffb91a0efccb7ec605a6df3a0e/mounts/shm
overlay          58G   58G     0 100% /var/lib/docker/overlay2/4fd846a365f9eec5c33500d3e2ef1c3b26f9c9f4e334f7b1e81d2d647d380767/merged
overlay          58G   58G     0 100% /var/lib/docker/overlay2/d1ea6d238faf301f09e2b28b806222f9564180351257803266c24b3e96b629bf/merged
overlay          58G   58G     0 100% /var/lib/docker/overlay2/f0a8eea282f8d2fe242994a036b4427ce4366c2518fe22f157765e38e27641d3/merged
overlay          58G   58G     0 100% /var/lib/docker/overlay2/5d12c8f19834bb388f49b7bddb7115f15042f02f7a0306ac33e7be3d77c1f826/merged
shm              64M  4.0K   64M   1% /var/lib/docker/containers/f8f7516e2ec6ec015c84570a8b3f7c8e8defc164a74a97ae70d06f33cfad145b/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/d7154f9c8860d0020f566dee55308e3684e85be9ec3cb07ad02bb9ab699fa59c/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/278029064ff63d7a13be18db0aea10753f8b5575e7134e63233d04bacafda9b0/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/7e44e36ddf2ceb9e595f69c24753e4cc88945f3c6c0201fcc5ae95a80a87f55a/mounts/shm
overlay          58G   58G     0 100% /var/lib/docker/overlay2/884949eb3da806108e290594a7de9d6fe5708131e30c9f777656772d7df0c49c/merged
shm              64M     0   64M   0% /var/lib/docker/containers/6c44c1077cf6206949db90ed0ab163c85b042c64e1aae655f88b8f258140a220/mounts/shm
overlay          58G   58G     0 100% /var/lib/docker/overlay2/ca637affde1bd3794db480b136faf49b113e6610564233b7b1f41bf4e58269ad/merged
shm              64M     0   64M   0% /var/lib/docker/containers/a92a48288e9c96348e3a2a98750968a015259ecea95b301c7126dd9bcb252232/mounts/shm
overlay          58G   58G     0 100% /var/lib/docker/overlay2/35c08a957e8fa14eff9ef902239d914b14518b5e11b86c26da174215a9787282/merged
shm              64M     0   64M   0% /var/lib/docker/containers/d957381142d86ba7c4cd7c0a2b45e5cc044a8fecf7b0ae9aee19c44955fd3bc1/mounts/shm
overlay          58G   58G     0 100% /var/lib/docker/overlay2/c50dd94e88e16bfef84d6d23e0966287020f0924bde61b00451b360a09572edf/merged
shm              64M     0   64M   0% /var/lib/docker/containers/ad4dd64d556e33929fc13fbf719184d544687848d723381e9eb2ac3c49f67dd2/mounts/shm
overlay          58G   58G     0 100% /var/lib/docker/overlay2/f2b3b1049f83ea060c003813886de91d275ae1436177e82c7f10ba6432b0c52a/merged
shm              64M     0   64M   0% /var/lib/docker/containers/a19e139d34e7552758352d287efd5f6638af95c8ce6b1b9b765c6e9dcc2040da/mounts/shm
tmpfs           799M     0  799M   0% /run/user/1013

This looks quite more familiar to me, in regard to the mountpoint / and the disk, which is full, being /dev/sda2

I'm afraid I have not a clue concerning the LVM...

michaelk · 07-23-2022, 05:00 PM

You are not using LVM and just one partition. You need to find what directory is the largest.

fviganeux · 07-23-2022, 07:00 PM

Quote:

Originally Posted by michaelk

You are not using LVM and just one partition. You need to find what directory is the largest.

Oh, that's what I'm up to, that's how I got to those command lines related to 'du' and when saw the 70G with the dot, I got confused... Does it mean that the root directory weights 70G? And, do you have a suggestion on how to get to the directory or directories that are the most occupied?

michaelk · 07-23-2022, 07:38 PM

Try using sudo or run as root
du -a / | sort -n -r | head -n 20

fviganeux · 07-24-2022, 12:21 PM

Quote:

Originally Posted by michaelk

Try using sudo or run as root
du -a / | sort -n -r | head -n 20

I'm done for!

Code:

sort: write failed: /tmp/sortOZkilb: No space left on device

I'll see what I can securely delete and try again...

computersavvy · 07-24-2022, 05:03 PM

Post 1 shows /var as 53 GB.
Post 5 shows /var/lib/docker with many containers and overlays.

Since you cannot run sort because it requires space in /tmp to write, why not investigate the extreme amount of data in /var and begin by deleting something there that is not needed

evo2 · 07-24-2022, 06:43 PM

Most likely docker. You can purge unused images

Code:

docker image prune

May want to look into running it with the -a option too.

Evo2.

fviganeux · 07-25-2022, 10:44 AM

Quote:

Originally Posted by computersavvy

Post 1 shows /var as 53 GB.
Post 5 shows /var/lib/docker with many containers and overlays.

Since you cannot run sort because it requires space in /tmp to write, why not investigate the extreme amount of data in /var and begin by deleting something there that is not needed

Great Indeed! Thanks, I'll see to it. I did no quite understand the results, I guess.
For what I see, there is no a straight method to get to the point, but you have to scrutinize along the disk.

fviganeux · 07-25-2022, 10:45 AM

Quote:

Originally Posted by evo2

Most likely docker. You can purge unused images

Code:

docker image prune

May want to look into running it with the -a option too.

Evo2.

Great, I'll speak with the 'owner' of the server. I work in a team on the 'server' side, while this guy is on development. I just create vms on a proxmox and they use them. But have never got with this problem before.

Thanks!

fviganeux · 07-30-2022, 07:09 AM

Hey! I'm alive, apologize for not writing back since last time.
Thing is I got nothing to say. Still am, but wanted to give some feedback, and of course, thank you all for the time invested in helping me with my noobiness and teaching me how to auscultate the disk!
Guys in charge of the machine we've been talking about are scratching their heads. Apparently there is a thing with the /var growing so much and the overlays and such.

I would say you've had enough from me for the time being. Dunno if you 'close' the thread, or if we can call it 'solved', but as far as I reckon, I'm more than thankful and plenty of new disk-space-knowledge.

Regards!

mrmazda · 07-30-2022, 04:37 PM

If you want an easy way to discover space consumed, try ncdu.