Disk space full, and can't remove files because it's full?

Legend of Ron · 06-29-2016, 06:54 PM

First off, Hello. This is my first post. I'm trying to learn Linux. I started with slackware64 14.1. I installed it and have been using it just fine for weeks. Now today, I put in a HDD, all good. Now I try to delete some files from it, well trash gets full. I open trash and delete it from there. Somewhere along the way desktop crashes, and most commands won't work. It keeps saying disk space full. Even remove commands.

I'm at a loss. I just don't known Linux well enough yet to troubleshoot it, I don't even know how to phrase the problem well enough to find a solution for my system.

As for the system. It's slackware 14.1 x64. Installed on a 120G SSD. Split into two partitions /dev/sda1 is just a small boot partition, the rest is an extended partition with 3 lvm luks encrypted partitions.

I checked disk usage and everything has space. I tried checking inodes, but from what I read, being encrypted can make that read zero.

perbh · 06-30-2016, 11:46 AM

If you try to delete too many files at a time - the disk may be used to hold the names. Just start with the biggest files you want/need to delete and delete them one at a time. As you release space, you can start to select more files ...
Oh - I wouldn't use the desktop to do it - use the console command line.

bassmadrigal · 06-30-2016, 12:31 PM

Depending on your partition layout, you could just remove everything in the /tmp/ directory. None of that is required for a properly running system.

Code:

rm -rf /tmp/*

If you have a separate /home/ partition, you'd need to find some large files to remove. As perbh mentioned, I would highly suggest using the commandline to remove stuff, as trying to use a filemanager may cause additional issues if it's trying to generate thumbnails or other similar files.

If you need to find out what's taking all your space, you can use the handy command below. Then once you find a large directory you'd like more info on, you can change into that directory and run the command again until you find some stuff you can delete.

Code:

du -hd 1 | sort -h

Legend of Ron · 06-30-2016, 03:43 PM

It actually ended up being btrfs. It was working fine for a while, so I forgot that could be an issue. I still couldn't get the workarounds to fix it though. So I went back to ext4. I'll play around with btrfs on a separate drive, vm, or something another time.

kjhambrick · 07-01-2016, 06:47 AM

Quote:

Originally Posted by Legend of Ron

First off, Hello. This is my first post.

Welcome to LQ Legend of Ron !

Quote:

I'm trying to learn Linux. I started with slackware64 14.1. I installed it and have been using it just fine for weeks. Now today, I put in a HDD, all good. Now I try to delete some files from it, well trash gets full. I open trash and delete it from there. Somewhere along the way desktop crashes, and most commands won't work. It keeps saying disk space full. Even remove commands.

What tool are you using to delete files ?

The reference to Trash sounds like a GUI tool at work.

Are you able to delete files from the command line ?

Quote:

As for the system. It's slackware 14.1 x64. Installed on a 120G SSD. Split into two partitions /dev/sda1 is just a small boot partition, the rest is an extended partition with 3 lvm luks encrypted partitions.

I checked disk usage and everything has space. I tried checking inodes, but from what I read, being encrypted can make that read zero.

bassmadrigal's and perbh's posts have some useful hints.

And now I need to study up on LUKS and inode counts ... IMO, it would be a nasty regression if LUKS hides or mis-reports inode counts !

Thanks for the info.

-- kjh

jpollard · 07-01-2016, 09:07 AM

Quote:

Originally Posted by Legend of Ron

It actually ended up being btrfs. It was working fine for a while, so I forgot that could be an issue. I still couldn't get the workarounds to fix it though. So I went back to ext4. I'll play around with btrfs on a separate drive, vm, or something another time.

Rebalancing didn't work? The usual cause is that the space for allocations gets split between the metadata and the actual data. Rebalancing is SUPPOSED to change the split, which would then allow you some space.

The problem sounds like too much GUI. That doesn't delete things very well - as it actually does a "mv src trash" type thing, which can require an expansion of the trash directory, if not a full copy of the file. Using the command line "rm" will delete files, and with a minimum (to no) overhead.

Gerardo Zamudio · 07-01-2016, 12:06 PM

When your disk is full I don't recommend using rm. That only deletes the pointer to the file but the data may still be on disk. It's best to find a big file, like a large log, and empty it out.

Code:

root@hades:~ # df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2        61G   23G   36G  39% /
/dev/sdc1       1.8T  929G  813G  54% /home
/dev/sda1        50G   48G  2.2G  96% /windows
tmpfs           3.9G  110M  3.8G   3% /dev/shm
/dev/sdb1       3.7T  1.5T  2.3T  40% /games

root@hades:~ # cd /var/log/
root@hades:/var/log # du -h --max-depth=1
24M     ./packages
1.4M    ./removed_scripts
5.7M    ./scripts
4.0K    ./uucp
4.0K    ./cups
4.0K    ./sa
14M     ./removed_packages
4.0K    ./httpd
248K    ./ConsoleKit
9.2G    ./libvirt
92K     ./setup
3.5M    ./sbopkg
4.0K    ./iptraf-ng
11M     ./teamviewer11
4.0K    ./samba
4.0K    ./nfsd
9.3G    .

root@hades:/var/log # cd libvirt/
root@hades:/var/log/libvirt # ls -lh
total 9.2G
-rw------- 1 root root 8.2G Jul  1 11:03 libvirtd.log
-rw------- 1 root root 659M May  3 04:40 libvirtd.log.1
-rw------- 1 root root  88M Apr 30 04:41 libvirtd.log.2.gz
-rw------- 1 root root 183M Apr 13 04:40 libvirtd.log.3.gz
-rw------- 1 root root  83M Mar  1 04:40 libvirtd.log.4.gz
drwxr-xr-x 2 root root 4.0K Jan  3 02:22 lxc/
drwxr-xr-x 2 root root 4.0K May 24 17:05 qemu/
drwxr-xr-x 2 root root 4.0K Jan  3 02:22 uml/

root@hades:/var/log/libvirt # > libvirtd.log

root@hades:/var/log/libvirt # ls -lh
total 1012M
-rw------- 1 root root    0 Jul  1 11:03 libvirtd.log
-rw------- 1 root root 659M May  3 04:40 libvirtd.log.1
-rw------- 1 root root  88M Apr 30 04:41 libvirtd.log.2.gz
-rw------- 1 root root 183M Apr 13 04:40 libvirtd.log.3.gz
-rw------- 1 root root  83M Mar  1 04:40 libvirtd.log.4.gz
drwxr-xr-x 2 root root 4.0K Jan  3 02:22 lxc/
drwxr-xr-x 2 root root 4.0K May 24 17:05 qemu/
drwxr-xr-x 2 root root 4.0K Jan  3 02:22 uml/

root@hades:/var/log/libvirt # df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2        61G   15G   44G  25% /
/dev/sdc1       1.8T  929G  813G  54% /home
/dev/sda1        50G   48G  2.2G  96% /windows
tmpfs           3.9G  110M  3.8G   3% /dev/shm
/dev/sdb1       3.7T  1.5T  2.3T  40% /games

I found the file /var/log/libvirt/libvirtd.log was 8.2GB. I redirected null to it to empty it and I gained that space back immediately.

kjhambrick · 07-01-2016, 12:20 PM

Interesting Gerardo Zamudio ...

Never thought of doing it that-a-way but next time I run into a full file system, I'll certainly do it your way (

if I remember

)

Off topic, but I have to ask ... ( Your log entries beg the question ), why are the libvirt logs so huge ?

-- kjh

Gerardo Zamudio · 07-01-2016, 12:28 PM

Eh, some sort of constant polling I guess.

Code:

root@hades:/var/log/libvirt # tail -f libvirtd.log
2016-07-01 16:02:07.714+0000: 861: info : virObjectUnref:259 : OBJECT_UNREF: obj=0x7fdbc40d1f80
2016-07-01 16:02:07.714+0000: 861: info : virNetServerClientSendMessageLocked:1405 : RPC_SERVER_CLIENT_MSG_TX_QUEUE: client=0x7fdbd8e60200 len=60 prog=536903814 vers=1 proc=16 type=1 status=0 serial=11178
2016-07-01 16:02:07.714+0000: 861: info : virEventPollUpdateHandle:152 : EVENT_POLL_UPDATE_HANDLE: watch=11 events=3
2016-07-01 16:02:07.714+0000: 861: info : virObjectUnref:259 : OBJECT_UNREF: obj=0x7fdbd8e2c0c0
2016-07-01 16:02:07.714+0000: 861: info : virObjectUnref:259 : OBJECT_UNREF: obj=0x7fdbd8e60200
2016-07-01 16:02:07.714+0000: 856: info : virEventPollDispatchHandles:507 : EVENT_POLL_DISPATCH_HANDLE: watch=1 events=1
2016-07-01 16:02:07.714+0000: 856: info : virEventPollRunOnce:641 : EVENT_POLL_RUN: nhandles=11 timeout=5000
2016-07-01 16:02:07.714+0000: 856: info : virEventPollDispatchHandles:507 : EVENT_POLL_DISPATCH_HANDLE: watch=11 events=2
2016-07-01 16:02:07.714+0000: 856: info : virEventPollUpdateHandle:152 : EVENT_POLL_UPDATE_HANDLE: watch=11 events=1
2016-07-01 16:02:07.714+0000: 856: info : virEventPollRunOnce:641 : EVENT_POLL_RUN: nhandles=11 timeout=5000
2016-07-01 16:02:10.717+0000: 856: info : virEventPollDispatchHandles:507 : EVENT_POLL_DISPATCH_HANDLE: watch=11 events=1
2016-07-01 16:02:10.717+0000: 856: info : virEventPollUpdateHandle:152 : EVENT_POLL_UPDATE_HANDLE: watch=11 events=1
2016-07-01 16:02:10.717+0000: 856: info : virNetServerClientDispatchRead:1148 : RPC_SERVER_CLIENT_MSG_RX: client=0x7fdbd8e60200 len=28 prog=536903814 vers=1 proc=6 type=0 status=0 serial=11179
2016-07-01 16:02:10.717+0000: 856: info : virEventPollUpdateTimeout:265 : EVENT_POLL_UPDATE_TIMEOUT: timer=2 frequency=5000
2016-07-01 16:02:10.717+0000: 856: info : virObjectRef:296 : OBJECT_REF: obj=0x7fdbd8e60200
2016-07-01 16:02:10.717+0000: 856: info : virObjectRef:296 : OBJECT_REF: obj=0x7fdbd8e2c0c0
2016-07-01 16:02:10.717+0000: 856: info : virEventPollUpdateHandle:152 : EVENT_POLL_UPDATE_HANDLE: watch=11 events=1
2016-07-01 16:02:10.717+0000: 856: info : virEventPollRunOnce:641 : EVENT_POLL_RUN: nhandles=11 timeout=5000
2016-07-01 16:02:10.717+0000: 866: info : virObjectRef:296 : OBJECT_REF: obj=0x7fdbc40d1f80
2016-07-01 16:02:10.717+0000: 866: info : virObjectRef:296 : OBJECT_REF: obj=0x7fdbc40d1f80
2016-07-01 16:02:10.717+0000: 866: info : virObjectRef:296 : OBJECT_REF: obj=0x7fdbd8e2cd30
2016-07-01 16:02:10.717+0000: 866: info : virObjectUnref:259 : OBJECT_UNREF: obj=0x7fdbd8e2cd30
2016-07-01 16:02:10.717+0000: 866: info : virObjectUnref:259 : OBJECT_UNREF: obj=0x7fdbc40d1f

I have plenty of space and the logs are taken care of by logrotate so I don't worry about it. If it wasn't for this thread I probably would have never even noticed it was even happening

jpollard · 07-01-2016, 12:43 PM

Quote:

Originally Posted by Gerardo Zamudio

When your disk is full I don't recommend using rm. That only deletes the pointer to the file but the data may still be on disk. It's best to find a big file, like a large log, and empty it out.

That is exactly how files are deleted. When the last pointer to the file is removed, the file is deleted.

When a file is NOT deleted, there are still pointers to the file.

jpollard · 07-01-2016, 12:46 PM

Quote:

Originally Posted by kjhambrick

Interesting Gerardo Zamudio ...

Never thought of doing it that-a-way but next time I run into a full file system, I'll certainly do it your way (

if I remember

)

Off topic, but I have to ask ... ( Your log entries beg the question ), why are the libvirt logs so huge ?

-- kjh

He must not be running a log rotate... His recommendation risks corrupting the log (don't EVER do that to a journald log).

bassmadrigal · 07-01-2016, 01:39 PM

Quote:

Originally Posted by jpollard

That is exactly how files are deleted. When the last pointer to the file is removed, the file is deleted.

When a file is NOT deleted, there are still pointers to the file.

This is true in the sense that nothing references the file anymore, but it doesn't actually delete anything. This isn't an issue with HDDs, because they'll just overwrite that sector when new data needs to be written, but SSDs require the sector to be zeroed out before it can be written, which running rm won't do. If you have an SSD, you need to trim the drive after the data is "deleted", as this prepares it for future writes. If you don't, when you try to write something else to that sector, you have to first clear it and then write it, doubling the time it takes to write. This is why fstrim or discard options in fstab are so important on SSDs.

But I will admit, I'm not familiar with btrfs or other similar setups, so it could be completely different there compared to normal HDD or SSD operations.

jpollard · 07-01-2016, 05:36 PM

Quote:

Originally Posted by bassmadrigal

This is true in the sense that nothing references the file anymore, but it doesn't actually delete anything. This isn't an issue with HDDs, because they'll just overwrite that sector when new data needs to be written, but SSDs require the sector to be zeroed out before it can be written, which running rm won't do. If you have an SSD, you need to trim the drive after the data is "deleted", as this prepares it for future writes. If you don't, when you try to write something else to that sector, you have to first clear it and then write it, doubling the time it takes to write. This is why fstrim or discard options in fstab are so important on SSDs.

That is handled by the SSD itself, not Linux. The fstrim option is the flag for the filesystem to provide that signal to the device.

Quote:

But I will admit, I'm not familiar with btrfs or other similar setups, so it could be completely different there compared to normal HDD or SSD operations.

As long as the filesystem supports SSD usage, there is no difference. And just truncating a file will NOT generate the signals to an SSD for the required activity - unless the filesystem ALREADY supports fstrim.

bassmadrigal · 07-01-2016, 05:43 PM

But it is still true that the file isn't "deleted" by simply removing the pointers. It only occurs on SSDs when trim is used, and on HDDs you need to manually do it, since there isn't something in place to do it (since there's really no need unless you need to securely remove the file).

jpollard · 07-01-2016, 05:51 PM

removing the pointers is all the filesystem needs. That is ALL that is needed.

The "trim" is only required by the device, and since it is a device activity, guess what - the filesystem has to do it.

If the filesystem doesn't support the trim, guess what - you can't do it at all. "manually do it" is called "writing data". Might be zeros, might not, but writing data is your activity.