[SOLVED] 95% full filesystem disallowing writes

mrmazda · 11-19-2018, 04:09 AM

Code:

# df /disks/esata2
Filesystem      1K-blocks       Used Available Use% Mounted on
/dev/sdc1      1937991472 1831610620 106380852  95% /disks/esata2
# df -h /disks/esata2
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdc1       1.9T  1.8T  102G  95% /disks/esata2
# mount | grep sdc1
/dev/sdc1 on /disks/esata2 type ext3 (rw,noatime,data=ordered)
# ls -ld /disks/esata2/
drwxr-xr-x 13 root root 4096 Jul  9 21:41 /disks/esata2/
# tune2fs -l /dev/sdc1
tune2fs 1.42.8 (20-Jun-2013)
Filesystem volume name:   s2tvm3mmvk
Last mounted on:          /media/hdd
Filesystem UUID:          e527cbfc-e43d-43b4-ae23-890e3e361a1a
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file
Filesystem flags:         signed_directory_hash 
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              122101760
Block count:              488378368
Reserved block count:     0
Free blocks:              26595213
Free inodes:              122100204
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      907
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   256
Filesystem created:       Wed Sep 27 14:44:41 2017
Last mount time:          Mon Nov 19 04:41:52 2018
Last write time:          Mon Nov 19 04:41:52 2018
Mount count:              6
Maximum mount count:      -1
Last checked:             Sun Nov 18 21:19:39 2018
Check interval:           0 (<none>)
Lifetime writes:          2209 GB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:	          128
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      6d3cf006-5e9e-44ab-8c10-774a2669ba35
Journal backup:           inode blocks

I don't see any clues in a dir listing of /disks/, the journal, or dmesg to explain what's preventing file writes on this filesystem. Even creation of a 0 byte file is disallowed. Is there something in the tune2fs -l output that explains? What else can I check?

FWIW, this disk is a brand new disk freshly cloned from a disk out of a STB satellite receiver that runs on an old 3.9.2 kernel and has a limited support set, and doesn't support EXT4. I initially configured the disk using mkfs.ext3 -I128 -m0..., probably with openSUSE 13.1, more than a year ago. Files on the disk are mostly HD h264 recordings, so rather large in size, and small in relative quantity.

Edit/Solved:
The mount point directory inexplicably had its immutable bit enabled.

hydrurga · 11-19-2018, 04:23 AM

By default, 5% of the blocks in an ext system are reserved for the root user, partially so that if the filesystem gets really full then you can still get in there with the root user and take steps to relieve the problem, but also to prevent filesystem fragmentation and resultant system performance issues.

You can change this value with tune2fs -m and -r.

The reserved blocks are only really intended for system drives. You can reduce the figure for data drives. Setting it to 0 won't affect performance much except if run for long periods of time with many file creates and deletes while the filesystem is almost full.

Edit:

I see now that the problem was something else and that your Reserved Block Count is 0 anyway. I'll leave this reply up here though in case it is of use to anyone...

mrmazda · 11-19-2018, 04:42 AM

Reserved blocks would have had nothing to do with the issue. Root could not write anything affected by the immutable bit on the parent directory. Reserved blocks were zero as the tune2fs output shows.