[SOLVED] Debian Jessie

wvv · 12-24-2017, 12:57 PM

Hello guys,

I am a Linux newbie with a Debian jessie openmediavault-based server used as a NAS.

last night I was uploading a large set of files (~20G) via nextcloud web interface. Only after it failed I realized that upload_tmp_dir was not properly set. In result I ended up having root partition full. Without reading much, I deleted temporary files hoping it would bring my space back. Only then I found out that those files are still open by php and mysql.

I tried killing processes which write to those files, truncating files themselves as well as rebooting to no avail. The space is still eaten up to 98% ( I only managed to reclaim a bit over 400MB through apt cleaning.

As soon as I try to kill the process, a new one seems to be starting (at least that's my impression) which uses deleted files.

I've spent last 4 hours trying to resolve this issue without luck as nothing seems to be working.

Could you please advise if there's anything I can do to get my disk space back?
Thanks a lot in advance!

Lsof +L1 output below

PHP Code:



COMMAND    PID                  USER   FD   TYPE DEVICE SIZE/OFF NLINK NODE NAME
mysqld   13358                 mysql    4u   REG    8,1        0     0   87 /tmp/ibbSdSAV (deleted)
mysqld   13358                 mysql    5u   REG    8,1        0     0  323 /tmp/ibbL2rDT (deleted)
mysqld   13358                 mysql    6u   REG    8,1        0     0  768 /tmp/ibXK61FR (deleted)
mysqld   13358                 mysql    7u   REG    8,1        0     0  778 /tmp/ibTcrpNN (deleted)
mysqld   13358                 mysql   11u   REG    8,1        0     0  953 /tmp/ibDJuJXL (deleted)
php5-fpm 14110                  root    4u   REG    8,1        0     0  147 /tmp/.ZendSem.sLCOOH (deleted)
php5-fpm 14112              www-data    4u   REG    8,1        0     0  147 /tmp/.ZendSem.sLCOOH (deleted)
php5-fpm 14113              www-data    4u   REG    8,1        0     0  147 /tmp/.ZendSem.sLCOOH (deleted)
php5-fpm 14114              www-data    4u   REG    8,1        0     0  147 /tmp/.ZendSem.sLCOOH (deleted)
php5-fpm 14115              www-data    4u   REG    8,1        0     0  147 /tmp/.ZendSem.sLCOOH (deleted)
php5-fpm 14116 openmediavault-webgui    4u   REG    8,1        0     0  147 /tmp/.ZendSem.sLCOOH (deleted)
php5-fpm 14117 openmediavault-webgui    4u   REG    8,1        0     0  147 /tmp/.ZendSem.sLCOOH (deleted)
php5-fpm 14118              www-data    4u   REG    8,1        0     0  147 /tmp/.ZendSem.sLCOOH (deleted)
php5-fpm 14119              www-data    4u   REG    8,1        0     0  147 /tmp/.ZendSem.sLCOOH (deleted)
php5-fpm 14676 openmediavault-webgui    4u   REG    8,1        0     0  147 /tmp/.ZendSem.sLCOOH (deleted)

PS. Ncdu shows that rootfs is only 3.5G (out of 20G) which is about how it was last night before the issue occurred. Df -h shows 98% while df -i shows 16%

ondoho · 12-25-2017, 03:50 AM

reboot, or restart all the services in question.

if that doesn't help, you really need to tell us where those files are, and how you tried deleting them.
sorry if i didn't fully understand; it seems like you got yourself into an unholy mess by taking a wrong turn early on.

wvv · 12-25-2017, 04:09 AM

Quote:

reboot, or restart all the services in question.

if that doesn't help, you really need to tell us where those files are, and how you tried deleting them.
sorry if i didn't fully understand; it seems like you got yourself into an unholy mess by taking a wrong turn early on.

I've already rebooted several times. Those files were in /tmp, they were created during PHP upload via web interface. I removed them normally using rm -rf. Those files are no longer on the filesystem, inodes reside in /proc/PID/fd, truncating them with > does not seem to give desired effect. Moreover, all these inodes are 1kB according to lsof.

I reclaimed additional 600MB through decreasing reserved space with tune2fs -m, but over 16GB is still missing.

To sum up, I have no idea where those 16GBs are, ncdu shows disk usage of around 3.5GB on root partition.

df -Th

Code:

Filesystem     Type      Size  Used Avail Use% Mounted on
udev           devtmpfs   10M     0   10M   0% /dev
tmpfs          tmpfs     1.5G   29M  1.5G   2% /run
/dev/sda1      ext4       20G   19G  947M  96% /
tmpfs          tmpfs     3.8G     0  3.8G   0% /dev/shm
tmpfs          tmpfs     5.0M     0  5.0M   0% /run/lock
tmpfs          tmpfs     3.8G     0  3.8G   0% /sys/fs/cgroup
/dev/sda2      ext4      666G  575G   57G  92% /STORAGE

df -Ti

Code:

Filesystem     Type       Inodes  IUsed    IFree IUse% Mounted on
udev           devtmpfs   976547    434   976113    1% /dev
tmpfs          tmpfs      980245    716   979529    1% /run
/dev/sda1      ext4      1310720 197023  1113697   16% /
tmpfs          tmpfs      980245      1   980244    1% /dev/shm
tmpfs          tmpfs      980245      4   980241    1% /run/lock
tmpfs          tmpfs      980245     15   980230    1% /sys/fs/cgroup
/dev/sda2      ext4     44351488 147556 44203932    1% /STORAGE

fatmac · 12-25-2017, 07:40 AM

Search for the files that you uploaded by the date or time stamp.

Code:

ls -t /

Go to init 1 so that nothing is mounted other than /, then delete them, & reboot.
Hopefully that should sort it out.

_roman_ · 12-25-2017, 07:47 AM

why did you not use tmpfs for /tmp ?

I use baobab to find where my space is used up.

MadeInGermany · 12-26-2017, 04:04 AM

Perhaps by coincidence some other big files were created?
Not even /var is separate.
Find recently created big files in the / filesystem

Code:

find / -xdev -size +10000 -mtime -30

wvv · 12-26-2017, 04:27 AM

Thanks for all your help guys, much appreciated!

It turns out I'm an idiot. My server got rebooted (most likely due to a short power outage), and in result encrypted drive used for backups was not automatically remounted. It happened on the day the backup was scheduled, so all the files were copied to an empty mountpoint (which I didn't scan with ndcu thinking it is of no purpose), I removed the failed backup files and remounted the drive which resolved the issue. I don't know why I didn't notice that the drive is unmounted in the first place...

Thanks again and sorry for being a pain

ondoho · 12-26-2017, 05:07 AM

glad you got it solved.
please mark it so.