LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 12-24-2017, 01:57 PM   #1
wvv
LQ Newbie
 
Registered: Dec 2017
Posts: 3

Rep: Reputation: Disabled
Debian Jessie - deleted files take up space after reboot


Hello guys,

I am a Linux newbie with a Debian jessie openmediavault-based server used as a NAS.

last night I was uploading a large set of files (~20G) via nextcloud web interface. Only after it failed I realized that upload_tmp_dir was not properly set. In result I ended up having root partition full. Without reading much, I deleted temporary files hoping it would bring my space back. Only then I found out that those files are still open by php and mysql.

I tried killing processes which write to those files, truncating files themselves as well as rebooting to no avail. The space is still eaten up to 98% ( I only managed to reclaim a bit over 400MB through apt cleaning.

As soon as I try to kill the process, a new one seems to be starting (at least that's my impression) which uses deleted files.

I've spent last 4 hours trying to resolve this issue without luck as nothing seems to be working.

Could you please advise if there's anything I can do to get my disk space back?
Thanks a lot in advance!

Lsof +L1 output below
PHP Code:
COMMAND    PID                  USER   FD   TYPE DEVICE SIZE/OFF NLINK NODE NAME
mysqld   13358                 mysql    4u   REG    8
,1        0     0   87 /tmp/ibbSdSAV (deleted)
mysqld   13358                 mysql    5u   REG    8,1        0     0  323 /tmp/ibbL2rDT (deleted)
mysqld   13358                 mysql    6u   REG    8,1        0     0  768 /tmp/ibXK61FR (deleted)
mysqld   13358                 mysql    7u   REG    8,1        0     0  778 /tmp/ibTcrpNN (deleted)
mysqld   13358                 mysql   11u   REG    8,1        0     0  953 /tmp/ibDJuJXL (deleted)
php5-fpm 14110                  root    4u   REG    8,1        0     0  147 /tmp/.ZendSem.sLCOOH (deleted)
php5-fpm 14112              www-data    4u   REG    8,1        0     0  147 /tmp/.ZendSem.sLCOOH (deleted)
php5-fpm 14113              www-data    4u   REG    8,1        0     0  147 /tmp/.ZendSem.sLCOOH (deleted)
php5-fpm 14114              www-data    4u   REG    8,1        0     0  147 /tmp/.ZendSem.sLCOOH (deleted)
php5-fpm 14115              www-data    4u   REG    8,1        0     0  147 /tmp/.ZendSem.sLCOOH (deleted)
php5-fpm 14116 openmediavault-webgui    4u   REG    8,1        0     0  147 /tmp/.ZendSem.sLCOOH (deleted)
php5-fpm 14117 openmediavault-webgui    4u   REG    8,1        0     0  147 /tmp/.ZendSem.sLCOOH (deleted)
php5-fpm 14118              www-data    4u   REG    8,1        0     0  147 /tmp/.ZendSem.sLCOOH (deleted)
php5-fpm 14119              www-data    4u   REG    8,1        0     0  147 /tmp/.ZendSem.sLCOOH (deleted)
php5-fpm 14676 openmediavault-webgui    4u   REG    8,1        0     0  147 /tmp/.ZendSem.sLCOOH (deleted
PS. Ncdu shows that rootfs is only 3.5G (out of 20G) which is about how it was last night before the issue occurred. Df -h shows 98% while df -i shows 16%

Last edited by wvv; 12-24-2017 at 02:01 PM.
 
Old 12-25-2017, 04:50 AM   #2
ondoho
LQ Addict
 
Registered: Dec 2013
Posts: 16,858
Blog Entries: 10

Rep: Reputation: 5013Reputation: 5013Reputation: 5013Reputation: 5013Reputation: 5013Reputation: 5013Reputation: 5013Reputation: 5013Reputation: 5013Reputation: 5013Reputation: 5013
reboot, or restart all the services in question.

if that doesn't help, you really need to tell us where those files are, and how you tried deleting them.
sorry if i didn't fully understand; it seems like you got yourself into an unholy mess by taking a wrong turn early on.
 
Old 12-25-2017, 05:09 AM   #3
wvv
LQ Newbie
 
Registered: Dec 2017
Posts: 3

Original Poster
Rep: Reputation: Disabled
Quote:
reboot, or restart all the services in question.

if that doesn't help, you really need to tell us where those files are, and how you tried deleting them.
sorry if i didn't fully understand; it seems like you got yourself into an unholy mess by taking a wrong turn early on.
I've already rebooted several times. Those files were in /tmp, they were created during PHP upload via web interface. I removed them normally using rm -rf. Those files are no longer on the filesystem, inodes reside in /proc/PID/fd, truncating them with > does not seem to give desired effect. Moreover, all these inodes are 1kB according to lsof.

I reclaimed additional 600MB through decreasing reserved space with tune2fs -m, but over 16GB is still missing.

To sum up, I have no idea where those 16GBs are, ncdu shows disk usage of around 3.5GB on root partition.

df -Th
Code:
Filesystem     Type      Size  Used Avail Use% Mounted on
udev           devtmpfs   10M     0   10M   0% /dev
tmpfs          tmpfs     1.5G   29M  1.5G   2% /run
/dev/sda1      ext4       20G   19G  947M  96% /
tmpfs          tmpfs     3.8G     0  3.8G   0% /dev/shm
tmpfs          tmpfs     5.0M     0  5.0M   0% /run/lock
tmpfs          tmpfs     3.8G     0  3.8G   0% /sys/fs/cgroup
/dev/sda2      ext4      666G  575G   57G  92% /STORAGE
df -Ti
Code:
Filesystem     Type       Inodes  IUsed    IFree IUse% Mounted on
udev           devtmpfs   976547    434   976113    1% /dev
tmpfs          tmpfs      980245    716   979529    1% /run
/dev/sda1      ext4      1310720 197023  1113697   16% /
tmpfs          tmpfs      980245      1   980244    1% /dev/shm
tmpfs          tmpfs      980245      4   980241    1% /run/lock
tmpfs          tmpfs      980245     15   980230    1% /sys/fs/cgroup
/dev/sda2      ext4     44351488 147556 44203932    1% /STORAGE

Last edited by wvv; 12-25-2017 at 05:39 AM.
 
Old 12-25-2017, 08:40 AM   #4
fatmac
Senior Member
 
Registered: Sep 2011
Location: Upper Hale, Surrey/Hants Border, UK
Posts: 4,029

Rep: Reputation: Disabled
Search for the files that you uploaded by the date or time stamp.
Code:
ls -t /
Go to init 1 so that nothing is mounted other than /, then delete them, & reboot.
Hopefully that should sort it out.
 
Old 12-25-2017, 08:47 AM   #5
_roman_
Member
 
Registered: Dec 2017
Location: _Austro_Bavaria_
Distribution: gentoo / linux mint
Posts: 433

Rep: Reputation: 29
why did you not use tmpfs for /tmp ?

I use baobab to find where my space is used up.
 
Old 12-26-2017, 05:04 AM   #6
MadeInGermany
Senior Member
 
Registered: Dec 2011
Location: Simplicity
Posts: 1,630

Rep: Reputation: 736Reputation: 736Reputation: 736Reputation: 736Reputation: 736Reputation: 736Reputation: 736
Perhaps by coincidence some other big files were created?
Not even /var is separate.
Find recently created big files in the / filesystem
Code:
find / -xdev -size +10000 -mtime -30
 
Old 12-26-2017, 05:27 AM   #7
wvv
LQ Newbie
 
Registered: Dec 2017
Posts: 3

Original Poster
Rep: Reputation: Disabled
Thanks for all your help guys, much appreciated!

It turns out I'm an idiot. My server got rebooted (most likely due to a short power outage), and in result encrypted drive used for backups was not automatically remounted. It happened on the day the backup was scheduled, so all the files were copied to an empty mountpoint (which I didn't scan with ndcu thinking it is of no purpose), I removed the failed backup files and remounted the drive which resolved the issue. I don't know why I didn't notice that the drive is unmounted in the first place...

Thanks again and sorry for being a pain
 
Old 12-26-2017, 06:07 AM   #8
ondoho
LQ Addict
 
Registered: Dec 2013
Posts: 16,858
Blog Entries: 10

Rep: Reputation: 5013Reputation: 5013Reputation: 5013Reputation: 5013Reputation: 5013Reputation: 5013Reputation: 5013Reputation: 5013Reputation: 5013Reputation: 5013Reputation: 5013
glad you got it solved.
please mark it so.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Zero out deleted files to recover disk space? JoeyB Linux - Newbie 3 03-23-2016 08:54 PM
Debian Jessie + Xen 4.3 doom0 reboot szistvan Linux - Virtualization and Cloud 1 03-15-2016 07:06 AM
Deleted Swap and Now Things are Going Wrong... (Debian Jessie) Alex Smith-Fanning Linux - Distributions 11 11-27-2014 07:01 PM
Deleted files on a SMB-share - Files gone, space still allocated - Fedora 10 Zwentendorf Linux - Server 4 03-01-2009 06:04 AM
Deleted files taking up space airwolf2 Linux - Newbie 2 12-09-2007 03:42 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 06:27 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration