LinuxQuestions.org
Latest LQ Deal: Complete CCNA, CCNP & Red Hat Certification Training Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 12-06-2017, 05:29 PM   #1
ss1100
LQ Newbie
 
Registered: Dec 2017
Posts: 10

Rep: Reputation: Disabled
df shows full root mount, du shows barely used


I know this kind of question has come up before, but none of the answers I've seen have fixed it.

I filled my root mount in a local VM (Oracle Virtualbox 5.2, CentoOS7) by using Oracle's datapump to import data from another database. I dropped the tablespace and data files and they no longer exist in the file system:

Code:
[root@localhost /]# du -sh
du: cannot access ./proc/8660/task/8660/fd/4: No such file or directory
du: cannot access ./proc/8660/task/8660/fdinfo/4: No such file or directory
du: cannot access ./proc/8660/fd/4: No such file or directory
du: cannot access ./proc/8660/fdinfo/4: No such file or directory
38G     .
Code:
[root@localhost /]# df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   76G   72G  3.9G  95% /
devtmpfs                 1.9G     0  1.9G   0% /dev
tmpfs                    1.9G  989M  908M  53% /dev/shm
tmpfs                    1.9G  8.7M  1.9G   1% /run
tmpfs                    1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/sda1               1014M  235M  780M  24% /boot
tmpfs                    380M     0  380M   0% /run/user/0
Code:
[root@localhost /]#  lsof +L1
COMMAND    PID   USER   FD   TYPE DEVICE SIZE/OFF NLINK    NODE NAME
firewalld  854   root    8u   REG   0,17     4096     0   18518 /dev/shm/ffiIAjc9T (deleted)
tuned     1239   root    7u   REG   0,17     4096     0   21002 /dev/shm/ffiQmR4Wq (deleted)
oracle    1929 oracle    8u   REG  253,0        0     0 4488313 /u01/app/oracle/product/11.2.0/dbhome_1/dbs/lkinstPRDDB1 (deleted)
oracle    1931 oracle    8u   REG  253,0        0     0 4488313 /u01/app/oracle/product/11.2.0/dbhome_1/dbs/lkinstPRDDB1 (deleted)
oracle    1935 oracle    8u   REG  253,0        0     0 4488313 /u01/app/oracle/product/11.2.0/dbhome_1/dbs/lkinstPRDDB1 (deleted)
oracle    1937 oracle    8u   REG  253,0        0     0 4488313 /u01/app/oracle/product/11.2.0/dbhome_1/dbs/lkinstPRDDB1 (deleted)
oracle    1939 oracle    8u   REG  253,0        0     0 4488313 /u01/app/oracle/product/11.2.0/dbhome_1/dbs/lkinstPRDDB1 (deleted)
oracle    1941 oracle    8u   REG  253,0        0     0 4488313 /u01/app/oracle/product/11.2.0/dbhome_1/dbs/lkinstPRDDB1 (deleted)
oracle    1943 oracle    8u   REG  253,0        0     0 4488313 /u01/app/oracle/product/11.2.0/dbhome_1/dbs/lkinstPRDDB1 (deleted)
oracle    1945 oracle    8u   REG  253,0        0     0 4488313 /u01/app/oracle/product/11.2.0/dbhome_1/dbs/lkinstPRDDB1 (deleted)
oracle    1947 oracle    8u   REG  253,0        0     0 4488313 /u01/app/oracle/product/11.2.0/dbhome_1/dbs/lkinstPRDDB1 (deleted)
oracle    1949 oracle    8u   REG  253,0        0     0 4488313 /u01/app/oracle/product/11.2.0/dbhome_1/dbs/lkinstPRDDB1 (deleted)
oracle    1951 oracle    8u   REG  253,0        0     0 4488313 /u01/app/oracle/product/11.2.0/dbhome_1/dbs/lkinstPRDDB1 (deleted)
oracle    1953 oracle    8u   REG  253,0        0     0 4488313 /u01/app/oracle/product/11.2.0/dbhome_1/dbs/lkinstPRDDB1 (deleted)
oracle    1955 oracle    8u   REG  253,0        0     0 4488313 /u01/app/oracle/product/11.2.0/dbhome_1/dbs/lkinstPRDDB1 (deleted)
oracle    1957 oracle    8u   REG  253,0        0     0 4488313 /u01/app/oracle/product/11.2.0/dbhome_1/dbs/lkinstPRDDB1 (deleted)
oracle    1959 oracle    8u   REG  253,0        0     0 4488313 /u01/app/oracle/product/11.2.0/dbhome_1/dbs/lkinstPRDDB1 (deleted)
oracle    1961 oracle    8u   REG  253,0        0     0 4488313 /u01/app/oracle/product/11.2.0/dbhome_1/dbs/lkinstPRDDB1 (deleted)
oracle    1963 oracle    8u   REG  253,0        0     0 4488313 /u01/app/oracle/product/11.2.0/dbhome_1/dbs/lkinstPRDDB1 (deleted)
oracle    2001 oracle    8u   REG  253,0        0     0 4488313 /u01/app/oracle/product/11.2.0/dbhome_1/dbs/lkinstPRDDB1 (deleted)
oracle    2015 oracle    8u   REG  253,0        0     0 4488313 /u01/app/oracle/product/11.2.0/dbhome_1/dbs/lkinstPRDDB1 (deleted)
oracle    2043 oracle    8u   REG  253,0        0     0 4488313 /u01/app/oracle/product/11.2.0/dbhome_1/dbs/lkinstPRDDB1 (deleted)
oracle    2045 oracle    8u   REG  253,0        0     0 4488313 /u01/app/oracle/product/11.2.0/dbhome_1/dbs/lkinstPRDDB1 (deleted)
oracle    2169 oracle    8u   REG  253,0        0     0 4488313 /u01/app/oracle/product/11.2.0/dbhome_1/dbs/lkinstPRDDB1 (deleted)
oracle    8176 oracle    8u   REG  253,0        0     0 4488313 /u01/app/oracle/product/11.2.0/dbhome_1/dbs/lkinstPRDDB1 (deleted)
oracle    8403 oracle    8u   REG  253,0        0     0 4488313 /u01/app/oracle/product/11.2.0/dbhome_1/dbs/lkinstPRDDB1 (deleted)
oracle    8432 oracle    8u   REG  253,0        0     0 4488313 /u01/app/oracle/product/11.2.0/dbhome_1/dbs/lkinstPRDDB1 (deleted)
oracle    8595 oracle    8u   REG  253,0        0     0 4488313 /u01/app/oracle/product/11.2.0/dbhome_1/dbs/lkinstPRDDB1 (deleted)
oracle    8597 oracle    8u   REG  253,0        0     0 4488313 /u01/app/oracle/product/11.2.0/dbhome_1/dbs/lkinstPRDDB1 (deleted)
I mounted root into a new directory to prove that there are no hidden files:
Code:
[root@localhost /]# cd /mnt
[root@localhost mnt]# du -sh
37G     .
Gparted shows that my partition is full. I've tried using "sync" 3 times as well.

Format type:

Code:
[root@localhost mnt]# df -T
Filesystem              Type     1K-blocks     Used Available Use% Mounted on
/dev/mapper/centos-root xfs       78747076 74739848   4007228  95% /
devtmpfs                devtmpfs   1925356        0   1925356   0% /dev
tmpfs                   tmpfs      1940940  1011948    928992  53% /dev/shm
tmpfs                   tmpfs      1940940     8876   1932064   1% /run
tmpfs                   tmpfs      1940940        0   1940940   0% /sys/fs/cgroup
/dev/sda1               xfs        1038336   240044    798292  24% /boot
tmpfs                   tmpfs       388188        0    388188   0% /run/user/0
Also, inodes:

Code:
[root@localhost mnt]# df -i
Filesystem               Inodes  IUsed   IFree IUse% Mounted on
/dev/mapper/centos-root 8273272 258338 8014934    4% /
devtmpfs                 481339    364  480975    1% /dev
tmpfs                    485235     99  485136    1% /dev/shm
tmpfs                    485235    478  484757    1% /run
tmpfs                    485235     16  485219    1% /sys/fs/cgroup
/dev/sda1                524288    380  523908    1% /boot
tmpfs                    485235      1  485234    1% /run/user/0
Treat me like a total Linux n00b, I'm far from an expert.
 
Old 12-06-2017, 07:58 PM   #2
mrmazda
Member
 
Registered: Aug 2016
Location: USA
Distribution: openSUSE, Knoppix, Debian, Mageia, Fedora, more
Posts: 326

Rep: Reputation: 86
Quote:
Originally Posted by ss1100 View Post
... I dropped the tablespace and data files...
How? Some apps behave like Windows, moving "deleted" files into a "Trash" directory, which must be emptied before the space they consume is freed.
 
1 members found this post helpful.
Old 12-06-2017, 08:25 PM   #3
ss1100
LQ Newbie
 
Registered: Dec 2017
Posts: 10

Original Poster
Rep: Reputation: Disabled
I dropped the tablespace including contents, then deleted the data files (thought contents would have included the files, but it doesn't) using rm. I checked for trash folders, but I didn't see any under /root or any /home/*/.local directories. Is there another place they could be?
 
Old 12-06-2017, 08:45 PM   #4
ss1100
LQ Newbie
 
Registered: Dec 2017
Posts: 10

Original Poster
Rep: Reputation: Disabled
Also note that the VM has been rebooted a few times.
 
Old 12-06-2017, 11:22 PM   #5
jefro
Moderator
 
Registered: Mar 2008
Posts: 17,395

Rep: Reputation: 2585Reputation: 2585Reputation: 2585Reputation: 2585Reputation: 2585Reputation: 2585Reputation: 2585Reputation: 2585Reputation: 2585Reputation: 2585Reputation: 2585
http://docs.cray.com/books/S-2377-22...029470303.html maybe to start.
 
1 members found this post helpful.
Old 12-07-2017, 12:06 AM   #6
ss1100
LQ Newbie
 
Registered: Dec 2017
Posts: 10

Original Poster
Rep: Reputation: Disabled
Since I'm working on the root partition, I booted into gparted, deactivated the root partition, and then opened the terminal and ran xfs_repair -n.

Summary of what I got:

Code:
user@debian:~$ sudo xfs_repair -n /dev/sda2
Phase 1 - find and verify superblock...
bad primary superblock - bad magic number !!!

attempting to find secondary superblock...
...
found candidate secodary superblock...
unable to verify superblock, continuing...
...
found candidate secodary superblock...
unable to verify superblock, continuing...
(^ this iterates maybe 6 or 7 times with lots of .'s in between)
...
Sorry, could not find valid secondary superblock
Exiting now.
This looks scary. I never knew there was ACTUALLY magic working behind the scenes...

EDIT: Just to be sure, this means I need to "If xfs_repair failed in phase 1, you must restore lost files from backups."?

EDIT EDIT: It doesn't feel right to me that I need to restore from backup when I can access the files while the device is mounted... I can scp files from it and some files do actually delete from it when I try and clear up space in df... If I need to restore from backup, I would appreciate a "teachable moment" being made out of this, I'm very confused...

Last edited by ss1100; 12-07-2017 at 12:16 AM. Reason: Read a little more, added some questions
 
Old 12-07-2017, 03:08 AM   #7
MadeInGermany
Member
 
Registered: Dec 2011
Location: Simplicity
Posts: 605

Rep: Reputation: 282Reputation: 282Reputation: 282
What are you doing? Your file system was full, not damaged.

If you have deleted "busy" files, they are freed when the process terminates. A simple reboot works.
 
1 members found this post helpful.
Old 12-07-2017, 09:14 AM   #8
ss1100
LQ Newbie
 
Registered: Dec 2017
Posts: 10

Original Poster
Rep: Reputation: Disabled
As stated, this issue has persisted multiple reboots. Anymore ideas, or do I need to go to backup?
 
Old 12-07-2017, 09:45 AM   #9
keefaz
LQ Guru
 
Registered: Mar 2004
Distribution: Slackware
Posts: 5,707

Rep: Reputation: 528Reputation: 528Reputation: 528Reputation: 528Reputation: 528Reputation: 528
About xfs_repair, it seams you attempted it on a physical partition (/dev/sda2) while your root filesystem is on /dev/mapper/centos-root which suggests a logical volume (that means it might not have same boundaries). I could be wrong though as I am not disk expert.
I would try ' xfs_repair -n /dev/mapper/centos-root ' instead
 
1 members found this post helpful.
Old 12-07-2017, 10:56 AM   #10
ss1100
LQ Newbie
 
Registered: Dec 2017
Posts: 10

Original Poster
Rep: Reputation: Disabled
What you're saying makes a lot of sense, but gparted isn't aware of the logical partition when booted as liveusb (iso) and I'm not sure how else to be able to run a check on the partition containing / because it's mounted...

Code:
[root@localhost ~]# xfs_repair -n /dev/mapper/centos-root
xfs_repair: /dev/mapper/centos-root contains a mounted and writable filesystem

fatal error -- couldn't initialize XFS library
EDIT: I am attempting to do this again after finding how to boot in single user mode.

Last edited by ss1100; 12-07-2017 at 11:09 AM. Reason: Found some stuff
 
Old 12-07-2017, 11:05 AM   #11
suicidaleggroll
LQ Guru
 
Registered: Nov 2010
Location: Colorado
Distribution: OpenSUSE, CentOS
Posts: 5,569

Rep: Reputation: 2129Reputation: 2129Reputation: 2129Reputation: 2129Reputation: 2129Reputation: 2129Reputation: 2129Reputation: 2129Reputation: 2129Reputation: 2129Reputation: 2129
XFS...I've had nothing but problems with that filesystem. Not this problem specifically, but phantom corruptions, hard reboots trashing the entire filesystem and requiring a full format, repairs almost always unsuccessful, etc. RHEL/CentOS should have NEVER switched to XFS for the default filesystem for v7, it is the most finicky and error-prone FS I've ever used on any OS.

I can't say if a bug in XFS is causing your problem, but if you get a chance, I highly recommend switching to eg: ext4.
 
1 members found this post helpful.
Old 12-07-2017, 11:14 AM   #12
ss1100
LQ Newbie
 
Registered: Dec 2017
Posts: 10

Original Poster
Rep: Reputation: Disabled
Started centos7 in single user mode as per https://ma.ttias.be/boot-in-single-u...ntos-7-rhel-7/:

This looks bad (attached)...
Attached Thumbnails
Click image for larger version

Name:	Capture.PNG
Views:	10
Size:	17.7 KB
ID:	26489  
 
Old 12-07-2017, 11:24 AM   #13
ss1100
LQ Newbie
 
Registered: Dec 2017
Posts: 10

Original Poster
Rep: Reputation: Disabled
I just went to restore from snapshot and then realized that the snapshots depended on the vmdk file still existing after I converted to a vdi (so I could resize it), so I basically have no backup at this point (it's a dev system, so it's not like we lose customer data, but we lose all my work on this vm).

If someone can tell me it's impossible to restore the data at this point, then I'll start rebuilding it, otherwise I want to continue trying to recover the instance if anyone can help?
 
Old 12-07-2017, 01:07 PM   #14
ss1100
LQ Newbie
 
Registered: Dec 2017
Posts: 10

Original Poster
Rep: Reputation: Disabled
I went ahead and nuked the VM and I'm starting over at this point. At least I learned some things.

Thanks everyone for your help!
 
Old 12-07-2017, 01:48 PM   #15
ss1100
LQ Newbie
 
Registered: Dec 2017
Posts: 10

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by suicidaleggroll View Post
XFS...I've had nothing but problems with that filesystem. Not this problem specifically, but phantom corruptions, hard reboots trashing the entire filesystem and requiring a full format, repairs almost always unsuccessful, etc. RHEL/CentOS should have NEVER switched to XFS for the default filesystem for v7, it is the most finicky and error-prone FS I've ever used on any OS.

I can't say if a bug in XFS is causing your problem, but if you get a chance, I highly recommend switching to eg: ext4.
I did create the new VM on ext4 partitions instead
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
WEBCAM: Stickam shows black screen when Cheese shows cam 449 Linux - Software 16 08-04-2009 04:23 AM
free -m shows 2GB used but gnome-system-monitor shows 1GB anjanesh Linux - General 1 03-04-2009 05:49 AM
df shows disk almost full, can't see why mhath Linux - General 6 12-14-2005 09:55 PM
snmp staus shows it running but on trying MRTG, it shows public@ipaddr not giving res swati220781 Linux - Networking 3 07-08-2004 06:32 PM
root partition shows 100% full in RH 7.2 dmcmurray5 Red Hat 2 05-18-2004 04:24 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 07:06 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration