[SOLVED] lost disk space

darkmanta · 08-15-2010, 06:24 AM

Quote:

Originally Posted by joec@home

Simple answer, the system will not report virtual open files, but df will. The following command will show the open files. Trace them back to the service that has them open and then stop and start the service. Do nor restart the service, but actually stop and then start them.

lsof |grep deleted

your suggestion:

Quote:

> lsof |grep deleted
mysqld 2331 mysql 4u REG 9,0 0 15418658 /tmp/ibfVpzes (deleted)
mysqld 2331 mysql 5u REG 9,0 21 15418659 /tmp/ibPtgim3 (deleted)
mysqld 2331 mysql 6u REG 9,0 0 15418660 /tmp/ibh1l1tE (deleted)
mysqld 2331 mysql 7u REG 9,0 0 15418662 /tmp/ibexN2Df (deleted)
mysqld 2331 mysql 11u REG 9,0 0 15418663 /tmp/ib7sNvGR (deleted)
gdm-binar 3062 root 10u REG 9,0 15000 15418760 /tmp/.pk11ipc1/coolkeypk11sE-Gate 0 0-0 (deleted)
gdm-binar 3134 root 10u REG 9,0 15000 15418760 /tmp/.pk11ipc1/coolkeypk11sE-Gate 0 0-0 (deleted)

After a mysql stop-start nothing happens:

Quote:

> lsof |grep deleted
gdm-binar 3062 root 10u REG 9,0 15000 15418760 /tmp/.pk11ipc1/coolkeypk11sE-Gate 0 0-0 (deleted)
gdm-binar 3134 root 10u REG 9,0 15000 15418760 /tmp/.pk11ipc1/coolkeypk11sE-Gate 0 0-0 (deleted)
mysqld 11908 mysql 4u REG 9,0 0 15418659 /tmp/ibLa4I4x (deleted)
mysqld 11908 mysql 5u REG 9,0 0 15418660 /tmp/ibwUre3u (deleted)
mysqld 11908 mysql 6u REG 9,0 0 15418661 /tmp/ib4s6J1r (deleted)
mysqld 11908 mysql 7u REG 9,0 0 15418662 /tmp/ibzvVI0o (deleted)
mysqld 11908 mysql 11u REG 9,0 0 15418663 /tmp/ib38SWOm (deleted)

and then:

Quote:

> df -h
Filesystem Size Used Avail Use% Mounted on
/dev/md0 95G 95G 0 100% /
tmpfs 940M 0 940M 0% /dev/shm
/dev/md2 51G 31G 18G 65% /home
/dev/md1 142G 2.8G 132G 3% /var

So GB of deleted files have been really deleted before... what is happening? I've never seen this king of problem!
Please help! Thank you.

jys17 · 08-17-2010, 04:20 PM

I'd do a full stop of mysqld, then run a lsof | grep deleted. That second column I believe are the process ID's, so while mysqld is stopped, do a ps and grep for those process ID's, to ensure they are no longer running, if the processes are still running you'll need to find out why, and either stop them or kill them.

darkmanta · 08-19-2010, 08:31 AM

Thank you jys17.

As told before, I don't think the problem is in deleted files in /tmp locked by a running program

before mysql stop

Quote:

> lsof |grep deleted
gdm-binar 3062 root 10u REG 9,0 15000 15418760 /tmp/.pk11ipc1/coolkeypk11sE-Gate 0 0-0 (deleted)
gdm-binar 3134 root 10u REG 9,0 15000 15418760 /tmp/.pk11ipc1/coolkeypk11sE-Gate 0 0-0 (deleted)
mysqld 11908 mysql 4u REG 9,0 0 15418659 /tmp/ibLa4I4x (deleted)
mysqld 11908 mysql 5u REG 9,0 0 15418660 /tmp/ibwUre3u (deleted)
mysqld 11908 mysql 6u REG 9,0 0 15418661 /tmp/ib4s6J1r (deleted)
mysqld 11908 mysql 7u REG 9,0 0 15418662 /tmp/ibzvVI0o (deleted)
mysqld 11908 mysql 11u REG 9,0 0 15418663 /tmp/ib38SWOm (deleted)

mysql stop

Quote:

> lsof |grep deleted
gdm-binar 3062 root 10u REG 9,0 15000 15418760 /tmp/.pk11ipc1/coolkeypk11sE-Gate 0 0-0 (deleted)
gdm-binar 3134 root 10u REG 9,0 15000 15418760 /tmp/.pk11ipc1/coolkeypk11sE-Gate 0 0-0 (deleted)
sh 13990 root 0r REG 9,1 2596 33816713 /var/spool/at/a02914014615e8 (deleted)
sh 13991 root 0r REG 9,0 72 15418698 /tmp/sh-thd-1282234010 (deleted)
clean_gra 13992 root 0r REG 9,0 72 15418698 /tmp/sh-thd-1282234010 (deleted)

mysql start

Quote:

> lsof |grep deleted
gdm-binar 3062 root 10u REG 9,0 15000 15418760 /tmp/.pk11ipc1/coolkeypk11sE-Gate 0 0-0 (deleted)
gdm-binar 3134 root 10u REG 9,0 15000 15418760 /tmp/.pk11ipc1/coolkeypk11sE-Gate 0 0-0 (deleted)
mysqld 14353 mysql 4u REG 9,0 0 15418661 /tmp/ibmXcNU3 (deleted)
mysqld 14353 mysql 5u REG 9,0 0 15418662 /tmp/ibWXu7eP (deleted)
mysqld 14353 mysql 6u REG 9,0 0 15418663 /tmp/ibHz5rzA (deleted)
mysqld 14353 mysql 7u REG 9,0 0 15418697 /tmp/ibgqRfUl (deleted)
mysqld 14353 mysql 11u REG 9,0 0 15418698 /tmp/ib9IRDI7 (deleted)

the full /tmp size is still 530Kb and the problem is still there, eve if I deleted 28Gb of backup files from /backup

Quote:

> df -h
Filesystem Size Used Avail Use% Mounted on
/dev/md0 95G 95G 0 100% /
tmpfs 940M 0 940M 0% /dev/shm
/dev/md2 51G 31G 18G 65% /home
/dev/md1 142G 2.8G 132G 3% /var

The problem is somewhere in the fs, I think

joec@home · 08-19-2010, 04:50 PM

With that much information you may be correct about an error in the file system. Have you tried running a manual fsck on the drives?

Quote:

http://www.cyberciti.biz/faq/linux-f...boot-sequence/

Frce fsck on next boot using shutdown command

The -F option force fsck on reboot, login as root and type the following command to reboot and run fsck:
# shutdown -rF now

darkmanta · 08-31-2010, 04:51 PM

Quote:

Originally Posted by joec@home

With that much information you may be correct about an error in the file system. Have you tried running a manual fsck on the drives?

Yes I did a

Quote:

touch /forcefsck

and reboot. The file-systems where checked with no error reported. But the problem is still there.

Quote:

> df -h
Filesystem Size Used Avail Use% Mounted on
/dev/md0 95G 95G 0 100% /

and

Quote:

Device file /dev/md0
RAID level Mirrored (RAID1)
Filesystem status Mounted on /
Usable size 102398208 blocks (97.65 GB)
Persistent superblock? Si
Chunk size Predefinito
RAID status clean
Partitions in RAID SATA device A partition 1
SATA device B partition 1

So my server suffers from many disfunctions because /tmp is so more writeable.

This is the first time I can't resolve a problem, even with the precious help of the forum people!

stevephillips79 · 09-02-2010, 12:55 AM

right click on your partition/properties/tools/check now

joec@home · 09-02-2010, 09:14 PM

The files sizes associated to the gdm-binar are also quite large. What happens when you shut down that process? You may have to back trace the command that started gdm-binar. So for example using "ps -elf" the 4th column is the process ID, and the 5th column is the parent process ID that called the process.

Wim Sturkenboom · 09-03-2010, 09:13 AM

If I'm not mistaken, your problem might be that a mounted partition hides data from you.

e.g. If you have 40GB in the directory /mnt and you mount an external HD on it, you will never see that 40GB, but df will still report it.

This can e.g. happen when you forget to mount and next make a backup on /mnt; or if you copied your home directory to a partition and forgot to clear the home directory before mounting on the home partition on /home.

darkmanta · 09-09-2010, 11:13 AM

Quote:

Originally Posted by joec@home

The files sizes associated to the gdm-binar are also quite large. What happens when you shut down that process? You may have to back trace the command that started gdm-binar. So for example using "ps -elf" the 4th column is the process ID, and the 5th column is the parent process ID that called the process.

thank you everybody.

I terminated gdm and nothing changed. Also renamed the gdm command so that it's impossible to startup again.
And I think that the file size of gdm is 15000.

Quote:

If I'm not mistaken, your problem might be that a mounted partition hides data from you.

Nope, I cleanly unmounted anything under /mnt and the result is still the same. I'm sure that I deleted many Gb of files without any change.

Wim Sturkenboom · 09-10-2010, 12:09 AM

Can you post output of mount command?

darkmanta · 12-21-2010, 04:10 PM

After many days of unresolved problems I reinstalled a new Debian system. This sysem seemed to work very well until I experienced the same problem than before: a growing filesystem occupation percent without evidence of new files or growing directories.

But...

Quote:

Originally Posted by Wim Sturkenboom

If I'm not mistaken, your problem might be that a mounted partition hides data from you.

e.g. If you have 40GB in the directory /mnt and you mount an external HD on it, you will never see that 40GB, but df will still report it.

This can e.g. happen when you forget to mount and next make a backup on /mnt; or if you copied your home directory to a partition and forgot to clear the home directory before mounting on the home partition on /home.

GREAT! That is! I got a failure in a mounted external directory under /mnt/condivision so for many days many backups went in the unmounted directory. And when I remounted the external directory all the files in the unmounted real directory where hidden.

So thank you very much for your suggestion!

Wim Sturkenboom · 01-24-2011, 02:40 AM

Love it when a plan comes together