I've got a weird problem on a few Linux RH machines. Here it is.

Just a minute ago, I was doing a database dump. I ran out of disk space during the dump and it quit.

I removed the partial dump file using "rm -f" but the disk space was not reclaimed. df -h still shows "0" bytes free. Restarting mysql doesn't help.

I have a similar problem on another machine with the same configuration. It ran out of disk space, so I truncated a table, then it still woudln't reclaim the disk space so I dropped that table, yet the disk space didnt change (though the table contained about 20GB of data).

In both cases, the mySQL data ranges from 5 to 50GB (dump/tables). Not sure if that makes a difference.

I've tried restarting mysql and the servers and it doesn't help. Anyone have any ideas? I'm totally stuck and tired of deleting files I actually need to reclaim space.

Thanks!
Luc

The first thing I would do is run fsck against the partition(s) in question to make sure that the file systems are intact.

-----------------------
Steve Stites

I've never used fsck yet but when I typed it in I got the following warning:

/dev/sda2 is mounted.

WARNING!!! Running e2fsck on a mounted filesystem may cause
SEVERE filesystem damage.

Do you really want to continue (y/n)?

You should run fsck against an unmounted file system. So:

umount /dev/sda2

before you run fsck. If you cannot umount /dev/sda2 then shutdown and boot a live CD. Run fsck from the live CD.

Also it is important to tell fsck what type of file system you are working on. At times fsck cannot figure it out and if fsck guesses wrong about the file system type it will usually make matters much worse. So assuming that /dev/sda2 is an ext3 file system you would want something like:

fsck -N -t ext3 /dev/sda2

-N tells fsck not to actually fix anything, just list what it thinks needs to be fixed. If you decide to go ahead and let fsck fix the file system then use the -y option to turn off fsck asking you for permission for each and every fix.

fsck -y -t ext3 /dev/sda2

If fsck finds orphans and file fragments which fsck does not know what they are then fsck places these files in lost+found and gives them a number for a name. So after you run fsck look in lost+found to see if you have any orphans.

Read the fsck man page before you boot the liveCD. Some liveCDs do not have man pages.

man fsck

--------------------------
Steve Stites

Thanks for the response and explanation.

I will have to run the fsck on the other box but on the first one with the delete file I figured out the problem thanks to help from a friend.

I issued a "head -n 100 dumpfile.sql" command on the dump file to make sure it's all good, but the new mysql dumps everything to one line, so i force quit (ctrl+c) the application. unfortunately it didn't release the lock on the dump file.

running

lsof | grep "dumpfile.sql" showed me that "head" still has a lock on it.
kill -9 pid_of_lock killed the process and released the memory.

The other mysql issue is a bit more complicated and will likely require running fsck to check the partition but i figure I would post a response to this in case anyone has the same issue.

thanks!
luc

Still open files is almost always the cause of space seemingly not being "released". lsof is generally a good way to find out what the go is.