Disk Space increase and clearing

Ubunter · 03-21-2012, 09:42 AM

Hello,
I'm using Centos 6.x, and I just detected today that mysql cannot start up due to critical disk quota.
The server is not hardly charged, indeed it's just runing asterisk in realtime, so all it's stored in mysql db.

When i went into the log directory to try to remove some of apache logs, i made a mistake and delete all over the log directory, so then I had to restor it with:

Quote:

find -type f -exec cat /dev/null > {} \;

But, even after deleting all over the /var/log directory, I still having my disk space in 88%, when I do:

Quote:

df -h
S.ficheros Size Used Avail Use% Montado en
rootfs 10G 8,3G 1,2G 88% /
/dev/root 10G 8,3G 1,2G 88% /
none 7,9G 168K 7,9G 1% /dev
/dev/md2 101G 188M 96G 1% /home
tmpfs 7,9G 0 7,9G 0% /dev/shm
/dev/root 10G 8,3G 1,2G 88% /var/named/chroot/etc/named
/dev/root 10G 8,3G 1,2G 88% /var/named/chroot/var/named
/dev/root 10G 8,3G 1,2G 88% /var/named/chroot/etc/named.conf
/dev/root 10G 8,3G 1,2G 88% /var/named/chroot/etc/named.rfc1912.zones
/dev/root 10G 8,3G 1,2G 88% /var/named/chroot/etc/rndc.key
/dev/root 10G 8,3G 1,2G 88% /var/named/chroot/usr/lib64/bind
/dev/root 10G 8,3G 1,2G 88% /var/named/chroot/etc/named.iscdlv.key
/dev/root 10G 8,3G 1,2G 88% /var/named/chroot/etc/named.root.key

I tried also to update the ossec log files, but no advance, why the disk quota still so overloaded, when I have nothing there??

I went also into the logrotate.conf and dicrease the rotate to 2...

I'm not so expert in this issue, sorry for stupid question, but how can i go ahead to look through this, and figure out the disk quota question?

MensaWater · 03-21-2012, 11:55 AM

That's a bit confusing but I'll try to go to what I think your basic issue is: You have space tied up in a filesystem as shown by df -h but NOT show by du -h?

This is commonly caused when a file that is being held "open" by some process is "deleted". What actually happens is that only the file "name" is deleted in such a case. The filesystem holds "inodes" for each file and when you delete the "name" the "inode" is left intact due to the "open". You can try to find what file is "open" by using lsof and looking for a large sized regular file with no name. You can then examine the process ID shown for that file and see if it is safe to stop (kill). If so killing the PID will free the "open" and the "inode" deletion will complete.

You can also simply reboot the server as that will kill all processes which will free the "open" file.

Ubunter · 03-21-2012, 12:05 PM

Thank you for your promt MensaWater, but I already have rebooted the server after deleting the log directory and recover it, I reboot to re-locate the logs and correct everything, but the disk space still the same...

Doing lsof, I get really huge list, but as I see there's nothing as no name...

Regards,

MensaWater · 03-21-2012, 12:35 PM

Odd layout you have there. All of your /dev/root entries are talking about the same filesystem so I'm assuming you're doing the bind mounts for the /var/named stuff but not sure why.

It shows root (/) which is mounted from /dev/root is 88% full.

It shows /home is only 1% full so you have plenty of free space there.

Since you've done the reboot the open file issue I spoke of wouldn't be the problem.

Common things to look for when trying to free space:
/tmp - Files here are "temporary" so can typically be deleted if not in use by running processes. (Running lsof against a file there will tell you if it is currently open.)
/var/tmp - Same thing - often this is a link back to /tmp.
/var/log - System log files such as /var/log/messages, /var/log/maillog etc... are there. Sometimes these can get quite big if something is going on.
Other things in /var as well - depends on setup. For example on RHEL/CentOS/Fedora /var/cache/yum can get quite big and can be cleaned with yum commands.
What kind of logs do you have under /var/named/chroot and its subdirectories. If you've enabled full BIND logging the logs here could get very large.

A good way to narrow down by directory level that I like to do is:
du -sk / |sort -n
That will show you the largest items under / in reverse order so the largest are at the bottom of the list. If for example this then displayed that /var was the largest you could do:
du -sk /var |sort -n
to see what is largest under var. If that showed you log was the largest you could do:
du -sk /var/log |sort -n
and so on... until you found what files to clean.

Another thing to look for is "core" dump files. Doing
find / -name core
will list core files. Run "file" command against each file found and if it shows it to be a core dump it is typically safe to delete. (Be sure to do the "file" command though - some applications these days create directories or files named core that are not simple core dumps and should not be deleted [Oracle e.g.])

Mail files (especially root's) can often become quite large if you're not reading the mail regularly and deleting messages so that would be another thing to check.

Ubunter · 03-21-2012, 01:48 PM

Quote:

Originally Posted by MensaWater

It shows /home is only 1% full so you have plenty of free space there.

Because the home directory it's empty, i don't use it, as the server is running asterisk&a2billing in real time, data are stored in database, and I have a small webs one in magento and other wordpress, so there's nothing to store in home for me...

Quote:

Common things to look for when trying to free space:
/tmp - Files here are "temporary" so can typically be deleted if not in use by running processes. (Running lsof against a file there will tell you if it is currently open.)

Sure, this was first what I did, as the server was stacked, and could do nothing, so first I delete the tmp to go ahead and see what's going there...

Quote:

/var/tmp - Same thing - often this is a link back to /tmp.
/var/log - System log files such as /var/log/messages, /var/log/maillog etc... are there. Sometimes these can get quite big if something is going on.

Done, thanks.

Quote:

Other things in /var as well - depends on setup. For example on RHEL/CentOS/Fedora /var/cache/yum can get quite big and can be cleaned with yum commands.

I have run yum clean-all etc... and as i said, lastly by error, i have deleted all over the /var/log directory with rm -rf *...

Quote:

What kind of logs do you have under /var/named/chroot and its subdirectories. If you've enabled full BIND logging the logs here could get very large.

I have

Quote:

ls -la /var/named/chroot
total 24
drwxr-x--- 6 root named 4096 ene 10 10:13 .
drwxr-x--- 6 root named 4096 ene 10 10:13 ..
drwxr-x--- 2 root named 4096 ene 10 10:13 dev
drwxr-x--- 4 root named 4096 mar 21 19:23 etc
drwxr-xr-x 3 root root 4096 jul 8 2011 usr
drwxr-x--- 6 root named 4096 ene 10 10:13 var

Quote:

A good way to narrow down by directory level that I like to do is:
du -sk / |sort -n

When I do this, I get an error strange message, it's in spanish, no idea why, and strange contain:

Quote:

du -sk / |sort -n
du: ATENCIÓN: Estructura de directorios circular.
Esto quiere decir seguramente que el sistema de ficheros está corrupto.
COMUNÍQUELO AL ADMINISTRADOR DEL SISTEMA.
El siguiente directorio es parte del ciclo:
«/var/named/chroot/var/named»

du: no se puede acceder a «/proc/4408/task/4408/fd/4»: No existe el fichero o el directorio
du: no se puede acceder a «/proc/4408/task/4408/fdinfo/4»: No existe el fichero o el directorio
du: no se puede acceder a «/proc/4408/fd/4»: No existe el fichero o el directorio
du: no se puede acceder a «/proc/4408/fdinfo/4»: No existe el fichero o el directorio
8514804

Quote:

That will show you the largest items under / in reverse order so the largest are at the bottom of the list. If for example this then displayed that /var was the largest you could do:
du -sk /var |sort -n

Here i get similar error:

Quote:

du -sk /var |sort -n
du: ATENCIÓN: Estructura de directorios circular.
Esto quiere decir seguramente que el sistema de ficheros está corrupto.
COMUNÍQUELO AL ADMINISTRADOR DEL SISTEMA.
El siguiente directorio es parte del ciclo:
«/var/named/chroot/var/named»

4947700 /var

Quote:

to see what is largest under var. If that showed you log was the largest you could do:
du -sk /var/log |sort -n

Here i get only this:

Quote:

du -sk /var/log |sort -n
2772 /var/log

Thanks in advance for your help

MensaWater · 03-21-2012, 02:07 PM

Sorry I left out the asterisk in the du command suggestions.

try du -sk /*, du -sk /var/*, du -sk /var/log/* each piped to the sort -n.

Not sure why you're getting Spanish (or Portuguese maybe?). /proc is a pseudo filesystem so you can ignore stuff about that. The other message about /var/named* would appear to be saying it is a circular reference which makes sense given that it is a bind mount of / - I'd probably ignore that as well.

Your original line where you said you "restored" using find doesn't make any sense to me. You can NOT just go and delete and/or truncate all files in a directory. Part of your foreign language output seems to be indicating corruption of some sort. It might be a good idea to boot to single user and do a full fsck of / to make sure it is OK.

Did /var/log get recreated after the reboot you did earlier?

chrism01 · 03-21-2012, 09:24 PM

Can you show us

Code:

cat /etc/fstab

fdisk -l    #lowercase L

Ubunter · 03-22-2012, 02:48 AM

Quote:

Originally Posted by chrism01

Can you show us

Code:

cat /etc/fstab

fdisk -l    #lowercase L

Thanks for your Prompt Cris, below is the result...

Quote:

cat /etc/fstab
/dev/md1 / ext4 errors=remount-ro 0 1
/dev/md2 /home ext4 defaults 0 2
/dev/sda3 none swap defaults 0 0
/dev/sdb3 none swap defaults 0 0
proc /proc proc defaults 0 0
sysfs /sys sysfs defaults 0 0
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts defaults 0 0

Quote:

fdisk -l

Disco /dev/sda: 120.0 GB, 120034123776 bytes
255 heads, 63 sectors/track, 14593 cylinders
Units = cilindros of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0002d25b

Disposit. Inicio Comienzo Fin Bloques Id Sistema
/dev/sda1 * 1 1306 10485760+ fd Linux raid autodetect
/dev/sda2 1306 14528 106204160 fd Linux raid autodetect
/dev/sda3 14528 14593 526304 82 Linux swap / Solaris

Disco /dev/sdb: 120.0 GB, 120034123776 bytes
255 heads, 63 sectors/track, 14593 cylinders
Units = cilindros of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00091352

Disposit. Inicio Comienzo Fin Bloques Id Sistema
/dev/sdb1 * 1 1306 10485760+ fd Linux raid autodetect
/dev/sdb2 1306 14528 106204160 fd Linux raid autodetect
/dev/sdb3 14528 14593 526304 82 Linux swap / Solaris

Disco /dev/md2: 108.8 GB, 108752994304 bytes
2 heads, 4 sectors/track, 26551024 cylinders
Units = cilindros of 8 * 512 = 4096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

El disco /dev/md2 no contiene una tabla de particiones válida

Disco /dev/md1: 10.7 GB, 10737352704 bytes
2 heads, 4 sectors/track, 2621424 cylinders
Units = cilindros of 8 * 512 = 4096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

El disco /dev/md1 no contiene una tabla de particiones válida

Regards,

chrism01 · 03-23-2012, 12:29 AM

I'm with MensaWater, it looks really funky (err I mean odd).
You seem to have bind-mounted lots, inc 2 copies of
/dev/root 10G 8,3G 1,2G 88% /var/named/chroot/etc/named
/dev/root 10G 8,3G 1,2G 88% /var/named/chroot/var/named

Code:

cat /etc/mtab

I'd recommend backing everything up, then doing a reboot & fsck, like he said

Ubunter · 03-23-2012, 03:13 AM

Quote:

Originally Posted by chrism01

I'm with MensaWater, it looks really funky (err I mean odd).
You seem to ahve bind-mounted lots, inc 2 copies of
/dev/root 10G 8,3G 1,2G 88% /var/named/chroot/etc/named
/dev/root 10G 8,3G 1,2G 88% /var/named/chroot/var/named

Sorry, I'm not expert in this issues, but this is due to the soft raid, no??

Code:

cat /etc/mtab

When I do this, i have a large list of errors, see below:

Quote:

cat /etc/mtab
rootfs / rootfs rw 0 0
/dev/root / ext4 rw,relatime,errors=remount-ro,barrier=1,data=ordered 0 0
/proc /proc proc rw,relatime 0 0
/sys /sys sysfs rw,relatime 0 0
none /dev devtmpfs rw,relatime,size=8184740k,nr_inodes=2046185,mode=755 0 0
devpts /dev/pts devpts rw,relatime,mode=600 0 0
/dev/md2 /home ext4 rw,relatime,barrier=1,data=ordered 0 0
tmpfs /dev/shm tmpfs rw,relatime 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0
/dev/root /var/named/chroot/etc/named ext4 rw,relatime,errors=remount-ro,barrier=1,data=ordered 0 0
/dev/root /var/named/chroot/var/named ext4 rw,relatime,errors=remount-ro,barrier=1,data=ordered 0 0
/dev/root /var/named/chroot/etc/named.conf ext4 rw,relatime,errors=remount-ro,barrier=1,data=ordered 0 0
/dev/root /var/named/chroot/etc/named.rfc1912.zones ext4 rw,relatime,errors=remount-ro,barrier=1,data=ordered 0 0
/dev/root /var/named/chroot/etc/rndc.key ext4 rw,relatime,errors=remount-ro,barrier=1,data=ordered 0 0
/dev/root /var/named/chroot/usr/lib64/bind ext4 rw,relatime,errors=remount-ro,barrier=1,data=ordered 0 0
/dev/root /var/named/chroot/etc/named.iscdlv.key ext4 rw,relatime,errors=remount-ro,barrier=1,data=ordered 0 0
/dev/root /var/named/chroot/etc/named.root.key ext4 rw,relatime,errors=remount-ro,barrier=1,data=ordered 0 0

I'd recommend backing everything up, then doing a reboot & fsck, like he said[/QUOTE]

But when i do the fsck, it's affecting my file system, should I re-reinstall the server, or backup my files in this case? Because I get this:

Quote:

fsck
fsck from util-linux-ng 2.17.2
e2fsck 1.41.12 (17-May-2010)
/dev/md1 está montado.

WARNING!!! The filesystem is mounted. If you continue you ***WILL***
cause ***SEVERE*** filesystem damage.

¿De verdad quiere continuar? (s/n)? no

revisión terminada.
e2fsck 1.41.12 (17-May-2010)
/dev/md2 está montado.

WARNING!!! The filesystem is mounted. If you continue you ***WILL***
cause ***SEVERE*** filesystem damage.

Thanks again,

MensaWater · 03-23-2012, 06:54 AM

The word "errors" in mtab is part of options on the mounts - it is not saying you have actual errors in the cat.

I would strongly suggest doing a backup before proceeding if possible. This is your root filesystem so if it becomes unusable you're going to lose everything.

You should generally NOT try to fsck a mounted filesystem. If you look at my recommendation I didn't say to just run fsck I said:

Quote:

It might be a good idea to boot to single user and do a full fsck of / to make sure it is OK.

In single user mode / is protected from a lot of things but has to be there and it is usually safe to do an fsck of it in that mode. Your other filesystems won't mount automatically in single user so you can run fsck on them. In your layout the only one besides / that you have to worry about doing fsck of is "/dev/md2 101G 188M 96G 1% /home" - as noted previously the other stuff is either pseudo mounts created at boot time or bind remounts of / itself.

Ubunter · 04-11-2012, 04:24 AM

Thanks for you all, guys, after several days turning, as this issue cause me sever problems in mysql, i had to remove the mysql log, clean the root mails, and run fsck several times, rebooting the server, lastly, now i got only 50% HD load, which i find reasonable...

So, many thanks, for you all again,

Regards,

MensaWater · 04-11-2012, 07:53 AM

Glad you got it fixed. Please go to thread tools and mark this as solved. It helps others with similar problems to find solutions more quickly in web searches.

Ubunter · 07-09-2012, 08:23 AM

Hello people,
I had to reopen this thread, as i got the same behaviour again this morning, within another crazy issue...

After that I found my server hd full, I delete the tmp files, then I reboot.. then, when rebooted, I just run fsck, and I accept all the suggested modification, then I got a request to reboot... After rebooting, the server is not reacheable any more, and no way to bring it up again...

I enter in the server in rescue mode, so I found all over my system files in the lost+found directory, and all over the rest empty, I mean, i found the disk empty just with one single directory as lost+found, so for that the server got stacked and cannot reboot anymore...

Please, any of you got such as behaviour, what can do, we're turning all the day, and this is the unique which we conclude, and I'm thinking just in coping the data from there, and reformat the server, but the mysql dbs, would be losed, and it's absolutly full damage in all...

any idea??

regards,

deepak_message · 07-10-2012, 02:18 AM

Why you run fsck? if you were not sure about the command.

find out any third party revocery software and revocer data first.

Best of luck :-)