Erroneous 'No space left on device' errors

riemann_noodles · 06-30-2008, 02:41 PM

Yesterday, mysql was unable to execute queries and failed with the error:

Code:

Can't create/write to file '/tmp/#sql_15b4_0.MYI' (Errcode: 28)]

Disk utilities all show plenty of space left on all devices, and permissions are set correctly on /tmp (root:root drwxrwxrwxt). Upon start mysqld gives the error:

Code:

open: No space left on device

I added a my.cnf entry, setting mysql's tmpdir = /var/lib/mysql/tmp.

Strangely, upon start mysqld still gives the "no space" error,
but operates normally.

We are also receiving a "No space left on device" error from arkeia:

Code:

********* 2008/06/30 10:51:16 START OF APPLICATION "arktrans Arkeia Network Backup 7.0.4", LOGLEVEL="20"
>>>>>>>>> LOG VARS RESET USING PREF, LOGLEVEL="10"
2008/06/30 10:51:24 E00320100 arktrans (4854) ark_open_for_write [L133]: Bad end (28: No space left on device)
2008/06/30 10:51:24 E14010420 arktrans (4854) df3_save [L1506]: bad end ("can't open file") (28: No space left on device)
2008/06/30 10:51:24 E14020030 arktrans (4854) df3_newids [L77]: bad end (9: Bad file descriptor)
2008/06/30 10:51:24 E00320100 arktrans (4854) ark_open_for_write [L133]: Bad end (28: No space left on device)
2008/06/30 10:51:24 E14010420 arktrans (4854) df3_save [L1506]: bad end ("can't open file") (28: No space left on device)
2008/06/30 10:51:24 E14020030 arktrans (4854) df3_newids [L77]: bad end (9: Bad file descriptor)
2008/06/30 10:51:24 E00320100 arktrans (4854) ark_open_for_write [L133]: Bad end (28: No space left on device)
2008/06/30 10:51:24 E14010420 arktrans (4854) df3_save [L1506]: bad end ("can't open file") (28: No space left on device)
2008/06/30 10:51:24 E14020030 arktrans (4854) df3_newids [L77]: bad end (9: Bad file descriptor)
2008/06/30 10:51:24 E00320100 arktrans (4854) ark_open_for_write [L133]: Bad end (28: No space left on device)
2008/06/30 10:51:24 E15010030 arktrans (4854) dc3_open [L119]: Bad end for /opt/arkeia/server/dbase/f3bkp/bkmaster (28: No space left on device)
2008/06/30 10:51:24 E15040080 arktrans (4854) do_save_bkm [L316]: Bad end
2008/06/30 10:51:24 E15040080 arktrans (4854) dc3_save_bkm [L497]: Bad end
2008/06/30 10:51:24 E15040150 arktrans (4854) dc3_add_bksid [L581]: Bad end
2008/06/30 10:51:24 E15040180 arktrans (4854) dc3_new_bksid [L633]: Bad end
2008/06/30 10:51:24 E00220020 arktrans (4854) sig_segv: INTERNAL ERROR: Got SEGV: segmentation fault

Kerberos is also giving a "No space left on device" in /tmp/ upon ticket creation.

At this point I am thoroughly perplexed. As I have noted, permissions are correct on /tmp, /var/lib/mysql, and in /opt/arkeia. There is plenty of space on all devices.

Anyone ever seen anything like this?

We are running Debian Etch.

jvj · 06-30-2008, 03:18 PM

I suspect that you're running out of inodes. What is the result of "df -i" ?

riemann_noodles · 06-30-2008, 03:22 PM

It looks like you're right:

Code:

Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1            1251712 1251712       0  100% /
tmpfs                 129512       2  129510    1% /lib/init/rw
udev                  129512     488  129024    1% /dev
tmpfs                 129512       3  129509    1% /dev/shm
/dev/sda6            1251712  148360 1103352   12% /usr
/dev/sda7            35291136   21862 35269274    1% /home
/dev/sda8            35241984   31055 35210929    1% /var

nitehawk · 07-01-2008, 07:03 PM

Quote:

Originally Posted by riemann_noodles

It looks like you're right:

Code:

Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1            1251712 1251712       0  100% /
tmpfs                 129512       2  129510    1% /lib/init/rw
udev                  129512     488  129024    1% /dev
tmpfs                 129512       3  129509    1% /dev/shm
/dev/sda6            1251712  148360 1103352   12% /usr
/dev/sda7            35291136   21862 35269274    1% /home
/dev/sda8            35241984   31055 35210929    1% /var

That's an awful lot of inodes used. Are you sure you don't have a lot of stale temp files sitting around that you can delete?

riemann_noodles · 07-02-2008, 03:15 PM

Unfortunately /tmp is virtually empty. I believe the culprit is arkeia in /opt. Turns out the server was powered down this weekend in the middle of a backup due to a loss of AC in the server room. Either that or a legitimate buildup of an immensely large number of arkeia pointer files.

fsck claims that the disk is clean and the inodes are legit.

nitehawk · 07-02-2008, 05:53 PM

Quote:

Originally Posted by riemann_noodles

Unfortunately /tmp is virtually empty. I believe the culprit is arkeia in /opt. Turns out the server was powered down this weekend in the middle of a backup due to a loss of AC in the server room. Either that or a legitimate buildup of an immensely large number of arkeia pointer files.

fsck claims that the disk is clean and the inodes are legit.

Have you done just a simple "find /opt" to see where all the files are?
I do remember having a script once that was stuck in recursion due to a subdirectory linked to itself.

The find revealed it right away.