LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Server (https://www.linuxquestions.org/questions/linux-server-73/)
-   -   Erroneous 'No space left on device' errors (https://www.linuxquestions.org/questions/linux-server-73/erroneous-no-space-left-on-device-errors-652632/)

riemann_noodles 06-30-2008 02:41 PM
Erroneous 'No space left on device' errors
 
Yesterday, mysql was unable to execute queries and failed with the error:
Code:
Can't create/write to file '/tmp/#sql_15b4_0.MYI' (Errcode: 28)]
Disk utilities all show plenty of space left on all devices, and permissions are set correctly on /tmp (root:root drwxrwxrwxt). Upon start mysqld gives the error:
Code:
open: No space left on device
I added a my.cnf entry, setting mysql's tmpdir = /var/lib/mysql/tmp.

Strangely, upon start mysqld still gives the "no space" error,
but operates normally.

We are also receiving a "No space left on device" error from arkeia:
Code:
********* 2008/06/30 10:51:16 START OF APPLICATION "arktrans Arkeia Network Backup 7.0.4", LOGLEVEL="20"
>>>>>>>>> LOG VARS RESET USING PREF, LOGLEVEL="10"
2008/06/30 10:51:24 E00320100 arktrans (4854) ark_open_for_write [L133]: Bad end (28: No space left on device)
2008/06/30 10:51:24 E14010420 arktrans (4854) df3_save [L1506]: bad end ("can't open file") (28: No space left on device)
2008/06/30 10:51:24 E14020030 arktrans (4854) df3_newids [L77]: bad end (9: Bad file descriptor)
2008/06/30 10:51:24 E00320100 arktrans (4854) ark_open_for_write [L133]: Bad end (28: No space left on device)
2008/06/30 10:51:24 E14010420 arktrans (4854) df3_save [L1506]: bad end ("can't open file") (28: No space left on device)
2008/06/30 10:51:24 E14020030 arktrans (4854) df3_newids [L77]: bad end (9: Bad file descriptor)
2008/06/30 10:51:24 E00320100 arktrans (4854) ark_open_for_write [L133]: Bad end (28: No space left on device)
2008/06/30 10:51:24 E14010420 arktrans (4854) df3_save [L1506]: bad end ("can't open file") (28: No space left on device)
2008/06/30 10:51:24 E14020030 arktrans (4854) df3_newids [L77]: bad end (9: Bad file descriptor)
2008/06/30 10:51:24 E00320100 arktrans (4854) ark_open_for_write [L133]: Bad end (28: No space left on device)
2008/06/30 10:51:24 E15010030 arktrans (4854) dc3_open [L119]: Bad end for /opt/arkeia/server/dbase/f3bkp/bkmaster (28: No space left on device)
2008/06/30 10:51:24 E15040080 arktrans (4854) do_save_bkm [L316]: Bad end
2008/06/30 10:51:24 E15040080 arktrans (4854) dc3_save_bkm [L497]: Bad end
2008/06/30 10:51:24 E15040150 arktrans (4854) dc3_add_bksid [L581]: Bad end
2008/06/30 10:51:24 E15040180 arktrans (4854) dc3_new_bksid [L633]: Bad end
2008/06/30 10:51:24 E00220020 arktrans (4854) sig_segv: INTERNAL ERROR: Got SEGV: segmentation fault
Kerberos is also giving a "No space left on device" in /tmp/ upon ticket creation.

At this point I am thoroughly perplexed. As I have noted, permissions are correct on /tmp, /var/lib/mysql, and in /opt/arkeia. There is plenty of space on all devices.

Anyone ever seen anything like this?

We are running Debian Etch.

jvj 06-30-2008 03:18 PM
I suspect that you're running out of inodes. What is the result of "df -i" ?

riemann_noodles 06-30-2008 03:22 PM
It looks like you're right:

Code:
Filesystem            Inodes  IUsed  IFree IUse% Mounted on
/dev/sda1            1251712 1251712      0  100% /
tmpfs                129512      2  129510    1% /lib/init/rw
udev                  129512    488  129024    1% /dev
tmpfs                129512      3  129509    1% /dev/shm
/dev/sda6            1251712  148360 1103352  12% /usr
/dev/sda7            35291136  21862 35269274    1% /home
/dev/sda8            35241984  31055 35210929    1% /var

nitehawk 07-01-2008 07:03 PM
Quote:
Originally Posted by riemann_noodles (Post 3199579)
It looks like you're right:

Code:
Filesystem            Inodes  IUsed  IFree IUse% Mounted on
/dev/sda1            1251712 1251712      0  100% /
tmpfs                129512      2  129510    1% /lib/init/rw
udev                  129512    488  129024    1% /dev
tmpfs                129512      3  129509    1% /dev/shm
/dev/sda6            1251712  148360 1103352  12% /usr
/dev/sda7            35291136  21862 35269274    1% /home
/dev/sda8            35241984  31055 35210929    1% /var
That's an awful lot of inodes used. Are you sure you don't have a lot of stale temp files sitting around that you can delete?

riemann_noodles 07-02-2008 03:15 PM
Unfortunately /tmp is virtually empty. I believe the culprit is arkeia in /opt. Turns out the server was powered down this weekend in the middle of a backup due to a loss of AC in the server room. Either that or a legitimate buildup of an immensely large number of arkeia pointer files.

fsck claims that the disk is clean and the inodes are legit.

nitehawk 07-02-2008 05:53 PM
Quote:
Originally Posted by riemann_noodles (Post 3201967)
Unfortunately /tmp is virtually empty. I believe the culprit is arkeia in /opt. Turns out the server was powered down this weekend in the middle of a backup due to a loss of AC in the server room. Either that or a legitimate buildup of an immensely large number of arkeia pointer files.

fsck claims that the disk is clean and the inodes are legit.

Have you done just a simple "find /opt" to see where all the files are?
I do remember having a script once that was stuck in recursion due to a subdirectory linked to itself. :o The find revealed it right away.


All times are GMT -5. The time now is 04:53 PM.