LinuxQuestions.org
Support LQ: Use code LQ3 and save $3 on Domain Registration
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices

Reply
 
Search this Thread
Old 06-30-2008, 02:41 PM   #1
riemann_noodles
Member
 
Registered: Feb 2005
Location: San Diego
Posts: 43

Rep: Reputation: 15
Erroneous 'No space left on device' errors


Yesterday, mysql was unable to execute queries and failed with the error:
Code:
Can't create/write to file '/tmp/#sql_15b4_0.MYI' (Errcode: 28)]
Disk utilities all show plenty of space left on all devices, and permissions are set correctly on /tmp (root:root drwxrwxrwxt). Upon start mysqld gives the error:
Code:
open: No space left on device
I added a my.cnf entry, setting mysql's tmpdir = /var/lib/mysql/tmp.

Strangely, upon start mysqld still gives the "no space" error,
but operates normally.

We are also receiving a "No space left on device" error from arkeia:
Code:
********* 2008/06/30 10:51:16 START OF APPLICATION "arktrans Arkeia Network Backup 7.0.4", LOGLEVEL="20"
>>>>>>>>> LOG VARS RESET USING PREF, LOGLEVEL="10"
2008/06/30 10:51:24 E00320100 arktrans (4854) ark_open_for_write [L133]: Bad end (28: No space left on device)
2008/06/30 10:51:24 E14010420 arktrans (4854) df3_save [L1506]: bad end ("can't open file") (28: No space left on device)
2008/06/30 10:51:24 E14020030 arktrans (4854) df3_newids [L77]: bad end (9: Bad file descriptor)
2008/06/30 10:51:24 E00320100 arktrans (4854) ark_open_for_write [L133]: Bad end (28: No space left on device)
2008/06/30 10:51:24 E14010420 arktrans (4854) df3_save [L1506]: bad end ("can't open file") (28: No space left on device)
2008/06/30 10:51:24 E14020030 arktrans (4854) df3_newids [L77]: bad end (9: Bad file descriptor)
2008/06/30 10:51:24 E00320100 arktrans (4854) ark_open_for_write [L133]: Bad end (28: No space left on device)
2008/06/30 10:51:24 E14010420 arktrans (4854) df3_save [L1506]: bad end ("can't open file") (28: No space left on device)
2008/06/30 10:51:24 E14020030 arktrans (4854) df3_newids [L77]: bad end (9: Bad file descriptor)
2008/06/30 10:51:24 E00320100 arktrans (4854) ark_open_for_write [L133]: Bad end (28: No space left on device)
2008/06/30 10:51:24 E15010030 arktrans (4854) dc3_open [L119]: Bad end for /opt/arkeia/server/dbase/f3bkp/bkmaster (28: No space left on device)
2008/06/30 10:51:24 E15040080 arktrans (4854) do_save_bkm [L316]: Bad end
2008/06/30 10:51:24 E15040080 arktrans (4854) dc3_save_bkm [L497]: Bad end
2008/06/30 10:51:24 E15040150 arktrans (4854) dc3_add_bksid [L581]: Bad end
2008/06/30 10:51:24 E15040180 arktrans (4854) dc3_new_bksid [L633]: Bad end
2008/06/30 10:51:24 E00220020 arktrans (4854) sig_segv: INTERNAL ERROR: Got SEGV: segmentation fault
Kerberos is also giving a "No space left on device" in /tmp/ upon ticket creation.

At this point I am thoroughly perplexed. As I have noted, permissions are correct on /tmp, /var/lib/mysql, and in /opt/arkeia. There is plenty of space on all devices.

Anyone ever seen anything like this?

We are running Debian Etch.

Last edited by riemann_noodles; 06-30-2008 at 02:42 PM.
 
Old 06-30-2008, 03:18 PM   #2
jvj
LQ Newbie
 
Registered: Apr 2008
Posts: 8

Rep: Reputation: 0
I suspect that you're running out of inodes. What is the result of "df -i" ?
 
Old 06-30-2008, 03:22 PM   #3
riemann_noodles
Member
 
Registered: Feb 2005
Location: San Diego
Posts: 43

Original Poster
Rep: Reputation: 15
It looks like you're right:

Code:
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1            1251712 1251712       0  100% /
tmpfs                 129512       2  129510    1% /lib/init/rw
udev                  129512     488  129024    1% /dev
tmpfs                 129512       3  129509    1% /dev/shm
/dev/sda6            1251712  148360 1103352   12% /usr
/dev/sda7            35291136   21862 35269274    1% /home
/dev/sda8            35241984   31055 35210929    1% /var
 
Old 07-01-2008, 07:03 PM   #4
nitehawk
LQ Newbie
 
Registered: Jun 2008
Distribution: Slackware
Posts: 11

Rep: Reputation: 0
Quote:
Originally Posted by riemann_noodles View Post
It looks like you're right:

Code:
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1            1251712 1251712       0  100% /
tmpfs                 129512       2  129510    1% /lib/init/rw
udev                  129512     488  129024    1% /dev
tmpfs                 129512       3  129509    1% /dev/shm
/dev/sda6            1251712  148360 1103352   12% /usr
/dev/sda7            35291136   21862 35269274    1% /home
/dev/sda8            35241984   31055 35210929    1% /var
That's an awful lot of inodes used. Are you sure you don't have a lot of stale temp files sitting around that you can delete?
 
Old 07-02-2008, 03:15 PM   #5
riemann_noodles
Member
 
Registered: Feb 2005
Location: San Diego
Posts: 43

Original Poster
Rep: Reputation: 15
Unfortunately /tmp is virtually empty. I believe the culprit is arkeia in /opt. Turns out the server was powered down this weekend in the middle of a backup due to a loss of AC in the server room. Either that or a legitimate buildup of an immensely large number of arkeia pointer files.

fsck claims that the disk is clean and the inodes are legit.
 
Old 07-02-2008, 05:53 PM   #6
nitehawk
LQ Newbie
 
Registered: Jun 2008
Distribution: Slackware
Posts: 11

Rep: Reputation: 0
Quote:
Originally Posted by riemann_noodles View Post
Unfortunately /tmp is virtually empty. I believe the culprit is arkeia in /opt. Turns out the server was powered down this weekend in the middle of a backup due to a loss of AC in the server room. Either that or a legitimate buildup of an immensely large number of arkeia pointer files.

fsck claims that the disk is clean and the inodes are legit.

Have you done just a simple "find /opt" to see where all the files are?
I do remember having a script once that was stuck in recursion due to a subdirectory linked to itself. The find revealed it right away.
 
  


Reply

Tags
cache, debian, etch, kerberos, mysql, permissions, tmp


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
No space left on device zmsc1 Linux From Scratch 7 03-28-2009 10:57 PM
no space left on device cjupiter2 Linux - General 2 09-03-2007 07:56 PM
No space left on device even though it has free space? enine Linux - General 8 05-30-2007 04:22 PM
No space left on device clintdavis Linux - Software 2 05-04-2006 10:38 PM
no space left on device, but it has enough!!! help!!! sarah21 Red Hat 3 02-03-2005 08:37 PM


All times are GMT -5. The time now is 04:45 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration