LinuxQuestions.org
Latest LQ Deal: Complete CCNA, CCNP & Red Hat Certification Training Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 02-07-2008, 05:05 AM   #1
rotezecke
LQ Newbie
 
Registered: Feb 2008
Location: Nimbin, Australia
Distribution: Debian, Mint, Mandriva
Posts: 27

Rep: Reputation: 15
root partition full, cannot edit fstab


Hi,
after I added another drive to my box (mandrake 10.0), and after i rewrote fstab, the GUI (gnome) doesnt fire up anymore. It used to automatically log on a user and do its thing (which is ripping mp3 files around the clock, for a community radio station). The box still works, mp3s are being written, but I cannot get passed the gnome login screen
I then hit ctrl alt F1 to get a command prompt. I can logon as root and discovered that my root partition is full. It is so full, I cannot even edit fstab. this is what i did:

memory test

free -m

total 218, used 129, free 88
buffer 47 used 141 free
swap 742, used 0, available 742

this seems to vary, another memory test said 216 used 2 free. but i doubt memory is the problem.

disk space test

df -h

(the following messages may not be fully accurate, as I have to type all this, no copy/paste!)

/dev/ide/host0/bus0/target0/lun0/part1 780M (size) / (mount point) 100% (usage)
(I assume this is hda1)
/dev/ide/host0/bus0/target0/lun0/part8 981M (size) /tmp (mount point)
2% (usage)
/dev/ide/host0/bus0/target0/lun0/part6 3.9G (size) /usr (mount point)
49% (usage)
/dev/ide/host0/bus0/target0/lun0/part7 20G (size) /var (mount point)
56% (usage)
/dev/ide/host0/bus0/target0/lun0/part9 48G (size) /streams (mount
point) 1% (usage)
/dev/ide/host0/bus1/target0/lun0/part1 367G (size) /audiologs (mount
point) 6% (usage)
I mounted the last one, hdc1.
The mount point /audiologs already existed (i deleted all files in there before I mounted the new drive), it used to be hda9. hda9 is now mounted at /streams. hda9 is currently not really in use.

disk usage test 'du'

cd /
du -scx * | sort -n

I thought du -x would only list the files on the partition I am on, but I get to see all partitions. Is there a way to look at only one partition? After all, I want to see what's going on on hda1.

I do not know how to enlarge a partition, not using commands anyway. And I doubt this is the cause of the problem, even though it may be a fix. remember, all I did was mount a new drive and write a new fstab.

I kept a backup of old fstab, but using that i still have the same problem. disk full, cannot load gnome. and i cannot edit fstab, when i try to save in vi it tells me: disk full?

when powering down the box there is one failed:

cannot write to mtab.tmp, disk full!

as far as i know mtab has the information that is printed with the mount command.

mount -l

/dev/ide/host0/bus0/target0/lun0/part1 on / type ext3 (rw)
none on /proc type proc (rw)
none on /proc/dev/usb type usbdevfs (rw)
none on /dev type devfs (rw)
non on /sys type sysfs (rw)
/dev/ide/host0/bus0/target0/lun0/part8 on /tmp type ext3 (rw)
/dev/ide/host0/bus0/target0/lun0/part6 on /usr type ext3 (rw)
/dev/ide/host0/bus0/target0/lun0/part7 on /var type ext3 (rw)
/dev/ide/host0/bus0/target0/lun0/part9 on /streams type ext3 (rw)
/dev/ide/host0/bus1/target0/lun0/part1 on /audiologs type ext3 (rw)
none on /mnt/hd type supermount
(rw,dev=/dev/ide/host0/bus0/target1/lun0/part1,fs=ext2:vfat,--,unmask=0,iocharset=iso8859-15,codepage=850)

this is another hd currently not in use, and not mentioned in fstab. I know nothing about supermount, fact is I didnt put it there. and finally there is my new disk - again!

none on /mnt/hd2 type supermount
(rw,dev=/dev/ide/host0/bus1/target0/lun0/part1,fs=ext2:vfat,--,unmask=0,iocharset=iso8859-15,codepage=850)

this i find bemusing, as hdc1 was last formatted in ext3. during my first mount attempt I accidentally formatted it in ext2. (i dont know how this could have happened as I used the mkfs.ext3 command) anyway, the drive was reformatted and is mounted at /audiologs type ext3 and still there is this supermount ext2:vfat entry.

my questions:
how can i find out what files are clogging up hda1? nothing new was installed.
what is supermount and why is hdc1 listed in mtab/fstab twice? the second entry (supermount ext2:vfat was made by the system, not by me)

I am a total newbie, so I would need some kind of linux for idiots instructions. Thanks heaps. rotezecke
 
Old 02-07-2008, 05:25 AM   #2
Simon Bridge
LQ Guru
 
Registered: Oct 2003
Location: Waiheke NZ
Distribution: Ubuntu
Posts: 9,211

Rep: Reputation: 198Reputation: 198
Well, you probably messed something up when you added the new drive.
You can use a live CD to edit the files.

The current fstab and the result of fdisk -l will help clear things up.
 
Old 02-07-2008, 05:26 AM   #3
oskar
Senior Member
 
Registered: Feb 2006
Location: Austria
Distribution: Ubuntu 12.10
Posts: 1,142

Rep: Reputation: 49
I don't think anybody still suggests dividing your system into a gazillion partitions anymore. A separate boot partition might be a good idea, if you want to be really careful. Everything else is overkill for a desktop PC. You have much too little space assigned to the system. If you have it on a separate partition, and you have about 500 gig, give it at least 10!
I would break up this whole mess altogether, and re-install with a big enough system partition (or move the system partition, which is beyond the scope of this post.

The answer to what is clogging up: everything. You have way too little space for a modern distribution. Delete some files on / that you don't need if you want to get into the gui again.

Last edited by oskar; 02-07-2008 at 05:28 AM.
 
Old 02-07-2008, 05:40 AM   #4
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 15,050

Rep: Reputation: 1915Reputation: 1915Reputation: 1915Reputation: 1915Reputation: 1915Reputation: 1915Reputation: 1915Reputation: 1915Reputation: 1915Reputation: 1915Reputation: 1915
Quote:
Originally Posted by rotezecke View Post
du -scx * | sort -n

I thought du -x would only list the files on the partition I am on, but I get to see all partitions. Is there a way to look at only one partition?
Try it as "du / -x --max-depth=1 | sort -nr"
If it was a "normal" ext3 there is 5% reserved for the root user for precisely this eventuality. Dunno what happened to you - I suspect this setup doesn't qualify as "normal" ...
I thought you might be getting slammed by logs, but not with that layout. Despite what oskar says there can be benefit in multiple partitions - didn't help you though. Gotta agree tho' - that root really is *way* too small; almost looks like a Debian recommendation I saw a while back.
I thought it was stupid then, and is more so today with cheap big disks.
 
Old 02-07-2008, 06:28 AM   #5
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,332
Blog Entries: 55

Rep: Reputation: 3533Reputation: 3533Reputation: 3533Reputation: 3533Reputation: 3533Reputation: 3533Reputation: 3533Reputation: 3533Reputation: 3533Reputation: 3533Reputation: 3533
Quote:
Originally Posted by oskar View Post
I don't think anybody still suggests dividing your system into a gazillion partitions anymore.
If you would have read his post carefully you see only his root partition got filled. Since he has a separate /usr and /tmp there can be only a few things going wrong like misplacing files, core dumps, like that. A two partition setup working for you doesn't mean it works for everybody. The "old" partitioning scheme is in no way obsolete.
 
Old 02-07-2008, 08:58 AM   #6
SlowCoder
Member
 
Registered: Oct 2004
Location: Southeast, U.S.A.
Distribution: Fedora (Desktop), CentOS (Server), Knoppix (Diags)
Posts: 934

Rep: Reputation: 38
Quote:
Originally Posted by oskar View Post
I don't think anybody still suggests dividing your system into a gazillion partitions anymore. A separate boot partition might be a good idea, if you want to be really careful. Everything else is overkill for a desktop PC.
I disagree.

When performing a complete OS reinstall, I find it very helpful to have my /home on a separate partition. I can reinstall my OS on my 20G partition, then remount my /home easily and efficiently. No need to reformat the entire drive.

And it also helps with any file system corruption, since they are generally contained within a single partition.
 
Old 02-07-2008, 10:24 AM   #7
dasy2k1
Member
 
Registered: Oct 2005
Location: 127.0.0.1
Distribution: Ubuntu 12.04 X86_64
Posts: 958

Rep: Reputation: 35
i just have 3 partitions,
/ swap and /home
(ok i have a few more for other distros / partitions, with shared /home and swap)
 
Old 02-07-2008, 11:16 AM   #8
oskar
Senior Member
 
Registered: Feb 2006
Location: Austria
Distribution: Ubuntu 12.10
Posts: 1,142

Rep: Reputation: 49
Quote:
Originally Posted by unSpawn View Post
If you would have read his post carefully you see only his root partition got filled. Since he has a separate /usr and /tmp there can be only a few things going wrong like misplacing files, core dumps, like that. A two partition setup working for you doesn't mean it works for everybody. The "old" partitioning scheme is in no way obsolete.

If you read my post carefully, you would know that I understood what the problem was. I just added that it is not necessary or even beneficial IMHO to have separate var usr and tmp partitions, especially if you don't know what your requirements are. If you know what you are doing, and why you are doing it, well then... - who am I to tell you otherwise.
But I think its a bad recommendation to new users, especially when it says that 1 gig is enough for / on a modern computer with a normal sized desktop distribution.
 
Old 02-07-2008, 11:25 AM   #9
oskar
Senior Member
 
Registered: Feb 2006
Location: Austria
Distribution: Ubuntu 12.10
Posts: 1,142

Rep: Reputation: 49
Quote:
Originally Posted by SlowCoder View Post
I disagree.

When performing a complete OS reinstall, I find it very helpful to have my /home on a separate partition. I can reinstall my OS on my 20G partition, then remount my /home easily and efficiently. No need to reformat the entire drive.

And it also helps with any file system corruption, since they are generally contained within a single partition.
Yes... but, for example: my / - system and programs takes up about 25 gig, so If you were to tell someone 20 is enough for / they might have to extend it too at some point. That's why a single / partition is a good idea for new users in my opinion.
You don't need to tell me what separate partitions are for. I know.
 
Old 02-07-2008, 04:02 PM   #10
SlowCoder
Member
 
Registered: Oct 2004
Location: Southeast, U.S.A.
Distribution: Fedora (Desktop), CentOS (Server), Knoppix (Diags)
Posts: 934

Rep: Reputation: 38
Quote:
Originally Posted by oskar View Post
Yes... but, for example: my / - system and programs takes up about 25 gig, so If you were to tell someone 20 is enough for / they might have to extend it too at some point.
Not that we're comparing the size of our systems ... But my / is only at 4.7G, and that includes everything but /home. My /home partition has 79G data. However, you are right in the opinion that a size that's good for some may not be good for others. But that's what we've got GParted for, right?

Quote:
Originally Posted by oskar View Post
That's why a single / partition is a good idea for new users in my opinion.
Ok, but that's not what your post said. You didn't specify new users.

Quote:
Originally Posted by oskar View Post
You don't need to tell me what separate partitions are for. I know.
No offense intended.

Last edited by SlowCoder; 02-07-2008 at 04:04 PM.
 
Old 02-08-2008, 12:37 AM   #11
rotezecke
LQ Newbie
 
Registered: Feb 2008
Location: Nimbin, Australia
Distribution: Debian, Mint, Mandriva
Posts: 27

Original Poster
Rep: Reputation: 15
Thanks for the interest in this post. I didnt want to start a discussion on partitioning strategies though. i am aware that / is too small, the system was set up years ago, by someone else, when disks were more pricy. however, until last week it all worked fine. the box serves as a member server, only ripping files and sharing them as read only over our network.

I moved a few files, not many, as I am not sure which ones I shouldnt touch.

mtab can now be written to during shutdown, and fstab can be edited.

gnome still does not want to come up.

memory is pretty full, and / still chockablock full. (during startup, partmon fails because of 0 kb available on /)

here are my fdisk -l results: (without the block size)

Disk /dev/hda: 80 GB.....(ok i skip this as well)

Device boot start end id system
/dev/hda1 * 1 101 83 linux
/dev/hda2 102 9733 5 extended
/dev/hda5 102 202 82 linux swap
/dev/hda6 203 711 83 linux
/dev/hda7 712 3260 83 linux
/dev/hda8 3261 3387 83 linux
/dev/hda9 3388 9733 83 linux

Disk /dev/hdb: 80 GB.....

Device boot start end id system
/dev/hdb1 * 1 9730 83 linux

Disk /dev/hdc: 400 GB.....

Device boot start end id system
/dev/hdc1 * 1 48641 83 linux

I dont get the hda2 entry. is this the primary vs logical partitioning?

fstab looks like this:

none /proc proc defaults 0 0
/dev/hda1 / ext3 defaults 1 1
/dev/hda8 /tmp ext3 defaults 1 2
/dev/hda6 /usr ext3 defaults 1 2
/dev/hda7 /var ext3 defaults 1 2
/dev/hda9 /streams ext3 defaults 0 0
/dev/hda5 swap swap defaults 0 0
/dev/hdc1 /audiologs ext3 defaults 1 2
/dev/hdd /mnt/cdrom0 iso9660 ro,user,noauto 0 0
/dev/fd0 /mnt/floppy0 auto rw,user,noauto 0 0
none /mnt/hd supermount dev=/dev/ide/host0/bus0/target1/lun0/part1,fs=ext2:vfat,--,unmask=0,iocharset=iso8859-15,kudzu,codepage=850 0 0
none /mnt/hd2 supermount dev=/dev/ide/host0/bus1/target0/lun0/part1,fs=ext2:vfat,--,unmask=0,iocharset=iso8859-15,kudzu,codepage=850 0 0

mind you there may be typoes. the last two entries came automatically. something seems to be fishy with my hdc1. how can I see how it is formatted? but that does not explain the sudden space problem.

Thanks for your input
 
Old 02-08-2008, 05:58 AM   #12
Simon Bridge
LQ Guru
 
Registered: Oct 2003
Location: Waiheke NZ
Distribution: Ubuntu
Posts: 9,211

Rep: Reputation: 198Reputation: 198
Technically, the correct action is to backup the data (just remove hdc, probably) and do a clean install of something better configured. Make sure you document the install, all your maintenance, and how you intend everything to work (which is what the last guy was supposed to do). Stay as close as feasible to the default installation. That way you will understand everything in there, and the next guy will too. (This sort of thing is why people change back to windows servers.)

That's a long-term solution.

So you added hdc1 to /audiologs (presumably to store ripped files). The partition that was audiologs, is not doing something else... you said, basically nothing right now? Remove it from fstab.

Now you can access the drive, there are things you can do...

You need to check to see which directory is hogging memory - check for large logs, tempfiles, whatever. Get rid of them. Move the biggest directories to their own partitions.

[rant severity="mild"]
I try not to be prejudiced in tech, particularly software, but I have never felt good about supermount. IMO: it creates problems later. I usually advise against using ReiserFS and LVM for similar reasons... all these things are good. They can create a smoother user experience and optimize... stuff. Trouble usually comes down the road, like when a new sysadmin has to change something, or something stops working.

Usually a user or an admin goes through a perfectly legit process, like you did, and ends with a broken system. The symptoms are usually weird, and really hard to troubleshoot properly via text. Identifying the change from expectation is normally the first step... in this case: supermount.

Next step is to learn all about supermount, LVM, whatever - figure out how you are supposed to add new drives and so on with the thing. This will tell you what you did.

http://mandrivausers.org/old_docs/ad...nt2.html#super
... in your case, supermount is not supposed to affect non-removable media. That's not what it is for. The two supermount entries are concerned with usb sticks and floppies. The idea is that you don't need to manually mount them. The optical drive is not included because you need to be able to play audio from CDs. So you probably have some other system to handle mounting data CDs - probably, you have to mount manually.

This leads us right back to eyeballing the root partition directories or following the "technically correct" solution. I know which one I'd do.
[/rant]

Of course the lesson here is - if you are a sysadmin: plan for the next guy. Document everything and make sure your system scales. Make sure that incoming admins can figure out the stuff the company relies on. Make sure it scales. Do not assume this person will check everything first.
- if you are an incoming admin, try to talk to the old one. Read the files. Make sure you understand the parts of the system the company relies on. Ask yourself what needs doing if you have some routine maintenance - what if a HDD dies suddenly? Check your assumptions: do not assume a standard setup: linux geeks like to tweak.

<pant - gasp>
I need a drink.
 
Old 02-10-2008, 12:50 AM   #13
Skull, Temple of
LQ Newbie
 
Registered: Feb 2008
Location: Nimbin
Distribution: Debian 4.0r2
Posts: 1

Rep: Reputation: 0
Cool

I am presuming you wanted to put in a storage disk and map it as an existing folder (off the / partition), so you don't have to change the program that writes the mp3's (the logger). Were ALL of the following steps followed?

The old folder you are mapping it to off / has to be empty, which means while adding the disk and transferring the contents of the existing folder, the new disk should be temporarily called (and mapped as) something else (like /audiologs-new). Don't map its permanent fstab entry until the old folder is empty (and contents transferred to the new folder/disk).

You'd have to temporarily stop the logger while doing this, and make sure it doesn't autostart before you're finished. If you have to reboot, stop the logger and clear any files it makes until its permanent folder is ready.

If any of those steps were missed, that could be the cause. On a multi partitioned disk, the / partition is usually small (less than 500MB even on big disks) and only holds the boot sector and the guts of system files. All other folders that appear to come off / are physically empty and mapped to other partitions (or whole disks) in fstab. The folder has to be empty to be mapped to another partition.
 
Old 02-13-2008, 01:05 AM   #14
rotezecke
LQ Newbie
 
Registered: Feb 2008
Location: Nimbin, Australia
Distribution: Debian, Mint, Mandriva
Posts: 27

Original Poster
Rep: Reputation: 15
Thanks guys,
the problem is now fixed. i mounted the new harddrive in a directory that wasnt empty. I didnt stop a process that tried writing to the directory. linux then shoved all the data into the root partition, but
du -xs /
didnt show it, but
df -h
certainly did.
I umounted the new drive, sure the old data reappeared, was subsequently deleted, and that was all.
thanks, again. it was certainly a lesson.
and by the way, the root partition is only 13 % used. (of 780M)
 
Old 02-13-2008, 01:18 AM   #15
jiml8
Senior Member
 
Registered: Sep 2003
Posts: 3,171

Rep: Reputation: 115Reputation: 115
As I read the post, "something happened".

Some time ago, I was having a problem with the partition that contained / filling up. I detected the problem before it became fatal, and did this and that, killing old logs etc to make more space, but it didn't solve the problem and / was filling up.

So I went searching with the find command, and located a subdirectory under /var that had literally millions of entries in it. At this time, I don't recall all the details, but it seems that a cron job was running once a minute, failing, and emailing a warning to root of the failure. The email was going via postfix, which was running on the system but which had never been set up to actually forward mail.

The result was that I had a mail queue in /var that grew, and grew, and grew. Each file was small, but there got to be a LOT of files, so many that I didn't have enough RAM to open the directory and look.

I wound up deleting the entire directory, then recreating it. I tracked down the cron and either fixed or killed it (don't recall) and the problem went away.

Moral of the story: if / is suddenly full, on a system that has been doing one job over and over for a long time, then go looking for a directory that has a huge number of entries, or a file that grows without limit. There very well could be something going on that you just don't know about.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
root partition full laan97ac Mandriva 6 07-23-2007 02:09 PM
root partition full, not sure how; where are my files? lefty.crupps MEPIS 10 01-31-2006 06:21 PM
Root partition full juleslinux Red Hat 4 01-12-2006 11:00 AM
root ('/') partition is FULL inon^ Linux - Hardware 2 08-08-2005 02:19 PM
Root partition is full, what now? Joe Soap Linux - Newbie 30 03-31-2005 12:38 PM


All times are GMT -5. The time now is 07:58 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration