Root system directory is full
 

Hi all,

Hope this is the right place to post this silly question. I was asked to check the available space of a server which is running Fedora Core release 3 (Heidelberg)
LSB_VERSION="1.3" and when I type df -h command, it shows the below info:

Could someone please tell me how to clean this up and what files do I need to delete? Sorry if this is a silly question, I'm very new to this.

Thanks in advance

Hi!

It seems that you still have 1.5GB available on the partition your root is mounted on:

/dev/hda1 4.0G 2.4G 1.5G 62% /

Two good places to look are /tmp and /var/log (although your /tmp doesn't look all the big). Note that /var/log may vary from distribution to distribution -- look for the logs if you don't have /var/log. If you do not have logrotate running, your system logs will be mammoth -- and, yes, you can simply empty them with
Although it doesn't appear that your user directories are in the root, look at the sizes of user directories, probably in /usr. Users do tend to gobble space and there may be some Really Large Files sitting in their directories; you can use find to locate them:
that will give a list of files larger than 10 MB (and then you go holler are folks, suggesting politely that the gzip utility is quite useful).

Hope this helps some.

Thanks tronayne and Bfresh for your replies.

Here is everything in my /var/log directory:

So if I understand correctly, I can just delete the .log files? Active files should not be deleted just truncate it if I'm not mistaken (I was doing a rm -rf for the .1 .2 .3,... files).

I found these under /usr:


But I found a directory /depot/server1/home in which I see directories corresponding to each user. When I ran that command you typed, it was complaining about the format:

Many users didn't have any files in those directories though, maybe I'm looking in the wrong place?

Thanks in advance for your help

After a week or two, yes you can delete the .1, .2., .3, etc. files if you're short on space. You may however want to add one or two to logrotate -- there are some that are being rotated, those with .1, .2, .3, etc. extensions. The messages log is a good candidate (it's just under 20 MB right now) that is not being rotated.

Go look in /etc. There should be a file, logrotate.conf, and a directory, logrotate.d. If you get into /etc/logrotate.d there should be one or more files that are used to rotate the logs every week to keep the individual sizes down. One of them may be named syslog and you can change it to add /var/log/messages so that log file will get rotated.

Bear in mind that your system may vary; see man logrotate to find out the location for these files if you do not immediately see a /etc/logroate.d directory.

Here's what the file /etc/logrorate.d/syslog looks like on my system:
Again, yours may vary and you might need to look around; if your file looks like that, you could simply insert /var/log/messages in the list as above and over the coming weekend it should get rotated and compressed. Don't just copy the above to your system, work with what your system has.

maillog is another good candidate for rotation.

I would leave the files and directories in /usr alone.
Looks like /depot/server/home is where your user directories are? Different, that.

It seems that your find utility just doesn't know about the -size 10M argument, so let's kill the same bird with a different stone. Take a look at the manual page for find, particularly the section on size.

You probably want to find files in /depot/server1/home that are bigger that 1 GB:
that will show you files larger than 10 MB and their actual size (you may get a large list) to discuss with users. If you add an additional zero (+104857600c), you'll get files larger than 100 MB.

Hope this helps some.

tronayne,

Thanks much for your very helpful reply.

This is the info in my syslog:

So looks like the messages are already a part of the rotation.

I found the users who happen to have large files on the network and contacted them all. So you are suggesting if they all clean up the space, I will have a clean root directory? This is the output of df -h:

These are everything I have under the root directory:

Thanks in advance

Well, your root partition is a little on the small side -- you do have 1.5G free, but over time that ain't a whole lot of wiggle room. You may -- may -- want to consider a larger drive and copying or reinstalling your existing system to it (which is not a trivial task). I typically make the root partition at least 10G (and I do not use a separate partition for /tmp); that results in about 7.5G of disk use. However, as long as you keep an eye on free space, you can probably get away with leaving well enough alone.

It's always a good thing (always) to keep an eye on the space users gobble up -- folks do tend to get and hang on to big stuff "just in case I might need it some day" and it never hurts to monitor that. Their use has nothing to do with your root space, but the day will come that somebody's going to fill up a partition and all heck will break loose (trust me: been there, they did that more than once). Also, have a look-see at what's in /tmp -- folks may be stashing big stuff in there and let it sit there forever. Oh, yeah, teach them about the benefits of gzip.

One thing you can do that will save you a little space is look at /etc/logrotate.conf. Mine looks like this (your will probably vary):
Of particular interest are the lines
If yours is like this, uncomment (remove the #) from compress. That will cause the rotated log files to be compressed with gzip; ain't much, but every little bit counts.

You know, disk drives are cheap -- a 500G drive for the system and another 500G drive for users is not a huge investment (and, no, you don't want to use terabyte drives yet; too new for longevity and reliability statistics to be meaningful). You now have a good statistical base to work from (your existing disk use as given by df -h) to give you a good starting place for adding or changing a drive or two.

Hope this helps some.

Hi tronayne,

Thanks again for your helpful reply. I looked at my /etc/logrotate.conf and it reads #compress, so I will remove the # sign.

Since users are using that machine all the time and I have never done anything like this, I'm not sure how I can replace the drive with a bigger one and not sure how to re-partition the existing drive. I know the person who put this box in service said you have to keep cleaning the root directory but he didn't tell me how I can do that. I constantly truncate the active logs and remove the .x files but that doesn't empty much space.

I would really appreciate it if you help me with this

Thanks

http://www.cyberciti.biz/faq/how-do-...sd-filesystem/
du : Disk usage

This shows the disk usage for all the files and directories (folders) in /var:

du -a /var | sort -n -r | head -n 10
1008372 /var
313236 /var/www
253964 /var/log
192544 /var/lib
152628 /var/spool
152508 /var/spool/squid
136524 /var/spool/squid/00
95736 /var/log/mrtg.log
74688 /var/log/squid
62544 /var/cache

You might need to sudo or su in order to see some directories.
Good luck.

Thanks tollingalong,
Here's the output I'm gettting:

1040600 /var
870180 /var/spool
862408 /var/spool/clientmqueue
99196 /var/lib
65952 /var/lib/rpm
45012 /var/lib/rpm/Packages
41332 /var/log
21484 /var/log/cups
16516 /var/www
16100 /var/lib/imap

Hi tronayne and everyone,

I have asked all ths user to move their files from the /name/test2. But the root directory % didn't go down and it even went up. Is there any way someone please help me and tell me what else to delete to empty up free space on the root directory?

This is my biggest fear that one day the root directory gets to 100% and the machine (therefore the LDAP server, DHCP server and everything else) goes down for good. Thank you in advance

Filesystem Size Used Avail Use% Mounted on
/dev/hda1 4.0G 2.7G 1.2G 70% /
/dev/md2 58G 13G 43G 23% /name/test1
/dev/md1 58G 54G 1.3G 98% /name/test2
/dev/md3 58G 33G 23G 59% /name/test3
/dev/md0 48G 37G 7.9G 83% /name/test4
none 506M 0 506M 0% /dev/shm
/dev/hda3 4.0G 41M 3.8G 2% /tmp
/dev/hda4 10G 2.2G 7.4G 23% /usr

Do the following:
du -a /name/test2 | sort -n -r | head -n 10
Please note the da -a <directory> means the directory you wish to find the largest files.

That should tell you which files are largest in the specific location you're in need.
If you cannot find enough files there is always the possibility a process is holding onto a file.

Hiya.

You do realize that the content of /name/test1 (test2, test3 and test4) have nothing to do with what's in the root partition -- they're mounted partitions and have wads of space available.

If I remember correctly, your user's home directories are in /home? That's your best candidate for relocation if that's so -- the size of user's home directories never goes down, only up and it's only a matter of time...

One candidate you have for moving /home is /dev/hda3, a 4.0G partition currently mounted as /tmp which appears to only be using 41M with 3.8G free. To consider doing that, you would want to do this (your size will be different, of course); the -sh shows a summary in human-readable form:
If this sounds good to you, let us know and we can walk you through how to do it; it's not difficult but has to be done step-by-step.

Bottom line, however, is that you're going to have to bite the bullet and add a drive or two and/or redo your system with larger partition sizes sooner or later. You do have time for now but the day will come when things get filled up, sorry.

Hope this helps some.

Thanks tronayne and tollingalong for your replies.

Tronayne,
I ran the df -h command again:

I understand that those drives are not parts of the root partition, but in /name/test2 there are all the users listed. Not sure why the root directory gets full if the users are in a different mounted drive.

The command you asked me to run for /home returned only 6.0M so I don't think it's the correct user directory:

Where else should I be looking for large files?

Thanks in advance

I've attached a file, biggest.sh, that you can use to find... uh, big files.

Download the file then enter
you'll have an executable utility named biggest.

Running it without arguments will show how to use it:
What you can do is
and that will show you large files throughout the file system (you can modify the minimum size with the -s option). You'll need to play with it a little to get a feel for what it's going to show you.

You really do have to realize that you're going to have to bite the bullet and work on reinstalling your system with a larger partition for root (more on the order of 10G or more) -- you don't have to do that as an emergency (hey, 71% is still 30%-ish free, eh), but you will have to do it sooner or later and it might be a good time to learn how before you hit the wall. I give root 15G and have 7G used (and /home is a mounted partition).

Hope this helps some.