Root filesystem full root cause

Febi881 · 07-16-2013, 01:00 AM

Dear All,

I am in a project to find a solution for root file system full (/) on Linux servers. I know that it is difficult to fix this as we don’t know who or what application will dump the files on root FS. But I was asked to find some sort of common solution for this problems or suggestions. Can anyone have gone through this before, If so please help me.

evo2 · 07-16-2013, 01:28 AM

Hi,

having separate /var and /tmp partitions will go a *long* way to stopping random processes filling up /.

Evo2.

unSpawn · 07-16-2013, 01:48 AM

...what evo2 said plus prevention like analyzing what gets logged (in the sense that multiple consecutive errors should be fixed instead of letting them fill up logs and maybe add proactive measures where this concerns brute force attacks) and remote log / free disk space monitoring with say Nagios.

Febi881 · 07-16-2013, 02:15 AM

/var /tmp & /home are created as a different file systems.

catkin · 07-16-2013, 03:02 AM

Have you found out which directory/ies the files filling the root file system are being created in?

Febi881 · 07-16-2013, 07:54 AM

I am serching for a common solution or suggessions. In my list i have 200 incidents which has 50 repeated issues. Is it anything that can impliment on all servers to reduce the incidents.

TobiSGD · 07-16-2013, 08:31 AM

If you have /var, /tmp and /home as separate partitions they shouldn't be anything that routinely writes to / to fill it up. If that is a case I would find out what program does this and fix that issue. Usually, on a system configured like yours, you should be able to mount / as read-only, preventing any writes to that partition.

Febi881 · 07-16-2013, 08:54 AM

Yes it was manually moved some files to the / file system by user or application team members. Once i informed them they removed the application files. Any work areound can u pls suggest?

frieza · 07-16-2013, 09:40 AM

Quote:

Originally Posted by Febi881

Yes it was manually moved some files to the / file system by user or application team members. Once i informed them they removed the application files. Any work areound can u pls suggest?

first of all, off topic, but please spell out your words (text speak such as 'u' and 'pls' are against the rules here FYI)
second, it has been mentioned in your situation that mounting / as read-only is a viable option in your situation, this would deny anyone or anything privilege to write to / in the first place. this seems to me the most simple solution short of writing a script that checks for new files and moves them to their respective owner's home directory, which would be cumbersome at best.

TobiSGD · 07-16-2013, 09:42 AM

Work around for team members to not write to /? Why do they have that capability in the first place, are they working with root-privileges? Give them restricted privileges, use sudo for giving limited root access to those that need it.

Febi881 · 07-16-2013, 08:37 PM

Thank you all for your suggestions. But just a question is there is any disadvantage to mount root File Syatem as in read-only mode?

TobiSGD · 07-16-2013, 08:51 PM

Yes, to change contents of it (like installing software or changing configuration files) you have to remount it r/w first, then go back to r/o later. Other than that, no.
The question still remains why your users are able to write to that partition. If they indeed work with root privileges there is nothing that hinders them to remount the partition themselves r/w to fill it up again. My first concern would be to fix that.

Febi881 · 07-16-2013, 08:57 PM

There are few users like Tivoli backup, few special customers they have root credentials etc. As per agreement they have those priviliages alredy.

I will put a suggession to remove those privliage as a part of prevention.

Febi881 · 07-17-2013, 04:25 AM

Just a question, if we have few NFS/NETAPP filesystems mounted on this root file system with application user as owners, will this read-only root file system will affect them from read-write operations.

TobiSGD · 07-17-2013, 06:12 AM

Quote:

Originally Posted by Febi881

Just a question, if we have few NFS/NETAPP filesystems mounted on this root file system with application user as owners, will this read-only root file system will affect them from read-write operations.

If you mount those filesystems rw then they are not affected.