LinuxQuestions.org
Visit the LQ Articles and Editorials section
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices

Reply
 
Search this Thread
Old 11-06-2011, 04:34 AM   #1
vzxen
Member
 
Registered: Jul 2010
Posts: 126

Rep: Reputation: 4
Disk full but after reboot 30GB available


Hi,

I have a CentOS 5.5 server with 50GB space.
Since three days I am facing a unique problem.
The Disk Usage is 18G.
After some 3-4 hours it touches 100% usage of the disk.
Restarting the OS shows 18GB usage again and then after 3-4 hours the same 100%.

What could be the reason behind this ?
I have even checked 'du -hs /' after it shows 100% usage.
But even 'du -hs /' shows 18-19G and not 50G

Can anyone help me with this ?
Is this is a virus ?
Should I reinstall the OS ?
 
Click here to see the post LQ members have rated as the most helpful post in this thread.
Old 11-06-2011, 04:51 AM   #2
salasi
Senior Member
 
Registered: Jul 2007
Location: Directly above centre of the earth, UK
Distribution: SuSE, plus some hopping
Posts: 3,896

Rep: Reputation: 774Reputation: 774Reputation: 774Reputation: 774Reputation: 774Reputation: 774Reputation: 774
Quote:
Originally Posted by vzxen View Post
Can anyone help me with this ?
Is this is a virus ?
Should I reinstall the OS ?
Who knows, probably not and certainly not (at this stage), in order.

Quote:
Originally Posted by vzxen View Post
After some 3-4 hours it touches 100% usage of the disk.
Restarting the OS shows 18GB usage again and then after 3-4 hours the same 100%.

What could be the reason behind this ?
At least at first blush, the most likely thing seems to be a log file growing at an insane rate (which, in turn, suggests that there is some kind of underlying problem, and the log file is being written to continuously) or some runaway process that is writing continuously.

The first thing to do is to try to make some observations before it has had chance to fill up completely:
  • is dmesg showing up millions of problems that are getting written somewhere (and, if so what are those problems)?
  • do you have any log files that are very large?
  • any files that are very large?
  • is there lots of unexpected io activity?
  • unusual pattern of open files

That should give a clue as to where to start.
 
Old 11-06-2011, 07:41 AM   #3
vzxen
Member
 
Registered: Jul 2010
Posts: 126

Original Poster
Rep: Reputation: 4
Hi,

The apache log files were growing insanely large.
But I set a CRON to delete the entire log file every minute.
The command :
rm -rf /var/log/httpd/xyz.com.err

Waiting for your reply.
 
Old 11-06-2011, 07:52 AM   #4
Thor_2.0
Senior Member
 
Registered: Nov 2007
Location: Somewhere on my hard drive...
Distribution: Manjaro
Posts: 2,204
Blog Entries: 23

Rep: Reputation: 279Reputation: 279Reputation: 279
Quote:
But I set a CRON to delete the entire log file every minute.
A very noble attempt to stop a tsunami...with a stop sign

You delete the very evidence of your problem...could we have a glimpse of the contents? Maybe that could help...

Thor
 
Old 11-06-2011, 11:40 AM   #5
vzxen
Member
 
Registered: Jul 2010
Posts: 126

Original Poster
Rep: Reputation: 4
Hi,

It had lots of PHP errors.
We upgraded PHP to 5.3.2 from 5.2.17
But if the file is deleted then shouldnt the space be emptied.
 
Old 11-06-2011, 12:19 PM   #6
yancek
Senior Member
 
Registered: Apr 2008
Distribution: PCLinux, Ubuntu, Slackware
Posts: 4,666

Rep: Reputation: 730Reputation: 730Reputation: 730Reputation: 730Reputation: 730Reputation: 730Reputation: 730
Quote:
But if the file is deleted then shouldnt the space be emptied.
If you mean the log file, yes the space will be emptied but not knowing the cause and changing it will just repeat the same errors and create another large log file. You need to look at the log file to see what the problem so you can correct it.
 
Old 11-06-2011, 12:40 PM   #7
vzxen
Member
 
Registered: Jul 2010
Posts: 126

Original Poster
Rep: Reputation: 4
Hi,

>> If you mean the log file, yes the space will be emptied but not knowing the cause

But at the moment I am not concerned about the PHP errors. I know why they are occuring. They are warnings of PHP and things like variables not declared.

If the file is deleted every minute there is no chance that the file will eat up the disk space right ?
 
Old 11-06-2011, 12:42 PM   #8
Thor_2.0
Senior Member
 
Registered: Nov 2007
Location: Somewhere on my hard drive...
Distribution: Manjaro
Posts: 2,204
Blog Entries: 23

Rep: Reputation: 279Reputation: 279Reputation: 279
Quote:
It had lots of PHP errors.
We upgraded PHP to 5.3.2 from 5.2.17
Okay, so you may either need to rewrite whatever pages are on the server or maybe...a backwards compatability setting...I'm willing to ding into the latter, if need be...

Is it a web hotel or just a private server with your stuff only...

Let's see some error lines if possible, please. I'm somewhat excited, here...

Quote:
If the file is deleted every minute there is no chance that the file will eat up the disk space right ?
Of course, but ... that's a bit like pumping the water out of a flooded basement and leaving the plumbing leaking...it does not solve your eminent problem, I'm afraid...

Thor

Last edited by Thor_2.0; 11-06-2011 at 12:44 PM.
 
Old 11-06-2011, 12:49 PM   #9
vzxen
Member
 
Registered: Jul 2010
Posts: 126

Original Poster
Rep: Reputation: 4
Hi,

The server has many users and the PHP errors canot be controlled sir.

> Of course, but ... that's a bit like pumping the water out of a flooded basement
> and leaving the plumbing leaking...it does not solve your eminent problem, I'm afraid...

At the moment the mysql server shuts every 3-4 hours saying no disk space left !
The 'du -hs /' shows 18GB usage.
I am worried its not a virus fooling the system no DiskSpace is left !
 
Old 11-06-2011, 01:11 PM   #10
Thor_2.0
Senior Member
 
Registered: Nov 2007
Location: Somewhere on my hard drive...
Distribution: Manjaro
Posts: 2,204
Blog Entries: 23

Rep: Reputation: 279Reputation: 279Reputation: 279
Quote:
I am worried its not a virus fooling the system no DiskSpace is left !
You're using Linux, a (live/active) virus is not to be worried about. Besides, a virus would'nt attrack attention to itself, and definetly not in such a manner...

Quote:
The server has many users and the PHP errors canot be controlled sir.
In that case, pumping (and hoping) is the better option, I'm affraid...

Good luck!

Thor
 
Old 11-06-2011, 03:21 PM   #11
btmiller
Senior Member
 
Registered: May 2004
Location: In the DC 'burbs
Distribution: Arch, Scientific Linux, Debian, Ubuntu
Posts: 4,082

Rep: Reputation: 299Reputation: 299Reputation: 299
Remember that when you delete a file in Linux, you're really just unlinking it (i.e. destroying the reference from the name in the filesystem to the object stored on disk). If some program has the file open, then the space will not be reclaimed until whatever process has it open closes it. Do you restart Apache after rm'ing the log files? If not, I suspect it's keeping them open and thus the disk space isn't reclaimed. This isn't a "virus", it's just the way that Linux works. If you want to confirm that a particular file is open, use lsof and see for yourself.

I agree with the above advice that the best thing to do would be to re-write your error-producing application. If that's truly impossible, you could always modify your Apache and/pr syslog configuration to not print PHP errors.
 
2 members found this post helpful.
Old 11-06-2011, 11:51 PM   #12
vzxen
Member
 
Registered: Jul 2010
Posts: 126

Original Poster
Rep: Reputation: 4
Hi,

> Remember that when you delete a file in Linux, you're really just unlinking it

Thanks for the advice. I have now simply emtied the file every minute via a CRON :
echo "" > /var/log/httpd/xyz.com.err

Hopefully this should solve the problem. Even lsof now shows that this file becomes of size zero and increases to a few KB in a minute and then goes to zero.

I will keep you posted if this doesnt help.
I will also see what we can do about the PHP errors coming.
One way I see is that we should switch off error reporting by default.
 
  


Reply

Tags
disk full


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
eth0 disappears after reboot/full shutdown shannonbay Linux - Hardware 4 03-30-2009 03:52 AM
Need to remove Linux from a laptop(30GB drive) for more Disk space for windows jah46 Linux - Newbie 3 10-13-2007 05:23 PM
disk stats not updating after deletion of file.. disk full joecool118 Linux - General 2 04-10-2006 12:23 PM
/home partition full (but) it's not on reboot reddazz Linux - Security 2 02-03-2005 08:12 AM
can I recover from disk full - without reboot ? can't open display dtimms Linux - Software 3 09-19-2004 12:27 AM


All times are GMT -5. The time now is 08:02 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration