LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 12-17-2016, 04:58 PM   #1
AdultFoundry
Member
 
Registered: Jun 2015
Posts: 282

Rep: Reputation: Disabled
Unusual fast growth of the root partition


I have a / partition of about 20GB in size and I've been using it for close to 12 months now. I did not have any problems and since yesterday it looks like it grows 1GB in size per 24 hours, which is an unusual thing. This part of the disk got full day before yesterday, I thought it just run out of space with time, so I deleted some files and more than 1GB was free. I had to do the same thing the next day (free up some space) and as I am getting towards the end of the day, only around 100MB are free, so close to 1GB of space has already filled up and this is very strange.

I figured out that the problem is large /var/lib/mysql/ibdata1 and I think that this is in fact what is growing too. It looks like somebody may be increasing it from an outside with a deliberate attack, but I am not sure. The system is secure and it is very very unlikely that somebody would be attacking me as I dont have very well known sites.
The space is running out though and I am not sure what this may be.

Is there any good way of finding out what this may be? I've already done the disk space checking part (files and directories, I have the list of the biggest ones from about three hours ago) but maybe this is something else?

Thanks.
 
Old 12-17-2016, 05:05 PM   #2
TheEzekielProject
Member
 
Registered: Dec 2016
Distribution: arch
Posts: 668

Rep: Reputation: 190Reputation: 190
Have a look at this Basically, that ibdata1 doesn't ever get cleaned is a known thing for Mysql. Simply Do a mysqldump of all databases, procedures, triggers etc except the mysql and performance_schema databases Drop all databases except the above 2 databases Stop mysql Delete ibdata1 and ib_log files Start mysql Restore from dump

Last edited by TheEzekielProject; 12-17-2016 at 05:07 PM.
 
Old 12-18-2016, 02:30 AM   #3
AdultFoundry
Member
 
Registered: Jun 2015
Posts: 282

Original Poster
Rep: Reputation: Disabled
Like I said I've been using this hosting / system for close to 12 months and I've never had any issues like that. I freed up around 1.1GB of disk space on the root partition and when I checked it now this is already occupied. So basically the root partition is running out of 1.1GB of space per 24 hours and this is very strange. I was not making any changes to any of the databases, as I was working on fixing this, so there is no reason for a database file to grow (the ibdata1) and especially for 1.1GB in less than a day time.

The /var/lib/mysql/ibdata1 was 13GB yesterday, when I started working on this, and it is now 14GB. There is no regular reason for a database to grow as my websites are slow and I have not been working on the sites. A growth of 10MB would be a very strange thing, I would say.

Last edited by AdultFoundry; 12-18-2016 at 02:34 AM.
 
Old 12-18-2016, 11:58 AM   #4
AdultFoundry
Member
 
Registered: Jun 2015
Posts: 282

Original Poster
Rep: Reputation: Disabled
Is it possible that entries like these in /etc/my.cnf may be causing this:

[mysql]
max_allowed_packet=100M
[mysqld]
max_allowed_packet=100M

These are not the default settings and I had to increase the default limit as I was not able to import one of the databases that I had with no errors (vBulletin database, and this program was storing image files in the database)?

Overall my /etc/my.cnf looks like this. Are there any problematic things in there, or probably not?

#My mod
[mysql]
max_allowed_packet=100M
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
#My mod
max_allowed_packet=100M
log_error=/var/log/mariadb/mariadb_error.log
slow_query_log_file = /var/log/mariadb/mariadb_slow.log
long_query_time = 2
log_queries_not_using_indexes = 1
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
# Settings user and group are ignored when systemd is used.
# If you need to run mysqld under a different user or group,
# customize your systemd unit file for mariadb according to the
# instructions in http://fedoraproject.org/wiki/Systemd

[mysqld_safe]
#Line below was log-error=/var/log/mariadb/mariadb.log
log-error=/var/log/mariadb/mariadb_error.log
pid-file=/var/run/mariadb/mariadb.pid
#
# include all files from the config directory
#
!includedir /etc/my.cnf.d
 
Old 12-18-2016, 03:28 PM   #5
AdultFoundry
Member
 
Registered: Jun 2015
Posts: 282

Original Poster
Rep: Reputation: Disabled
/var/lib/mysql/ibdata1 grows more than 1GB per 24hours since three days. This is the day three now. Does this sound like a usual thing?
 
Old 12-18-2016, 04:25 PM   #6
astrogeek
Moderator
 
Registered: Oct 2008
Distribution: Slackware [64]-X.{0|1|2|37|-current} ::12<=X<=15, FreeBSD_12{.0|.1}
Posts: 6,277
Blog Entries: 24

Rep: Reputation: 4225Reputation: 4225Reputation: 4225Reputation: 4225Reputation: 4225Reputation: 4225Reputation: 4225Reputation: 4225Reputation: 4225Reputation: 4225Reputation: 4225
Here is a rather complete discussion of the problem, with suggested solutions.

The problem is due to InnoDB tablespace requirements, in particular that it must keep a complete snapshot of the DB for open transactions in case a ROLLBACK should be required. If you have long running transaction queries, then you have an open snapshot for each.

The above link tells you how to identify those long running queries.

From the above page:

Quote:
How can I solve the problem?

The answer to this question is easy. If you can still commit that query, do it. If not you’ll have to kill the thread to start the rollback process. That will just stop ibdata1 from growing but it is clear that your software has a bug or someone made a mistake...

Is there any way to recover the used space?

No, it is not possible at least in an easy and fast way. InnoDB tablespaces never shrink…

Summary

When the ibdata1 file is growing too fast within MySQL it is usually caused by a long running transaction that we have forgotten about. Try to solve the problem as fast as possible (commiting or killing a transaction) because you won’t be able to recover the wasted disk space without the painfully slow mysqldump process.

Monitoring the database to avoid these kind of problems is also very recommended.
 
Old 12-19-2016, 05:57 AM   #7
AdultFoundry
Member
 
Registered: Jun 2015
Posts: 282

Original Poster
Rep: Reputation: Disabled
If somebody has the same problem, just move the /var/lib/mysql to a different location on your system, where you have more space. I was looking into shrinking an LV, adding it where the space was needed, but this is not a needed thing, and it can be risky. It takes a lot of work to prepare it if you dont work on things like this every day.

Running out of space on a disk because of /var/lib/mysql/ibdata1 (and possibly other files there)? Just move the whole thing where you have space. On Centos 7 I had to add these location to the /etc/php.ini file (two places) and this is not described in most of the articles on the Internet.
 
Old 12-19-2016, 06:22 AM   #8
AdultFoundry
Member
 
Registered: Jun 2015
Posts: 282

Original Poster
Rep: Reputation: Disabled
The problem that I had was caused by disabling the captcha on registration to a vBulletin site. Some people detected it and started filling up the forum with some automated posts. The database got 231 larger than it was before. I disabled the captcha because the images were not working because of some problem with the site (the support of the program cant fix it they say, I will probably drop the whole site). The space on the / has run out and it started messing up other websites on this hosting plan.

Last edited by AdultFoundry; 12-19-2016 at 07:17 AM.
 
1 members found this post helpful.
Old 12-19-2016, 06:46 AM   #9
AdultFoundry
Member
 
Registered: Jun 2015
Posts: 282

Original Poster
Rep: Reputation: Disabled
This can be deleted...

Last edited by AdultFoundry; 12-19-2016 at 07:03 AM.
 
Old 12-19-2016, 04:55 PM   #10
sundialsvcs
LQ Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 10,706
Blog Entries: 4

Rep: Reputation: 3949Reputation: 3949Reputation: 3949Reputation: 3949Reputation: 3949Reputation: 3949Reputation: 3949Reputation: 3949Reputation: 3949Reputation: 3949Reputation: 3949
Also note that "MySQL is a rather-compelling reason to use LVM = Logical Volume Management!"

When some mount-point, be it "/" or what-have-you, starts running out of space, LVM allows you to allocate space on another physical volume to expand this logical volume. There is no longer a one-to-one correspondence between a mount-point and a single physical resource.

You do not have to shut down the machine to add space in this way.

LVM also allows you to plan which volumes will be responsible for storing various types of information. If your machine hardware is capable of multiple simultaneous I/O transfer operations (as most are, these days), then you can distribute the information to take full advantage of this. Very important for "I/O-intensive" machines like SQL servers.

Last edited by sundialsvcs; 12-19-2016 at 04:56 PM.
 
Old 12-20-2016, 02:43 AM   #11
AdultFoundry
Member
 
Registered: Jun 2015
Posts: 282

Original Poster
Rep: Reputation: Disabled
@sundialsvcs - I am aware of that, I've been reading about it for the last 2-3 days, and I read more than 10 Linux books (more than a year ago), the best books that are out there. In my situation, I dont work on system administration every day, I have a self-managed hosting plan because it is less expensive and it enables me to work on making websites, and also possibly programming, later on. If I would have to pay like $150 or $200 per month, this would not make any sense, basically I would not be able to afford this kind of money for this at this point. Right now I pay $42 per month and I can easily do what I want. When you dont work on it every day (like a support person in a hosting company, lets say, or some other jobs related to this), it takes a while to get familair with the situation and all the commands that need to be used. Things like this are not written or described in the books. But it is a good situation, this is where you actually learn. 2-4 additional situations like this and I will on the next level of this, having a better understanding of how things work. I also need to work on understanding the commands like cp, find, things like this. I would like, for example, get a list of biggest files and another one of the biggest directories on /, that would look like this:

10G /var/lib/mysql/some-name
9.5G /var/lib/mysql/some-other-name
7.1G /var/log/httpd/error_log

Just the human readable size, name of the file with the directory path, and nothing else. And this would be organized from the largest on / towards the smaller ones, and it could be limited not to the top 50 or 100 entries, lets say, but everything above 100MB or so (or some other number). I've been reading about these commnads on the Internet and when something like this is needed, it takes a while to figure it out. find is faster than some other commands like this, I remember that I was waiting for like 20 minutes for one of these commands to complete. And the same for the directories on / - list just like this but for the directories. The next time I have a problem, I check where the biggest files are, based on that I would check the size of the databases, then look at the problematic site and I would know what is going on.

I wasted some time for researching and preparing the LVM shrinking and this was not a needed thing, although it may be in the future so this is ok. I still had issues with very high server load when I was turning off the pc yesterday, but for some reason, everything works as it used to know (the load average displayed in one of the scripts is 0.32 instead of over a hundred like it was yesterday). I guess the things came back to normal by itself, in some way, I am not sure. This is where I am lacking the experience and the knowledge too. There is a lot left to learn about it and this is a good thing. All the databases (like the things that could be read on the MariaDb sites or MySql site), programming, some kind of improvements, optimization, software design (LAMP, as this is what I am interested in).

If somebody is familiar, what would you be looking into as far as high / very high server load caused by mysqld / mysql? It came back to normal now, but it could be good to check and figure out what was going on. There was some problem there with something, I can say, and I should be able to check it in some place, like logs, but I did not get any info about it from there.

This is how it looked yesterday:
http://prnt.sc/dlheuq

And this is what it is now / today:
http://prntscr.com/dln01w

It would just work itself out, by itself? I went to mysql and I entered the command "SHOW ALL PROCESSES" or something like this and it was all related to one site, which is a pinterest clone script, which is not of a very good quality. I then went to Google Analytics Live for this website and even though this was not showing any visitors to this site for like 3-5 minutes period lets say, the load caused by the mysqld/mysql processes of this website was very high. Like I said, it is all normal today, and maybe there is a way to check what this was. I have the sysstat program installed, I think. Also the one that should be mailing a snapshot of a day to the administrator, but I am not sure. I think that I kept it installed but it has not been sending any emails, I think. I have less than 1,000 people per day to my sites overall, so I dont worry about it for now, but if the things would be going as they were at some kind of workplace, I would be terminated for the work that I did...

Last edited by AdultFoundry; 12-20-2016 at 02:46 AM.
 
Old 12-20-2016, 03:21 AM   #12
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 7,381
Blog Entries: 3

Rep: Reputation: 3772Reputation: 3772Reputation: 3772Reputation: 3772Reputation: 3772Reputation: 3772Reputation: 3772Reputation: 3772Reputation: 3772Reputation: 3772Reputation: 3772
Quote:
Originally Posted by AdultFoundry View Post
I would like, for example, get a list of biggest files and another one of the biggest directories on /, that would look like this:

10G /var/lib/mysql/some-name
9.5G /var/lib/mysql/some-other-name
7.1G /var/log/httpd/error_log

Just the human readable size, name of the file with the directory path, and nothing else.
The utility du can do a lot of that. See the options -s and -h in particular.

Code:
du -sh /var/lib/mysql/*
If you would like it sorted, then pipe it through sort.

Code:
du -sh /var/lib/mysql/* | sort -k1,1rh -k2,2
If you need all that but just the top five offenders, then use also head

Code:
du -sh /var/lib/mysql/* | sort -k1,1rh -k2,2 | head -n 5
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Resizing partition in VHD file - root partition first, swap second (*Advanced*) dpesios Linux - Virtualization and Cloud 2 05-18-2012 05:50 AM
Resizing partition in VHD file - root partition first, swap second (*Advanced*) dpesios Linux - Virtualization and Cloud 3 05-17-2012 11:53 AM
Do you see anything unusual in my partition info (FC1) slackist Linux - Newbie 2 06-01-2004 10:38 AM
Root desktop is fast, Users are slow BajaNick Linux - General 5 02-10-2004 08:22 PM
Need help getting root access - Unusual Situation solo545 Linux - Security 11 08-17-2002 08:06 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 05:16 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration