I have a / partition of about 20GB in size and I've been using it for close to 12 months now. I did not have any problems and since yesterday it looks like it grows 1GB in size per 24 hours, which is an unusual thing. This part of the disk got full day before yesterday, I thought it just run out of space with time, so I deleted some files and more than 1GB was free. I had to do the same thing the next day (free up some space) and as I am getting towards the end of the day, only around 100MB are free, so close to 1GB of space has already filled up and this is very strange.

I figured out that the problem is large /var/lib/mysql/ibdata1 and I think that this is in fact what is growing too. It looks like somebody may be increasing it from an outside with a deliberate attack, but I am not sure. The system is secure and it is very very unlikely that somebody would be attacking me as I dont have very well known sites.
The space is running out though and I am not sure what this may be.

Is there any good way of finding out what this may be? I've already done the disk space checking part (files and directories, I have the list of the biggest ones from about three hours ago) but maybe this is something else?

Thanks.

Have a look at this Basically, that ibdata1 doesn't ever get cleaned is a known thing for Mysql. Simply Do a mysqldump of all databases, procedures, triggers etc except the mysql and performance_schema databases Drop all databases except the above 2 databases Stop mysql Delete ibdata1 and ib_log files Start mysql Restore from dump

Like I said I've been using this hosting / system for close to 12 months and I've never had any issues like that. I freed up around 1.1GB of disk space on the root partition and when I checked it now this is already occupied. So basically the root partition is running out of 1.1GB of space per 24 hours and this is very strange. I was not making any changes to any of the databases, as I was working on fixing this, so there is no reason for a database file to grow (the ibdata1) and especially for 1.1GB in less than a day time.

The /var/lib/mysql/ibdata1 was 13GB yesterday, when I started working on this, and it is now 14GB. There is no regular reason for a database to grow as my websites are slow and I have not been working on the sites. A growth of 10MB would be a very strange thing, I would say.

Is it possible that entries like these in /etc/my.cnf may be causing this:

[mysql]
max_allowed_packet=100M
[mysqld]
max_allowed_packet=100M

These are not the default settings and I had to increase the default limit as I was not able to import one of the databases that I had with no errors (vBulletin database, and this program was storing image files in the database)?

Overall my /etc/my.cnf looks like this. Are there any problematic things in there, or probably not?

#My mod
[mysql]
max_allowed_packet=100M
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
#My mod
max_allowed_packet=100M
log_error=/var/log/mariadb/mariadb_error.log
slow_query_log_file = /var/log/mariadb/mariadb_slow.log
long_query_time = 2
log_queries_not_using_indexes = 1
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
# Settings user and group are ignored when systemd is used.
# If you need to run mysqld under a different user or group,
# customize your systemd unit file for mariadb according to the
# instructions in http://fedoraproject.org/wiki/Systemd

[mysqld_safe]
#Line below was log-error=/var/log/mariadb/mariadb.log
log-error=/var/log/mariadb/mariadb_error.log
pid-file=/var/run/mariadb/mariadb.pid
#
# include all files from the config directory
#
!includedir /etc/my.cnf.d

/var/lib/mysql/ibdata1 grows more than 1GB per 24hours since three days. This is the day three now. Does this sound like a usual thing?

Here is a rather complete discussion of the problem, with suggested solutions.

The problem is due to InnoDB tablespace requirements, in particular that it must keep a complete snapshot of the DB for open transactions in case a ROLLBACK should be required. If you have long running transaction queries, then you have an open snapshot for each.

The above link tells you how to identify those long running queries.

From the above page:

If somebody has the same problem, just move the /var/lib/mysql to a different location on your system, where you have more space. I was looking into shrinking an LV, adding it where the space was needed, but this is not a needed thing, and it can be risky. It takes a lot of work to prepare it if you dont work on things like this every day.

Running out of space on a disk because of /var/lib/mysql/ibdata1 (and possibly other files there)? Just move the whole thing where you have space. On Centos 7 I had to add these location to the /etc/php.ini file (two places) and this is not described in most of the articles on the Internet.

The problem that I had was caused by disabling the captcha on registration to a vBulletin site. Some people detected it and started filling up the forum with some automated posts. The database got 231 larger than it was before. I disabled the captcha because the images were not working because of some problem with the site (the support of the program cant fix it they say, I will probably drop the whole site). The space on the / has run out and it started messing up other websites on this hosting plan.

1 members found this post helpful.

This can be deleted...

Also note that "MySQL is a rather-compelling reason to use LVM = Logical Volume Management!"

When some mount-point, be it "/" or what-have-you, starts running out of space, LVM allows you to allocate space on another physical volume to expand this logical volume. There is no longer a one-to-one correspondence between a mount-point and a single physical resource.

You do not have to shut down the machine to add space in this way.

LVM also allows you to plan which volumes will be responsible for storing various types of information. If your machine hardware is capable of multiple simultaneous I/O transfer operations (as most are, these days), then you can distribute the information to take full advantage of this. Very important for "I/O-intensive" machines like SQL servers.

@sundialsvcs - I am aware of that, I've been reading about it for the last 2-3 days, and I read more than 10 Linux books (more than a year ago), the best books that are out there. In my situation, I dont work on system administration every day, I have a self-managed hosting plan because it is less expensive and it enables me to work on making websites, and also possibly programming, later on. If I would have to pay like $150 or $200 per month, this would not make any sense, basically I would not be able to afford this kind of money for this at this point. Right now I pay $42 per month and I can easily do what I want. When you dont work on it every day (like a support person in a hosting company, lets say, or some other jobs related to this), it takes a while to get familair with the situation and all the commands that need to be used. Things like this are not written or described in the books. But it is a good situation, this is where you actually learn. 2-4 additional situations like this and I will on the next level of this, having a better understanding of how things work. I also need to work on understanding the commands like cp, find, things like this. I would like, for example, get a list of biggest files and another one of the biggest directories on /, that would look like this:

10G /var/lib/mysql/some-name
9.5G /var/lib/mysql/some-other-name
7.1G /var/log/httpd/error_log

Just the human readable size, name of the file with the directory path, and nothing else. And this would be organized from the largest on / towards the smaller ones, and it could be limited not to the top 50 or 100 entries, lets say, but everything above 100MB or so (or some other number). I've been reading about these commnads on the Internet and when something like this is needed, it takes a while to figure it out. find is faster than some other commands like this, I remember that I was waiting for like 20 minutes for one of these commands to complete. And the same for the directories on / - list just like this but for the directories. The next time I have a problem, I check where the biggest files are, based on that I would check the size of the databases, then look at the problematic site and I would know what is going on.

I wasted some time for researching and preparing the LVM shrinking and this was not a needed thing, although it may be in the future so this is ok. I still had issues with very high server load when I was turning off the pc yesterday, but for some reason, everything works as it used to know (the load average displayed in one of the scripts is 0.32 instead of over a hundred like it was yesterday). I guess the things came back to normal by itself, in some way, I am not sure. This is where I am lacking the experience and the knowledge too. There is a lot left to learn about it and this is a good thing. All the databases (like the things that could be read on the MariaDb sites or MySql site), programming, some kind of improvements, optimization, software design (LAMP, as this is what I am interested in).

If somebody is familiar, what would you be looking into as far as high / very high server load caused by mysqld / mysql? It came back to normal now, but it could be good to check and figure out what was going on. There was some problem there with something, I can say, and I should be able to check it in some place, like logs, but I did not get any info about it from there.

This is how it looked yesterday:
http://prnt.sc/dlheuq

And this is what it is now / today:
http://prntscr.com/dln01w

It would just work itself out, by itself? I went to mysql and I entered the command "SHOW ALL PROCESSES" or something like this and it was all related to one site, which is a pinterest clone script, which is not of a very good quality. I then went to Google Analytics Live for this website and even though this was not showing any visitors to this site for like 3-5 minutes period lets say, the load caused by the mysqld/mysql processes of this website was very high. Like I said, it is all normal today, and maybe there is a way to check what this was. I have the sysstat program installed, I think. Also the one that should be mailing a snapshot of a day to the administrator, but I am not sure. I think that I kept it installed but it has not been sending any emails, I think. I have less than 1,000 people per day to my sites overall, so I dont worry about it for now, but if the things would be going as they were at some kind of workplace, I would be terminated for the work that I did...

The utility du can do a lot of that. See the options -s and -h in particular.

If you would like it sorted, then pipe it through sort.

If you need all that but just the top five offenders, then use also head