Mysql2::Error: Lost connection to MySQL server during query

NotionCommotion · 03-27-2014, 09:36 AM

I am at my wits end! I am receiving the following error:

Code:

[root@michaels gitlab]# sudo -u git -H bundle exec rake gitlab:setup RAILS_ENV=production
This will create the necessary database tables and seed the database.
You will lose any previous data stored in the database.
Do you want to continue (yes/no)? yes

gitlabhq_production already exists
-- enable_extension("plpgsql")
   -> 0.0013s
bla bla bla
-- add_index("namespaces", ["type"], {:name=>"index_namespaces_on_type", :using=>:btree})
   -> 0.2003s
-- create_table("notes", {:force=>true})
rake aborted!
Mysql2::Error: Lost connection to MySQL server during query: CREATE TABLE `notes` (`id` int(11) DEFAULT NULL auto_increment PRIMARY KEY, `note` text, `noteable_type` varchar(255), `author_id` int(11), `created_at` datetime, `updated_at` datetime, `project_id` int(11), `attachment` varchar(255), `line_code` varchar(255), `commit_id` varchar(255), `noteable_id` int(11), `system` tinyint(1) DEFAULT 0 NOT NULL, `st_diff` text) ENGINE=InnoDB
/home/git/gitlab/vendor/bundle/ruby/2.0.0
bla bla bla
/gems/activerecord-4.0.3/lib/active_record/railties/databases.rake:260:in `block (3 levels) in <top (required)>'
/home/git/gitlab/lib/tasks/gitlab/setup.rake:17:in `setup_db'
/home/git/gitlab/lib/tasks/gitlab/setup.rake:4:in `block (2 levels) in <top (required)>'
Tasks: TOP => db:schema:load
(See full trace by running task with --trace)
[root@michaels gitlab]#

I am using mysql55.x86_64 , 5.5.36-1.ius.centos6.

I've tried modifing /etc/my.cnf as follows, but it produces no change:

Uncomment innodb_log_file_size and change from 8M to 1G
Uncomment innodb-buffer-pool-size and change from 128M to 4G
Change max-allowed-packet from 16M to 32M

What could be the cause, and how can I troubleshoot it?

Thank you

szboardstretcher · 03-27-2014, 09:41 AM

First of all,.. Gitlab reccomends using postgres. Not sure how useful this program will be without it.

Quote:

We do not recommend using MySQL due to various issues. For example, case (in)sensitivity and problems that suggested fixes have.

But anyway, it looks like that rake task is trying to use postgres connections? (pgsql) instead of mysql.

https://github.com/gitlabhq/gitlabhq...nstallation.md
https://github.com/gitlabhq/gitlabhq...abase_mysql.md

Have you followed all the steps in those guides? Including the part that says 'skip this if you dont want to use postgres, and use mysql instead'?

NotionCommotion · 03-27-2014, 10:25 AM

Quote:

Originally Posted by szboardstretcher

First of all,.. Gitlab reccomends using postgres. Not sure how useful this program will be without it.

But anyway, it looks like that rake task is trying to use postgres connections? (pgsql) instead of mysql.

https://github.com/gitlabhq/gitlabhq...nstallation.md
https://github.com/gitlabhq/gitlabhq...abase_mysql.md

Have you followed all the steps in those guides? Including the part that says 'skip this if you dont want to use postgres, and use mysql instead'?

I successfully did this same install on another server, so it should work with MySQL. To the best of my knowledge, I've followed https://github.com/gitlabhq/gitlab-r...install/centos exactly, and skipped the steps regarding postgres.

I see what you are talking about regarding '-- enable_extension("plpgsql")'. I don't recall whether this came up on the other server successful install. Before executing the command which results in an error, I executed the following which should not use postgres.

Code:

sudo -u git -H /usr/local/bin/bundle install --deployment --without development test postgres aws

I've tried starting the install process over, but had the same results. Could there be any configuration that I need to manually remove? I would rather not rebuild the entire server just really start over.

What I've read about the "Lost connection to MySQL server during query" error is it might be memory related.

Any suggestions would be very appreciated.

Thank you

NotionCommotion · 03-27-2014, 10:44 AM

Could this be related?

Code:

[root@michaels git]# sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-shell.git -b v1.9.1
Initialized empty Git repository in /home/git/gitlab-shell/.git/
remote: Counting objects: 1150, done.
remote: Compressing objects: 100% (703/703), done.
remote: Total 1150 (delta 677), reused 725 (delta 400)
Receiving objects: 100% (1150/1150), 160.40 KiB, done.
Resolving deltas: 100% (677/677), done.
warning: Remote branch v1.9.1 not found in upstream origin, using HEAD instead
[root@michaels git]# cd gitlab-shell

szboardstretcher · 03-27-2014, 11:41 AM

That just means you are in a detached HEAD mode... explanation in the warning:

Quote:

git clone https://gitlab.com/gitlab-org/gitlab-shell.git -b v1.9.1
Cloning into 'gitlab-shell'...
:blah:
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b new_branch_name

Do you have a moment to show the versions that you are using here? of git, mysql, ruby,.. any other requirements? Or are you certain that you are meeting the requirements?

NotionCommotion · 03-27-2014, 12:00 PM

Quote:

Originally Posted by szboardstretcher

Do you have a moment to show the versions that you are using here? of git, mysql, ruby,.. any other requirements? Or are you certain that you are meeting the requirements?

I think you are on to something!!!! git is only 1.7.1. I am going to post a new question about this. Hopefully, this is the problem. Thank you!

On a side note, per https://dev.mysql.com/doc/refman/5.0...onnection.html

Quote:

C.5.2.3 Lost connection to MySQL server

There are three likely causes for this error message.

Usually it indicates network connectivity trouble and you should check the condition of your network if this error occurs frequently. If the error message includes “during query,” this is probably the case you are experiencing.

I will look into the git version before proceeding with this.

szboardstretcher · 03-27-2014, 12:12 PM

Np. If you are using an old ruby version with an old gem, that might explain the mysql stuff. I would suggest following the guide and upgrading Ruby to 2.0.x and Git to 1.7.10+ where needed.

NotionCommotion · 03-27-2014, 01:07 PM

Ugg!

I deleted the git user (and its home directory) and the database and started over.

Still the same errors. Sometimes I get "rake aborted! Errno::EBADF: Bad file descriptor: SHOW TABLES LIKE 'projects'", but mostly it is the "Lost connection to MySQL server during query" error

In regards to versions of MySQL, Ruby, and git, I should be fine on all of them.

Could it be a network issue as the link I posted to mysql suggested? How could I troubleshoot it.

Thank you

Code:

[root@michaels gitlab]# mysql -u root -p                                        Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 30
Server version: 5.5.36 Distributed by The IUS Community Project

Copyright (c) 2000, 2014, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> exit
Bye
[root@michaels gitlab]# git --version
git version 1.8.3.1
[root@michaels gitlab]# ruby -v
ruby 2.0.0p451 (2014-02-24 revision 45167) [x86_64-linux]
[root@michaels gitlab]#

NotionCommotion · 03-28-2014, 08:42 AM

http://omegadelta.net/2014/03/15/the-rails-grim-reaper/

I ended up solving the problem by commenting out "reaping_frequency: 10" in /home/git/config/database.yml.

Why this solved the issue, I am still not 100% certain.

Thanks for your help

divzero · 03-28-2014, 12:09 PM

Hey, I'm glad my blog post was helpful

The "reaper" is meant to remove "dead" connections but it seems to be very buggy and removes live ones too.

The reaper setting is default OFF in rails now (the rails team made that change due to bugs like the ones we're having), but it looks like a bunch of sites like Heroku copied some config from when it was default ON.

Really unless you are having problems with dead/leaking connections, you don't need the reaper – it's a really bad setting. And if you *are* having issues, then there must be a better way to solve them than enabling the reaper.

NotionCommotion · 03-28-2014, 01:45 PM

Thanks divzero,

Yes your blog was very helpful! Amazing how difficult it was to find this solution. I was changing buffer sizes, etc, etc in mysql configuration.

The buggyness (if that is a word) appears to even be dependent upon hardware.

divzero · 03-28-2014, 11:39 PM

NotionCommotion, you're welcome

Yeah I know the feeling, because most of the debugging advice for that error message are generic MySQL tips, I ended up tweaking at least a dozen config variables for MySQL with no luck. Only when I found a rails-specific reference to this error message did the penny drop.

I rolled out the fix a while ago, and the server has been running fine since so I'm confident that was the issue.

Out of curiosity, why did you have the reaper enabled? For me, it was due to following some Heroku docs.

NotionCommotion · 03-29-2014, 07:05 AM

Quote:

Originally Posted by divzero

Out of curiosity, why did you have the reaper enabled? For me, it was due to following some Heroku docs.

I did not explicitly enable the reaper. I was installing gitlab per https://github.com/gitlabhq/gitlab-r...install/centos, and it was preset in /home/git/gitlab/config/database.yml.

I will not soon forgot about the reaper.

Thanks again.

divzero · 03-30-2014, 07:49 AM

Quote:

Originally Posted by NotionCommotion

I will not soon forgot about the reaper.

Neither will I.

I'm going to contact Heroku and petition them to change their recommendations, I suggest you do the same with gitlab. I'm actually surprised more people are not encountering this issue…