Sudden start problem with mysql

spaceuser · 07-22-2008, 02:51 PM

I use mysql for a web gallery, but yesterday it seems to stop working, so I tried to restart mysql and webserver, but mysql fail to start. The syslog have the following info about it:

Quote:

Jul 22 13:11:12 webserver mysqld_safe[3587]: started
Jul 22 13:11:12 webserver mysqld[3590]: ^G/usr/sbin/mysqld: Can't read dir of '/tmp/' (Errcode: 13)
Jul 22 13:11:12 webserver mysqld[3590]: ^G/usr/sbin/mysqld: File '/var/log/mysql/mysql-bin.index' not found (Errcode: 13)
Jul 22 13:11:12 webserver mysqld[3590]: 080722 13:11:12 [ERROR] Aborting
Jul 22 13:11:12 webserver mysqld[3590]:
Jul 22 13:11:12 webserver mysqld[3590]: 080722 13:11:12 [Note] /usr/sbin/mysqld: Shutdown complete
Jul 22 13:11:12 webserver mysqld[3590]:
Jul 22 13:11:12 webserver mysqld_safe[3592]: ended
Jul 22 13:11:29 webserver /etc/init.d/mysql[3727]: 0 processes alive and '/usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf ping' resulted in
Jul 22 13:11:29 webserver /etc/init.d/mysql[3727]: ^G/usr/bin/mysqladmin: connect to server at 'localhost' failed
Jul 22 13:11:29 webserver /etc/init.d/mysql[3727]: error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)'
Jul 22 13:11:29 webserver /etc/init.d/mysql[3727]: Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!
Jul 22 13:11:29 webserver /etc/init.d/mysql[3727]:

I have searched for the error and have checked /tmp and all seems ok. I don't know what to do. I would appreciate any comment on this.

I run the above on a Debian 4 server.

TIA

TB0ne · 07-22-2008, 03:15 PM

Have you check the /var/log/mysql directory, and the /var/run/mysqld directory? Are they there, and have they changed?

Also, check your MySQl configuration file, and see where it stores database files. Off the top of my head I don't remember, but I think it may be /var/spool/mysql, but please verify. Make sure that the directory exists. If it got nuked, you'll lose everything.

trickykid · 07-22-2008, 03:30 PM

By default, mysql database files will reside in /var/lib/mysql

You need to check your mysql log files to find out the problem. But these lines are of concern:

Code:

Jul 22 13:11:12 webserver mysqld[3590]: ^G/usr/sbin/mysqld: Can't read dir of '/tmp/' (Errcode: 13)
Jul 22 13:11:12 webserver mysqld[3590]: ^G/usr/sbin/mysqld: File '/var/log/mysql/mysql-bin.index' not found (Errcode: 13)

Seems to me some needed files did get wiped. I hope you've been making backups of your database(s)?

jlinkels · 07-22-2008, 08:29 PM

Does the ^G stand for 'bell'?

Both error messages are equally disturbing. Why would mysql not be able to read on /tmp?

Check the permissions in the /tmp dir. The sticky bit must be set. This is how it should look like:
drwxrwxrwt 11 root root 11264 2008-07-22 21:06 /tmp

Did you check whether none of yours disks is full? Or inaccessible? Or not mounted? Or damaged?

Don't worry about the mysql-bin.index. It is not very important for your database, only if you want to roll back transactions. If mysql insists of having this file, I guess you could create an empty file. Or, if there are still file in that directory, put the file list in:

Code:

/var/log/mysql/mysql-bin.000158
/var/log/mysql/mysql-bin.000159
/var/log/mysql/mysql-bin.000160
/var/log/mysql/mysql-bin.000161
/var/log/mysql/mysql-bin.000162
/var/log/mysql/mysql-bin.000163
/var/log/mysql/mysql-bin.000164
/var/log/mysql/mysql-bin.000165
/var/log/mysql/mysql-bin.000166
/var/log/mysql/mysql-bin.000167
/var/log/mysql/mysql-bin.000168
/var/log/mysql/mysql-bin.000169
/var/log/mysql/mysql-bin.000170
/var/log/mysql/mysql-bin.000171

But then again, I think that file is not the problem, it is your read access in general.

jlinkels

spaceuser · 07-23-2008, 01:25 PM

Thanks for all answers, but all seems ok when I check.

The log says that it couldn't read the /tmp,
^G/usr/sbin/mysqld: Can't read dir of '/tmp/' (Errcode: 13)

My /tmp is as it should, as seen below

Quote:

drwxrwxrwt 5 root root 4096 2008-07-21 05:38 tmp

I have also plenty of diskspace left also,

Quote:

webserver:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 2.3G 955M 1.3G 43% /
tmpfs 507M 0 507M 0% /lib/init/rw
udev 10M 64K 10M 1% /dev
tmpfs 507M 0 507M 0% /dev/shm
/dev/sda6 5.3G 3.9G 1.2G 77% /home
/dev/sdb1 32G 2.0G 28G 7% /home2
webserver:~#

It says in log that it couldn't find /var/log/mysql/mysql-bin.index, but the file is there at the right path.

Quote:

webserver:~# ls -l /var/log/mysql
total 3032
-rw-rw---- 1 mysql adm 7498 2008-07-11 06:25 mysql-bin.000363
-rw-rw---- 1 mysql adm 1332844 2008-07-11 19:33 mysql-bin.000364
-rw-rw---- 1 mysql adm 117 2008-07-11 19:33 mysql-bin.000365
-rw-rw---- 1 mysql adm 510697 2008-07-11 19:33 mysql-bin.000366
-rw-rw---- 1 mysql adm 272 2008-07-11 19:33 mysql-bin.000367
-rw-rw---- 1 mysql adm 538 2008-07-11 19:33 mysql-bin.000368
-rw-rw---- 1 mysql adm 695 2008-07-11 19:33 mysql-bin.000369
-rw-rw---- 1 mysql adm 117 2008-07-11 19:33 mysql-bin.000370
-rw-rw---- 1 mysql adm 64015 2008-07-12 06:25 mysql-bin.000371
-rw-rw---- 1 mysql adm 187609 2008-07-15 06:25 mysql-bin.000372
-rw-rw---- 1 mysql adm 40455 2008-07-16 06:25 mysql-bin.000373
-rw-rw---- 1 mysql adm 3923 2008-07-16 09:09 mysql-bin.000374
-rw-rw---- 1 mysql adm 259923 2008-07-17 06:25 mysql-bin.000375
-rw-rw---- 1 mysql adm 22419 2008-07-18 06:25 mysql-bin.000376
-rw-rw---- 1 mysql adm 51715 2008-07-18 18:29 mysql-bin.000377
-rw-rw---- 1 mysql adm 440063 2008-07-20 06:25 mysql-bin.000378
-rw-rw---- 1 mysql adm 109464 2008-07-21 05:19 mysql-bin.000379
-rw-rw---- 1 mysql adm 544 2008-07-20 06:25 mysql-bin.index

Quote:

webserver:~# ls -l /var/run
total 48
srw-rw-rw- 1 root root 0 2008-07-21 05:33 acpid.socket
drwxr-xr-x 2 root root 4096 2007-06-17 22:37 apache2
-rw-r--r-- 1 root root 5 2008-07-21 05:36 apache2.pid
-rw-r--r-- 1 root root 5 2008-07-21 05:33 crond.pid
---------- 1 root root 0 2008-07-21 05:33 crond.reboot
-rw-r--r-- 1 root root 5 2008-07-21 05:32 klogd.pid
-rw-r--r-- 1 root root 355 2008-07-21 05:32 motd
drwxr-xr-x 2 mysql root 4096 2008-07-21 05:19 mysqld
drwxr-xr-x 2 root root 4096 2008-07-21 05:32 network
drwxr-xr-x 2 root root 4096 2007-03-05 17:38 sshd
-rw-r--r-- 1 root root 5 2008-07-21 05:33 sshd.pid
-rw-r--r-- 1 root root 5 2008-07-21 05:32 syslogd.pid
-rw-rw-r-- 1 root utmp 4608 2008-07-23 13:12 utmp
webserver:~#

Quote:

/var/

drwxr-xr-x 2 root root 4096 2008-07-22 07:03 backups
drwxr-xr-x 8 root root 4096 2007-08-19 13:39 cache
drwxr-xr-x 20 root root 4096 2008-07-11 19:33 lib
drwxrwsr-x 2 root staff 4096 2006-10-28 16:06 local
drwxrwxrwt 3 root root 4096 2008-07-21 05:32 lock
drwxr-xr-x 7 root root 4096 2008-07-23 06:25 log
drwxrwsr-x 2 root mail 4096 2007-08-19 13:34 mail
drwxr-xr-x 2 root root 4096 2007-08-19 13:34 opt
drwxr-xr-x 6 root root 4096 2008-07-21 05:36 run
drwxr-xr-x 3 root root 4096 2007-08-19 13:36 spool
drwxrwxrwt 2 root root 4096 2006-10-28 16:06 tmp
drwxr-xr-x 4 root root 4096 2008-07-11 19:45 www
webserver:~# ls -l /var/spool
total 4
drwxr-xr-x 3 root root 4096 2007-08-19 13:36 cron
lrwxrwxrwx 1 root root 7 2007-08-19 13:34 mail -> ../mail
webserver:~#

More ideas would be appreciated.

trickykid · 07-23-2008, 01:29 PM

What does your mysql logs indicate if any instead of the syslog logs?

spaceuser · 07-23-2008, 02:23 PM

That's also strange, since they are empty,

Quote:

-rw-r----- 1 mysql adm 0 2008-07-11 19:33 mysql.err
-rw-r----- 1 mysql adm 0 2008-07-23 06:25 mysql.log
-rw-r----- 1 mysql adm 20 2008-07-22 07:03 mysql.log.1.gz
-rw-r----- 1 mysql adm 20 2008-07-20 06:25 mysql.log.2.gz
-rw-r----- 1 mysql adm 20 2008-07-18 06:25 mysql.log.3.gz
-rw-r----- 1 mysql adm 20 2008-07-17 06:25 mysql.log.4.gz
-rw-r----- 1 mysql adm 20 2008-07-16 06:25 mysql.log.5.gz
-rw-r----- 1 mysql adm 20 2008-07-15 06:25 mysql.log.6.gz
-rw-r----- 1 mysql adm 20 2008-07-12 06:25 mysql.log.7.gz

Another strange thing is that this lib is empty.

Quote:

webserver:~# ls -l /var/run/mysqld | more
total 0
webserver:~#

I have an Ubuntu server as well, and it has two files in same lib, and Ubuntu is based on Debian, so I assume it should have the same files on the same place. In that case, how could files being deleted?

spaceuser · 07-24-2008, 10:10 PM

To continue checking I made a new mysql install on another Debian Etch server, and then compared all mysql files, and I can't figure out what the problem could be anyway.

Any more hints... anyone?