LinuxQuestions.org - [SOLVED] Mysql not starting

- Linux - Newbie (https://www.linuxquestions.org/questions/linux-newbie-8/)

- - Mysql not starting (https://www.linuxquestions.org/questions/linux-newbie-8/mysql-not-starting-4175529446/)

Mysql not starting

mysql stopped working over the weekend and I can't get it to restart on my vps
I'm getting this error.

141229 10:00:36 [Note] Plugin 'FEDERATED' is disabled.
141229 10:00:36 InnoDB: The InnoDB memory heap is disabled
141229 10:00:36 InnoDB: Mutexes and rw_locks use GCC atomic builtins
141229 10:00:36 InnoDB: Compressed tables use zlib 1.2.3.4
^G/usr/sbin/mysqld: Can't create/write to file '/tmp/ibHw49jH' (Errcode: 28)
141229 10:00:36 InnoDB: Error: unable to create temporary file; errno: 28
141229 10:00:36 [ERROR] Plugin 'InnoDB' init function returned error.
141229 10:00:36 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
141229 10:00:36 [ERROR] Unknown/unsupported storage engine: InnoDB
141229 10:00:36 [ERROR] Aborting

I've been trying every post i can find but am not getting any further so any help would be amazing

Code:

^G/usr/sbin/mysqld: Can't create/write to file '/tmp/ibHw49jH' (Errcode: 28)

I think it's fairly basic. Error number 28 is "No space left on device."

/tmp is full. Go figure.

My understanding is limited and that's what I thought but when I run df -h

Filesystem Size Used Avail Use% Mounted on
/dev/vda3 49G 19G 27G 42% /
udev 490M 4.0K 490M 1% /dev
tmpfs 200M 224K 199M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 498M 0 498M 0% /run/shm
/dev/vda1 461M 37M 401M 9% /boot

which seems to show lots of space

tmpfs is not /tmp - That's a newbie banana skin, I'm afraid. Your check is interesting, though.

Code:

tmpfs 200M 224K 199M 1% /run

My box shows tmpfs the size of half the total ram fitted in tmpfs. I have 6G; tmpfs is 2.9G. With mysqld loading up it maybe ran out of memory. You can check ram with free and top, (each with man pages). A command like

free -ms 1

will print a free memory report every 1 second (ctrl_c to quit). So

free -ms 1 && <command to start mysql> would tell you how the ram survives as mysql starts. I suspect you have not enough ram in your vm.

free -ms 1 brings back

total used free shared buffers cached
Mem: 995 650 345 0 163 97
-/+ buffers/cache: 388 606
Swap: 476 0 476

and after deleting some log files yesterday mysql seems to be running

Code:

 sudo netstat -tap | grep mysql

tcp        0      0 localhost:mysql        *:*                    LISTEN      1340/mysqld

but I'm still getting the same error on my site

SQLSTATE[HY000]: General error: 1 Can't create/write to file '/tmp/#sql_53c_0.MYI' (Errcode: 28)

which again points to low disk space or inability to write to /tmp

(by the way you have no idea how much I appreciate your help on this)

why not have a look around in /tmp?

Code:

ls -a /tmp

The directory can fill up with crap - I get reams of hidden files with names like
..xfsm-ICE-<random_crap> session manager files
Symlinks 5<random_crap> - pages printed via cups
lilo<random_crap> lilo sessions I presume
dhcp<random_crap> dhcp logons

also .tmp and .part files in abundance. I have this in rc.local to keep the crap down

Code:

rm -rf /tmp/{*.part,*.tmp,gpg*,5*}

that cleans them out every time I boot. Do not do a global rm -rf on /tmp ever. Some of that stuff is/may be needed.

There doesn't seem to be very much there at all

Code:

ls -a /tmp

.  ..  .ICE-unix  .X11-unix

ok, cd there as root. Check what the mysql user is and try

Code:

su <mysql_user>

touch /tmp/somefile

How does that go?

touch /tmp/somefile
touch: cannot touch `/tmp/somefile': No space left on device

That looks to be your problem, which I told you about in post #2. You have the box. Go figure.

If you exit from mysql-user and use root, is there still no space? It could be write permissions (Unlikely)

As a workaround, you may be able to set $TMPDIR to ~/ where they will have write permissions.

Code:

export TMPDIR=~/

Nope same answer with root

This is where my I'm getting lost, The file system shows I have 27g available but there is no space in the /tmp directory, looking in the /tmp directory there doesn't seem to be many files there either.

If I Move the temp directory would that solve the space problem or is that just if there were permission issues.

Again thanks for your patients

It could be a directory issue, or a filesystem problem. Mioving it sounds good

Code:

cd /

du -sh /tmp  #tells the size

mv tmp crap

install -dm 0777 tmp

moved the directory when I got to

Code:

install -dm 0777 tmp

install: cannot change permissions of `tmp': No such file or directory

Tried

Quote:

sudo install -dm 0777 tmp
install: cannot change permissions of `tmp': No such file or directory

Tried restarting mysql

Code:

sudo service mysql restart

mysql stop/waiting

start: Job failed to start

I've just noticed the the error message on the site has now changed to
SQLSTATE[HY000] [2002] Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

when i checked the sql erro file if found this

Code:

141229  9:00:02 [Note] /usr/sbin/mysqld: Normal shutdown



141229  9:00:02 [Note] Event Scheduler: Purging the queue. 0 events

141229  9:00:04  InnoDB: Starting shutdown...

141229  9:00:08  InnoDB: Shutdown completed; log sequence number 4760563155

141229  9:00:08 [Note] /usr/sbin/mysqld: Shutdown complete



141229  9:00:09 [Note] Plugin 'FEDERATED' is disabled.

141229  9:00:09 InnoDB: The InnoDB memory heap is disabled

141229  9:00:09 InnoDB: Mutexes and rw_locks use GCC atomic builtins

141229  9:00:09 InnoDB: Compressed tables use zlib 1.2.3.4

141229  9:00:09 InnoDB: Initializing buffer pool, size = 128.0M

141229  9:00:09 InnoDB: Completed initialization of buffer pool

141229  9:00:09 InnoDB: highest supported file format is Barracuda.

^G/usr/sbin/mysqld: Can't create/write to file '/tmp/ibKuG8ir' (Errcode: 28)

141229  9:00:09  InnoDB: Error: unable to create temporary file; errno: 28

141229  9:00:09  InnoDB: Assertion failure in thread 139994462189376 in file dict0dict.c line 732

InnoDB: Failing assertion: dict_foreign_err_file

InnoDB: We intentionally generate a memory trap.

InnoDB: Submit a detailed bug report to http://bugs.mysql.com.

InnoDB: If you get repeated assertion failures or crashes, even

InnoDB: immediately after the mysqld startup, there may be

InnoDB: corruption in the InnoDB tablespace. Please refer to

InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html

InnoDB: about forcing recovery.

09:00:09 UTC - mysqld got signal 6 ;

This could be because you hit a bug. It is also possible that this binary

or one of the libraries it was linked against is corrupt, improperly built,

or misconfigured. This error can also be caused by malfunctioning hardware.

We will try our best to scrape up some info that will hopefully help

diagnose the problem, but since we have already crashed,

something is definitely wrong and this may fail.



key_buffer_size=16777216

read_buffer_size=131072

max_used_connections=0

max_threads=151

thread_count=0

connection_count=0

It is possible that mysqld could use up to

key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 346681 K  bytes of memory

Hope that's ok; if not, decrease some variables in the equation.



Thread pointer: 0x0

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong...

stack_bottom = 0 thread_stack 0x30000

/usr/sbin/mysqld(my_print_stacktrace+0x29)[0x7f5300869039]

for information, post the output of

Code:

ls -a /