Hello, I have an interesting segfault that is occurring whenever mysql is restarted. The program in question retries mysql queries if the database is unavailable in addition to reconnecting if a connection goes bad by setting the OPT_RECONNECT option. I have provided a snapshot of what is going on when I restart MySQL below.

It doesn't appear that the MYSQL connection object is NULL but I am wondering if somehow I am referencing an invalid memory location or something. Any help in diagnosing where to start looking would be greatly appreciated! Thanks!

Brian



[root@server-a dev]# /etc/init.d/mysql restart
Shutting down MySQL........ [ OK ]
Starting MySQL.*** glibc detected *** /home/user/a.out: double free or corruption (!prev): 0x000000000b64dd00 ***
======= Backtrace: =========
/lib64/libc.so.6[0x3739471634]
/lib64/libc.so.6(cfree+0x8c)[0x3739474c5c]
/home/user/a.out[0x44f227]
/home/user/a.out[0x452699]
/home/user/a.out[0x44f3fe]
/home/user/a.out[0x42d313]
/home/user/a.out[0x42f89e]
/home/user/a.out(__gxx_personality_v0+0x44d)[0x409835]
/lib64/libpthread.so.0[0x373a0062f7]
/lib64/libc.so.6(clone+0x6d)[0x37394d1b6d]
======= Memory map: ========
00400000-005a7000 r-xp 00000000 08:03 3797972 /home/user/a.out
007a7000-007e2000 rw-p 001a7000 08:03 3797972 /home/user/a.out
007e2000-007e5000 rw-p 007e2000 00:00 0
0b636000-0b6a4000 rw-p 0b636000 00:00 0
40f67000-40f68000 ---p 40f67000 00:00 0
40f68000-41968000 rw-p 40f68000 00:00 0
41dce000-41dcf000 ---p 41dce000 00:00 0
41dcf000-427cf000 rw-p 41dcf000 00:00 0
477d7000-477d8000 ---p 477d7000 00:00 0
477d8000-481d8000 rw-p 477d8000 00:00 0
481d8000-481d9000 ---p 481d8000 00:00 0
481d9000-48bd9000 rw-p 481d9000 00:00 0
48bd9000-48bda000 ---p 48bd9000 00:00 0
48bda000-495da000 rw-p 48bda000 00:00 0
495da000-495db000 ---p 495da000 00:00 0
495db000-49fdb000 rw-p 495db000 00:00 0
3739000000-373901a000 r-xp 00000000 08:03 2387691 /lib64/ld-2.5.so
373921a000-373921b000 r--p 0001a000 08:03 2387691 /lib64/ld-2.5.so
373921b000-373921c000 rw-p 0001b000 08:03 2387691 /lib64/ld-2.5.so
373974d000-373974e000 rw-p 0014d000 08:03 2387692 /lib64/libc-2.5.so
373974e000-3739753000 rw-p 373974e000 00:00 0
3739c00000-3739c82000 r-xp 00000000 08:03 2387702 /lib64/libm-2.5.so
3739e82000-3739e83000 rw-p 00082000 08:03 2387702 /lib64/libm-2.5.so
373a000000-373a015000 r-xp 00000000 08:03 2387697 /lib64/libpthread-2.5.so
373a215000-373a216000 rw-p 00015000 08:03 2387697 /lib64/libpthread-2.5.so
373a216000-373a21a000 rw-p 373a216000 00:00 0
374aa00000-374aae6000 r-xp 00000000 08:03 3816642 /usr/lib64/libstdc++.so.6.0.8
374acee000-374ad00000 rw-p 374acee000 00:00 0
374ae00000-374ae0d000 r-xp 00000000 08:03 2387703 /lib64/libgcc_s-4.1.2-20080102.so.1
2aaaac000000-2aaaac021000 rw-p 2aaaac000000 00:00 0
2aaaac021000-2aaab0000000 ---p 2aaaac021000 00:00 0
2b31820fa000-2b3182104000 r-xp 00000000 08:03 2387420 /lib64/libnss_files-2.5.so
2b3182304000-2b3182305000 rw-p 0000a000 08:03 2387420 /lib64/libnss_files-2.5.so
7fff289cb000-7fff289e1000 rw-p 7fff289cb000 00:00 0 [stack]
ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0 [vdso]
[ OK ]


[root@server-a dev]# gdb -c ./core.24618 /home/user/a.out

Reading symbols from /lib64/libpthread.so.0...done.
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /usr/lib64/libstdc++.so.6...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libm.so.6...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libgcc_s.so.1...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/libc.so.6...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libnss_files.so.2...done.
Loaded symbols for /lib64/libnss_files.so.2
Core was generated by `/home/user/a.out'.
Program terminated with signal 6, Aborted.
#0 0x0000003739430155 in raise () from /lib64/libc.so.6
(gdb) bt
#0 0x0000003739430155 in raise () from /lib64/libc.so.6
#1 0x0000003739431bf0 in abort () from /lib64/libc.so.6
#2 0x000000373946a38b in __libc_message () from /lib64/libc.so.6
#3 0x0000003739471634 in _int_free () from /lib64/libc.so.6
#4 0x0000003739474c5c in free () from /lib64/libc.so.6
#5 0x000000000044f227 in end_server (mysql=0x7fff289dd600) at client.c:949
#6 0x0000000000452699 in cli_advanced_command (mysql=0x7fff289dd600, command=<value optimized out>, header=0x0,
header_length=0, arg=0x0, arg_length=0, skip_check=1 '\001', stmt=0x0) at client.c:812
#7 0x000000000044f37a in mysql_close (mysql=0x7fff289dd600) at client.c:2755
#8 0x000000000044f3fe in mysql_close (mysql=0x41966aa0) at client.c:2770
#9 0x00000000004524ba in mysql_reconnect (mysql=0x41966aa0) at client.c:2581
#10 0x00000000004526fd in cli_advanced_command (mysql=0x41966aa0, command=COM_QUERY, header=0x0, header_length=0,
arg=0x41966100 "SELECT ComponentID FROM Component WHERE IPAddress='10.10.75.5' AND Hostname='server-a' AND Source=1",
arg_length=103, skip_check=1 '\001', stmt=0x0) at client.c:781
#11 0x000000000044ed71 in mysql_send_query (mysql=0x41966aa0,
query=0x41966100 "SELECT ComponentID FROM Component WHERE IPAddress='10.10.75.5' AND Hostname='server-a' AND Source=1", length=103) at client.c:2894
#12 0x000000000044edd9 in mysql_real_query (mysql=0x602a, query=0x602d <Address 0x602d out of bounds>, length=6)
at client.c:2905
#13 0x000000000042b42f in do_query (conn=0x41966aa0, lm=0xb6365b0,
msg=0x41966100 "SELECT ComponentID FROM Component WHERE IPAddress='10.10.75.5' AND Hostname='server-a' AND Source=1",
maxfail=60, logfreq=10, retry=1) at dosql.cpp:20
#14 0x000000000042d313 in handleControlMessage (conn=0x41966aa0, lm=0xb6365b0, source=<value optimized out>,
type=<value optimized out>, payload=0xb6899d0 "3", oproc=<value optimized out>, ip=0xb6397f8 "10.10.75.5",
host=0xb65a500 "server-a", ets=0x41966a50 "20111215134037", connts=0x7e25d0 "20111215134007") at mysqllib.cpp:293
#15 0x000000000042f89e in manualHB (c=
{net = {vio = 0x0, buff = 0x0, buff_end = 0xb63c5f0 "�\034", write_pos = 0xb63a5f0 "\001", read_pos = 0xb63a5f0 "\001", fd = 4, remain_in_buf = 0, length = 0, buf_length = 0, where_b = 0, max_packet = 8192, max_packet_size = 1073741824, pkt_nr = 1, compress_pkt_nr = 0, write_timeout = 31536000, read_timeout = 31536000, retry_count = 1, fcntl = 0, return_status = 0x0, reading_or_writing = 0 '\0', save_char = 0 '\0', unused0 = 0 '\0', unused = 0 '\0', compress = 0 '\0', unused1 = 0 '\0', query_cache_query = 0x0, last_errno = 2002, error = 2 '\002', unused2 = 0 '\0', return_errno = 0 '\0', last_error = "Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)", '\0' <repeats 429 times>, sqlstate = "HY000", extension = 0x0}, connector_fd = 0x0, host = 0xb64d740 "localhost", user = 0x0, passwd = 0x0, unix_socket = 0xb64d750 "/var/lib/mysql/mysql.sock---Type <return> to continue, or q <return> to quit---0x000000000b64dd00
", server_version = 0xb64d770 "5.1.46-community", host_info = 0x0, info = 0x0, db = 0x0, charset = 0x7ad860, fields = 0x0, field_alloc = {free = 0x0, used = 0x0, pre_alloc = 0x0, min_malloc = 32, block_size = 8160, block_num = 4, first_block_usage = 0, error_handler = 0}, affected_rows = 18446744073709551615, insert_id = 0, extra_info = 0, thread_id = 23, packet_length = 0, port = 3306, client_flag = 41613, server_capabilities = 63487, protocol_version = 10, field_count = 0, server_status = 2, server_language = 8, warning_count = 0, options = {connect_timeout = 0, read_timeout = 0, write_timeout = 0, port = 0, protocol = 0, client_flag = 0, host = 0x0, user = 0x0, password = 0x0, unix_socket = 0x0, db = 0x0, init_commands = 0x0, my_cnf_file = 0x0, my_cnf_group = 0x0, charset_dir = 0x0, charset_name = 0x0, ssl_key = 0x0, ssl_cert = 0x0, ssl_ca = 0x0, ssl_capath = 0x0, ssl_cipher = 0x0, shared_memory_base_name = 0x0, max_allowed_packet = 0, use_ssl = 0 '\0', compress = 0 '\0', named_pipe = 0 '\0', rpl_probe = 0 '\0', rpl_parse = 0 '\0', no_master_reads = 0 '\0', separate_thread = 0 '\0', methods_to_use = MYSQL_OPT_CONNECT_TIMEOUT, client_ip = 0x0, secure_auth = 0 '\0', report_data_truncation = 0 '\0', local_infile_init = 0, local_infile_read = 0, local_infile_end = 0, local_infile_error = 0, local_infile_userdata = 0x0, extension = 0x0}, status = MYSQL_STATUS_READY, free_me = 0 '\0', reconnect = 116 't', scramble = "eyQf1^fgP<ypDDgZo=$8", rpl_pivot = 1 '\001', master = 0x7fff289dd600, next_slave = 0x7fff289dd600, last_used_slave = 0x0, last_used_con = 0x41966aa0, stmts = 0x0, methods = 0x7e0a40, thd = 0x0, unbuffered_fetch_owner = 0x0, info_buffer = 0x0, extension = 0x0}, source=65281, lm=0xb6365b0, connts=0x7e25d0 "20111215134007",
compname=0x4a4534 "main", hbcount=3, ip=0xb6397f8 "10.10.75.5") at mysqllib.cpp:206
#16 0x0000000000409835 in doHeartbeat () at main.cpp:668
#17 0x000000373a0062f7 in start_thread () from /lib64/libpthread.so.0
#18 0x00000037394d1b6d in clone () from /lib64/libc.so.6
(gdb) x 0x000000000b64dd00
0xb64dd00: 0x3974ebf0
(gdb)

Hi.

Did you tried to compile your program with `-g' option to gcc and then debug it in gdb? I suspect you've forgot to set some pointer to NULL after freeing it.

Thanks for the response, I did compile with debugging turned on and have been using gdb. It definitely has something to do with the connection object after MySQL restarts. My work around at the moment is to replace the persistent connection with a new connection everytime this code executes. It eliminates the stacktrace (although at the cost of not being as efficient).

Hi.
I feel deja vu now.. Do you have some explicit calls to `free(some_var)' in your code? Do you set `some_var=NULL;' afterwards?

1 members found this post helpful.

must be a glitch in the matrix!

No I don't explicitly call 'free' or set my connection variable to NULL, at least I don't believe. I'll double check. I know for sure that the MYSQL connection object is valid (because I'm constantly using it), its during a restart that I'm relying on the mysql RECONNECT option to handle the condition where the database drops offline for a minute.

This line of the core dump has me believing that somehow my connection has gotten hosed:

#12 0x000000000044edd9 in mysql_real_query (mysql=0x602a, query=0x602d <Address 0x602d out of bounds>, length=6)
at client.c:2905

Hi.
According to this prototype
your `query' (a string) is invalid pointer. Where is that pointer comes from? What that string is supposed to contain (there should be 6 characters)? This link mention that you can identify connection failures by checking return value of mysql_real_query().

I believe that in your case the problem is not with connection, but with invalid query string. It is freed somewhere else in the code, possibly on reconnection event..

1 members found this post helpful.

Thanks firstfire, I don't remember if I used a char* or a string type for the query. i will check on it monday.

the query is built using a 'string' type which is converted to a char * by calling the c_str() function. It is then passed to the mysql_real_query. So I don't think it is a memory allocation problem with the query.

Well, to check that simply pass constant string query (I mean something in double quotes; don't forget to specify its length) to mysql_real_query() and see if program crashes on mysql restart.

I replaced the query string with a constant, quoted string such as "select count(*) from tableA" and the segfault still occurs after restarting mysql... it doesn't make sense. I am wondering if there is some sort of bug in the reconnect code with this version of mysql (5.1.46).

Hi.
Can you post core dump for this case?

All I have is what I posted at the top of the thread, which shows the glibc double free corruption message and the analysis of the core file in gdb. Is there something else that would be helpful?