analyzing C program core dump using GDB (Cannot access memory)

m4rtin · 08-31-2010, 06:24 AM

Analyzing /var/log/messages, a process restart caught my attention. According to log file it was restarted thanks to ABRT signal and so there had to be a dump fie. I analyzed core dump using gdb utility and the results are here(lines 100 - 1199 are removed because of LQ 30000 characters limitation):

Code:

(gdb) core pid.core.0
Core was generated by `rpd'.
Program terminated with signal 6, Aborted.
#0  0x88d86437 in ?? ()
(gdb) bt
#0  0x88d86437 in ?? ()
#1  0x88c9b204 in ?? ()
#2  0x000007ea in ?? ()
#3  0x00000006 in ?? ()
#4  0x7fffffdf in ?? ()
#5  0x88c9b1db in ?? ()
#6  0x00000292 in ?? ()
#7  0x88d99354 in ?? ()
#8  0xbfbed8e8 in ?? ()
#9  0x88d851d5 in ?? ()
#10 0x00000006 in ?? ()
#11 0xbfbed8b8 in ?? ()
#12 0x00000000 in ?? ()
#13 0x88d851a2 in ?? ()
#14 0x00000001 in ?? ()
#15 0x88b0ca00 in ?? ()
#16 0xffffffdf in ?? ()
#17 0xffffffff in ?? ()
#18 0xffffffff in ?? ()
#19 0xffffffff in ?? ()
#20 0x88ae94d9 in ?? ()
#21 0x88b0ca00 in ?? ()
#22 0x00000a00 in ?? ()
#23 0xbfbed840 in ?? ()
#24 0x000000a3 in ?? ()
#25 0x88d99354 in ?? ()
#26 0x0891c950 in ?? ()
#27 0x1865b000 in ?? ()
#28 0xbfbed8f8 in ?? ()
#29 0x88d619c7 in ?? ()
#30 0x24282702 in ?? ()
#31 0x00000000 in ?? ()
#32 0xbfbed9c8 in ?? ()
#33 0x084ee22c in ?? ()
#34 0x0891c2f8 in ?? ()
#35 0x0891c354 in ?? ()
#36 0x00000987 in ?? ()
#37 0x0891c950 in ?? ()
#38 0x09bc4000 in ?? ()
#39 0x1311b6a0 in ?? ()
#40 0x0001d938 in ?? ()
#41 0x00000001 in ?? ()
#42 0x00000000 in ?? ()
#43 0x01000000 in ?? ()
#44 0x093d0aa8 in ?? ()
#45 0x02050000 in ?? ()
#46 0x09bc4000 in ?? ()
#47 0x1311b6a0 in ?? ()
#48 0xbfbeda28 in ?? ()
#49 0x0858329e in ?? ()
#50 0x1865b000 in ?? ()
#51 0x00000000 in ?? ()
#52 0x00000000 in ?? ()
#53 0x00000000 in ?? ()
#54 0x00000000 in ?? ()
#55 0x00000000 in ?? ()
#56 0xbfbed968 in ?? ()
#57 0x08598132 in ?? ()
#58 0x00000000 in ?? ()
#59 0x23a5512c in ?? ()
---Type <return> to continue, or q <return> to quit---
#60 0x0000dac8 in ?? ()
#61 0x0857b954 in ?? ()
#62 0x00000000 in ?? ()
#63 0x00000000 in ?? ()
#64 0xbfbed9af in ?? ()
#65 0x0001d9b0 in ?? ()
#66 0x08cf7280 in ?? ()
#67 0x01d5394c in ?? ()
#68 0x0001afde in ?? ()
#69 0x08ce6f50 in ?? ()
#70 0x08ce51e4 in ?? ()
#71 0x0000000c in ?? ()
#72 0xbfbed9c8 in ?? ()
#73 0x0819505f in ?? ()
#74 0x08ce51e4 in ?? ()
#75 0xbfbed9c8 in ?? ()
#76 0x081ecba3 in ?? ()
#77 0x08d11aa0 in ?? ()
#78 0x00000000 in ?? ()
#79 0x00000000 in ?? ()
#80 0x00000000 in ?? ()
#81 0x24282702 in ?? ()
#82 0x09bc4000 in ?? ()
#83 0x00000000 in ?? ()
#84 0xbfbeda18 in ?? ()
#85 0x0853aec0 in ?? ()
#86 0x1865b000 in ?? ()
#87 0x09bc4000 in ?? ()
#88 0x200ed1a7 in ?? ()
#89 0x00000002 in ?? ()
#90 0x00000005 in ?? ()
#91 0x00000000 in ?? ()
#92 0x00000000 in ?? ()
#93 0x00000000 in ?? ()
#94 0x24282754 in ?? ()
#95 0x0000000c in ?? ()
#96 0x09bc4200 in ?? ()
#97 0x08577533 in ?? ()
#98 0x1964b0b8 in ?? ()
#99 0x24282774 in ?? ()
#1200 0x00000000 in ?? ()
#1201 0x00000000 in ?? ()
#1202 0x00000000 in ?? ()
#1203 0x00000000 in ?? ()
#1204 0x00000000 in ?? ()
#1205 0x00000000 in ?? ()
#1206 0x00000000 in ?? ()
#1207 0x00000000 in ?? ()
#1208 0x00000000 in ?? ()
#1209 0x00000000 in ?? ()
#1210 0x00000000 in ?? ()
#1211 0x00000000 in ?? ()
#1212 0x00000000 in ?? ()
#1213 0x00000000 in ?? ()
#1214 0x00000000 in ?? ()
#1215 0x00000000 in ?? ()
#1216 0x00000000 in ?? ()
#1217 0x00000000 in ?? ()
#1218 0x00000000 in ?? ()
#1219 0x00000000 in ?? ()
#1220 0x00000000 in ?? ()
#1221 0x00000000 in ?? ()
#1222 0x00000000 in ?? ()
#1223 0x00000000 in ?? ()
#1224 0x00000000 in ?? ()
#1225 0x00000000 in ?? ()
#1226 0x00000000 in ?? ()
#1227 0x00000000 in ?? ()
#1228 0x00000000 in ?? ()
#1229 0x00000000 in ?? ()
#1230 0x00000000 in ?? ()
#1231 0x00000000 in ?? ()
#1232 0x00000000 in ?? ()
#1233 0x00000000 in ?? ()
#1234 0x00000000 in ?? ()
#1235 0x00000000 in ?? ()
#1236 0x00000000 in ?? ()
#1237 0x00000000 in ?? ()
#1238 0x00000000 in ?? ()
#1239 0x00000000 in ?? ()
#1240 0x00000000 in ?? ()
#1241 0x00000000 in ?? ()
#1242 0x00000000 in ?? ()
#1243 0x00000000 in ?? ()
#1244 0x00000000 in ?? ()
#1245 0x00000000 in ?? ()
#1246 0x00000000 in ?? ()
#1247 0x00000000 in ?? ()
#1248 0x00000000 in ?? ()
#1249 0x00000000 in ?? ()
#1250 0x00000000 in ?? ()
#1251 0x00000000 in ?? ()
#1252 0x00000000 in ?? ()
#1253 0x00000000 in ?? ()
#1254 0x00000000 in ?? ()
#1255 0x00000000 in ?? ()
#1256 0x00000000 in ?? ()
#1257 0x00000000 in ?? ()
#1258 0x00000000 in ?? ()
#1259 0x00000000 in ?? ()
---Type <return> to continue, or q <return> to quit---
#1260 0x00000000 in ?? ()
#1261 0x00000000 in ?? ()
#1262 0x00000000 in ?? ()
#1263 0x00000000 in ?? ()
#1264 0x00000000 in ?? ()
#1265 0x00000000 in ?? ()
#1266 0x00000000 in ?? ()
#1267 0x00000000 in ?? ()
#1268 0x00000000 in ?? ()
#1269 0x00000000 in ?? ()
#1270 0x00000000 in ?? ()
#1271 0x00000000 in ?? ()
#1272 0x00000000 in ?? ()
#1273 0x00000000 in ?? ()
#1274 0x00000000 in ?? ()
#1275 0x00000000 in ?? ()
#1276 0x00000000 in ?? ()
#1277 0x00000000 in ?? ()
#1278 0x00000000 in ?? ()
#1279 0x00000000 in ?? ()
#1280 0x00000000 in ?? ()
#1281 0x00000000 in ?? ()
#1282 0x00000000 in ?? ()
#1283 0x00000000 in ?? ()
#1284 0x00000000 in ?? ()
#1285 0x00000000 in ?? ()
#1286 0x00000000 in ?? ()
#1287 0x00000000 in ?? ()
#1288 0x00000000 in ?? ()
#1289 0x00000000 in ?? ()
#1290 0x00000000 in ?? ()
#1291 0x00000000 in ?? ()
#1292 0x00000000 in ?? ()
#1293 0x00000000 in ?? ()
#1294 0x00000000 in ?? ()
#1295 0x00000000 in ?? ()
#1296 0x00000000 in ?? ()
#1297 0x00000000 in ?? ()
#1298 0x00000000 in ?? ()
#1299 0x00000000 in ?? ()
#1300 0x00000000 in ?? ()
#1301 0x00000000 in ?? ()
#1302 0x00000000 in ?? ()
#1303 0x00000000 in ?? ()
#1304 0x00000000 in ?? ()
#1305 0x00000000 in ?? ()
#1306 0x00000000 in ?? ()
#1307 0x00000000 in ?? ()
#1308 0x00000000 in ?? ()
#1309 0x00000000 in ?? ()
#1310 0x00000000 in ?? ()
#1311 0x00000000 in ?? ()
#1312 0x00000000 in ?? ()
#1313 0x00000000 in ?? ()
#1314 0x00000000 in ?? ()
#1315 0x00000000 in ?? ()
#1316 0x00000000 in ?? ()
#1317 0x00000000 in ?? ()
#1318 0x00000000 in ?? ()
#1319 0x00000000 in ?? ()
---Type <return> to continue, or q <return> to quit---
#1320 0x00000000 in ?? ()
#1321 0x00000000 in ?? ()
#1322 0x00000000 in ?? ()
#1323 0x00000000 in ?? ()
#1324 0x00000000 in ?? ()
#1325 0x00000000 in ?? ()
#1326 0x00000000 in ?? ()
#1327 0x00000000 in ?? ()
#1328 0x00000000 in ?? ()
#1329 0x00000000 in ?? ()
#1330 0x00000000 in ?? ()
#1331 0x00000000 in ?? ()
#1332 0x00000000 in ?? ()
#1333 0x00000000 in ?? ()
#1334 0x00000000 in ?? ()
#1335 0x00000000 in ?? ()
#1336 0x00000000 in ?? ()
#1337 0x00000000 in ?? ()
#1338 0x00000000 in ?? ()
#1339 0x00000000 in ?? ()
#1340 0x00000000 in ?? ()
#1341 0x00000000 in ?? ()
#1342 0x00000000 in ?? ()
#1343 0x00000000 in ?? ()
#1344 0x00000000 in ?? ()
#1345 0x00000000 in ?? ()
#1346 0x00000000 in ?? ()
#1347 0x00000000 in ?? ()
#1348 0x00000000 in ?? ()
#1349 0x00000000 in ?? ()
#1350 0x00000000 in ?? ()
#1351 0x00000000 in ?? ()
#1352 0x00000000 in ?? ()
#1353 0x00000000 in ?? ()
#1354 0x00000000 in ?? ()
#1355 0x00000000 in ?? ()
#1356 0x00000000 in ?? ()
#1357 0x00000000 in ?? ()
#1358 0x00000000 in ?? ()
#1359 0x00000000 in ?? ()
#1360 0x00000000 in ?? ()
#1361 0x00000000 in ?? ()
#1362 0x00000000 in ?? ()
#1363 0x00000000 in ?? ()
#1364 0x00000000 in ?? ()
#1365 0x00000000 in ?? ()
#1366 0x00000000 in ?? ()
#1367 0x00000000 in ?? ()
#1368 0x00000000 in ?? ()
#1369 0x00000000 in ?? ()
#1370 0x00000000 in ?? ()
#1371 0x00000000 in ?? ()
#1372 0x00000000 in ?? ()
#1373 0x00000000 in ?? ()
#1374 0x00000000 in ?? ()
#1375 0x00000000 in ?? ()
#1376 0x00000000 in ?? ()
#1377 0x00000000 in ?? ()
#1378 0x00000000 in ?? ()
#1379 0x00000000 in ?? ()
---Type <return> to continue, or q <return> to quit---
#1380 0x00000000 in ?? ()
#1381 0x00000000 in ?? ()
#1382 0x00000000 in ?? ()
#1383 0x00000000 in ?? ()
#1384 0x00000000 in ?? ()
#1385 0x00000000 in ?? ()
#1386 0x00000000 in ?? ()
#1387 0x00000000 in ?? ()
#1388 0x00000000 in ?? ()
#1389 0x00000000 in ?? ()
#1390 0x00000000 in ?? ()
#1391 0x00000000 in ?? ()
#1392 0x00000000 in ?? ()
#1393 0x00000000 in ?? ()
#1394 0x00000000 in ?? ()
#1395 0x00000000 in ?? ()
#1396 0x00000000 in ?? ()
#1397 0x00000000 in ?? ()
#1398 0x00000000 in ?? ()
#1399 0x00000000 in ?? ()
#1400 0x00000000 in ?? ()
#1401 0x00000000 in ?? ()
#1402 0x00000000 in ?? ()
#1403 0x00000000 in ?? ()
#1404 0x00000000 in ?? ()
#1405 0x00000000 in ?? ()
#1406 0x00000000 in ?? ()
#1407 0x00000000 in ?? ()
#1408 0x00000000 in ?? ()
#1409 0x00000000 in ?? ()
#1410 0x00000000 in ?? ()
#1411 0x00000000 in ?? ()
#1412 0x00000000 in ?? ()
#1413 0x00000000 in ?? ()
#1414 0x00000000 in ?? ()
#1415 0x00000000 in ?? ()
#1416 0x00000000 in ?? ()
#1417 0x00000000 in ?? ()
#1418 0x00000000 in ?? ()
#1419 0x00000000 in ?? ()
#1420 0x00000000 in ?? ()
#1421 0x00000000 in ?? ()
#1422 0x00000000 in ?? ()
#1423 0x00000000 in ?? ()
#1424 0x00000000 in ?? ()
#1425 0x00000000 in ?? ()
#1426 0x00000000 in ?? ()
#1427 0x00000000 in ?? ()
#1428 0x00000000 in ?? ()
#1429 0x00000000 in ?? ()
#1430 0x00000000 in ?? ()
#1431 0x00000000 in ?? ()
#1432 0x00000000 in ?? ()
#1433 0x00000000 in ?? ()
#1434 0x00000000 in ?? ()
#1435 0x00000000 in ?? ()
#1436 0x00000000 in ?? ()
#1437 0x00000000 in ?? ()
#1438 0x00000000 in ?? ()
#1439 0x00000000 in ?? ()
---Type <return> to continue, or q <return> to quit---
#1440 0x00000000 in ?? ()
#1441 0x00000000 in ?? ()
#1442 0x00000000 in ?? ()
#1443 0x00000000 in ?? ()
#1444 0x00000000 in ?? ()
#1445 0x00000000 in ?? ()
#1446 0x00000000 in ?? ()
#1447 0x00000000 in ?? ()
#1448 0x00000000 in ?? ()
#1449 0x00000000 in ?? ()
#1450 0x00000000 in ?? ()
#1451 0x00000000 in ?? ()
#1452 0x00000000 in ?? ()
#1453 0x00000000 in ?? ()
#1454 0x00000000 in ?? ()
#1455 0x00000000 in ?? ()
#1456 0x00000000 in ?? ()
#1457 0x00000000 in ?? ()
#1458 0x00000000 in ?? ()
#1459 0x00000000 in ?? ()
#1460 0x00000000 in ?? ()
#1461 0x00000000 in ?? ()
#1462 0x00000000 in ?? ()
#1463 0x00000000 in ?? ()
#1464 0x00000000 in ?? ()
#1465 0x00000000 in ?? ()
#1466 0x00000000 in ?? ()
#1467 0x00000000 in ?? ()
#1468 0x00000000 in ?? ()
#1469 0x00000000 in ?? ()
#1470 0x00000000 in ?? ()
#1471 0x00000000 in ?? ()
#1472 0x00000000 in ?? ()
#1473 0x00000000 in ?? ()
#1474 0x00000000 in ?? ()
#1475 0x00000000 in ?? ()
#1476 0x00000000 in ?? ()
#1477 0x00000000 in ?? ()
#1478 0x102454ff in ?? ()
#1479 0x2024448d in ?? ()
#1480 0x5440f750 in ?? ()
#1481 0x00020000 in ?? ()
#1482 0x688e0375 in ?? ()
#1483 0x01a1b814 in ?? ()
#1484 0xcd500000 in ?? ()
#1485 0x90feeb80 in ?? ()
#1486 0x102454ff in ?? ()
#1487 0x1424448d in ?? ()
#1488 0x5440f750 in ?? ()
#1489 0x00020000 in ?? ()
#1490 0x688e0375 in ?? ()
#1491 0x0158b814 in ?? ()
#1492 0xcd500000 in ?? ()
#1493 0x90feeb80 in ?? ()
#1494 0x102454ff in ?? ()
#1495 0x1424448d in ?? ()
#1496 0x1840f750 in ?? ()
#1497 0x00020000 in ?? ()
#1498 0x688e0375 in ?? ()
#1499 0x0067b844 in ?? ()
---Type <return> to continue, or q <return> to quit---
#1500 0xcd500000 in ?? ()
#1501 0x90feeb80 in ?? ()
#1502 0xbfbedef4 in ?? ()
#1503 0x00000002 in ?? ()
#1504 0xbfbedf00 in ?? ()
#1505 0x00000000 in ?? ()
Cannot access memory at address 0xbfbef000
(gdb)

What might be the cause of this crash? Or is it possible to further debug this problem(it's not open source program)?

raconteur · 08-31-2010, 01:41 PM

The symbols have been stripped from the executable so the core file is not very useful. It is possible, but extremely tedious, to trace through a core file with a stripped executable or library but you do need the source to make any meaningful use of it and if you have that it just makes sense to build an unstripped version for debugging.

So the answers are a) there is no reasonable way to tell why the program crashed. b) there is no reasonable way to further debug this crash instance.

Mara · 08-31-2010, 03:10 PM

It seems to me (because of many 0 addresses) that the program went out of stack (stack overflow, overwritten return address or similar problem). If not, 1500 functions on backtrace suggest a deep (infinite?) recursion. But without the debug symbols it'll be very hard to find out exactly.

johnsfine · 08-31-2010, 03:27 PM

Expanding on what Mara said:

Quote:

Originally Posted by m4rtin

Code:

#0  0x88d86437 in ?? ()
#1  0x88c9b204 in ?? ()
#2  0x000007ea in ?? ()

Line #2 in that backtrace is garbage. Once one line is garbage, it is unlikely that any further lines are OK.

I wouldn't say with confidence that lines #0 and #1 are OK, but they are the most that might be OK in that backtrace.

So you can be pretty sure that something trashed some copy of the frame pointer, so gdb can't give a correct backtrace.

If you knew a bit about gdb and asm code, you could look at the disassembly around addresses 0x88d86437 and 0x88c9b204 to see if that code is plausible enough to be valid backtrace levels.

If they're not plausible, you're totally lost. But even if they are, you're pretty much lost. If they are valid stack frames then the object that overran is two frames up the stack from where the crash happened, which makes it likely the bug was relatively long before the crash in execution history. Those are hard to find even when you have symbols and source code.

Quote:

What might be the cause of this crash? Or is it possible to further debug this problem(it's not open source program)?

If you knew assembler well, you could probably easily find the data structure on the stack that had the overrun that caused the crash (I'm assuming but can't be certain that a stacked data structure overrun caused this crash). Then you could look at the data structure. If it happens to contain readable text, you might guess what chunk of data the program tried to process that exceeds some unenforced limit. That's all the most optimistic answer to your question.

The more realistic answer is that it is impossible to diagnose. If you were someone for whom such diagnosis is really hard (rather than impossible) you wouldn't have needed to ask.