Hi,

Our server is crashing frequently on RHEL and the binary is not made with debug option although we do not strip it. We are getting cores but am not sure from the stack where is the exact problem. We are trying to go deeper and see the assembly calls and interpret them. Please help. I need to know how to interpret the stack when shown in assembly.

Here are the details (merchant is the name of the binary)
$gdb merchant core gives:

Program terminated with signal 11, Segmentation fault

#0 0x081949f3 in basic_uoutputstream<unsigned char>::write ()
(gdb) where
#0 0x081949f3 in basic_uoutputstream<unsigned char>::write ()
#1 0x081a8db8 in DerEncoded::derEncode ()
#2 0x081acf7d in ASN1Seq::derEncode ()
#3 0x081a866c in ASN1Encodable::derEncode1 ()
#4 0x0815a230 in Key::verifyHash ()
#5 0x0815581f in Key::verify ()
#6 0x08155e0d in Key::x509Verify ()
#7 0x0813d755 in ServerCertificateContext::verifySignature ()
#8 0x0814425f in CertStore::verifyPairEx ()
#9 0x0814918a in CertStore::verifyCertChainEx ()
#10 0x08144086 in CertStore::verifyCertChain ()
#11 0x0812f839 in SSL_verify_callback ()
#12 0x081d5c8e in ssl_verify_cert_chain ()
#13 0x083e5340 in ?? ()

And, so the crash is happening in write() function ,which is being passed a character buffer and length of the buffer. We want to see what arguments are being passed.

If for Frame 0, we do a disassemble, we get this:

(gdb) disassemble

Dump of assembler code for function _ZN19basic_uoutputstreamIhE5writeEPKhi:
0x081949dc <_ZN19basic_uoutputstreamIhE5writeEPKhi+0>: push %ebp
0x081949dd <_ZN19basic_uoutputstreamIhE5writeEPKhi+1>: mov %esp,%ebp
0x081949df <_ZN19basic_uoutputstreamIhE5writeEPKhi+3>: push %edi
0x081949e0 <_ZN19basic_uoutputstreamIhE5writeEPKhi+4>: push %esi
0x081949e1 <_ZN19basic_uoutputstreamIhE5writeEPKhi+5>: push %ebx
0x081949e2 <_ZN19basic_uoutputstreamIhE5writeEPKhi+6>: sub $0xc,%esp
0x081949e5 <_ZN19basic_uoutputstreamIhE5writeEPKhi+9>: mov 0x8(%ebp),%edi
0x081949e8 <_ZN19basic_uoutputstreamIhE5writeEPKhi+12>: mov 0xc(%ebp),%esi
0x081949eb <_ZN19basic_uoutputstreamIhE5writeEPKhi+15>: mov 0x10(%ebp),%ebx
0x081949ee <_ZN19basic_uoutputstreamIhE5writeEPKhi+18>: jmp 0x8194a01 <_ZN19basic_uoutputstreamIhE5writeEPKhi+37>
0x081949f0 <_ZN19basic_uoutputstreamIhE5writeEPKhi+20>: sub $0x8,%esp
0x081949f3 <_ZN19basic_uoutputstreamIhE5writeEPKhi+23>: movzbl (%esi),%edx
0x081949f6 <_ZN19basic_uoutputstreamIhE5writeEPKhi+26>: mov (%edi),%eax
0x081949f8 <_ZN19basic_uoutputstreamIhE5writeEPKhi+28>: push %edx
0x081949f9 <_ZN19basic_uoutputstreamIhE5writeEPKhi+29>: push %edi
0x081949fa <_ZN19basic_uoutputstreamIhE5writeEPKhi+30>: call *0x18(%eax)
0x081949fd <_ZN19basic_uoutputstreamIhE5writeEPKhi+33>: inc %esi
0x081949fe <_ZN19basic_uoutputstreamIhE5writeEPKhi+34>: add $0x10,%esp
0x08194a01 <_ZN19basic_uoutputstreamIhE5writeEPKhi+37>: mov %ebx,%eax
0x08194a03 <_ZN19basic_uoutputstreamIhE5writeEPKhi+39>: dec %ebx
0x08194a04 <_ZN19basic_uoutputstreamIhE5writeEPKhi+40>: test %eax,%eax
0x08194a06 <_ZN19basic_uoutputstreamIhE5writeEPKhi+42>: jg 0x81949f0 <_ZN19basic_uoutputstreamIhE5writeEPKhi+20>
0x08194a08 <_ZN19basic_uoutputstreamIhE5writeEPKhi+44>: lea 0xfffffff4(%ebp),%esp
0x08194a0b <_ZN19basic_uoutputstreamIhE5writeEPKhi+47>: pop %ebx
0x08194a0c <_ZN19basic_uoutputstreamIhE5writeEPKhi+48>: pop %esi
0x08194a0d <_ZN19basic_uoutputstreamIhE5writeEPKhi+49>: mov %edi,%eax
0x08194a0f <_ZN19basic_uoutputstreamIhE5writeEPKhi+51>: pop %edi
0x08194a10 <_ZN19basic_uoutputstreamIhE5writeEPKhi+52>: leave
0x08194a11 <_ZN19basic_uoutputstreamIhE5writeEPKhi+53>: ret
0x08194a12 <_ZN19basic_uoutputstreamIhE5writeEPKhi+54>: mov %esi,%esi
End of assembler dump.


Can this dump help us? How to navigate this and interpret this?

Thanks

Biswa

Maybe rebuild your server properly and add debugging prints into log files ?

Hi Sergei,

The intent of the question was to know how to manage and interpret the core using gdb when the ordinary stack does not help. I need help to understand the disassembly output of gdb for example or from the output of gdb's stack, somehow manage to know what are the parameters being passed to the function which is crashing.

Thanks
Biswa

Regarding parameter passing - I think there is a standard convention covering this in C/C++, and I think in C++ it is different ('this' is passed too when it's not a class function).

You can either look it up or write a small piece of C++ code and get annotated (C++ + assembly listing) from 'gcc'.

Even if/when you know the above, it's not for sure you'll be able to debug the issue statically - stack is a volatile thing, so even if you have it dumped somewhere in the 'core' file, it's not for sure its contents reflect what was on it when the failing function was called. I.e. stack may be overwritten.

So, again, if it is your server and if you know the crashing function, better change/wrap it in order to be able to see what is passed to it by standard C++ means.

...

For starters, is the value stack pointer known ? I.e. does the 'core' file have this info in it ?

I forget the gdb command to display the register values. Find that in gdb documentation and display the register values from your core dump.

In the disassembly you posted, the code moves the object address (of the stream object) into edi, and moves the buffer address into esi, and the length into ebx. Then it subtracts at least one from the length in ebx before seg faulting on an access to the buffer.

So the esi and ebx values may tell you a lot about the nature of the malfunction. ESI might be a reasonable looking address that is a multiple of 0x1000, in which case it is likely you overflowed the end of a legitimate buffer. Then ebx would be one less than the amount by which you overflowed.

More likely esi is zero or some other unreasonable value in which case you just have a bad buffer pointer.

For further understanding of the issue, you may need to get the ESP value then dump a lot of memory starting at the address pointed to by esp. You can find the return addresses and ebp values in the memory dump to see the call boundaries in the stack frame. Then you can see calling arguments and local variables at each level.

Parameter passing is simple in 32 bit x86. For functions that don't have the frame pointer optimized away, reading the stack is simple. The crashing function did not have its frame pointer optimized away, but I don't know for sure about functions further up the stack.

The EBP register is the frame pointer. Starting at the address pointed to by EBP there is a sequence of values:
At offset 0: The value of the frame pointer of the caller.
At offset 4: The return address of the caller.
At offset 8: The leftmost arg. But be aware that the this pointer, in any function taking a this pointer, is an implicit leftmost arg before the leftmost arg that appears in your source code.
Beyond offset 8 are the rest of the args.
At negative offsets from the frame pointer, there may be local variables.

Many bugs (not likely this one) trash the stack making this kind of diagnosis hard.

Any C or C++ function is free to overwrite its own calling args if the compiler decides that is convenient. So the values you see in the stack dump might not be the actual calling args.
It is very unlikely for a function to overwrite its calling args unless:
1) The original source code modifies its args
and
2) The function has enough args and local variables that the compiler runs out of registers in generating the code.
Neither of those is true about the function that crashed in the first post. But that might be true of some function up the call stack.

Thanks Johnsfine. You seemed to have hit the bull's eye!! I will take your cues and most likely that should help in solving this issue.

In the disassembly output that I had given, I noticed another thing which supports your view.
(gdb) x /x $esi
0x84b6000: Cannot access memory at address 0x84b6000

And, also note in the disassembly of frame 0:
0x081949f3 <_ZN19basic_uoutputstreamIhE5writeEPKhi+23>: movzbl (%esi),%edx

And then, a gdb>where gives:
(gdb) where
#0 0x081949f3 in basic_uoutputstream<unsigned char>::write ()
#1 0x081a8db8 in DerEncoded::derEncode ()

I think the above observations point to the fact that esi's value was corrupt and thus the buffer being passed was a bad one. Is not it?

You also mentioned about decrement by 1. As you see, the crash happened at write() and write() is simply this:

template<> basic_uoutputstream<impl>&
basic_uoutputstream<impl>::write(const ochar_type * buf, int n)
{
while(n-- > 0)
put(*buf++);
return *this;
}

Is that a problem?

Thanks again for your brilliant observation!!

--Biswa

Why/what for do you have 'const' qualifier ?

Your posts would be more readable if you used code tags around both source code and any large chunks of gdb output.

Is what a problem?

Maybe you already understood this. If not:
At the point of failure, the code being executed corresponds to *buf in the C++ source code, and the esi register contained buf, while the ebx register contained n.

As is obvious from the C++ source code, the value of n in the ebx register has been decremented at least once compared to the value of n passed from the calling function.

The passed values of buf and n are available at 0xc(%ebp) and 0x10(%ebp) so you can tell whether the fault occurred on the first pass through that loop.

Because the buffer is read only from the point of view of a write() function. The write() function reads from the buffer and writes to the stream.

Thanks again Johnsfine. I am not very conversant of assembly.
Can we be sure from the outputs I had given, that the buffer being passed to write() has an invalid address? Can we check whether that is NULL?

I add here some more information on the same disassemble output.
(gdb) x /x 0xc
0xc: 0x00000000
(gdb) x /x 0x10
0x10: 0x00000000

(gdb) x /x $esi
0x84b6000: Cannot access memory at address 0x84b6000

(gdb) x /x $ebx
0x8174177 <_ZN14ArcotExtensionC2ERK13basic_ustringIhEP9CertStoreS3_S3_+1359>: 0xb3d7e824

Do the above help in getting any useful information? If I can be sure that a NULL buffer is causing this crash then probably I can do a defensive check in write() where I will check for NULL. But, if that is not NULL but an invalid address then I cannot do this defensive check.


--Biswa

???

Suppose you implement such a check, and from time to time it will catch NULL. So, there will be no write and no crash, but will your program work correctly ? I.e. since the program calls 'write' is supposed to write something - isn't it ?

The main question in my last comment was whether the outputs I gave there indicate that the buffer passed to write() is NULL or not? And, does it also say anything on the length of the buffer(2nd argument)?

Thanks

Biswa

From what you just posted, it looks like the buffer address was valid, but the length was garbage (far too large), so it ran past the end of the buffer.

You should figure out the gdb commands needed to dump a big chunk of memory pointed to by esp.