[SOLVED] Stumped while learning ASM. I know what to do to make it work, but can't grasp why.

Hi everybody.
I am learning x86_64 Linux assembly language. I got through the basics, making syscalls, comparisons and conditional jumps, accessing memory, registers, all that stuff. However, now I am trying to learn to write programs which use external libraries. Calling other functions is no problem, but when another piece of code is calling my function, that is where I am confused.
According to the amd64 ABI and Linux calling convention, I need to save some specific registers including RBP and RSP, which are the stack base pointer and stack pointer.
But if I don't use the stack, and all the functions that I call are responsible to save those registers correctly, shouldn't everything work if I never push or pop anything or modify those registers?
Even if something goes wrong, I think it would segfault on the ret instruction, but that is not what happens, my program segfaults at the first function call instead. I can't understand why that happens.

Also, I can get everything to run properly by simply pushing RBP onto the stack at the beginning of the function and popping it off at the end. So does the segfault have something to do with the state of the stack?

I have been searching for the answer, and some resources come close, but nothing explains why that needs to be done.
The best one I found so far was this: http://cs.nyu.edu/courses/fall11/CSC.../x64-intro.pdf
It almost explains it, but doesn't quite get there.

Others show what needs to be done, but don't explain why, like this one: http://vikaskumar.org/amd64/sample.htm

Also, compiling a small C program and examining the resultant ASM code shows the same thing: PUSH RBP and then set RBP to RSP. Which I think I understand, but I can't figure out why it segfaults before the RET instruction if I don't do that.

Originally Posted by prushik But if I don't use the stack, and all the functions that I call are responsible to save those registers correctly, shouldn't everything work if I never push or pop anything or modify those registers?

Yes. If you never modify those registers (even if you do use the stack) then you don't need to save/restore them. If other functions you call modify those registers, it is their responsibility to restore them.

my program segfaults at the first function call instead. I can't understand why that happens.

At the function call? Or do you mean somewhere inside that function?

Also, I can get everything to run properly by simply pushing RBP onto the stack at the beginning of the function and popping it off at the end. So does the segfault have something to do with the state of the stack?

Sounds like stack alignment. In x86_64, you shouldn't simply call another function on entry to your function. The call into your function misaligned the stack by 8 bytes. You need to push another 8 bytes (or any odd multiple of 8 bytes) onto the stack before calling another function.

I can't figure out why it segfaults before the RET instruction if I don't do that.

Very few functions would crash due to stack misalignment. Some would just run slower. Others wouldn't care at all. But I don't have a guess other than stack misalignment based on your description. If stack misalignment is the problem, it should crash deep inside the other function you called on some operation that depends on stack alignment.

You "fixed" the problem by saving and restoring rbp, creating what I think is an illusion that you needed to save/restore rbp. Try pushing/popping r11 instead of rbp. We know nothing cares about saving/restoring r11, so when that also "fixes" the problem, you have pretty much confirmed it was stack alignment.

Originally Posted by johnsfine Yes. If you never modify those registers (even if you do use the stack) then you don't need to save/restore them. If other functions you call modify those registers, it is their responsibility to restore them.

Ok, sounds good.

Originally Posted by johnsfine At the function call? Or do you mean somewhere inside that function?

Probably inside the function.

Originally Posted by johnsfine Sounds like stack alignment. In x86_64, you shouldn't simply call another function on entry to your function. The call into your function misaligned the stack by 8 bytes. You need to push another 8 bytes (or any odd multiple of 8 bytes) onto the stack before calling another function.

Stack alignment. Good call. You are sharp my friend.

Originally Posted by johnsfine Very few functions would crash due to stack misalignment. Some would just run slower. Others wouldn't care at all. But I don't have a guess other than stack misalignment based on your description. If stack misalignment is the problem, it should crash deep inside the other function you called on some operation that depends on stack alignment.

After reading your comment, I did some research and found that some SSE instructions will segfault if the stack is misaligned. I am calling SDL, which is built with SSE and SSE2 instructions enabled. So that sounds plausible. Awesome.

Originally Posted by johnsfine You "fixed" the problem by saving and restoring rbp, creating what I think is an illusion that you needed to save/restore rbp. Try pushing/popping r11 instead of rbp. We know nothing cares about saving/restoring r11, so when that also "fixes" the problem, you have pretty much confirmed it was stack alignment.

I think you are right. I did try your suggestion of using r11 instead and it worked fine, so think your answer is correct. Thank you. I understand much better now.

However, this seems a bit awkward now, I need to align the stack by pushing some random data onto, even though I don't need the data. Is there a more graceful way to align the stack?

Originally Posted by prushik ...Is there a more graceful way to align the stack?

Explicitly modify stack pointer.

Personally I prefer push/pop for stack alignment (preferably with a comment saying just for stack alignment).

Maybe you think it is more graceful to use

  sub $8,%rsp
  ...
  add $8,%rsp

That is a valid opinion if you like.
Sometimes that code is also faster than push/pop but very unlikely to make a difference that matters and there are many factors (instruction alignment etc.) that might overwhelm the superficial time difference and even reverse which version is slightly faster.

Originally Posted by Sergei Steshenko Explicitly modify stack pointer.

Sounds good, I can just do something like:
sub rsp,8

If I do that, then I have to restore it also before I return, should I use:
add rsp,8

If I have to restore rsp at the end, isn't it just as graceful to save rsp on the stack and pop it off at the end?

I haven't used x86 assembly in a while, but you don't align a stack by adding a constant value to it - what if it's already aligned? You usually OR the stack pointer - e.g. OR it with 0xfffffff0 to align a 32 bit stack pointer by 16 bytes. And IIRC, you save %esp as %ebp on function entry for x86, so you shouldn't need undo what you've done at the end (again, my x86 assembler is rusty...).

Originally Posted by JohnGraham I haven't used x86 assembly in a while, but you don't align a stack by adding a constant value to it - what if it's already aligned? You usually OR the stack pointer - e.g. OR it with 0xfffffff0 to align a 32 bit stack pointer by 16 bytes. And IIRC, you save %esp as %ebp on function entry for x86, so you shouldn't need undo what you've done at the end (again, my x86 assembler is rusty...).

Your discussion (excepting some details) applies when you want higher alignment than is standard for the environment in which your code will be run.

The situation of this thread was standard x86_64 ABI, in which the stack is 16 byte aligned before each call, so on entry to a function it is assumed to be 8 bytes off from 16 byte aligned. So you can subtract 8 from rsp in order to restore 16 byte aligned.

Originally Posted by JohnGraham I haven't used x86 assembly in a while, but you don't align a stack by adding a constant value to it - what if it's already aligned? You usually OR the stack pointer - e.g. OR it with 0xfffffff0 to align a 32 bit stack pointer by 16 bytes.

And by OR you of course meant AND.

Originally Posted by prushik If I have to restore rsp at the end, isn't it just as graceful to save rsp on the stack and pop it off at the end?

SUB and ADD is faster then PUSH and POP, and besides, even though PUSH RSP and POP RSP will work correctly, they look a bit weird (at least to me) since PUSH and POP respectively decrement and increment RSP register.

Thanks everybody. I am marking this thread as solved since you guys answered my question. Thanks.

Your discussion (excepting some details) applies when you want higher alignment than is standard for the environment in which your code will be run. The situation of this thread was standard x86_64 ABI, in which the stack is 16 byte aligned before each call, so on entry to a function it is assumed to be 8 bytes off from 16 byte aligned. So you can subtract 8 from rsp in order to restore 16 byte aligned.

Ah, I see - fair enough.

And by OR you of course meant AND.

I did indeed - silly me :-P