Segfault trying to pass more than 6 arguments to a function (x86_64 ASM)
 

Hi everybody.
I'm still working on learning ASM, you helped me before so maybe you can help me out again.
I am trying to call a C function from my ASM code. The function is part of SDL. I have used this function before in C so I know what I am supposed to do with it, but I can't figure out how to call it the same way in ASM.

The problem is that the function takes 8 arguments. Since I am on x86_64 Linux, that means I have to pass the first 6 in registers and the last two on the stack.

I didn't think this would be a big deal, but I just can't get it to work. To compound the issue, I am also trying to extract data to pass to this function from a structure which was allocated by SDL. So I created these structures (YASM Syntax):
Which I believe correspond to structures as defined by SDL.h

I then initialize SDL as I did before and then call the function like this:
This is almost an exact duplicate of my C function which did the same thing. However, only 6 arguments are passed to the function SDL_CreateRGBSurface. If I uncomment the last two arguments, the program will segfault. Really strange, but I'm sure I did something terribly and obviously wrong.
Can anybody see what it is?

Thanks!

I'll read the rest of your post after writing this answer. But just what you said so far has an answer you ought to be in the habit of using:

If you have ANY doubt about the ABI details for calling some function, code the call in C, then look at the asm code created by the C compiler.

Hopefully you know how to use the -S option to gcc and/or one of the many other ways to see the compiler output as asm code.

Edit: The above is still an important tool you should use when you have any ABI doubts, but after reading your whole post, I think the problem is more likely to be in the SDL_PixelFormat struct rather than in the mechanism of passing parameters to that function.

Yes, I have tried that. It seemed that the output was essentially the same as what I wrote by hand. Except that it did it in a slightly different way (subtract 16 from rsp, then mov the arguments into rsp and rsp+8), my two push instructions are the same thing as that. Even if I accidentally pass them in the wrong order, it should give strange results, but they would be valid values and it shouldn't segfault. I even tried pushing two 0's onto the stack and it still segfaults. Yet, if I don't push anything onto the stack then it doesn't crash at all.

How was fmt declared, and what do you think the above two lines are doing?

I can't think of anything fmt might be that makes this use of fmt consistent with your later use of fmt.

I never used yasm, so I might be guessing wrong on the syntax. Is there a memory reference implicit there?

fmt is allocated in the .bss section like this:
fmt resq 1
It's a pointer.
screen is an instance of SDL_Surface, which is a structure that contains a pointer to SDL_PixelFormat. It gets allocated and initialized by SDL in SDL_SetVideoMode. I set fmt to the address of the SDL_PixelFormat structure which is inside the structure and then try to access the members of the SDL_PixelFormat structure.
That's what I was trying to do.

Way off!

Assembler doesn't work like that at all.

Forget fmt. You need the pointer in a register and you had (see below) the pointer in a register.

I don't know yasm syntax exactly, but instead of
etc. You need something like
or maybe
Even though I'm not sure of yasm syntax, the same concepts apply to any form of x86_64 assembler. The assembler specific syntax is a minor detail.

To use a pointer in assembler, the pointer must be in a register and you must specify a valid addressing mode that combines that register with the desired offset.

I also looked up SDL_PixelFormat with google and I see your asm version of that struct does not match the standard layout.

Also, setting up R11 before an unrelated call doesn't work, since the unrelated call clobbers that register.

Also, your syntax was wrong again in:
Why did you select an assembler syntax other than gas, for use on Linux, when you didn't know any asm syntax?

If you were using gas syntax, it would be a lot easier to compare your own efforts with the output of gcc. If you were using gas syntax, more experts in this forum could help you.

Yasm may be a great tool for those who learned asm on Windows and don't want to relearn the syntax. But obviously you don't know yasm syntax yet.

Alright, thanks, that is a step in the right direction at least. I modified my code as you suggested, removing fmt and using instead a register, r11 (is it safe for me to use that particular register?). It still segfaults, but now it segfaults instead at the ret instruction at the end of the function. Which means that I am not cleaning up the stack properly right?
my code looks like this:
Now I noticed something new this time. I had to use the qword directive or it complained about the operand of the push instruction being the wrong size. But according to the function's definition, those last 4 arguments are only 32 bits wide, does that matter for me or am I passing the arguments correctly?
I think I must not be since the program segfaults at ret.

Instead of
you are supposed to use
That error was the reason the ret crashed.

In case you missed my last edits to the earlier post, your "screen+SDL_Surface.format" was wrong. What is correct depends on how screen was defined and initialized. Also I'm pretty sure your declaration of SDL_PixelFormat was wrong, both alignment issues and having the wrong size for Amask etc.

There are some details I'm not 100% certain of: In 64-bit mode, I believe a push from memory cannot read just a 32-bit value. I know it must write a 64-bit value. That called function expects a 32-bit value on the stack and the next 32-bits wasted. If Amask had been properly declared as 32 bits, I would have used
But you probably don't care about the excess memory read and the garbage put into the unused bits, so your version isn't terribly wrong:

You have a choice of style (which sometimes impacts performance):

vs.