Segfault trying to pass more than 6 arguments to a function (x86_64 ASM)

Hi everybody.
I'm still working on learning ASM, you helped me before so maybe you can help me out again.
I am trying to call a C function from my ASM code. The function is part of SDL. I have used this function before in C so I know what I am supposed to do with it, but I can't figure out how to call it the same way in ASM.

The problem is that the function takes 8 arguments. Since I am on x86_64 Linux, that means I have to pass the first 6 in registers and the last two on the stack.

I didn't think this would be a big deal, but I just can't get it to work. To compound the issue, I am also trying to extract data to pass to this function from a structure which was allocated by SDL. So I created these structures (YASM Syntax):

struc	SDL_PixelFormat
	.palette		resq	1
	.BitsPerPixel	resb	1
	.BytesPerPixel	resb	1
	.Rloss		resb	1
	.Gloss		resb	1
	.Bloss		resb	1
	.Aloss		resb	1
	.Rshift		resb	1
	.Gshift		resb	1
	.Bshift		resb	1
	.Ashift		resb	1
	.Rmask		resq	1
	.Gmask		resq	1
	.Bmask		resq	1
	.Amask		resq	1
	.colorkey		resq	1
	.alpha		resb	1
endstruc

struc	SDL_Surface
	.flags		resq	1
	.format		resq	1
	.w			resq	1
	.h			resq	1
	.pitch		resd	1
	.pixels		resq	1
				resq 1
				resq	1
	.clip_rect.x	resd 1
	.clip_rect.y	resd	1
	.clip_rect.w	resd	1
	.clip_rect.h	resd	1
				resq	1
				resq	1
				resq	1
				resq	1
	.refcount		resq	1
endstruc

Which I believe correspond to structures as defined by SDL.h

I then initialize SDL as I did before and then call the function like this:

	CreateSurface:
		push	rbp		;stack alignment
		mov		rbp,rsp

		mov		r11,screen+SDL_Surface.format
		mov		[fmt],r11;

		call		PSF_GetGlyphSize

		mov		rdi,0
		mov		rsi,8
		mov		rdx,rax
		mov		rcx,fmt+SDL_PixelFormat.BitsPerPixel
		mov		r8,fmt+SDL_PixelFormat.Rmask
		mov		r9,fmt+SDL_PixelFormat.Gmask
;		push	fmt+SDL_PixelFormat.Amask
;		push	fmt+SDL_PixelFormat.Bmask
		call		SDL_CreateRGBSurface
		
		pop		rbp
		ret

This is almost an exact duplicate of my C function which did the same thing. However, only 6 arguments are passed to the function SDL_CreateRGBSurface. If I uncomment the last two arguments, the program will segfault. Really strange, but I'm sure I did something terribly and obviously wrong.
Can anybody see what it is?

Thanks!

Originally Posted by prushik I have used this function before in C so I know what I am supposed to do with it, but I can't figure out how to call it the same way in ASM.

I'll read the rest of your post after writing this answer. But just what you said so far has an answer you ought to be in the habit of using:

If you have ANY doubt about the ABI details for calling some function, code the call in C, then look at the asm code created by the C compiler.

Hopefully you know how to use the -S option to gcc and/or one of the many other ways to see the compiler output as asm code.

Edit: The above is still an important tool you should use when you have any ABI doubts, but after reading your whole post, I think the problem is more likely to be in the SDL_PixelFormat struct rather than in the mechanism of passing parameters to that function.

Originally Posted by johnsfine I'll read the rest of your post after writing this answer. But just what you said so far has an answer you ought to be in the habit of using: If you have ANY doubt about the ABI details for calling some function, code the call in C, then look at the asm code created by the C compiler. Hopefully you know how to use the -S option to gcc and/or one of the many other ways to see the compiler output as asm code.

Yes, I have tried that. It seemed that the output was essentially the same as what I wrote by hand. Except that it did it in a slightly different way (subtract 16 from rsp, then mov the arguments into rsp and rsp+8), my two push instructions are the same thing as that. Even if I accidentally pass them in the wrong order, it should give strange results, but they would be valid values and it shouldn't segfault. I even tried pushing two 0's onto the stack and it still segfaults. Yet, if I don't push anything onto the stack then it doesn't crash at all.

Originally Posted by prushik mov r11,screen+SDL_Surface.format mov [fmt],r11;

How was fmt declared, and what do you think the above two lines are doing?

I can't think of anything fmt might be that makes this use of fmt consistent with your later use of fmt.

Originally Posted by prushik ; push fmt+SDL_PixelFormat.Amask ; push fmt+SDL_PixelFormat.Bmask

I never used yasm, so I might be guessing wrong on the syntax. Is there a memory reference implicit there?

Originally Posted by johnsfine How was fmt declared, and what do you think the above two lines are doing? I can't think of anything fmt might be that makes this use of fmt consistent with your later use of fmt.

fmt is allocated in the .bss section like this:
fmt resq 1
It's a pointer.
screen is an instance of SDL_Surface, which is a structure that contains a pointer to SDL_PixelFormat. It gets allocated and initialized by SDL in SDL_SetVideoMode. I set fmt to the address of the SDL_PixelFormat structure which is inside the structure and then try to access the members of the SDL_PixelFormat structure.
That's what I was trying to do.

Originally Posted by prushik then try to access the members of the SDL_PixelFormat structure. That's what I was trying to do.

Way off!

Assembler doesn't work like that at all.

Forget fmt. You need the pointer in a register and you had (see below) the pointer in a register.

I don't know yasm syntax exactly, but instead of

mov  rcx,fmt+SDL_PixelFormat.BitsPerPixel

etc. You need something like

mov  rcx,[R11+SDL_PixelFormat.BitsPerPixel]

or maybe

mov  rcx,SDL_PixelFormat.BitsPerPixel[R11]

Even though I'm not sure of yasm syntax, the same concepts apply to any form of x86_64 assembler. The assembler specific syntax is a minor detail.

To use a pointer in assembler, the pointer must be in a register and you must specify a valid addressing mode that combines that register with the desired offset.

I also looked up SDL_PixelFormat with google and I see your asm version of that struct does not match the standard layout.

Also, setting up R11 before an unrelated call doesn't work, since the unrelated call clobbers that register.

Also, your syntax was wrong again in:

mov  r11,screen+SDL_Surface.format

Why did you select an assembler syntax other than gas, for use on Linux, when you didn't know any asm syntax?

If you were using gas syntax, it would be a lot easier to compare your own efforts with the output of gcc. If you were using gas syntax, more experts in this forum could help you.

Yasm may be a great tool for those who learned asm on Windows and don't want to relearn the syntax. But obviously you don't know yasm syntax yet.

Originally Posted by johnsfine Way off! Assembler doesn't work like that at all. Forget fmt. You need the pointer in a register and you have the pointer in a register. I don't know yasm syntax exactly, but instead of mov rcx,fmt+SDL_PixelFormat.BitsPerPixel etc. You need something like mov rcx,[R11+SDL_PixelFormat.BitsPerPixel] Even though I'm not sure of yasm syntax, the same concepts apply to any form of x86_64 assembler. The assembler specific syntax is a minor detail. To use a pointer in assembler, the pointer must be in a register and you must specify a valid addressing mode that combines that register with the desired offset.

Alright, thanks, that is a step in the right direction at least. I modified my code as you suggested, removing fmt and using instead a register, r11 (is it safe for me to use that particular register?). It still segfaults, but now it segfaults instead at the ret instruction at the end of the function. Which means that I am not cleaning up the stack properly right?
my code looks like this:

	CreateSurface:
		push	rbp		;stack alignment
		mov		rbp,rsp

		call		PSF_GetGlyphSize

		mov		r11,screen+SDL_Surface.format

		mov		rdi,0
		mov		rsi,8
		mov		rdx,rax
		mov		rcx,[r11+SDL_PixelFormat.BitsPerPixel]
		mov		r8,[r11+SDL_PixelFormat.Rmask]
		mov		r9,[r11+SDL_PixelFormat.Gmask]
		push	qword [r11+SDL_PixelFormat.Amask]
		push	qword [r11+SDL_PixelFormat.Bmask]
		call		SDL_CreateRGBSurface
		
		pop		rbp
		ret

Now I noticed something new this time. I had to use the qword directive or it complained about the operand of the push instruction being the wrong size. But according to the function's definition, those last 4 arguments are only 32 bits wide, does that matter for me or am I passing the arguments correctly?
I think I must not be since the program segfaults at ret.

Instead of

pop rbp

you are supposed to use

leave

That error was the reason the ret crashed.

In case you missed my last edits to the earlier post, your "screen+SDL_Surface.format" was wrong. What is correct depends on how screen was defined and initialized. Also I'm pretty sure your declaration of SDL_PixelFormat was wrong, both alignment issues and having the wrong size for Amask etc.

There are some details I'm not 100% certain of: In 64-bit mode, I believe a push from memory cannot read just a 32-bit value. I know it must write a 64-bit value. That called function expects a 32-bit value on the stack and the next 32-bits wasted. If Amask had been properly declared as 32 bits, I would have used

mov   eax, [r11+SDL_PixelFormat.Amask]
push  rax

But you probably don't care about the excess memory read and the garbage put into the unused bits, so your version isn't terribly wrong:

push  qword [r11+SDL_PixelFormat.Amask]

You have a choice of style (which sometimes impacts performance):

sub   rsp, 24
...
mov   eax, ...Bmask]
mov   [rsp], eax
mov   eax, ...Amask]
mov   [rsp+8], eax
...
add   rsp, 24

vs.

push  rbp
mov   rbp, rsp
...
push ...
push ...
...
leave