LinuxQuestions.org
Visit the LQ Articles and Editorials section
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices



Reply
 
Search this Thread
Old 07-02-2013, 01:29 PM   #1
prushik
Member
 
Registered: Mar 2009
Location: Pennsylvania
Distribution: Lubuntu, BetterLinux
Posts: 370

Rep: Reputation: 29
Segfault trying to pass more than 6 arguments to a function (x86_64 ASM)


Hi everybody.
I'm still working on learning ASM, you helped me before so maybe you can help me out again.
I am trying to call a C function from my ASM code. The function is part of SDL. I have used this function before in C so I know what I am supposed to do with it, but I can't figure out how to call it the same way in ASM.

The problem is that the function takes 8 arguments. Since I am on x86_64 Linux, that means I have to pass the first 6 in registers and the last two on the stack.

I didn't think this would be a big deal, but I just can't get it to work. To compound the issue, I am also trying to extract data to pass to this function from a structure which was allocated by SDL. So I created these structures (YASM Syntax):
Code:
struc	SDL_PixelFormat
	.palette		resq	1
	.BitsPerPixel	resb	1
	.BytesPerPixel	resb	1
	.Rloss		resb	1
	.Gloss		resb	1
	.Bloss		resb	1
	.Aloss		resb	1
	.Rshift		resb	1
	.Gshift		resb	1
	.Bshift		resb	1
	.Ashift		resb	1
	.Rmask		resq	1
	.Gmask		resq	1
	.Bmask		resq	1
	.Amask		resq	1
	.colorkey		resq	1
	.alpha		resb	1
endstruc

struc	SDL_Surface
	.flags		resq	1
	.format		resq	1
	.w			resq	1
	.h			resq	1
	.pitch		resd	1
	.pixels		resq	1
				resq 1
				resq	1
	.clip_rect.x	resd 1
	.clip_rect.y	resd	1
	.clip_rect.w	resd	1
	.clip_rect.h	resd	1
				resq	1
				resq	1
				resq	1
				resq	1
	.refcount		resq	1
endstruc
Which I believe correspond to structures as defined by SDL.h

I then initialize SDL as I did before and then call the function like this:
Code:
	CreateSurface:
		push	rbp		;stack alignment
		mov		rbp,rsp

		mov		r11,screen+SDL_Surface.format
		mov		[fmt],r11;

		call		PSF_GetGlyphSize

		mov		rdi,0
		mov		rsi,8
		mov		rdx,rax
		mov		rcx,fmt+SDL_PixelFormat.BitsPerPixel
		mov		r8,fmt+SDL_PixelFormat.Rmask
		mov		r9,fmt+SDL_PixelFormat.Gmask
;		push	fmt+SDL_PixelFormat.Amask
;		push	fmt+SDL_PixelFormat.Bmask
		call		SDL_CreateRGBSurface
		
		pop		rbp
		ret
This is almost an exact duplicate of my C function which did the same thing. However, only 6 arguments are passed to the function SDL_CreateRGBSurface. If I uncomment the last two arguments, the program will segfault. Really strange, but I'm sure I did something terribly and obviously wrong.
Can anybody see what it is?

Thanks!
 
Old 07-02-2013, 01:36 PM   #2
johnsfine
Guru
 
Registered: Dec 2007
Distribution: Centos
Posts: 5,139

Rep: Reputation: 1127Reputation: 1127Reputation: 1127Reputation: 1127Reputation: 1127Reputation: 1127Reputation: 1127Reputation: 1127Reputation: 1127
Quote:
Originally Posted by prushik View Post
I have used this function before in C so I know what I am supposed to do with it, but I can't figure out how to call it the same way in ASM.
I'll read the rest of your post after writing this answer. But just what you said so far has an answer you ought to be in the habit of using:

If you have ANY doubt about the ABI details for calling some function, code the call in C, then look at the asm code created by the C compiler.

Hopefully you know how to use the -S option to gcc and/or one of the many other ways to see the compiler output as asm code.

Edit: The above is still an important tool you should use when you have any ABI doubts, but after reading your whole post, I think the problem is more likely to be in the SDL_PixelFormat struct rather than in the mechanism of passing parameters to that function.

Last edited by johnsfine; 07-02-2013 at 01:44 PM.
 
Old 07-02-2013, 01:44 PM   #3
prushik
Member
 
Registered: Mar 2009
Location: Pennsylvania
Distribution: Lubuntu, BetterLinux
Posts: 370

Original Poster
Rep: Reputation: 29
Quote:
Originally Posted by johnsfine View Post
I'll read the rest of your post after writing this answer. But just what you said so far has an answer you ought to be in the habit of using:

If you have ANY doubt about the ABI details for calling some function, code the call in C, then look at the asm code created by the C compiler.

Hopefully you know how to use the -S option to gcc and/or one of the many other ways to see the compiler output as asm code.
Yes, I have tried that. It seemed that the output was essentially the same as what I wrote by hand. Except that it did it in a slightly different way (subtract 16 from rsp, then mov the arguments into rsp and rsp+8), my two push instructions are the same thing as that. Even if I accidentally pass them in the wrong order, it should give strange results, but they would be valid values and it shouldn't segfault. I even tried pushing two 0's onto the stack and it still segfaults. Yet, if I don't push anything onto the stack then it doesn't crash at all.
 
Old 07-02-2013, 01:47 PM   #4
johnsfine
Guru
 
Registered: Dec 2007
Distribution: Centos
Posts: 5,139

Rep: Reputation: 1127Reputation: 1127Reputation: 1127Reputation: 1127Reputation: 1127Reputation: 1127Reputation: 1127Reputation: 1127Reputation: 1127
Quote:
Originally Posted by prushik View Post
Code:
        mov        r11,screen+SDL_Surface.format
        mov        [fmt],r11;
How was fmt declared, and what do you think the above two lines are doing?

I can't think of anything fmt might be that makes this use of fmt consistent with your later use of fmt.
 
Old 07-02-2013, 01:51 PM   #5
johnsfine
Guru
 
Registered: Dec 2007
Distribution: Centos
Posts: 5,139

Rep: Reputation: 1127Reputation: 1127Reputation: 1127Reputation: 1127Reputation: 1127Reputation: 1127Reputation: 1127Reputation: 1127Reputation: 1127
Quote:
Originally Posted by prushik View Post
Code:
;        push    fmt+SDL_PixelFormat.Amask
;        push    fmt+SDL_PixelFormat.Bmask
I never used yasm, so I might be guessing wrong on the syntax. Is there a memory reference implicit there?

Last edited by johnsfine; 07-02-2013 at 01:54 PM.
 
Old 07-02-2013, 01:55 PM   #6
prushik
Member
 
Registered: Mar 2009
Location: Pennsylvania
Distribution: Lubuntu, BetterLinux
Posts: 370

Original Poster
Rep: Reputation: 29
Quote:
Originally Posted by johnsfine View Post
How was fmt declared, and what do you think the above two lines are doing?

I can't think of anything fmt might be that makes this use of fmt consistent with your later use of fmt.
fmt is allocated in the .bss section like this:
fmt resq 1
It's a pointer.
screen is an instance of SDL_Surface, which is a structure that contains a pointer to SDL_PixelFormat. It gets allocated and initialized by SDL in SDL_SetVideoMode. I set fmt to the address of the SDL_PixelFormat structure which is inside the structure and then try to access the members of the SDL_PixelFormat structure.
That's what I was trying to do.
 
Old 07-02-2013, 01:59 PM   #7
johnsfine
Guru
 
Registered: Dec 2007
Distribution: Centos
Posts: 5,139

Rep: Reputation: 1127Reputation: 1127Reputation: 1127Reputation: 1127Reputation: 1127Reputation: 1127Reputation: 1127Reputation: 1127Reputation: 1127
Quote:
Originally Posted by prushik View Post
then try to access the members of the SDL_PixelFormat structure.
That's what I was trying to do.
Way off!

Assembler doesn't work like that at all.

Forget fmt. You need the pointer in a register and you had (see below) the pointer in a register.

I don't know yasm syntax exactly, but instead of
Code:
mov  rcx,fmt+SDL_PixelFormat.BitsPerPixel
etc. You need something like
Code:
mov  rcx,[R11+SDL_PixelFormat.BitsPerPixel]
or maybe
Code:
mov  rcx,SDL_PixelFormat.BitsPerPixel[R11]
Even though I'm not sure of yasm syntax, the same concepts apply to any form of x86_64 assembler. The assembler specific syntax is a minor detail.

To use a pointer in assembler, the pointer must be in a register and you must specify a valid addressing mode that combines that register with the desired offset.

I also looked up SDL_PixelFormat with google and I see your asm version of that struct does not match the standard layout.

Also, setting up R11 before an unrelated call doesn't work, since the unrelated call clobbers that register.

Also, your syntax was wrong again in:
Code:
mov  r11,screen+SDL_Surface.format
Why did you select an assembler syntax other than gas, for use on Linux, when you didn't know any asm syntax?

If you were using gas syntax, it would be a lot easier to compare your own efforts with the output of gcc. If you were using gas syntax, more experts in this forum could help you.

Yasm may be a great tool for those who learned asm on Windows and don't want to relearn the syntax. But obviously you don't know yasm syntax yet.

Last edited by johnsfine; 07-02-2013 at 02:21 PM.
 
Old 07-02-2013, 02:16 PM   #8
prushik
Member
 
Registered: Mar 2009
Location: Pennsylvania
Distribution: Lubuntu, BetterLinux
Posts: 370

Original Poster
Rep: Reputation: 29
Quote:
Originally Posted by johnsfine View Post
Way off!

Assembler doesn't work like that at all.

Forget fmt. You need the pointer in a register and you have the pointer in a register. I don't know yasm syntax exactly, but instead of
Code:
mov  rcx,fmt+SDL_PixelFormat.BitsPerPixel
etc. You need something like
Code:
mov  rcx,[R11+SDL_PixelFormat.BitsPerPixel]

Even though I'm not sure of yasm syntax, the same concepts apply to any form of x86_64 assembler. The assembler specific syntax is a minor detail.

To use a pointer in assembler, the pointer must be in a register and you must specify a valid addressing mode that combines that register with the desired offset.
Alright, thanks, that is a step in the right direction at least. I modified my code as you suggested, removing fmt and using instead a register, r11 (is it safe for me to use that particular register?). It still segfaults, but now it segfaults instead at the ret instruction at the end of the function. Which means that I am not cleaning up the stack properly right?
my code looks like this:
Code:
	CreateSurface:
		push	rbp		;stack alignment
		mov		rbp,rsp

		call		PSF_GetGlyphSize

		mov		r11,screen+SDL_Surface.format

		mov		rdi,0
		mov		rsi,8
		mov		rdx,rax
		mov		rcx,[r11+SDL_PixelFormat.BitsPerPixel]
		mov		r8,[r11+SDL_PixelFormat.Rmask]
		mov		r9,[r11+SDL_PixelFormat.Gmask]
		push	qword [r11+SDL_PixelFormat.Amask]
		push	qword [r11+SDL_PixelFormat.Bmask]
		call		SDL_CreateRGBSurface
		
		pop		rbp
		ret
Now I noticed something new this time. I had to use the qword directive or it complained about the operand of the push instruction being the wrong size. But according to the function's definition, those last 4 arguments are only 32 bits wide, does that matter for me or am I passing the arguments correctly?
I think I must not be since the program segfaults at ret.
 
Old 07-02-2013, 02:29 PM   #9
johnsfine
Guru
 
Registered: Dec 2007
Distribution: Centos
Posts: 5,139

Rep: Reputation: 1127Reputation: 1127Reputation: 1127Reputation: 1127Reputation: 1127Reputation: 1127Reputation: 1127Reputation: 1127Reputation: 1127
Instead of
Code:
pop rbp
you are supposed to use
Code:
leave
That error was the reason the ret crashed.

In case you missed my last edits to the earlier post, your "screen+SDL_Surface.format" was wrong. What is correct depends on how screen was defined and initialized. Also I'm pretty sure your declaration of SDL_PixelFormat was wrong, both alignment issues and having the wrong size for Amask etc.

There are some details I'm not 100% certain of: In 64-bit mode, I believe a push from memory cannot read just a 32-bit value. I know it must write a 64-bit value. That called function expects a 32-bit value on the stack and the next 32-bits wasted. If Amask had been properly declared as 32 bits, I would have used
Code:
mov   eax, [r11+SDL_PixelFormat.Amask]
push  rax
But you probably don't care about the excess memory read and the garbage put into the unused bits, so your version isn't terribly wrong:
Code:
push  qword [r11+SDL_PixelFormat.Amask]

Last edited by johnsfine; 07-02-2013 at 02:48 PM.
 
1 members found this post helpful.
Old 07-02-2013, 02:53 PM   #10
johnsfine
Guru
 
Registered: Dec 2007
Distribution: Centos
Posts: 5,139

Rep: Reputation: 1127Reputation: 1127Reputation: 1127Reputation: 1127Reputation: 1127Reputation: 1127Reputation: 1127Reputation: 1127Reputation: 1127
You have a choice of style (which sometimes impacts performance):

Code:
sub   rsp, 24
...
mov   eax, ...Bmask]
mov   [rsp], eax
mov   eax, ...Amask]
mov   [rsp+8], eax
...
add   rsp, 24
vs.

Code:
push  rbp
mov   rbp, rsp
...
push ...
push ...
...
leave
 
1 members found this post helpful.
  


Reply

Tags
asm, sdl, segfault, stack, x86_64


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] How can i pass an entire Array into a function parameter by value? (no pointer pass) esgol Programming 26 07-31-2012 12:34 PM
How to pass all arguments to a function stormcloud Linux - General 4 07-08-2010 04:15 AM
[SOLVED] How to pass arguments to a function using a loop 10110111 Programming 2 03-01-2010 04:24 PM
pass arguments and return values from a function in UNIX gaynut Programming 4 07-10-2008 02:56 AM
How to pass structure as function arguments ssg14j Programming 2 08-20-2005 10:59 PM


All times are GMT -5. The time now is 10:34 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration