Segfault trying to pass more than 6 arguments to a function (x86_64 ASM)
ProgrammingThis forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
Segfault trying to pass more than 6 arguments to a function (x86_64 ASM)
Hi everybody.
I'm still working on learning ASM, you helped me before so maybe you can help me out again.
I am trying to call a C function from my ASM code. The function is part of SDL. I have used this function before in C so I know what I am supposed to do with it, but I can't figure out how to call it the same way in ASM.
The problem is that the function takes 8 arguments. Since I am on x86_64 Linux, that means I have to pass the first 6 in registers and the last two on the stack.
I didn't think this would be a big deal, but I just can't get it to work. To compound the issue, I am also trying to extract data to pass to this function from a structure which was allocated by SDL. So I created these structures (YASM Syntax):
This is almost an exact duplicate of my C function which did the same thing. However, only 6 arguments are passed to the function SDL_CreateRGBSurface. If I uncomment the last two arguments, the program will segfault. Really strange, but I'm sure I did something terribly and obviously wrong.
Can anybody see what it is?
I have used this function before in C so I know what I am supposed to do with it, but I can't figure out how to call it the same way in ASM.
I'll read the rest of your post after writing this answer. But just what you said so far has an answer you ought to be in the habit of using:
If you have ANY doubt about the ABI details for calling some function, code the call in C, then look at the asm code created by the C compiler.
Hopefully you know how to use the -S option to gcc and/or one of the many other ways to see the compiler output as asm code.
Edit: The above is still an important tool you should use when you have any ABI doubts, but after reading your whole post, I think the problem is more likely to be in the SDL_PixelFormat struct rather than in the mechanism of passing parameters to that function.
I'll read the rest of your post after writing this answer. But just what you said so far has an answer you ought to be in the habit of using:
If you have ANY doubt about the ABI details for calling some function, code the call in C, then look at the asm code created by the C compiler.
Hopefully you know how to use the -S option to gcc and/or one of the many other ways to see the compiler output as asm code.
Yes, I have tried that. It seemed that the output was essentially the same as what I wrote by hand. Except that it did it in a slightly different way (subtract 16 from rsp, then mov the arguments into rsp and rsp+8), my two push instructions are the same thing as that. Even if I accidentally pass them in the wrong order, it should give strange results, but they would be valid values and it shouldn't segfault. I even tried pushing two 0's onto the stack and it still segfaults. Yet, if I don't push anything onto the stack then it doesn't crash at all.
How was fmt declared, and what do you think the above two lines are doing?
I can't think of anything fmt might be that makes this use of fmt consistent with your later use of fmt.
fmt is allocated in the .bss section like this:
fmt resq 1
It's a pointer.
screen is an instance of SDL_Surface, which is a structure that contains a pointer to SDL_PixelFormat. It gets allocated and initialized by SDL in SDL_SetVideoMode. I set fmt to the address of the SDL_PixelFormat structure which is inside the structure and then try to access the members of the SDL_PixelFormat structure.
That's what I was trying to do.
then try to access the members of the SDL_PixelFormat structure.
That's what I was trying to do.
Way off!
Assembler doesn't work like that at all.
Forget fmt. You need the pointer in a register and you had (see below) the pointer in a register.
I don't know yasm syntax exactly, but instead of
Code:
mov rcx,fmt+SDL_PixelFormat.BitsPerPixel
etc. You need something like
Code:
mov rcx,[R11+SDL_PixelFormat.BitsPerPixel]
or maybe
Code:
mov rcx,SDL_PixelFormat.BitsPerPixel[R11]
Even though I'm not sure of yasm syntax, the same concepts apply to any form of x86_64 assembler. The assembler specific syntax is a minor detail.
To use a pointer in assembler, the pointer must be in a register and you must specify a valid addressing mode that combines that register with the desired offset.
I also looked up SDL_PixelFormat with google and I see your asm version of that struct does not match the standard layout.
Also, setting up R11 before an unrelated call doesn't work, since the unrelated call clobbers that register.
Also, your syntax was wrong again in:
Code:
mov r11,screen+SDL_Surface.format
Why did you select an assembler syntax other than gas, for use on Linux, when you didn't know any asm syntax?
If you were using gas syntax, it would be a lot easier to compare your own efforts with the output of gcc. If you were using gas syntax, more experts in this forum could help you.
Yasm may be a great tool for those who learned asm on Windows and don't want to relearn the syntax. But obviously you don't know yasm syntax yet.
Forget fmt. You need the pointer in a register and you have the pointer in a register. I don't know yasm syntax exactly, but instead of
Code:
mov rcx,fmt+SDL_PixelFormat.BitsPerPixel
etc. You need something like
Code:
mov rcx,[R11+SDL_PixelFormat.BitsPerPixel]
Even though I'm not sure of yasm syntax, the same concepts apply to any form of x86_64 assembler. The assembler specific syntax is a minor detail.
To use a pointer in assembler, the pointer must be in a register and you must specify a valid addressing mode that combines that register with the desired offset.
Alright, thanks, that is a step in the right direction at least. I modified my code as you suggested, removing fmt and using instead a register, r11 (is it safe for me to use that particular register?). It still segfaults, but now it segfaults instead at the ret instruction at the end of the function. Which means that I am not cleaning up the stack properly right?
my code looks like this:
Now I noticed something new this time. I had to use the qword directive or it complained about the operand of the push instruction being the wrong size. But according to the function's definition, those last 4 arguments are only 32 bits wide, does that matter for me or am I passing the arguments correctly?
I think I must not be since the program segfaults at ret.
In case you missed my last edits to the earlier post, your "screen+SDL_Surface.format" was wrong. What is correct depends on how screen was defined and initialized. Also I'm pretty sure your declaration of SDL_PixelFormat was wrong, both alignment issues and having the wrong size for Amask etc.
There are some details I'm not 100% certain of: In 64-bit mode, I believe a push from memory cannot read just a 32-bit value. I know it must write a 64-bit value. That called function expects a 32-bit value on the stack and the next 32-bits wasted. If Amask had been properly declared as 32 bits, I would have used
Code:
mov eax, [r11+SDL_PixelFormat.Amask]
push rax
But you probably don't care about the excess memory read and the garbage put into the unused bits, so your version isn't terribly wrong:
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.