LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 11-18-2012, 08:10 PM   #1
HellSinker
LQ Newbie
 
Registered: Nov 2012
Posts: 3

Rep: Reputation: Disabled
Segmentation Fault when calling ASM from C


Hello, I'm currently in the process of migrating from windows to linux.

I've written a fairly simple pair of programs to help me get aquainted with the differences in the environments.

I have a simple ASM program which I list here

Code:
[BITS 64]
GLOBAL Test
Test:
mov QWORD [rsp+8], r8
mov QWORD [rsp+16], r9
mov QWORD [rsp+24], rcx
mov QWORD [rsp+32], rdx
sub rsp, 40
add rsp, 40
mov rax, 1974
ret 0
For starters, I am curious if we even still store our parameters the same way, ie. on the stack with sub rsp and add rsp.

now I have another program written in C, listed here

Code:
#include <stdio.h>
void main()
{
printf("%u\r\n",Test(0,0,0,0));
}
compiling the ASM with NASM, and the C with CC, linking with GCC produces an executable that works (i.e. it will display the number '1974') but also seg faults on exit.

Could someone explain the reason for this Segmentation Fault?

I am using SUSE 12.2 Studio, with very little more than the JeoS base...

Last edited by HellSinker; 11-18-2012 at 08:12 PM. Reason: typos
 
Old 11-19-2012, 03:03 AM   #2
pan64
Senior Member
 
Registered: Mar 2012
Location: Hungary
Distribution: debian i686 (solaris)
Posts: 4,929

Rep: Reputation: 1306Reputation: 1306Reputation: 1306Reputation: 1306Reputation: 1306Reputation: 1306Reputation: 1306Reputation: 1306Reputation: 1306Reputation: 1306
probably you not need add rsp, 40, but I'm not really sure.
You can insert a second line into the c code (another printf) and execute step by step in debug mode
 
Old 11-19-2012, 04:47 AM   #3
linosaurusroot
Member
 
Registered: Oct 2012
Distribution: OpenSuSE,RHEL,Fedora,OpenBSD
Posts: 808
Blog Entries: 2

Rep: Reputation: 203Reputation: 203Reputation: 203
At the end of main() you have no exit() so it will be trying return - which is perhaps what fails after your stack manipulation.

Really there's no need for assembler in most linux work (a few fields of work excepted). C is the lowest level most people need and they prefer to use higher levels such as Perl and Python.
 
Old 11-19-2012, 07:36 AM   #4
johnsfine
Guru
 
Registered: Dec 2007
Distribution: Centos
Posts: 5,107

Rep: Reputation: 1114Reputation: 1114Reputation: 1114Reputation: 1114Reputation: 1114Reputation: 1114Reputation: 1114Reputation: 1114Reputation: 1114
Quote:
Originally Posted by HellSinker View Post
For starters, I am curious if we even still store our parameters the same way, ie. on the stack with sub rsp and add rsp.
Store the parameters any valid way you like, but you ought to know that the standard for passing parameters is different. (There is no standard for storing parameters within the asm function.)

AMD defined a parameter passing standard when inventing the X86-64 architecture. Linux still uses that standard. Microsoft decided to ignore the standard and invent their own. In this particular case, Microsoft's version is technically superior to the X86-64 ABI standard (especially regarding floating point), but ignoring a standard (even an inferior one) causes problems for those like you who want to port asm source. It caused an even bigger problem (eventually solved) for the people who created the 64 bit version of mingw.

In integer/pointer part of the Microsoft standard, the first four parameters are passed in rcx, rdx, r8 and r9. Then the function must preserve rdi, rsi, rbx, rbp and r12-r15.

In the integer/pointer part of the X86-64 ABI standard (what you need to use with Linux) the first six parameters are passed in rdi, rsi, rdx, rcx, r8 and r9. Then the function must preserve rbx, rbp and r12-r15.

The floating point side of those standards differs more.

Quote:
Could someone explain the reason for this Segmentation Fault?
Quote:
Originally Posted by HellSinker View Post
Code:
[BITS 64]
GLOBAL Test
Test:
mov QWORD [rsp+8], r8
mov QWORD [rsp+16], r9
mov QWORD [rsp+24], rcx
mov QWORD [rsp+32], rdx
sub rsp, 40
add rsp, 40
mov rax, 1974
ret 0
I expect you wanted to sub rsp,40 before using positive offsets from rsp to store (the wrong) registers.

The fact that you have the wrong registers would matter if your asm function tried to do something useful, but doesn't matter yet.

I'm not certain exactly how your main function was compiled so I don't know its exact stack layout. The locations you overwrote (rsp+8 through rsp+39 on entry to your asm function) are parts of the stack of that main function. You obviously overwrote at least one of the saved rbp or rip in main's stack frame, so that when main tries to return you get a seg fault.

Quote:
Originally Posted by linosaurusroot View Post
At the end of main() you have no exit() so it will be trying return - which is perhaps what fails after your stack manipulation.
That is correct (and pretty obvious given that the printf worked). I hope the OP read my long post far enough to see why that stack overwrite was wrong, rather than just infer the stack manipulation was wrong as the only plausible reason the return from main() might crash.

Quote:
Originally Posted by linosaurusroot View Post
Really there's no need for assembler in most linux work (a few fields of work excepted). C is the lowest level most people need and they prefer to use higher levels such as Perl and Python.
We at LQ answer a lot of "how to" questions with "this is why you shouldn't even want to". I've answered quite a few that way myself. But I think it is a really rotten answer this time.

There are lots of good reasons for the OP to want to do what he asks. Writing ASM functions to be called from C, is a very useful learning exercise that will help you with other programming and debugging even if you never write a line of ASM for production use. The OP seems to have ASM code already written and this is a porting question. So the decision (right or wrong) to use ASM is "water under the bridge". Please answer constructively to help solve the OP's problem, rather than posting non constructive discouragement.

Last edited by johnsfine; 11-19-2012 at 08:04 AM.
 
3 members found this post helpful.
Old 11-19-2012, 11:55 PM   #5
HellSinker
LQ Newbie
 
Registered: Nov 2012
Posts: 3

Original Poster
Rep: Reputation: Disabled
Thankyou for your answers.
 
Old 11-20-2012, 07:39 AM   #6
sundialsvcs
Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 5,401

Rep: Reputation: 1119Reputation: 1119Reputation: 1119Reputation: 1119Reputation: 1119Reputation: 1119Reputation: 1119Reputation: 1119Reputation: 1119
The most common reason for "seg-faults during cross language calls" (of any sort) is the so-called calling convention: that is to say, precisely how parameters are passed between them.

Also note that the most-common place to find assembly code is within an asm{} directive within a C/C++ program.
 
Old 11-21-2012, 07:06 AM   #7
resetreset
Senior Member
 
Registered: Mar 2008
Location: India
Distribution: Dynebolic, Ubuntu 10.10
Posts: 1,320

Rep: Reputation: 62
This is a VERY IMPORTANT question: where did you pick up 64-bit assembly? Are there any books?
 
Old 11-21-2012, 07:11 AM   #8
resetreset
Senior Member
 
Registered: Mar 2008
Location: India
Distribution: Dynebolic, Ubuntu 10.10
Posts: 1,320

Rep: Reputation: 62
Also, I don't get what the "sub rsp,40" and "add rsp,40" are for.

What is "ret 0"? What's the "0" for?
 
Old 11-21-2012, 07:24 AM   #9
johnsfine
Guru
 
Registered: Dec 2007
Distribution: Centos
Posts: 5,107

Rep: Reputation: 1114Reputation: 1114Reputation: 1114Reputation: 1114Reputation: 1114Reputation: 1114Reputation: 1114Reputation: 1114Reputation: 1114
Quote:
Originally Posted by resetreset View Post
where did you pick up 64-bit assembly? Are there any books?
Since I already knew x86-32 assembly and many other assembly languages, it was easiest for me to learn X86-64 from reference manuals (pdf files downloaded from AMD's website) rather than look for any resource that teaches assembly using X86-64.

The most useful of those reference manuals is a pdf file with 24594 in its name (inside the pdf file, the document name is AMD64 Architecture Programmer's Manual Volume 3: General-Purpose and System Instructions)

I just did a google search for AMD 24594 and the first hit looks like it is a current link to that pdf file, but I didn't test it. My old links to AMD pages that link to the full set of these manuals are no longer valid.

Other useful pdf files in that set included 24592, 24593, 26568 and 26569.

Quote:
Originally Posted by resetreset View Post
I don't get what the "sub rsp,40" and "add rsp,40" are for.
In the exact code quoted earlier in this thread, those instructions are of course useless.

If the sub rsp,40 had been in its intended place before the instructions that use positive offsets of rsp for storage, then its purpose is to make those positive offsets OK to use.

In user mode X86-64, you can safely use stack space via negative offsets of rsp. So you might never need the sub from rsp at all. But if your asm function calls any other functions, you need to adjust rsp to account for your stack use before calling another function.

If you change rsp that way, you need to restore it before returning, so that is the purpose of the add rsp,40.


Quote:
What is "ret 0"? What's the "0" for?
The 0 there is useless. A nonzero value there would change the meaning to a form of return not used by the standard calling convention. See 24594 for specifics.

Without the 0, the ret instruction would perform the same operation as with the 0. With the 0, that asm syntax implies a three byte instruction that does the same job as the usual one byte ret instruction. I don't know whether the nasm assembler optimizes that. There are a few cases where an assembler emits the shorter version of an instruction even when the programmer has coded a functionally equivalent longer instruction.

Last edited by johnsfine; 11-21-2012 at 07:42 AM.
 
Old 11-21-2012, 07:27 AM   #10
resetreset
Senior Member
 
Registered: Mar 2008
Location: India
Distribution: Dynebolic, Ubuntu 10.10
Posts: 1,320

Rep: Reputation: 62
Actually I was asking the OP, not you John But anyway, that's bad news, cuz I really prefer to get my info from books, preferably ones written in a conversational style (was spoiled by Michael Abrash ).

Also, could you answer my question above about the "sub rsp, 40" and the "ret 0"?
 
Old 11-21-2012, 07:56 AM   #11
johnsfine
Guru
 
Registered: Dec 2007
Distribution: Centos
Posts: 5,107

Rep: Reputation: 1114Reputation: 1114Reputation: 1114Reputation: 1114Reputation: 1114Reputation: 1114Reputation: 1114Reputation: 1114Reputation: 1114
Quote:
Originally Posted by resetreset View Post
Actually I was asking the OP, not you John
Maybe the OP will also answer. But based on the first post, I would not estimate the OP ever did "pick up 64-bit assembly". I didn't bother to ask how the OP came to be porting asm code he doesn't really understand. I didn't need to know why in order to give appropriate help.

I expect the OP's partial understanding of the subject at some point came from some online tutorial. But maybe there is some book. I've never looked for one.

Quote:
But anyway, that's bad news, cuz I really prefer to get my info from books, preferably ones written in a conversational style
I think the audience for a book on x86-64 assembly would be much smaller than it was (even than it still is) for books on obsolete architecture assembly language.

Very few programmers learn x86-64 assembly and I think most of those are self motivated enough to get what they need from reference manuals.

Surprisingly many college courses still teach asm. But every one of those classes I've ever heard of teaches an obsolete architecture. This sustains the market for books on the obsolete architectures. I assume the instructors are neither self motivated enough nor intelligent enough to learn x86-64 asm themselves. They teach obsolete stuff because that is all they know how to teach.

Quote:
Also, could you answer my question above about the "sub rsp, 40" and the "ret 0"?
I tend to press "submit" part way through typing a post to avoid loosing too much in case of malfunction, then I edit to finish typing. So you saw my post before I finished. Reread my earlier post to see the rest of the answer.
 
Old 11-21-2012, 08:02 AM   #12
resetreset
Senior Member
 
Registered: Mar 2008
Location: India
Distribution: Dynebolic, Ubuntu 10.10
Posts: 1,320

Rep: Reputation: 62
Gosh, you're a lifesaver John, thanks

By the way, may I PM you? I'd like to make friends....
 
Old 11-21-2012, 08:29 AM   #13
johnsfine
Guru
 
Registered: Dec 2007
Distribution: Centos
Posts: 5,107

Rep: Reputation: 1114Reputation: 1114Reputation: 1114Reputation: 1114Reputation: 1114Reputation: 1114Reputation: 1114Reputation: 1114Reputation: 1114
Quote:
Originally Posted by resetreset View Post
By the way, may I PM you?
OK, but I usually prefer discussions in forums to private discussions. On technical subjects, it is nice to leave a trail that others who search for it later might find.
 
Old 11-21-2012, 09:03 AM   #14
johnsfine
Guru
 
Registered: Dec 2007
Distribution: Centos
Posts: 5,107

Rep: Reputation: 1114Reputation: 1114Reputation: 1114Reputation: 1114Reputation: 1114Reputation: 1114Reputation: 1114Reputation: 1114Reputation: 1114
Quote:
Originally Posted by johnsfine View Post
But every one of those classes I've ever heard of teaches an obsolete architecture.
It is so easy to use google to prove myself wrong. Maybe I should have tried that google search earlier.

I just found
http://www.cs.uaf.edu/2012/fall/cs30..._21_stack.html

I tried searching something from the first post that I thought would be an exact match in the online tutorial that I was guessing was the source of the OP's coding style. Zero hits!

Trying a slightly looser search for something similar, I still found no online tutorial on x86-64 asm (which doesn't mean it doesn't exist, just that I'm not searching well).

But that looser search did find the above page that seems to be the lecture notes from a lecture in the middle of an asm class that uses (some) x86-64 asm (what I said I'd never heard of). The top level link is
http://www.cs.uaf.edu/2012/fall/cs301/

The start of that series seems to focus on PowerPC asm and only uses x86-64 for contrast. I didn't read all the notes to see how much of the later focus is on x86-64

Last edited by johnsfine; 11-21-2012 at 09:08 AM.
 
Old 11-22-2012, 12:04 AM   #15
resetreset
Senior Member
 
Registered: Mar 2008
Location: India
Distribution: Dynebolic, Ubuntu 10.10
Posts: 1,320

Rep: Reputation: 62
John, I PM'ed you..... no reply? Please PM me back, don't email, cause I lost the password to that address long time ago.....
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
C program calling assembly - results in segmentation fault Filipe Programming 2 11-03-2008 11:53 AM
Qt: seg fault when calling treewidget Ephracis Programming 8 02-23-2007 05:38 AM
yast segmentation fault, system freezing - nvidia driver at fault? BaltikaTroika Suse/Novell 2 12-02-2005 09:34 AM
Segmentation fault with asm("int 0x0f") justthething Programming 1 08-02-2004 01:43 PM
calling all C, ASM in trouble orgcandman Programming 1 06-24-2002 02:07 PM


All times are GMT -5. The time now is 06:51 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration