Was working on some test code which would allow me to call functions in a C source file from an Assembly source file.
It was getting some really bizarre results, which had nothing to do with the parameters I fed it.
The NASM code in question:
Code:
SUB RSP, 2 * 8
MOV [RSP + 8], BYTE 2
MOV [RSP], BYTE 2
CALL u_CdeclTester
ADD RSP, 16
(The starting portion is equivalent to PUSHing to stack.
The C Code in question:
Code:
uint16_t u_CdeclTester(const uint16_t val1, const uint16_t val2) {
uint32_t X = u_Testinator();
X++;
if ((val1 == 1) && (val2 == 2)) return 0;
return val1 + val2;
}
When provided the aforementioned inputs, it would return 104, so I decided to decompile the code with objdump. This is the x86-64 disassembly of the entire program, shortened to just the above C code:
Code:
40113e: 55 push rbp
40113f: 48 89 e5 mov rbp,rsp
401142: 48 83 ec 18 sub rsp,0x18 {{allocate room for 3 variables?}}
401146: 89 fa mov edx,edi
401148: 89 f0 mov eax,esi
40114a: 66 89 55 ec mov WORD PTR [rbp-0x14],dx {{overwrite memory?}}
40114e: 66 89 45 e8 mov WORD PTR [rbp-0x18],ax
401152: b8 00 00 00 00 mov eax,0x0
401157: e8 cf ff ff ff call 40112b <putc@plt+0x10b>
40115c: 89 45 fc mov DWORD PTR [rbp-0x4],eax
40115f: 83 45 fc 01 add DWORD PTR [rbp-0x4],0x1
401163: 66 83 7d ec 01 cmp WORD PTR [rbp-0x14],0x1
401168: 75 0e jne 401178 <putc@plt+0x158>
40116a: 66 83 7d e8 02 cmp WORD PTR [rbp-0x18],0x2
40116f: 75 07 jne 401178 <putc@plt+0x158>
401171: b8 00 00 00 00 mov eax,0x0
401176: eb 0a jmp 401182 <putc@plt+0x162>
401178: 0f b7 55 ec movzx edx,WORD PTR [rbp-0x14]
40117c: 0f b7 45 e8 movzx eax,WORD PTR [rbp-0x18]
401180: 01 d0 add eax,edx
401182: c9 leave
401183: c3 ret
I understand that with CDECL, all parameters are passed through the stack, and that the caller has to manage RAX, RCX and RDX. Given the layout of the C code, I would expect it to at least start like this:
Code:
PUSH RBP
MOV RBP, RSP
MOV DX, [RBP + 0x16] # load parameter 2
MOV AX, [RBP + 0x08] # load parameter 1
...
But as you can see in the disassembly, it sets DX and AX to the Destination and Source registers respectively, and then goes on to
overwrite the memory positions that should have been assigned to the parameters (not only being really weird, but also violating the
const attribute applied to the parameters).
I also decompiled it into the nastier AT&T syntax, and it still did the same operation, so it isn't on the fault of the disassembler in objdump. Furthermore, when I set RDI and RSI to zero, the code returned zero.
The compiling code is as follows:
Code:
rm -f "test"
nasm -Wall -t -g -f elf64 -Ox "test.asm" -o "test.o"
gcc -Wall -std=c11 -Wno-attributes -c "utils.c" -o "utils.o"
ld -O9 -s -L/usr/lib/x86_64-linux-gnu --dynamic-linker=/lib64/ld-linux-x86-64.so.2 "utils.o" "test.o" -o "test" -call_shared -lc
rm "test.o" "utils.o"
Compilation finished successfully.
Is there some compiling flag that I missed? Is LD not linking the object code properly? Or is it something else entirely?
I'm completely stumped with this. It's stalled my progress, and no matter what I change, it still uses the RSI/RDI and still screws up what it's supposed to return.
