Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum. |
Notices |
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
Are you new to LinuxQuestions.org? Visit the following links:
Site Howto |
Site FAQ |
Sitemap |
Register Now
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
|
 |
|
11-20-2008, 03:38 PM
|
#1
|
LQ Newbie
Registered: Nov 2008
Posts: 18
Rep:
|
g++ on 64bit machine alignment issue
Hi everybody,
We just moved our application from 32bit Fedora to 64bit Fedora 9. We changed one field on several structs from unsigned int to long. Then the struct size changed unexpected.
It looks like the structs don't align by 4 any more. It aligns by 8 now. For example one of the struct was 88 bytes before and with one field changed from unsigned int to long, the size changed to 96 but not 92.
What I cannot understand is if there is no use of long, the struct size didn't changed even it was not a size by 8. For example one struct only have a int inside, the size still 4 on 64bit machine.
Tried few compile flag, such as -fpack-struct=4, -malign-double. no luck on them.
Any advice on this? thanks,
|
|
|
11-20-2008, 04:04 PM
|
#2
|
Senior Member
Registered: May 2008
Location: planet earth
Distribution: Debian
Posts: 1,732
Rep:
|
I don't see what's unexpected here. By default the compiler will align to 8 for a 64-bit machine. Also, the GCC documentation states very clearly that 'long' defaults to 32 on a 32-bit machine and 64 on a 64-bit machine. Why is the size and alignment of the structs important?
|
|
|
11-20-2008, 04:11 PM
|
#3
|
LQ Newbie
Registered: Nov 2008
Posts: 18
Original Poster
Rep:
|
Quote:
Originally Posted by pinniped
I don't see what's unexpected here. By default the compiler will align to 8 for a 64-bit machine. Also, the GCC documentation states very clearly that 'long' defaults to 32 on a 32-bit machine and 64 on a 64-bit machine. Why is the size and alignment of the structs important?
|
Yes, we know the long will be 8 bytes in 64bit. But we want to keep the alignment in 4 bytes not in 8 bytes. We need to send those structs to other clients through the socket so we need to make sure the size is correct.
|
|
|
11-20-2008, 07:58 PM
|
#4
|
Senior Member
Registered: May 2008
Location: planet earth
Distribution: Debian
Posts: 1,732
Rep:
|
If you're sending structs through sockets using a pointer to a struct and a sizeof(struct) then your software has fatal deficiencies; there are far bigger problems than simply aligning 8-byte-wide variables to a 4-byte address boundary. You need to rethink what you want to achieve and how you're going about it because at the moment I get the impression that the software is working more by accident than by design.
|
|
|
11-20-2008, 09:08 PM
|
#5
|
LQ Newbie
Registered: Nov 2008
Posts: 18
Original Poster
Rep:
|
Quote:
Originally Posted by pinniped
If you're sending structs through sockets using a pointer to a struct and a sizeof(struct) then your software has fatal deficiencies; there are far bigger problems than simply aligning 8-byte-wide variables to a 4-byte address boundary. You need to rethink what you want to achieve and how you're going about it because at the moment I get the impression that the software is working more by accident than by design.
|
Why you think it is working more by accident than by design? Could you please give me more details about your concerns?
Do you have idea for my 4 bytes alignment without any code changes?
|
|
|
11-20-2008, 09:36 PM
|
#6
|
LQ Newbie
Registered: Nov 2008
Posts: 18
Original Poster
Rep:
|
Quote:
Originally Posted by pinniped
If you're sending structs through sockets using a pointer to a struct and a sizeof(struct) then your software has fatal deficiencies; there are far bigger problems than simply aligning 8-byte-wide variables to a 4-byte address boundary. You need to rethink what you want to achieve and how you're going about it because at the moment I get the impression that the software is working more by accident than by design.
|
Why you think it is working more by accident than by design? Could you please give me more details about your concerns?
Do you have idea for my 4 bytes alignment without any code changes?
Last edited by linux001; 11-20-2008 at 09:37 PM.
|
|
|
11-20-2008, 10:05 PM
|
#7
|
Senior Member
Registered: May 2008
Location: planet earth
Distribution: Debian
Posts: 1,732
Rep:
|
Well, -mno-align-double (rather than -malign-double, which is the default on x86_64) the data *might* align to single-word boundaries, but you need to test that. This behavior is not guaranteed by any standards and may change between compiler versions.
With your current design for sending the info in the struct you may have data ordering, padding, and alignment issues between different compilers or versions of compilers or even the same compiler running on different CPUs (for example, x86 vs AMD64). A few years ago I was helping someone out with similar issues (sending structs over a socket); he was getting corrupted data at one end because of padding. In fact, your current issues with alignment suggest you would also have a padding issue (if not a data order issue). Another problem is with your (new) use of 'long' which with GCC may be 32 or 64 bits. I can't imagine what that 'long' would be used for except to store a pointer, and sending pointers over sockets is rather pointless; even if you use shared memory, the pointers can be retrieved by other means rather than by sending on the socket.
|
|
|
11-21-2008, 08:54 AM
|
#8
|
LQ Newbie
Registered: Nov 2008
Posts: 18
Original Poster
Rep:
|
Quote:
Originally Posted by pinniped
Well, -mno-align-double (rather than -malign-double, which is the default on x86_64) the data *might* align to single-word boundaries, but you need to test that. This behavior is not guaranteed by any standards and may change between compiler versions.
With your current design for sending the info in the struct you may have data ordering, padding, and alignment issues between different compilers or versions of compilers or even the same compiler running on different CPUs (for example, x86 vs AMD64). A few years ago I was helping someone out with similar issues (sending structs over a socket); he was getting corrupted data at one end because of padding. In fact, your current issues with alignment suggest you would also have a padding issue (if not a data order issue). Another problem is with your (new) use of 'long' which with GCC may be 32 or 64 bits. I can't imagine what that 'long' would be used for except to store a pointer, and sending pointers over sockets is rather pointless; even if you use shared memory, the pointers can be retrieved by other means rather than by sending on the socket.
|
It is a good chance to learn more from expert now. I thought the data ordering for a struct should be always the same for all compilers. I cannot image why different compiler can generate different data orders. The padding and alignment are related. Usually the padding is applied to follow the alignment requirement. So if we make sure there is no extra padding (means the sender and receiver both know the size and data order of the data struct), the communication should be fine, I think.
We actually pass binary data through socket. we start using long and moving to 64bit machine because the value in that field is too big for int. The receiver received data and know the first 8bytes is a number and then converted it. I did see anything dangerous here.
Just tested the -mno-aligned-double, it doesn't work.
One think bother me is the issue we have is not consistence. We have a struct only have one int inside. The sizeof result is 4 but not 8. If the compiler does a 8 bytes alignment, the size of above struct should be 8 bytes. Cannot understand.
|
|
|
11-21-2008, 09:38 AM
|
#9
|
LQ Guru
Registered: Dec 2007
Distribution: Centos
Posts: 5,286
|
Quote:
Originally Posted by linux001
We actually pass binary data through socket. we start using long and moving to 64bit machine because the value in that field is too big for int. The receiver received data and know the first 8bytes is a number and then converted it.
|
Check your g++ documentation for __attribute__ ((packed))
In my copy that is in section 5.34
When you need to use struct's in communication between differently compiled programs, it is a good idea to manage the packing and alignment issues for those struct's directly in the declaration of the struct's, using attributes such as packed (maybe on the whole structure, and/or maybe on parts of the structure).
That has significant advantages vs. using compile command switches to change the packing rules for the entire compilation unit.
|
|
|
11-21-2008, 10:27 AM
|
#10
|
Senior Member
Registered: Jun 2008
Distribution: debian, ubuntu, sidux
Posts: 1,127
Rep: 
|
In addition to the other suggestions you might also want to switch to explicit sizes, int32_t uint32_t int64_t uint64_t, these are typically defined in stdint.h.
|
|
|
11-21-2008, 11:02 AM
|
#11
|
LQ Newbie
Registered: Nov 2008
Posts: 18
Original Poster
Rep:
|
Quote:
Originally Posted by johnsfine
Check your g++ documentation for __attribute__ ((packed))
In my copy that is in section 5.34
When you need to use struct's in communication between differently compiled programs, it is a good idea to manage the packing and alignment issues for those struct's directly in the declaration of the struct's, using attributes such as packed (maybe on the whole structure, and/or maybe on parts of the structure).
That has significant advantages vs. using compile command switches to change the packing rules for the entire compilation unit.
|
This is one reason I am worry about the code change to add __attribute__((packed)). http://digitalvampire.org/blog/index...d/#comment-445
I prefer to tell compiler what the alignment will be. But it looks like g++ doesn't work that way.
|
|
|
11-21-2008, 11:03 AM
|
#12
|
LQ Newbie
Registered: Nov 2008
Posts: 18
Original Poster
Rep:
|
Quote:
Originally Posted by estabroo
In addition to the other suggestions you might also want to switch to explicit sizes, int32_t uint32_t int64_t uint64_t, these are typically defined in stdint.h.
|
yes, good point. I will change code to use them.
|
|
|
11-21-2008, 11:48 AM
|
#13
|
LQ Guru
Registered: Dec 2007
Distribution: Centos
Posts: 5,286
|
Quote:
Originally Posted by linux001
|
1) Notice that it says the problem isn't significant in x86 nor x86_64. You're not using ia64 are you? If you were, then you actually need some inefficient code to access the structure you want to build.
2) Also notice the suggestion (there but not very clearly) to use __attribute__((aligned(x))) to undo the undesired part of the effects of __attribute__((packed)). I'm not entirely sure myself how those interact. You might need to experiment a little. If I understand correctly, you want the entire structure and each 4 or 8 byte element of the structure to have 4 byte alignment. I'm pretty sure you can do that.
3) Also notice where it says "unless you know you need it". In fact, you do know you need it. Even if you knew how to get the same effect with a compiler switch, you shouldn't want to:
You don't want the performance hit of applying this to structures that don't need it.
You really don't want the compatibility hit of applying this to structures declared in library hpp files that must be binary compatible with the same structures in library .so files.
You don't want the maintainability hit of having a structure declaration with an unusual requirement, but having that unusual requirement implemented in a makefile far from that declaration source code.
With more thought, I probably could find a bunch of further reasons you really shouldn't do this with a compiler switch.
|
|
|
11-21-2008, 01:47 PM
|
#14
|
LQ Newbie
Registered: Nov 2008
Posts: 18
Original Poster
Rep:
|
Quote:
Originally Posted by johnsfine
1) Notice that it says the problem isn't significant in x86 nor x86_64. You're not using ia64 are you? If you were, then you actually need some inefficient code to access the structure you want to build.
2) Also notice the suggestion (there but not very clearly) to use __attribute__((aligned(x))) to undo the undesired part of the effects of __attribute__((packed)). I'm not entirely sure myself how those interact. You might need to experiment a little. If I understand correctly, you want the entire structure and each 4 or 8 byte element of the structure to have 4 byte alignment. I'm pretty sure you can do that.
3) Also notice where it says "unless you know you need it". In fact, you do know you need it. Even if you knew how to get the same effect with a compiler switch, you shouldn't want to:
You don't want the performance hit of applying this to structures that don't need it.
You really don't want the compatibility hit of applying this to structures declared in library hpp files that must be binary compatible with the same structures in library .so files.
You don't want the maintainability hit of having a structure declaration with an unusual requirement, but having that unusual requirement implemented in a makefile far from that declaration source code.
With more thought, I probably could find a bunch of further reasons you really shouldn't do this with a compiler switch.
|
You convinced me now. Thanks.
|
|
|
11-21-2008, 04:30 PM
|
#15
|
Senior Member
Registered: May 2008
Location: planet earth
Distribution: Debian
Posts: 1,732
Rep:
|
Is the __attribute__((packed)) part of the C specification or is it a gnuism?
I don't know if the data ordering in a struct is governed by any C specs, and it's been a long time since I've used anything other than GCC, but data ordering within the struct was an issue - some compilers could shuffle around data to avoid excessive packing.
If you use any tricks to shape your struct just the way you want, make sure they're in the C specification; otherwise you're just doing contortion acts to save a little time writing the code which can actually guarantee that data is sent just the way you like.
|
|
|
All times are GMT -5. The time now is 12:36 AM.
|
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
|
Latest Threads
LQ News
|
|