Porting C Program to Linux & Problem with Size of ICMP Packet

SeymourButts · 01-06-2010, 09:26 PM

I have a C program that has been running for years on Solaris platform. I have compiled it on Linux 2.6 successfully.

The program sits on a RAW socket and continually waits and processes custom ICMP packets that arrive from different devices.

When the recvfrom() drops through from an incoming packet, the return (rc) says it read a 256 byte icmp type packet that I sent in. That's all good.

I pass the recvfrom buffer to a function and cast it to a struct ip pointer. At this point, when I check the size of the packet using ip_len, it now shows 5121 bytes as the packet size.

I don't understand how the packet grew from 256 bytes to 5121 byes. I am really out of ideas on why this is happening. On Solaris, the packet size is what is should be but not on Linux.

Any ideas you may have would be very helpful. Thank you.

I will include the main code pieces below.

Code:

char recv_buffer [8184];
..
icmp_sd = socket (AF_INET, SOCK_RAW, IPPROTO_ICMP);
..
setsockopt(icmp_sd,SOL_SOCKET,SO_RCVBUF,(char *)&iReceive,sizeof(iReceive));
..
rc = recvfrom (icmp_sd, recv_buffer, iLen,iFlag,&sFrom, &iFromLen);
..
printf(log_message,"Packet Arrived. Bytes Read:[%d]\n",rc); /* 256 bytes */
..
recv_buffer[rc] = '\0';
..
process_msg(recv_buffer, icmp_sd);

/* Function */
int process_msg (char *buffer, int iFd)
{
struct ip *ip_rcv_ptr; 
..
/* Initialize IP pointer */
ip_rcv_ptr = (struct ip *) buffer;
..
printf("Length of Packet: %d\n", ip_rcv_ptr->ip_len); /* 5121 bytes */
..
}

SeymourButts · 01-07-2010, 06:09 AM

Crap, I meant to post this in the programming forum. I would be overjoyed if a moderator would move this to the programming forum.

orgcandman · 01-07-2010, 01:49 PM

A few nits.

Code:

char recv_buffer [8184];

Make this 2048. While it's *possible* to receive a packet that is > 2048 bytes, in practice you'll rarely find packets exceeding roughly 1490. 2048 is nice and "round" and does the job just fine. (additionally, an 8k memory region on the stack is a pagefault waiting to happen)

Code:

..
setsockopt(icmp_sd,SOL_SOCKET,SO_RCVBUF,(char *)&iReceive,sizeof(iReceive));

Would be nice to see what you set the receive buffer size to be.

Code:

..
rc = recvfrom (icmp_sd, recv_buffer, iLen,iFlag,&sFrom, &iFromLen);

What are your flags?

Code:

..
recv_buffer[rc] = '\0';

Why are you doing this? No need to 0 out packet buffer contents. You're not using str*() operations (right?)

Code:

..
printf("Length of Packet: %d\n", ip_rcv_ptr->ip_len); /* 5121 bytes */

ntohs() baby!

You'll need to remember to change network order to host order. Unless your machine is big endian.

-Aaron

SeymourButts · 01-07-2010, 02:59 PM

Thanks for the reply. Receive buffer size is set to the same 8184 as the recv_buffer char array.

How to check if server is big endian?

Status Update:

I found out that there seems to be differences in data types in the ip & icmp structs between Solaris & Linux. Too bad the structs don't use the same data types.

On Solaris, struct ip.ip_len is a short and on Linux it's a u_short. Honestly, I am not clear on the difference, but clearly it is causing issues. I assume other members of struct ip or struct icmp are also causing me heartaches.

I added the following htons() call which did shrink the ip_len down to a size I was expecting. However, I am still having issues with memory faults due to differences in sizes.

Code:

/* Initialize IP pointer */
ip_rcv_ptr = (struct ip *) buffer;

/* Convert ip_len */
ip_rcv_ptr->ip_len = htons(ip_rcv_ptr->ip_len);

jf.argentino · 01-08-2010, 03:25 AM

Quote:

How to check if server is big endian?

You don't have to! Data exchange over ethernet must be done in big endian, that's the law. Since ultra-sparc cpu are big-endian, you never see this bug until you port your code on a littl-endian system (x86 cpu). By the way, if you really need to know which endianness is used, use macro defined in <endian.h>.

Quote:

On Solaris, struct ip.ip_len is a short and on Linux it's a u_short. Honestly, I am not clear on the difference, but clearly it is causing issues

"u_short" goes from 0 to 65535, "short" goes from -32768 to 32767. And the bug is Solaris side since the maximum IP length _IS_ 65536 bytes. But I don't think this is causing any issue in your case since you don't use size over 32767 bytes...

SeymourButts · 01-08-2010, 06:31 AM

If I don't adjust ip.ip_len using htons() then the number of bytes is 5121 even though the number of bytes read off socket is 256.

Code:

rc = recvfrom (icmp_sd, recv_buffer, iLen,iFlag,&sFrom, &iFromLen);

printf("Bytes Read=%d\n",rc); /* 256 bytes */

/* Initialize IP pointer */
ip_rcv_ptr = (struct ip *) recv_buffer;

printf("Before htons() ip_len=%d\n", ip_rcv_prt->ip_len); /* 5151 */

/* Convert ip_len */
ip_rcv_ptr->ip_len = htons(ip_rcv_ptr->ip_len);

printf("After htons() ip_len=%d\n", ip_rcv_prt->ip_len); /* 256 */

Maybe calling htons() is really just a bandaid for some other issue that I have not discovered yet. Using htons() appears to correct the length of the packet pulled off the socket.

I can't tell you how long I have been trying to get this working correctly on Linux.

SeymourButts · 01-08-2010, 09:34 AM

Now I am really confused. If I am sending an icmp packet to my program from a test program the byte order should not be an issue because both programs are compiled and running on the same Linux server (little endian).

I still can't explain why the the ip_rcv_ptr->ip_len is 5121 bytes after setting it to a struct ip ptr when the size of the packet read is 256 bytes. So frustrating.

jf.argentino · 01-08-2010, 09:51 AM

Because, on sending part, the IP stack handle the endianness to be conform to the ETHERNET spec, then when it create the IP header, its content is in big endian, as is the transport header (icmp, tcp, udp..., maybe not the transport layer, I'm not an OSI guru)

SeymourButts · 01-08-2010, 01:35 PM

I think I may have a grasp of what I need to do.

If I am sending a packet, I need to use htons() or htonl() on the int or byte members of struct ip or struct icmp.

If I am receiving a packet, I need to use ntohs() or ntohl() on the int or byte members of struct ip or struct icmp.

Plan on giving that a try later to see if fixes the problem.

jf.argentino · 01-11-2010, 02:15 AM

Quote:

If I am sending a packet, I need to use htons() or htonl() on the int or byte members of struct ip or struct icmp.

If I am receiving a packet, I need to use ntohs() or ntohl() on the int or byte members of struct ip or struct icmp.

That's almost it... But htons and ntohs doesn't apply to bytes but shorts (16bits), bytes does not have endianness...
I recall you that you need to do this on the ip / icmp headers since you're using RAW socket level.
A last clue, use int8_t, int16_t,.. types family, defined in stdint.h

SeymourButts · 01-15-2010, 07:53 AM

Quote:

Originally Posted by jf.argentino

That's almost it... But htons and ntohs doesn't apply to bytes but shorts (16bits), bytes does not have endianness...
I recall you that you need to do this on the ip / icmp headers since you're using RAW socket level.
A last clue, use int8_t, int16_t,.. types family, defined in stdint.h

Can you explain a little more about "use int8_t, int16_t"?

jf.argentino · 01-15-2010, 08:06 AM

When you need to specify a type by its size, using int8_t, int16_t... is the only reliable way to do it since the C99 standard doesn't require that "sizeof (char) == 1", "sizeof (short) == 2" nor "sizeof (long) == 4". For example, with gcc, long are 32 bits on fedora 32 bits, and 64 bits on fedora 64 bits.
These type are mandatory for a (reasonable) C99 conformant compiler / standard library, see <stdint.h> for more details.