Hi Forum

I transmit a 100kb file in a single server, single client application( with a windows client and Linux server).
Before transmitting, the data is fragmented into smaller packets each of 8kb size. Each packet is assigned with a sequence number so that data can be recombined correctly at receiver end. On the Windows client side I first wait for the acknowledge sent by the linux server for each correctly received packet and then send the next packet. This method is works sometimes, but sometimes does not work. Client does not receive acknowledgment on expected time, and I fail to recombine the data in correct sequence.What could be the probable error. Kindly guide how a smooth transmission can be ensured?

Thanking in advance
Nancy

I think we'd need to see the code. At the very least, the key parts and just confirm which lang, which distro. Are you writing your own protocol? Can you not use eg scp, Samba, NFS, FTP etc?

Or, more to the point, are you doing this because you're trying to solve the problem presented by this thread?

Because if that's the problem you're trying to solve, I've read that thread, and some of the responses there are sufficiently wrong to scare me, and others are right on the button.

YOur original question in that thread was this:
You meant TCP, of course, and the answer is this:

TCP acts as a stream. The data doesn't move at the speed of light. It's quite possible (and often happens) that a program receiving TCP data can process the data faster than it comes in. So you'll get part of the data you want, and you'll just have to wait for the rest. It's up to you to know how much data you're expecting. You start filling up a buffer with this data, and every time you get more data, you add it to the buffer. Simple as that.

If that was your concern, you don't need to spend one more minute on this project of artificially splitting and recombining the data. Just collect it as it comes.

So let's review some of the things in the other thread that might motivate you to split up the data artificially:
No. Don't bother. Just keep filling up the buffer until it contains as much data as you were expecting.
True but irrelevant. When you do a read() or recv(), you will get as much data as has been collected for you so far, no matter how many TCP packets are involved. You shouldn't even worry about TCP packets; they're an implementation detail you can forget about.
No, if you're using TCP, it's the job of the TCP stack to cut it into smaller chunks as necessary. That's one of the neat things about TCP.
I'm guessing here that anticipating a buggy TCP stack is premature, until you understand what you're doing. TCP stacks which are that buggy are rather rare. No programs I've seen which use TCP concern themselves with workarounds for buggy TCP stacks.
That's a performance issue. It's one you might be able to fine tune, but get your program working first. :)
This is not a bug. You should expect this.
Again, this is TCP's job. Just keep read()ing or recv()ing until you get all the data you were expecting.
Bingo! Absolutely correct! "Outside the realm" means outside the application's control and outside the application's concern.
Again, correct. But no need to worry about flushing; with or without closing the socket, what the sender write()s or send()s will come along soon enough.
But that's not what you're trying to do, right? You're just trying to get all the data?

The Application has limitations not to use other protocols.
I am not writing a new protocol, just want to use TCP/IP to transfer large size data.
Language used is "C"


void *NetCtrlThread(void *param)
{
int l_soc = -1;

netthrd_running = 1;
netthd_stop = 0;

fprintf(stderr, "NetCtrlThread():start\r\n");
// ネットワーク制御
while (!netthd_stop/*1*/) {
// listenソケット生成
if ((l_soc = NetListen(NET_PORT_NO)) < 0) {
// 暫定　エラー処理
fprintf(stderr, "NetCtrlThread():listen error\r\n");
sleep(1);
continue;
}
fprintf(stderr, "NetCtrlThread():listen ok\r\n");

// accept
if ((NetSocket = NetAccept(l_soc)) < 0) {
// 暫定　エラー処理
fprintf(stderr, "NetCtrlThread():accept error\r\n");
close(l_soc);
sleep(1);
continue;
}
fprintf(stderr, "NetCtrlThread():accept ok\r\n");

// listenソケットクローズ
close(l_soc);

// select
struct timeval timeout;
fd_set mask;
int width;
int len, err;

err = 0;
while (!netthd_stop/*1*/) {
// 受信
// fprintf(stderr, "NetCtrlThread():select():start\r\n");
struct timespec tim_req, tim_rem;
tim_req.tv_sec = 0;
tim_req.tv_nsec = 2 * 1000 * 1000; // 1mSec周期
FD_ZERO(&mask);
FD_SET(NetSocket, &mask);
width = NetSocket + 1;
timeout.tv_sec = NET_TIME_OUT_SEC; // select タイムアウト時間
timeout.tv_usec = NET_TIME_OUT_USEC;
switch (select(width, &mask, NULL, NULL, &timeout)) {
case -1: // エラー
fprintf(stderr, "NetCtrlThread():select():error\r\n");
err = -1;
break;

case 0: // timeout
// fprintf(stderr, "NetCtrlThread():select():timeout\r\n");
break;

default:
if (FD_ISSET(NetSocket, &mask)) {
fprintf(stderr, "NetCtrlThread():FD_ISEET():OK\r\n");


nanosleep(&tim_req, &tim_rem);
if ((len = recv(NetSocket, NetSocRcvBuff, sizeof(NetSocRcvBuff), 0)) < 0) {
fprintf(stderr, "NetCtrlThread():recv():error\r\n");
// 受信エラー
err = -1;
// break ;
}else if(len == 0){ // Editted on 26-1-2009 to print the closing of connection
fprintf(stderr, "NetCtrlThread():recv():errormsg = Connection has been closed:\r\n");
err = -1;
// break ;
} // Edit end
else {
char buff[256];

memset(buff, 0 ,sizeof(buff));
memcpy(buff, NetSocRcvBuff, /*len*/4);
fprintf(stderr, "NetCtrlThread():recv()k:msg=%d : %s\r\n",len, buff);
// 今回受信データをアプリケーション用受信バッファにコピー
if (NetRcvSize + len > NET_RCV_BUFF_SIZE) { // バッファのガード
NetRcvSize = 0;
}

// Sending back acknowlegement of data
unsigned short type, chksum = 0 ;
unsigned long size = 0 ;
int retval ;
type = (NetSocRcvBuff[1]<<8)+NetSocRcvBuff[2] ;
size = (NetSocRcvBuff[4]<<24)+(NetSocRcvBuff[5]<<16)+(NetSocRcvBuff[6]<<8)+NetSocRcvBuff[7] ;
chksum = (NetSocRcvBuff[8]<<8)+NetSocRcvBuff[9] ;
fprintf(stderr, "NetCtrlThread() : recv (): size = :%d\r\n", size) ;
int temp = len - 10 ;
fprintf(stderr, "NetCtrlThread() : recv (): len - size = :%d\r\n", temp) ;


if(size != (len-10)){
fprintf(stderr, "NetCtrlThread():recv(): Data Received is not complete :\r\n");
//nanosleep(&tim_req, &tim_rem);
// enter in while loop to call the recv function again.
memcpy(&NetRcvBuff[0], &NetSocRcvBuff[0], 4); // コピー
}else{
memcpy(&NetRcvBuff[0], &NetSocRcvBuff[0], len); // コピー
if((retval = CheckData(type, size, chksum, /*&NetSocRcvBuff[10]*/&NetRcvBuff[10]))!=0){
fprintf(stderr, "CheckData():Error: msg= :%d\r\n", retval);
NetRcvBuff[3] = 0x4E ;
fprintf(stderr, "NetCtrlThread():recv()k:msg= : %s Data received is not correct \r\n");
}
else{
NetRcvBuff[3] = 0x41 ;
fprintf(stderr, "NetCtrlThread():recv()k:msg= : Correct data received, size=:%d checksum=:%d\r\n", size,chksum);
}

}


// memcpy(&NetRcvBuff[NetRcvSize], NetSocRcvBuff, 4); // コピー
NetRcvSize += 4;

// 受信データを検査してコマンド受信確定かどうかチェック
// コマンド受信確定であればコマンドに対応した処理を実行

#if 1 // for debug

NetSndReq(NetRcvBuff, NetRcvSize);
NetRcvSize = 0;


#endif
} // end of else


}
break;
}
if (err < 0) {
fprintf(stderr, "NetCtrlThread():error = :%d\r\n",err);
break;
}

// 送信
if (NetSndSize) {
pthread_mutex_lock(&NetBuffMutex);
if ((len = send(NetSocket, NetSocSndBuff, NetSndSize, 0)) < 0) {
fprintf(stderr, "NetCtrlThread():send():error\r\n");
// 送信エラー
err = -2;
}
else {
fprintf(stderr, "NetCtrlThread():send()k\r\n");
}
NetSndSize = 0;
pthread_mutex_unlock(&NetBuffMutex);
}
if (err < 0) {
fprintf(stderr, "NetCtrlThread():error = :%d\r\n",err);
break;
}
}
close(NetSocket); // ソケットクローズ
NetSocket = 0;
sleep(1);
}


netthrd_running = 0;

return NULL;
}

Ok, then. Just be prepared that when you come back from recv(), len might be greater than 0 but less than sizeof()NetSocRcvBuff). In that case, you need to go back and reset timeout() (because some operating systems will change that in the select() itself), do the select(), and do the recv().

Of course, for subsequent times through this new loop, you need to adjust downward the number of bytes you want to receive, and adjust upward the address of where the data should go; that is, in subsequent times through this new loop, the data shouldn't go to the very beginning of NetSocRcvBuff, because that would wipe out the data you've gotten in a previous time through the loop.

When the number of bytes you get back is equal to the remaining number of bytes you were asking for, you know you're done.

Hi -

I agree completely with wje_lq. I'd like to elaborate on a couple of points:

1. You're apparently writing your program using standard sockets, in C. That's an excellent choice - one of many, many good options you have to solve the same problem.

2. Your socket seems to be SOCK_STREAM (i.e., TCP streams, not UDP datagrams). That, too, is an excellent choice. It means that TCP will make sure you get your data in the right order, or it will return an error message to you.

In other words - if you use TCP sockets, you check for errors, and you check for partial reads, then you *should* be pretty much guaranteed that you've got all the data, in the right order.

3. TCP will *not* help you with byte ordering.
A 4-byte integer, for example, might appear "backwards" if the sending host has a different byte order than the receiving host.

4. TCP will *not* help you with "partial reads". You need to know in advance how many bytes to expect, or you need to look for some kind of delimiter or "end of file marker" to know when you stop reading.

Strong suggestion: please take a look at Beej's Guide:

http://beej.us/guide/bgnet/

'Hope that helps .. PSM

Dear wje_lq

Thank you very much for your valuable guidance and examining the problem so deeply. You are superb!!!!
I am very much thankful to all who helped me to solve this issue.

Thanks and regards

Nancy