LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
LinkBack Search this Thread
Old 01-27-2009, 09:15 AM   #1
NancyT
LQ Newbie
 
Registered: Nov 2008
Posts: 15

Rep: Reputation: 0
faulty transmission of 100 kb data using TCP/IP protocol


Hi Forum

I transmit a 100kb file in a single server, single client application( with a windows client and Linux server).
Before transmitting, the data is fragmented into smaller packets each of 8kb size. Each packet is assigned with a sequence number so that data can be recombined correctly at receiver end. On the Windows client side I first wait for the acknowledge sent by the linux server for each correctly received packet and then send the next packet. This method is works sometimes, but sometimes does not work. Client does not receive acknowledgment on expected time, and I fail to recombine the data in correct sequence.What could be the probable error. Kindly guide how a smooth transmission can be ensured?

Thanking in advance
Nancy
 
Old 01-27-2009, 06:47 PM   #2
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.5, Centos 5.10
Posts: 16,093

Rep: Reputation: 1995Reputation: 1995Reputation: 1995Reputation: 1995Reputation: 1995Reputation: 1995Reputation: 1995Reputation: 1995Reputation: 1995Reputation: 1995Reputation: 1995
I think we'd need to see the code. At the very least, the key parts and just confirm which lang, which distro. Are you writing your own protocol? Can you not use eg scp, Samba, NFS, FTP etc?
 
Old 01-27-2009, 08:22 PM   #3
wje_lq
Member
 
Registered: Sep 2007
Location: Mariposa
Distribution: Debian lenny, Slackware 12
Posts: 806

Rep: Reputation: 161Reputation: 161
Or, more to the point, are you doing this because you're trying to solve the problem presented by this thread?

Because if that's the problem you're trying to solve, I've read that thread, and some of the responses there are sufficiently wrong to scare me, and others are right on the button.

YOur original question in that thread was this:
Quote:
When I send a large size file (100kb) to the server the data is not received as a single TSP. It arrives in 4-5 packets of different sizes.
How can I receive the complete data without any changes..??
You meant TCP, of course, and the answer is this:

TCP acts as a stream. The data doesn't move at the speed of light. It's quite possible (and often happens) that a program receiving TCP data can process the data faster than it comes in. So you'll get part of the data you want, and you'll just have to wait for the rest. It's up to you to know how much data you're expecting. You start filling up a buffer with this data, and every time you get more data, you add it to the buffer. Simple as that.

If that was your concern, you don't need to spend one more minute on this project of artificially splitting and recombining the data. Just collect it as it comes.

So let's review some of the things in the other thread that might motivate you to split up the data artificially:
Quote:
Is it a good approach to devide the data in some packets of same size and assign some serial number to each of them. Then at the receiver end again combine the packets according to the erial numbers assigned..?
No. Don't bother. Just keep filling up the buffer until it contains as much data as you were expecting.
Quote:
TCP/IP max message size is 65535 bytes, including header size, thus in mean for you a maximum payload of 65507 bytes in UDP datagram, and I think something near for a TCP packet.
True but irrelevant. When you do a read() or recv(), you will get as much data as has been collected for you so far, no matter how many TCP packets are involved. You shouldn't even worry about TCP packets; they're an implementation detail you can forget about.
Quote:
If what you want to send trough an IP network is over this value, this is your job to cut it into smaller chunk.
No, if you're using TCP, it's the job of the TCP stack to cut it into smaller chunks as necessary. That's one of the neat things about TCP.
Quote:
some IP stack are buggy then recommended maximum size for datagrams is 8192 bytes.
I'm guessing here that anticipating a buggy TCP stack is premature, until you understand what you're doing. TCP stacks which are that buggy are rather rare. No programs I've seen which use TCP concern themselves with workarounds for buggy TCP stacks.
Quote:
And if you want to minimizing packet fragmentation, the max size must be inferior to the minimum MTU of your network (generally MTU is 1500 bytes).
That's a performance issue. It's one you might be able to fine tune, but get your program working first. :)
Quote:
on the output window I see the data arrivec in 3-4 loops.I display the no of bytes returned with each successful call of recv(). So I found that the large data is not coming at once.
This is not a bug. You should expect this.
Quote:
can you please guide how can I devide the packets ensuring that they will be received properly. Secondly how can I aggregate the packets in proper sequence
Again, this is TCP's job. Just keep read()ing or recv()ing until you get all the data you were expecting.
Quote:
TCP is a stream oriented protocol. The sender shoves its data into the pipeline, and the receiver gets the data out in the same order, guaranteed. The size and timing of the chunks that get transmitted along the way are outside the realm of the application.
Bingo! Absolutely correct! "Outside the realm" means outside the application's control and outside the application's concern.
Quote:
The receiver either must know a priori how much data the stream is to contain, or read iteratively until it reads EOF. The sender can cause the receiver to read EOF by closing the socket. Closing the socket should force it to be flushed, I think.
Again, correct. But no need to worry about flushing; with or without closing the socket, what the sender write()s or send()s will come along soon enough.
Quote:
If your message must be receivable as a single datagram, you will have to use UDP
But that's not what you're trying to do, right? You're just trying to get all the data?
 
Old 01-27-2009, 09:07 PM   #4
NancyT
LQ Newbie
 
Registered: Nov 2008
Posts: 15

Original Poster
Rep: Reputation: 0
Key parts of source code

The Application has limitations not to use other protocols.
I am not writing a new protocol, just want to use TCP/IP to transfer large size data.
Language used is "C"


void *NetCtrlThread(void *param)
{
int l_soc = -1;

netthrd_running = 1;
netthd_stop = 0;

fprintf(stderr, "NetCtrlThread():start\r\n");
// ネットワーク制御
while (!netthd_stop/*1*/) {
// listenソケット生成
if ((l_soc = NetListen(NET_PORT_NO)) < 0) {
// 暫定 エラー処理
fprintf(stderr, "NetCtrlThread():listen error\r\n");
sleep(1);
continue;
}
fprintf(stderr, "NetCtrlThread():listen ok\r\n");

// accept
if ((NetSocket = NetAccept(l_soc)) < 0) {
// 暫定 エラー処理
fprintf(stderr, "NetCtrlThread():accept error\r\n");
close(l_soc);
sleep(1);
continue;
}
fprintf(stderr, "NetCtrlThread():accept ok\r\n");

// listenソケットクローズ
close(l_soc);

// select
struct timeval timeout;
fd_set mask;
int width;
int len, err;

err = 0;
while (!netthd_stop/*1*/) {
// 受信
// fprintf(stderr, "NetCtrlThread():select():start\r\n");
struct timespec tim_req, tim_rem;
tim_req.tv_sec = 0;
tim_req.tv_nsec = 2 * 1000 * 1000; // 1mSec周期
FD_ZERO(&mask);
FD_SET(NetSocket, &mask);
width = NetSocket + 1;
timeout.tv_sec = NET_TIME_OUT_SEC; // select タイムアウト時間
timeout.tv_usec = NET_TIME_OUT_USEC;
switch (select(width, &mask, NULL, NULL, &timeout)) {
case -1: // エラー
fprintf(stderr, "NetCtrlThread():select():error\r\n");
err = -1;
break;

case 0: // timeout
// fprintf(stderr, "NetCtrlThread():select():timeout\r\n");
break;

default:
if (FD_ISSET(NetSocket, &mask)) {
fprintf(stderr, "NetCtrlThread():FD_ISEET():OK\r\n");


nanosleep(&tim_req, &tim_rem);
if ((len = recv(NetSocket, NetSocRcvBuff, sizeof(NetSocRcvBuff), 0)) < 0) {
fprintf(stderr, "NetCtrlThread():recv():error\r\n");
// 受信エラー
err = -1;
// break ;
}else if(len == 0){ // Editted on 26-1-2009 to print the closing of connection
fprintf(stderr, "NetCtrlThread():recv():errormsg = Connection has been closed:\r\n");
err = -1;
// break ;
} // Edit end
else {
char buff[256];

memset(buff, 0 ,sizeof(buff));
memcpy(buff, NetSocRcvBuff, /*len*/4);
fprintf(stderr, "NetCtrlThread():recv()k:msg=%d : %s\r\n",len, buff);
// 今回受信データをアプリケーション用受信バッファにコピー
if (NetRcvSize + len > NET_RCV_BUFF_SIZE) { // バッファのガード
NetRcvSize = 0;
}

// Sending back acknowlegement of data
unsigned short type, chksum = 0 ;
unsigned long size = 0 ;
int retval ;
type = (NetSocRcvBuff[1]<<8)+NetSocRcvBuff[2] ;
size = (NetSocRcvBuff[4]<<24)+(NetSocRcvBuff[5]<<16)+(NetSocRcvBuff[6]<<8)+NetSocRcvBuff[7] ;
chksum = (NetSocRcvBuff[8]<<8)+NetSocRcvBuff[9] ;
fprintf(stderr, "NetCtrlThread() : recv (): size = :%d\r\n", size) ;
int temp = len - 10 ;
fprintf(stderr, "NetCtrlThread() : recv (): len - size = :%d\r\n", temp) ;


if(size != (len-10)){
fprintf(stderr, "NetCtrlThread():recv(): Data Received is not complete :\r\n");
//nanosleep(&tim_req, &tim_rem);
// enter in while loop to call the recv function again.
memcpy(&NetRcvBuff[0], &NetSocRcvBuff[0], 4); // コピー
}else{
memcpy(&NetRcvBuff[0], &NetSocRcvBuff[0], len); // コピー
if((retval = CheckData(type, size, chksum, /*&NetSocRcvBuff[10]*/&NetRcvBuff[10]))!=0){
fprintf(stderr, "CheckData():Error: msg= :%d\r\n", retval);
NetRcvBuff[3] = 0x4E ;
fprintf(stderr, "NetCtrlThread():recv()k:msg= : %s Data received is not correct \r\n");
}
else{
NetRcvBuff[3] = 0x41 ;
fprintf(stderr, "NetCtrlThread():recv()k:msg= : Correct data received, size=:%d checksum=:%d\r\n", size,chksum);
}

}


// memcpy(&NetRcvBuff[NetRcvSize], NetSocRcvBuff, 4); // コピー
NetRcvSize += 4;

// 受信データを検査してコマンド受信確定かどうかチェック
// コマンド受信確定であればコマンドに対応した処理を実行

#if 1 // for debug

NetSndReq(NetRcvBuff, NetRcvSize);
NetRcvSize = 0;


#endif
} // end of else


}
break;
}
if (err < 0) {
fprintf(stderr, "NetCtrlThread():error = :%d\r\n",err);
break;
}

// 送信
if (NetSndSize) {
pthread_mutex_lock(&NetBuffMutex);
if ((len = send(NetSocket, NetSocSndBuff, NetSndSize, 0)) < 0) {
fprintf(stderr, "NetCtrlThread():send():error\r\n");
// 送信エラー
err = -2;
}
else {
fprintf(stderr, "NetCtrlThread():send()k\r\n");
}
NetSndSize = 0;
pthread_mutex_unlock(&NetBuffMutex);
}
if (err < 0) {
fprintf(stderr, "NetCtrlThread():error = :%d\r\n",err);
break;
}
}
close(NetSocket); // ソケットクローズ
NetSocket = 0;
sleep(1);
}


netthrd_running = 0;

return NULL;
}
 
Old 01-27-2009, 09:48 PM   #5
wje_lq
Member
 
Registered: Sep 2007
Location: Mariposa
Distribution: Debian lenny, Slackware 12
Posts: 806

Rep: Reputation: 161Reputation: 161
Ok, then. Just be prepared that when you come back from recv(), len might be greater than 0 but less than sizeof()NetSocRcvBuff). In that case, you need to go back and reset timeout() (because some operating systems will change that in the select() itself), do the select(), and do the recv().

Of course, for subsequent times through this new loop, you need to adjust downward the number of bytes you want to receive, and adjust upward the address of where the data should go; that is, in subsequent times through this new loop, the data shouldn't go to the very beginning of NetSocRcvBuff, because that would wipe out the data you've gotten in a previous time through the loop.

When the number of bytes you get back is equal to the remaining number of bytes you were asking for, you know you're done.
 
Old 01-27-2009, 11:33 PM   #6
paulsm4
Guru
 
Registered: Mar 2004
Distribution: SusE 8.2
Posts: 5,863
Blog Entries: 1

Rep: Reputation: Disabled
Hi -

I agree completely with wje_lq. I'd like to elaborate on a couple of points:

1. You're apparently writing your program using standard sockets, in C. That's an excellent choice - one of many, many good options you have to solve the same problem.

2. Your socket seems to be SOCK_STREAM (i.e., TCP streams, not UDP datagrams). That, too, is an excellent choice. It means that TCP will make sure you get your data in the right order, or it will return an error message to you.

In other words - if you use TCP sockets, you check for errors, and you check for partial reads, then you *should* be pretty much guaranteed that you've got all the data, in the right order.

3. TCP will *not* help you with byte ordering.
A 4-byte integer, for example, might appear "backwards" if the sending host has a different byte order than the receiving host.

4. TCP will *not* help you with "partial reads". You need to know in advance how many bytes to expect, or you need to look for some kind of delimiter or "end of file marker" to know when you stop reading.

Strong suggestion: please take a look at Beej's Guide:

http://beej.us/guide/bgnet/

'Hope that helps .. PSM
 
Old 01-28-2009, 01:07 AM   #7
NancyT
LQ Newbie
 
Registered: Nov 2008
Posts: 15

Original Poster
Rep: Reputation: 0
Dear wje_lq

Thank you very much for your valuable guidance and examining the problem so deeply. You are superb!!!!
I am very much thankful to all who helped me to solve this issue.

Thanks and regards

Nancy
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: BSDCan 2008: Stream Control Transmission Protocol LXer Syndicated Linux News 0 05-17-2008 07:00 PM
Secure data transmission introuble Programming 10 04-27-2007 02:31 AM
Data Rate transmission discrepancy suggesting hijack drmjh Linux - Security 13 03-23-2007 04:29 PM
Recovering data on RAID 5 with semi-faulty hdd davidsaxton Linux - Hardware 2 09-16-2006 03:45 PM
project on secure stream control transmission protocol(SCTP) monindra Fedora - Installation 1 05-29-2005 03:47 AM


All times are GMT -5. The time now is 01:03 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration