Hi,

My application developed using C and running on Linux.

A process has a char buffer(4096 bytes) which contains a log statement which is a comma seperated string of the following information:
loglevel, errorlevel, Module name, function name, description of the error etc.
(Variable length string and can go upto 4096 characters)

Now my requirement is to send this comma seperated string to a different process through a FIFO.

My doubt is,
1) is it better to use write(fifofile,buffer,4096) or
2) should i use any other function like fputs or fwrite(fifofile,strlen(buffer))?

If i go by option 1, i would read 4096 bytes safely at the reading process. Also it is equal to the PIPE_BUF so, the process wont be interrupted by any signal upto PIPE_BUF size. But i am not sure whether this is an efficient way, because we are reading/writing lot of unnecessary characters.

Buf if I do Fwrite(strlen(buffer)), while reading from FIFO i am not sure how many bytes to look for. I cannot use fgets because it might stop reading if it sees a new line, and it would become difficult if the log statement itself contains a new line in between.

Please guide me with the proper function to read from or write into a FIFO.

Thanks,
Sathya

I'd open a FILE buffer with fdopen and write the formatted output directly to there with fprintf instead of using write directly.
ta0kira

As ta0kira says, you're much better off opening a stream and using the *printf family of functions rather than using a raw write.

As for reading, there are any number of ways of formatting the output. You could prefix the message with the length e.g. fprintf(fifo, "%d%s", strlen(msg), msg) and use fscanf(fifo, "%d", &len) to read back the length, then fread to suck up len bytes.

Alternatively, you could end each log message with a unique delimiter character/string; a blank line, "EOF\n", "%%\n", whatever. Then just loop over fgets until you get the delimiter string.

I have written applications that use fifos to send data to other processes. I used read/write because I was sending structured data and not strings. I just make sure the first couple of fields identify the structure type and size. It worked well for me. For strings, you may want to use what the previous posters indicated though.

If you are sending log data, you may want to look into using log4cxx. This is a logger library that allows you to easily redirect where logging data goes by simply changing configuration files. It already has built in logging levels you can use. There is some up front work required but it is really full featured. For example you can log to a file, socket, or memory or whatever just by changing the config file and not messing with your code directly. I have used this before and it works well. It may only work with C++ though (not sure). When using that library I use ostringstream to format the strings first since that logger does not accept printf style statements.

When i use fprintf(server_fifo,"%d%s",length,arr) while writing and fscanf(pipe_fd,"%d",&length);
printf("Length = %d\n", length);
fread((void *)arr,sizeof(char),length,pipe_fd); while reading, I am not able to read the length properly. The length value is totally a junk value. I dont understand why.

Also I have a doubt as to whether this fprintf(with a %d%s) is an atomic operation or not.

Please clarify.

Thanks,
Sathya

Could it be that your loglevel is being read in with the length string (if it is a numeric value)? Do you have something in between the two that will cause fscanf to know that loglevel is distinct?

Actually my array contains string as below:

"1000,CR,[04/16/08 13:28:45] [9238] [cobmaknk] [cobm_iProcessParseErrorPayment], M.14: ,Installing converter failed,START processing Payment, 0 pending\n"

When i say, fprintf(fifo,"%d%s",length,arr), the length(151 characters) gets appended to the string, and while reading using fscanf I am getting 1511000.

I think i should read the length using %s only, and then convert it to integer using atoi.

The problem is your log message begins with a number so fscanf will read that too. You're writing "1511000....", this is a single stream of characters so fscanf can't magically know it was written as two numbers. It will read any characters valid for "%d", i.e. any sequence of denary digits. Simply seperate the length with a non-digit character, "%d %s", "%d:%s" etc.

You will of course have to account for that character when you read the message (i.e. read the length, then read one character and discard it).

i.e.

I tried using fscanf to send length and the string seperated by a comma character, and read out the length and the extra character, and finally the "length" bytes. This works properly. I tried using 1 server, and 1000 client instances. Then also i am able to write and read very fast. Below is the code for that.

I have one more doubt. I read that write to a fifo is an atomic operation upto PIPE_BUF bytes. But i am not sure, if it is atomic for any write/read functions (i.e fprintf, fscanf, fgetc, fputc etc or is it specific for just the two low level calls read/write)

My the test run i did, i could see the output properly with 1000 clients, but i would require confirmation on that.

Please clarify.

<Code>
ServerSide:
while(1) {
read_res = fscanf(pipe_fd,"%d%c",&length,&c);
if (read_res == EOF)
{
memset(arr,'\0',4096);length=0;continue;
}
read_res = fread((void *)arr,sizeof(char),length,pipe_fd);
if (read_res ==length)
printf("bytes read =%d %s\n",read_res,arr);
else
perror("read error");
}
ClientSide:
write_res = fprintf(server_fifo,"%d,%s",length,arr);
</Code>

Thanks,
Sathya

When i run the read as a continuous process(in an infinite loop), i observe that the fscanf returns EOF when there is nothing to read from the FIFO. Fscanf doesnt get blocked when there is nothing to read from FIFO. This consumes CPU.

So what can be done to reduce cpu consumption? Should i make the process sleep for sometime and check the FIFO every few minutes?

(For my requirement, since i would be logging only critical errors, this process will not get data in the FIFO all the time. It will get only during error conditions). So i want it to be blocked rather than polling continuously.

Please clarify.

Thanks,
Sathya

Indeed, writing to a fifo is atomic in that a single write will not be interleaved with data written to the same pipe from another process. However, the stdio stream functions (fprintf etc) add another layer on top of the low-level read/write. I don't think fprintf in general can guarantee atomic operation, but if you add an fflush after the fprintf and ensure the total size of the data written is less than PIPE_BUF (which you have already done), then it should be atomic.

You should use select(2) or poll(2) to wait on the pipe, see the man pages on these. These will block until data is ready to read (or write or until for a given timeout). There is also a "tutorial" man page for select (select_tut) which gives a few examples. Either will do, poll is IMO a bit more straightforward to use.

This gives an example of select and poll usage.

The difficult part about doing it that way is error recovery. One accidental byte in the wrong place can throw off everything else if you don't somehow account for bad data and find a way to realign input with the structured output of the other process. At least with line I/O you can reset at the next line if there's an error. Or you might just use datagram sockets instead.setlinebuf is a good replacement for fflush after each line. If you need multiple lines to stay together as a group, you should probably use fcntl and F_GETLK + F_SETLKW to lock the file.

I think you're misusing "atomic" here, also. Formatted output is not atomic because it is possible for another thread or process to observe a state where the called function is partially complete. System calls such as write are atomic in that nothing can observe an interim state where write has started and hasn't completed. It seems what you're talking about is just getting it all to fit into a single write operation. I don't think the size of the pipe buffer (1 page?) has bearing over atomicity. If the buffer is full, the internally-called write operation will just block.
ta0kira

A pipe is (just) a continuous stream, and it's entirely up to you to impose whatever protocol you need to impose upon it. All that you can be sure of is that, when you write a chunk of data to the pipe, all of it will be written, in one atomic operation.

You should therefore make sure that you write the data in a format that can be understood and verified. For instance, you might write a known delimiter, the length, a known delimiter, the data, and another known delimiter. The program that reads the pipe knows what data-format to expect and, thanks to the various delimiters, is able to verify that it did in fact receive it (and to scream horribly if it does not).

But remember: you're not the first person on the planet to have a need to read-and-write data through a pipe. Therefore, make sure that you are not re-inventing any wheels. You probably are.

As suggested, i went in for a select().
I opened an named pipe fd in read/nonblocking mode, and then associated a stream using fdopen

int pipe1_fd = open("/tmp/myfirstfifo", open_mode);
FILE *pipe_fd= fdopen(pipe1_fd,"r");

First time, the read gets blocked on select with a time out of 20 secs as set in the function call.
Once there is something in the pipe, select returns and FD_ISSET succeeds.

But after the first success and read from pipe,the while loop continues and the next select always returns 1 instead of blocking again for 20 secs. I am not able to reset the fds' value. How can i do that? What went wrong? Is it because i am mixing file descriptor and the stream?

The code is

main()
{
fd_set fds;
while(1)
{
if(isready(pipe_fd,&fds)==1)
// there is data to process...
// read using fscanf and fread
else
// there is no data to process. need to block on select
}
}


isready(int fd, fd_set *fds)
{
FD_ZERO(fds);
FD_SET(pipe_fd,fds);
rc = select(fd+1, fds, NULL, NULL, NULL);
if rc<0
return -1
FD_ISSET(fd,fds)?1:0;
}


Please clarify

Thanks a lot for helping me out till now. All these information were very useful

Sathya

in the select function call, i had set tv.sec=20 and passed &tv instead of NULL.
Sorry for the cut and paste error..

Please clarify my doubt.

Thanks
Sathya