xhi: Pipes do store data, it is just extremely volatile. If you don't read from your pipes but keep writing to them, they will overflow

tuxdev do you know how to get around my issue?

tuxdev, thanks for the info. that is good to know.

Centinul, how long are these log lines ?

they're length changes.. it's probably anywhere from 180 - 250 characters. I really thought this method was going to work. I got really excited when my regular expressions were done and working. As soon as I tried to run it with the while loop around it, it starts working really funny like I described. Very discouraging. I'm beginning to wonder if what I want to do is possible.

hmm, yeh i dont know for sure. is it possible that getline reads the data from the pipe before your logger is done writing to the pipe? you know like jumping in and reading a half line before the whole line is written.

you should test the return value of getline and see if it is 0 or -1, -1 is error or eof, 0 is just no chars read. so that might help some and you could also throw a few prints in there and see how much it actually reading on that second read.

otherwise it looks like it should work (disregarding concurrency issues between the logger, the pipe, and the reader), id see if you are getting a -1 or 0 first..

I wonder how "tail" does this, cause I can run "tail -f" to monitor my syslog realtime.

ok I ran the program while printing out the getline code that is returned. It never returned a 0 or a -1. It always seem to return the length of the line (which in this case was either 210 or 211 characters). It still is acting in that weird fashion. Also I tend to notice that the pipe sometimes likes to die.

I think ultimately I think I need a way to get the information from the file when it changes. I'm not sure if pipes are the answer. It seems like they are to volatile and not reliable. With that said is there anyway I can accomplish reading data from a file continuously in real time?

There HAS to be a way to do this reliably and effectively. I can't really come up with any other ideas. The only other one I have is some sort of buffer mechanism where 5 - 10 lines at a time can be read into a buffer. I wouldn't know how to implement it though because I wouldn't know where the last line I used stopped.

::EDIT:: I figured out the problem. I was passing an offset by reference when doing my string parsing. I forgot to reset it to zero when I ended the string parsing. It all seems to work now.

Regarding pipes, a pipe is a kernel construct of a fixed size (the size is hardcoded into the kernel, default is 4K (or maybe 1 page, not sure)). Note sure what is meant by "volatile", no data is lost, once the pipe is full, a writing process will block until someone reads from the other end (or in non-blocking mode, returns with -1 and errno==EAGAIN). Of course, it's up to the writing process to buffer any unwritten data.

wait, so it wont overflow then. it will either block until read from or fail.

that seems to be opposite of what tuxdev mentioned earlier. dammit, now i may have to do research.. unless we can come to an agreement of who is right here.

Actually, we both be in agreement, just he explained more. Methinks a pipe is volatile because it be only in memory, and in normal case, quite active. Me essentially wanted ta say it be a Bad Thing for a pipe to reach its limit.

Someone tell me why this approach, which I thought was obvious, doesn't fly.
Read the file from beginning to end.
Call ftell(), to find out where in the file you are.
Save the result in one of:
a.) disk file
b.) chunk of shared memory
c.) memory queue
d.) anywhere that has a persistant lifetime
Close file and quit.
For all subsequent iterations:
Open the file.
Read the saved position.
fseek() to the required position.
Read to end-of-file
Get new position from ftell()
Save position, close file, quit.

This seems to make use of facilites created specifically for this purpose.
What am I missing?

--- rod.