System Logging - File Writing
 

Hey all,

I'm writing some system log software for Linux for a project and I am almost done but there is one piece that I am missing. The software needs to write the application messages to the log file and depending on user requirements (max log file size, number of messages, etc) remove the oldest message from the log. So I'm using a circular buffer to hold the messages. The question is when should I write to the log file? I cannot write every time data is modified in the buffer, that would be too taxing on the system. I am aware of the syslog software and was looking through the source to see when it writes to file, but was not able to find what I was looking for. Any ideas?

Thanks,
Nate

I know this isn't the answer you are looking for, but why don't you just use syslog?

As far as taxing the system goes, what language are you writing this in? Writing isn't necessarily done synchronously - for this reason some languages have commands like flush() and close() which force writing to be done.

Good question. The company I'm working for wants some features that syslog has. To include these features I would have needed to make a front end application to intercept messages. At that point I was basically writing the entire application anyhow, so we decided that it'd be best just to leave the busybox version currently running on the system there and utilize our own.

Hi -

Reinventing the wheel is seldom a good solution ... unless you have absolutely no alternative.

Please scan this article, and see if it might be applicable:

http://www.ducea.com/2006/06/06/rota...nux-log-files/

IMHO .. PSM

Thanks for the article Paul, while informative, it wasn't quite what I'm looking for. I'll try to be a little clearer. What I need is an algorithm for when to write to a file in a system log setting. When writing to disk is not possible every time data changes, what is the best approach?

Before you make the assumption that 'I cannot write every time data is modified in the buffer, that would be too taxing on the system', why don't you simply do it that way and see if your assumption is correct. You may be surprised to find how little load this adds. The filesystem is generally able to use algorithms employing buffering that does a good job of optimizing disk access efficiency.
Another factor that may make a difference is the type of filesystem used. Most have some particular emphasis on performance, and using the one that matches your requirements may be all the fine-tuning that your purpose requires.
---- rod.

The purpose of this system log is to monitor applications that are running on this board. We have a lot of them and they could all be spitting out a lot of messages at once. Writing every time a message is received is just not an option.

I guess what I'll do is create a series of flags (log file hasn't been written to in x minutes, buffer has x bytes of data more, etc) that will trigger a file write. It was my initial idea and seems very primitive, but I don't see any other solutions out there.