Get hourly log files

goboxe · 05-12-2006, 08:53 AM

Hi,

I am not very familiar with UNIX scripting.
Can someone give me pointers on how I can achive the
process below?

I need to retrieve the hourly log entries from a daily log file. The log file is generated from VoizBridge server which only can generate daily log file.

For generating the hourly entries, I think I should copy the daily log every hour and do some processing on the latest log and compare with the older log...

Does anyone have experience in doing so?

Thanks,
G

taylor_venable · 05-12-2006, 10:16 AM

I don't know anything about VoizBridge, but an easy solution for any logfile would be:

Code:

cp new.log old.log && \
cp /var/log/voizbridge.log new.log && \
diff -o old.log new.log | grep '^> ' | sed '/^> /s/^> //' > hourly_`date +%Y%m%d-%H%M%S`.log

This will keep a copy of the last log and put any differences into uniquely-named hourly log files. But note that when the logfile rolls over, you'll end up with an empty file.

goboxe · 05-12-2006, 10:21 AM

Thanks Taylor.

Do you mean that if I combine/concat all the hourly files from 00hrs to 23hrs, it will have output similar to full daily log?

Anyway I will try this and post the feedback.

Cheers,
G

taylor_venable · 05-12-2006, 10:28 AM

Yes, that's correct. And actually, forget what I said about the hourly difference being empty when the logfile gets rolled over; that won't happen. Sorry about that, I should have thought a little bit more before I said it.

goboxe · 05-12-2006, 11:15 AM

Taylor, you are superb!

It works as expected.

Another help, now how I can get this to run every hour?

Thanks in advance.

Below I share the results:

hourly_20060512-115201.log

Code:

[201263]	12/26/2005 00:01:05	0	12/26/2005 00:01:06	0	H323-H323

hourly_20060512-115313.log

Code:

[201259]	12/26/2005 00:00:37	12/26/2005 00:01:14	12/26/2005 00:01:17	3	H323-H323
[201267]	12/26/2005 00:01:27	0	12/26/2005 00:01:29	0	H323-H323
[201260]	12/26/2005 00:00:56	0	12/26/2005 00:01:29	0	H323-H323
[201269]	12/26/2005 00:01:32	0	12/26/2005 00:01:34	0	H323-H323

hourly_20060512-115409.log

Code:

[201244]	12/25/2005 23:59:40	12/25/2005 23:59:52	12/26/2005 00:01:34	102	H323-H323
[201270]	12/26/2005 00:01:34	0	12/26/2005 00:01:35	0	H323-H323
[201254]	12/26/2005 00:00:25	12/26/2005 00:00:36	12/26/2005 00:01:35	59	H323-H323
[201271]	12/26/2005 00:01:36	0	12/26/2005 00:01:39	0	H323-H323
[201264]	12/26/2005 00:01:13	12/26/2005 00:01:20	12/26/2005 00:01:41	21	H323-H323

nhydra · 05-12-2006, 11:37 AM

You can use "cron" command. More information can be found with "man cron"
Here you can take a closer look about it.
http://www.ussg.iu.edu/usail/automation/cron.html

goboxe · 05-12-2006, 12:19 PM

Thanks for your help nhydra. I really appreciate it!

frankjoshua · 05-15-2006, 12:33 PM

This may work for you:

cat `ls /location/of/logs/access.\`date +%F.\`[0-9]*` | diff -a - /location/of/logs/access.log | grep ^\> | cut -c 3- > /location/of/logs/access.`date +%F.%k`

This was used for apache logs but should work for anything. Put in cron to run every hour. Replace the word access with your log file name.

sundialsvcs · 05-15-2006, 02:58 PM

If the program that is generating the logfile keeps the logfile open and does not flush pending writes to disk, then you would not be able to get anything from the file until it had been closed. But if that is not the case, you can read the entries from the file which represent the current time-range that you wish, while the log remains open.

As others have said, the cron daemon can execute any sort of scheduled task. There are a variety of tools, such as grep and awk, which are excellent for parsing text-files. (Of the two, awk is much more versatile and deserves the half-hour or so that it takes to bone-up on it...)

goboxe · 05-15-2006, 09:20 PM

frankjoshua, also wd like to thakns for ur suggestion. I'll try it.