Truncating Script

blu_alchemist · 09-13-2007, 01:17 AM

I have a server log that outputs

Code:

`/bin/date +%H%M` `/bin/date +%d` $SDB $SDC $SDD $SDE"

and the log file looks:

Code:

0540 13 33 33 35 36
0541 13 33 33 35 35
0542 13 33 33 35 36
0543 13 33 33 35 35
0544 13 32 33 35 35
0545 13 33 33 35 36
0546 13 33 33 35 36
0547 13 33 33 35 36
0548 13 33 33 35 35

The second value is the day in 1-31 format. Now I want to truncate the file at midnight so I only have that days data in this specific log file.

The original log file has thousands of entries that spans weeks and I am looking for a script that will take massive.log and take only the latest days info and make daily.log.

Ex:

Code:

cat massive.log | grep '/bin/date +%d' > daily.log

Something along that line would work perfectly EXCEPT I cannot use the '/bin/date' part in the command. I have seen many console examples such as

Code:

apt-get remove -f 'dpkg -l | grep i18n'

As a way to remove the searched packages but once again the console says cannot find 'dpkg -l | grep i18n' is their something obvious im missing??

colucix · 09-13-2007, 02:38 AM

Code:

cat massive.log | grep '/bin/date +%d' > daily.log

The correct syntax is with back quotes instead of single quotes

Code:

cat massive.log | grep `/bin/date +%d` > daily.log

Anyway, take in mind that this will match any line containing the number of the current day at any position in the file. Maybe you will not get the expected result.

blu_alchemist · 09-13-2007, 04:52 PM

All these years of computer use and I just realized theirs a '`' under that their tilde key...

Yea theirs that small problem of any times with 13xx, x13x, xx13 are also included in the file.

awk '{print $2}' would only include the second line but cut out all the others while doing so. An if statement might be able to filter out all non 2 digit numbers but im not so good at automatic scripting.

unSpawn · 09-13-2007, 05:44 PM

Quote:

Originally Posted by blu_alchemist

Now I want to truncate the file at midnight so I only have that days data in this specific log file.

Wouldn't it be easier to logrotate the logfile then? Or zap the log and restart this service?

blu_alchemist · 09-13-2007, 06:40 PM

I prefer to have the absolute minimum amount of packages installed on my servers and currently I have 100% usable systems in under 500mb. Makes things very affordable to run a raid 1+0 system with solid state media for maximum os performance.

I agree that my current method of manging log files is grossly inadequate but I was still hoping for an answer to this post as it will solve several other issues.

Im reading up on log rotate and if it will suit my needs I may add it to my pkg list but Im unsure if it can manage a massive 45000line log file and separate it into daily updates every hour.

Maybe someone knows of a full management suit that provides, CPU/swap/mem/thread/eth-traffic/drive and sys temperatures and would make log files for monthly(hourly samples) weekly(30min samples) daily(5min samples) and yet graph them onto an html page without using python, or a PNG image.