Perl question

dnoy · 12-26-2007, 07:30 PM

I have a bunch of log files that begin with unix time and im trying to convert it all to human readable. Im trying to run a program that will look at my log files (data.txt) and take each line (i have thousands of lines) and convert the unix time into human readable. This means that i want the unix time to be taken out of the line and replaced with human readable. The below program is what i have so far. This is my first program so i apologize if my programing/documentation suck.

#!/usr/bin/perl
$file = '/data/data.txt';
$outputfl = '/data/output.txt';
open (INFO, $file);
$num=1;
open (OUTPUT, ">$outputfl");
#@lines = <INFO>;
#close (INFO);
#print @lines;
#while ($line = <INFO>)
while (<INFO>)
{
system ("\`echo OUTPUT\|cut -d' ' -f1"); --doesnt work
#this is how its done in bash -- `echo $line|cut -d' ' -f1
#I also need to run date --date=@$mydate +"%Y%m%d-%X %Z" and
#change the unix time to human readable
print system ("$_|cut -d'");
$num ++;
print "\n$num";
}

ghostdog74 · 12-26-2007, 08:13 PM

show your sample log file and your expected output

dnoy · 12-26-2007, 11:44 PM

Below are the logs im analyzing:

124376766.026 RELEASE -1 FFFFFFFF 4734C557F9315105CA6BE0FA56B94D55 200 1124276674 -1 -1 unknown -1/0 POST http://reports.hotbar.com/reports/hotbar/4.0/HbRpt.dll
1124392388.975 RELEASE -1 FFFFFFFF 810FFBF233584C330353CF0A8C31F5D2 503 -1 -1 -1 unknown -1/813 POST http://log.cc.cometsystems.com/dss/cc.2_0_0.report_u

I run this command:
echo 124376766.026 RELEASE -1 FFFFFFFF 4734C557F9315105CA6BE0FA56B94D55 200 1124276674 -1 -1 unknown -1/0 POST http://reports.hotbar.com/reports/hotbar/4.0/HbRpt.dll|cut -d' ' -f1

and get

124376766.026

then i run

date --date=@124376766.026 +"%Y%m%d-%X %Z" --this and the other command are bash, i would appreciate if anyone could help me convert them to perl or should i just use system ()?

and get

19731210-06:06:06 AM MST

i want to take that output and put it back into my logs and replace the unix time so they look like this:

19731210-06:06:06 AM MST RELEASE -1 FFFFFFFF 4734C557F9315105CA6BE0FA56B94D55 200 1124276674 -1 -1 unknown -1/0 POST http://reports.hotbar.com/reports/hotbar/4.0/HbRpt.dll

Thank you,

Hobbletoe · 12-27-2007, 09:35 AM

Try this ...

Code:

#!/usr/bin/perl
${file} = '/data/data.txt';
${outputfl} = '/data/output.txt';
open (INFO, ${file});
open (OUTPUT, ">${outputfl}");
while (<INFO>)
{
 @{line_in}=split;
 @{line_in[0]}=`date --date=\@@{line_in[0]} +'%Y%m%d-%X %Z'`;
 chomp @{line_in[0]};
 print(OUTPUT join(' ', @{line_in}));
 print(OUTPUT "/n");
}
close INFO;
close OUTPUT;

I split the fields out into an array (by default, split assumes a space and the $_ variable. It could have been split(' ', $_) just as easily. Then, through my research this morning, system will only return the success/failure of the command (i.e., 1, 0, -1, 256 (error number)). You need to use the backticks to return the output. I had to chomp that ${line_in[0]} to get rid of the return. Then it was a simple matter of printing all back out.

As a note, you will see that I enclose all of my variables in braces {}. I do this so that I have better control of my variables, and know exactly where they end. I once tried to echo a variable with some letters directly after it. Say $abcd where only $a is my variable. Perl assumed that I wanted to use the entire thing as a variable. So, ${a}bcd corrected my issue for me, and I just stuck with it and use it everywhere.

dnoy · 12-29-2007, 02:00 PM

I ran the program with the following input in data.txt

387702480 RELEASE -1 FFFFFFFF 4734C557F9315105CA6BE0FA56B94D55 200 1124276674 -1 -1 unknown -1/0 POST http://reports.hotbar.com/reports/hotbar/4.0/HbRpt.dll
387702480 RELEASE -1 FFFFFFFF 810FFBF233584C330353CF0A8C31F5D2 503 -1 -1 -1 unknown -1/813 POST http://log.cc.cometsystems.com/dss/cc.2_0_0.report_u
387702480 RELEASE -1 FFFFFFFF 4734C557F9315105CA6BE0FA56B94D55 200 1124276674 -1 -1 unknown -1/0 POST http://reports.hotbar.com/reports/hotbar/4.0/HbRpt.dllo
387702480 RELEASE -1 FFFFFFFF 4734C557F9315105CA6BE0FA56B94D55 200 1124276674 -1 -1 unknown -1/0 POST http://reports.hotbar.com/reports/hotbar/4.0/HbRpt.dll
387702480 RELEASE -1 FFFFFFFF 4734C557F9315105CA6BE0FA56B94D55 200 1124276674 -1 -1 unknown -1/0 POST http://reports.hotbar.com/reports/hotbar/4.0/HbRpt.dll
387702480 RELEASE -1 FFFFFFFF 4734C557F9315105CA6BE0FA56B94D55 200 1124276674 -1 -1 unknown -1/0 POST http://reports.hotbar.com/reports/hotbar/4.0/HbRpt.dll
387702480 RELEASE -1 FFFFFFFF 4734C557F9315105CA6BE0FA56B94D55 200 1124276674 -1 -1 unknown -1/0 POST http://reports.hotbar.com/reports/hotbar/4.0/HbRpt.dll
387702480 RELEASE -1 FFFFFFFF 4734C557F9315105CA6BE0FA56B94D55 200 1124276674 -1 -1 unknown -1/0 POST http://reports.hotbar.com/reports/hotbar/4.0/HbRpt.dll
387702480 RELEASE -1 FFFFFFFF 4734C557F9315105CA6BE0FA56B94D55 200 1124276674 -1 -1 unknown -1/0 POST http://reports.hotbar.com/reports/hotbar/4.0/HbRpt.dll
387702480 RELEASE -1 FFFFFFFF 4734C557F9315105CA6BE0FA56B94D55 200 1124276674 -1 -1 unknown -1/0 POST http://reports.hotbar.com/reports/hotbar/4.0/HbRpt.dll
387702480 RELEASE -1 FFFFFFFF 4734C557F9315105CA6BE0FA56B94D55 200 1124276674 -1 -1 unknown -1/0 POST http://reports.hotbar.com/reports/hotbar/4.0/HbRpt.dll
387702480 RELEASE -1 FFFFFFFF 4734C557F9315105CA6BE0FA56B94D55 200 1124276674 -1 -1 unknown -1/0 POST http://reports.hotbar.com/reports/hotbar/4.0/HbRpt.dll
387702480 RELEASE -1 FFFFFFFF 4734C557F9315105CA6BE0FA56B94D55 200 1124276674 -1 -1 unknown -1/0 POST http://reports.hotbar.com/reports/hotbar/4.0/HbRpt.dll
1113548880 RELEASE -1 FFFFFFFF 4734C557F9315105CA6BE0FA56B94D55 200 1124276674 -1 -1 unknown -1/0 POST http://google.com

my output looks like this:

19820415-12:08:00 AM MST RELEASE -1 FFFFFFFF 4734C557F9315105CA6BE0FA56B94D55 200 1124276674 -1 -1 unknown -1/0 POST http://reports.hotbar.com/reports/ho...20415-12:08:00 AM MST RELEASE -1 FFFFFFFF 810FFBF233584C330353CF0A8C31F5D2 503 -1 -1 -1 unknown -1/813 POST http://log.cc.cometsystems.com/dss/c...20415-12:08:00 AM MST RELEASE -1 FFFFFFFF 4734C557F9315105CA6BE0FA56B94D55 200 1124276674 -1 -1 unknown -1/0 POST http://reports.hotbar.com/reports/ho...20415-12:08:00 AM MST RELEASE -1 FFFFFFFF 4734C557F9315105CA6BE0FA56B94D55 200 1124276674 -1 -1 unknown -1/0 POST http://reports.hotbar.com/reports/ho...20415-12:08:00 AM MST RELEASE -1 FFFFFFFF 4734C557F9315105CA6BE0FA56B94D55 200 1124276674 -1 -1 unknown -1/0 POST http://reports.hotbar.com/reports/ho...20415-12:08:00 AM MST RELEASE -1 FFFFFFFF 4734C557F9315105CA6BE0FA56B94D55 200 1124276674 -1 -1 unknown -1/0 POST http://reports.hotbar.com/reports/ho...20415-12:08:00 AM MST RELEASE -1 FFFFFFFF 4734C557F9315105CA6BE0FA56B94D55 200 1124276674 -1 -1 unknown -1/0 POST http://reports.hotbar.com/reports/ho...20415-12:08:00 AM MST RELEASE -1 FFFFFFFF 4734C557F9315105CA6BE0FA56B94D55 200 1124276674 -1 -1 unknown -1/0 POST http://reports.hotbar.com/reports/ho...20415-12:08:00 AM MST RELEASE -1 FFFFFFFF 4734C557F9315105CA6BE0FA56B94D55 200 1124276674 -1 -1 unknown -1/0 POST http://reports.hotbar.com/reports/ho...20415-12:08:00 AM MST RELEASE -1 FFFFFFFF 4734C557F9315105CA6BE0FA56B94D55 200 1124276674 -1 -1 unknown -1/0 POST http://reports.hotbar.com/reports/ho...20415-12:08:00 AM MST RELEASE -1 FFFFFFFF 4734C557F9315105CA6BE0FA56B94D55 200 1124276674 -1 -1 unknown -1/0 POST http://reports.hotbar.com/reports/ho...20415-12:08:00 AM MST RELEASE -1 FFFFFFFF 4734C557F9315105CA6BE0FA56B94D55 200 1124276674 -1 -1 unknown -1/0 POST http://reports.hotbar.com/reports/ho...20415-12:08:00 AM MST RELEASE -1 FFFFFFFF 4734C557F9315105CA6BE0FA56B94D55 200 1124276674 -1 -1 unknown -1/0 POST http://reports.hotbar.com/reports/ho...50415-01:08:00 AM MDT RELEASE -1 FFFFFFFF 4734C557F9315105CA6BE0FA56B94D55 200 1124276674 -1 -1 unknown -1/0 POST http://google.com/n

as you can see the first line looks good but every other line is not right. the date is inserted into the middle. Also the last date for google doesnt even show up. Any suggestions?

Thank you

syg00 · 12-29-2007, 04:28 PM

Try "\n" in place of "/n"

chrism01 · 12-29-2007, 09:19 PM

Date conversion in Perl:

Code:

use Time::localtime;
$tm = localtime($time);
printf("Dateline: %02d:%02d:%02d-%04d/%02d/%02d\n",
    $tm->hour, $tm->min, $tm->sec, $tm->year+1900,
    $tm->mon+1, $tm->mday);

Hobbletoe: re "enclose all of my variables in braces {}.", please don't, it hurts the eyes and distracts the guy who comes after you (if you do this at work).

dnoy · 12-29-2007, 09:29 PM

the \n did the trick, i cant believe i missed that one

Thanks for everyones help

bigearsbilly · 01-04-2008, 02:25 PM

or

Code:

use POSIX;
print strftime "%Y%m%d-%X %Z", localtime 124376766.026

19731210-13:06:06 GMT

bigearsbilly · 01-04-2008, 03:06 PM

Code:

#!/usr/bin/perl -n
use POSIX;

($time, $line) = split " ", $_, 2;

print strftime "%Y%m%d-%X %Z $line", localtime $time;

Code:

[billy@daffy lxq]$ ./1.pl 1
19731210-13:06:06 GMT RELEASE -1 FFFFFFFF 4734C557F9315105CA6BE0FA56B94D55 200 1124276674 -1 -1 unknown -1/0 POST http://reports.hotbar.com/reports/hotbar/4.0/HbRpt.dll
20050818-20:13:08 BST RELEASE -1 FFFFFFFF 810FFBF233584C330353CF0A8C31F5D2 503 -1 -1 -1