combine multiple awk statements into single line

tlegend33 · 12-04-2015, 03:00 PM

I'm mining logs in an attempt to create a pipe delimited file. I have the below awk statement, but I can't seem to get the appropriate output onto the correct line:

Quote:

awk '/nz_migrate table/ {print $3}/.....migration process started/ {print $5,$6} end' tony.log
table1
2015-11-29 07:00:09
table2
table3
2015-11-29 07:00:20
table4
2015-11-29 07:00:48
table5
table6
2015-11-29 07:01:19
table7
table8
2015-11-29 07:01:31
table9
table10
2015-11-29 07:01:41

The desired output is:

Quote:

table1 2015-11-29 07:00:09
table2
table3 2015-11-29 07:00:20
table4 2015-11-29 07:00:48
table5
table6 2015-11-29 07:01:19
table7
table8 2015-11-29 07:01:31
table9
table10 2015-11-29 07:01:41

Thanks!

berndbausch · 12-04-2015, 07:19 PM

print always ends with ORS, usually a newline character. Use printf instead. See also my signature.

What is the end for, by the way?

tlegend33 · 12-07-2015, 08:01 AM

The end was most likely left over from previous attempts at this bit of functionality. printf gets everything into a single line. I need to have my output as above. Some tables will have migrate dates while others do not. If a table has a migrate date, then I need a new line. If a table does not have a migrate date, I also need a new line to start with the next table name. Thanks for your help and also the link provided. I will continue to research.

Code:

awk  '/nz_migrate table/ {printf $3}/.....migration process                              started/ {printf $5,$6}' tony.log
table1 2015-11-29 07:00:09table2table3 2015-11-29 07:00:20table4 2015-11-29 07:00:48table5table6 2015-11-29 07:01:19table7table8 2015-11-29 07:01:31table9table10 2015-11-29 07:01:41

syg00 · 12-07-2015, 08:06 AM

If you want a newline, add it to the printf - "\n".

berndbausch · 12-07-2015, 08:06 AM

Convert the first print into printf. Add code that prints a newline when the input line doesn't contain /migration process started/.

tlegend33 · 12-07-2015, 12:12 PM

I believe I've already converted the first print to a printf. Additionally, that last input line will not always contain /migration process started/. My biggest challenge throughout the logfile is that not all the input will be identical in format. Thanks for your assistance, I will continue to research.

tlegend33 · 12-07-2015, 12:14 PM

ie

Code:

nz_migrate table table1

.....processing table 1 of 1396
.....truncating the target table
.....migration process                              started at  2015-11-29 07:00:09
.....estimated # of records                                     2
.....nzload starting             ( thread 1 of 1 )
.....unloading data              ( thread 1 of 1 )
.....data flowing.....
.....unload results              ( thread 1 of 1 )              INSERT 0 2
.....unload finished             ( thread 1 of 1 )              elapsed seconds: 3
.....nzload finished             ( thread 1 of 1 )              elapsed seconds: 3
.....nzload successful           ( thread 1 of 1 )
.....migration process                              ended at    2015-11-29 07:00:12
.....data flow finished
.....actual # of records unloaded                               2
.....
.....migration completed                                        TOTAL seconds: 3
.....
.....cksum process                                  started at  2015-11-29 07:00:12
.....cksum process                                  ended at    2015-11-29 07:00:14
.....confirmed cksum: 0.0 2 table1
.....
.....cksum completed                                            TOTAL seconds: 2
.....
.....launching generate statistics (in the background)

nz_migrate table table2

.....processing table 2 of 1396
.....truncating the target table
.....source table is empty, no data to migrate
.....actual # of records unloaded                               0
.....
.....migration completed                                        TOTAL seconds: 0

MadeInGermany · 12-07-2015, 12:38 PM

It's a bit tricky to get the newlines alright, especially in the first and the last section (output line).
The following uses a variable nl that is empty in the first line. At the END a pending newline is printed.

Code:

awk '/nz_migrate table/ {printf "%s", nl $3} /.....migration process/ {printf "|%s %s", $5, $6} {nl="\n"} END {printf nl}' logfile

tlegend33 · 12-07-2015, 02:04 PM

This has me very close to where I need to be:

Code:

awk  '/nz_migrate table/ {printf $3"|"}/.....migration process                              started/{if ($1) print $5" "$6} end {printf "\n"}' tony.log
table1|2015-11-29 07:00:09
table2|table3|2015-11-29 07:00:20
table4|2015-11-29 07:00:48
table5|table6|2015-11-29 07:01:19
table7|table8|2015-11-29 07:01:31
table9|table10|2015-11-29 07:01:41

needs to be:

Code:

table1|2015-11-29 07:00:09
table2|
table3|2015-11-29 07:00:20
table4|2015-11-29 07:00:48
table5|
table6|2015-11-29 07:01:19
table7|
table8|2015-11-29 07:01:31
table9|
table10|2015-11-29 07:01:41

I want the 'if' statement to function where if the string is not found, then to printf "\n".

tlegend33 · 12-07-2015, 03:12 PM

This still gives me the same result:

Code:

awk  '/nz_migrate table/ {printf $3"|"}/.....migration process                              started/{if (length($1)==0) {print "\n";} else {print $5" "$6;}} end {printf "\n"}' tony.log
table1|2015-11-29 07:00:09
table2|table3|2015-11-29 07:00:20
table4|2015-11-29 07:00:48
table5|table6|2015-11-29 07:01:19
table7|table8|2015-11-29 07:01:31
table9|table10|2015-11-29 07:01:41

Does awk have something for a null condition? table2 should have a null 'migration process' in this scenario.

syg00 · 12-07-2015, 04:44 PM

There are innumerable ways to fix this. Testing for a condition that doesn't exist is not a good one.
Simplest probably just to force a newline first in the initial printf. Will give you a null line as first line of output, but in need that can be worked around using something like the nl in post #8.
Better to use regex rather than a fixed number of blanks in the test too ... output formats have been known to change without warning.

berndbausch · 12-07-2015, 05:25 PM

You can actually check for the absence of a pattern, see http://www.gnu.org/software/gawk/man...ssion-Patterns or use the !~ operator.

But it might be better to work with next statements in the actions - that is, when you are done with the processing of /migrate table/ lines, next; when you are done with the proceesing of /migration process/ lines, next, and then print a newline.

Depends on the overall logic of the program.

grail · 12-07-2015, 07:01 PM

Code:

awk '/nz_migrate/{printf (x?"\n":"")$3"|";x=1}/migration process\s*started/{print $5,$6;x=0}'

Can probably be tidied up

syg00 · 12-08-2015, 01:22 AM

I like that too grail, but difficult to explain to someone not well versed.

tlegend33 · 12-08-2015, 12:00 PM

syg00, you are correct. I am not terribly well versed on awk programming. However, I sincerely appreciate everyone's comments on my issue. Grails solution provides what I need however I submitted only two search conditions for my scripts as I thought it would a good jumping off point for the several other strings I need to search for also:

Code:

 awk '
> /nz_migrate/{printf (x?"\n":"")$3"|";x=2} \
> /.....migration process                              started/{print $5" "$6"|";x=1 \
> /.....migration process                              ended/{print $5" "$6"|";x=0 \
> }' tony.log
awk: cmd. line:3: /.....migration process                              ended/{print $5" "$6"|";x=0 \
awk: cmd. line:3:  ^ syntax error
awk: cmd. line:3: /.....migration process                              ended/{print $5" "$6"|";x=0 \
awk: cmd. line:3:                                                            ^ syntax error
awk: cmd. line:5: }
awk: cmd. line:5:  ^ unexpected newline or end of string

I'm researching arrays so I can see how to incorporate grail's suggestions to additional criteria. Thanks to all who replied. I'll get this figured out.

Thanks!