LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 12-04-2015, 03:00 PM   #1
tlegend33
LQ Newbie
 
Registered: Sep 2013
Posts: 20

Rep: Reputation: Disabled
combine multiple awk statements into single line


I'm mining logs in an attempt to create a pipe delimited file. I have the below awk statement, but I can't seem to get the appropriate output onto the correct line:


Quote:
awk '/nz_migrate table/ {print $3}/.....migration process started/ {print $5,$6} end' tony.log
table1
2015-11-29 07:00:09
table2
table3
2015-11-29 07:00:20
table4
2015-11-29 07:00:48
table5
table6
2015-11-29 07:01:19
table7
table8
2015-11-29 07:01:31
table9
table10
2015-11-29 07:01:41

The desired output is:

Quote:
table1 2015-11-29 07:00:09
table2
table3 2015-11-29 07:00:20
table4 2015-11-29 07:00:48
table5
table6 2015-11-29 07:01:19
table7
table8 2015-11-29 07:01:31
table9
table10 2015-11-29 07:01:41
Thanks!
 
Old 12-04-2015, 07:19 PM   #2
berndbausch
Senior Member
 
Registered: Nov 2013
Location: Tokyo
Distribution: Redhat/Centos, Ubuntu, Raspbian, Fedora, Alpine, Cirros, OpenSuse/SLES
Posts: 3,281

Rep: Reputation: 858Reputation: 858Reputation: 858Reputation: 858Reputation: 858Reputation: 858Reputation: 858
print always ends with ORS, usually a newline character. Use printf instead. See also my signature.

What is the end for, by the way?
 
Old 12-07-2015, 08:01 AM   #3
tlegend33
LQ Newbie
 
Registered: Sep 2013
Posts: 20

Original Poster
Rep: Reputation: Disabled
The end was most likely left over from previous attempts at this bit of functionality. printf gets everything into a single line. I need to have my output as above. Some tables will have migrate dates while others do not. If a table has a migrate date, then I need a new line. If a table does not have a migrate date, I also need a new line to start with the next table name. Thanks for your help and also the link provided. I will continue to research.

Code:
awk  '/nz_migrate table/ {printf $3}/.....migration process                              started/ {printf $5,$6}' tony.log
table1 2015-11-29 07:00:09table2table3 2015-11-29 07:00:20table4 2015-11-29 07:00:48table5table6 2015-11-29 07:01:19table7table8 2015-11-29 07:01:31table9table10 2015-11-29 07:01:41

Last edited by tlegend33; 12-07-2015 at 08:10 AM.
 
Old 12-07-2015, 08:06 AM   #4
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 18,069

Rep: Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899
If you want a newline, add it to the printf - "\n".
 
Old 12-07-2015, 08:06 AM   #5
berndbausch
Senior Member
 
Registered: Nov 2013
Location: Tokyo
Distribution: Redhat/Centos, Ubuntu, Raspbian, Fedora, Alpine, Cirros, OpenSuse/SLES
Posts: 3,281

Rep: Reputation: 858Reputation: 858Reputation: 858Reputation: 858Reputation: 858Reputation: 858Reputation: 858
Convert the first print into printf. Add code that prints a newline when the input line doesn't contain /migration process started/.
 
Old 12-07-2015, 12:12 PM   #6
tlegend33
LQ Newbie
 
Registered: Sep 2013
Posts: 20

Original Poster
Rep: Reputation: Disabled
I believe I've already converted the first print to a printf. Additionally, that last input line will not always contain /migration process started/. My biggest challenge throughout the logfile is that not all the input will be identical in format. Thanks for your assistance, I will continue to research.
 
Old 12-07-2015, 12:14 PM   #7
tlegend33
LQ Newbie
 
Registered: Sep 2013
Posts: 20

Original Poster
Rep: Reputation: Disabled
ie

Code:
nz_migrate table table1

.....processing table 1 of 1396
.....truncating the target table
.....migration process                              started at  2015-11-29 07:00:09
.....estimated # of records                                     2
.....nzload starting             ( thread 1 of 1 )
.....unloading data              ( thread 1 of 1 )
.....data flowing.....
.....unload results              ( thread 1 of 1 )              INSERT 0 2
.....unload finished             ( thread 1 of 1 )              elapsed seconds: 3
.....nzload finished             ( thread 1 of 1 )              elapsed seconds: 3
.....nzload successful           ( thread 1 of 1 )
.....migration process                              ended at    2015-11-29 07:00:12
.....data flow finished
.....actual # of records unloaded                               2
.....
.....migration completed                                        TOTAL seconds: 3
.....
.....cksum process                                  started at  2015-11-29 07:00:12
.....cksum process                                  ended at    2015-11-29 07:00:14
.....confirmed cksum: 0.0 2 table1
.....
.....cksum completed                                            TOTAL seconds: 2
.....
.....launching generate statistics (in the background)

nz_migrate table table2

.....processing table 2 of 1396
.....truncating the target table
.....source table is empty, no data to migrate
.....actual # of records unloaded                               0
.....
.....migration completed                                        TOTAL seconds: 0
 
Old 12-07-2015, 12:38 PM   #8
MadeInGermany
Senior Member
 
Registered: Dec 2011
Location: Simplicity
Posts: 1,179

Rep: Reputation: 533Reputation: 533Reputation: 533Reputation: 533Reputation: 533Reputation: 533
It's a bit tricky to get the newlines alright, especially in the first and the last section (output line).
The following uses a variable nl that is empty in the first line. At the END a pending newline is printed.
Code:
awk '/nz_migrate table/ {printf "%s", nl $3} /.....migration process/ {printf "|%s %s", $5, $6} {nl="\n"} END {printf nl}' logfile

Last edited by MadeInGermany; 12-07-2015 at 12:49 PM. Reason: quick-edit distorted everything...pipe-delimited
 
Old 12-07-2015, 02:04 PM   #9
tlegend33
LQ Newbie
 
Registered: Sep 2013
Posts: 20

Original Poster
Rep: Reputation: Disabled
This has me very close to where I need to be:

Code:
awk  '/nz_migrate table/ {printf $3"|"}/.....migration process                              started/{if ($1) print $5" "$6} end {printf "\n"}' tony.log
table1|2015-11-29 07:00:09
table2|table3|2015-11-29 07:00:20
table4|2015-11-29 07:00:48
table5|table6|2015-11-29 07:01:19
table7|table8|2015-11-29 07:01:31
table9|table10|2015-11-29 07:01:41
needs to be:

Code:
table1|2015-11-29 07:00:09
table2|
table3|2015-11-29 07:00:20
table4|2015-11-29 07:00:48
table5|
table6|2015-11-29 07:01:19
table7|
table8|2015-11-29 07:01:31
table9|
table10|2015-11-29 07:01:41
I want the 'if' statement to function where if the string is not found, then to printf "\n".
 
Old 12-07-2015, 03:12 PM   #10
tlegend33
LQ Newbie
 
Registered: Sep 2013
Posts: 20

Original Poster
Rep: Reputation: Disabled
This still gives me the same result:

Code:
awk  '/nz_migrate table/ {printf $3"|"}/.....migration process                              started/{if (length($1)==0) {print "\n";} else {print $5" "$6;}} end {printf "\n"}' tony.log
table1|2015-11-29 07:00:09
table2|table3|2015-11-29 07:00:20
table4|2015-11-29 07:00:48
table5|table6|2015-11-29 07:01:19
table7|table8|2015-11-29 07:01:31
table9|table10|2015-11-29 07:01:41
Does awk have something for a null condition? table2 should have a null 'migration process' in this scenario.

Last edited by tlegend33; 12-07-2015 at 03:14 PM.
 
Old 12-07-2015, 04:44 PM   #11
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 18,069

Rep: Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899
There are innumerable ways to fix this. Testing for a condition that doesn't exist is not a good one.
Simplest probably just to force a newline first in the initial printf. Will give you a null line as first line of output, but in need that can be worked around using something like the nl in post #8.
Better to use regex rather than a fixed number of blanks in the test too ... output formats have been known to change without warning.
 
Old 12-07-2015, 05:25 PM   #12
berndbausch
Senior Member
 
Registered: Nov 2013
Location: Tokyo
Distribution: Redhat/Centos, Ubuntu, Raspbian, Fedora, Alpine, Cirros, OpenSuse/SLES
Posts: 3,281

Rep: Reputation: 858Reputation: 858Reputation: 858Reputation: 858Reputation: 858Reputation: 858Reputation: 858
You can actually check for the absence of a pattern, see http://www.gnu.org/software/gawk/man...ssion-Patterns or use the !~ operator.

But it might be better to work with next statements in the actions - that is, when you are done with the processing of /migrate table/ lines, next; when you are done with the proceesing of /migration process/ lines, next, and then print a newline.

Depends on the overall logic of the program.
 
Old 12-07-2015, 07:01 PM   #13
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,719

Rep: Reputation: 3034Reputation: 3034Reputation: 3034Reputation: 3034Reputation: 3034Reputation: 3034Reputation: 3034Reputation: 3034Reputation: 3034Reputation: 3034Reputation: 3034
Code:
awk '/nz_migrate/{printf (x?"\n":"")$3"|";x=1}/migration process\s*started/{print $5,$6;x=0}'
Can probably be tidied up
 
Old 12-08-2015, 01:22 AM   #14
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 18,069

Rep: Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899
I like that too grail, but difficult to explain to someone not well versed.
 
Old 12-08-2015, 12:00 PM   #15
tlegend33
LQ Newbie
 
Registered: Sep 2013
Posts: 20

Original Poster
Rep: Reputation: Disabled
syg00, you are correct. I am not terribly well versed on awk programming. However, I sincerely appreciate everyone's comments on my issue. Grails solution provides what I need however I submitted only two search conditions for my scripts as I thought it would a good jumping off point for the several other strings I need to search for also:

Code:
 awk '
> /nz_migrate/{printf (x?"\n":"")$3"|";x=2} \
> /.....migration process                              started/{print $5" "$6"|";x=1 \
> /.....migration process                              ended/{print $5" "$6"|";x=0 \
> }' tony.log
awk: cmd. line:3: /.....migration process                              ended/{print $5" "$6"|";x=0 \
awk: cmd. line:3:  ^ syntax error
awk: cmd. line:3: /.....migration process                              ended/{print $5" "$6"|";x=0 \
awk: cmd. line:3:                                                            ^ syntax error
awk: cmd. line:5: }
awk: cmd. line:5:  ^ unexpected newline or end of string
I'm researching arrays so I can see how to incorporate grail's suggestions to additional criteria. Thanks to all who replied. I'll get this figured out.

Thanks!
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
awk split single column into multiple columns based on RS wolverene13 Programming 11 11-01-2012 05:07 PM
BASH or AWK: extract columns in multiple files and combine to a single file cristalp Programming 2 03-15-2012 11:55 AM
Combine output of select statements on different tables. linuxlover.chaitanya Linux - Newbie 2 11-16-2011 01:25 AM
multiple pattern search in a single file line by line saheervc Linux - Newbie 2 09-01-2010 11:45 PM
merge multiple lines of a single file into one line groverrajiv Linux - Newbie 4 05-26-2004 02:38 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 07:51 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration