LinuxQuestions.org
View the Most Wanted LQ Wiki articles.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices



Reply
 
Search this Thread
Old 01-18-2012, 04:37 PM   #1
inversecow
LQ Newbie
 
Registered: Aug 2005
Location: Victoria, British Columbia - Canada
Distribution: RHEL 5.5
Posts: 16

Rep: Reputation: 0
Question BASH: Reformatting text output to join lines


Hello all,

I am working on a script to collect I/O metrics and am having trouble reformatting output from the iostat command.
Specifically, some "normally single line" rows of data are being placed as double line results (OUTPUT 1).
I expect this has to do with output formatting rules within the utility itself, but have been unable to "unify" the two lines back into one.

What are some recommended solutions to obtain a desirable result (EG: RESULT 1)?
I would like to use tools suitable to include in a command pipeline if possible (EG: sed, tr, awk, etc).
The spacing is less important as I plan to cut the results up with awk and convert to CSV.

# OUTPUT 1
Code:
[20:42:45:user@hostname:~]
$ /usr/bin/iostat -d -x
...
cciss/c0d0        1.44    22.00  3.19  6.78   273.78   190.25    46.53     0.36   36.42   3.82   3.81
cciss/c0d0p1
0.00     0.00  0.00  0.00     0.03     0.00    18.12     0.00   27.78  26.17   0.01
cciss/c0d0p2
0.55     7.65  1.08  1.54   103.03    69.58    66.01     0.16   61.01   4.80   1.26
cciss/c0d0p3
0.88    14.35  2.11  5.24   170.72   120.67    39.62     0.20   27.68   4.22   3.10
...
# RESULT 1
Code:
[20:42:45:user@hostname:~]
$ /usr/bin/iostat -d -x | $EXTRA_COMMANDS
...
cciss/c0d0        1.44    22.00  3.19  6.78   273.78   190.25    46.53     0.36   36.42   3.82   3.81
cciss/c0d0p1      0.00     0.00  0.00  0.00     0.03     0.00    18.12     0.00   27.78  26.17   0.01
cciss/c0d0p2      0.55     7.65  1.08  1.54   103.03    69.58    66.01     0.16   61.01   4.80   1.26
cciss/c0d0p3      0.88    14.35  2.11  5.24   170.72   120.67    39.62     0.20   27.68   4.22   3.10
...

Thanks for your time.

Last edited by inversecow; 01-18-2012 at 04:40 PM.
 
Old 01-18-2012, 04:59 PM   #2
thesnow
Member
 
Registered: Nov 2010
Location: Minneapolis, MN
Distribution: Ubuntu, Red Hat, Mint
Posts: 126

Rep: Reputation: 28
Does this happen only when you output to screen, or also when you redirect to a file?
 
Old 01-18-2012, 05:01 PM   #3
inversecow
LQ Newbie
 
Registered: Aug 2005
Location: Victoria, British Columbia - Canada
Distribution: RHEL 5.5
Posts: 16

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by thesnow View Post
Does this happen only when you output to screen, or also when you redirect to a file?
Hello thesnow,

Thanks for the above reply.
Indeed this is a problem which occurs in both scenarios (screen && file).
I am writing to a file with a simple STDOUT redirection (
Code:
command > file.txt
).
 
Old 01-18-2012, 07:04 PM   #4
Nominal Animal
Senior Member
 
Registered: Dec 2010
Location: Finland
Distribution: Xubuntu, CentOS, LFS
Posts: 1,723
Blog Entries: 3

Rep: Reputation: 943Reputation: 943Reputation: 943Reputation: 943Reputation: 943Reputation: 943Reputation: 943Reputation: 943
Quote:
Originally Posted by inversecow View Post
Specifically, some "normally single line" rows of data are being placed as double line results (OUTPUT 1).
It seems iostat is splitting the line because the device name is too long. The splits occur at the start of the line, just after the device name, and not later on in the line.

Quote:
Originally Posted by inversecow View Post
What are some recommended solutions to obtain a desirable result (EG: RESULT 1)?
Pipe through
Code:
awk '{ if (NF>1) printf("%s\n", $0); else printf("%s\t", $0); }'
Quote:
Originally Posted by inversecow View Post
The spacing is less important as I plan to cut the results up with awk and convert to CSV.
In that case, you can use a "pre-rule" to merge those lines, so your awk script will see correct records:
Code:
awk '(NF==1) { getline temp; $0 = $0 "\t" temp; }
     { # your actual rule goes here
       printf("\"%s\",%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s\n", $1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12);
     }'
Note that before running the iostat and the awk script, you should probably set LANG=C and LC_ALL=C in the environment, to make sure the output does not depend on locale. For example, in my default locale, the decimal point is , . If you use Bash, start it with
Code:
#!/bin/bash
export LANG=C LC_ALL=C
To set it for just the iostat call, use
Code:
env LANG=C LC_ALL=C iostat -d -x
 
1 members found this post helpful.
Old 01-18-2012, 07:25 PM   #5
inversecow
LQ Newbie
 
Registered: Aug 2005
Location: Victoria, British Columbia - Canada
Distribution: RHEL 5.5
Posts: 16

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by Nominal Animal View Post
It seems iostat is splitting the line because the device name is too long. The splits occur at the start of the line, just after the device name, and not later on in the line.


Pipe through
Code:
awk '{ if (NF>1) printf("%s\n", $0); else printf("%s\t", $0); }'
In that case, you can use a "pre-rule" to merge those lines, so your awk script will see correct records:
Code:
awk '(NF==1) { getline temp; $0 = $0 "\t" temp; }
     { # your actual rule goes here
       printf("\"%s\",%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s\n", $1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12);
     }'
Note that before running the iostat and the awk script, you should probably set LANG=C and LC_ALL=C in the environment, to make sure the output does not depend on locale. For example, in my default locale, the decimal point is , . If you use Bash, start it with
Code:
#!/bin/bash
export LANG=C LC_ALL=C
To set it for just the iostat call, use
Code:
env LANG=C LC_ALL=C iostat -d -x
Hello Nominal Animal,

Thanks for the in-depth response.
This fixed my issue beautifully! :-D
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
need help to slit multiple lines of text output at whitspaces in shell Ryanjon7 Programming 4 10-20-2011 10:44 PM
bash text to variable accessing individual text lines patolfo Programming 11 05-11-2010 11:21 AM
Join lines in text file vidyashankara Linux - General 10 12-21-2009 04:17 PM
[SOLVED] BASH - how to substitute many lines in a text file at once carolflb Programming 2 10-20-2009 11:21 AM
join every three lines of a text file powah Programming 8 02-02-2007 12:40 AM


All times are GMT -5. The time now is 02:39 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration