Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question?
If it is not in the man pages or the how-to's this is the place! |
Notices |
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
Are you new to LinuxQuestions.org? Visit the following links:
Site Howto |
Site FAQ |
Sitemap |
Register Now
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
|
 |
01-14-2011, 04:31 AM
|
#1
|
LQ Newbie
Registered: Jan 2011
Posts: 6
Rep:
|
Sed adds a space instead of tab at end of line
The objective is to read a file line by line, add a tab at end of each line and add a value(number) after the tab.
My script:
Code:
i=0
val=44
while read line
do
#Ignore empty lines
case "$line" in
"") echo >> Report.tmp.tsv; continue;;
esac
case "$i" in
0)
dt=`date +%d-%m-%y`
echo $line | sed s/$/'\t'$dt/ >> Report.tmp.tsv;;
1) echo $line | sed s/$/'\t'$val/ >> Report.tmp.tsv;;
2) echo $line | sed s/$/'\t'$val/ >> Report.tmp.tsv;;
3) echo $line | sed s/$/'\t'$val/ >> Report.tmp.tsv;;
esac
i=$(($i+1))
val=$(($val+1))
done < Report.tsv
rm Report.tsv
mv Report.tmp.tsv Report.tsv
Report.tsv before running the script for the first time:
Code:
Test Line 1
This is Line number two
Line number is three
Fourth line of original report
At the end of first run, each line of Report.tsv gets appended by a space instead of a tab.
On the other hand each line of Report.tsv gets appended by tab at the end of second run onwards.
This was realized when Report.tsv was imported in open office spreadsheet.
First set of appended values get merged into the original column (Strings) and the subsequent appended values fall in distinct columns.
What is wrong here? Please guide.
Thanks in advance.
|
|
|
01-14-2011, 04:59 AM
|
#2
|
LQ Guru
Registered: Aug 2001
Location: Fargo, ND
Distribution: SuSE AMD64
Posts: 15,733
|
Sed is a line editor. A line will end with a new-line. Using "s/$/\t/" will only add a tab before the end of the line.
You can build up lines and then change all of the newlines (except the last) to tabs. This is usually done by adding a line to the Hold buffer; recalling the hold buffer; and performing a global replace "s/\n/\t/".
Here is an example, extracting one of the records of the lspci output, and replacing the newlines with tabs:
Code:
/sbin/lspci -v | sed -n '/Network controller/,/^$/{ /^$/!H
/^$/{H;g;s/\n/\t/gp}}'
Note the use of braces to group commands together when you want to perform more than one sed command inside a subrange. The `g' flag is needed at the end of the substitute command `s' to substitute all of the newlines in the line.
If all you want to do is replace all the newlines in a file with tabs, you could use the `tr' program instead:
tr '\n' '\t' <original_file >newfile
Last edited by jschiwal; 01-14-2011 at 05:00 AM.
|
|
|
01-14-2011, 05:58 AM
|
#3
|
LQ Newbie
Registered: Jan 2011
Posts: 6
Original Poster
Rep:
|
jschiwal,
Really appreciate your quick reply.
I didn't quite understand why sed adds a space (not new line) at the end of a line when it has been asked to add a tab.
When I run my script third time, first two appended columns get separated by spaces (which were by tabs earlier) and third by a proper tab.
This is extremely confusing.
Also the same sed command works like a charm if I ask it to add a comma instead of \t at end of each line.
The sed-hold buffer example you presented didn't work for me. Get "sed: -e expression #1, char 25: extra characters after command" error. I am researching more on sed and hold buffer.
And [tr '\n' '\t'] replaces every single \n by a \t so it isn't really helpful for me.
Again, thanks much for the reply.
|
|
|
01-14-2011, 06:15 AM
|
#4
|
Senior Member
Registered: Jan 2010
Posts: 2,020
|
Hi,
are you sure that it is not a setting in OpenOffice that malforms the file?
Can you post the output of the following command
Code:
od -c Report.tmp.tsv
directly after the sed's have been apllied?
|
|
|
01-14-2011, 06:22 AM
|
#5
|
Member
Registered: Mar 2008
Location: N. W. England
Distribution: Mandriva
Posts: 360
Rep: 
|
Tabs need to be quoted to survive being echoed
Code:
line=abc$'\t'123 # $'\t' is a tab character in bash
echo $line
abc 123 # tab has changed to a space
echo "$line"
abc 123 # tab has been preserved
|
|
1 members found this post helpful.
|
01-14-2011, 06:36 AM
|
#6
|
LQ Guru
Registered: Aug 2001
Location: Fargo, ND
Distribution: SuSE AMD64
Posts: 15,733
|
I cut and pasted my posted example. It didn't have an error.
If your file is highly structured, consider using awk instead of sed. However, using tabs as record separators instead of field separators is very odd. Normally, tabs separate fields in a record, and newlines separate records.
Also, put your sed commands in double quotes if you use bash variables. An alternative is to enclose fixed text in single quotes, and variables in double quotes. You need to do the latter if you use `$' in a sed command meaning end of line.
Code:
head kmenu.trace | sed "s/^/$Date\t/"
14-01-11 execve("/usr/bin/kmenuedit", ["/usr/bin/kmenuedit"], [/* 92 vars */]) = 0
14-01-11 brk(0) = 0x602000
14-01-11 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f0049ead000
14-01-11 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
Last edited by jschiwal; 01-14-2011 at 06:37 AM.
|
|
1 members found this post helpful.
|
01-14-2011, 06:47 AM
|
#7
|
LQ Newbie
Registered: Jan 2011
Posts: 6
Original Poster
Rep:
|
Quote:
Originally Posted by crts
Hi,
are you sure that it is not a setting in OpenOffice that malforms the file?
Can you post the output of the following command
Code:
od -c Report.tmp.tsv
directly after the sed's have been apllied?
|
In fact open office asks me which delimiter to use before hand upon which I select tab.
Output of
Code:
od -c Report.tmp.tsv
after 1st run:
Code:
0000000 T e s t L i n e o n e \t 1 4
0000020 - 0 1 - 1 1 \n T h i s i s L
0000040 i n e n u m b e r t w o \t 4
0000060 5 \n L i n e n u m b e r i s
0000100 t h r e e \t 4 6 \n F o u r t h
0000120 l i n e o f o r i g i n a
0000140 l r e p o r t \t 4 7 \n
0000154
Above file (generated after the first round of sed applies) opens in open office with tab as delimiter successfully.
Following is the out put of
Code:
od -c Report.tmp.tsv
after second run:
Code:
0000000 T e s t L i n e o n e 1 4
0000020 - 0 1 - 1 1 \t 1 4 - 0 1 - 1 1 \n
0000040 T h i s i s L i n e n u m
0000060 b e r t w o 4 5 \t 4 5 \n L i
0000100 n e n u m b e r i s t h r
0000120 e e 4 6 \t 4 6 \n F o u r t h
0000140 l i n e o f o r i g i n a l
0000160 r e p o r t 4 7 \t 4 7 \n
0000176
This one when opened in open office with tab as delimiter; first set of appended values get merged with the row headers.
Thanks.
|
|
|
01-14-2011, 07:34 AM
|
#8
|
LQ Newbie
Registered: Jan 2011
Posts: 6
Original Poster
Rep:
|
Quote:
Originally Posted by Kenhelm
Tabs need to be quoted to survive being echoed
Code:
line=abc$'\t'123 # $'\t' is a tab character in bash
echo $line
abc 123 # tab has changed to a space
echo "$line"
abc 123 # tab has been preserved
|
I understand.
I'm indeed on bash.
Wrote this script to test:
Code:
line="Temporary line number one"
echo $line
val=55
some=`echo $line | sed s/$/'\t'$val/`
echo "$some"
some=`echo $some | sed s/$/'\t'$val/`
echo "$some"
output:
Code:
Temporary line number one
Temporary line number one 55
Temporary line number one 55 55
Looks like for each new tab all previous tabs are lost in bash shell.
If I wrap $some by more than one set of quotes (") all the tabs are lost including the last one.
Wonder how can I solve this easily.
Thanks.
|
|
|
01-14-2011, 07:41 AM
|
#9
|
LQ Newbie
Registered: Jan 2011
Posts: 6
Original Poster
Rep:
|
jschiwal
I completely agree that tabs are normally used to separate fields and new-lines to separate records.
This articular report is to be updated on a nightly basis where it should be easy to compare values for subsequent days. Hence the upside down design!
Thanks for the pointer to awk and importance of quotes in sed.
|
|
|
01-14-2011, 07:42 AM
|
#10
|
Senior Member
Registered: Jan 2010
Posts: 2,020
|
After your post #7 I realize that you are running the script twice on the same file. I initially assumed that you are processing several different files with your script. In this case double-quoting like
should take care of the issue, as suggested by kenhelm.
|
|
|
01-14-2011, 07:45 AM
|
#11
|
LQ Newbie
Registered: Jan 2011
Posts: 6
Original Poster
Rep:
|
Following script seems to have done the trick:
Code:
i=0
val=44
while read line
do
#Ignore empty lines
case "$line" in
"") echo >> Report.tmp.tsv; continue;;
esac
case "$i" in
0)
dt=`date +%d-%m-%y`
some=`echo "$line" | sed s/$/'\t'$dt/`
echo "$some" >> Report.tmp.tsv;;
1) some=`echo "$line" | sed s/$/'\t'$val/`
echo "$some" >> Report.tmp.tsv;;
2) some=`echo "$line" | sed s/$/'\t'$val/`
echo "$some" >> Report.tmp.tsv;;
3) some=`echo "$line" | sed s/$/'\t'$val/`
echo "$some" >> Report.tmp.tsv;;
esac
i=$(($i+1))
val=$(($val+1))
done < Report.tsv
rm Report.tsv
mv Report.tmp.tsv Report.tsv
Output of od -c Report.tsv:
Code:
0000000 T e s t L i n e o n e \t 1 4
0000020 - 0 1 - 1 1 \t 1 4 - 0 1 - 1 1 \n
0000040 T h i s i s L i n e n u m
0000060 b e r t w o \t 4 5 \t 4 5 \n L i
0000100 n e n u m b e r i s t h r
0000120 e e \t 4 6 \t 4 6 \n F o u r t h
0000140 l i n e o f o r i g i n a l
0000160 r e p o r t \t 4 7 \t 4 7 \n
0000176
Thanks.
|
|
|
01-14-2011, 07:54 AM
|
#12
|
Senior Member
Registered: Jan 2010
Posts: 2,020
|
Quote:
Originally Posted by kaprasanna1
I
If I wrap $some by more than one set of quotes (") all the tabs are lost including the last one.
Wonder how can I solve this easily.
|
Not sure why you want to use 'multiple' enclosing quotes, however, if you use an even number of pairs of double-quotes then you are actually not enclosing your variable.
Example:
Code:
echo "" $line ""
^^ opening and closing quote.
^^ opening and closing quote.
As you can see, the quotes are not interpreted as outer and inner quotes. Do you want to echo a the quotes? As in
Code:
echo "$line"
"content of line"
Then you will have to escape the double-quotes when you assign the value to line:
Code:
line="\"content of line\""
Please elaborate a bit more on the situation that triggers this issue. I am not sure if I fully understand what you are trying to do.
|
|
|
01-14-2011, 10:24 AM
|
#13
|
LQ Guru
Registered: Sep 2009
Location: Perth
Distribution: Arch
Posts: 10,031
|
Maybe this can give you some ideas:
Code:
i=0
val=44
while read line
do
#Ignore empty lines
[[ -z $line ]] && continue
case $((i++)) in
0) dt=`date +%d-%m-%y`;;
[1-3]) (( dt = val++ ));;
esac
echo "$line" | sed "s/$/\t$dt/" >> Report.tmp.tsv
done < Report.tsv
rm Report.tsv
mv Report.tmp.tsv Report.tsv
|
|
|
All times are GMT -5. The time now is 03:07 AM.
|
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
|
Latest Threads
LQ News
|
|