LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices



Reply
 
Search this Thread
Old 11-22-2010, 09:53 AM   #1
XXLRay
Member
 
Registered: May 2010
Posts: 126

Rep: Reputation: 16
How to replace newline pattern in file by other newline pattern in a shell script


I have several (vhdl) files containing a pattern with newline characters that I need to replace by another pattern that also contains newline characters.

I start with something like:
Code:
    port (
      A : in std_logic_vector(width - 1 downto 0);
      Z : out std_logic;
    );
I want to replace it by something like:
Code:
    port (
      A : in std_logic_vector(width - 1 downto 0);
      B : in std_logig;
      Z : out std_logic;
    );
(I need to paste some lines)

As I need to do this (very) often I want to use a shell script.

I tried:

1.
Code:
#!/bin/sh
sed -i 's/      A : in std_logic_vector(width - 1 downto 0);
      Z : out std_logic;/    A : in std_logic_vector(width - 1 downto 0);
      B : in std_logig;
      Z : out std_logic;/g' ./testfile
result:
Code:
sed: -e expression #1, char 5: unterminated `s' command
2.
Code:
#!/bin/sh
sed -i 's/      A : in std_logic_vector(width - 1 downto 0);\n      Z : out std_logic;/    A : in std_logic_vector(width - 1 downto 0);\n      B : in std_logig;\n      Z : out std_logic;/g' ./testfile
result:
File remains unchanged

3.
Code:
#!/bin/sh
awk '{sub(/      A : in std_logic_vector(width - 1 downto 0);
      Z : out std_logic;/,"    A : in std_logic_vector(width - 1 downto 0);
      B : in std_logig;
      Z : out std_logic;")}1' ./testfile
result:
Code:
unexpected newline or end of string
4.
Code:
#!/bin/sh
awk '{sub(/      A : in std_logic_vector(width - 1 downto 0);\n      Z : out std_logic;/,"    A : in std_logic_vector(width - 1 downto 0);\n      B : in std_logig;\n      Z : out std_logic;")}1' ./testfile
result:
Displays the unchanged testfile

Are there any suggestions how I can automate the pattern replacement?

Code:
$ sed --version
GNU sed version 4.1.5
 
Old 11-22-2010, 10:34 AM   #2
colucix
Moderator
 
Registered: Sep 2003
Location: Bologna
Distribution: CentOS 6.5 OpenSuSE 12.3
Posts: 10,509

Rep: Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957
A multi-line pattern is recognized only if you store all the lines into the pattern space. Using the following sed command, basically you store all the lines into the pattern space separated by newline, perform the substitution and finally print the result:
Code:
sed -n '1h; 1!H; g; s/      A : in std_logic_vector(width - 1 downto 0);\n      Z : out std_logic;/      A : in std_logic_vector(width - 1 downto 0);\n      B : in std_logig;\n      Z : out std_logic;/g; $p' testfile
Problems may arise if the file is very large, since all the content will be stored into the pattern space. Moreover, this one-liner is a monster, due to the length of the patterns. Maybe you can change the logic, for example by adding text after the line containing 'A :' but it depends on the actual task to achieve.
 
1 members found this post helpful.
Old 11-23-2010, 02:54 AM   #3
XXLRay
Member
 
Registered: May 2010
Posts: 126

Original Poster
Rep: Reputation: 16
Wow, thank you. It seems to work. Until now I only tested a small file. The final file really is quite big (more than 50,000 lines). I will see how it works for it.

I only have one remark for others following my attempt. To perform the action in place (i.e. directly inside the file) you have to add 'i' as parameter:
Code:
sed -ni '1h; 1!H; g; s/foo\nbar/foo\nfoobar\nfoo/g; $p' ./testfile
This will turn the content of ./testfile
Code:
foo
bar
into
Code:
foo
foobar
bar
 
Old 11-23-2010, 07:15 AM   #4
XXLRay
Member
 
Registered: May 2010
Posts: 126

Original Poster
Rep: Reputation: 16
Just to give an update:

I applied the method to the 50,000 lines file. It took about 15 minutes on a Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz but at least it works.
 
Old 11-23-2010, 01:16 PM   #5
Kenhelm
Member
 
Registered: Mar 2008
Location: N. W. England
Distribution: Mandriva
Posts: 333

Rep: Reputation: 141Reputation: 141
The sed code is slow because it applies a 's' command at each cycle and so uses 50000 of them.
Either one of these should be faster; they both read all of the file into the pattern space and then use just one 's' command.
Code:
sed -in '1h; 1!H; ${g; s/foo\nbar/foo\nfoobar\nfoo/g;p}' ./testfile

sed -i ':a $!{N;ba}; s/foo\nbar/foo\nfoobar\nfoo/g' ./testfile
 
2 members found this post helpful.
Old 11-24-2010, 02:54 AM   #6
XXLRay
Member
 
Registered: May 2010
Posts: 126

Original Poster
Rep: Reputation: 16
I tried the two proposals and both work nearly in zero time. Thank you.
 
Old 11-24-2010, 05:51 PM   #7
crts
Senior Member
 
Registered: Jan 2010
Posts: 1,604

Rep: Reputation: 446Reputation: 446Reputation: 446Reputation: 446Reputation: 446
large files

Hi,

though your problem is solved I'd like to add another solution to address the large file "issue".
Code:
sed -i '/foo/ ! {/bar/ {x; /foo/ i\
foobar
x}};h' /path/to/file
This will not hold the complete file in memory to process it. It will insert the line 'foobar' if the lines 'foo' and 'bar' are in consecutive order.

As for the large file "issue":
It is not really an issue. Suppose every line contains 100 characters. with 50000 lines the filesize would be appr. 5MB. This is still negligible. So there is nothing wrong with the previous solutions. In fact, Kenhelm's is probably the fastest. So consider the above code just as complementary "backup" in the highly unlikely case that you might have to process a file of several hundred MB's in size.

PS: You might want to change the single quotes to double quotes and use variables for the patterns 'foo' and 'bar'. This way you keep the 'sed' readable if the pattern is big as in your case. You also gain flexibility.
 
Old 11-25-2010, 04:13 AM   #8
XXLRay
Member
 
Registered: May 2010
Posts: 126

Original Poster
Rep: Reputation: 16
Quote:
Originally Posted by crts View Post
You might want to change the single quotes to double quotes and use variables for the patterns 'foo' and 'bar'. This way you keep the 'sed' readable if the pattern is big as in your case. You also gain flexibility.
For a pattern like:
Code:
foo bar
foo
bar
bar foo
Would this look like:
Code:
#!/bin/sh
FILE="./testfile"
START="foo bar
foo"
END="bar
bar foo"
INSERTION="foobar"
sed -i "/$START/ ! {/$END/ {x; /$BEGINNING/ i\$INSERTION x}};h" $FILE
???
 
Old 11-26-2010, 04:03 AM   #9
crts
Senior Member
 
Registered: Jan 2010
Posts: 1,604

Rep: Reputation: 446Reputation: 446Reputation: 446Reputation: 446Reputation: 446
Quote:
Originally Posted by XXLRay View Post
For a pattern like:
Code:
foo bar
foo
bar
bar foo
Would this look like:
Code:
#!/bin/sh
FILE="./testfile"
START="foo bar
foo"
END="bar
bar foo"
INSERTION="foobar"
sed -i "/$START/ ! {/$END/ {x; /$BEGINNING/ i\$INSERTION x}};h" $FILE
???
No, if you have patterns that spread over a variable number of lines, then it is probably best to stick with the substitution command solution.
BTW, this
Code:
sed -i "/$START/ ! {/$END/ {x; /$BEGINNING/ i\$INSERTION x}};h" $FILE
is not correct. It is important to have it in this format:
Code:
sed -i "/$START/ ! {/$END/ {x; /$BEGINNING/ i\
$INSERTION
x}};h" $FILE
Otherwise sed will think that 'x' is also some content that needs to be inserted. But it is really a command.

Here is how you use the whole pattern in memory alternative with multiline patterns:
Code:
START="foo bar\nfoo"
END="bar\nbar foo"
INSERTION="foobar"

sed -e ":a $ ! {N;ba}; s/$START\n$END/$START\n$INSERTION\n$END/g" /path/to/file
Notice, that you use '\n' instead of a "real" newline (hitting enter) in your patterns.
 
1 members found this post helpful.
Old 11-29-2010, 08:57 AM   #10
XXLRay
Member
 
Registered: May 2010
Posts: 126

Original Poster
Rep: Reputation: 16
Ah - ok thank you. I just misunderstood what you wrote and was already suspicious.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
How to replace string pattern with multi-line text in bash script? brumela Linux - Newbie 6 04-21-2011 07:56 AM
Shell script to delete file if a pattern is found and restart servers sun81 Linux - Newbie 2 07-22-2010 03:24 PM
Sed to read the file and replace and insert the pattern saurabhchokshi Programming 2 06-12-2009 02:15 PM
VI replacement: newline for newline not working! jhwilliams Linux - Software 3 08-16-2007 07:11 PM
vim: search pattern / replace with file naflan Programming 5 01-07-2006 04:16 PM


All times are GMT -5. The time now is 03:59 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration