Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game. |
Notices |
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
Are you new to LinuxQuestions.org? Visit the following links:
Site Howto |
Site FAQ |
Sitemap |
Register Now
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
|
 |
|
06-10-2016, 06:10 AM
|
#1
|
Member
Registered: Sep 2005
Posts: 201
Rep:
|
sed search for a string, duplicate original and replace the string
I have a file where I have some placeholders. I want
to duplicate the lines with a matching string, do a replacement on the new line and carry on.
For example, original file:
blablablaa
blablablaa
blablablaa
blablablaa
blablablaa<replaceme>blablablaa
blablablaa
blablablaa
now I can replace the <replaceme> with sed "s#<replaceme>#REPLACEMENT#"
but what I want is:
blablablaa
blablablaa
blablablaa
blablablaa
blablablaa<replaceme>blablablaa
blablablaaREPLACEMENTblablablaa
blablablaa
blablablaa
In the end I want to do it several times and in the end remove this original line, so that I have:
blablablaa
blablablaa
blablablaa
blablablaa
blablablaaREPLACEMENT_1blablablaa
blablablaaREPLACEMENT_2blablablaa
blablablaaREPLACEMENT_3blablablaa
blablablaaREPLACEMENT_4blablablaa
blablablaa
blablablaa
How can I accomplish this with sed?
|
|
|
06-10-2016, 06:22 AM
|
#2
|
LQ Veteran
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 21,333
|
With great difficulty. Why the insistence on sed ?.
|
|
1 members found this post helpful.
|
06-10-2016, 10:43 AM
|
#3
|
Member
Registered: Sep 2005
Posts: 201
Original Poster
Rep:
|
That was the first that came into my mind - whereas I can accomplish few pathetic string operations with sed I am completely illitarate when it comes to awk or such things.
|
|
|
06-10-2016, 09:32 PM
|
#4
|
LQ 5k Club
Registered: Oct 2003
Location: Melbourne
Distribution: Slackware64-15.0
Posts: 6,525
|
You can use back references in sed to go from the original file to the intermediate.
Code:
sed 's#\(.*\)<replaceme>\(.*\)#\1<replaceme>\2\n\1REPLACEMENT\2#' <original>
Or, more concisely, using the & character that represents what was matched
Code:
sed 's#\(.*\)<replaceme>\(.*\)#&\n\1REPLACEMENT\2#' <original>
Last edited by allend; 06-10-2016 at 09:53 PM.
|
|
|
06-11-2016, 12:01 AM
|
#5
|
LQ Veteran
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 21,333
|
Sed is good at substitution(s), but that incrementing number requirement makes it just an ugly hack. There are people here that will take that as a challenge - not me ....
I'm normally averse to just giving out solutions, but this tossed up some interesting side notes. Here's some awk that should hopefully be reasonably readable
Code:
awk '{if (!/<replaceme>/) {print ; next} ; for (i=1; i<5; i++) { a=gensub(/<replaceme>/, "REPLACEMENT_"i, "1") ; print a}} ' input.file
Some assumptions I took:
- always substitute the found string.
- only first found on each line changed.
- 4 new lines substituted always.
|
|
|
06-11-2016, 12:36 AM
|
#6
|
Member
Registered: Sep 2005
Posts: 201
Original Poster
Rep:
|
Sorry, I think it confused a bit that I made an increment into the replacement text. Actually I have a file with these placeholders.
I have another file with replacement strings. I read this second file line by line and want to duplicate the original line with placeholders and
simultaneously do the replacement for the duplicated line. This way when I get the next replacement line I can repeat the procedure.
So maybe a more accurate description would be:
Template with placeholders:
blablablaablablablaablablablaa
blablablaablablablaa
blablablaa<placeholder>blablablaablablablaa
blablablaablablablaablablablaa<placeholder>
blablablaa
replacement strings in the second file:
first_replacement
another_one
After first loop the file should look like:
blablablaablablablaablablablaa
blablablaablablablaa
blablablaa<placeholder>blablablaablablablaa
blablablaa first_replacementblablablaablablablaa
blablablaablablablaablablablaa<placeholder>
blablablaablablablaablablablaa first_replacement
blablablaa
After the second loop:
blablablaablablablaablablablaa
blablablaablablablaa
blablablaa<placeholder>blablablaablablablaa
blablablaa another_oneblablablaablablablaa
blablablaa first_replacementblablablaablablablaa
blablablaablablablaablablablaa<placeholder>
blablablaablablablaablablablaa another_one
blablablaablablablaablablablaa first_replacement
blablablaa
In the end I can delete the lines with placeholders. The amount of replacement strings in the second file may vary...
I think the solution might be already presented - I'll muck around a bit and we'll see...
The framework for the file operations I already have:
Code:
#!/bin/bash
rm -rf tmp.txt
touch tmp.txt
while read line; do
sed for duplication of lines with <placeholder> and replacing <placeholder> with $line in the duplicate
done < $1
sed to delete the lines with <placeholder>
Last edited by Jykke; 06-11-2016 at 12:49 AM.
|
|
|
06-11-2016, 01:31 AM
|
#7
|
Member
Registered: Sep 2005
Posts: 201
Original Poster
Rep:
|
Ok, both seds from allend came very close to hacking it. The only problem is that it would not work if there were multiple <placeholder> on the same line.
Additionally I intend a second replacement operation with the counter, so one iteration more:
sourcefile:
blabla<id>blabla<placeholder>bla
blablabla<id>,<id>blabla
blablablablablablablablabla
replacement strings:
first_replacement
another_replacement
source should become:
blabla<id>blabla<placeholder>bla
blabla 1blabla first_replacementbla
blabla 2blabla second_replacementbla
blablabla<id>,<id>blabla
blablabla 1, 1blabla
blablabla 2, 2blabla
blablablablablablablablabla
Code:
#!/bin/bash
i=1
while read line; do
sed -i 's#\(.*\)<placeholder>\(.*\)#&\n\1'$line'\2#' $2
sed -i 's#\(.*\)<id>\(.*\)#&\n\1'$i'\2#' $2
i=$((i+1))
done < $1
sed to delete the lines with <placeholder>
The & is a good idea, but somehow the replacement need to be modified.
I could ease it a bit by saying that as a matter of fact there are limited amount of replacements,
actually a formating and I think I'll get there but a universal solution would be more educating 
|
|
|
06-11-2016, 08:03 AM
|
#8
|
LQ 5k Club
Registered: Oct 2003
Location: Melbourne
Distribution: Slackware64-15.0
Posts: 6,525
|
The task has morphed from your original post. I suggest that you look at alternative tools such as awk or perl.
With awk, you could read the replacement strings into an array, then add lines as required while iterating over the array.
Code:
awk 'BEGIN {i=0;while ((getline s < "replacement_strings.txt") > 0) {i++;r[i]=s}} {print} /<placeholder>/ {for (i in r) {s=gensub(/<placeholder>/,r[i],"1");s=gensub(/<id>/,i,"g",s);print s}} /<id>,/ {for (i in r) {s=gensub(/<id>/,i,"g"); print s}}' sourcefile
|
|
|
06-13-2016, 08:14 AM
|
#9
|
LQ Guru
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 11,091
|
Kindly remember a few things . . .
- You have a dozen programming languages to choose from, and any of them are just a #!shebang away. sed, like "bash scripting," is suitable for only the most-trivial applications (IMHO), and, while awk is an advance over this, "the world is your oyster."
- "One-liners," generally, are Evil. (IMHO.) It's okay for your solution to consist of more than one file.
- Above all, design and write your solution to be durable, flexible, and maintainable. Please don't write "write-only code."
I've more-or-less made my living out of cleaning-up after people. 
|
|
1 members found this post helpful.
|
06-13-2016, 08:49 AM
|
#10
|
Moderator
Registered: Jan 2005
Location: Central Florida 20 minutes from Disney World
Distribution: SlackwareŽ
Posts: 13,971
|
Moderator response
Moved: This thread is more suitable in <Programming> and has been moved accordingly to help your thread/question get the exposure it deserves.
|
|
|
06-13-2016, 09:35 AM
|
#11
|
LQ 5k Club
Registered: Oct 2003
Location: Melbourne
Distribution: Slackware64-15.0
Posts: 6,525
|
If the replacement strings are in a file named 'replacement_strings.txt',
and the source file is named 'sourcefile',
then a file named 'replace.awk' containing the code below (multiline version of post #8)
can be used with the command 'awk -f replace.awk sourcefile',
to achieve the requested output.
Code:
# Read contents of replacement_strings.txt file into array r[]
BEGIN {
i=0
while ((getline s < "replacement_strings.txt") > 0)
{i++;r[i]=s}}
# Echo all lines from input file
{print}
# If line contains '<placeholder>', add lines with substitutions
# for <placeholder> for each element of array r[] and
# for <id> with a numeric identifier of the element of the array
/<placeholder>/ {
for (i in r)
{s=gensub(/<placeholder>/,r[i],"1")
s=gensub(/<id>/,i,"g",s)
print s}}
# If line contains '<id>,', add lines with substitutions
# for <id> with a numeric identifier for each element of array r[]
/<id>,/ {
for (i in r)
{s=gensub(/<id>/,i,"g")
print s}}
Last edited by allend; 06-13-2016 at 09:44 AM.
|
|
|
06-13-2016, 10:36 AM
|
#12
|
Member
Registered: Aug 2013
Location: Sweden
Distribution: Debian, Arch, Red Hat, CentOS
Posts: 773
|
Quote:
Originally Posted by sundialsvcs
"One-liners," generally, are Evil. (IMHO.) It's okay for your solution to consist of more than one file.[*] Above all, design and write your solution to be durable, flexible, and maintainable.
|
^This should be axiomatic. Well written!
Best regards,
HMW
|
|
|
06-13-2016, 03:56 PM
|
#13
|
LQ Guru
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 11,091
|
Quote:
Originally Posted by allend
If the replacement strings are in a file named 'replacement_strings.txt',
[...]
then ...
|
"Hear! Hear!"
IMHO, a very critical point has been made here: "the solution has now been generalized!"
" If When(!), maybe someday many times in the future, someone needs to replace something else," they might only need to add a single line into a text file.
Furthermore, in order to convince themselves that their actions will produce the expected result, they need only examine the [i]well-formatted, easily readable, awk file with comments(!)" in order to reliably understand what's going on.
... and "if, somehow, that was what I could actually count on, in my job" ...  ...
(a) I would feel like I had just died and gone to Heaven, and ...
(b) I probably wouldn't have a job. 
|
|
|
06-20-2016, 09:46 AM
|
#14
|
Senior Member
Registered: Jan 2004
Posts: 1,420
Rep: 
|
Quote:
Originally Posted by sundialsvcs
"One-liners," generally, are Evil. (IMHO.) It's okay for your solution to consist of more than one file.[*]
|
I'm curious why your against one-liners.
I've seen other posts where people brag that they have optimize others code into one line and I've also seen job posts where they ask "what is the one-liner you are proud of most and why?"
|
|
|
06-29-2016, 06:48 AM
|
#15
|
LQ Guru
Registered: Aug 2004
Location: Sydney
Distribution: Rocky 9.2
Posts: 18,425
|
One liners are a fun exercise at home to see what you can do, but as a long time worker in the industry, in a commercial env they are (generally) a pita; hard to debug and maintain.
Unintended side-effects are a common problem, as are unexpected results with impure inputs.
K.I.S.S. https://en.wikipedia.org/wiki/KISS_principle
Last edited by chrism01; 06-29-2016 at 06:51 AM.
|
|
|
All times are GMT -5. The time now is 02:18 AM.
|
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
|
Latest Threads
LQ News
|
|