LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 11-02-2021, 10:05 AM   #1
pedropt
Member
 
Registered: Aug 2014
Distribution: Devuan
Posts: 324

Rep: Reputation: Disabled
Reading variable and deleting multiple lines with sed at same time


Hi , i have an issue here , i am building a script where i input the lines i want to delete from a file , however if is not done at same time then sed will not delete the line i want if i do it on a loop to delete .

Imagine this code :

Text file with line numbers to delete (The numbers are not always the same)

filename: lines.tmp
Quote:
3
5
2
79
23
If i send sed to delete line 3 on a loop then when it goes for line 5 the content of that line is already in line 4 because line 3 was delete previously .

Based on that next this code
Code:
var1=$(wc -l lines.tmp | awk '{print$1}')
for i in  $(seq "$var1")
do
var2=$(sed -n ${i}p lines.tmp)

# out.file is the file where lines will be deleted
sed -i -e ${var2}d out.file
done
If i use this code then sed will only delete correctly the first file but all others will be incorrect .

Is there a way to write this code to make sed delete everything on first run .

There is also an alternative for this to run well , witch is deleting from last number line to backwards , i think .
I would like some opinions if there is a better alternative to do it .

Thank you

Last edited by pedropt; 11-02-2021 at 10:06 AM.
 
Old 11-02-2021, 11:18 AM   #2
michaelk
Moderator
 
Registered: Aug 2002
Posts: 22,081

Rep: Reputation: 4430Reputation: 4430Reputation: 4430Reputation: 4430Reputation: 4430Reputation: 4430Reputation: 4430Reputation: 4430Reputation: 4430Reputation: 4430Reputation: 4430
I would delete last to first but you also have the problem where the numbers are not in numerical order. I think the simplest would be to sort your lines.tmp via a for loop.

Code:
for i in $(sort -nr lines.tmp)
do
  echo $i
done
 
Old 11-02-2021, 11:28 AM   #3
shruggy
Senior Member
 
Registered: Mar 2020
Posts: 3,035

Rep: Reputation: Disabled
Instead of calling sed in a loop, make the file with line numbers a sed script, and call sed on it with sed -f. I.e. the file lines.tmp would be
Code:
3d
5d
2d
79d
23d
Then
Code:
sed -i -f lines.tmp out.file

Last edited by shruggy; 11-02-2021 at 11:32 AM.
 
3 members found this post helpful.
Old 11-02-2021, 11:57 AM   #4
pedropt
Member
 
Registered: Aug 2014
Distribution: Devuan
Posts: 324

Original Poster
Rep: Reputation: Disabled
nice one shruggy , did not knew that was possible with sed , gonna try out here .
 
Old 11-02-2021, 02:06 PM   #5
pedropt
Member
 
Registered: Aug 2014
Distribution: Devuan
Posts: 324

Original Poster
Rep: Reputation: Disabled
Your solution worked Shruggy , thank you .

Code:
sed -i 's/$/d/' lines.tmp # had to add the d at the end of each line so sed recognize it
sed -i -f lines.tmp out.file
 
Old 11-02-2021, 02:45 PM   #6
danielbmartin
Senior Member
 
Registered: Apr 2010
Location: Apex, NC, USA
Distribution: Mint 17.3
Posts: 1,841

Rep: Reputation: 649Reputation: 649Reputation: 649Reputation: 649Reputation: 649Reputation: 649
sed with the -f option is convenient and powerful. I think it is expensive (in execution time) if the control file is large. Is this true? Remarks from knowledgeable people are invited.

I wrote a solution which does not use the -f option.

With this InFile ...
Code:
Acura
Buick
discard
Chevrolet
Dodge
Edsel
Ford
discard this too
GMC
Honda
Infiniti
Jaguar
Kia
Lincoln
garbage
Mercedes
Nissan
Opel
get rid of this
Peugeot
... and this Line Delete file ...
Code:
3
15
19
8
... this sed ...
Code:
sed "{$(sed 's/$/d;/' $DFile)}" $InFile >$OutFile
... produced this OutFile ...
Code:
Acura
Buick
Chevrolet
Dodge
Edsel
Ford
GMC
Honda
Infiniti
Jaguar
Kia
Lincoln
Mercedes
Nissan
Opel
Peugeot
Daniel B. Martin

.
 
1 members found this post helpful.
Old 11-02-2021, 03:34 PM   #7
shruggy
Senior Member
 
Registered: Mar 2020
Posts: 3,035

Rep: Reputation: Disabled
Depending on how lines.tmp is being generated, it may be viable to make it a sed script from the get-go rather than adding d to the end of each line with an extra sed command. Then I don't see how using -f would be inferior in execution time.

But if we cannot control the format of lines.tmp then the solution by danielbmartin indeed will be preferable. Although I would simplify it by omitting the curly braces and the semicolon:
Code:
sed "$(sed 's/$/d/' $DFile)" $InFile >$OutFile
For a very large control file, I don't know if paste is more efficient than sed
Code:
paste -dd $DFile /dev/null
I suspect there's no much difference.

OTOH, if the control file were created in Windows and had CRLF as EOL then I'd use tr and kill two birds with one stone
Code:
tr \\r d <$DFile
 
3 members found this post helpful.
Old 11-02-2021, 05:09 PM   #8
danielbmartin
Senior Member
 
Registered: Apr 2010
Location: Apex, NC, USA
Distribution: Mint 17.3
Posts: 1,841

Rep: Reputation: 649Reputation: 649Reputation: 649Reputation: 649Reputation: 649Reputation: 649
Quote:
Originally Posted by shruggy View Post
... I would simplify it by omitting the curly braces and the semicolon:
Code:
sed "$(sed 's/$/d/' $DFile)" $InFile >$OutFile
Nice! You made the code cleaner and enhanced readability.

Daniel B. Martin

.
 
Old 11-02-2021, 06:55 PM   #9
pedropt
Member
 
Registered: Aug 2014
Distribution: Devuan
Posts: 324

Original Poster
Rep: Reputation: Disabled
Thank you both for the new options , however i will stay with first shruggy solution by the fact that i dont get more than 40 lines on a file and i will probably need to delete at most 10 .
the script here is working perfectly , however i will take note here of the other options in case i need to use them in future .
 
Old 11-02-2021, 07:56 PM   #10
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 20,059

Rep: Reputation: 3671Reputation: 3671Reputation: 3671Reputation: 3671Reputation: 3671Reputation: 3671Reputation: 3671Reputation: 3671Reputation: 3671Reputation: 3671Reputation: 3671
Nicely creative, by why has everyone persisted with using (multiple) sed ?. awk can handle it all simply - not to mention perl et al.
 
Old 11-02-2021, 09:01 PM   #11
danielbmartin
Senior Member
 
Registered: Apr 2010
Location: Apex, NC, USA
Distribution: Mint 17.3
Posts: 1,841

Rep: Reputation: 649Reputation: 649Reputation: 649Reputation: 649Reputation: 649Reputation: 649
Quote:
Originally Posted by syg00 View Post
Nicely creative, by why has everyone persisted with using (multiple) sed ?. awk can handle it all simply - not to mention perl et al.
The subject line of the thread specifies sed.

Daniel B. Martin

.
 
Old 11-02-2021, 09:32 PM   #12
shruggy
Senior Member
 
Registered: Mar 2020
Posts: 3,035

Rep: Reputation: Disabled
Quote:
Originally Posted by syg00 View Post
awk can handle it all simply - not to mention perl et al.
I don't see how awk is simpler in this case:
Code:
awk -v d="$DFile" '
  BEGIN {
    while ( (getline l <d) > 0 ) _[l]=1
    close(d)
  }
  !(NR in _)' "$InFile"
 
Old 11-02-2021, 10:44 PM   #13
danielbmartin
Senior Member
 
Registered: Apr 2010
Location: Apex, NC, USA
Distribution: Mint 17.3
Posts: 1,841

Rep: Reputation: 649Reputation: 649Reputation: 649Reputation: 649Reputation: 649Reputation: 649
With this InFile ...
Code:
Acura
Buick
discard
Chevrolet
Dodge
Edsel
Ford
discard this too
GMC
Honda
Infiniti
Jaguar
Kia
Lincoln
garbage
Mercedes
Nissan
Opel
get rid of this
Peugeot
... and this Line Delete file ...
Code:
3
15
19
8
... this awk ...
Code:
awk '{if (NR==FNR) D[$0]=1
 else if (!D[FNR]) print}'  \
$DFile $InFile >$OutFile
... produced this OutFile ...
Code:
Acura
Buick
Chevrolet
Dodge
Edsel
Ford
GMC
Honda
Infiniti
Jaguar
Kia
Lincoln
Mercedes
Nissan
Opel
Peugeot
Daniel B. Martin

.

Last edited by danielbmartin; 11-02-2021 at 10:51 PM. Reason: Improve readability
 
2 members found this post helpful.
Old 11-03-2021, 02:11 AM   #14
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 17,201

Rep: Reputation: 5822Reputation: 5822Reputation: 5822Reputation: 5822Reputation: 5822Reputation: 5822Reputation: 5822Reputation: 5822Reputation: 5822Reputation: 5822Reputation: 5822
Quote:
Originally Posted by pedropt View Post
Based on that next this code
Code:
var1=$(wc -l lines.tmp | awk '{print$1}')
for i in  $(seq "$var1")
do
var2=$(sed -n ${i}p lines.tmp)

# out.file is the file where lines will be deleted
sed -i -e ${var2}d out.file
done
This is an extremely over-complicated way to read a file line by line and do nothing but print the selected lines from another file.
What about bash?
Code:
while read -r var2;
do
    DEL[$var2]=1
done < lines.tmp
i=1
while read -r a_line;
dp
    [[ -n "${DEL[$i]}" ]] || echo "$a_line"
    ((i++))
done < out.file > result.file
this is not tested
 
Old 11-03-2021, 04:59 AM   #15
shruggy
Senior Member
 
Registered: Mar 2020
Posts: 3,035

Rep: Reputation: Disabled
Quote:
Originally Posted by danielbmartin View Post
... this awk ...
Nice. This could be made a one-liner, but then it would be less readable:
Code:
awk 'NR==FNR{D[$0]=1;next}!D[FNR]' "$DFile" "$InFile"
Quote:
Originally Posted by pan64 View Post
What about bash?
Don't forget about mapfile:
Code:
#!/bin/bash
mapfile -t DEL <'lines.tmp'
while read -r a_line
do [[ ${DEL[*]} =~ (^| )$((++i))( |$) ]] || printf %s\\n "$a_line"
done <'out.file'

Last edited by shruggy; 11-04-2021 at 03:21 AM.
 
2 members found this post helpful.
  


Reply

Tags
sed


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Problem with reading multiple lines from multiple files VijayB Linux - Newbie 2 11-22-2014 09:31 AM
Sed issue with finding a match on two sequential lines and deleting the previous line ulto Programming 3 10-26-2014 07:26 PM
[SOLVED] CAT command | multiple lines to multiple lines udiubu Programming 11 10-28-2011 07:09 AM
[SOLVED] Deleting the same user in differents servers at the same time... W_Chevezman Linux - Software 3 02-25-2011 02:57 PM
deleting duplicate lines without deleting first instance of the duplicated line jkeertir Linux - Newbie 2 02-07-2011 07:55 AM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 04:52 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration