awk and sed headaches

Drosera_capensis · 04-03-2020, 04:21 AM

Hello everyone, I am trying to set up a nested while loop to select a string of text and print the string row the row after as an output.

I could not manage to do it with awk, as it print the whole document ($0?) and not only the string and the line below.

Code:

cat list_1 | while read g;  
 do
  cat list_2 | while read s;
   do 
    touch ./output/$g
    awk '$0==$s {print $0}' ./*/*/$g > ./output/$g ;
  done
done

I have tried as well with sed, and this command does not work with a variable, where it works with /regexp/.

Code:

cat list_1 | while read g;  
do
 cat list_2 | while read s;
  do 
   touch ./output/$g
   sed -n '/$s/, +1p' ./*/*/$g > ./output/$g ; 
  done
done

Could you indicate me what could be the mistake here?

pan64 · 04-03-2020, 04:32 AM

yes, both awk and sed can read stdin.
So the cat piped into while will be first read by while, but afterward the sed or awk will continue to read it and the while will not get anything more ....
You need to close stdin like:

Code:

sed .... </dev/null > file
# or
awk .... </dev/null > file

furthermore $s is not evaluated between ' signs.
Would be nice to use shellcheck to check your script.

Turbocapitalist · 04-03-2020, 04:53 AM

The single quotes mean that the string is literal not interpreted. Thus above you have an actual $s being passed to AWK instead of the contents of $s. If you use double quotes, then you will have to escape the AWK built-in variables such as $0. Instead you might look at the -v, aka --assign, option:

Code:

cat list_1 | while read g;  
 do
  cat list_2 | while read s;
   do 
    touch ./output/$g
    awk --assign s="$s" '$0==s {print $0}' ./*/*/$g > ./output/$g ;
  done
done

Or maybe (untested)

Code:

cat list_1 | while read g;  
 do
  touch ./output/$g
  awk 'NR==FNR {a[$0]++; next} a[$0] {print $0}' list_2 ./*/*/$g > ./output/$g ;
done

The double-quote versus single quote difference also applies to sed too.

Drosera_capensis · 04-04-2020, 06:26 AM

Thank you very much for your answers. The following script is almost working Turbocapitalist, thanks.
The matches /regexp/ are printed along with the following line (getline).
But, there is still a single line of no-match printed (without the second line).
Would you have an idea why?

Code:

cat list_1 | while read g;  
 do
  cat list_2 | while read s;
   do 
    touch ./output/$g
    awk --assign s="$s" '$0==s {print $0; getline; print $0}'  ./*/*/$g > ./output/$g ;
  done
done

Thanks for the links pan64. I have used shellcheck and it looks a formidable tool. Except that some advises it gives are not compatible with the command.
For exemple, replacing:

Code:

cat list_1 | while read g

with:

Code:

while read g < list_1

Is preventing the command to work. Same thing for the advice to put all the variables between double quotes "$g". I guess it is a tool to use carefully.

Turbocapitalist · 04-04-2020, 06:52 AM

Quote:

Originally Posted by Drosera_capensis

Thank you very much for your answers. The following script is almost working Turbocapitalist, thanks.
The matches /regexp/ are printed along with the following line (getline).
But, there is still a single line of no-match printed (without the second line).
Would you have an idea why?

It is a little hard now to visualize your goal. Can you show a handful of lines from each of the two files, sanitized if necessary, along with the output you desire?

pan64 · 04-04-2020, 08:43 AM

Quote:

Originally Posted by Drosera_capensis

Code:

cat list_1 | while read g

with:

Code:

while read g < list_1

Is preventing the command to work.

yes, would be nice to explain better, that should work

Quote:

Originally Posted by Drosera_capensis

Same thing for the advice to put all the variables between double quotes "$g". I guess it is a tool to use carefully.

Again, what shellcheck suggests is usually really good. There is always a help for every suggestion, so you can easily understand the reason and also the possible exceptions.

teckk · 04-04-2020, 11:23 AM

Quote:

while read g < list_1

Code:

echo -e "One\nTwo\nThree\nFour" > MyTest.txt

while read g; do
    echo "$g"
done < MyTest.txt

One
Two
Three
Four

MadeInGermany · 04-05-2020, 05:29 AM

Code:

while read g < list_1

The redirection is for the read command.
Because it is within the while loop, it opens the file (and reads the 1st line) for each loop cycle.
In contrast, the following redirects the entire loop

Code:

while
  read g
do
  ...
done < list_1

The shell opens the file once and redirects stdin.
Now each command that reads from stdin actually reads from the file. For each loop cycle the read command reads a new line.
The file is closed when the loop terminates, at the end or by a break.

For the advanced users:
the early Bourne shell allowed to switch order

Code:

< list_1 read

And all shells are compatible.
But only the zsh does, as a logical consequence, allow the same for a code block:

Code:

< list_1 while
  read g
do
  ...
done

Not only while-do-done is a code block.
Also for-do-done and if-then-fi are code blocks.
And { } is an artificial code block. An example:

Code:

{
read line1
while
  read line
do
  ...
done
} < inputfile

Again only zsh allows

Code:

< inputfile {
...
}