LinuxQuestions.org
Latest LQ Deal: Complete CCNA, CCNP & Red Hat Certification Training Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 09-19-2017, 03:23 PM   #1
MensaWater
LQ Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, CoreOS, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 7,070
Blog Entries: 14

Rep: Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250
while loop, bash arrays and multiline awk ambiguous redirect


So the situation was I was trying to populate a couple of arrays within a while loop. The arrays populated but could not be used outside the while loop.

Based on reading that was because I was starting with the following:

Code:
awk -F" " '/ourcompany.re/ {if ($9 ~ /^ourcompany.re/) print $9
 if ($10 ~ /^ourcompany.re/) print $10
 if ($11 ~ /^ourcompany.re/ ) print $11 }' $SFTPLIST  | while read REMFILE
do ...
done
In the above SFTPLIST is a file that list files we see on a remote sftp site. Creation of that file is not an issue.

The multiline awk syntax used is one I've done many times when I want to find information that might be found on various lines or various fields. It in fact works.

The issue based on reading I've done is that piping into while makes it run the while in a subshell. Within the while loop it does populate 2 separate arrays successfully.

However, apparently due to the subshell these variables are not accessible after the "done" by the rest of the program that isn't inside the while loop.

The fix is purportedly to instead do a redirect into the end of the loop rather than a pipe into the start. Ostensibly something like:

Code:
while read REMFILE
do ...
done <  awk -F" " '/ourcompany.re/ {if ($9 ~ /^ourcompany.re/) print $9
 if ($10 ~ /^ourcompany.re/) print $10
 if ($11 ~ /^ourcompany.re/ ) print $11 }' $SFTPLIST
My problem is that no matter how I encapsulate the multiline awk (parentheses, $(), quotes etc...) it simply fails. The best I got was a message about ambiguous redirect.

There were various web hits about that message but they were all related to > (redirect out) rather than the < (redirect in) I was trying to do.

Finally I surrendered and used the loop to write into temporary files instead of the arrays. Later in the script I was able to read in from those files to do what I'd wanted to do from the arrays so my script is working.

My question:
How would I redirect in from the multiline awk syntax shown?

Last edited by MensaWater; 09-20-2017 at 09:04 AM.
 
Old 09-19-2017, 03:38 PM   #2
smallpond
Senior Member
 
Registered: Feb 2011
Location: Massachusetts, USA
Distribution: CentOS 6 (pre-systemd)
Posts: 2,850

Rep: Reputation: 757Reputation: 757Reputation: 757Reputation: 757Reputation: 757Reputation: 757Reputation: 757
Unless it is a large number of lines, you can save the awk output in a bash variable rather than a file, then do a for loop on the variable.
 
Old 09-19-2017, 04:24 PM   #3
MensaWater
LQ Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, CoreOS, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 7,070
Blog Entries: 14

Original Poster
Rep: Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250
Actually I had tried that but had same issue.



Code:
VAR=$(awk -F" " '/ourcompany.re/ {if ($9 ~ /^ourcompany.re/) print $9
 if ($10 ~ /^ourcompany.re/) print $10
 if ($11 ~ /^ourcompany.re/ ) print $11 }' $SFTPLIST)
It appears to me my issue has to do with the awk not being understood to be multiline which is why it says ambiguous redirect. I'd tried doing line continuations "" at end of the first 2 lines along with other escaping/quoting/parentheses all to no avail.

So the question would be similar:
How do I redirect multiline awk into a single variable?
 
Old 09-19-2017, 04:52 PM   #4
smallpond
Senior Member
 
Registered: Feb 2011
Location: Massachusetts, USA
Distribution: CentOS 6 (pre-systemd)
Posts: 2,850

Rep: Reputation: 757Reputation: 757Reputation: 757Reputation: 757Reputation: 757Reputation: 757Reputation: 757
This should not create a subshell, since it doesn't redirect or pipe:

Code:
for $REMFILE in $VAR
do
  -- your code --
done
 
Old 09-19-2017, 05:25 PM   #5
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 3,618

Rep: Reputation: 1577Reputation: 1577Reputation: 1577Reputation: 1577Reputation: 1577Reputation: 1577Reputation: 1577Reputation: 1577Reputation: 1577Reputation: 1577Reputation: 1577
Quote:
Originally Posted by MensaWater View Post
Code:
while read REMFILE
do ...
done <  awk -F" " '/ourcompany.re/ {if ($9 ~ /^ourcompany.re/) print $9
 if ($10 ~ /^ourcompany.re/) print $10
 if ($11 ~ /^ourcompany.re/ ) print $11 }' $SFTPLIST
Close, but the syntax for process substitution is "<( list )", and that needs to be separate from the "<" that indicates stdin redirection:
Code:
while read REMFILE
do ...
done <  <(awk -F" " '/ourcompany.re/ {if ($9 ~ /^ourcompany.re/) print $9
 if ($10 ~ /^ourcompany.re/) print $10
 if ($11 ~ /^ourcompany.re/ ) print $11 }' $SFTPLIST)
Note that the whitespace between the two "<" characters is essential, since "<<" means something else.
 
1 members found this post helpful.
Old 09-20-2017, 08:58 AM   #6
MensaWater
LQ Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, CoreOS, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 7,070
Blog Entries: 14

Original Poster
Rep: Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250
Quote:
Originally Posted by smallpond View Post
This should not create a subshell, since it doesn't redirect or pipe:

Code:
for $REMFILE in $VAR
do
  -- your code --
done
You missed the point of my response which was to say I couldn't assign the multiline awk as $VAR.
 
Old 09-20-2017, 09:03 AM   #7
MensaWater
LQ Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, CoreOS, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 7,070
Blog Entries: 14

Original Poster
Rep: Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250
Quote:
Originally Posted by rknichols View Post
Close, but the syntax for process substitution is "<( list )", and that needs to be separate from the "<" that indicates stdin redirection:
Code:
while read REMFILE
do ...
done <  <(awk -F" " '/ourcompany.re/ {if ($9 ~ /^ourcompany.re/) print $9
 if ($10 ~ /^ourcompany.re/) print $10
 if ($11 ~ /^ourcompany.re/ ) print $11 }' $SFTPLIST)
Note that the whitespace between the two "<" characters is essential, since "<<" means something else.
Thanks.

Last night I happened to mention my issue at a local RedHat meeting and a developer there suggested exactly the same thing. I'll give it a try.
 
Old 09-20-2017, 10:18 AM   #8
MensaWater
LQ Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, CoreOS, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 7,070
Blog Entries: 14

Original Poster
Rep: Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250
Thanks, rknichols. What you suggested solved the issue perfectly.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Bash: ambiguous redirect justmy2cents Programming 7 07-27-2017 01:33 PM
Bash: Ambiguous Redirect in script Firy Programming 4 05-30-2013 09:33 AM
bash returns ambiguous redirect with date in filename rstoutmna Programming 5 09-13-2011 08:37 PM
[SOLVED] bash ambiguous redirect wakatana Programming 2 03-09-2011 10:35 AM
append to end of each file -- -bash: *: ambiguous redirect allele Linux - Software 7 06-03-2009 02:57 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 04:23 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration