LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices


Reply
  Search this Thread
Old 02-21-2012, 11:06 AM   #1
towheedm
Member
 
Registered: Sep 2011
Location: Trinidad & Tobago
Distribution: Debian Stretch
Posts: 612

Rep: Reputation: 125Reputation: 125
A question on Shell Quoting


I've been trying to figure out why this is happening with quotes and now I seem to be lost.

I'm setting a variable (in BASH) to hold a directory tree listing:
Code:
x=`ls -R ~/dir1 | grep ":$" | sed 's/://'`
Listing var x gives:
Code:
echo $x
/home/towheed/dir1 /home/towheed/dir1/dir2 /home/towheed/dir1/dir2/dir3
But listing var x with it quoted, gives:
Code:
/home/towheed/dir1
/home/towheed/dir1/dir2
/home/towheed/dir1/dir2/dir3
Why does quoting the variable generate the newline?

Thanks.
 
Old 02-21-2012, 01:02 PM   #2
uhelp
Member
 
Registered: Nov 2011
Location: Germany, Bavaria, Nueremberg area
Distribution: openSUSE, Debian, LFS
Posts: 205

Rep: Reputation: 43
Your "echo" command gets a string from the var.
But it's default behavior is to suppress the interpretation of sequences like "\n".
So it does, what it is designed to: it concatenates all given strings (disregarding the contained newlines), inserts a space between the strings and outputs it.
This takes place with your "echo $x".

If you quote "$var" echo gets one big string containing newline characters and as it has nothing to concatenate it outputs the string, as it is and the contained "\n" are interpreted in a normal fashion.

Some remarks.
Never parse the output of "ls" "ps" and alike.
They depend on language settings and are not portable.
In fact, if using the output of "ls" you are NOT dealing with filenames, instead you are dealing with representations of filenames.
Use find instead.
"find . -type d " gives a newline separated list of all found directories.

And to avoid the quoting hell ALWAYS quote each and every var.
Use "$var" and never, never $var.

If the use of unquoted vars seems to fit, be sure: your script logic is wrong. Definitely!

Another point is, that there are loads of "echo"s out.
It could be a binary, a bash builtin or even an alias.
If you are going to examine this deeper take care to know, which kind of "echo" you are using.
"env echo" calls it from the path.
"echo" only god i.e. the sysadmin knows.
erm. Should know.

Last edited by uhelp; 02-21-2012 at 01:15 PM. Reason: added problem builtin vs. bin
 
Old 02-21-2012, 01:24 PM   #3
jhwilliams
Senior Member
 
Registered: Apr 2007
Location: Portland, OR
Distribution: Debian, Android, LFS
Posts: 1,168

Rep: Reputation: 211Reputation: 211Reputation: 211
You are observing word splitting resulting from the parameter expansion of $x, itself.

From the bash man page:

Quote:
Parameter Expansion
The `$' character introduces parameter expansion,
...
Quote:
Word Splitting
The shell scans the results of parameter expansion, command substitution, and arithmetic expansion
that did not occur within double quotes for word splitting.
...
So, word splitting usually happens to the output of a parameter expansion, except when it is in double quotes. Since newline is in $IFS, it gets stripped.

Last edited by jhwilliams; 02-21-2012 at 01:28 PM.
 
Old 02-21-2012, 01:27 PM   #4
jhwilliams
Senior Member
 
Registered: Apr 2007
Location: Portland, OR
Distribution: Debian, Android, LFS
Posts: 1,168

Rep: Reputation: 211Reputation: 211Reputation: 211
Quote:
Originally Posted by uhelp View Post
Your "echo" command gets a string from the var.
But it's default behavior is to suppress the interpretation of sequences like "\n".
So it does, what it is designed to: it concatenates all given strings (disregarding the contained newlines), inserts a space between the strings and outputs it.
This isn't quite right.

Echo respects newlines in input strings:
Code:
echo 'a
b'
a
b
The suppression of special chars (e.g. \n) has to do with when it is received literally to echo:
Code:
echo 'a\nb'
a\nb
vs.
Code:
echo -e 'a\nb'
a
b
 
Old 02-21-2012, 01:51 PM   #5
uhelp
Member
 
Registered: Nov 2011
Location: Germany, Bavaria, Nueremberg area
Distribution: openSUSE, Debian, LFS
Posts: 205

Rep: Reputation: 43
Quote:
But it's default behavior is to suppress the interpretation of sequences like "\n".
Maybe I did not write clear enough.

What you mentioned is right.
But this does not resolve the question asked.

Try this with the defined var x :
Code:
echo -e $x | od --format=ax2
echo  $x | od --format=ax2
echo -e "$x" | od --format=ax2
echo "$x" | od --format=ax2
 
Old 02-21-2012, 02:02 PM   #6
jhwilliams
Senior Member
 
Registered: Apr 2007
Location: Portland, OR
Distribution: Debian, Android, LFS
Posts: 1,168

Rep: Reputation: 211Reputation: 211Reputation: 211
@uhelp You're right, I guess I did not actually answer the question. Also, od is a useful tool -- I had almost forgot about it. Glad you reintroduced me. :-)

Quote:
Originally Posted by towheedm View Post
Why does quoting the variable generate the newline?
So the answer here is that it doesn't generate the newline. Rather, by not quoting the variable, the shell performs word splitting on the output of the parameter expansion.
 
Old 02-21-2012, 02:16 PM   #7
towheedm
Member
 
Registered: Sep 2011
Location: Trinidad & Tobago
Distribution: Debian Stretch
Posts: 612

Original Poster
Rep: Reputation: 125Reputation: 125
Thanks for all the responses. So a few weeks ago, I realized I did not know regex's as much as I thought I did. Now this. :-)

Quote:
If the use of unquoted vars seems to fit, be sure: your script logic is wrong. Definitely!
I'm using it in a shell script:

Code:
mydirs="`ls -R ~/dir1 | grep ":$" | sed 's/://'`"
for x in $mydirs ; do
  <commandlist>
done
Of course, in the above, quoting $mydirs would cause the commandlist to fail.

I'll change to "find" once I'm clear on this.

Thanks again
 
Old 02-21-2012, 02:22 PM   #8
jhwilliams
Senior Member
 
Registered: Apr 2007
Location: Portland, OR
Distribution: Debian, Android, LFS
Posts: 1,168

Rep: Reputation: 211Reputation: 211Reputation: 211
Quote:
Originally Posted by towheedm View Post
I'll change to "find" once I'm clear on this.
I was going to say -- this is a bit clearer:
Code:
find ~/dir1 -type d
 
Old 02-21-2012, 02:37 PM   #9
towheedm
Member
 
Registered: Sep 2011
Location: Trinidad & Tobago
Distribution: Debian Stretch
Posts: 612

Original Poster
Rep: Reputation: 125Reputation: 125
Sorry for the confusion, I meant once I was a bit clearer on the whole quoting stuff.

I was just asking if not quoting in the for..done loop was OK, since quoting the var definitely would not work.
 
Old 02-21-2012, 03:06 PM   #10
uhelp
Member
 
Registered: Nov 2011
Location: Germany, Bavaria, Nueremberg area
Distribution: openSUSE, Debian, LFS
Posts: 205

Rep: Reputation: 43
Code:
# this gives you an array named "mydirs"
# addressing one item can be  ¼{mydirs[$i] where i is an integer.
# index starts at zero
mydirs=(  $( find <yourDir> -type d )   )

# if you want each entry concatenated with a space in between use ${mydirs[*]}
for x in "${mydirs[*]}" ; do
  <commandlist>  "$x"  #this is safe even if directory name contain spaces or 
                       # other unusual characters. It is ensured 
                       #that only a directory name will be given
done
 
Old 02-21-2012, 06:55 PM   #11
towheedm
Member
 
Registered: Sep 2011
Location: Trinidad & Tobago
Distribution: Debian Stretch
Posts: 612

Original Poster
Rep: Reputation: 125Reputation: 125
Thank you. That's a more fool-proof method. I've gotten so accustomed to not using spaces in dir/file names anymore, I did not consider it.
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Processing a command line literally (without shell quoting) paulux Programming 5 09-01-2010 02:46 AM
shell quoting within a function dazdaz Linux - Software 4 09-03-2007 11:01 AM
bash - quoting RGummi Linux - General 3 10-21-2006 03:06 PM
Quick question: quoting text in sed overbored Linux - Software 0 06-24-2004 01:23 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - General

All times are GMT -5. The time now is 12:19 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration