LinuxQuestions.org
Help answer threads with 0 replies.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 11-09-2016, 06:59 AM   #1
break_da_funk
LQ Newbie
 
Registered: Apr 2016
Posts: 15

Rep: Reputation: Disabled
Shell arguments in variable vs direct string


Hello Community!
Let's say that we have some script which counts its arguments number:
arguments_count.sh:
Code:
#!/bin/sh
echo "Number of arguments="$#
and some test script:
test.sh
Code:
#!/bin/sh
my_args="1 2 3 '4 5' 6"
echo "Count of arguments when using my_args:"
./arguments_count.sh $my_args
echo "Count of arguments when using direct string:"
./arguments_count.sh 1 2 3 '4 5' 6
Output after execute test.sh is:
Quote:
Count of arguments when using my_args:
Number of arguments=6
Count of arguments when using direct string:
Number of arguments=5
What should I do to get output for "my_args" the same as for direct string?
 
Old 11-09-2016, 07:34 AM   #2
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,912

Rep: Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513
Well first...

If you run the script with debugging (-vx) options you would see why.

You are getting what you asked for:
Code:
$ sh -vx test.sh
#!/bin/sh
my_args="1 2 3 '4 5' 6"
+ my_args='1 2 3 '\''4 5'\'' 6'
echo "Count of arguments when using my_args:"
+ echo 'Count of arguments when using my_args:'
Count of arguments when using my_args:
./arguments_count.sh $my_args
+ ./arguments_count.sh 1 2 3 ''\''4' '5'\''' 6
Number of arguments=6
echo "Count of arguments when using direct string:"
+ echo 'Count of arguments when using direct string:'
Count of arguments when using direct string:
./arguments_count.sh 1 2 3 '4 5' 6
+ ./arguments_count.sh 1 2 3 '4 5' 6
Number of arguments=5
Note the line
Code:
+ ./arguments_count.sh 1 2 3 ''\''4' '5'\''' 6
The quotes cancel out... and you get 1 2 3 4 5 6.

if you change the script "arguments_count.sh" to:
Code:
#!/bin/sh
echo "Number of arguments="$#
echo "  args are= " $*
i=1
while [ "$1" != "" ]; do
    echo "P$i="$1
    shift
    i=$(($i+1))
done
you get to see what the result is.

The issue is understanding how the shell expands variables for use. And it does get tricky to pass arguments with "special" characters embedded (space/tab - as well as others like ()&; ...).

What you saw was that the '4 is passed as a single argument... and 5' was passed as the other.

But when passed directly, the '4 5' was interpreted as you expected.

It takes practice -- and quite frequently making a test case is the only way to see what actually happens.

This is also one of the reasons shell scripts cannot be considered for "secure" use. It is too easy to overlook a loophole (try embedding the string `ls` as a parameter and see what happens...)
 
Old 11-09-2016, 07:34 AM   #3
MensaWater
LQ Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, CoreOS, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 7,831
Blog Entries: 15

Rep: Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669
Escape or quote the quotes:

Escape:
./arguments_count.sh 1 2 3 \'4 5\' 6

Quote (note this is single quote followed by single quote NOT double quote):
./arguments_count.sh 1 2 3 ''4 5'' 6

Often when running things within levels (e.g. a script within a script) you have to escape or quote to be sure the literal things you are passing (in this case your single quotes around 4 5) make it through to the next level.
 
Old 11-09-2016, 07:38 AM   #4
break_da_funk
LQ Newbie
 
Registered: Apr 2016
Posts: 15

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by jpollard View Post
Well first...

If you run the script with debugging (-vx) options you would see why.

You are getting what you asked for:
Code:
$ sh -vx test.sh
#!/bin/sh
my_args="1 2 3 '4 5' 6"
+ my_args='1 2 3 '\''4 5'\'' 6'
echo "Count of arguments when using my_args:"
+ echo 'Count of arguments when using my_args:'
Count of arguments when using my_args:
./arguments_count.sh $my_args
+ ./arguments_count.sh 1 2 3 ''\''4' '5'\''' 6
Number of arguments=6
echo "Count of arguments when using direct string:"
+ echo 'Count of arguments when using direct string:'
Count of arguments when using direct string:
./arguments_count.sh 1 2 3 '4 5' 6
+ ./arguments_count.sh 1 2 3 '4 5' 6
Number of arguments=5
Note the line
Code:
+ ./arguments_count.sh 1 2 3 ''\''4' '5'\''' 6
The quotes cancel out... and you get 1 2 3 4 5 6.

if you change the script "arguments_count.sh" to:
Code:
#!/bin/sh
echo "Number of arguments="$#
echo "  args are= " $*
i=1
while [ "$1" != "" ]; do
    echo "P$i="$1
    shift
    i=$(($i+1))
done
you get to see what the result is.

The issue is understanding how the shell expands variables for use. And it does get tricky to pass arguments with "special" characters embedded (space/tab - as well as others like ()&; ...).

What you saw was that the '4 is passed as a single argument... and 5' was passed as the other.

But when passed directly, the '4 5' was interpreted as you expected.

It takes practice -- and quite frequently making a test case is the only way to see what actually happens.

This is also one of the reasons shell scripts cannot be considered for "secure" use. It is too easy to overlook a loophole (try embedding the string `ls` as a parameter and see what happens...)
Thank You for reply.
How can I get this to work well?
 
Old 11-09-2016, 08:22 AM   #5
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,912

Rep: Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513
This is not obvious... and may not be the best way...

If you replace the following line in the test.sh script
Code:
./arguments_count.sh $my_args
with

Code:
eval ./arguments_count.sh $my_args
You get the same answer as the direct string returns. This happens because you are forcing an additional evaluation of the strings... and that converts the '4 and 5' parameters into the string '4 5'.

This approach may not even work in all cases - as the space character between the 4 and 5 will have been evaluated, so if you expected to pass a tab character it won't be there as previous evaluations removed it.

This will not help if have multiple nested structures...

Easier to use perl or python..

Last edited by jpollard; 11-09-2016 at 08:24 AM. Reason: typo
 
Old 11-09-2016, 12:53 PM   #6
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: Rocky Linux
Posts: 4,779

Rep: Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212
Quote:
Originally Posted by jpollard View Post
This is not obvious... and may not be the best way...
Code:
eval ./arguments_count.sh $my_args
It is also horribly insecure if there is any possibility of untrusted content in $my_args, such as:
Code:
my_args='1 2 `do_something_evil` 4 5'
Using an array instead of a simple variable is one way to handle this situation.
Code:
my_args=(1 2 3 '4 5' 6)
./arguments_count.sh "${my_args[@]}"
That will expand into
Code:
./arguments_count.sh "1" "2" "3" "4 5" "6"
 
1 members found this post helpful.
Old 11-09-2016, 01:12 PM   #7
break_da_funk
LQ Newbie
 
Registered: Apr 2016
Posts: 15

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by rknichols View Post
Using an array instead of a simple variable is one way to handle this situation.
Code:
my_args=(1 2 3 '4 5' 6)
./arguments_count.sh "${my_args[@]}"
That will expand into
Code:
./arguments_count.sh "1" "2" "3" "4 5" "6"
Solution should be POSIX compatible :/
I parsed argument in single quatas into separate variable and then use, it is working good, unfortunately is not so flexible. Tomorrow I will post details about my solution (with example code). Maybe someone will point me improvements.
 
Old 11-09-2016, 01:43 PM   #8
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: Rocky Linux
Posts: 4,779

Rep: Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212
Quote:
Originally Posted by rknichols View Post
Code:
my_args=(1 2 3 '4 5' 6)
./arguments_count.sh "${my_args[@]}"
Quote:
Originally Posted by break_da_funk View Post
Solution should be POSIX compatible :/
Works fine under "bash --posix" and also when bash is invoked as "sh".
 
Old 11-09-2016, 04:20 PM   #9
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,912

Rep: Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513
Quote:
Originally Posted by rknichols View Post
Works fine under "bash --posix" and also when bash is invoked as "sh".
Yeah ... but the bash shell is not POSIX... even with the --posix option.

The POSIX bourne shell doesn't have arrays...

Last edited by jpollard; 11-09-2016 at 04:21 PM.
 
Old 11-10-2016, 12:12 AM   #10
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 10,007

Rep: Reputation: 3191Reputation: 3191Reputation: 3191Reputation: 3191Reputation: 3191Reputation: 3191Reputation: 3191Reputation: 3191Reputation: 3191Reputation: 3191Reputation: 3191
As it has not really been explained why this is necessary it is more difficult to know why we need to mold this particular answer in this way.

So my suggestion would be to quote the variable and thereby preserve the quotes and then write your own function to parse the data to your own specification.
 
Old 11-10-2016, 01:07 AM   #11
break_da_funk
LQ Newbie
 
Registered: Apr 2016
Posts: 15

Original Poster
Rep: Reputation: Disabled
Hello Community!
Below You can find my solution for the problem:
test_parse.sh:
Code:
#!/bin/sh
my_args="1 2 3 '4 5' 6"
echo "Count of arguments when using my_args:"
./arguments_count.sh $my_args
echo "Count of arguments when using direct string:"
./arguments_count.sh 1 2 3 '4 5' 6

#split my_args, depends on single quote char
my_args_before_quote=""
my_args_in_quote=""
my_args_after_quote=""
my_args_quote_pos=`expr index "$my_args" "'"`	#find single quote char pos
if [ $my_args_quote_pos -gt 0 ]; then		#if founded then split my_args
	my_args_before_quote=`echo ${my_args:0:$((my_args_quote_pos-1))}`	#store args before single quote char
	my_args_in_quote=`echo ${my_args:$my_args_quote_pos}`			#store data from first quote to the end of the string
	my_args_quote_pos=`expr index "$my_args_in_quote" "'"`			#find second single quote char pos
	if [ $my_args_quote_pos -gt 0 ]; then					#if founded then split rest of string after quote char
		my_args_in_quote_length=`expr length "$my_args_in_quote"`	#length of string which contains second quote and after it
		if [ $((my_args_in_quote_length-$my_args_quote_pos)) -gt 0 ]; then	#if there is something after quote char
			my_args_after_quote="`echo ${my_args_in_quote:$my_args_quote_pos:$my_args_in_quote_length}`"	#store substring from second quote char to the end of the string
		fi
		my_args_in_quote=`echo ${my_args_in_quote:0:$((my_args_quote_pos-1))}`	#store substring in single quotes
	fi
	echo "Count of arguments when using parse/split method:"
	./arguments_count.sh $my_args_before_quote "$my_args_in_quote" $my_args_after_quote	#check
fi
I will be very grateful if You analyse my code and propose improvement.
Problem with my solution is that is only allow parsing arguments string which consist only one (or none) expression with single quotas. Anyway my problem is solved but not for future.

Short explanation of the root cause:
I have some configuration file. One of the scripts analyses the file and searches for some daemon arguments. When the script finds the arguments it runs the demon with some fixed arguments combined with founded arguments. User can write in configuration file arguments which consist single quoted text (argument in single quotes is short script indeed). Some time ago there was a change of the read and write configuration method. Now I found that some old configuration doesn't work properly because of the problem with interpretation of single quotes in variable.
If You are interested in details it is about pppd daemon and argument "connect <connect_script>"(check man for pppd). You can create connect script and put path to it. You can also put some short script in single quotas instead of the path. For example:
connect_script:
Code:
#!/bin/sh
chat -v -f /somefile.chat
and You can run pppd as:
Code:
pppd arg1 arg2 arg3 connect /path/to/connect_script arg4 arg5 &
but You can also run pppd as:
Code:
pppd arg1 arg2 arg3 connect 'chat -v -f /somefile.chat' arg4 arg5&
Result of the two pppd execution is the same.
I cannot change old configuration file and replace quotas with path to script. That is why I have to modify method of read daemon arguments.
Shell which uses daemon arguments read script is POSIX compatible.

Last edited by break_da_funk; 11-10-2016 at 01:14 AM.
 
Old 11-10-2016, 02:04 AM   #12
break_da_funk
LQ Newbie
 
Registered: Apr 2016
Posts: 15

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by break_da_funk View Post
Hello Community!
...
Finally, with the help of my friend (some dodger from Radom city), we found solution:
test.sh:
Code:
#!/bin/sh
my_args="1 2 3 '4 5' 6"
echo "Count of arguments when using my_args:"
echo "./arguments_count.sh $my_args" > ./my_args.sh	#put whole command into a file
sh ./my_args.sh						#run the file
echo "Count of arguments when using direct string:"
echo "./arguments_count.sh 1 2 3 '4 5' 6" > ./my_string.sh	#put whole command into a file
sh ./my_string.sh						#run the file
Thank You all for posts and help.

Last edited by break_da_funk; 11-10-2016 at 02:05 AM.
 
Old 11-10-2016, 02:05 AM   #13
pan64
LQ Addict
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 21,842

Rep: Reputation: 7309Reputation: 7309Reputation: 7309Reputation: 7309Reputation: 7309Reputation: 7309Reputation: 7309Reputation: 7309Reputation: 7309Reputation: 7309Reputation: 7309
usually this:
Code:
my_args_in_quote=`echo ${my_args:$my_args_quote_pos}`
is deprecated, not suggested at all.
First, you need to use ", next `echo something` is usually equal to something without echo and ` (backtick). Also backtick makes some tricks on the command in between backticks, so you need to take extra care.
[[ ]] is preferred (instead of [ ] )

So your code should look like this:
Code:
my_args_before_quote="${my_args:0:$((my_args_quote_pos-1))}"	#store args before single quote char
my_args_in_quote="${my_args:$my_args_quote_pos}"			#store data from first quote to the end of the string
my_args_quote_pos=$(expr index "$my_args_in_quote" "'")			#find second single quote char pos
if [[ $my_args_quote_pos -gt 0 ]]; then					#if founded then split rest of string after quote char
	my_args_in_quote_length=${#my_args_in_quote}	                #length of string which contains second quote and after it  <<< do not need expr length here
	if [[ $my_args_in_quote_length -gt $my_args_quote_pos ]]; then	#if there is something after quote char
		my_args_after_quote="${my_args_in_quote:$my_args_quote_pos:$my_args_in_quote_length}"	#store substring from second quote char to the end of the string
	fi
	my_args_in_quote="${my_args_in_quote:0:$((my_args_quote_pos-1))}"	#store substring in single quotes
fi
I would say this will not work the same way as before, so you need to check it

looks like you want to modify the original behaviour (of bash), so just go back to the original question:
Code:
#!/bin/sh
my_args="1 2 3 '4 5' 6"
echo "Count of arguments when using my_args:"
./arguments_count.sh $my_args    # result was 6
echo "Count of arguments when using direct string:"
./arguments_count.sh 1 2 3 '4 5' 6  # result was 5
You need to understand how the entered string was evaluated. In the second case it is quite simple, you will have 5 arguments
1
2
3
4 5 << here ' ' will protect the space between 4 and 5 to use as separator
6

in the first case you will have
1
2
3
'4
5'
6
because " will force the interpreter to use ' as is, loosing its special meaning (as it was used in the first case).
Also you may try:
./arguments_count.sh "$my_args"
which will return 1, because the whole string will be used "in one", as a single argument.

Every time you start a command this argument parsing will occur and that's why embedding argument lists into each other is not really trivial (backtick will also do something like this) - although not impossible. You need to keep embedded commands+its arguments in one as long as it is just passed to another command.
The usual way is to protect " against evaluation by escaping it (using \ and/or ').

http://wiki.bash-hackers.org/syntax/quoting
http://stackoverflow.com/questions/9...ring-quotation
http://www.tldp.org/LDP/abs/html/quotingvar.html
http://stackoverflow.com/questions/1...quoted-strings
(and obviously you can find a lot of other examples)

I hope this helps a bit
 
1 members found this post helpful.
Old 11-10-2016, 02:42 AM   #14
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 10,007

Rep: Reputation: 3191Reputation: 3191Reputation: 3191Reputation: 3191Reputation: 3191Reputation: 3191Reputation: 3191Reputation: 3191Reputation: 3191Reputation: 3191Reputation: 3191
Probably just me not following, but are you saying that the current config files will have the line:
Code:
chat -v -f /somefile.chat
And that this along with some default set of args need to be passed to pppd? If so, won't all the data already be split into default args plus newly found information from config which can then
be quoted?

I would have thought that if trying to future proof you would also need to consider the scenario where there are more than 1 set of quoted information?
 
Old 11-10-2016, 02:48 AM   #15
break_da_funk
LQ Newbie
 
Registered: Apr 2016
Posts: 15

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by grail View Post
Probably just me not following, but are you saying that the current config files will have the line:
Code:
chat -v -f /somefile.chat
And that this along with some default set of args need to be passed to pppd? If so, won't all the data already be split into default args plus newly found information from config which can then
be quoted?

I would have thought that if trying to future proof you would also need to consider the scenario where there are more than 1 set of quoted information?
3xYes.

There can be max 3 sets with quotes (pppd 2.4.7):
connect 'something'
disconnect 'else'
init 'another'

But solution with "storing whole command into another script" does the trick.

Last edited by break_da_funk; 11-10-2016 at 03:00 AM.
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
How to pass string variable as command parameter in shell script? prasad1990 Linux - Newbie 6 03-23-2015 12:48 AM
[SOLVED] Bash Shell Script - Store a variable as a string not an integer RML1992 Linux - General 8 09-12-2012 09:19 AM
how to read shell variable in /etc/auto.direct map for autofs service UltraSoul Linux - General 8 06-05-2009 07:21 AM
Need shell script to concatenate a string and a variable into a directory name AwesomeMachine Linux - Newbie 2 05-07-2006 03:42 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 11:30 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration