LinuxQuestions.org - [SOLVED] Shell arguments in variable vs direct string

Page 1 of 2

Show 50 post(s) from this thread on one page

- Linux - Newbie (https://www.linuxquestions.org/questions/linux-newbie-8/)

- - Shell arguments in variable vs direct string (https://www.linuxquestions.org/questions/linux-newbie-8/shell-arguments-in-variable-vs-direct-string-4175593215/)

break_da_funk

11-09-2016 06:59 AM

Shell arguments in variable vs direct string

Hello Community!
Let's say that we have some script which counts its arguments number:
arguments_count.sh:

Code:

#!/bin/sh

echo "Number of arguments="$#

and some test script:
test.sh

Code:

#!/bin/sh

my_args="1 2 3 '4 5' 6"

echo "Count of arguments when using my_args:"

./arguments_count.sh $my_args

echo "Count of arguments when using direct string:"

./arguments_count.sh 1 2 3 '4 5' 6

Output after execute test.sh is:

Quote:

Count of arguments when using my_args:
Number of arguments=6
Count of arguments when using direct string:
Number of arguments=5

What should I do to get output for "my_args" the same as for direct string?

jpollard

11-09-2016 07:34 AM

Well first...

If you run the script with debugging (-vx) options you would see why.

You are getting what you asked for:

Code:

$ sh -vx test.sh

#!/bin/sh

my_args="1 2 3 '4 5' 6"

+ my_args='1 2 3 '\''4 5'\'' 6'

echo "Count of arguments when using my_args:"

+ echo 'Count of arguments when using my_args:'

Count of arguments when using my_args:

./arguments_count.sh $my_args

+ ./arguments_count.sh 1 2 3 ''\''4' '5'\''' 6

Number of arguments=6

echo "Count of arguments when using direct string:"

+ echo 'Count of arguments when using direct string:'

Count of arguments when using direct string:

./arguments_count.sh 1 2 3 '4 5' 6

+ ./arguments_count.sh 1 2 3 '4 5' 6

Number of arguments=5

Note the line

Code:

+ ./arguments_count.sh 1 2 3 ''\''4' '5'\''' 6

The quotes cancel out... and you get 1 2 3 4 5 6.

if you change the script "arguments_count.sh" to:

Code:

#!/bin/sh

echo "Number of arguments="$#

echo "  args are= " $*

i=1

while [ "$1" != "" ]; do

    echo "P$i="$1

    shift

    i=$(($i+1))

done

you get to see what the result is.

The issue is understanding how the shell expands variables for use. And it does get tricky to pass arguments with "special" characters embedded (space/tab - as well as others like ()&; ...).

What you saw was that the '4 is passed as a single argument... and 5' was passed as the other.

But when passed directly, the '4 5' was interpreted as you expected.

It takes practice -- and quite frequently making a test case is the only way to see what actually happens.

This is also one of the reasons shell scripts cannot be considered for "secure" use. It is too easy to overlook a loophole (try embedding the string `ls` as a parameter and see what happens...)

MensaWater

11-09-2016 07:34 AM

Escape or quote the quotes:

Escape:
./arguments_count.sh 1 2 3 \'4 5\' 6

Quote (note this is single quote followed by single quote NOT double quote):
./arguments_count.sh 1 2 3 ''4 5'' 6

Often when running things within levels (e.g. a script within a script) you have to escape or quote to be sure the literal things you are passing (in this case your single quotes around 4 5) make it through to the next level.

break_da_funk

11-09-2016 07:38 AM

Quote:

Originally Posted by jpollard (Post 5628718)

Well first...

If you run the script with debugging (-vx) options you would see why.

You are getting what you asked for:

Code:

$ sh -vx test.sh

#!/bin/sh

my_args="1 2 3 '4 5' 6"

+ my_args='1 2 3 '\''4 5'\'' 6'

echo "Count of arguments when using my_args:"

+ echo 'Count of arguments when using my_args:'

Count of arguments when using my_args:

./arguments_count.sh $my_args

+ ./arguments_count.sh 1 2 3 ''\''4' '5'\''' 6

Number of arguments=6

echo "Count of arguments when using direct string:"

+ echo 'Count of arguments when using direct string:'

Count of arguments when using direct string:

./arguments_count.sh 1 2 3 '4 5' 6

+ ./arguments_count.sh 1 2 3 '4 5' 6

Number of arguments=5

Note the line

Code:

+ ./arguments_count.sh 1 2 3 ''\''4' '5'\''' 6

The quotes cancel out... and you get 1 2 3 4 5 6.

if you change the script "arguments_count.sh" to:

Code:

#!/bin/sh

echo "Number of arguments="$#

echo "  args are= " $*

i=1

while [ "$1" != "" ]; do

    echo "P$i="$1

    shift

    i=$(($i+1))

done

Thank You for reply.
How can I get this to work well?

jpollard

11-09-2016 08:22 AM

This is not obvious... and may not be the best way...

If you replace the following line in the test.sh script

Code:

./arguments_count.sh $my_args

with

Code:

eval ./arguments_count.sh $my_args

You get the same answer as the direct string returns. This happens because you are forcing an additional evaluation of the strings... and that converts the '4 and 5' parameters into the string '4 5'.

This approach may not even work in all cases - as the space character between the 4 and 5 will have been evaluated, so if you expected to pass a tab character it won't be there as previous evaluations removed it.

This will not help if have multiple nested structures...

Easier to use perl or python..

rknichols

11-09-2016 12:53 PM

Quote:

Originally Posted by jpollard (Post 5628729)

This is not obvious... and may not be the best way...

Code:

eval ./arguments_count.sh $my_args

It is also horribly insecure if there is any possibility of untrusted content in $my_args, such as:

Code:

my_args='1 2 `do_something_evil` 4 5'

Using an array instead of a simple variable is one way to handle this situation.

Code:

my_args=(1 2 3 '4 5' 6)

./arguments_count.sh "${my_args[@]}"

That will expand into

Code:

./arguments_count.sh "1" "2" "3" "4 5" "6"

break_da_funk

11-09-2016 01:12 PM

Quote:

Originally Posted by rknichols (Post 5628774)

Using an array instead of a simple variable is one way to handle this situation.

Code:

my_args=(1 2 3 '4 5' 6)

./arguments_count.sh "${my_args[@]}"

That will expand into

Code:

./arguments_count.sh "1" "2" "3" "4 5" "6"

Solution should be POSIX compatible :/
I parsed argument in single quatas into separate variable and then use, it is working good, unfortunately is not so flexible. Tomorrow I will post details about my solution (with example code). Maybe someone will point me improvements.

rknichols

11-09-2016 01:43 PM

Quote:

Originally Posted by rknichols (Post 5628774)

Code:

my_args=(1 2 3 '4 5' 6)

./arguments_count.sh "${my_args[@]}"

Quote:

Originally Posted by break_da_funk (Post 5628782)

Solution should be POSIX compatible :/

Works fine under "bash --posix" and also when bash is invoked as "sh".

jpollard

11-09-2016 04:20 PM

Quote:

Originally Posted by rknichols (Post 5628788)

Works fine under "bash --posix" and also when bash is invoked as "sh".

Yeah ... but the bash shell is not POSIX... even with the --posix option.

The POSIX bourne shell doesn't have arrays...

grail

11-10-2016 12:12 AM

As it has not really been explained why this is necessary it is more difficult to know why we need to mold this particular answer in this way.

So my suggestion would be to quote the variable and thereby preserve the quotes and then write your own function to parse the data to your own specification.

break_da_funk

11-10-2016 01:07 AM

Hello Community!
Below You can find my solution for the problem:
test_parse.sh:

Code:

#!/bin/sh

my_args="1 2 3 '4 5' 6"

echo "Count of arguments when using my_args:"

./arguments_count.sh $my_args

echo "Count of arguments when using direct string:"

./arguments_count.sh 1 2 3 '4 5' 6



#split my_args, depends on single quote char

my_args_before_quote=""

my_args_in_quote=""

my_args_after_quote=""

my_args_quote_pos=`expr index "$my_args" "'"`        #find single quote char pos

if [ $my_args_quote_pos -gt 0 ]; then                #if founded then split my_args

        my_args_before_quote=`echo ${my_args:0:$((my_args_quote_pos-1))}`        #store args before single quote char

        my_args_in_quote=`echo ${my_args:$my_args_quote_pos}`                        #store data from first quote to the end of the string

        my_args_quote_pos=`expr index "$my_args_in_quote" "'"`                        #find second single quote char pos

        if [ $my_args_quote_pos -gt 0 ]; then                                        #if founded then split rest of string after quote char

                my_args_in_quote_length=`expr length "$my_args_in_quote"`        #length of string which contains second quote and after it

                if [ $((my_args_in_quote_length-$my_args_quote_pos)) -gt 0 ]; then        #if there is something after quote char

                        my_args_after_quote="`echo ${my_args_in_quote:$my_args_quote_pos:$my_args_in_quote_length}`"        #store substring from second quote char to the end of the string

                fi

                my_args_in_quote=`echo ${my_args_in_quote:0:$((my_args_quote_pos-1))}`        #store substring in single quotes

        fi

        echo "Count of arguments when using parse/split method:"

        ./arguments_count.sh $my_args_before_quote "$my_args_in_quote" $my_args_after_quote        #check

fi

I will be very grateful if You analyse my code and propose improvement.
Problem with my solution is that is only allow parsing arguments string which consist only one (or none) expression with single quotas. Anyway my problem is solved but not for future.

Short explanation of the root cause:
I have some configuration file. One of the scripts analyses the file and searches for some daemon arguments. When the script finds the arguments it runs the demon with some fixed arguments combined with founded arguments. User can write in configuration file arguments which consist single quoted text (argument in single quotes is short script indeed). Some time ago there was a change of the read and write configuration method. Now I found that some old configuration doesn't work properly because of the problem with interpretation of single quotes in variable.
If You are interested in details it is about pppd daemon and argument "connect <connect_script>"(check man for pppd). You can create connect script and put path to it. You can also put some short script in single quotas instead of the path. For example:
connect_script:

Code:

#!/bin/sh

chat -v -f /somefile.chat

and You can run pppd as:

Code:

pppd arg1 arg2 arg3 connect /path/to/connect_script arg4 arg5 &

but You can also run pppd as:

Code:

pppd arg1 arg2 arg3 connect 'chat -v -f /somefile.chat' arg4 arg5&

Result of the two pppd execution is the same.
I cannot change old configuration file and replace quotas with path to script. That is why I have to modify method of read daemon arguments.
Shell which uses daemon arguments read script is POSIX compatible.

break_da_funk

11-10-2016 02:04 AM

Quote:

Originally Posted by break_da_funk (Post 5628932)

Hello Community!
...

Finally, with the help of my friend (some dodger from Radom city), we found solution:
test.sh:

Code:

#!/bin/sh

my_args="1 2 3 '4 5' 6"

echo "Count of arguments when using my_args:"

echo "./arguments_count.sh $my_args" > ./my_args.sh        #put whole command into a file

sh ./my_args.sh                                                #run the file

echo "Count of arguments when using direct string:"

echo "./arguments_count.sh 1 2 3 '4 5' 6" > ./my_string.sh        #put whole command into a file

sh ./my_string.sh                                                #run the file

Thank You all for posts and help.

pan64

11-10-2016 02:05 AM

usually this:

Code:

my_args_in_quote=`echo ${my_args:$my_args_quote_pos}`

is deprecated, not suggested at all.
First, you need to use ", next `echo something` is usually equal to something without echo and ` (backtick). Also backtick makes some tricks on the command in between backticks, so you need to take extra care.
[[ ]] is preferred (instead of [ ] )

So your code should look like this:

Code:

my_args_before_quote="${my_args:0:$((my_args_quote_pos-1))}"        #store args before single quote char

my_args_in_quote="${my_args:$my_args_quote_pos}"                        #store data from first quote to the end of the string

my_args_quote_pos=$(expr index "$my_args_in_quote" "'")                        #find second single quote char pos

if [[ $my_args_quote_pos -gt 0 ]]; then                                        #if founded then split rest of string after quote char

        my_args_in_quote_length=${#my_args_in_quote}                        #length of string which contains second quote and after it  <<< do not need expr length here

        if [[ $my_args_in_quote_length -gt $my_args_quote_pos ]]; then        #if there is something after quote char

                my_args_after_quote="${my_args_in_quote:$my_args_quote_pos:$my_args_in_quote_length}"        #store substring from second quote char to the end of the string

        fi

        my_args_in_quote="${my_args_in_quote:0:$((my_args_quote_pos-1))}"        #store substring in single quotes

fi

I would say this will not work the same way as before, so you need to check it

looks like you want to modify the original behaviour (of bash), so just go back to the original question:

Code:

#!/bin/sh

my_args="1 2 3 '4 5' 6"

echo "Count of arguments when using my_args:"

./arguments_count.sh $my_args    # result was 6

echo "Count of arguments when using direct string:"

./arguments_count.sh 1 2 3 '4 5' 6  # result was 5

You need to understand how the entered string was evaluated. In the second case it is quite simple, you will have 5 arguments
1
2
3
4 5 << here ' ' will protect the space between 4 and 5 to use as separator
6

in the first case you will have
1
2
3
'4
5'
6
because " will force the interpreter to use ' as is, loosing its special meaning (as it was used in the first case).
Also you may try:
./arguments_count.sh "$my_args"
which will return 1, because the whole string will be used "in one", as a single argument.

Every time you start a command this argument parsing will occur and that's why embedding argument lists into each other is not really trivial (backtick will also do something like this) - although not impossible. You need to keep embedded commands+its arguments in one as long as it is just passed to another command.
The usual way is to protect " against evaluation by escaping it (using \ and/or ').

http://wiki.bash-hackers.org/syntax/quoting
http://stackoverflow.com/questions/9...ring-quotation
http://www.tldp.org/LDP/abs/html/quotingvar.html
http://stackoverflow.com/questions/1...quoted-strings
(and obviously you can find a lot of other examples)

I hope this helps a bit

grail

11-10-2016 02:42 AM

Probably just me not following, but are you saying that the current config files will have the line:

Code:

chat -v -f /somefile.chat

And that this along with some default set of args need to be passed to pppd? If so, won't all the data already be split into default args plus newly found information from config which can then
be quoted?

I would have thought that if trying to future proof you would also need to consider the scenario where there are more than 1 set of quoted information?

break_da_funk

11-10-2016 02:48 AM

Quote:

Originally Posted by grail (Post 5628950)

Probably just me not following, but are you saying that the current config files will have the line:

Code:

chat -v -f /somefile.chat

3xYes.

There can be max 3 sets with quotes (pppd 2.4.7):
connect 'something'
disconnect 'else'
init 'another'

But solution with "storing whole command into another script" does the trick.

All times are GMT -5. The time now is 07:26 AM.

Page 1 of 2

Show 50 post(s) from this thread on one page