LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 09-02-2012, 04:32 PM   #1
Nagy Szilveszter
LQ Newbie
 
Registered: Sep 2012
Posts: 11

Rep: Reputation: Disabled
Question Excaping quotes problem


Hi all,

I have a problem that I can't solve. I'll post it here, maybe someone can help me.

So here is it:
I have a test folder with some subfolders:

Code:
# find .
.
./etc
./etc/a
./a
./a/b
./e
./c
./c/d
I want to list all folders and files except the etc folder and its contents
Using the following command, and i get exactly what i want:

Code:
# find . ! -wholename "./etc*"
.
./a
./a/b
./e
./c
./c/d
Till this point it's fine. But inside a bash script I NEED THIS AS STRING because i'm building up the condition based on some internal values. Now look at this:

Code:
# cond='! -wholename "./etc*"'
And when i run find again...

Code:
# find . $cond
.
./etc
./etc/a
./a
./a/b
./e
./c
./c/d
...it lists the etc folder and its contents which it shouldn't.

I'm sure it's a quotation problem, and I tried all variations I know, but wasn't able to solve the problem.
Where is the mistake?

I appreciate your help
Szilvi
 
Old 09-02-2012, 06:42 PM   #2
Tinkster
Moderator
 
Registered: Apr 2002
Location: in a fallen world
Distribution: slackware by choice, others too :} ... android.
Posts: 23,067
Blog Entries: 11

Rep: Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910
Hi, welcome to LQ!

Try
Code:
eval find . $cond
 
Old 09-02-2012, 07:25 PM   #3
kakaka
Member
 
Registered: Sep 2003
Posts: 382

Rep: Reputation: 87
Hi Nagy Szilveszter!

You can sometimes see what the bash parser is doing with a given line from a script, by placing:

Code:
set -x
in your script.

Doing that with the command line you are using, it turns out that the issue is the difference between the way bash expands ./etc* and the way find -wholename expands it.

With your directory structure, bash only matches ./etc

So when you use virtually any form of quoting or evaluation where bash does it's matching, bash will replace it with the matches. You need to force a situation where the asterisk stays in place until it is fully passed to find -wholename. Running a script with these lines:

Code:
set -x
find . ! -wholename "./etc*"
cond='. ! -wholename ./etc\*'
echo "${cond}" | xargs find
eval find $cond
will show you how the parser is interpreting each line. Either the xargs approach, or the eval approach produce the same output as the command you started with, as long as the asterisk is escaped.

HTH.

Last edited by kakaka; 09-03-2012 at 08:13 AM.
 
1 members found this post helpful.
Old 09-03-2012, 01:21 AM   #4
Nagy Szilveszter
LQ Newbie
 
Registered: Sep 2012
Posts: 11

Original Poster
Rep: Reputation: Disabled
Thanks a lot both of you! It works with eval.... the xargs is a also great but i need to evolve to understand it.

The "set -x" is a really useful thing!
 
Old 09-03-2012, 02:50 AM   #5
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian + kde 4 / 5
Posts: 6,837

Rep: Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981
Quote:
Originally Posted by Nagy Szilveszter View Post
But inside a bash script I NEED THIS AS STRING...
No you don't. It's fairly rare that you would actually need to store the quoted values in a variable for use elsewhere. And as in most such cases your attempt to do so it's due to an improper desire to store a command (or list of arguments) inside a scalar variable.

I'm trying to put a command in a variable, but the complex cases always fail!
http://mywiki.wooledge.org/BashFAQ/050


The best option is generally to set up a function of some kind. But in this case it's also possible to use an array.

Code:
cond=( '!' '-wholename' './etc*' )
find . "${cond[@]}"
If you explain your "conditions" in detail, perhaps we could help you whip up an appropriate function instead.

PS: Stay far away from eval unless you know exactly what you're doing (and if you think you need it, most likely you're wrong).
 
1 members found this post helpful.
Old 09-03-2012, 03:52 AM   #6
Nagy Szilveszter
LQ Newbie
 
Registered: Sep 2012
Posts: 11

Original Poster
Rep: Reputation: Disabled
David's solution also works fine.

Would you please explain why using eval is not a good option? (Just to learn)


And here is the real-world scenario:
I am using this script for refreshing website content from SVN. I want to delete the old directory first BUT skipping the directories where is have configurations, logs, etc. Every project has a DELSKIP parameter in which are listed the subfolders that shouldn't be deleted. The code looks like this:

Code:
DELSKIP="etc var"
...

echo "Deleting old directory..."
if [ "$DELCOND" = "" -a "$DELSKIP" !- "" ]; then
    DELCOND=' ! -wholename $DESTDIR '
    for SKIP in $DELSKIP; do
        DELCOND=$DELCOND' ! wholename "$DESTDIR/'$SKIP'*" '
    done
fi
echo -e "find $DESTDIR $DELCOND -delete" >>$LOGFILE 2>>$LOGFILE
eval find $DESTDIR $DELCOND -delete >>$LOGFILE 2>>$LOGFILE
...

echo "-Exporting SVN $SVNURL to $DESTDIR" >>$LOGFILE
svn export $SVNURL $DESTDIR --username=$SVNusr --password=$SVNpwd --force --no-auth-cache --non-interactive -q >>$LOGFILE 2>>$LOGFILE

With David's solution i don't know how to build up the DELCOND as an array
It would be another great lesson if you'd help me

Thanks

Last edited by Nagy Szilveszter; 09-03-2012 at 03:53 AM.
 
Old 09-03-2012, 04:12 AM   #7
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 9,905

Rep: Reputation: 2921Reputation: 2921Reputation: 2921Reputation: 2921Reputation: 2921Reputation: 2921Reputation: 2921Reputation: 2921Reputation: 2921Reputation: 2921Reputation: 2921
Why don't you protect those files by setting them readonly?
 
Old 09-03-2012, 09:18 AM   #8
kakaka
Member
 
Registered: Sep 2003
Posts: 382

Rep: Reputation: 87
Nagy Szilveszter,

I realize that in your most recent message you've shown ... to indicate that portions of your script were omitted. But if I understand what you are trying to accomplish, even for those portions of the script that you showed, you might be happier if it looked a little more like this:

Code:
set -x

DELSKIP="etc var"
DESTDIR='./new'
LOGFILE='/tmp/refresh.log'

echo -e "\nAt `date`, Deleting old directory..." >>$LOGFILE
if [ \( "$DELCOND" == "" \)  -a  \(  "$DELSKIP" != "" \)  ]; then
    DELCOND=' ! -wholename '$DESTDIR
    for SKIP in $DELSKIP; do
        DELCOND=$DELCOND' ! -wholename "'$DESTDIR/$SKIP'*" '
    done
fi

echo -e "find $DESTDIR $DELCOND -delete" &>>$LOGFILE
eval find $DESTDIR $DELCOND -delete &>>$LOGFILE
 
Old 09-04-2012, 01:27 AM   #9
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian + kde 4 / 5
Posts: 6,837

Rep: Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981
First, I'm going to post a few of my standard cut&paste replies concerning common scripting errors. Your script extract displays them all.

1) QUOTE ALL OF YOUR VARIABLE SUBSTITUTIONS. You should never leave the quotes off a parameter expansion unless you explicitly want the resulting string to be word-split by the shell (globbing patterns are also expanded). This is a vitally important concept in scripting, so train yourself to do it correctly now. You can learn about the exceptions later.

http://mywiki.wooledge.org/Arguments
http://mywiki.wooledge.org/WordSplitting
http://mywiki.wooledge.org/Quotes

2) When using bash or ksh, it's recommended to use [[..]] for string/file tests, and ((..)) for numerical tests. Avoid using the old [..] test unless you specifically need POSIX-style portability.

http://mywiki.wooledge.org/ArithmeticExpression
http://mywiki.wooledge.org/BashFAQ/031
http://wiki.bash-hackers.org/commands/classictest
http://wiki.bash-hackers.org/syntax/...nal_expression
http://wiki.bash-hackers.org/syntax/arith_expr

3) Since environment variables are generally all upper-case, it's good practice to keep your own user variables in lower-case or mixed-case to help differentiate them.


Now, to build DELCOND as an array, just do the same thing as I posted before. Only use +=() to add entries to an existing array (assuming you want to create a list of them for find to use, instead of setting it new.

Code:
DELSKIP=( etc var )

if [[ -z "$DELCOND" && -n "$DELSKIP" ]]; then

    DELCOND=( '!' -path "$DESTDIR" )

    for SKIP in "${DELSKIP[@]}"; do
        DELCOND+=( '!' -path "$DESTDIR/$SKIP*" )
    done
fi

find "$DESTDIR" "${DELCOND[@]}" -delete >>"$LOGFILE" 2>&1
Notice also how you can simplify the redirection of stderr. Just send it to the current setting of stdout (just previously defined as $LOGFILE).

You also missed a hyphen in front of wholename, BTW, and you should be using -path instead anyway. Actually, your whole find command as it stands is probably wrong (e.g. wholename/path matches entire directory paths, so the pattern argument needs to be designed to properly match them.). You should test the syntax thoroughly before inserting it into your code.

The basic point is, scalar variables store single values. Any time you have a list of related entries, you should use an array instead.

How can I use array variables?
http://mywiki.wooledge.org/BashFAQ/005


Finally, concerning eval, start by carefully reading the link I gave you.

eval works by parsing the line twice. The first pass expands all the variables and command substitutions (i.e. it can execute code in the process) to build the final command, and then that command is parsed again on the second pass and executed.

The main danger lies in that this operation is not transparent. Malicious or poorly-written code can be inserted and executed without your knowledge, sometimes to deleterious effect. This means that you should never use it on any line that contains variables or command substitutions (or even processes file names or contents) that are not under your complete control; where you don't know exactly what they contain, and exactly how the line gets built and executed, on both passes.

Besides, using eval in a situation like this is equivalent to using a sledgehammer to drive home a finishing nail. It's way too powerful for the purpose, and liable to cause damage in the process. There are nearly always other techniques available, particularly in modern, advanced shells like bash and ksh, that are usually safer and more efficient.

eval is still occasionally needed in POSIX scripts, which are more limited in the shell features they can command. But even then I maintain that people depend on it much more often than they should.

Last edited by David the H.; 09-04-2012 at 01:33 AM. Reason: minor fixes
 
2 members found this post helpful.
Old 09-04-2012, 02:05 AM   #10
Nagy Szilveszter
LQ Newbie
 
Registered: Sep 2012
Posts: 11

Original Poster
Rep: Reputation: Disabled
Right, precise, to the point - you deserve 2 reps for this.

I completely agree with all you pointed out, it's just like how i use it in other programming languages...i just didn't know bash can do all these too, i found bash to be too poor and always had to find an alternative solution to achieve what i wanted...but now i see bash is MUCH more than we've learnt in school...
 
Old 09-04-2012, 04:33 AM   #11
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian + kde 4 / 5
Posts: 6,837

Rep: Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981
Thank you for the compliment.

People experienced in higher level languages sometimes show a tendency to discount or even denigrate shell scripting. Sure it's not the most efficient or "cleanest" form of programming, but it is programming. Shell scripting languages do offer a turing complete set of programming features, and in place of libraries you have access to just about any cli command available to your system. More modern shells like bash, ksh, and zsh also offer a large number of built-in features that make it even more convenient.

Like all languages it has its strengths and weaknesses. I certainly wouldn't recommend using the shell to create a program for general distribution, or for doing heavy lifting work like running a relational database or somesuch. But for the average home user like me it provides an easy-to-learn and easy-to-use way of creating simple programs for personal use and system maintenance.
 
Old 09-04-2012, 05:18 AM   #12
Nagy Szilveszter
LQ Newbie
 
Registered: Sep 2012
Posts: 11

Original Poster
Rep: Reputation: Disabled
I corrected my whole script taking in consideration your advices and i have one more question regarding the "1) QUOTE ALL OF YOUR VARIABLE SUBSTITUTIONS." issue:

When i have something like this:

Quote:
PROJECTSVNURL="$SVNBASEDIR/$PRJDIR/trunk"
Should i write it this way? (kind of ugly...)

Quote:
PROJECTSVNURL="$SVNBASEDIR""/""$PRJDIR""/trunk"
Is there any danger/possibility of mistake using the first form?

Last edited by Nagy Szilveszter; 09-04-2012 at 05:20 AM.
 
Old 09-04-2012, 05:20 AM   #13
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 9,905

Rep: Reputation: 2921Reputation: 2921Reputation: 2921Reputation: 2921Reputation: 2921Reputation: 2921Reputation: 2921Reputation: 2921Reputation: 2921Reputation: 2921Reputation: 2921
no, they are exactly the same
 
1 members found this post helpful.
Old 09-04-2012, 10:16 PM   #14
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian + kde 4 / 5
Posts: 6,837

Rep: Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981
Actually, the name=value variable setting expression is one of those few exceptions where quoting is optional. Word-splitting/globbing is not done on variable and command substitutions here. You only need to escape whitespace in the initial unexpanded string to ensure that it comprises a single token.

Code:
$ var1='a b c'
$ var2='d e f'
$ var3=$var1\ $var2

$ echo "[$var3]"
[a b c d e f]
Quotes are needed to protect the raw spaces in var1 & var2, but after they are inside the variables, they do not have to be protected when setting var3. The space between them does, however.

For example purposes I used a backslash to do that, but actually the general best practice is to simply quote the longest string possible. var3="$var1 $var2" is more easily readable than what I wrote above, as it provides a clear, at-a-glance visual grouping of the entire value. Only rarely should you need to concatenate separate strings together, or quote only part of a string.


The other two main quoting exceptions, BTW, are with the variable right after the case keyword, and variables used in [[..]] tests (with the partial exception of the right-hand-side, where it controls globbing & regex behavior).
 
Old 09-05-2012, 06:19 AM   #15
Nagy Szilveszter
LQ Newbie
 
Registered: Sep 2012
Posts: 11

Original Poster
Rep: Reputation: Disabled
That \ thingy doesn't work for me at this part:

RunAfter is a parameter where i tell what other tasks to do after the export

Quote:
# RunAfter="curl http://myserver.tld/generate.php ; curl http://myserver.tld/translate.php"
Trying to run it:
Quote:
# $RunAfter
curl: (6) Couldn't resolve host ';'
curl: (6) Couldn't resolve host 'curl'
OUT: Translated.
So the first script didn't run.
-----

The wrong way works:

Quote:
# eval $RunAfter

OUT: Generated. OUT: Translated.

I need to escape the ; in the middle, so the bash will recognize it as command separator.

Quote:
# RunAfter="curl http://myserver.tld/generate.php \; curl http://myserver.tld/translate.php"


# $RunAfter
curl: (6) Couldn't resolve host '\;'
curl: (6) Couldn't resolve host 'curl'
OUT: Translated.

Last edited by Nagy Szilveszter; 09-05-2012 at 06:25 AM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Yet Another Bash Quotes Within Quotes Issue tboyer Linux - Software 17 11-03-2012 12:17 PM
translate value from single quotes to double quotes venkateshrupineni Linux - Newbie 2 06-14-2012 04:03 PM
Problems with quotes and double quotes Andruha Slackware 6 01-02-2010 05:44 PM
Using single quotes vs double quotes in PHP strings vharishankar Programming 6 07-11-2005 12:41 PM
PostNuke install, magic quotes problem HippieCat Linux - Software 0 02-21-2005 05:06 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 10:48 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration