Excaping quotes problem
 

Hi all,

I have a problem that I can't solve. I'll post it here, maybe someone can help me.

So here is it:
I have a test folder with some subfolders:

I want to list all folders and files except the etc folder and its contents
Using the following command, and i get exactly what i want:

Till this point it's fine. But inside a bash script I NEED THIS AS STRING because i'm building up the condition based on some internal values. Now look at this:

And when i run find again...

...it lists the etc folder and its contents which it shouldn't.

I'm sure it's a quotation problem, and I tried all variations I know, but wasn't able to solve the problem.
Where is the mistake?

I appreciate your help
Szilvi

Hi, welcome to LQ!

Try

Hi Nagy Szilveszter!

You can sometimes see what the bash parser is doing with a given line from a script, by placing:

in your script.

Doing that with the command line you are using, it turns out that the issue is the difference between the way bash expands ./etc* and the way find -wholename expands it.

With your directory structure, bash only matches ./etc

So when you use virtually any form of quoting or evaluation where bash does it's matching, bash will replace it with the matches. You need to force a situation where the asterisk stays in place until it is fully passed to find -wholename. Running a script with these lines:

will show you how the parser is interpreting each line. Either the xargs approach, or the eval approach produce the same output as the command you started with, as long as the asterisk is escaped.

HTH.

Thanks a lot both of you! It works with eval.... the xargs is a also great but i need to evolve to understand it.

The "set -x" is a really useful thing!

No you don't. It's fairly rare that you would actually need to store the quoted values in a variable for use elsewhere. And as in most such cases your attempt to do so it's due to an improper desire to store a command (or list of arguments) inside a scalar variable.

I'm trying to put a command in a variable, but the complex cases always fail!
http://mywiki.wooledge.org/BashFAQ/050


The best option is generally to set up a function of some kind. But in this case it's also possible to use an array.

If you explain your "conditions" in detail, perhaps we could help you whip up an appropriate function instead.

PS: Stay far away from eval unless you know exactly what you're doing (and if you think you need it, most likely you're wrong).

David's solution also works fine.

Would you please explain why using eval is not a good option? (Just to learn)


And here is the real-world scenario:
I am using this script for refreshing website content from SVN. I want to delete the old directory first BUT skipping the directories where is have configurations, logs, etc. Every project has a DELSKIP parameter in which are listed the subfolders that shouldn't be deleted. The code looks like this:


With David's solution i don't know how to build up the DELCOND as an array
It would be another great lesson if you'd help me

Thanks

Why don't you protect those files by setting them readonly?

Nagy Szilveszter,

I realize that in your most recent message you've shown ... to indicate that portions of your script were omitted. But if I understand what you are trying to accomplish, even for those portions of the script that you showed, you might be happier if it looked a little more like this:

First, I'm going to post a few of my standard cut&paste replies concerning common scripting errors. Your script extract displays them all.

1) QUOTE ALL OF YOUR VARIABLE SUBSTITUTIONS. You should never leave the quotes off a parameter expansion unless you explicitly want the resulting string to be word-split by the shell (globbing patterns are also expanded). This is a vitally important concept in scripting, so train yourself to do it correctly now. You can learn about the exceptions later.

http://mywiki.wooledge.org/Arguments
http://mywiki.wooledge.org/WordSplitting
http://mywiki.wooledge.org/Quotes

2) When using bash or ksh, it's recommended to use [[..]] for string/file tests, and ((..)) for numerical tests. Avoid using the old [..] test unless you specifically need POSIX-style portability.

http://mywiki.wooledge.org/ArithmeticExpression
http://mywiki.wooledge.org/BashFAQ/031
http://wiki.bash-hackers.org/commands/classictest
http://wiki.bash-hackers.org/syntax/...nal_expression
http://wiki.bash-hackers.org/syntax/arith_expr

3) Since environment variables are generally all upper-case, it's good practice to keep your own user variables in lower-case or mixed-case to help differentiate them.


Now, to build DELCOND as an array, just do the same thing as I posted before. Only use +=() to add entries to an existing array (assuming you want to create a list of them for find to use, instead of setting it new.

Notice also how you can simplify the redirection of stderr. Just send it to the current setting of stdout (just previously defined as $LOGFILE).

You also missed a hyphen in front of wholename, BTW, and you should be using -path instead anyway. Actually, your whole find command as it stands is probably wrong (e.g. wholename/path matches entire directory paths, so the pattern argument needs to be designed to properly match them.). You should test the syntax thoroughly before inserting it into your code.

The basic point is, scalar variables store single values. Any time you have a list of related entries, you should use an array instead.

How can I use array variables?
http://mywiki.wooledge.org/BashFAQ/005


Finally, concerning eval, start by carefully reading the link I gave you.

eval works by parsing the line twice. The first pass expands all the variables and command substitutions (i.e. it can execute code in the process) to build the final command, and then that command is parsed again on the second pass and executed.

The main danger lies in that this operation is not transparent. Malicious or poorly-written code can be inserted and executed without your knowledge, sometimes to deleterious effect. This means that you should never use it on any line that contains variables or command substitutions (or even processes file names or contents) that are not under your complete control; where you don't know exactly what they contain, and exactly how the line gets built and executed, on both passes.

Besides, using eval in a situation like this is equivalent to using a sledgehammer to drive home a finishing nail. It's way too powerful for the purpose, and liable to cause damage in the process. There are nearly always other techniques available, particularly in modern, advanced shells like bash and ksh, that are usually safer and more efficient.

eval is still occasionally needed in POSIX scripts, which are more limited in the shell features they can command. But even then I maintain that people depend on it much more often than they should.

Right, precise, to the point - you deserve 2 reps for this.

I completely agree with all you pointed out, it's just like how i use it in other programming languages...i just didn't know bash can do all these too, i found bash to be too poor and always had to find an alternative solution to achieve what i wanted...but now i see bash is MUCH more than we've learnt in school...

Thank you for the compliment.

People experienced in higher level languages sometimes show a tendency to discount or even denigrate shell scripting. Sure it's not the most efficient or "cleanest" form of programming, but it is programming. Shell scripting languages do offer a turing complete set of programming features, and in place of libraries you have access to just about any cli command available to your system. More modern shells like bash, ksh, and zsh also offer a large number of built-in features that make it even more convenient.

Like all languages it has its strengths and weaknesses. I certainly wouldn't recommend using the shell to create a program for general distribution, or for doing heavy lifting work like running a relational database or somesuch. But for the average home user like me it provides an easy-to-learn and easy-to-use way of creating simple programs for personal use and system maintenance.

I corrected my whole script taking in consideration your advices and i have one more question regarding the "1) QUOTE ALL OF YOUR VARIABLE SUBSTITUTIONS." issue:

When i have something like this:

Should i write it this way? (kind of ugly...)

Is there any danger/possibility of mistake using the first form?

no, they are exactly the same

Actually, the name=value variable setting expression is one of those few exceptions where quoting is optional. Word-splitting/globbing is not done on variable and command substitutions here. You only need to escape whitespace in the initial unexpanded string to ensure that it comprises a single token.

Quotes are needed to protect the raw spaces in var1 & var2, but after they are inside the variables, they do not have to be protected when setting var3. The space between them does, however.

For example purposes I used a backslash to do that, but actually the general best practice is to simply quote the longest string possible. var3="$var1 $var2" is more easily readable than what I wrote above, as it provides a clear, at-a-glance visual grouping of the entire value. Only rarely should you need to concatenate separate strings together, or quote only part of a string.


The other two main quoting exceptions, BTW, are with the variable right after the case keyword, and variables used in [[..]] tests (with the partial exception of the right-hand-side, where it controls globbing & regex behavior).

That \ thingy doesn't work for me at this part:

RunAfter is a parameter where i tell what other tasks to do after the export

Trying to run it:
So the first script didn't run.
-----

The wrong way works:


I need to escape the ; in the middle, so the bash will recognize it as command separator.