[SOLVED] Shell Scripting: Scraping Public IP and Emailing - [: too many arguments

zer0signal · 07-05-2011, 12:18 AM

Thank you guys very much! by messing with the code from your suggestions, i have at least accomplished no more false alerting for 2 days now and removed erroneous messages being received! I appreciate this!

Cheers!

Oh yeah! - btw I am using BASH, I really should have expressed that in the beginning! My apologies! the reason it is labeled ".sh" is because whenever i have to wipe or back a system... its alot easier to just search for a .sh extension, than trying to remember where I placed all my scripts or remember the name of them.... Though I usually tend to get them backed up! But makes it easier in an emergency!

konsolebox · 07-05-2011, 07:02 AM

I haven't told guys but [[ is similarly available in major shells like bash, ksh, pdksh and zsh.

Quote:

Originally Posted by Nominal Animal

set -f

Is it generally implemented in all sh? I didn't really care for it much since I was thinking about compatibility. i.e. ash, dash, heirloom sh, etc. That should be helpful if that's the case and I should reconsider.

Quote:

Originally Posted by Nominal Animal

Unless you do use an ancient version of sh, like I do.

I did use them many times just to complete something.

Quote:

Originally Posted by Nominal Animal

Code:

grep exp BIGFILE.TXT | while read -r LINE ; do ... ; done

does stream the grep results line by line, but the loop body is a subshell (making it difficult to pass results outside). The -r parameter tells the shell to not interpret backslash escapes.

Yep I didn't consider that method since it will put the loop inside a subshell thus losing values in variables when it ends.

Quote:

Originally Posted by Nominal Animal

I personally like the Bash-specific <( list ) a lot. It solves very cleanly the aforementioned result-passing problem. However, the way it does it -- the expression resolves to a file name, with the "file" containing the data -- is perhaps a bit surprising. For example, this command will work just fine:

Code:

dd if=<( for A in one two three ; do echo "$A" ; done )

Yes it's very helpful. e.g.:

Code:

while read -r -u 3 A && read -r -u 4 B; do
    echo "$A -- $B"
done 3< <(exec "do something") 4< <(exec "do something")

I like using exec inside <() when executing external commands just to make it a little more efficient; not letting it summon another subprocess.

I also remember using it with mkisofs or cdrecord to burn audio tracks that are being converted inside <() on the fly.

I had 17 to 19 instances of <() in one command. I think what was inside <() was sox. Not sure though. That was just 2004 (bash 2.05b or 3.0) and it was stable already.

Update: It was sox. I made a post of it 2 years later: http://www.linuxquestions.org/questi...5/#post2347884

Can't do that in other shells

The code was still quirky and it wasn't really complicated. What was I saying there LOL

Quote:

Originally Posted by Nominal Animal

/proc/self/fd/

Yeah.. and sometimes it's /dev/* and rarely a named pipe as well..

Nominal Animal · 07-05-2011, 09:42 PM

Quote:

Originally Posted by grail

I am not sure where you got this sentiment from?

The link, in my opinion, just ignores quoting and escaping rules altogether. It states that "[[ is a new improved version of it" (referring to [), and only refers to quoting or escaping as something you do not need to do if you use [[ instead of [. For example, "With [ the variable needs to be quoted". In my opinion, the link also gives the impression that quoting and escaping is something certain commands require.

Quoting and escaping rules are something you have to do in shell scripts -- for external command or script execution if nothing else -- in all shells. The only exception I'm aware of is [[.

Between shells, there are only are minimal differences in escaping rules; I believe the quoting rules are identical. These rules are something a shell script writer should learn first, because they are applied everywhere (except for [[). If you apply the rules always, even when technically not required, your scripts will work regardless of input. If you fail to apply the rules, your scripts will break for some input.

I'm definitely not criticising the use of Bash over a POSIX shell. Indeed, I thought the example code David the H. showed is very nice. It is a very good example in my opinion. I prefer Bash for shell scripts myself.

I only have a problem with recommending the use of [[. I seem to be the only one who believes [[ might reduce the incentive to learn the quoting and escaping rules, and that's fine. I understand that viewpoint. I just disagree. But don't worry, I'm not going to raise this issue again; I will just quietly keep avoiding [[ in my own example scripts, and ignoring the responses recommending replacing [ with [[.

I wonder if it would be worthwhile to write a short guide to quoting and escaping in shell scripts, with an introduction explaining its importance and prevalence, and point script writers to it whenever the quoting/escaping problems crop up?

Quote:

Originally Posted by konsolebox

Is it (set -f) generally implemented in all sh?

Yes, -f (disable pathname expansion) and -e (exit immediately if any nonzero exit status occurs) (and their counterparts +f and +e) are universally portable. They are defined in the POSIX shell spec (and thus guaranteed to not go away anytime soon), but also are implemented in all the shells I have access to, including the ancient sh in SunOS 5.10.

I avoid the other options, they're not as portable.

When using set , just remember that a subshell will inherit the settings from its parent, and changing them won't change them in the parent. In a loop body, changing the setting once will persist over iterations. (Change it at the start of the first iteration, and it will stay that way for the rest of the loop.)

grail · 07-06-2011, 01:17 AM

Quote:

I wonder if it would be worthwhile to write a short guide to quoting and escaping in shell scripts, with an introduction explaining its importance and prevalence, and point script writers to it whenever the quoting/escaping problems crop up?

I think that is a fantastic idea. I know I personally struggled with quoting and understanding 'full' or 'partial'
made no sense at all for ages. Personally it was only on writing more and then helping people here that made me acknowledge
the differences

PS. I could even see this as a sticky in the Programming section (or the like)

konsolebox · 07-06-2011, 05:18 AM

Quote:

Originally Posted by Nominal Animal

Yes, -f (disable pathname expansion) and -e (exit immediately if any nonzero exit status occurs) (and their counterparts +f and +e) are universally portable. They are defined in the POSIX shell spec (and thus guaranteed to not go away anytime soon), but also are implemented in all the shells I have access to, including the ancient sh in SunOS 5.10.

I avoid the other options, they're not as portable.

When using set , just remember that a subshell will inherit the settings from its parent, and changing them won't change them in the parent. In a loop body, changing the setting once will persist over iterations. (Change it at the start of the first iteration, and it will stay that way for the rest of the loop.)

So even in that ancient sh in SunOS.. I guess I'll try removing my doubt about it from now on. Thanks for explaining. It saved me from installing all those shells again

@Nominal Animal, @grail: Topics about having many invalid arguments in a command were already discussed many times here in LQ. I also think that it's not a bad idea to make a compiled note about it. Wonder if it's already done somewhere..

David the H. · 07-06-2011, 08:28 AM

Don't assume that the guys at Greg's wiki are soft on quoting. They may not have put much focus on it on the [[ page, but on other pages they strongly emphasize just just how important it is.

http://mywiki.wooledge.org/WordSplitting
http://mywiki.wooledge.org/Quotes
http://mywiki.wooledge.org/ShellHallOfShame

Indeed, their "arguments" page is probably the most useful single page I've ever read on this topic. It goes into great detail to explain exactly why you have to quote things the way you do. Everybody should read this page.

http://mywiki.wooledge.org/Arguments

I believe that what they really want to focus on there is not so much teaching any single "best practice", but more that you should try to thoroughly understand everything about the shell...not just the syntax, but what it's doing underneath. You need to understand why it works the way it does if you really want to script well.

I just recently read through their Bash Guide, for example, and I don't think I've ever read a primer that so deftly explained the key concepts while still being easily readable. From now on it's going to be the first scripting guide I recommend to newcomers.

http://mywiki.wooledge.org/BashGuide