LinuxQuestions.org
LinuxAnswers - the LQ Linux tutorial section.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 12-04-2009, 08:06 PM   #1
GrapefruiTgirl
Guru
 
Registered: Dec 2006
Location: underground
Distribution: Slackware64
Posts: 7,594

Rep: Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550
Angry #!/bin/sh vs #!/bin/bash -- script executes differently; each way = different bugs


So far I haven't found the exact answer I'm looking for; I don't even really know what the question is though various mailing lists and threads elsewhere tell me that when bash is invoked by the #!/bin/sh method, it starts in some sort of sh compatibility mode.
I presume that this would mean that a script written to be invoked with bin/sh would be compatible with other shells, not just bash and not necessarily just on Linux.

Anyhow...

So, keeping in mind that I'd like to if at all possible be sure my script will work on all sh or bash shells, consider the following:

I have been trying to debug a weird problem involving a gawk statement. I've been fiddling with this weird problem for days now.

My script starts with #!/bin/sh.

Now, in an effort to debug, I tried running the script with a commandline like:

shell# bash -x ./script

instead of the ./ usual; I discovered that the FIRST weird gawk problem does not occur now. Instead, a different gawk statement acts weird, AND/BUT: The script executes to completion, terminating with an error because of the GARBAGE input I gave it.

Why should it not execute to completion? Because I am testing its sanity checks which make the script fail if given garbage input. So to test, I give it garbage input and run it. When started with /bin/sh the gawk problem screws the works long before execution would otherwise terminate, and when I start it with /bin/bash the gawk works fine, but a bunch of other crap all fails, the garbage is ignored, and the script runs to completion :/

What to do??

I don't know if I'm looking for an absolute answer, but for now, what are the big implications/differences between /bin/bash and /bin/sh? Have you run into a situation like this?

FWIW, the pesky gawk line is like so:

Code:
ip=$(echo $1 | gawk -v FS=. '/\//{ gsub("/",".") ; print and($1,$5)"."and($2,$6)"."and($3,$7)"."and($4,$8) }' )
Here it is in context; here's lines 1240-1247 inclusinve, from the script:
Code:
get_if_from_ip () {

# comment: NOTE: during debugging, I removed doublequotes from the FS=. below in the gawk. removed ; before "print" too.
echo "$LINENO: just entered get_if_from_ip() and 1=$1"
echo "this is line 1244 and the gawk is about to happen on this line";ip=$(echo $1 | gawk -v FS=. '/\//{ gsub("/",".") ; print and($1,$5)"."and($2,$6)"."and($3,$7)"."and($4,$8) }' )
echo "this is an echo statement on line 1245"
echo "Line $LINENO: just gawked 1; now ip=$ip PRESS ENTER to cont.."
read debugjunks
..and when it's executed with # bash ./script or # sh ./script there's no trouble; with ./ however, the line runs as though it's missing a closing quote, or closing backtick, or closing whatever, and the following line(s) of the script actually becomes part of the resulting value of $ip.

Watch this bunch of console output when via # sh -- I sedded a $LINENO and/or small message echo onto every line of the script, so nearly every line outputs a line number to the console -- everything below comes from the script, except the bold text:
NOTE: IP address gummed up for safety reasons.
Quote:
NOTE: these first few lines are where the pesky gawk statement is first executed, on line 1244.
(742 done)
(885): about to jump to get_if_from_ip() with arg=
(887 done)
(888)rule 1: side_if[1]=1243: just entered get_if_from_ip() and $1=00.000.162.205
..but not till HERE do I see the message at the start of that line, nor the PRESS ENTER message after that line. See next bold para for more.
this is line 1244 and the gawk is about to happen on this line
this is an echo statement on line 1245
Line 1246_ just gawked $1; now ip= PRESS ENTER to cont..
(1249 done)
(1251 done)
(1254 done)
(1256 done)
(1254 done)
(1256 done)
(1254 done)
(1256 done)

(1259 done)
(742 done)
(1014 done)
Rule 1:1: eth0 00.000.162.205 garbage accept
(742 done)
(1017 done)
(1018 done)
(1019 done)
Clients: ip 192.168.2.3 Device: eth0
(742 done)
(1020 done)
(1021 done)
(1022 done)
Servers: ip 00.000.162.205 Device:
1243: just entered get_if_from_ip() and $1=00.000.162.205
this is line 1244 and the gawk is about to happen on this line
this is an echo statement on line 1245
NOTE script stops right HERE, waiting for me to press enter, because the gawk statement
has "absorbed" the following 2 lines of code, including the echo statement and `read` statement
on line 1246. I don't get to actually SEE the next line (1246) until I hit ENTER (thereby "completing" the borked gawk statement)

Line 1246: just gawked 1; now ip= PRESS ENTER to cont..
(1249 done)
(1251 done)
(1254 done)
(1256 done)
(1254 done)
(1256 done)
(1254 done)
(1256 done)

(1259 done)
(742 done)
(1023 done)
Services: GARBAGE Targets: ACCEPT

(742 done)
(1024 done)
(1030 done)
(1031 done)
(1032 done)
(1033 done)
# NOTE: See all these "bad substitution" errors below? Look inside the ${} -- those 4-digit numbers and the word "done" come from echo statements like: echo "$LINENO done" elsewhere in the program! Somehow they are getting turned into variables. wtf!?
./rc.script.013: line 1040: ${1243_[6]}: bad substitution
./rc.script.013: line 1040: ${get_if_from_ip()[6]}: bad substitution
./rc.script.013: line 1040: $1: cannot assign in this way
./rc.script.013: line 1040: ${1244[6]}: bad substitution
./rc.script.013: line 1040: ${1245[6]}: bad substitution
./rc.script.013: line 1040: ${1246_[6]}: bad substitution
./rc.script.013: line 1040: ${1;[6]}: bad substitution
./rc.script.013: line 1040: ${cont..[6]}: bad substitution
./rc.script.013: line 1040: ${(1249[6]}: bad substitution
./rc.script.013: line 1040: ${done)[6]}: bad substitution
./rc.script.013: line 1040: ${(1251[6]}: bad substitution
./rc.script.013: line 1040: ${done)[6]}: bad substitution
./rc.script.013: line 1040: ${(1254[6]}: bad substitution
./rc.script.013: line 1040: ${done)[6]}: bad substitution
./rc.script.013: line 1040: ${(1256[6]}: bad substitution
./rc.script.013: line 1040: ${done)[6]}: bad substitution
./rc.script.013: line 1040: ${(1254[6]}: bad substitution
./rc.script.013: line 1040: ${done)[6]}: bad substitution
./rc.script.013: line 1040: ${(1256[6]}: bad substitution
./rc.script.013: line 1040: ${done)[6]}: bad substitution
./rc.script.013: line 1040: ${(1254[6]}: bad substitution
./rc.script.013: line 1040: ${done)[6]}: bad substitution
./rc.script.013: line 1040: ${(1256[6]}: bad substitution
./rc.lutelwall.013: line 1040: ${done)[6]}: bad substitution
./rc.script.013: line 1040: ${(1259[6]}: bad substitution
./rc.script.013: line 1040: ${done)[6]}: bad substitution
and here's the script output when executed with identical GARBAGE input, except this time it's executed at the commandline:

Quote:
# ./rc.script.013 start | tee debug.B.013
...
... # normal execution so far..
(794 done)
(795 done)
(793 done)
(794 done)
(795 done)
(793 done)
(794 done)
(795 done)
(793 done)
(794 done)
(795 done)
(973 done)
(975 done)
(976 done)
(977 done)
(978 done)
(979 done)
Rule: (eth0 00.000.162.205 garbage accept)
Error-- wrong target "ETH0~00.000.162.205~GARBAGE~ACCEPT"; ** RULE SKIPPED **
(742 done) #NOTE: 1) the Error line above should have been parsed at the tildes by gawk (didn't happen) and 2) this line right here is AFAICT the same point of execution that I started the copy of output in the first [ quote ] block above.
Config option 'IgnoreErrors' is set to false. Due to an error, script is exiting now.
(742 done)
(748 done)
(749 done) # last line
..and as we can see, based on the line numbers outputted, the execution has taken a totally different course. The script DOES terminate sort-of as it should, BUT: on the line where it says RULE SKIPPED, "target" should be "accept", not that whole string; the tildes were supposed to have been gawked out (result of a different failing gawk behavior)

So I suppose, there are two questions here:

1) What's the big diff's between bash and sh;
and
2) How does bash vs sh affect interpretation + execution of things like gawk statements, shell expansion, etc.? Website links are welcome.
3) Does anyone see anything about that gawk statement way up there, that might cause it to act weird under any circumstances? It runs fine in the console (bash).

And if all the above makes no sense at all to you, and you have not one iota of a clue as to wth is going on here in the least, or you have never heard of bash, please reserve funny commentary for a bit later, after there has been maybe some educated feedback -- thank you!

PS - This post is part <rant> and part post and may not make much sense -- after all the copying, pasting & explaining, I'm lost...

And, maybe there's some little simple obvious thing(s) I've missed in here, that would eliminate all this weirdness.

NOTE: there are typos/brainos above somewhere.

Sasha

Last edited by GrapefruiTgirl; 12-04-2009 at 08:11 PM.
 
Old 12-04-2009, 08:50 PM   #2
dxqcanada
Member
 
Registered: Sep 2006
Location: Canada
Distribution: Gentoo
Posts: 702

Rep: Reputation: 43
/bin/sh is usually a pointer to a shell.
Code:
# ls -l /bin/sh
If you are writing a shell script for a specific shell interpreter ... then state it explicitly ( #!/bin/bash )
 
Old 12-04-2009, 10:23 PM   #3
GrapefruiTgirl
Guru
 
Registered: Dec 2006
Location: underground
Distribution: Slackware64
Posts: 7,594

Original Poster
Rep: Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550
FWIW, I am FINALLY starting to figure out wth is going on with the script, though I have not yet tested my findings using the different shell invocations;

The large part of my trouble involves (our old favorites!) different instances of when/where to use quotes/backticks/brackets:
Quote:
[ $(something) ] && ...
[ "$(something)" ] && ...
[ "$(eval echo \${${some_thing[x]}})" ] && ...
if [ "${some_thing[x]}" = "$(eval echo \${${some_otherthing[y]}[z]})" ]; then
..in other words, depending on the type of shell expansion expected, and whether or not it is inside [] and whether of not is is being compared to something else, the shell expects different items around the items to be expanded.
When the proper items are not surrounding the expansion stuff, WEIRD things happen: other lines of the script get sucked into the expansion process..

Likely there's more, but this is where I'm at this moment.
 
Old 12-04-2009, 10:46 PM   #4
alunduil
Member
 
Registered: Feb 2005
Location: San Antonio, TX
Distribution: Gentoo
Posts: 684

Rep: Reputation: 62
Have you seen The Advanced Bash Scripting Guide? It might give you some guidance on some of the things you're doing.

Regards,

Alunduil
 
Old 12-05-2009, 01:26 AM   #5
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,551
Blog Entries: 28

Rep: Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176
Hello Sasha

Bash behaves differently when invoked as sh as detailed here (scroll down to "Invoked with name sh") where it says "When invoked as sh, Bash enters posix mode after the startup files are read".

Details of differences: bash/POSIX.

If tracing (-x) propagates to pipelines (by definition including single commands) in $( ) and ` ` then the output from var="$( <pipeline> )" and var="` <pipeline> `"includes the trace output and almost certainly breaks the script. It used to behave this way in ksh and POSIX shell and, anyway a lot of trace output is befuddlifusing so I got out of the habit of using it and now insert set -x and set +x around the section to be debugged.
 
1 members found this post helpful.
Old 12-05-2009, 01:41 AM   #6
gnashley
Amigo developer
 
Registered: Dec 2003
Location: Germany
Distribution: Slackware
Posts: 4,768

Rep: Reputation: 477Reputation: 477Reputation: 477Reputation: 477Reputation: 477
Using backticks instead of $() should help. If you really want to write it so it is portable, then run it using dash or ash. dash claims to be completely POSIX-compatible. Even when running bash as 'sh' it will still have capabilities which are not compatible with other shells.
 
1 members found this post helpful.
Old 12-05-2009, 04:28 AM   #7
H_TeXMeX_H
Guru
 
Registered: Oct 2005
Location: $RANDOM
Distribution: slackware64
Posts: 12,928
Blog Entries: 2

Rep: Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269
Quote:
Originally Posted by catkin View Post
Hello Sasha

Bash behaves differently when invoked as sh as detailed here (scroll down to "Invoked with name sh") where it says "When invoked as sh, Bash enters posix mode after the startup files are read".

Details of differences: bash/POSIX.

If tracing (-x) propagates to pipelines (by definition including single commands) in $( ) and ` ` then the output from var="$( <pipeline> )" and var="` <pipeline> `"includes the trace output and almost certainly breaks the script. It used to behave this way in ksh and POSIX shell and, anyway a lot of trace output is befuddlifusing so I got out of the habit of using it and now insert set -x and set +x around the section to be debugged.
Interesting, I didn't know that. I always use '/bin/sh'.

As for the script, I think you might want to look for a better way to do it, maybe with awk.
 
Old 12-05-2009, 05:18 AM   #8
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,551
Blog Entries: 28

Rep: Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176
Might be worth trying
Code:
ip="$(echo $1 | gawk -v FS=. '/\//{ gsub("/",".") ; print and($1,$5)"."and($2,$6)"."and($3,$7)"."and($4,$8) }' )"
 
Old 12-05-2009, 10:17 AM   #9
GrapefruiTgirl
Guru
 
Registered: Dec 2006
Location: underground
Distribution: Slackware64
Posts: 7,594

Original Poster
Rep: Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550
bash debugging.

Quote:
Originally Posted by catkin View Post
Hello Sasha

Bash behaves differently when invoked as sh as detailed here (scroll down to "Invoked with name sh") where it says "When invoked as sh, Bash enters posix mode after the startup files are read".

Details of differences: bash/POSIX.
Perfect -- those may be just the links I need; will check them out today!
Quote:
If tracing (-x) propagates to pipelines (by definition including single commands) in $( ) and ` ` then the output from var="$( <pipeline> )" and var="` <pipeline> `"includes the trace output and almost certainly breaks the script. It used to behave this way in ksh and POSIX shell and, anyway a lot of trace output is befuddlifusing so I got out of the habit of using it and now insert set -x and set +x around the section to be debugged.
You're right, I found that turning -x debugging ON broke a lot of stuff :/ and was not as helpful as I would have liked. The output is less than intuitive.
One handy thing I have found though, to be used when running with -x turned on, is to set your PS4 prompt like so:

Code:
export PS4='(LINE:${LINENO} FUNC:${FUNCNAME[0]} [shlvl:${SHLVL} subsh:${BASH_SUBSHELL} error:$?]
'
set -x
And now you get line numbers, function name, error return ($?) and a couple other useful things from the debug output. NOTE the newline -- it's supposed to be there.

At this time, I'm back to where everything works perfectly -- a significant part of my problem, was that, a couple days ago, I merrily went willy-nilly putting double-quotes around a bunch of $(...) operations; it screwed a lot of stuff. dunno how many times I have to learn that lesson: Change ONE thing, then test!!

Thanks!
Sasha
 
1 members found this post helpful.
Old 12-05-2009, 10:29 AM   #10
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,551
Blog Entries: 28

Rep: Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176
Quote:
Originally Posted by GrapefruiTgirl View Post
One handy thing I have found though, to be used when running with -x turned on, is to set your PS4 prompt like so:

Code:
export PS4='(LINE:${LINENO} FUNC:${FUNCNAME[0]} [shlvl:${SHLVL} subsh:${BASH_SUBSHELL} error:$?]
'
set -x
Neat!
Quote:
Originally Posted by GrapefruiTgirl View Post
At this time, I'm back to where everything works perfectly -- a significant part of my problem, was that, a couple days ago, I merrily went willy-nilly putting double-quotes around a bunch of $(...) operations; it screwed a lot of stuff. dunno how many times I have to learn that lesson: Change ONE thing, then test!!
MMC but it's surprising that double-quoting $( ) screwed things up, unless you were not assigning the output to a variable and needed it to be parsed into separate words ...
 
Old 12-05-2009, 10:42 AM   #11
GrapefruiTgirl
Guru
 
Registered: Dec 2006
Location: underground
Distribution: Slackware64
Posts: 7,594

Original Poster
Rep: Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550
Quote:
Originally Posted by catkin View Post
...surprising that double-quoting $( ) screwed things up, unless you were not assigning the output to a variable and needed it to be parsed into separate words ...
Some cases, where a variable was trapping the results, worked fine, but some didn't.
the problem was in (for example) [ bracketed tests ] when there was multiply-embedded shell substitution or expansion, such as:

[ "$(eval echo \${${some_thing[x]}})" ]

in cases like this, the expansion/substitution failed, causing very bizarre results.

PS - what's MMC?

Sasha

Last edited by GrapefruiTgirl; 12-05-2009 at 10:45 AM.
 
Old 12-07-2009, 12:26 AM   #12
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.5, Centos 5.10
Posts: 16,301

Rep: Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037
sounds like you're having fun. I've got a couple of suggestions that may(!) help.

1.double brackets [[ ]] is generally more robust
Quote:
Using the [[ ... ]] test construct, rather than [ ... ] can prevent many logic errors in scripts. For example, the &&, ||, <, and > operators work within a [[ ]] test, despite giving an error within a [ ] construct.
http://tldp.org/LDP/abs/html/testcon...ml#DBLBRACKETS

2. if I get to the point where I need serious debugging info I use

set -xv

just before the trouble, or just as the 2nd line after the shell declaration; and you can turn them off in the usual *nix way

set +xv
 
1 members found this post helpful.
Old 12-07-2009, 01:19 AM   #13
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,551
Blog Entries: 28

Rep: Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176
Quote:
Originally Posted by GrapefruiTgirl View Post
Some cases, where a variable was trapping the results, worked fine, but some didn't.
the problem was in (for example) [ bracketed tests ] when there was multiply-embedded shell substitution or expansion, such as:

[ "$(eval echo \${${some_thing[x]}})" ]

in cases like this, the expansion/substitution failed, causing very bizarre results.

PS - what's MMC?

Sasha
As chrism01 suggested [[ ]] might solve that problem. MMC="made me chuckle". I learned that from my daughter who was very supercilious about me not knowing it so it's comforting to find I am not alone
 
Old 12-07-2009, 02:27 PM   #14
GrapefruiTgirl
Guru
 
Registered: Dec 2006
Location: underground
Distribution: Slackware64
Posts: 7,594

Original Poster
Rep: Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550
OK, cool -- thanks you guys for the tips about the double brackets; I've read of it MANY times but never found that I actually needed to do it, although I have encountered the operator errors within the [ .. ] constructs, and generally got around it by switching operator type, such as from | to -o or from != to -ne. I always wrote this off as bash telling me that I was using the wrong operator for the type of variable, i.e. trying to compare $string to $integer ]. I had pretty much given up on < > and switched to -lt -gt instead, but this double-bracket thing will have me re-examining this stuff.


Thanks!!
Sasha
 
Old 12-07-2009, 02:38 PM   #15
tuxdev
Senior Member
 
Registered: Jul 2005
Distribution: Slackware
Posts: 2,014

Rep: Reputation: 115Reputation: 115
For doing stuff that's mathematical, it's actually better to use (()) arithmetic test construct
 
  


Reply

Tags
bash, gawk, sh


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Really strange problem - RedHat /bin/sh vs /bin/bash leeharris Programming 4 02-13-2009 06:35 AM
How to add paths to sbin, bin/bash, bin sh for users gopi.d Fedora 3 12-07-2007 12:47 AM
Change /bin/bash to /bin/zsh Smokey Slackware 12 07-14-2004 01:06 AM
why did bash 2.05b install delete /bin/bash & "/bin/sh -> bash"? johnpipe Linux - Software 2 06-06-2004 06:42 PM
bin/bash:usr/bin/lpr NO SUCH FILE OR DIRECTORY Adibe_Hamm Linux - Newbie 3 10-14-2003 02:30 AM


All times are GMT -5. The time now is 12:03 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration