[SOLVED] Same bash script has different behavior in 2 separate scripts. Thoughts?

uncorrupt3d · 03-17-2020, 02:28 PM

Hey guys,

So I've been working on a really large project for work and I can't seem to crack this bug right now... So I'm trying to get my script to verify all users' home directories exist, using the following code:

Code:

#!/bin/bash
grep -E -v '^(halt|sync|shutdown)' /etc/passwd | awk -F: '($7 != "'"$which nologin)"'" && $7 != "/bin/false") { print $1 " " $6 }' | while read -r user dir; 
do if [ ! -d "$dir" ]; 
then echo "The home directory ($dir) of user $user does not exist.";
fi;
done;

When I run this as an individual shell script, I get several outputs that look somewhat like:

Code:

The home directory (HOME_DIR_PATH) of user USERNAME does not exist.

However, when I remove the #!/bin/bash line and copy the rest of the text into a function inside my shell script (same exact thing), I get the following behavior on an infinite loop:

Code:

The home directory () of user does not exist.

Do you guys have any ideas? I've tried running a separate script with this chunk of bash code from my project to resolve this, rewriting the logic entirely and using shell code checkers- I originally had a syntax issue with awk.

Your help matters a ton! Thx

MensaWater · 03-17-2020, 04:26 PM

It is because you're piping into your while loop at the beginning rather than redirecting at the end. With the pipe the variables are local and can't be used elsewhere such as within a function.

Code:

while read -r user dir
do if [ ! -d "$dir" ] 
then echo "The home directory ($dir) of user $user does not exist."
fi
done < <grep -E -v '^(halt|sync|shutdown)' /etc/passwd | awk -F: '($7 != "'"$which nologin)"'" && $7 != "/bin/false") { print $1 " " $6 }'

Note there is a space between the two less than signs. The first less than is a redirect into the done.
The second less than is "process substitution" based on the grep - you may need to encapsulate the grep and what follows in parentheses for it work.

uncorrupt3d · 03-17-2020, 06:01 PM

Thanks! This seems to work, I'll bump the thread if this part of the function throws any more errors.

MadeInGermany · 03-18-2020, 02:31 AM

I would prefer pure shell builtins

Code:

# The following line is clumsy
wnologin=$(which nologin)

while IFS=":" read user pw uid gid gecos dir shell
do
  case $user in (halt|sync|shutdown) continue;; esac
  case $shell in ($wnologin|/bin/false) continue;; esac
  if [ ! -d "$dir" ]
  then
    echo "The home directory ($dir) of user $user does not exist."
  fi
done < /etc/passwd

uncorrupt3d · 03-18-2020, 07:28 AM

I tried both of the methods, and apparently it's still not working in my test lab vm. I'm beginning to wonder if it's an environment thing with CentOS 8... Because it worked at home.

boughtonp · 03-18-2020, 08:00 AM

The version MadeInGermany posted is definitely clearer.

It also makes it easy to add debug statements in, to test at which point things are differing from expectations...

pan64 · 03-18-2020, 08:26 AM

when you use pipe in shell scripts you will automatically force bash to use child processes.
So all your awk/grep/... will run in a new shell, but also your while loop will be put in a subshell.

Additionally post #2 looks incorrect without parenthesis:

Code:

while read -r user dir
do if [ ! -d "$dir" ] 
then echo "The home directory ($dir) of user $user does not exist."
fi
done < <(grep -E -v '^(halt|sync|shutdown)' /etc/passwd | awk -F: '($7 != "'"$which nologin)"'" && $7 != "/bin/false") { print $1 " " $6 }')

looks much better - and probably works
(this is just a comment: this grep and awk can be combined into one single awk)

shruggy · 03-18-2020, 08:53 AM

If you just copied the MadeInGermany's version without checking the syntax, be aware that there's a typo in the second case statement:

contine → continue

grail · 03-18-2020, 09:06 AM

I have to say that on any relatively new system, I feel as though there are plenty of issues to be had with some of the items in this script.
Specifically:

1. Many systems these days have /bin, /sbin and /usr/sbin as symbolic links to /usr/bin. Now depending on your PATH value, which will return the first place it finds any command.
So using 'which nologin' is erroneous at best, here's an example on my home machine:

Code:

$ echo $PATH
/home/grail/.rbenv/shims:/usr/sbin:/sbin:/bin:/usr/games:/home/grail/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/home/grail/bin:/home/grail/go/bin
# I have highlighted the first (s)bin entry in my PATH
$ which nologin
/usr/sbin/nologin
$ awk -F: '$7 ~ /nologin/{print $7}' /etc/passwd
/usr/bin/nologin
/usr/bin/nologin
/usr/bin/nologin
/usr/bin/nologin
/usr/bin/nologin
/usr/bin/nologin
/usr/bin/nologin
/usr/bin/nologin
/usr/bin/nologin
/usr/bin/nologin
/usr/bin/nologin
/usr/bin/nologin
/usr/bin/nologin
/sbin/nologin
/sbin/nologin
/sbin/nologin
/sbin/nologin
/sbin/nologin
/bin/nologin
/usr/bin/nologin
/sbin/nologin
/usr/bin/nologin
/usr/bin/nologin
/sbin/nologin
/usr/bin/nologin
/sbin/nologin
/sbin/nologin
/sbin/nologin
/sbin/nologin
/usr/bin/nologin
/usr/bin/nologin
/usr/bin/nologin

As you can see, I have a large number of users using nologin from a variety of places and one would think all should be excluded??

2. /bin/false -- see above as users could have /sbin/false, /usr/bin/false and /usr/sbin/false ... again, should you be limiting yourself?

3. grep -E -v '^(halt|sync|shutdown)' -- what if you have a user that starts with one of these words but has additional characters? (eg. sync-backup)

4. Try not to string multiple commands together when you are doubling up on abilities

As you have a solution in all bash, here's another alternative in awk:

Code:

awk -F: '$1 !~ /^(halt|sync|shutdown)$/ && $7 !~ /(nologin|false)/ && system("[[ -d "$6" ]]"){print "tada -- "$0}' /etc/passwd

michaelk · 03-18-2020, 09:13 AM

What do you mean by not work? What is your test lab VM?

I don't have CentOS 8 installed yet but the only difference I noticed with MadeInGermany's script as compared to CentOS 7 is which nologin returns /usr/sbin/nologin but /etc/passwd uses /sbin/nologin for a regular user.

By default which (without any options) returns the first match and with CentOS7 /usr/sbin is in the path before /sbin for a regular user. So yes it could be an environment thing.

I would not use a variable named dir since it is an actual command but that does not seem to break the script.

A bit to late grail beat me to the punch...

Kenhelm · 03-18-2020, 11:59 AM

If there is a function called 'read' in the script it will replace the shell builtin and could cause an infinite loop.

Code:

read () {
        # This is a function which always returns true
        return 0
        }

grep -E -v '^(halt|sync|shutdown)' /etc/passwd | awk -F: '($7 != "'"$which nologin)"'" && $7 != "/bin/false") { print $1 " " $6 }' | while read -r user dir; 
do if [ ! -d "$dir" ]; 
then echo "The home directory ($dir) of user $user does not exist.";
fi;
done;

The home directory () of user  does not exist.
The home directory () of user  does not exist.
The home directory () of user  does not exist.
The home directory () of user  does not exist.
The home directory () of user  does not exist.
The home directory () of user  does not exist.
The home directory () of user  does not exist.
....               # infinite loop

There is a missing parenthesis

Code:

"$which nologin)"
# should be
"$(which nologin)"

uncorrupt3d · 03-18-2020, 12:00 PM

My test lab vm is a fresh CentOS 8 server install, and my home CentOS 8 vm is the same exact thing. I just tried grail's solution and it didn't work either... I just get the same infinite loop.

It's important to stress that my code already works, even if it's not proper. It's just not working inside CentOS 8 when paired with other bash functions I have defined, which is the intended goal.

I'm throwing it into a function and calling it from another one, because that's what my script needs to do. I can run all this and have it work in the shell, or when I throw this portion of the bash script into it's own script it runs fine- but I cannot invoke it from my main script. That's what I'm trying to understand.

Thanks again for your help! This problem has stumped my coworkers too... Should I try it on a different distro like debian and see if it's just a problem with my code?

pan64 · 03-18-2020, 01:08 PM

this is not related to the distro, although it may depend on the version of some tools you use (for example bash).
I would suggest you:
1. add set -xv to see what's happening - and try to post some debug output. Probably helps
2. try shellcheck, it can be useful too.
3. also would be nice to prepare a sample script to demonstrate what works and what failed. If we could try it we could also repair it.

MadeInGermany · 03-18-2020, 01:21 PM

Here I avoid the problematic which

Code:

# for debugging uncomment the next line
#set -x

while IFS=":" read user pw uid gid gecos dir shell
do
  case $user in (halt|sync|shutdown) continue;; esac
  case $shell in (*/nologin|*/false) continue;; esac
  if [ -n "$dir" ] && [ ! -d "$dir" ]
  then
    echo "The home directory ($dir) of user $user does not exist."
  fi
done < /etc/passwd

As said before, for debugging use set -x or set -vx

astrogeek · 03-18-2020, 01:25 PM

Quote:

Originally Posted by pan64

3. also would be nice to prepare a sample script to demonstrate what works and what failed. If we could try it we could also repair it.

This, indeed! It would be very helpful to others, and yourself, to work up a simplest complete test case which demonstrates the problem so that others can reproduce it, better understand the problem and suggest a solution. Descriptions about the problem are not nearly as useful as an example which demonstrates the problem. Preparation of a simplest example often leads you to a better understanding and solution in the process - a valuable troubleshooting methid in itself.

I would suggest you review the Site FAQ for guidance in posting your questions and general forum usage. Especially, read the link in that page, How To Ask Questions The Smart Way. The more effort you put into understanding your problem and framing your questions, the better others can help!