LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Newbie (https://www.linuxquestions.org/questions/linux-newbie-8/)
-   -   Is getent a better option than parsing the /etc/passwd file through while loop (https://www.linuxquestions.org/questions/linux-newbie-8/is-getent-a-better-option-than-parsing-the-etc-passwd-file-through-while-loop-819723/)

vinaytp 07-14-2010 01:20 AM

Is getent a better option than parsing the /etc/passwd file through while loop
 
Dear All,

I am trying to fetch the user records like:

SHELL
COMMENTS
HOME_DIR
PrimaryGroup
Secondarygrouplist

from /etc/passwd file and display in a format. I have written following script

Code:

for __NAME__ in `awk -F: '{printf " "$1}' /etc/passwd`
do
USID=`getent passwd $__NAME__` | cut -d: -f3
COMMENTS=`getent passwd $__NAME__` | cut -d: -f5
HOME_DIR=`getent passwd $__NAME__` | cut -d: -f6
SHELL=`getent passwd $__NAME__` | cut -d: -f7
GROUP=$(groups $__NAME__| cut -d ':' -f 2 | cut -d ' ' -f 2)
secgrplist=$(groups $__NAME__ | sed 's/ /,/3' | cut -d ',' -f 2 | sed 's/ /,/g')
if [ "$password" == '!' ];then ENABLE="false";else ENABLE="true";fi;
RESULT=__NAME__:$__NAME__:__ENABLE__:$ENABLE:COMMENTS:$COMMENTS:USID:$USID:SHELL:$SHELL:HOME_DIR:$HOME_DIR:SECONDARYGROUP:$secgrplist:GROUP:$GROUP:__UID__:$__NAME__;
echo "RESULT_START $RESULT RESULT_END";

If there are around 1000 users. Does this script proves to be effecient ?

Is there any other better way to accomplish this?

raskin 07-14-2010 01:55 AM

What you do seems quite inefficient (and also you put ` wrong).

Code:

USID=`getent passwd $__NAME__` | cut -d: -f3
I guess entire right-hand side should be contained in "`"

Now, you iterate over all entries in /etc/passwd. I'd do it with
Code:

( while read; do
  USID="$(echo "$REPLY" | cut -d: -f3)"
  [..]
done ) < /etc/passwd


vinaytp 07-14-2010 02:15 AM

Quote:

Originally Posted by raskin (Post 4032598)
What you do seems quite inefficient (and also you put ` wrong).

Code:

USID=`getent passwd $__NAME__` | cut -d: -f3
I guess entire right-hand side should be contained in "`"

Many thanks for your response.

Sorry, I have pasted quite old code.

Code:

for __NAME__ in `awk -F: '{printf " "$1}' /etc/passwd`;do
USID=$(getent passwd $__NAME__ | cut -d: -f3)
COMMENTS=$(getent passwd $__NAME__ | cut -d: -f5)
HOME_DIR=$(getent passwd $__NAME__ | cut -d: -f6)
SHELL=$(getent passwd $__NAME__ | cut -d: -f7)
GROUP=$(groups $__NAME__| cut -d ':' -f2 | cut -d ' ' -f2)
secgrplist=$(groups $__NAME__ | groups $__NAME__ | sed 's/ /,/g' | cut -d, -f4-)
password=$(getent shadow $__NAME__ | cut -d: -f2)
if [ "$password" == '!!' ];then ENABLE="false";else ENABLE="true";fi;
RESULT=__NAME__:$__NAME__:__ENABLE__:$ENABLE:COMMENTS:$COMMENTS:USID:$USID:SHELL:$SHELL:HOME_DIR:$HOME_DIR:SECONDARYGROUP:$secgrplist:GROUP:$GROUP:__UID__:$__NAME__;
echo "RESULT_START $RESULT RESULT_END";done

Quote:

Now, you iterate over all entries in /etc/passwd. I'd do it with
Code:

( while read; do
  USID="$(echo "$REPLY" | cut -d: -f3)"
  [..]
done ) < /etc/passwd


For me this code looks pretty ineffecient as you have to traverse through all the entries of /etc/passwd in while loop.

But In my above pasted code I will get all usernames at once through awk -F: '{printf " "$1}' /etc/passwd

And I use getent to find the details of user which internally finds it through administrative databases like passwd and shadow. Instead of passing /etc/passwd through while loop.

raskin 07-14-2010 02:29 AM

In the beginning I would recommend also getting "groups" entry into a variable, and parse the line taken from variable then.

Let us now try to compare what actually happens.

I suggest reading each passwd line once, and parse it in memory. You say that retrieving one field a time is better. Unfortunately, what actually happens is that the first awk invocation will read entire /etc/passwd file. Also, getent will read entire /etc/passwd pefix until the needed line - and you retrieve the same line multiple times. Also your code spawns more processes.

By the way, on my machine the only reasonable way to get user's primary group is using passwd entry and then getting an entry from /etc/group file (here getent will fit nice). "groups" just prints all groups the user is in without obvious emphasis for primary group.

vinaytp 07-14-2010 02:36 AM

Thanks for your inputs raskin.

I have tried in a different way now.
mostly this could be faster...

Code:

awk -F: '{printf " RESULT_START __NAME__:"$1":USID:"$3":COMMENTS:"$5":HOME_DIR:"$6":SHELL:"$7" RESULT_END\n"}' /etc/passwd
This works pretty well, but now the problem is to incorporate primary and secondary group information into the above command.

I have to execute system() function of awk here to execute groups command.
Can you please help me with this ?

raskin 07-14-2010 02:40 AM

Well, I don't use awk much (and so I have never seen system() function).. Actually, is it just so time-critical code to optimize it beyond removing obvious problems?

vinaytp 07-14-2010 04:01 AM

Quote:

Originally Posted by raskin (Post 4032622)
Well, I don't use awk much (and so I have never seen system() function).. Actually, is it just so time-critical code to optimize it beyond removing obvious problems?

Yes, We need to reduce the time of execution as much as possible.

raskin 07-14-2010 04:12 AM

I guess spawning a couple of processes per line is not too fast, too. I guess writing it all in a compiled language is what you need if it is really time-critical. Also, you'd better parse the files linearly on load (you need all entries anyway) and put results in a data structure more suitable for search than a linear list.

grail 07-14-2010 05:50 AM

Well here are a couple of alternatives:

bash:
Code:

#!/bin/bash

while IFS=":" read user password uid gid comment home shell
do
        grps=($(groups $user))

        [[ -n ${grps[3]} ]] && second=$(echo "${grps[@]:3}" | sed 's/ /,/g')

        echo "RESULT_START __NAME__:$user:USID:$uid:COMMENTS:$comments:HOME_DIR:$home:SHELL:$shell:PRIMARY_GRP:${grps[2]}:SECONDARY_GRPS:${second[@]// /,}:RESULT_END"
        second=
done<$1

awk:
Code:

#!/usr/bin/awk -f

BEGIN{ FS=":"}

{
        while((getline grps < "/etc/group")>0)
        {
                if(grps ~ "."$1)
                        if(sec ~ /./)
                                sec=sec","gensub(":.*","","g",grps)
                        else
                                sec=gensub(":.*","","g",grps)

                if(grps ~ $4)
                        prim=gensub(":.*","","g",grps)
        }

        close("/etc/group")

        print "RESULT_START __NAME__:"$1":USID:"$3":COMMENTS:"$5":HOME_DIR:"$6":SHELL:"$7":PRIMARY_GRP:"prim":SECONDARY_GRPS:"sec":RESULT_END"
        prim=""
        sec=""
}

Both called as follows:
Code:

./script /etc/passwd

vinaytp 07-14-2010 06:02 AM

Assigning variables in awk
 
Thanks for your reply grail..

Will try this once I get back to my linux environment

vinaytp 07-14-2010 07:00 AM

Thanks a lot grail...Both the scripts are working great...can you suggest me any good awk turotial.

I am trying to understand your script..

Thanks again to both grail and raskin...

grail 07-14-2010 07:24 AM

I normally just follow this -> http://www.gnu.org/manual/gawk/html_node/index.html

If you have any questions I am happy to help :)

vinaytp 07-14-2010 07:31 AM

Quote:

Originally Posted by grail (Post 4032781)
I normally just follow this -> http://www.gnu.org/manual/gawk/html_node/index.html

If you have any questions I am happy to help :)

Thanks for the tutorial.
Yes, I have doubts in 2nd script

Code:

if(sec ~ /./)
I could not understand this line, variable sec will be empty at the beginning. How can you compare it with "." ?

Code:

if(grps ~ "."$1)
similarly in the above line will it compare grps with .concatenated with first field ?

grail 07-14-2010 07:54 AM

Yes I can see how that is a little confusing, so here we go:

sec variable - this is to store the secondary groups that a user "may" be a part of, remembering that you do not have to have more than your primary group.

so to explain:
Quote:

if(sec ~ /./)
This is simply a test to see if sec has anything stored in it, ie at least one character ... hence the .

grps variable - the variable stores each line one at a time from the file /etc/group (this part is easy)

so to explain:
Quote:

if(grps ~ "."$1)
Here $1 is the name of the user from the file /etc/passwd so to test if a user is part of any other group the name must appear on the line but not at the start of the line, therefore there should be at least one character (actually many but testing for one is sufficient) prior to the users name.

As an example, my username is grail so the following line shows me as a member of the following group:
Code:

dialout:x:20:grail
So our test would now see that there is a ":" prior to name grail so this is a secondary group.

However the following group is not secondary:
Code:

grail:x:1000:
As there are no characters prior to name it is not a secondary group

As of typing this I have realised the logic is slightly floored because if you put all users into the primary group of 'users' this will show them as both a primary and secondary of the group 'users' ... :banghead:

Oh well ... something to play with :)

vinaytp 07-14-2010 09:32 AM

Thanks grail...You explained it very well..

I learnt a lot about awk today from you...I have written the script in different way..

Please shed your views on this..

For me it looks very optimized as it does not require any loop execution..

Code:


#!/usr/bin/awk -f
BEGIN { FS=":" }
{
"groups "$1"| cut -d ':' -f2 | cut -d ' ' -f2" | getline GROUP
"groups "$1"| sed 's/ /,/g' | cut -d, -f4-" | getline secgrplist
print "RESULT_START __NAME__:"$1":USID:"$3":COMMENTS:"$5":HOME_DIR:"$6":SHELL:"$7":PRIMARY_GRP:"GROUP":SECONDARY_GRPS:"secgrplist":RESULT_END"
}

Thanks again grail...


All times are GMT -5. The time now is 03:55 AM.