[SOLVED] Is getent a better option than parsing the /etc/passwd file through while loop

vinaytp · 07-14-2010, 01:20 AM

Dear All,

I am trying to fetch the user records like:

SHELL
COMMENTS
HOME_DIR
PrimaryGroup
Secondarygrouplist

from /etc/passwd file and display in a format. I have written following script

Code:

for __NAME__ in `awk -F: '{printf " "$1}' /etc/passwd`
do
USID=`getent passwd $__NAME__` | cut -d: -f3
COMMENTS=`getent passwd $__NAME__` | cut -d: -f5
HOME_DIR=`getent passwd $__NAME__` | cut -d: -f6
SHELL=`getent passwd $__NAME__` | cut -d: -f7
GROUP=$(groups $__NAME__| cut -d ':' -f 2 | cut -d ' ' -f 2)
secgrplist=$(groups $__NAME__ | sed 's/ /,/3' | cut -d ',' -f 2 | sed 's/ /,/g')
if [ "$password" == '!' ];then ENABLE="false";else ENABLE="true";fi;
RESULT=__NAME__:$__NAME__:__ENABLE__:$ENABLE:COMMENTS:$COMMENTS:USID:$USID:SHELL:$SHELL:HOME_DIR:$HOME_DIR:SECONDARYGROUP:$secgrplist:GROUP:$GROUP:__UID__:$__NAME__;
echo "RESULT_START $RESULT RESULT_END";

If there are around 1000 users. Does this script proves to be effecient ?

Is there any other better way to accomplish this?

raskin · 07-14-2010, 01:55 AM

What you do seems quite inefficient (and also you put ` wrong).

Code:

USID=`getent passwd $__NAME__` | cut -d: -f3

I guess entire right-hand side should be contained in "`"

Now, you iterate over all entries in /etc/passwd. I'd do it with

Code:

( while read; do
  USID="$(echo "$REPLY" | cut -d: -f3)"
  [..]
done ) < /etc/passwd

vinaytp · 07-14-2010, 02:15 AM

Quote:

Originally Posted by raskin

What you do seems quite inefficient (and also you put ` wrong).

Code:

USID=`getent passwd $__NAME__` | cut -d: -f3

I guess entire right-hand side should be contained in "`"

Many thanks for your response.

Sorry, I have pasted quite old code.

Code:

for __NAME__ in `awk -F: '{printf " "$1}' /etc/passwd`;do
USID=$(getent passwd $__NAME__ | cut -d: -f3)
COMMENTS=$(getent passwd $__NAME__ | cut -d: -f5)
HOME_DIR=$(getent passwd $__NAME__ | cut -d: -f6)
SHELL=$(getent passwd $__NAME__ | cut -d: -f7)
GROUP=$(groups $__NAME__| cut -d ':' -f2 | cut -d ' ' -f2)
secgrplist=$(groups $__NAME__ | groups $__NAME__ | sed 's/ /,/g' | cut -d, -f4-)
password=$(getent shadow $__NAME__ | cut -d: -f2)
if [ "$password" == '!!' ];then ENABLE="false";else ENABLE="true";fi;
RESULT=__NAME__:$__NAME__:__ENABLE__:$ENABLE:COMMENTS:$COMMENTS:USID:$USID:SHELL:$SHELL:HOME_DIR:$HOME_DIR:SECONDARYGROUP:$secgrplist:GROUP:$GROUP:__UID__:$__NAME__;
echo "RESULT_START $RESULT RESULT_END";done

Quote:

Now, you iterate over all entries in /etc/passwd. I'd do it with

Code:

( while read; do
  USID="$(echo "$REPLY" | cut -d: -f3)"
  [..]
done ) < /etc/passwd

For me this code looks pretty ineffecient as you have to traverse through all the entries of /etc/passwd in while loop.

But In my above pasted code I will get all usernames at once through awk -F: '{printf " "$1}' /etc/passwd

And I use getent to find the details of user which internally finds it through administrative databases like passwd and shadow. Instead of passing /etc/passwd through while loop.

raskin · 07-14-2010, 02:29 AM

In the beginning I would recommend also getting "groups" entry into a variable, and parse the line taken from variable then.

Let us now try to compare what actually happens.

I suggest reading each passwd line once, and parse it in memory. You say that retrieving one field a time is better. Unfortunately, what actually happens is that the first awk invocation will read entire /etc/passwd file. Also, getent will read entire /etc/passwd pefix until the needed line - and you retrieve the same line multiple times. Also your code spawns more processes.

By the way, on my machine the only reasonable way to get user's primary group is using passwd entry and then getting an entry from /etc/group file (here getent will fit nice). "groups" just prints all groups the user is in without obvious emphasis for primary group.

vinaytp · 07-14-2010, 02:36 AM

Thanks for your inputs raskin.

I have tried in a different way now.
mostly this could be faster...

Code:

awk -F: '{printf " RESULT_START __NAME__:"$1":USID:"$3":COMMENTS:"$5":HOME_DIR:"$6":SHELL:"$7" RESULT_END\n"}' /etc/passwd

This works pretty well, but now the problem is to incorporate primary and secondary group information into the above command.

I have to execute system() function of awk here to execute groups command.
Can you please help me with this ?

raskin · 07-14-2010, 02:40 AM

Well, I don't use awk much (and so I have never seen system() function).. Actually, is it just so time-critical code to optimize it beyond removing obvious problems?

vinaytp · 07-14-2010, 04:01 AM

Quote:

Originally Posted by raskin

Well, I don't use awk much (and so I have never seen system() function).. Actually, is it just so time-critical code to optimize it beyond removing obvious problems?

Yes, We need to reduce the time of execution as much as possible.

raskin · 07-14-2010, 04:12 AM

I guess spawning a couple of processes per line is not too fast, too. I guess writing it all in a compiled language is what you need if it is really time-critical. Also, you'd better parse the files linearly on load (you need all entries anyway) and put results in a data structure more suitable for search than a linear list.

grail · 07-14-2010, 05:50 AM

Well here are a couple of alternatives:

bash:

Code:

#!/bin/bash

while IFS=":" read user password uid gid comment home shell 
do
	grps=($(groups $user))

	[[ -n ${grps[3]} ]] && second=$(echo "${grps[@]:3}" | sed 's/ /,/g')

	echo "RESULT_START __NAME__:$user:USID:$uid:COMMENTS:$comments:HOME_DIR:$home:SHELL:$shell:PRIMARY_GRP:${grps[2]}:SECONDARY_GRPS:${second[@]// /,}:RESULT_END"
	second=
done<$1

awk:

Code:

#!/usr/bin/awk -f

BEGIN{ FS=":"}

{
	while((getline grps < "/etc/group")>0)
	{
		if(grps ~ "."$1)
			if(sec ~ /./)
				sec=sec","gensub(":.*","","g",grps)
			else 
				sec=gensub(":.*","","g",grps)

		if(grps ~ $4)
			prim=gensub(":.*","","g",grps)
	}

	close("/etc/group")

	print "RESULT_START __NAME__:"$1":USID:"$3":COMMENTS:"$5":HOME_DIR:"$6":SHELL:"$7":PRIMARY_GRP:"prim":SECONDARY_GRPS:"sec":RESULT_END"
	prim=""
	sec=""
}

Both called as follows:

Code:

./script /etc/passwd

vinaytp · 07-14-2010, 06:02 AM

Thanks for your reply grail..

Will try this once I get back to my linux environment

vinaytp · 07-14-2010, 07:00 AM

Thanks a lot grail...Both the scripts are working great...can you suggest me any good awk turotial.

I am trying to understand your script..

Thanks again to both grail and raskin...

grail · 07-14-2010, 07:24 AM

I normally just follow this -> http://www.gnu.org/manual/gawk/html_node/index.html

If you have any questions I am happy to help

vinaytp · 07-14-2010, 07:31 AM

Quote:

Originally Posted by grail

I normally just follow this -> http://www.gnu.org/manual/gawk/html_node/index.html

If you have any questions I am happy to help

Thanks for the tutorial.
Yes, I have doubts in 2nd script

Code:

if(sec ~ /./)

I could not understand this line, variable sec will be empty at the beginning. How can you compare it with "." ?

Code:

if(grps ~ "."$1)

similarly in the above line will it compare grps with .concatenated with first field ?

grail · 07-14-2010, 07:54 AM

Yes I can see how that is a little confusing, so here we go:

sec variable - this is to store the secondary groups that a user "may" be a part of, remembering that you do not have to have more than your primary group.

so to explain:

Quote:

if(sec ~ /./)

This is simply a test to see if sec has anything stored in it, ie at least one character ... hence the .

grps variable - the variable stores each line one at a time from the file /etc/group (this part is easy)

so to explain:

Quote:

if(grps ~ "."$1)

Here $1 is the name of the user from the file /etc/passwd so to test if a user is part of any other group the name must appear on the line but not at the start of the line, therefore there should be at least one character (actually many but testing for one is sufficient) prior to the users name.

As an example, my username is grail so the following line shows me as a member of the following group:

Code:

dialout:x:20:grail

So our test would now see that there is a ":" prior to name grail so this is a secondary group.

However the following group is not secondary:

Code:

grail:x:1000:

As there are no characters prior to name it is not a secondary group

As of typing this I have realised the logic is slightly floored because if you put all users into the primary group of 'users' this will show them as both a primary and secondary of the group 'users' ...

Oh well ... something to play with

vinaytp · 07-14-2010, 09:32 AM

Thanks grail...You explained it very well..

I learnt a lot about awk today from you...I have written the script in different way..

Please shed your views on this..

For me it looks very optimized as it does not require any loop execution..

Code:

#!/usr/bin/awk -f
BEGIN { FS=":" }
{
"groups "$1"| cut -d ':' -f2 | cut -d ' ' -f2" | getline GROUP
"groups "$1"| sed 's/ /,/g' | cut -d, -f4-" | getline secgrplist
print "RESULT_START __NAME__:"$1":USID:"$3":COMMENTS:"$5":HOME_DIR:"$6":SHELL:"$7":PRIMARY_GRP:"GROUP":SECONDARY_GRPS:"secgrplist":RESULT_END"
}

Thanks again grail...