LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 07-14-2010, 01:20 AM   #1
vinaytp
Member
 
Registered: Apr 2009
Location: Bengaluru, India
Distribution: RHEL 5.4, 6.0, Ubuntu 10.04
Posts: 704

Rep: Reputation: 55
Is getent a better option than parsing the /etc/passwd file through while loop


Dear All,

I am trying to fetch the user records like:

SHELL
COMMENTS
HOME_DIR
PrimaryGroup
Secondarygrouplist

from /etc/passwd file and display in a format. I have written following script

Code:
for __NAME__ in `awk -F: '{printf " "$1}' /etc/passwd`
do
USID=`getent passwd $__NAME__` | cut -d: -f3
COMMENTS=`getent passwd $__NAME__` | cut -d: -f5
HOME_DIR=`getent passwd $__NAME__` | cut -d: -f6
SHELL=`getent passwd $__NAME__` | cut -d: -f7
GROUP=$(groups $__NAME__| cut -d ':' -f 2 | cut -d ' ' -f 2)
secgrplist=$(groups $__NAME__ | sed 's/ /,/3' | cut -d ',' -f 2 | sed 's/ /,/g')
if [ "$password" == '!' ];then ENABLE="false";else ENABLE="true";fi;
RESULT=__NAME__:$__NAME__:__ENABLE__:$ENABLE:COMMENTS:$COMMENTS:USID:$USID:SHELL:$SHELL:HOME_DIR:$HOME_DIR:SECONDARYGROUP:$secgrplist:GROUP:$GROUP:__UID__:$__NAME__;
echo "RESULT_START $RESULT RESULT_END";
If there are around 1000 users. Does this script proves to be effecient ?

Is there any other better way to accomplish this?
 
Old 07-14-2010, 01:55 AM   #2
raskin
Senior Member
 
Registered: Sep 2005
Location: France
Distribution: approximately NixOS (http://nixos.org)
Posts: 1,900

Rep: Reputation: 68
What you do seems quite inefficient (and also you put ` wrong).

Code:
USID=`getent passwd $__NAME__` | cut -d: -f3
I guess entire right-hand side should be contained in "`"

Now, you iterate over all entries in /etc/passwd. I'd do it with
Code:
( while read; do
  USID="$(echo "$REPLY" | cut -d: -f3)"
  [..]
done ) < /etc/passwd
 
Old 07-14-2010, 02:15 AM   #3
vinaytp
Member
 
Registered: Apr 2009
Location: Bengaluru, India
Distribution: RHEL 5.4, 6.0, Ubuntu 10.04
Posts: 704

Original Poster
Rep: Reputation: 55
Quote:
Originally Posted by raskin View Post
What you do seems quite inefficient (and also you put ` wrong).

Code:
USID=`getent passwd $__NAME__` | cut -d: -f3
I guess entire right-hand side should be contained in "`"
Many thanks for your response.

Sorry, I have pasted quite old code.

Code:
for __NAME__ in `awk -F: '{printf " "$1}' /etc/passwd`;do
USID=$(getent passwd $__NAME__ | cut -d: -f3)
COMMENTS=$(getent passwd $__NAME__ | cut -d: -f5)
HOME_DIR=$(getent passwd $__NAME__ | cut -d: -f6)
SHELL=$(getent passwd $__NAME__ | cut -d: -f7)
GROUP=$(groups $__NAME__| cut -d ':' -f2 | cut -d ' ' -f2)
secgrplist=$(groups $__NAME__ | groups $__NAME__ | sed 's/ /,/g' | cut -d, -f4-)
password=$(getent shadow $__NAME__ | cut -d: -f2)
if [ "$password" == '!!' ];then ENABLE="false";else ENABLE="true";fi;
RESULT=__NAME__:$__NAME__:__ENABLE__:$ENABLE:COMMENTS:$COMMENTS:USID:$USID:SHELL:$SHELL:HOME_DIR:$HOME_DIR:SECONDARYGROUP:$secgrplist:GROUP:$GROUP:__UID__:$__NAME__;
echo "RESULT_START $RESULT RESULT_END";done
Quote:
Now, you iterate over all entries in /etc/passwd. I'd do it with
Code:
( while read; do
  USID="$(echo "$REPLY" | cut -d: -f3)"
  [..]
done ) < /etc/passwd
For me this code looks pretty ineffecient as you have to traverse through all the entries of /etc/passwd in while loop.

But In my above pasted code I will get all usernames at once through awk -F: '{printf " "$1}' /etc/passwd

And I use getent to find the details of user which internally finds it through administrative databases like passwd and shadow. Instead of passing /etc/passwd through while loop.
 
Old 07-14-2010, 02:29 AM   #4
raskin
Senior Member
 
Registered: Sep 2005
Location: France
Distribution: approximately NixOS (http://nixos.org)
Posts: 1,900

Rep: Reputation: 68
In the beginning I would recommend also getting "groups" entry into a variable, and parse the line taken from variable then.

Let us now try to compare what actually happens.

I suggest reading each passwd line once, and parse it in memory. You say that retrieving one field a time is better. Unfortunately, what actually happens is that the first awk invocation will read entire /etc/passwd file. Also, getent will read entire /etc/passwd pefix until the needed line - and you retrieve the same line multiple times. Also your code spawns more processes.

By the way, on my machine the only reasonable way to get user's primary group is using passwd entry and then getting an entry from /etc/group file (here getent will fit nice). "groups" just prints all groups the user is in without obvious emphasis for primary group.
 
Old 07-14-2010, 02:36 AM   #5
vinaytp
Member
 
Registered: Apr 2009
Location: Bengaluru, India
Distribution: RHEL 5.4, 6.0, Ubuntu 10.04
Posts: 704

Original Poster
Rep: Reputation: 55
Thanks for your inputs raskin.

I have tried in a different way now.
mostly this could be faster...

Code:
awk -F: '{printf " RESULT_START __NAME__:"$1":USID:"$3":COMMENTS:"$5":HOME_DIR:"$6":SHELL:"$7" RESULT_END\n"}' /etc/passwd
This works pretty well, but now the problem is to incorporate primary and secondary group information into the above command.

I have to execute system() function of awk here to execute groups command.
Can you please help me with this ?
 
Old 07-14-2010, 02:40 AM   #6
raskin
Senior Member
 
Registered: Sep 2005
Location: France
Distribution: approximately NixOS (http://nixos.org)
Posts: 1,900

Rep: Reputation: 68
Well, I don't use awk much (and so I have never seen system() function).. Actually, is it just so time-critical code to optimize it beyond removing obvious problems?
 
Old 07-14-2010, 04:01 AM   #7
vinaytp
Member
 
Registered: Apr 2009
Location: Bengaluru, India
Distribution: RHEL 5.4, 6.0, Ubuntu 10.04
Posts: 704

Original Poster
Rep: Reputation: 55
Quote:
Originally Posted by raskin View Post
Well, I don't use awk much (and so I have never seen system() function).. Actually, is it just so time-critical code to optimize it beyond removing obvious problems?
Yes, We need to reduce the time of execution as much as possible.
 
Old 07-14-2010, 04:12 AM   #8
raskin
Senior Member
 
Registered: Sep 2005
Location: France
Distribution: approximately NixOS (http://nixos.org)
Posts: 1,900

Rep: Reputation: 68
I guess spawning a couple of processes per line is not too fast, too. I guess writing it all in a compiled language is what you need if it is really time-critical. Also, you'd better parse the files linearly on load (you need all entries anyway) and put results in a data structure more suitable for search than a linear list.
 
Old 07-14-2010, 05:50 AM   #9
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,507

Rep: Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890
Well here are a couple of alternatives:

bash:
Code:
#!/bin/bash

while IFS=":" read user password uid gid comment home shell 
do
	grps=($(groups $user))

	[[ -n ${grps[3]} ]] && second=$(echo "${grps[@]:3}" | sed 's/ /,/g')

	echo "RESULT_START __NAME__:$user:USID:$uid:COMMENTS:$comments:HOME_DIR:$home:SHELL:$shell:PRIMARY_GRP:${grps[2]}:SECONDARY_GRPS:${second[@]// /,}:RESULT_END"
	second=
done<$1
awk:
Code:
#!/usr/bin/awk -f

BEGIN{ FS=":"}

{
	while((getline grps < "/etc/group")>0)
	{
		if(grps ~ "."$1)
			if(sec ~ /./)
				sec=sec","gensub(":.*","","g",grps)
			else 
				sec=gensub(":.*","","g",grps)

		if(grps ~ $4)
			prim=gensub(":.*","","g",grps)
	}

	close("/etc/group")

	print "RESULT_START __NAME__:"$1":USID:"$3":COMMENTS:"$5":HOME_DIR:"$6":SHELL:"$7":PRIMARY_GRP:"prim":SECONDARY_GRPS:"sec":RESULT_END"
	prim=""
	sec=""
}
Both called as follows:
Code:
./script /etc/passwd
 
Old 07-14-2010, 06:02 AM   #10
vinaytp
Member
 
Registered: Apr 2009
Location: Bengaluru, India
Distribution: RHEL 5.4, 6.0, Ubuntu 10.04
Posts: 704

Original Poster
Rep: Reputation: 55
Assigning variables in awk

Thanks for your reply grail..

Will try this once I get back to my linux environment

Last edited by vinaytp; 07-14-2010 at 06:04 AM.
 
Old 07-14-2010, 07:00 AM   #11
vinaytp
Member
 
Registered: Apr 2009
Location: Bengaluru, India
Distribution: RHEL 5.4, 6.0, Ubuntu 10.04
Posts: 704

Original Poster
Rep: Reputation: 55
Thanks a lot grail...Both the scripts are working great...can you suggest me any good awk turotial.

I am trying to understand your script..

Thanks again to both grail and raskin...
 
Old 07-14-2010, 07:24 AM   #12
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,507

Rep: Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890
I normally just follow this -> http://www.gnu.org/manual/gawk/html_node/index.html

If you have any questions I am happy to help
 
Old 07-14-2010, 07:31 AM   #13
vinaytp
Member
 
Registered: Apr 2009
Location: Bengaluru, India
Distribution: RHEL 5.4, 6.0, Ubuntu 10.04
Posts: 704

Original Poster
Rep: Reputation: 55
Quote:
Originally Posted by grail View Post
I normally just follow this -> http://www.gnu.org/manual/gawk/html_node/index.html

If you have any questions I am happy to help
Thanks for the tutorial.
Yes, I have doubts in 2nd script

Code:
if(sec ~ /./)
I could not understand this line, variable sec will be empty at the beginning. How can you compare it with "." ?

Code:
if(grps ~ "."$1)
similarly in the above line will it compare grps with .concatenated with first field ?

Last edited by vinaytp; 07-14-2010 at 07:33 AM.
 
Old 07-14-2010, 07:54 AM   #14
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,507

Rep: Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890
Yes I can see how that is a little confusing, so here we go:

sec variable - this is to store the secondary groups that a user "may" be a part of, remembering that you do not have to have more than your primary group.

so to explain:
Quote:
if(sec ~ /./)
This is simply a test to see if sec has anything stored in it, ie at least one character ... hence the .

grps variable - the variable stores each line one at a time from the file /etc/group (this part is easy)

so to explain:
Quote:
if(grps ~ "."$1)
Here $1 is the name of the user from the file /etc/passwd so to test if a user is part of any other group the name must appear on the line but not at the start of the line, therefore there should be at least one character (actually many but testing for one is sufficient) prior to the users name.

As an example, my username is grail so the following line shows me as a member of the following group:
Code:
dialout:x:20:grail
So our test would now see that there is a ":" prior to name grail so this is a secondary group.

However the following group is not secondary:
Code:
grail:x:1000:
As there are no characters prior to name it is not a secondary group

As of typing this I have realised the logic is slightly floored because if you put all users into the primary group of 'users' this will show them as both a primary and secondary of the group 'users' ...

Oh well ... something to play with
 
Old 07-14-2010, 09:32 AM   #15
vinaytp
Member
 
Registered: Apr 2009
Location: Bengaluru, India
Distribution: RHEL 5.4, 6.0, Ubuntu 10.04
Posts: 704

Original Poster
Rep: Reputation: 55
Thanks grail...You explained it very well..

I learnt a lot about awk today from you...I have written the script in different way..

Please shed your views on this..

For me it looks very optimized as it does not require any loop execution..

Code:
#!/usr/bin/awk -f
BEGIN { FS=":" }
{
"groups "$1"| cut -d ':' -f2 | cut -d ' ' -f2" | getline GROUP
"groups "$1"| sed 's/ /,/g' | cut -d, -f4-" | getline secgrplist
print "RESULT_START __NAME__:"$1":USID:"$3":COMMENTS:"$5":HOME_DIR:"$6":SHELL:"$7":PRIMARY_GRP:"GROUP":SECONDARY_GRPS:"secgrplist":RESULT_END"
}
Thanks again grail...
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
mount.cifs credentials=/path-to-passwd-file not parsing xxx_anuj_xxx Linux - Server 1 11-10-2009 03:44 AM
getent passwd problems carlos23 Linux - Server 1 01-24-2009 01:17 AM
getent passwd only pulls local info - getent group works? epoh Linux - Server 2 03-14-2008 07:56 AM
nss_ldap not working (getent passwd) WindowBreaker Slackware 2 06-27-2006 02:19 AM
getent passwd and wibinfo -u not working bahadur Linux - Networking 0 06-14-2004 12:20 AM


All times are GMT -5. The time now is 02:24 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration