LinuxQuestions.org - [SOLVED] Command to merge linewise from 2 files

- Linux - Newbie (https://www.linuxquestions.org/questions/linux-newbie-8/)

- - Command to merge linewise from 2 files (https://www.linuxquestions.org/questions/linux-newbie-8/command-to-merge-linewise-from-2-files-915156/)

mandyapenguin

11-23-2011 01:12 PM

Command to merge linewise from 2 files

Hi..Everybody,
I am newbie in shell script, I want to know when the user password will expire. for that I made a small shell script like below

Quote:

It will create users.txt and expire.txt files and format will be like below

Quote:

cat users.txt
user1
user2
user3
cat expire.txt
Dec 23, 2011
Oct 25, 2012
May 31, 2011

Now I need a command in the same shell script which will create a newfile and arrange the format like below

Quote:

user1 Dec 23, 2011
user2 Oct 25, 2012
user3 May 31, 2011

So please help me how it can be done?

David the H.

11-23-2011 03:06 PM

First things first. We have GOT to fix up what you already have.

Code:

users=$(grep /home /etc/passwd | grep -i "sh" | cut -d: -f 1 >> users.txt)

There are three main problems here.

1) It unnecessarily uses multiple tools to do what one instance of the proper tool can do. awk is designed precisely for doing this kind of job.

2) You're sending all the output of the commands into users.txt, which means that the users variable gets nothing.

3) Even if we fix #2, multiple outputs should not be stored in a single variable. This is a job for an array.

Code:

users=( $( awk -F':' '/\/home.*sh/ { print $1 }' /etc/passwd ) )

There are probably also better ways to get a list of user names, but we won't worry about that now. Don't worry about exporting it to a text file yet either. We'll take care of that later.

Code:

expire_date=$(grep /home /etc/passwd | grep -i "sh" | cut -d: -f 1 | while read user; do chage -l $user | sed -n '/Last/,+2p' | grep -v Last | grep -v inactive | cut -d: -f 2 >> expire.txt; done)

Holy freaking cow. Congratulations. This has got to be the most convoluted single command I've ever seen. :hattip: ;) Why do you go to the trouble to use sed to grab three lines, only to use grep to remove two of them again? All you really want is the "Password expires" line, don't you? So why not just grab that line directly?

Anyway, if you have the user list properly stored in an array, then it becomes trivial to get the date you want.

Code:

for i in "${!users[@]}"; do



    expire_date[i]=$( chage -l "${users[i]}" | grep '^Password expires' )

    expire_date[i]=${expire_date[i]##*:}



done

I could've used awk again instead of grep, but I decided to go with parameter substitution instead. No need for overkill.

Now you have matching array indexes for name and date, and can easily echo them to text files individually or as a whole.

Code:

for i in "${!users[@]}" ; do



        echo "${users[i]}" >> users.txt

        echo "${expire_date[i]}" >>expire.txt

        echo "${users[i]} ${expire_date[i]}" >>list.txt



done

This can even be done in the same loop that sets the date array, of course.

Check out the BashGuide for a good intro to scripting. You'll want to read the section on arrays in particular:

http://mywiki.wooledge.org/BashGuide

Juako

11-23-2011 03:07 PM

You could do both in one pass,

Code:

while read user; do echo "user: $user, expires: $(chage -l $user | sed -rn '2 s/^.*: (.*)$/\1/p')"; done< <(sed -rn 's/^([^:]*):.*$/\1/gp' /etc/passwd)

The command starts at the end:

Code:

< <(sed -rn 's/^([^:]*):.*$/\1/gp' /etc/passwd)

Filters the names and sends the output to the while ... done loop.

Each while...loop round reads one name, and echoes the name followed by a command substitution where 1) chage -l is called and 2) sed filters the last part after the : on the second line (the "expires" line).

Aka: no need for intermediate files.

mandyapenguin

11-23-2011 09:10 PM

know the user's expiration date

Quote:

#!/bin/bash
users=( $( awk -F':' '/\/home.*sh/ { print $1 }' /etc/passwd ) )
for i in "${!users[@]}"; do
expire_date[i]=$( chage -l "${users[i]}" | grep '^Password expires' )
expire_date[i]=${expire_date[i]##*:}
done
for i in "${!users[@]}" ; do
echo "${users[i]}" >> users.txt
echo "${expire_date[i]}" >> expire.txt
echo "${users[i]} ${expire_date[i]}" >> list.txt
done

Excellent, Thank you very much master. You saved me my lot of time in struggling to make the format in new file that I had requested like below.

Quote:

cat list.txt
user1 Dec 23, 2011
user2 Oct 25, 2012
user3 May 31, 2011

Now I need to segregate only those users which are already expired and going to expire which user expire date is less than 15 days from today.
Hope you definitely help me in this.
once again thank you very much master.

David the H.

11-24-2011 12:55 PM

Glad to help out.

Please use [code][/code] tags around your code, to preserve formatting and to improve readability. Do NOT use quote tags, which don't protect whitespace.

Your new requirement is a bit more of a challenge, but not too hard, as long as you have the gnu version of date (standard on almost all Linux distros*). With it, you can convert the dates to epoch format (seconds passed since Jan 1, 1970). This allows you to do simple numeric comparisons between different times.

Code:

now=$( date +%s )

soon=$( date -d "15 days " +%s )



for i in "${!expire_date[@]}" ; do



    userdate=$( date -d "${expire_date[i]}" +%s 2>/dev/null )





    if [[ ${expire_date[i]} == "never" ]]; then



          echo "Password for ${user[i]} is set to never expire"



    elif (( userdate < now )); then



          echo "Password for ${user[i]} has expired"



    elif (( userdate <= soon )); then



          diff=$(( ( userdate - now ) / 60 / 60 / 24 ))

          echo "Password for ${user[i]} will expire in $diff days"



    else



          echo "Password for ${user[i]} will not expire soon"



    fi



done

Note that date's -d option can handle many "natural English" style entries, but it's not perfect. It's possible that there could be input strings that it can't understand. My current username is set to "never", for example, which is why I had to set date to discard errors (with 2>/dev/null), and included a specific test for it. If there are other possible values, make sure the script can handle them as well.

By the way, to convert an epoch number back to another format, you have to prefix it with "@"

Code:

datestring='1322159922'

date -d "@$datestring" "+%b %d, %Y"

*The date implementations on other unixes generally don't have -d, so if you aren't on Linux you'll probably have to find some other tool for converting arbitrary dates, such as perl.

salemhouda

11-25-2011 01:23 AM

Try this command

Code:

$paste users.txt expire.txt > merged.txt

mandyapenguin

11-26-2011 09:59 PM

Code:

while read user; do echo "user: $user, expires: $(chage -l $user | sed -rn '2 s/^.*: (.*)$/\1/p')"; done< <(sed -rn 's/^([^:]*):.*$/\1/gp' /etc/passwd)

Super, Excellent master.

Code:

#!/bin/bash

while read user; do echo "user: $user, expires: $(chage -l $user | sed -rn '2 s/^.*: (.*)$/\1/p')"; done< <(sed -rn 's/^([^:]*):.*$/\1/gp' /etc/passwd) | grep -v never > list.txt

This is what I really expected.

Code:

bash pe

echo $?

0

cat list.txt

user: kumar, expires: Dec 23, 2011

user: roopa, expires: Oct 25, 2012

user: user1, expires: May 31, 2011

user: user2, expires: Dec 24, 2011

Code:

chage -l roopa | sed -rn '2 s/^.*: (.*)$/\1/p'

Oct 25, 2012

Code:

sed -rn 's/^([^:]*):.*$/\1/gp' /etc/passwd

root

daemon

bin

sys

sync and so on

Could you please explain for these two
sed -rn '2 s/^.*: (.*)$/\1/p'
sed -rn 's/^([^:]*):.*$/\1/gp' /etc/passwd

Or could you please send me some url links by which I can be also like you in shell scripts.

mandyapenguin

11-26-2011 10:50 PM

get mail about password expiration

H..David master.
Sorry for the late response. I could not see your kind reply since I was at out of station.
Please excuse me. I am spoiling your valuable time with this thread. You are really genius.

Code:

bash pass_expire

Password for kumar will not expire soon

Password for roopa has expired

Password for user1 will not expire soon

Password for user2 will expire in 3 days

Code:

date

Sun Nov 27 10:21:56 IST 2011

Code:

chage -l user2

Last password change                                        : Nov 01, 2011

Password expires                                        : Dec 01, 2011

Password inactive                                        : never

Account expires                                                : never

Minimum number of days between password change                : 0

Maximum number of days between password change                : 30

Number of days of warning before password expires        : 7

The last one thing I would like to expect from you. Is it possible to send mails once in a day via crontab to related people to change their password whose account is going to expire within 15 days with this below message

Code:

Hi.(related username)

The password for the user account "related_account_name_which_is_going_to_expire" is going to expire. Please change the password before it get expired. To change the passowrd follow the below steps

1) ssh username@IP_address

2) passwd  (give current passowrd and update with new one. Don't use previous and simple password

3) chage -l $USER (note down the next expiration date and change it before it get expired using same above steps

4) exit

Sometimes one mail ID will be having many accounts, for example
kumar=kumar@domain.com
roopa=roopa@domain.com
user1=user@domain.com
user2=user@domain.com
Now the user1 and user2 is having the same mail ID i.e user@domain.com. Now the user@domain.com should get mail to change only for user2 account. This is my last question in this thread. I hope you definitely help me in this.

David the H.

11-27-2011 05:33 AM

This may be the first time I've ever been called a "genius". :hattip: But everything posted so far is well-understood by anyone with a reasonable amount of scripting experience. You'll get there too, eventually.

sed, grep and many other commands use regular expressions to parse out sections of text. There are many good tutorials available both online and in printed form. Just find one that looks good to you and study it until you at least understand the basic techniques, such as the lines juako gave. You'll be glad you did.

As for mailing out the results, it's certainly not hard to do. But I'd probably set it up as a separate mail script that launches the first, or at least reads the output file from it, and sends the messages out based on that. Then just set cron to launch that script once per day.

I'm not too familiar with cli mail programs though, so you'll have to figure out the exact commands yourself. Post what you come up with here and we'll be glad to look it over and give advice.

mandyapenguin

11-27-2011 08:14 AM

1 Attachment(s)

Hi..David master, Thanks for your kind reply.
Before adding body message.

Code:

bash pass_expire 

Password for roopa will expire in 11 days.

Password for user2 will expire in 1 days.

Password for user3 has expired

Password for user4 has expired

Please find the attachment. I added body message. Now the helpdesk members can get 4 mails i.e 2 about going to expire and other 2 about expired. Meanwhile the related accounts are not able to get mails. Please help me to send mails to both i.e helpdesk and the related accounts. And if I add "chage -l $USER" in the body message it showing the currently logged in user in the mails instead of "$USER". I also tried chage -l "$USER", but still it showing currently logged in user in the related field. So at last I put "chage -l username" only in the body message.

David the H.

11-29-2011 01:18 PM

Well, as I said, I don't know much about how mail works, so I don't really know what you need to do to get it to send the right things to the right people.

I'm also not sure what you mean by sending messages to both "helpdesk and related". It's not clear to me who is "helpdesk" and who is "related". Could you elaborate please?

The $USER variable is preset by the shell to the name of the user who owns the shell. In this case it's the name of the user running the script, and the value is expanded before the message is sent. You need to use the ${users[i]} array value for the name you want to supply to the message. But then again you should know that, as you're already doing it on the first line.

In any case I think the script would be cleaner, and the text more easily modifiable, if you used a variable or function to supply the default message. A function would probably be the most flexible:

Code:



printmess(){

        #$1 is expired/expsoon, $2 is the username



        case $1 in

                expired) echo "Password for $2 has expired." ;;

                expsoon) echo "Password for $2 will expire in $diff days." ;;

        esac

        echo "To change the password follow the below steps"

        echo "  1) ssh username@IP_address"

        echo "  2) passwd  (type current password and update with new one, Don't use previous and simple password)"

        echo "  3) chage -l username (note down the next expiration date and change it before it get expired using the same above steps."

        echo "  4) exit"

        echo "Regards"

        echo "root@domain"

}





for i in "${!expire_date[@]}" ; do



        userdate=$( date -d "${expire_date[i]}" +%s 2>/dev/null )

        if (( userdate < now )); then

                printmess expired "${users[i]}" | mail -s "password expire" "$admins" "$users"

        elif (( userdate <= soon )); then

                diff=$(( ( userdate - now ) / 60 / 60 / 24 ))

                printmess expsoon "${users[i]}" | mail -s "password expire" "$admins" "$users"

        fi

done

mandyapenguin

12-01-2011 09:16 AM

1 Attachment(s)

Hi.. David master, Thanks for your kind reply.
You are helping me a lot. Thank you very much.
Please excuse me, if you could not understand from my explanation.
You are already near to that what I am expecting. I don't want you to confuse with helpdesk members, and you just leave it.
Here I will tell clearly.
The related users are nothing but the mail IDs for the user accounts from the passwd file, and those mail IDs for the user accounts as below.
For example

Code:

roopa=roopa@domain.com

kumar=kumar@domain.com

Please see these below

Code:

Password for roopa will expire in 2 days.

Password for kumar has expired.

Now mail ID called "roopa@domain.com" should get mail only about roopa account and other mail ID i.e "kumar@domain.com" account should get mail about kumar account.
Now if I add as only "mail -s "password expire" "$users" the related mail IDs are not getting mail. Some mail ID will get mails only then, when system user account and webmail account both are same and only if I add as

Code:

mail -s "password expire" "${users[i]}@domainname

but some user accounts will be having their project name also like kumar_proj_name at that time they will not get mails since the mail ID itself does not exist.
I can also get mail if I add like below
admins=mymailID@domain.com
"mail -s "password expire" "$admins" "${users[i]}@domainname"
Please find the attachment, I have removed also "$admins" because again I don't want you to get confuse with $admins and $users. It is enough, if the related mail IDs are getting mails about related accounts.
I hope, now you are not confused and help me in this.

David the H.

12-01-2011 08:16 PM

I'm still not completely sure I follow you.

I think I see your main issue now though. If "users" is defined as an array, then "$users" is the same as "${users[0]}". So you're sending everything to the same address. If you want the full array list, you need to use "${users[@]}". Unless of course you want to send it to that user only, in which case use "${users[i]}".

For the second part, are you saying that some users have to have their messages sent to different addresses from the default?

If so, we'd have to use some kind of a test to compare the username to a preset list, and send the message to a different address if there's a match. Assuming you're using bash 4+, an associative array would probably be the best option.

Code:

#define a list of exceptions to the default mail addresses

declare -A xpns

xpns=(

        [roopa]='roopa@domain.com'

        [kumar]='kumar@domain.com'

)

It might be a good idea instead to save the list to a separate "config" file and source it, so you don't have to edit the script itself (You can also store other default values in it, like "IP").

Code:

### external file (exceptions.txt) ###

xpns[roopa]='roopa@domain.com'

xpns[kumar]='kumar@domain.com'



### script ###

#define a list of exceptions to the default mail address.

#import values from the file "exceptions.txt"

declare -A xpns

if [[ -r exceptions.txt ]];

        . exceptions.txt

else

        echo "Warning: can't find or read the exceptions file!" >&2

fi

Then in the loop, you can easily test to see if the current name is in the exception list.

Code:

if [[ "${!xpns[*]}" =~ "${users[i]}" ]]; then

        address="${xpns[${users[i]}]}"

else

        address="${users[i]}"

fi



printmess expired "${users[i]}" | mail -s "password expire" "$address"

(This is assuming again that I understand the mail command correctly. The last argument is supposed to be either the username or full address to send to, correct?)

It's even possible to shorten the full if test to this:

Code:

address="${xpns[${users[i]}]:-${users[i]}}"

The last argument means it will use the name from the exception list if it exists, and defaults to to the username if it doesn't (it's the ${var:-default} substitution pattern).

All times are GMT -5. The time now is 11:06 PM.