Duplicate output looping through multiple values in while loop bash

bluethundr · 09-12-2018, 12:13 PM

I'm trying to loop through the contents of 3 files using a while loop in bash. They contain a list of names, aws accounts and aws account numbers.

But the loop isn't correct and it keeps repeating the first name from the list and the first aws environment from the list.

This is the output I see:

Code:

    AWS user name: aadatiya does not exist in AWS account: company-lab
    AWS user name: aadatiya does not exist in AWS account: company-lab
    AWS user name: aadatiya does not exist in AWS account: company-lab
    AWS user name: aadatiya does not exist in AWS account: company-lab
    AWS user name: aadatiya does not exist in AWS account: company-lab
    AWS user name: aadatiya does not exist in AWS account: company-lab
    AWS user name: aadatiya does not exist in AWS account: company-lab
    AWS user name: aadatiya does not exist in AWS account: company-lab

These are the files I'm trying to read from:

Code:

    aws_users_all="source_files/aws_users_all.txt"
    aws_env_list="source_files/aws_environments_all.txt"
    aws_account_numbers="source_files/aws_account_numbers.txt"

And this is the code with the incorrect loop:

Code:

    ofile=source_files/aws_access_keys/company-aws-access-keys-all-accounts.csv
    while IFS= read -r aws_user_name
      do 
        while IFS= read -r aws_key
        do
          while IFS= read -r aws_account_num
          do
            user_lives_here=$(aws iam get-user --user-name "$aws_user_name" --profile="$aws_key" 2> /dev/null  | jq -r '.User.UserName')
            if [[ -z "$user_lives_here" ]]; then
              printf "AWS user name: %s does not exist in AWS account: %s\\n" "$aws_user_name" "$aws_key"
            else
              echo "$aws_user_name,$user_access_key1,$key1_date_created,$key1_last_used,$key1AgeDays,$aws_key,$aws_account_num" >> $ofile
          fi
          done < "$aws_account_numbers"
        done < "$aws_env_list"
      done < "$aws_users_all"

If I take out one level (the account numbers level) the script behaves as expected and produces this output:

Code:

    AWS user name: aadatiya does not exist in AWS account: company-lab
    AWS user name: aadatiya does not exist in AWS account: company-bill
    AWS user name: aadatiya does not exist in AWS account: company-stage
    AWS user name: aadatiya does not exist in AWS account: company-dlab
    AWS user name: aadatiya does not exist in AWS account: company-nonprod
    AWS user name: aadatiya does not exist in AWS account: company-prod
    AWS user name: aadatiya does not exist in AWS account: company-govcloud-admin-nonprod
    AWS user name: abigailcharles does not exist in AWS account: company-lab
    AWS user name: abigailcharles does not exist in AWS account: company-bill
    AWS user name: abigailcharles does not exist in AWS account: company-stage
    AWS user name: abigailcharles does not exist in AWS account: company-dlab
    AWS user name: abigailcharles does not exist in AWS account: company-nonprod
    AWS user name: abigailcharles does not exist in AWS account: company-prod
    AWS user name: abigailcharles does not exist in AWS account: company-govcloud-admin-nonprod

I just commented out this level and it works:

Code:

      #while IFS= read -r aws_account_num
      #do
      #done

How can I do this correctly so that I loop through each name, aws account and aws account number so that each entry shows once?

rtmistler · 09-12-2018, 02:36 PM

I'm going to move this to the Programming forum to gain it better exposure.

Not totally sure because I don't use IFS, but that is a special shell variable. I'm not sure if you intended to use it versus not.

From a purely programming perspective, you are using the same variable in nested loops. I don't do that and it may be what is causing you grief. Perhaps establish three different loop variables? But if the syntax does require the use of IFS (internal field separator?) then OK. As I say, I have no experience programming using that variable.

ntubski · 09-12-2018, 06:25 PM

Quote:

Originally Posted by bluethundr

How can I do this correctly so that I loop through each name, aws account and aws account number so that each entry shows once?

I think your loop is correct, you just forgot to print the account number:

Code:

              printf "AWS user name: %s does not exist in AWS account: %s, number: %s\\n" "$aws_user_name" "$aws_key" "$aws_account_num"

MadeInGermany · 09-12-2018, 10:41 PM

You have nested the loops, so it repeats name and aws account for each aws account number.

grail · 09-13-2018, 01:14 AM

I believe the others have spotted your issue, but what would have made it easier would be if you also supplied input data.
You can obscure this as much as needed, but it would help paint the overall picture.

You would also need to provide an idea of how the data looks, ie. can 1 user have multiple accounts? can each account be for multiple account numbers? the list goes on
Remember, for you the data and code makes perfect sense due to how long you have been working on it, however we can only go on the information supplied.

Finally, you do not show us the desired output assuming all three loops were acting as intended.

astrogeek · 09-13-2018, 02:06 AM

All responders have made good points, but as grail has pointed out no one here actually knows what your data looks like or what you expect as a final result. As such we could easily end up with a script that runs perfectly, but does not produce the result you want!

Usually the best way to pose such questions is to provide a sample of the input data, for each of the files you are using, suitably scrubbed if necessary. Then an example of what you want the final result to look like, including mention of anything which should not be included in the result. That way we can see if the nested loops will produce what you want or if you should consider a different approach to the problem.

For example, as pointed out by MadeInGermany, the nested loops will at best produce the complete cartesian product of all names joined with all accounts and all account numbers. That may, or may not be what you actually want.