[SOLVED] How to break array and extract individual components in a shell script

sysmicuser · 03-04-2019, 06:54 PM

Hey Guys,

I have a shell script below, basically, the intent of the script is to enable firewall rule on remote Windows servers for WinRM so that ANsible can perform configuration management.

Code:

#!/bin/bash

set -x
PROCESS_ID=$$

LOGFILE=/tmp/"${PROCESS_ID}"

az vm list --output=table |tee -a >> "${LOGFILE}"

az login --service-principal -u $APP_ID --password $PASSWORD --tenant $TENANT_ID

# declare an array variable
declare -a host_list=$(cat "${LOGFILE}"|grep NXT|awk '{print $1}')
echo $host_list
## now loop through the above array
for i in "${host_list[@]}"
do
case ${i} in
*AD* )
  export resource_group="rg-ad"
  ;;
*AOS* | *WEB* )
  export resource_group="rg-app"
  ;;
*SQL* | *FS* )
  export resource_group="rg-sql"
  ;;
*APP* | *STG* )
  export resource_group="rg-remote"
  ;;
*BI* | *SSRS* )
  export resource_group="rg-bi"
  ;;
*)
  Message="Invalid hostname , please check"
  ;;
esac
   echo "Installing Windows Remote Management $i with resource group $resource_group "


   az vm run-command invoke --command-id RunPowerShellScript --resource-group "${resource_group}" --name "${i}" --scripts '(New-Object -TypeName System.Net.WebClient).DownloadFile("https://raw.githubusercontent.com/ansible/ansible/devel/examples/scripts/ConfigureRemotingForAnsible.ps1", "$env:temp\ConfigureRemotingForAnsible.ps1")'

   az vm run-command invoke --command-id RunPowerShellScript --resource-group "${resource_group}" --name "${i}" --scripts 'powershell.exe -ExecutionPolicy ByPass -File "$env:temp\ConfigureRemotingForAnsible.ps1"'
   az vm run-command invoke --command-id RunPowerShellScript --resource-group "${resource_group}" --name "${i}" --scripts 'netsh advfirewall firewall add rule name="Allow WinRM (Http)" dir=in localport=5985 protocol=tcp action=allow enable=yes'
   az vm run-command invoke --command-id RunPowerShellScript --resource-group "${resource_group}" --name "${i}" --scripts 'netsh advfirewall firewall add rule name="Allow WinRM (Https)" dir=in localport=5986 protocol=tcp action=allow enable=yes'

done

exit 0

When it is executed

Code:

./temp.sh
+ PROCESS_ID=13879
+ LOGFILE=/tmp/13879
+ az vm list --output=table
+ tee -a
+ az login "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
[
  {
    "cloudName": "AzureCloud",
      "type": "servicePrincipal"
    }
  }
]
++ cat /tmp/13879
++ awk '{print $1}'
++ grep NXT
+ declare -a 'host_list=NXTPREPAOS-1
NXTPREPAOS-2
NXTPREPAOS-3
NXTPREPAOS-4
NXTPREPAOS-5
NXTPREPSQL-1
NXTPREPSQL-2'
+ echo NXTPREPAOS-1 NXTPREPAOS-2 NXTPREPAOS-3 NXTPREPAOS-4 NXTPREPAOS-5 NXTPREPSQL-1 NXTPREPSQL-2
NXTPREPAOS-1 NXTPREPAOS-2 NXTPREPAOS-3 NXTPREPAOS-4 NXTPREPAOS-5 NXTPREPSQL-1 NXTPREPSQL-2
+ for i in '"${host_list[@]}"'
+ case ${i} in
+ export resource_group=rg-app
+ resource_group=rg-app
+ echo 'Installing Windows Remote Management NXTPREPAOS-1
NXTPREPAOS-2
NXTPREPAOS-3
NXTPREPAOS-4
NXTPREPAOS-5
NXTPREPSQL-1
NXTPREPSQL-2 with resource group rg-app '
Installing Windows Remote Management NXTPREPAOS-1
NXTPREPAOS-2
NXTPREPAOS-3
NXTPREPAOS-4
NXTPREPAOS-5
NXTPREPSQL-1
NXTPREPSQL-2 with resource group rg-app
+ az vm run-command invoke --command-id RunPowerShellScript --resource-group rg-app --name 'NXTPREPAOS-1
NXTPREPAOS-2
NXTPREPAOS-3
NXTPREPAOS-4
NXTPREPAOS-5
NXTPREPSQL-1
NXTPREPSQL-2' --scripts '(New-Object -TypeName System.Net.WebClient).DownloadFile("https://raw.githubusercontent.com/ansible/ansible/devel/examples/scripts/ConfigureRemotingForAnsible.ps1", "$env:temp\ConfigureRemotingForAnsible.ps1")'
Operation failed with status: 'Bad Request'. Details: 400 Client Error: Bad Request for url: https://management.azure.com/subscriptions/xxxxxxxxxxxxxxxxxxxxxxxxx/resourceGroups/rg-app/providers/Microsoft.Compute/virtualMachines/NXTPREPAOS-1%0ANXTPREPAOS-2%0ANXTPREPAOS-3%0ANXTPREPAOS-4%0ANXTPREPAOS-5%0ANXTPREPSQL-1%0ANXTPREPSQL-2/runCommand?api-version=2018-10-01

I want to dynamically get the no of VM's as it could change as per requirement. I know the problem is a whitespace character in the array, how to solve?

sysmicuser · 03-04-2019, 09:13 PM

Made it simpler, all what we want is individual hostnames from an array.

Code:

#!/bin/bash
set -x
PROCESS_ID=$$

LOGFILE=/tmp/"${PROCESS_ID}"

az vm list --output=table |tee -a >> "${LOGFILE}"

az login "xxxxxxxxxxxxxx"

# declare an array variable
declare -a host_list=$(cat "${LOGFILE}"|grep NXT|awk '{print $1}')
echo $host_list

## now loop through the above array
for i in "${host_list[@]}"
do
targetedhost=$(echo $i|cut -d ' ' -f1)
case ${targetedhost} in
*AD* )
  export resource_group="rg-ad"
  ;;
*AOS* | *WEB* )
  export resource_group="rg-app"
  ;;
*SQL* | *FS* )
  export resource_group="rg-sql"
  ;;
*APP* | *STG* )
  export resource_group="rg-remote"
  ;;
*BI* | *SSRS* )
  export resource_group="rg-bi"
  ;;
*)
  Message="Invalid hostname , please check"
  ;;
esac
   echo "Installing Windows Remote Management $targetedhost with resource group $resource_group "

done

exit 0

Output: Looks good but it not going for next hostname until end of the array:

Code:

./temp.sh
+ PROCESS_ID=15311
+ LOGFILE=/tmp/15311
+ az vm list --output=table
+ tee -a
+ az login "xxxxxxxxxxxxxxxxxxxxxxxxxxx"
[
  {
    "cloudName": "AzureCloud",
    "name": "Visual Studio Enterprise",
    }
  }
]
++ cat /tmp/15311
++ grep NXT
++ awk '{print $1}'
+ declare -a 'host_list=NXTPREPAOS-1
NXTPREPAOS-2
NXTPREPAOS-3
NXTPREPAOS-4
NXTPREPAOS-5
NXTPREPSQL-1
NXTPREPSQL-2'
+ echo NXTPREPAOS-1 NXTPREPAOS-2 NXTPREPAOS-3 NXTPREPAOS-4 NXTPREPAOS-5 NXTPREPSQL-1 NXTPREPSQL-2
NXTPREPAOS-1 NXTPREPAOS-2 NXTPREPAOS-3 NXTPREPAOS-4 NXTPREPAOS-5 NXTPREPSQL-1 NXTPREPSQL-2
+ for i in '"${host_list[@]}"'
++ echo NXTPREPAOS-1 NXTPREPAOS-2 NXTPREPAOS-3 NXTPREPAOS-4 NXTPREPAOS-5 NXTPREPSQL-1 NXTPREPSQL-2
++ cut -d ' ' -f1
+ targetedhost=NXTPREPAOS-1
+ case ${targetedhost} in
+ export resource_group=rg-app
+ resource_group=rg-app
+ echo 'Installing Windows Remote Management NXTPREPAOS-1 with resource group rg-app '
Installing Windows Remote Management NXTPREPAOS-1 with resource group rg-app
+ exit 0

As you would see above we got the hostname and it finished it need to loop through, why it is not going through?

orbea · 03-04-2019, 10:38 PM

Your quoted array is not splitting, is this intentional?

For example.

Code:

$ foo='this is a test'
$ for i in ${foo[@]}; do echo $i; done
this
is
a
test
$ for i in "${foo[@]}"; do echo $i; done
this is a test

sysmicuser · 03-04-2019, 11:05 PM

@orbea

Thanks, heaps for that, I cannot believe that double quotes around the array were causing the grief! I am now confused, normally it a good practice to reference a variable like "${env}", so this does not hold true for an array?

Please enlighten

Thank you

NevemTeve · 03-04-2019, 11:12 PM

It depends your goal: it you don't wish your variable space-splitted, use qutes: "$env", otherwise don't: $env or ${env}

sysmicuser · 03-04-2019, 11:26 PM

Aah! Got it, Thanks @Nevem Teve!

grail · 03-05-2019, 12:48 AM

This is an extremely bad example.

You should in fact double quote if it was actually an array.
As it is a simple string the ${[@]} expander is not being used correctly hence the output issues.
Here is a quick demo using orbea's example:

Code:

foo='this is a test'
for i in ${foo[@]}; do echo $i; done
for i in "${foo[@]}"; do echo $i; done


fooa=(this 'is a' test)

for i in ${fooa[@]}; do echo $i; done
for i in "${fooa[@]}"; do echo $i; done

The additional quotes inside the array are to prove why you need quotes around an array

NevemTeve · 03-05-2019, 02:04 AM

It would be nice to see an mcve.
https://stackoverflow.com/help/mcve

orbea · 03-05-2019, 07:40 AM

Quote:

Originally Posted by grail

The additional quotes inside the array are to prove why you need quotes around an array

Thanks for the additional examples, this is what I get for almost never using bash specific arrays.

ntubski · 03-05-2019, 08:03 AM

In this case, it probably makes more sense to use a while read loop (and drop useless use of cat and grep):

Code:

awk '/NXT/ {print $1}' "$LOGFILE" | while read i