Using 2 arrays in a For loop with different number setS?

finalturismo · 09-08-2022, 09:06 PM

Ok so i was just trying this out, but its obviously not working -.-
how can i add another letter or number for count?
like i want to have another number increase for the pcs arrays.... instead of i i want N

For example

Code:

for ((i = 0; n=0; i <${#server[@]} + n<${#pcs[@]}; i++ n++))

Something like that

michaelk · 09-08-2022, 09:33 PM

Not sure exactly what you are trying to accomplish. Maybe two loops? Or maybe a while loop instead of a for with whatever variables you want use to increment and test for.

Code:

for ((i=0; i<${#server[@]}; i++)) do;
   for ((n=0; n<${#pcs[@]}; n++)) do;
   ...
   done
done

evo2 · 09-08-2022, 09:35 PM

Quote:

Originally Posted by finalturismo

Ok so i was just trying this out, but its obviously not working -.-
how can i add another letter or number for count?
like i want to have another number increase for the pcs arrays.... instead of i i want N

For example

Code:

for ((i = 0; n=0; i <${#server[@]} + n<${#pcs[@]}; i++ n++))

Something like that

Not sure what the "+" is doing there. May be you want "&&"? Also you need to use "," if you want multiple statements in control section of the for loop.
Eg.

Code:

#!/bin/bash
server=( "sa" "sb" "sc")
pcs=( "pa" "pb" )

for ((i = 0, n=0; i <${#server[@]} && n<${#pcs[@]}; i++, n++)) {
    echo "----"
    echo "i = $i"
    echo "n = $n"
    echo "server is ${server[i]}"
    echo "pc is ${server[n]}"
}

Evo2.

evo2 · 09-08-2022, 09:39 PM

Quote:

Originally Posted by michaelk

Not sure exactly what you are trying to accomplish. Maybe two loops? Or maybe a while loop instead of a for with whatever variables you want use to increment and test for.

Code:

for ((i=0; i<${#server[@]}; i++)) do;
   for ((n=0; n<${#pcs[@]}; n++) do;
   ...
   done
done

Ahh, nested loops, maybe that is what OP is actually asking!

finalturismo · 09-08-2022, 11:12 PM

Quote:

Originally Posted by evo2

Ahh, nested loops, maybe that is what OP is actually asking!

Ya thats literally perfect. Now i just got to parallel process these variables. I have been trying for past day to get it and i just seem to be getting my ass kicked XD

Code:

cpumodel[$i]=$(ssh ${server[i]} -p ${sshport[i]} lscpu | grep -i -E -m1  "model name" | sed 's&model name:&&Ig' | sed 's&(r)&&Ig'| sed 's&@.*&&Ig' | xargs );\
	totalcores[$i]=$(ssh ${server[i]} -p ${sshport[i]} lscpu | grep -i "core(s)\|socket(s)" | sed 's&core.*:&&gI' | sed 's&socket.*:&&gI' | sed 's& &&gI' | xargs | awk '{print $1*$2}' | xargs );\
	cpuuseallcores[$i]=$(ssh ${server[i]} -p ${sshport[i]} mpstat  -P ALL | awk '{print "<td><font color=orange>"$6"</font color>""<font color=teal>%</font color></td>"}' | tail +4 | sed '1~4 s&^&<tr align=left>&gI' | sed '0~4 s&$&</tr>&gI' | sed '$s&$&</tr>&' | xargs );\
	corespersocket[$i]=$(ssh ${server[i]} -p ${sshport[i]} lscpu | grep -i "core(s) per socket" | awk '{print $4}'|xargs);\
	threadpercore[$i]=$(ssh ${server[i]} -p ${sshport[i]} lscpu | grep -i "thread(s) per core" | awk '{print $4}' | xargs);\
	cpumhz[$i]=$(ssh ${server[i]} -p ${sshport[i]} lscpu | grep -i -E -m1  "max mhz" | sed 's&CPU max MHz:&&Ig' | sed 's&.0000&&Ig' | awk '{print "<font color=orange>"$1"</font color>""<font color=teal>MHz</font color>"}'| xargs );\
	cpusockets[$i]=$(ssh ${server[i]} -p ${sshport[i]} lscpu | grep -i "socket(s)" | awk '{print $2}'| xargs);\
	cpu1temp[$i]=$(ssh ${server[i]} -p ${sshport[i]} sensors | grep -i "package id 0" | awk '{print $4}' | sed 's&+&&gI' | xargs);\
	cpu2temp[$i]=$(ssh ${server[i]} -p ${sshport[i]} sensors | grep -i "package id 1" | awk '{print $4}' | sed 's&+&&gI' | xargs);\
	subnet[$i]=$(ssh ${server[i]} -p ${sshport[i]} hostname -I | sed 's& &\n&' | grep -v "${wanip[i]}" | awk -F'.' -vOFS='.' '{$NF=0}1;'| xargs );\
	totalcpuusage[$i]=$(ssh ${server[i]} -p ${sshport[i]} grep 'cpu ' /proc/stat | awk '{usage=($2+$4)*100/($2+$4+$5)} END {print usage "%"}'| xargs );\
	ramtotal[$i]=$(ssh ${server[i]} -p ${sshport[i]} free --giga -h |grep -i -v "swap" | grep -i "g" | awk '{print $2"B"}' | xargs );\
	ramused[$i]=$(ssh ${server[i]} -p ${sshport[i]} free --giga -h  |grep -i -v "swap" | grep -i "g" | awk '{print $3"B"}' | xargs );\
	ramascache[$i]=$(ssh ${server[i]} -p ${sshport[i]} free --giga -h |grep -i -v "swap" | grep -i "g" | awk '{print $6"B"}' | xargs );\
	ramfree[$i]=$(ssh ${server[i]} -p ${sshport[i]} free --giga -h | grep -i -v "swap" | grep -i "g"  | awk '{print $4"B"}' | xargs );\
	zfsarcram[$i]=$(ssh ${server[i]} -p ${sshport[i]} arc_summary | grep -i "arc size" | awk '{print "<font color=orange>"$6"</font color><font color=teal>"$7"</font color>"}' | xargs );\
	zfsarchit[$i]=$(ssh ${server[i]} -p ${sshport[i]} arc_summary | grep -i "cache hit ratio" | awk '{print "<font color=orange>"$4"</font color><font color=teal>"$5"</font color>"}' | xargs );\

evo2 · 09-08-2022, 11:29 PM

Quote:

Originally Posted by finalturismo

Ya thats literally perfect. Now i just got to parallel process these variables. I have been trying for past day to get it and i just seem to be getting my ass kicked XD

Code:

cpumodel[$i]=$(ssh ${server[i]} -p ${sshport[i]} lscpu | grep -i -E -m1  "model name" | sed 's&model name:&&Ig' | sed 's&(r)&&Ig'| sed 's&@.*&&Ig' | xargs );\
	totalcores[$i]=$(ssh ${server[i]} -p ${sshport[i]} lscpu | grep -i "core(s)\|socket(s)" | sed 's&core.*:&&gI' | sed 's&socket.*:&&gI' | sed 's& &&gI' | xargs | awk '{print $1*$2}' | xargs );\
	cpuuseallcores[$i]=$(ssh ${server[i]} -p ${sshport[i]} mpstat  -P ALL | awk '{print "<td><font color=orange>"$6"</font color>""<font color=teal>%</font color></td>"}' | tail +4 | sed '1~4 s&^&<tr align=left>&gI' | sed '0~4 s&$&</tr>&gI' | sed '$s&$&</tr>&' | xargs );\
	corespersocket[$i]=$(ssh ${server[i]} -p ${sshport[i]} lscpu | grep -i "core(s) per socket" | awk '{print $4}'|xargs);\
	threadpercore[$i]=$(ssh ${server[i]} -p ${sshport[i]} lscpu | grep -i "thread(s) per core" | awk '{print $4}' | xargs);\
	cpumhz[$i]=$(ssh ${server[i]} -p ${sshport[i]} lscpu | grep -i -E -m1  "max mhz" | sed 's&CPU max MHz:&&Ig' | sed 's&.0000&&Ig' | awk '{print "<font color=orange>"$1"</font color>""<font color=teal>MHz</font color>"}'| xargs );\
	cpusockets[$i]=$(ssh ${server[i]} -p ${sshport[i]} lscpu | grep -i "socket(s)" | awk '{print $2}'| xargs);\
	cpu1temp[$i]=$(ssh ${server[i]} -p ${sshport[i]} sensors | grep -i "package id 0" | awk '{print $4}' | sed 's&+&&gI' | xargs);\
	cpu2temp[$i]=$(ssh ${server[i]} -p ${sshport[i]} sensors | grep -i "package id 1" | awk '{print $4}' | sed 's&+&&gI' | xargs);\
	subnet[$i]=$(ssh ${server[i]} -p ${sshport[i]} hostname -I | sed 's& &\n&' | grep -v "${wanip[i]}" | awk -F'.' -vOFS='.' '{$NF=0}1;'| xargs );\
	totalcpuusage[$i]=$(ssh ${server[i]} -p ${sshport[i]} grep 'cpu ' /proc/stat | awk '{usage=($2+$4)*100/($2+$4+$5)} END {print usage "%"}'| xargs );\
	ramtotal[$i]=$(ssh ${server[i]} -p ${sshport[i]} free --giga -h |grep -i -v "swap" | grep -i "g" | awk '{print $2"B"}' | xargs );\
	ramused[$i]=$(ssh ${server[i]} -p ${sshport[i]} free --giga -h  |grep -i -v "swap" | grep -i "g" | awk '{print $3"B"}' | xargs );\
	ramascache[$i]=$(ssh ${server[i]} -p ${sshport[i]} free --giga -h |grep -i -v "swap" | grep -i "g" | awk '{print $6"B"}' | xargs );\
	ramfree[$i]=$(ssh ${server[i]} -p ${sshport[i]} free --giga -h | grep -i -v "swap" | grep -i "g"  | awk '{print $4"B"}' | xargs );\
	zfsarcram[$i]=$(ssh ${server[i]} -p ${sshport[i]} arc_summary | grep -i "arc size" | awk '{print "<font color=orange>"$6"</font color><font color=teal>"$7"</font color>"}' | xargs );\
	zfsarchit[$i]=$(ssh ${server[i]} -p ${sshport[i]} arc_summary | grep -i "cache hit ratio" | awk '{print "<font color=orange>"$4"</font color><font color=teal>"$5"</font color>"}' | xargs );\

Hmm.. you know that there is a whole bunch of existing well established solutions for cluster monitoring?

finalturismo · 09-08-2022, 11:52 PM

Quote:

Originally Posted by evo2

Hmm.. you know that there is a whole bunch of existing well established solutions for cluster monitoring?

That i do know, but thats just a small part of the script. It does a bunch of other custom stuff that i need and it puts everything in 1 email every 24 hours so i can see all my client machines and servers. It makes life much easier, well thats the goal anyway but i seem to be stuck on parallel processing. XD

finalturismo · 09-09-2022, 01:05 AM

Anyone got a solution for the variables in parallel?

pan64 · 09-09-2022, 01:11 AM

https://linux.die.net/man/1/parallel

Turbocapitalist · 09-09-2022, 01:27 AM

Quote:

Originally Posted by finalturismo

Anyone got a solution for the variables in parallel?

I would run one SSH session per host and inside that do each command so that the end result is captured outside of SSH as an array, perhaps simulate an array of arrays even. Then you could just append & to the end of each such line for parallel action. wait in bash is a little special and might help. Though GNU parellel might be of use, too, if you have to set a maximum limit for the number of parallel processes.

finalturismo · 09-09-2022, 02:58 AM

Quote:

Originally Posted by Turbocapitalist

I would run one SSH session per host and inside that do each command so that the end result is captured outside of SSH as an array, perhaps simulate an array of arrays even. Then you could just append & to the end of each such line for parallel action. wait in bash is a little special and might help. Though GNU parellel might be of use, too, if you have to set a maximum limit for the number of parallel processes.

I have been trying just about everything and i cant seem to get it to go. But since you said ssh, i think that sounds like the issue.

You have a recommendation on how i might do this? or an example, I am still a coder n00b for the most part.

What you guys could do in a day still probably takes me a week or 2. Still learning how to get properly setup.

boughtonp · 09-09-2022, 05:44 AM

Quote:

Originally Posted by finalturismo

Quote:

Originally Posted by evo2

Hmm.. you know that there is a whole bunch of existing well established solutions for cluster monitoring?

That i do know, but thats just a small part of the script. It does a bunch of other custom stuff that i need and it puts everything in 1 email every 24 hours so i can see all my client machines and servers. It makes life much easier, well thats the goal anyway but i seem to be stuck on parallel processing. XD

Quote:

Originally Posted by finalturismo

I have been trying just about everything and i cant seem to get it to go. But since you said ssh, i think that sounds like the issue.

You have a recommendation on how i might do this? or an example, I am still a coder n00b for the most part.

What you guys could do in a day still probably takes me a week or 2. Still learning how to get properly setup.

!!!

The existing well established tools evo2 is referring to almost certainly already do everything you need them to, but can also be configured/scripted for any additional functionality that they don't do out of the box, they already exist, are way more efficient, effective, stable, and contain far far fewer bugs than what you are currently struggling to write.

The most important thing to learn is using the right tools for the job.

michaelk · 09-09-2022, 09:48 AM

Code:

cpumodel[$i]=$(ssh ... );\
totalcores[$i]=$(ssh ... );\

The easiest might be to just run each as a background job i.e.

Code:

 
cpumodel[$i]=$(ssh ... ) &
totalcores[$i]=$(ssh ... ) &

You are creating 17 concurrent login ssh sessions which normally should not be a problem unless some limits have been set.

As posted I think, you can run a single ssh session to capture the data and "post process" back on the local PC.

Code:

for loops; do
  ssh  ${server[i]} -p ${sshport[i]} "lscpu; mpstat;" > server${$i}.log &
done
for loops; do
  cpumodel[$i]=$(grep ... server${$i}.log ...)
  ...
done

Also posted you really need to use the right tool for the job.

MadeInGermany · 09-09-2022, 02:56 PM

A for loop takes one list and one variable.
A while loop is more versatile:

Code:

i=0; n=0
while (( i < ${#server[@]} && n < ${#pcs[@]} ))
do
  ...
  (( i++ )); (( n++ ))
done

However if i and n always have the same value, why have them separate at all?
Further, I see no relation to your ssh task.

Code:

cpumodel[$i]=$(ssh ... ) &

I think this won't work. The backgrounded task will run in a forked sub-shell and that won't copy back its variables to the main shell.

Code:

cpumodel[$i]=$(ssh ${server[i]} -p ${sshport[i]} lscpu | grep -i -E -m1  "model name" | sed 's&model name:&&Ig' | sed 's&(r)&&Ig'| sed 's&@.*&&Ig' | xargs )
totalcores[i]=$(ssh ${server[i]} -p ${sshport[i]} lscpu | grep -i "core(s)\|socket(s)" | sed 's&core.*:&&gI' | sed 's&socket.*:&&gI' | sed 's& &&gI' | xargs | awk '{print $1*$2}' | xargs )
cpuuseallcores[$i]=$(ssh ${server[i]} -p ${sshport[i]} mpstat  -P ALL | awk '{print "<td><font color=orange>"$6"</font color>""<font color=teal>%</font color></td>"}' | tail +4 | sed '1~4 s&^&<tr align=left>&gI' | sed '0~4 s&$&</tr>&gI' | sed '$s&$&</tr>&' | xargs )

This is inefficient.
If you have each value on a separate line then consider one ssh call, like the following:

Code:

{
read cpumodel[i]
read totalcores[i]
read cpuuseallcores[i]
} < <(
ssh ${server[i]} -p ${sshport[i]} /bin/bash -s <<"REMOTESCRIPT"
lscpu | grep -i -E -m1 "model name" | sed 's&model name:&&Ig' | sed 's&(r)&&Ig'| sed 's&@.*&&Ig' | xargs
lscpu | grep -i "core(s)\|socket(s)" | sed 's&core.*:&&gI' | sed 's&socket.*:&&gI' | sed 's& &&gI' | xargs | awk '{print $1*$2}' | xargs
mpstat -P ALL | awk '{print "<td><font color=orange>"$6"</font color>""<font color=teal>%</font color></td>"}' | tail +4 | sed '1~4 s&^&<tr align=left>&gI' | sed '0~4 s&$&</tr>&gI' | sed '$s&$&</tr>&' | xargs
REMOTESCRIPT
)

michaelk · 09-09-2022, 03:21 PM

Quote:

I think this won't work. The backgrounded task will run in a forked sub-shell and that won't copy back its variables to the main shell.

Oops, your correct.