[SOLVED] How to dynamically create array with different namesNum to them

BW-userx · 08-09-2019, 10:29 AM

As I am now having FreeBSD on my laptop too, I found that there ARG_MAX is less than Linux is, so I am getting this error too many args when I run this one script.

'line 175: /usr/local/bin/shuf: Argument list too long'

So I am trying to put together a function that splits up the paths/filename into dynamically created arrays that are created when a magic number is reached so it does not spill over the ARG_MAX of the system.

using a simple number assignment to the array to cause it to be
array1
array2
array3
ect

while filling them as I go.

this is as far as I've gotten because it does not like how I am trying to assign a number to a array var and adding to it on the fly.

this is just a function within a larger script.

the error is

Code:

/home/userx/bin/ranImg: line 69: syntax error near unexpected token `$d'
/home/userx/bin/ranImg: line 69: `

the bold part is line 69

Code:

wd1=$HOME/data/ScreenResizedImages
wd2=$HOME/data/variety
wd3=$HOME/data/wallhaven-papers
SysColors=$HOME/bin/colorsOfASystem

 

subdivideImageDir()
{
	Dcount="$(find "$wd1" -type d -mindepth 2 -maxdepth 2 | wc -l)"
	echo $Dcount
	#ls gives new line per each entry. To try and compensate
	#for that put each entry into an element of an array
	#then just use one element to get the dir names in order
	# to the files in the last sub-dir
	#knowing the file tree layout helps to do this. 
	#wd1 goes to root/parent dir
	#
	#To dynamically get the sub-directories names in order to
	#traverse down into the sub-directories two (2) deep.
	#left side, right side
	#wd1 first set of sub-directories splits into two directories,
	#then take one of them sub-directories and go into that one
	#first . left side then right side, 
	#getting the files paths out and putting them into n array.
	
	#Then using the length of the path and file to get its character length
	# of the string, each. 
	#
	#then using that as a template for length of to total
	#ARG_MAX allowed in the system minus a fair amount to try
	#and keep it from spilling over the total amount allowed
	#within the system.
	#
	#Set up arrays filled with path and dir names each
	subdir1+=( $(ls "$wd1") )
	#subdir1[0] left side
	#subdir1[1] right side
	
	#gets dir names of bottom directories
	oneMoreDeep+=( $(ls "$wd1"/"${subdir1[0]}") )
	echo ${oneMoreDeep[6]}
	echo "len = ${#oneMoreDeep[@]}"
	#to comtrol the depth amount of directoires
	dlen=${#oneMoreDeep[@]}
	#put them together
	completePath=$wd1/${subdir1[1]}/${oneMoreDeep[3]}
	echo $completePath
	echo "going in loop"
        
        #try to create arrays on the fly
	cnt=0
	while read d
	do
		if [[ $cnt -lt "3" ]] ; then
			array"$cnt"+=( $d )
			((cnt++))
		else
			break
		fi	
	done < <(find $completePath -type f)
	
        #try to print out the created arrays on the fly to fill
	ct=0
	for ((a=0;a<${#array$ct[@]};a++)) ; do
		echo "${array$ct[$a]}"

		if [[ $a = "${#array$ct[@]}" ]] ; then
			((cn++))
		fi

		[[ $ct = $cnt ]] && break
	done
	exit


	#
	chk4+=( $(ls "$wd1"/"${subdir1[0]}"/"${chk2[0]}" ) )
	#get ARG_MAX minus an fair amount 
	#source https://www.cyberciti.biz/faq/linux-unix-arg_max-maximum-length-of-arguments/
	
	getSysArgMax=$( expr `getconf ARG_MAX` - `env|wc -c` - `env|wc -l` \* 4 - 2048)
	
	
	echo "final"
	echo "${chk2[1]}"
	echo;echo
	echo "${chk3[4]}"
	echo
	echo "${chk4[0]}"
	echo
	echo "$wd1"/"${chk1[0]}"/"${chk2[1]}"/"${chk4[0]}"
	len="$wd1"/"${chk1[0]}"/"${chk2[0]}"/"${chk4[0]}"
	echo "${#len}"
	#get into the dir that is the furiest sub-directory in holding
	#the files. Then take the files out of that until total ARG_MAX
	# is reach, then start over using a diffent array to store the paths
	# to the files. 
	#chk1[0] left side
	#chk1[1] right side
	 
	
	
	exit
	
}
subdivideImageDir

pan64 · 08-09-2019, 10:55 AM

did you try to insert set -xv and check what's happening?
did you try shellcheck?

NevemTeve · 08-09-2019, 10:59 AM

You could check the source of program xargs.

BW-userx · 08-09-2019, 11:23 AM

Quote:

Originally Posted by pan64

did you try to insert set -xv and check what's happening?
did you try shellcheck?

shellcheck

Code:

 for ((a=0;a<${#array$ct[@]};a++)) ; do
                    ^-- SC2154: array is referenced but not assigned (did you mean 'array1'?).
                            ^-- SC1087: Use braces when expanding arrays, e.g. ${array[idx]} (or ${var}[.. to quiet).
 
Line 57:
                echo "${array$ct[$a]}"
                             ^-- SC1087: Use braces when expanding arrays, e.g. ${array[idx]} (or ${var}[.. to quiet).
 
Line 58:
                if [[ $a = "${#array$ct[@]}" ]] ; then
                                    ^-- SC1087: Use braces when expanding arrays, e.g. ${array[idx]} (or ${var}[.. to quiet).

It as to do with how I am assigning the $cnt to the array name to get my array1, array2 etc... as I already suspected before shellcheck said so.

Just hoping someone knows that tick to this, saving me some time is all.

it seems a simple task to tag a number on the end of the array name, but obviously is not.

pan64 · 08-09-2019, 11:32 AM

I don't really understand how ARG_MAX, the syntax error and array handling comes together

BW-userx · 08-09-2019, 11:44 AM

Quote:

Originally Posted by pan64

I don't really understand how ARG_MAX, the syntax error and array handling comes together

ARG_MAX = max allowed character length = path/filename in a dir that is being put into one array totaling over the ARG_MAX allowed.

so Instead of trying to shove all of the paths to each individual file into one array split up the sub-directories and there files into separate arrays, making the array only hold almost up to that max allowed character length total between all of the elements in the (one) array.

get length of string for each string consisting of path/filename gives total characters in that string. then just keep a running total until ARG_MAX is reached then create another array and start over until ARG_MAX is reached repeat until entire sud-dir are read into all of arrays needed,

array0
array1
array2
etc...

that link in the function shows how to calculate ARG_MAX minus a fair amount to not over run the max allowed character length.

#source https://www.cyberciti.biz/faq/linux-...-of-arguments/

the syntext error I think you're talking about is my trying to assign a number to an array name while trying to fill that array and change its number at the same time.

simple loop
pseudo code

Code:

cnt=0

MAX_ARG=$( expr `getconf ARG_MAX` - `env|wc -c` - `env|wc -l` \* 4 - 2048)


while read d
 totalARG_MAX+=${#d}
array$cnt+=( $d ) 

#to change the array name by the number 
if [[ $totalARG_MAX = $MAX_ARG ]] ; then
    ((cnt++))
fi
done< <(find files to add to arrays)

( that loop is actually better then what I've already written out)

this is the actual reason for the error(s)

Code:

array$cnt
       ^

it does not know what that is, what is inside of it when written as an array in referencing it and its values in the elements.

Code:

${array$cnt[@]}

#can be 
${array1[@]}
${array2[@]}
etc

but it throws that error(s) instead.

teckk · 08-09-2019, 10:12 PM

Quote:

So I am trying to put together a function that splits up the paths/filename into dynamically created arrays

If I understand what you are wanting.

Code:

#300 items in array
Big_array=({1..300})

array1=()
array2=()
array3=()
array4=()

cnt=10

for i in "${Big_array[@]}"; do

    if [ ${#array1} -lt $cnt ]; then
        array1=""$array1" "$i""
         
    elif [ ${#array2} -lt $cnt ]; then
        array2=""$array2" "$i"" 
        
    elif [ ${#array3} -lt $cnt ]; then
        array3=""$array3" "$i""
         
    else
        array4=""$array4" "$i""
        
    fi
    
done

echo "array1 is "$array1""
echo "array2 is "$array2""
echo "array3 is "$array3""
echo "array4 is "$array4""

teckk · 08-10-2019, 08:28 AM

Code:

#Put all files in dir in 10 item arrays

cd /some/where

#Number of items in array
start=0
end=10

#array of all files in dir
files=($(echo *))

#split files into 10 item arrays
array_name=({a..z})
for i in "${array_name[@]}"; do
    array="array"$i""
    array=""$array" = $(echo ${files[@]:$start:$end})"
    echo "$array"
    echo
    start=$(($start + 10))
    sleep 1
done

Just an example for splitting array of files into smaller arrays.
If that what you are doing?

BW-userx · 08-10-2019, 09:51 AM

Quote:

Originally Posted by teckk

Code:

#Put all files in dir in 10 item arrays

cd /some/where

#Number of items in array
start=0
end=10

#array of all files in dir
files=($(echo *))

#split files into 10 item arrays
array_name=({a..z})
for i in "${array_name[@]}"; do
    array="array"$i""
    array=""$array" = $(echo ${files[@]:$start:$end})"
    echo "$array"
    echo
    start=$(($start + 10))
    sleep 1
done

Just an example for splitting array of files into smaller arrays.
If that what you are doing?

It is to just make some simple changes to mrxvtrc colors and bg image.

Code:

#!/usr/bin/env bash
#Jun. 11, 2019
#Michael Heras

#to give a random background image to
#mrxvt terminal
#by chaning its resource file
#.mrxvtrc

#set -xv

wd1=$HOME/data/ScreenResizedImages
wd2=$HOME/data/variety
wd3=$HOME/data/wallhaven-papers
SysColors=$HOME/bin/colorsOfASystem

#echo "
#wd1Count=$( find $wd1 -type f | wc -l )
#wd2Count=$( find $wd2 -type f | wc -l )
#wd3Count=$( find $wd3 -type f | wc -l )
#"


#get count of totals in the dir, put in an array
totals=(
$( find $wd3 -type f | wc -l )
$( find $wd1 -type f | wc -l )
$( find $wd2 -type f | wc -l )
)

smallest=${totals[0]}

for((i=0;i<${#totals[@]};i++))
do
#logic for smallest number
##
##  Get the lest amount to use
## that number to fill final array from
## total arrays used
###############
if [ ${totals[$i]} -lt $smallest ]; then
	smallest=${totals[$i]}
fi
done
echo $smallest
 
#logic for greatest number
#elif [ ${nos[$i]} -gt $greatest ]; then
#greatest=${nos[$i]}
##fi
#done

RandomColor()
{
	cat "$SysColors" | shuf | shuf > tempColors
	mv tempColors "$SysColors"
	mapfile -t colors < "$SysColors"
}


MixRandomImages()
{
	#Heidy Images
	mapfile -t temp1 < <(find "$wd1" -type f | shuf )
	#Wallpaper Images
	mapfile -t temp2 < <(find "$wd2" -type f -name "*.jpg" | shuf )
	#wallhaven-papers 
	mapfile -t temp3 < <(find "$wd3" -type f -name "*.jpg" | shuf )
	
	
	
MAX_ARG=$( expr `getconf ARG_MAX` - `env|wc -c` - `env|wc -l` \* 4 - 4096 ) # 2048 ) 
echo "1: MAX_ARG=$MAX_ARG"
	
while read d
do
	totalARGMAX=$(( $totalARGMAX + ${#d} ))
	#echo "$totalARGMAX"
	#echo "MAX_ARG $MAX_ARG"
	
	
	
	if [[ "$totalARGMAX" -ge "$MAX_ARG" ]] ; then
		((cnt++))
		#totalARGMAX=0
		echo;echo
		echo "$cnt :: $totalARGMAX"
		echo;echo
		sleep 1
		echo "Breaking "
		break
	else
		myarray+=( "$d" )
	fi
	

	
done < <(find "$wd1" -type f)

	#takes half of the amount of images in wd2 and adds that
	#and temp1 into an array. 
	#-e, --echo treat each ARG as an input line 
	#-n, --head-count=COUNT output at most COUNT lines 
	#	temp5=( $( shuf -e -n "$(($(ls "$wd2" | wc -l)/2))" "${temp1[@]}" ) )
	#take the lowest amount in all three dir, and give same amount
	#of other images into the mix
#	echo "
#	Temp1
#	${#temp1[@]}
#	Smallest
#	$smallest
	
#	"
	 ### HERE IS WHERE I AM GETTING THE 
	 #
	 #   /usr/local/bin/shuf: Argument list too long'
	 #
	 ###################################################
	#temp4=( $( shuf -e -n "$smallest" "${temp1[@]}" ) )
	
	#changing the array this one uses 
	temp4=( $( shuf -e -n "$smallest" "${myarray[@]}" ) )
#	echo "
#	${#temp4[@]}"
	 
#		echo "
#	Temp2
#	${#temp2[@]}
#	Smallest
#	$smallest
	
#	"
	temp5=( $( shuf -e -n "$smallest" "${temp2[@]}" ) )

#	echo "
#	${#temp5[@]}"
	 
#		echo "
#	Temp3
#	${#temp3[@]}
#	Smallest
#	$smallest
#	
#	"
	temp6=( $( shuf -e -n "$smallest" "${temp3[@]}" ) )

#	echo "
#	${#temp6[@]}"
		
	#all all 3 image arrays into one arrays
	ranArray1=( "${temp5[@]}" "${temp4[@]}"  "${temp6[@]}" )	
	
	#shuffle them up a few times
	for i in {1..4}
	do
		for g in 1
		do
			sh1=( $(shuf -e "${ranArray1[@]}") )
		done
		#empty array so it can be filled again with 
		#newly shuffed array
		unset ranArray1
		ranArray1=( ${sh1[@]} )
		#same here empty this array to be used again
		#to fill with newly shuffed results. 
		unset sh1	
	done
	
	#Add shuffled array into a main array to pick from
	ImageArray=( "${ranArray1[@]}" )
	echo "$totalARGMAX"
	echo "MAX_ARG $MAX_ARG"
}

#write changes to mrxvt terminal config file
UpdateMrxvtConfig()
{
	sed -ibak 's|Mrxvt.background:.*|Mrxvt.background: '"$( echo -e ${colors[ $RANDOM % ${#colors[@]} ]})"'|' $HOME/.mrxvtrc
	sed -ibak 's|Mrxvt.tabBackground:.*|Mrxvt.tabBackground: '"$( echo -e ${colors[ $RANDOM % ${#colors[@]} ]})"'|' $HOME/.mrxvtrc
	sed -ibak 's|Mrxvt.cursorColor: .*|Mrxvt.cursorColor: '"$( echo -e ${colors[ $RANDOM % ${#colors[@]} ]})"'|' $HOME/.mrxvtrc
	sed -ibak 's|Mrxvt.foreground: .*|Mrxvt.foreground:  '"$( echo -e ${colors[ $RANDOM % ${#colors[@]} ]})"'|' $HOME/.mrxvtrc
	sed -ibak 's|Mrxvt.Pixmap:.*|Mrxvt.Pixmap: '"$( echo -e ${ImageArray[ $RANDOM % ${#ImageArray[@]} ]})"';80x80|' $HOME/.mrxvtrc
}
#run functions
RandomColor
MixRandomImages
UpdateMrxvtConfig

#start the terminal.
mrxvt &

IN FreeBSD ARG_MAX

Code:

$ getconf ARG_MAX 
262144

$ echo $(( $(getconf ARG_MAX) - $(env | wc -c) ))
 261372

$ expr `getconf ARG_MAX` - `env|wc -c` - `env|wc -l` \* 4 - 2048
259204

In ArcoLinux ARG_MAX

Code:

$ getconf ARG_MAX
2097152

$ echo $(( $(getconf ARG_MAX) - $(env | wc -c) ))
2095916

$ expr `getconf ARG_MAX` - `env|wc -c` - `env|wc -l` \* 4 - 2048
2093688

FreeBSD has a significantly lesser amount of total characters allowed to be taken in off of the Command line. 1,834,484 less so adding path/filename of 16100 files and growing takes up and over runs FreeBSD's ARG_MAX limitation.

so because it is a changing number of files within the directories holding the images. In order to give the script the full amount to pick from, and keep from over running the max allowed ARG_MAX.

I'd fill an array until the ARG_MAX is met, then move to another array and fill that one until the same takes place, and keep adding an array to take from the total until that is depleted.

the math behind it can be done. the simple loop to do it is easy, the assignment of unique names for array on the fly is the hard part.
ie
array1
array2
etc being created as needed then filled until ARG_MAX is met then another array is created and same done to it.

grail · 08-10-2019, 04:09 PM

My thoughts are two fold, assuming bash is not restricted by the same limit you could simply pass the list of files to your function and use $@ and $# appropriately,
or if the value is the same then, then I would fill the array and use the same process for parsing:

Code:

#!/usr/bin/env bash

parse_n()
{
	local -a items
	local total inc

	items=("$@")
	total=$#
	inc=5

	for (( i = 0; i < total; i += inc ))
	do
		printf "%s\n" "${items[@]:i:inc}"
		echo "-------------"
	done
}

parse_n *

Above assumes no issue passing to bash function, otherwise:

Code:

#!/usr/bin/env bash

parse_n()
{
	local -a items
	local total inc

	items=(*)
	total=${#items[*]}
	inc=5

	for (( i = 0; i < total; i += inc ))
	do
		printf "%s\n" "${items[@]:i:inc}"
		echo "-------------"
	done
}

parse_n

I have chosen the arbitrary number of 5, but you get the idea

BW-userx · 08-10-2019, 06:41 PM

it's

Quote:

limit for the command line length. UNIX / Linux / BSD system has a limit on how many bytes can be used for the command line argument and environment variables. You need to use the getconf command to query system configuration variable called ARG_MAX.

that means counting from the prompt $ to the right there are a set amount of chars that can be taken in at one time, whence that limit has been reached then an error is thrown, and a message to report same error.

using that limit of chars that can take up an entire array. not words, but chars.

Code:

#get the max number of chars minus a fair amount for a safe buffer zone.

MAX_ARG=$( expr `getconf ARG_MAX` - `env|wc -c` - `env|wc -l` \* 4 - 4096 ) # 2048 ) 


echo "1: MAX_ARG=$MAX_ARG"
	
while read d
do
   #keep track of amount of chars being taken in by
   #using a running length of strings, added together
   #to give a total of chard being taken in.
   totalARGMAX=$(( $totalARGMAX + ${#d} ))
	
    #echo "$totalARGMAX"
    #echo "MAX_ARG $MAX_ARG"
	
    if [[ "$totalARGMAX" -ge "$MAX_ARG" ]] ; then
		((cnt++))
		#totalARGMAX=0
		echo;echo
		echo "$cnt :: $totalARGMAX"
		echo;echo
		sleep 1
		echo "Breaking "
		break
	else
		myarray+=( "$d" )
	fi
	

	
done < <(find "$wd1" -type f)

that function takes it up to one time of total allowed chars that can be taken into the array by the limits of ARG_MAX. Whence that max is met, then another array needs to be put into place to continute taking in the path/filenames until the complete source has been taken into the batch of arrays needed to store all of the files.

It would be better written like this, if the solution was present.

Code:

#!/usr/bin/env bash

#get the max number of chars minus a fair amount for a safe buffer zone.

MAX_ARG=$( expr `getconf ARG_MAX` - `env|wc -c` - `env|wc -l` \* 4 - 4096 ) # 2048 ) 


echo "1: MAX_ARG=$MAX_ARG"
	
while read d
do
   #keep track of amount of chars being taken in by
   #using a running length of strings, added together
   #to give a total of chard being taken in.
   totalARGMAX=$(( $totalARGMAX + ${#d} ))
	
	#provide a means to change the name of the array
	#so it can continue storing the path/file 
	#into an "array" of arrays until 
	#all files have been put into the
	#cluster/batch of arrays needed to
	#complete the storing of path/files
	#
	
    if [[ "$totalARGMAX" -ge "$MAX_ARG" ]] ; then
	     #when max chars haev been reach change the change
	     #the name of array here, to provide and 
	     #new storage area for the remaning files.
	     
			((newNum++))
			
			#reset count
			totalARGMAX=0
			
	fi
	
	myarray$newNum+=( "$d" )
	
done < <(find /path/to/files -type f)

the naming of an array like this gives an error due to the way bash sees it.

Code:

num=3
array$num=( 1 2 3 4 5 6 )

which the name of the array should translate to array3. But it does not.

pan64 · 08-11-2019, 03:46 AM

again, I know exatly what is ARG_MAX, I know bash ARRAYS, just I don't understand how comes these things together.
I think it is a totally wrong approach and you want to construct something useless (= that is the syntax error) caused by a limitation.
The solution could be to avoid passing such a huge array (but its name), or use a different language.

You can easily implement a perl/python/whatever oneliner to invoke any program and pass the full array to it without these limitations.
From the other hand there is a command xargs to do the split you are trying to implement.

What you try to implement in bash is a very good exercise to learn bash and to understand what should not be written in bash.

grail · 08-11-2019, 04:19 AM

My point was to make a simple sum, ie you say you have ~16100 files to process, so pick a number, maybe 5000 or 2000 and process in clumps

teckk · 08-11-2019, 10:16 AM

That's what post 8 does. Takes 100's-1000's of files and puts them into 10 item array. Then do what you want with each array. So that you don't reach a item limit.

Loop through the arrays and do the same thing with each.

teckk · 08-11-2019, 10:30 AM

Quote:

So I am trying to put together a function that splits up the paths/filename into dynamically created arrays

Code:

files=(/my/path/a.txt
/my/path/b.txt
/my/path/c.txt
/my/path/d.txt
/my/path/e.txt
/my/path/f.txt
/my/path/g.txt
/my/path/h.txt
/my/path/i.txt
/my/path/j.txt)

#Something will only do ARG_MAX=2

array1="${files[@]:0:2}"
echo "$array1"
/my/path/a.txt /my/path/b.txt

array2="${files[@]:2:2}"
echo "$array2"
/my/path/c.txt /my/path/d.txt