[SOLVED] bash select items not in array

eamesj · 01-02-2013, 12:43 PM

Hi all,

I'm building an array in bash and i would like to perform actions on everything that is NOT in the array

#! /bin/bash
while read -r input ; do
[[ $input == quit ]] && break
array+= ( "$input" )
done
for index (NOT)in ${!array[@]}; do
do something
done

anyone know how i can do this?
thanks,

millgates · 01-02-2013, 03:40 PM

Maybe a stupid question, but what are the items/indices that are "not in array"?

grail · 01-02-2013, 11:26 PM

Firstly, please use [code][/code] tags around your code.

The way you have constructed the array it will have no holes in it so simply use the size of the array and all numbers after that, which will be an infinite for loop.

eamesj · 01-03-2013, 10:28 AM

Sorry, I suppose it wasn't clear enough.

normal coding... create array and do something to every item in the list...

Code:

#! /bin/bash
echo "enter items into list to process, use new line for multiple items"
# create array of items. quit exits list builder
while read -r input ; do
   [[ $input == quit ]] && break 
   array+= ( "$input" )
done
# do something to all items in array
for index in ${!array[@]}; do
   do something 
done

- i would like to be able to do something in loop for items not in the array. for example a list of directories, I would like the user to make a list of items (directories) to exclude and perform an action on all directories except for the ones in the list.

so id like something like...

Code:

#! /bin/bash
echo "enter items into list to NOT process, use new line for multiple items"
# create array of items. quit exits list builder
while read -r input ; do
   [[ $input == quit ]] && break 
   array+= ( "$input" )
done
# do something to all items not in array
for index notin ${!array[@]}; do
   do something 
done

I'm not sure that an array like this is the best way to go about this so if anyone has any suggestions id appreciate it.

Hope that's a bit clearer

Grail, afraid i'm really not sure what you mean by having 'holes' in an array can you clarify your "use the size of the array and all numbers after that, which will be an infinite for loop." ?

thanks,

millgates · 01-03-2013, 11:43 AM

The problem is, you can only iterate over something that is in a list. To do something with everything not in a list is like if I told you to name all things in the universe with the exception of say, my car, your dog and a pencil on grail's table. The question only makes sense if you have two lists - then you may for example want to do something for all items in list1 except for those that are also in list2.
So, for example to do something with all files in a directory that are not in list you may reason as follows:

Code:

for fname in *; do 
    if (fname is not in list); then
        do_something;
    fi
done;

grail · 01-03-2013, 11:49 AM

Quote:

Grail, afraid i'm really not sure what you mean by having 'holes' in an array can you clarify your "use the size of the array and all numbers after that, which will be an infinite for loop." ?

Part of your code says:

Code:

for index notin ${!array[@]}; do

By asking for indexes not in the array you either create an array with indexes missing, ie 0,1,3,6,7,9
This would then mean you have "holes" at positions - 2,4,5,8 and nothing in any index after 9

However, you create your array using the following:

Code:

array+= ( "$input" )

So unless this is an existing array, you are not adding to one that already has "holes" hence there are none and all up until the last index will have data associated with it.
This then implies that the size of the array, found using:

Code:

${#array[*]}

would give you the size of the array which would also be the point from which the last index would have no data and so it and all after it (to infinity)
would be the numbers you are looking for (although I would guess this is not the case)

So your current example seems to be the same as the original so I am still at a loss as to what it is you actually wish to do?

On reading your question, it sounds like you may need to arrays / lists and whatever is not in one of them needs to be dealt with???

danielbmartin · 01-03-2013, 12:13 PM

One way to do this is adopt the convention that any entry which is prefixed with a dash (minus sign) should not be processed.

Try this ...

Code:

# Fill the array named "list" with user inputs.
echo "Enter list items, one per line, then enter <null> to escape..."  
    echo "Prefix any list item  with dash (-) to signify 'no processing'." 
for (( j=1; j<9999; j++ ))
  do
    # Solicit user response
    read reply
    if [ -z "$reply" ]   # Test for null entry
        then break
    fi
    list[$j]=$reply  # Save the user response
  done
 
# Regurgitate the list contents.
echo; echo "These are your orders..."
for (( j=1; j<=${#list[@]}; j++ ))
  do
    if [ '-' == ${list[$j]:0:1} ]   # Test first character
       then echo "DON'T process "${list[$j]:1}
       else echo "DO process " ${list[$j]}
    fi
  done

My test results ...

Code:

Enter list items, one per line, then enter <null> to escape...
Prefix any list item  with dash (-) to signify 'no processing'.
able
-baker
charlie


These are your orders...
DO process  able
DON'T process baker
DO process  charlie

Daniel B. Martin

eamesj · 01-03-2013, 01:18 PM

ok, so if i understand correctly. Going with the directory example (perform an action on all directories except those in a list specified by the user)

I would need to create an array of all of the sub-directories within a given directory

e.g.
/home/users/user1
/home/users/user2
/home/users/user3
/home/users/user4

have the script ask for directories not to be processed and create an array out of it:

Code:

#! /bin/bash
# create an array of all directories..
declare -a dirs
i=1
for d in /home/users/*
do
   dirs[i++]="${d%/}"
done

echo "enter items into list to process, use new line for multiple items"
# create array of items. quit exits list builder
while read -r input ; do
   [[ $input == quit ]] && break 
   array+= ( "$input" )
done

and then compare one array with the other ?

How can i compare the arrays? or there a better method of doing this?

eamesj · 01-03-2013, 02:13 PM

that'll work like a charm, thanks danielbmartin... very tidy

eamesj · 01-03-2013, 02:53 PM

I jumped the gun a little in making this solved...

although that solution works fune, its a bit of a pain for the user to enter all of the directories if there are a lot. Right now I've got two arrays, one for the complete set of directories (dirs) and one listbuilder (list). Is there a way to compare one with the other so that if an entry in (dirs) does not exist in (list) then add a '-' and therefore not process?

Code:

declare -a dirs
i=1
for directory in /home/users/*/
do
    dirs[i++]="${directory}"
done
echo "There are ${#dirs[@]} dirs in the current path"
for((i=1;i<=${#dirs[@]};i++)); do
    echo "${dirs[i]}"
done

echo -e "\n\n"

# Fill the array named "list" with user inputs.
echo "Enter list items, one per line, then press enter <null> to escape..."  
    echo "Prefix any list item  with dash (-) to signify 'no processing':" 
for (( i=1; i<=${#dirs[@]}; i++ )); do

# Solicit user response
    read reply
    if [ -z "$reply" ]   # Test for null entry
        then break
    fi
    list[$i]=$reply  # Save the user response
  done

# Regurgitate the list contents.
echo; echo "These are your orders..."
for (( i=1; i<=${#list[@]}; i++ )); do
    if [ '-' == ${list[$i]:0:1} ]   # Test first character
    then echo "DON'T process "${list[$i]:1}
    else echo "DO process " ${list[$i]}
    fi
done

danielbmartin · 01-03-2013, 02:58 PM

Quote:

Originally Posted by eamesj

Right now I've got two arrays, one for the complete set of directories (dirs) and one listbuilder (list). Is there a way to compare one with the other so that if an entry in (dirs) does not exist in (list) then add a '-' and therefore not process?

Perhaps you are seeking the comm command.

Code:

daniel@daniel-desktop:~$ comm --help
Usage: comm [OPTION]... FILE1 FILE2
Compare sorted files FILE1 and FILE2 line by line.

With no options, produce three-column output.  Column one contains
lines unique to FILE1, column two contains lines unique to FILE2,
and column three contains lines common to both files.

  -1              suppress lines unique to FILE1
  -2              suppress lines unique to FILE2
  -3              suppress lines that appear in both files

  --check-order     check that the input is correctly sorted, even
                      if all input lines are pairable
  --nocheck-order   do not check that the input is correctly sorted
  --output-delimiter=STR  separate columns with STR
      --help     display this help and exit
      --version  output version information and exit

Note, comparisons honor the rules specified by `LC_COLLATE'.

Daniel B. Martin

millgates · 01-03-2013, 03:22 PM

Quote:

Originally Posted by danielbmartin

One way to do this is adopt the convention that any entry which is prefixed with a dash (minus sign) should not be processed.

This is quite nice but I fail to see how this solves the issue. You still have to manually mark all items you don't want to process. Why would you even write those entries to stdin?

Code:

declare -a dirs
i=1
for directory in /home/users/*/
do
    dirs[i++]="${directory}"
done

the same can be done as

Code:

dirs=(/home/users/*/)

Do you need those arrays stored anyway?
Also, maybe you should tell us what are you trying to achieve in the first place. Then we can recommend you a better tool for the job. What are the two lists you have? Are they in files? What do you want to do with all of them?

another example:

Code:

#!/bin/bash

declare -A exclude;

echo "enter filenames to exclude from processing, one per line, end with Ctrl+D:"

while read -r line; do
    exclude["$line"]=" "
done

for fname in *; do
    [[ -z ${exclude["$fname"]} ]] && do_something_with_file
done

danielbmartin · 01-03-2013, 05:11 PM

Perhaps you will find this snippet useful.

InFile1 ...

Code:

alan.txt
barney.txt
charlie.txt
david.txt
edgar.txt
frank.txt

InFile2 ...

Code:

alan.txt
charlie.txt
frank.txt

Code ...

Code:

# Display contents of InFile1.
echo; echo "InFile1 ..."; cat $InFile1

# Display contents of InFile2.
echo; echo "InFile2 ..."; cat $InFile2

echo
echo "Identify files which exist in InFile1 and *not* in InFile2,"
echo " and prefix them with a dash."
comm -2 $InFile1 $InFile2 |sed 's/^/-/; s/-\t//'

My test results ...

Code:

InFile1 ...
alan.txt
barney.txt
charlie.txt
david.txt
edgar.txt
frank.txt

InFile2 ...
alan.txt
charlie.txt
frank.txt

Identify files which exist in InFile1 and *not* in InFile2,
 and prefix them with a dash.
alan.txt
-barney.txt
charlie.txt
-david.txt
-edgar.txt
frank.txt

Daniel B. Martin