A command for making this organization from a dir with a lot of files?

dedec0 · 04-05-2019, 12:37 PM

Hello,

I have a directory called file.set which contains 1000 thousand files (the number may vary), and we may assume no subfolder exist. I want to organize these files in directories in new folders, with at most 50 files (number may also vary). These directories should be created inside file.set with names "001.050", "051.100", etc.

Do you know a command for this? Or writing a shell script is the best option?

scasey · 04-05-2019, 12:52 PM

I think you'd need to write a script to do that using mkdir and mv commands.

dedec0 · 04-05-2019, 02:08 PM

I am creating a script now using Bash. I know (and have used before) that I can create a 'for' like this, to iterate on each file on a dir, ordered by name:

Code:

# list the $path dir
for fileName in $path/*; do
    echo $fileName
done

This 'for' is not appropriate for the algorithm i am using, though.

How do I obtain an array with each file in a position?

files=$path/*

and

files="$path/*"

do not work.

Or my problem is how i am trying to access the file names? With:

Code:

echo -e "mv $files[$n] $dest"         # where n is a var with a number

dedec0 · 04-05-2019, 03:30 PM

Quote:

Originally Posted by dedec0

I am creating a script now using Bash. I know (and have used before) that I can create a 'for' like this, to iterate on each file on a dir, ordered by name:

Code:

# list the $path dir
for fileName in $path/*; do
    echo $fileName
done

This 'for' is not appropriate for the algorithm i am using, though.

How do I obtain an array with each file in a position?

files=$path/*

and

files="$path/*"

do not work.

Or my problem is how i am trying to access the file names? With:

Code:

echo -e "mv $files[$n] $dest"         # where n is a var with a number

Is there something better than navigating throw all files, building an array, like

Code:

arqs=()
for i in $dir/*; do
    arqs+=("$i")
done

does?

scasey · 04-05-2019, 05:25 PM

Nothing wrong with a for loop, IMO.
Assuming that $path is populated correctly:

Code:

for file in $(ls -1 $path/) ;do 
     echo $file
done

That's a -one, not a -ell
Note that $file only contains the name of the file itself.

On the other hand, I guess I don't really understand how your

Code:

for fileName in $path/*

even works at all...but my comment about the for loop stands...
It will iterate through the file names for you just fine.

dedec0 · 04-05-2019, 06:18 PM

I have made a script. But it has a problem i could not solve, maybe you can help.

This is its code:

Code:

#!/bin/bash
#   org: Organiza uma pasta com vários arquivos, criando várias subpastas com
# um número máximo de arquivos

# Versão: 2019,04,05,19,25,00 UTC-3
# Version that only prints the commands that would be done

# Print help
ajuda()
{
    echo Uso:
    echo -e "\torg [pasta]\n"
    echo -e   "Argumento:\n"
    echo -e   "\t[pasta]\t\ta pasta que contém vários arquivos a serem organizados.\n\t"
    return
}

# Processa argumento, possivelmente decidindo o diretório escolhido
if  (( ($# == 0) )); then
    ajuda
    exit 0
fi
if (( $# == 1 )); then
    if  [[ $1 == '-h' ]] ||
	[[ $1 == '--help' ]] ||
        [[ $1 == '--ajuda' ]];
	then 
        ajuda
	exit 0
    else
	dir="$1"
    fi
else
    dir="."
fi
#echo dir=$dir

# There is no better way to make an array with each file in dir?
arqs=()
for i in $dir/*; do
    arqs+=("$i")
done
#numArqs=${#arqs[@]}
#echo $numArqs arquivos
#arqs=$dir/*

n=0 				# In each folder, the number of a file
#qPorPasta=50;			# Files per dir (max)
qPorPasta=5;			# Files per dir (max)

#echo $qPorPasta
max=$qPorPasta
for (( max=qPorPasta; max<${#arqs[@]}; max+=qPorPasta )); do
    dest=`printf "%03i" $n`
    dest="$dest."
    dest="$dest"`printf "%03i" $max`
    #echo dest = \"$dest\"
    echo mkdir $dest
    #mkdir $dest
    for((; n<max; n++)); do
	if [[ $n -lt ${#arqs[@]} ]]; then
	    echo -e "mv ${arqs[n]} $dest"
	    #mv ${arqs[n]} $dest
	else
	    echo max=$max
	    for(( max+=qPorPasta; n<max; n++)); do
		if [[ $n -lt ${#arqs[@]} ]]; then
		    echo -e "mv ${arqs[n]} $dest"
		    #mv ${arqs[n]} $dest
		fi
	    done
	    break 3;
	fi
    done
done

exit 0

And these commands may help you making a test like mine:

Code:

cd /dev/shm mkdir t; cd t;
# 23 files from '00' to '22':
for (( i=0; i<23; i++)); do touch `printf "%02d" $i`; done

Executing the script in the t dir does not move the last set of files. The output (which would be the commands done) is this:

Code:

/dev/shm/org .    # 'org' is the script
mkdir 000.005
mv ./00 000.005
mv ./01 000.005
mv ./02 000.005
mv ./03 000.005
mv ./04 000.005
mkdir 005.010
mv ./05 005.010
mv ./06 005.010
mv ./07 005.010
mv ./08 005.010
mv ./09 005.010
mkdir 010.015
mv ./10 010.015
mv ./11 010.015
mv ./12 010.015
mv ./13 010.015
mv ./14 010.015
mkdir 015.020
mv ./15 015.020
mv ./16 015.020
mv ./17 015.020
mv ./18 015.020
mv ./19 015.020

BW-userx · 04-05-2019, 06:34 PM

this is a script I use to devide up x amount of dirs and putting all of the parent sud-directores into a set up to 4 other parent dirs.

Code:

#!/bin/bash


working_dir=/run/media/userx/1TB/ARedoneMusic_Sorted

move=( /run/media/userx/1TB/Music_2_do1 /run/media/userx/1TB/Music_2_do2 /run/media/userx/1TB/Music_2_do3 /run/media/userx/1TB/Music_2_do4 )
#move=( /run/media/userx/2TBInternal/Music_2_do1 /run/media/userx/2TBInternal/Music_2_do2 /run/media/userx/2TBInternal/Music_2_do3 /run/media/userx/2TBInternal/Music_2_do4 )
t=0



#remove empty directories from working working_dir

while read d ;
do
  rmdir -v "$d"
done <<< "$(find "$working_dir" -type d)"



mkdir -pv /run/media/userx/1TB/Music_2_do1
mkdir -pv /run/media/userx/1TB/Music_2_do2
mkdir -pv /run/media/userx/1TB/Music_2_do3
mkdir -pv /run/media/userx/1TB/Music_2_do4



echo
totalDir="$(find "$working_dir"  -maxdepth 1 -mindepth 1   -type d  | wc -l)"

divDir=$(($totalDir / 4))

echo "Total DIr         Div/4
$totalDir               $divDir"

countDn=0

    while read f
    do
                    if  [[ $countDn -le $divDir ]] ; then
                    {
                        mv -f  "$f" "${move[$t]}"
                        ((countDn++))
                    }
                    fi

        echo "t $t"

        [[ $countDn = $divDir ]] && { ((t++)) ; countDn=0 ; }

     [[ $t -eq 4 ]] &&
     { echo "$t $countDn $divDir" ; exit ; }



done <<< "$(find "$working_dir" -maxdepth 1 -mindepth 1 -type d)"

basically what you want to do is something similar like run a loop and a count on files, if x amount of files reached then create a new dir then start putting files into that one, reset count then loop until same x amount reached then create another dir then repeat process.

Code:

#!/bin/bash

working_dir=/whatever/
move2=/destanation->

count=0
newDir=0
xamount=100

#create 1st dir
mkdir "$move2"_"$newDir"

while read f
do

mv -f "$f" "$move2"_"$newdir"
((count++))

#reset count add 1 to dir name, create new dir
[[ "$count" -eq $xamount ]] && { count=0 ; ((newDir++)) ; mkdir "$move2"_"$newDir" ; }

done <<<"$(find "$working_dir" -type f )"

adds a number to end of whatever you name your dirs you want your files in.

not tested.

rereading that last post, question?
you are reading all of the files first then putting the file path/name into an array first, then using that array to get the x amount of files and moving them into a different directory then getting the next set of files and doing the same until you're out of files?

the way I wrote it, it just going though the dir, putting the files into dir created dynamically as it loops through the files, when it gets to the end it stops no matter how many files left because it goes until eof.

scasey · 04-05-2019, 06:34 PM

You're processing only when the count is less than the number of entries in the array, so you're not processing the last entry. Try:

Code:

for (( max=qPorPasta; max<${#arqs[@]+1}; max+=qPorPasta )); do

or

Code:

for (( max=qPorPasta; max<=${#arqs[@]}; max+=qPorPasta )); do

Not sure if that's the only issue.

allend · 04-05-2019, 10:54 PM

I would suggest adding a counter to the loop so that a remainder test can be used for branching.

Code:

j=50
for i in {1..501}; do
  if (( $i % 50 == 0 )); then
    j=$(( $i + 50 ))
  elif (( $i % 50 == 1 )); then
    echo
    echo "$i-$j"
    # make next directory here
  fi
  echo -n "$i "
done