looping rsync to rename files at destination help?

lleb · 03-23-2023, 05:24 PM

Thank you for reading.

My goal is to take a large number of files from directory A into directory B, not always on the same server, using rsync and set a specific name at destination that is numbered.

Code:

#!/bin/bash

for i in *
do
	i=1
	rsync -aviS --progress *.avi /mnt/Exports/Backup/TEST/test.S01E0${i}
	i=i+1
done

The extension is not important, its just what im testing with atm.

the output that the above is generating is not what i desire:

1. it creates a new subdirectory under /mnt/Exports/Backup/TEST/ called test.S01E01
2. after creating the directory, that it should not be creating according to the rsync command, it is than moving the files from A to B/test.S01E01/, nothing is renamed in the process.

When I manually do this with:

Code:

rsync -aviS --progress foo\ \bad\ chars.avi /path/to/new/foo.no.bads.avi

it does exactly that into directory B as desired. This is time consuming and something that should be automated. I am horrid at coding, but thought a simple for loop should do the trick, my for loop is bad and wrong.

Looking for better guidance than RTFM or google as I've already spent far to much time RTFM and as not a programmer, not fully following examples, and my google foo fails as I dont know what to ask to get the correct output.

Thank you in advance for guidance, education, and direction.

michaelk · 03-23-2023, 05:36 PM

Code:

for FILE in *.avi
do
  FILE_fixed=$(remove bad characters)
  rsync -aviS  "$FILE" /path/to/new/"$FILE_fixed"
done

Here is the basic loop... There are many ways to fix bad characters... Can you post an example?

lleb · 03-23-2023, 06:05 PM

michaelk,

I have either ran a bit more complicated code that a friend made for me, i will post that in a bit, or just the manual rsync command above. What I want to avoid is renaming the files before the rsync command, source files in directory A need to stay are they are named.

rename code

Code:

#!/bin/bash

# loop to scan entire directory for all file extension
# remove original file name and number 0001.ext - XXXX.ext
# using BASH 4.0 or greater should be able to use 'tr' to convert from
# uppercase to lowercase letters in the file
# Code from friend <25 July, 2021>
# NOTE: this will also move directories to lowercase and NN.####.foo
# ex:  Old/  will move to NN.####.old/

###########################################################################
N=1
for OLDNAME in *	# OLDNAME is the original name of the file
do
    # convert from upper to lower case
    LOWER="$( echo "${OLDNAME}" | tr '[:upper:]' '[:lower:]')"
    # EXT is file extension to be left in place
    EXT="${LOWER##*.}"
    # check if file has an extension, if not do nothing and move on
    if [[ -n "$EXT" ]]
    then
        EXT=".$EXT"
    fi
    # NEWNAME will be 2 padded 0's with original file type 0001.ext  
    NEWNAME="$( printf '%02d%s' $N "$EXT" )"
    #NEWNAME="$( printf '%04d.%s' $N "$EXT" )"
    # if loop to check that the original file name is not already 0001.ext
    # if not !=, than perform the mv function to 0001.ext
    if [[ "${OLDNAME}" != "${NEWNAME}" ]]
    then
        mv -v -- "${OLDNAME}" "${NEWNAME}"
    fi
    N=$((N+1))
	
done
###########################################################################

# Code updated with help from friend <July 25, 2021>
# convert webp to jpg and remove original file if not error using &&
# must have ffmpeg installed on system for these lines to work
for x in ls *.webp; do
	ffmpeg -loglevel panic -i "$x" "${x%.webp}".jpg && rm "$x"
done

# convert webm to mkv and remove original file if not error using &&
for x in ls *.webm; do
	ffmpeg -loglevel panic -i "$x" "${x%.webm}".mkv && rm "$x"
done
###########################################################################
# with new information from above code provided by friend
# removed multiple loops and replaced with single loop below
# me July 25, 2021

# prepend file XXXX.ext with NN.
# replace NN. with correct file name
for f in *
do
    mv "$f" "NN$f" 
done
###########################################################################

# move *.sh back to Numbers.sh and *.txt to grab.txt
# NOTE: make sure only 1 each .txt and .sh file in the directory or this
#       will fail
mv *.sh Numbers.sh
# mv *.txt grab.txt
###########################################################################

This is a much bigger hammer than I want to take to the files in directory A. Currently I will rsync *.(extension type) /path/to/destination/ than i will modify the above script to rename and number everything in directory B.

When the number of files to move is limited I will do so as the way I typed in the OP. Thus my desire to use a for loop with rsync to move the files. That should, in theory, both clean up the files in A as well as provide the desired output and ordered numbering in B.

michaelk · 03-23-2023, 06:20 PM

If that rename section of code still works for you then just replace the mv with rsync. Pull that one section of code to a new script if I understand what you are trying to accomplish.

Code:

    if [[ "${OLDNAME}" != "${NEWNAME}" ]]
    then
        mv -v -- "${OLDNAME}" "${NEWNAME}"
    fi

Replace with

Code:

rsync ... "${OLDNAME}" /path/to/new/"${NEWNAME}"

lleb · 03-23-2023, 06:55 PM

that seems to be renaming before it moves the files to B. I am sending from A to B if that matters.

I have this 99.96% working

Code:

#!/bin/bash

###########################################################################
N=1
for OLDNAME in *	# OLDNAME is the original name of the file
do
    # convert from upper to lower case
    LOWER="$( echo "${OLDNAME}" | tr '[:upper:]' '[:lower:]')"
    # EXT is file extension to be left in place
    EXT="${LOWER##*.}"
    # check if file has an extension, if not do nothing and move on
    if [[ -n "$EXT" ]]
    then
        EXT=".$EXT"
    fi
    # NEWNAME will be 2 padded 0's with original file type 0001.ext  
    NEWNAME="$( printf '%02d%s' $N "$EXT" )"
    SNEWNAME=Name."${NEWNAME}"
    
    if [[ "${OLDNAME}" != "${NEWNAME}" ]]
    then
    	rsync -aviS --progress "${OLDNAME}" /path/to/foo/"${SNEWNAME}"
    fi
    N=$((N+1))
done

question, so from what i understand reading this code, if a file has no extension it should be ignored. when i ran this script it moved a file without an extension to B directory? why? What did I goof?

FYI, you rock michaelk.

michaelk · 03-23-2023, 07:26 PM

Code:

#!/bin/bash

###########################################################################
N=1
for OLDNAME in *	# OLDNAME is the original name of the file
do
    # convert from upper to lower case
    LOWER="$( echo "${OLDNAME}" | tr '[:upper:]' '[:lower:]')"
    # EXT is file extension to be left in place
    EXT="${LOWER##*.}"
    # check if file has an extension, if not do nothing and move on
    if [[ -n "$EXT" ]]
    then
        EXT=".$EXT"
    else
        # no extension go to next file
        continue
    fi
    # NEWNAME will be 2 padded 0's with original file type 0001.ext  
    NEWNAME="$( printf '%02d%s' $N "$EXT" )"
    SNEWNAME=Name."${NEWNAME}"
    
    rsync -aviS --progress "${OLDNAME}" /path/to/foo/"${SNEWNAME}"

    N=$((N+1))
done

I don't think you need the if oldname = newname but if it does not work as desired you can always add it back in. If no extension I added an else which executes a continue. That basically skips the rest of the code in the loop and therefore the file is not copied.

I consider myself a novice programmer despite my experience...

lleb · 03-23-2023, 08:04 PM

hmm, seems better. It is still moving to script over as NEWNAME.z and for some odd reason the rm command is not removing said file from B

Code:

#!/bin/bash
PATH=foo
###########################################################################
N=1
for OLDNAME in *        # OLDNAME is the original name of the file
do
    # convert from upper to lower case
    LOWER="$( echo "${OLDNAME}" | /usr/bin/tr '[:upper:]' '[:lower:]')"
    # EXT is file extension to be left in place
    EXT="${LOWER##*.}"
    # check if file has an extension, if not do nothing and move on
    if [[ -n "$EXT" ]]
    then
        EXT=".$EXT"
    else
        # contineu if no extension for file
        continue
    fi
    # NEWNAME will be 2 padded 0's with original file type 0001.ext
    NEWNAME="$( printf '%02d%s' $N "$EXT" )"
    SNEWNAME=Name"${NEWNAME}"

    if [[ "${OLDNAME}" != "${NEWNAME}" ]]
    then
        /usr/local/bin/rsync -aviS --progress "${OLDNAME}" $PATH"${SNEWNAME}"
    fi
    N=$((N+1))
done
##########################################################################
/usr/rm ${PATH}*.z

Also for whatever reason after adding that slight change of continue the system demanded full pathing. should have done that from the start i suppose.

michaelk · 03-23-2023, 10:24 PM

Quote:

rsync -aviS --progress foo\ \bad\ chars.avi /path/to/new/foo.no.bads.avi

What do you want the new file name format to look like?

lleb · 03-24-2023, 07:24 AM

Quote:

Originally Posted by michaelk

What do you want the new file name format to look like?

Sorry for the delay in reply. The output of the new script is working for the most part. Only parts not working are:

1. it is moving a copy of the script over that i have named 'z' no extension
2. even with the script portion that is supposed to ignore files without extensions it is still touching all files
3. knowing that, the last line should clean up the mess and it is not.

a clean format will be a name without special characters or spaces, numbered in order, with the same extension as the original file.

michaelk · 03-24-2023, 08:58 AM

Code:

    # replace spaces with underscore
    OLDNAME="${OLDNAME// /_}"
    remove non printable characters and convert to lowercase
    LOWER=$( echo "${OLDNAME}" | tr -dc '[:print:]' | tr -dc [:alpha:]_. | tr '[:upper:]' '[:lower:]')
    # EXT is file extension to be left in place
    EXT="${LOWER##*.}"
    # strip base name
    base="${LOWER%%.*}"
    # check if file has an extension, if not do nothing and move on
    if [[ ! "$LOWER" == "$EXT" ]]
    then
        EXT=".$EXT"
        echo "ext=$EXT"
    else
        continue
    fi
    
    NEWNAME="$( printf '%s%02d%s' $base $N "$EXT" )"
 
    echo "$NEWNAME"

Not sure how you want to "string" the new name together.

MadeInGermany · 03-28-2023, 08:46 AM

There is a bug in

Code:

    EXT="${LOWER##*.}"
    # check if file has an extension, if not do nothing and move on
    if [[ -n "$EXT" ]]

$EXT becomes $LOWER if there is no match (no dot)!
The fix was already in the previous post.

Code:

    EXT="${LOWER##*.}"
    # check if file has an extension, if not do nothing and move on
    if [[ "$LOWER" != "$EXT" ]]

BTW shouldn't both EXT,base variables take the rightmost dot? Then it must be

Code:

    EXT="${LOWER##*.}" # Maximum match from the left
    # strip base name
    base="${LOWER%.*}" # Minimum match from the right