automatically renamiremoving illegal characters recrusively

wallagh · 04-27-2010, 06:06 PM

Hello,

I would like to replace all illegal WIndows characters in filenames, recursively. They should be replaces with underscore _
The illegal characters are:
? < > \ : * | ”

The reason being, that I can not backup my music files (genrated based on tags on an embedded linux music server) to a WIndows based Home server, because the filenames contain illegal characters (that is: for WIndows illegal, not for Linux).

Can anybody help me?
Btw. my "linux"system has only bash, nothing fancy.

Regards,

Serge W.

jschiwal · 04-27-2010, 07:32 PM

You can use character classes in your variable expansion.

mv -v "$file" "${file//[*?<>\\]/_}"

The backslash needs to be escaped with a backslash. The `^' character must not be the first character in the list if it is one of the characters to match. The `]' character needs to be the first character in the list or preceded with a backslash to prevent it from ending the list.

wallagh · 04-28-2010, 06:00 AM

Does thsi work recursively into all sub directories?

Quote:

Originally Posted by jschiwal

You can use character classes in your variable expansion.

mv -v "$file" "${file//[*?<>\\]/_}"

The backslash needs to be escaped with a backslash. The `^' character must not be the first character in the list if it is one of the characters to match. The `]' character needs to be the first character in the list or preceded with a backslash to prevent it from ending the list.

catkin · 04-28-2010, 08:36 AM

No -- that command moves a single file as in mv old_name new_name, all the -v does is make it verbose so it tells you what it is doing.

To apply it to all the files in a directory and its subdirectories and its subdirectories ... and its subdirectories the most robust technique (replace $dir as required) is

Code:

while read -r -d '' file
do
   mv -v "$file" "${file//[*?<>\\]/_}"
done < <(find $dir -type f -print0)

David the H. · 04-28-2010, 12:05 PM

I'd like to politely point out that the batch renaming of files is an exceedingly common request that has been repeatedly brought up and answered both here and elsewhere on the net. A few minutes with the LQ site search and/or Google would certainly have given you all the information you need to accomplish your goal.

Thus it's generally a good idea to first perform a few searches to discover what information is already available before starting a new thread.

Thank you.

wallagh · 04-28-2010, 01:56 PM

I understand you get tiered of questions being asked over and over again. However: I have been looking for an answer for quit a while. I have found many answers, but none was to the exact question that I had
So it was no lazyness. Perhaps ignorance....

Quote:

Originally Posted by David the H.

I'd like to politely point out that the batch renaming of files is an exceedingly common request that has been repeatedly brought up and answered both here and elsewhere on the net. A few minutes with the LQ site search and/or Google would certainly have given you all the information you need to accomplish your goal.

Thus it's generally a good idea to first perform a few searches to discover what information is already available before starting a new thread.

Thank you.

David the H. · 04-29-2010, 09:34 AM

I understand. But in that case you should make it clear that you've already done some work. Perhaps show us some of the things that you've already tried. Most of us are delighted to help out people who demonstrate that they've given it their best shot, but just can't seem to get it to work out.

There really are many ways to go about this. There are several cli (and gui) batch renaming tools, for example, including at least two called simply "rename". You haven't mentioned what distro you're using, but the perl package on Debian-based distros includes a "rename" that uses the standard "s///" replacement structure.

You could combine this with find to target files recursively, for example.

Code:

find ~./musicdir -name "*.mp3" -exec rename -v 's/[?<>\\:*|"]/_/g' '{}' \+

wallagh · 04-30-2010, 04:04 AM

Quote:

Originally Posted by David the H.

I understand. But in that case you should make it clear that you've already done some work. Perhaps show us some of the things that you've already tried. Most of us are delighted to help out people who demonstrate that they've given it their best shot, but just can't seem to get it to work out.

There really are many ways to go about this. There are several cli (and gui) batch renaming tools, for example, including at least two called simply "rename". You haven't mentioned what distro you're using, but the perl package on Debian-based distros includes a "rename" that uses the standard "s///" replacement structure.

You could combine this with find to target files recursively, for example.

Code:

find ~./musicdir -name "*.mp3" -exec rename -v 's/[?<>\\:*|"]/_/g' '{}' \+

The Linux version that I use is embedded on my Hifidelipor music server. It simply states "linux version 2.4.26").
It does not have CC, or rename. It does have Bash.

I have tried many combinations of the sugestions, but not seem to work 100%. Most of the time, the script starts to complain about "can not rename directory" of "can not rename file".
SO in the end it is only 70% effective....
BTW I have also "replaced ":" for "\"" to replace the quot instead of the :

The latest script that I have been trying is:

Code:

#! /bin/bash


ONE=1                     # For getting singular/plural right (see below).
number=0                  # Keeps track of how many files actually renamed.
FOUND=0                   # Successful return value.
while read -r -d '' filename
do                                #      
echo "$filename" | grep -q ":"         #  Check whether filename
     if [ $? -eq $FOUND ]                   # contains :
              then
               fname=$filename                      # Strip off path.
               n=`echo $fname | sed -e "s/:/_/g"`   # Substitute dash f
               mv "$fname" "$n"                     # Do the actual rena
               let "number += 1"

	fi

done < <(find $dir -type f -print0)

if [ "$number" -eq "$ONE" ]                 # For correct grammar.
             then
               echo "$number file renamed."
             else
               echo "$number files renamed."
fi

exit 0

jschiwal · 05-05-2010, 01:01 AM

Look at your program as having two parts. First to select files that need renaming. And second, the renaming itself.

Your last example only changes a single character. Perhaps if you posted what some of the filenames were, or posted which illegal characters need renaming might be of help. The mv command I posted took a guess.

Also post a name of a file that didn't get renamed.

catkin · 05-06-2010, 04:50 AM

This might do the job

Code:

#! /bin/bash

dir=$1
if [[ ! -d $dir ]];then
    echo "Directory '$dir' does not exist or is not reachable" >&2
    exit 1
fi

number=0                  # Keeps track of how many files actually renamed.
while read -r -d '' filename
do
    new_filename=$filename
    for char in '?' '<' '>' '\' ':' '*' '|' '"'
    do
        eval new_filename='${new_filename//\'$char'/_}'
    done
    if [[ $new_filename != $filename ]]; then
        if [[ -e "$new_filename" ]]; then
            echo "$new_filename already exists.  Skipping $filename" >&2
        else
            echo mv "$filename" "$new_filename"  # Remove echo after testing
            [[ $? ]] && let number++
        fi
    fi
done < <(find $dir -type f -print0)

case $number in
    1 )
        echo "$number file renamed."
        ;;
    * )
        echo "$number files renamed."
esac

exit 0

Bash's eval command is relatively slow so if there are many files to rename it may be worth experimenting to find out if it is faster to use tr once rather than the loop

Code:

    for char in '?' '<' '>' '\' ':' '*' '|' '"'
    do
        eval new_filename='${new_filename//\'$char'/_}'
    done

wallagh · 05-07-2010, 05:23 AM

Quote:

Originally Posted by jschiwal

Look at your program as having two parts. First to select files that need renaming. And second, the renaming itself.

Your last example only changes a single character. Perhaps if you posted what some of the filenames were, or posted which illegal characters need renaming might be of help. The mv command I posted took a guess.

Also post a name of a file that didn't get renamed.

I think you are right about the "single character" change: I already discovered that filenames with >1 illegal character failed!

The solution that I find was.... change those by hand..... About 150 filenames, so it took me a night to do so.

ANyway thanks for all the help!

MTK358 · 05-07-2010, 08:03 AM

I don't know how to recursively iterate over files, but this should rename them:

Code:

newname=$(dirname "$oldname")'/'$(basename "$oldname" | sed 's/[?<>\\:|*"]/_/g')

if [ -e "$newname" ]
then
	echo 'error: file '"$newname"' already exists'
	echo -n '[C]lobber, [S]kip (default), or [A]bort: '
	read input
	if [ "$input" = 'c' ]
	then
		mv "$oldname" "$newname"
	elif [ "$input" = 'a' ]
	then
		exit 0
	fi
else
	mv "$oldname" "$newname"
fi