Avoiding name clashes when batch-renaming files

Bob.A-12 · 01-05-2015, 10:11 AM

Hello,

I'm new to this forum and am hoping someone could help me with a scripting issue I'm having. I'm trying to write a script that'll rename all my photos according to the date/time taken provided by EXIF;

e.g. 100_1234.jpg -> 20150101_203015.jpg

I have 253 JPGs in my working directory and when I run the script, the files are renamed properly but I end up with only 239 JPGs. When the script is run, files are being overwritten due to name-clashes with other files but this isn't something I know how to fix. Is it possible to have duplicates renamed instead of overwritten, for example, as;

-Two different files with the same EXIF data-

100_1234.jpg -> 20150101_203015.jpg (First file)
101_2335.jpg -> 20150101_203015_1.jpg (Second file)

I have tried both "mv --backup=numbered" and "set -o noclobber" switches but the overwrites still occur. Apologies if this is worded badly and many thanks!

rknichols · 01-05-2015, 11:16 AM

The "--backup=numbered" option in the mv command should prevent losing files, but of course that previously duplicated name will no longer end with a ".jpg" extension (it will be, for example, "xxx.jpg.~1~) and your programs that deal with photos probably won't recognize it.

Setting the shell's "noclobber" option with "set -o noclobber" won't affect the operation of the mv command. That option just keeps the shell itself from overwriting files when you use output redirection ("echo good-by >my_very_important_file"). You would have to use the "--noclobber" (or "-n") option in the mv command itself to keep it from overwriting files.

Bob.A-12 · 01-05-2015, 11:33 AM

Hi rknichols, thanks for responding to my post. I've previously used the "--backup" switch but only in one-liners and not scripts. I'm not too bothered about the system not recognising the photos due to the trailing .~1~ suffix. Will try the "--backup" in a new script and see if it works. Thanks, Bob.

josephj · 01-06-2015, 12:04 PM

Since you usually can't control how a camera (or, worse, multiple cameras!) names files, duplicate file names can easily happen. The previous post shows how to deal with them once they have already occurred.

If you potentially have a lot of these, look into the fdupes utility. It will help you find and fix them.

However, prevention is the best medicine. What I do:

1) I have a master directory for all managed photos.
2) Under that, I have a directory for each camera.
3) Every time I get photos off a camera, I get all of them and delete them from the camera.
This makes sure I don't try to save them again which could potentially cause more duplicates
(or, even worse, cause me to overwrite one file with another one and lose it completely.)
4) I look at the photos as files and find the newest file.
5) I create a new directory under the relevant camera's directory and name it
yyyymmdd (year, month, day of the newest file) and put the photos there.
6) I manage all the photos under the master directory with a photo manager. I use KPhotoAlbum
because it's great for tagging and finding photos and has a bunch of great features.
7) Whenever I copy a photo for a project or to work on it, I copy it outside of the master directory
so it won't be managed. Later, I can move it back in with a new name or
(much less likely) replacing the original.

By doing things this way, I never get any duplicate collisions because they can't occur in the same directory.

Before I figured this out, I had huge problems with duplicates and wrote a bunch of scripts to repair things. I wasted a ton of time.

Miati · 01-06-2015, 09:24 PM

This is a problem I came across awhile back when I had someone constantly send me photos with the same filename (photo.jpg) lol. I was writing a fully automated script for uploading so I knew that was going to be a problem. While loops come to the rescue.

Code:

filename="foo.jpg"
new_filename=$(code to create filename from exif - i'd be interested in seeing it, I could use it)
random_number="$((RANDOM%99))"

if [ -e new/location/"$new_filename" ]
	then
		while [ -e new/location/"$random_number"_"$new_filename" ] # Ensure file does not already exist.
			do 
				random_number="$((RANDOM%99))"
		done
		cp "$filename" new/location/"$random_number"_"$new_filename"
	else
		cp "$filename" new/location/"$new_filename"
fi

cp "$filename" new/location/"$random_number"_"$new_filename"

# If file exists, run while loop with random numbers until it doesn't exist
# then copy file over, else just copy original file over
#

Since the while loop will never exit if the file exists, the file can never be overwritten. Now the random number goes to 99 (as you should see), so 99 identical names could be a problem, add a 9 to fix

igadoter · 01-06-2015, 10:25 PM

Before renaming file put it supposed to be name into variable $FILENAME. $FILENAME will be a prefix for new file. Match existed files with this $FILENAME. Now there is one subtlety. There can be several files with the same prefix $FILENAME. New file will be named $FILENAME_$SUFF.jpg
and $SUFF is determined as $SUFF=number_of_already_existed_files + 1. For the $SUFF you need something like this

Code:

SUFF=$((`ls "$FILENAME"* | wc -l` + 1))

Try this in you ~/

Code:

$ touch newfile_$((`ls | wc -l` +1)).jpg
$ touch newfile_$((`ls | wc -l`+1)).jpg

Finally

Code:

mv 102_345.jpg "$FILENAME"_"$SUFF".jpg

josephj · 01-07-2015, 12:19 PM

@Miati - That's a good solution with one reservation. Since you're using random numbers, by default, the renamed pictures will sort randomly. It would be nice to have them sort in order received or date order - which takes a bit more code.

Also, (just my personal preference), I would use the number as a suffix rather than a prefix. This would be especially useful if files had several different base names. Then they would stay grouped (by default) by those names by default.

Here's some code I used as part of my renaming strategy. It doesn't show exactly what you want, but all the framework is there.

I no longer have the old exif.py installed, so this code no longer works, but I found the equivalent

EXIF.py

https://github.com/ianare/exif-py

Code:

#!/bin/bash

## Rename picture files by prepending the name with
## Picture Creation time from embedded exif data

## assumes it's in the directory where the files to be renamed are
## arg 1 is file name to be renamed

NAME=basename "$1"

if [ ! -w "${NAME}" ]
then
  echo "Can't rename [${NAME}]"
  exit 1
fi

INFO=($(/usr/share/apps/imagerotation/exif.py "${1}" | grep DateTime:))
##echo [${INFO[2]}] [${INFO[3]}]

DATE=${INFO[2]##ASCII=}
TIME=${INFO[3]}
FILE="${DATE}_${TIME}_$1"

echo mv "${1}" ${FILE}

josephj · 01-07-2015, 01:04 PM

@igadoter - Nice idea, but it's not recommended to use the output of ls in scripts. It has way too many quirks and portability issues. E.g. if your directory contains other directories or symlinks, your code will get unexpected results.

See:
http://mywiki.wooledge.org/BashGuide/Practices

about half way down the page.

It's better to use find instead. E.g.:

Code:

find . -maxdepth 1 -type f -name '*'

or you could use '*.jpg' instead of just '*' to be even safer.

Miati · 01-07-2015, 01:22 PM

Quote:

I would use the number as a suffix rather than a prefix.

That's true. With them as suffix I believe it should sort more effectively (if the exif data is the same, does it really matter what order the suffix is while sorting?)

However, this wouldn't be a big issue, just have to write the line a little more clever.

something like

Code:

"${new_filename%.*}_$random_number.${new_filename##*.}"

$ new_filename=test.jpg
$ random_number=$RANDOM
$ echo ${new_filename%.*}_$random_number.${new_filename##*.}
test_22767.jpg

Basically, the first segment strips the extension from the filename string. Then puts in _ and $random_number, then a . then the filename extension jpg

Also, thanks for the link. I did some looking around & it turns out identify makes for a pretty easy exif extractor.
To get the date created from exif:

Code:

$ identify -format "%[EXIF:DateTimeOriginal]" file.jpg
2014:11:28 22:59:40

Norseman01 · 01-10-2015, 04:47 PM

[QUOTE=josephj;5296444]Since you usually can't control how a camera (or, worse, multiple cameras!) names files...
============================

I use a similar method, but first I need to clarify what I'm running into as the duplicate problem.

Problem: Same file first name with different last name.
Example: dscn1234.jpg (the photo), dscn1234.txt (metadata), dscn1234.pdf (of photo)
yr/mo/day-hr:mn:sc.jpg (photo from Smartphone), ...txt (meta), ...pdf (of photo)
Problem: Rapid file can give several photos the same name to the same second. Camera or SmartPhone
will usually add something to differentiate them. BUT - the method can be different from
taking source to taking source.

SO: as I said, I use something similar to (#4) Joseph.

I copy to a scratch section keeping the files on the source until I am done.

I run a loop that straightens up Window'$ badly formed names, then cleans up the method that is
used to designate anther to the same second name. Once that is done I run the files through a sort
routine and process them in name sorted order keeping the previous name and checking for duplicates.
In my case - same first but different last name.
I process the .jpg (.nef ...whatever) with a simple Python file that I wrote that sets the disk
file stamp to the photo's internal EXIF date/time. I DO NOT run this program on files with
Date/Time file names. I DO NOT run this on "source added to name" files that have not been
through the "cleaner". Files that are not deemed Image do not get disk time stamps changed. That
is easily managed because they don't have EXIF data.

Files with Date/Time stamp names get their disk time stamps set to match. This can include "movies".

Sorting and storing is pretty much identical to "josephj"s Master area/Year/... method.
I use:
..YEAR
....YrMoDa (20150101)
......operator (me/Wife)
........camera (Mine/Hers
..........images
........phone (you get the idea)
..........images
dupe from operator and down for the day if more than one was taking photos.

and fill images area with images, relevant text files of travel dialogue and specific image data
and with .pdfs of the Images, .txts and the directory listing and combine the pdfs into
a "visual directory file" for quick searches. The individual pdfs are then deleted
and the original sources (picture taking device storage media) is reformatted.
Do that last part last and you will shed fewer tears.

All that is done with bash, Python (with EXIF and PIL installed), and gs (ghostscript)
At storage time or later I use GIMP for touch-ups.

If you haven't taken the time to learn a few of the programming things Linux offers - now
is the time to start. Much of the Day-to-Day can be automated.

Just how automated can this be??? Type it's name and press Enter and: MkDiskNdx.scr,
My lead script of all 3,133 characters (including tabs, spaces and comments), can process
the entire computer system. All partitions of all drives connected.

Norseman01