ProgrammingThis forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
Ok, I'm at my wits end with a rather simple little script and no matter how many incantations of IFS/read fiddling I do, I can't figure it out.
It's basically a simple batch script that invokes ffmpeg on numerous files in a set location, converting them to another format. Problem is since some of them have come from Windows machines, there's every likelihood they could have spaces in them. I was lead to believe that using the -exec feature of find is the safest way to go about this, and it's what I've done in the past with success.
Script in it's entirety is :-
Code:
#!/bin/bash
# Where are the files to process (input files)?
INPUT_DIR=/home/video/batch/in/tompeg2
# Where do you want to create the final files (output files)?
OUTPUT_DIR=/home/video/batch/out/mpeg2
# If a file fails to convert, where shall we move the original input file so it's not attempted again, but not deleted?
FAILED_DIR=/home/video/batch/in/failed
# If a file converts ok, where shall we move the original input file to afterwards?
SUCCESS_DIR=/home/video/batch/in/processed
# Initialise the dryrun flag; don't change this here (it's a parameter)
DRYRUN=0
FILE=""
# Where should we log all our activity?
LOG=/home/video/batch/mpeg2batch.log
# THAT'S IT FOR CONFIGURABLE VARIABLES - DON'T EDIT BELOW UNLESS YOU KNOW WHAT YOU'RE DOING!
function debug {
echo "`(date +"%d/%m/%Y %H:%M:%S")` $1" >> $LOG
}
function debugall {
echo "`(date +"%d/%m/%Y %H:%M:%S")` $1"
echo "`(date +"%d/%m/%Y %H:%M:%S")` $1" >> $LOG
}
# Parse parameters
for i in $@;
do
case "$i" in
"-dryrun" | "-dry")
printf "\ndryrun : no files will be created, changed, moved or deleted\n"
DRYRUN=1
;;
"-help" | "-h" | "-?")
printf "\nParameters are :-\n -dryrun Perform dryrun only - do not create or change any files (non-destructive run)\n\n"
exit;
;;
esac
done
debugall "================================================================================="
debugall "Starting batch job - convert to MPEG2"
debugall " inbound directory : ${INPUT_DIR}"
debugall " outbound directory : ${OUTPUT_DIR}"
find $INPUT_DIR -maxdepth 1 -print0 -type f -name '*' | while read -d $'\0' FILE;
do
if [ -f "${FILE}" ] && [ -s "${FILE}" ]; then
echo "[${FILE}]"
fi
done
find $INPUT_DIR -maxdepth 1 -print0 -type f -name '*' | while read -d $'\0' FILE;
do
# Let's just make sure the file isn't empty (zero byte length), and also make sure we ignore directories too
if [ -f "${FILE}" ] && [ -s "${FILE}" ]; then
debugall "Processing file ${FILE}"
# FILENAME is the name of the file itself, rather than the full path and filename
FILENAME=`basename "${FILE}"`
# ...and the same, but without whatever extension is on the end (.dvr-ms, .mpeg, etc)
FILENAME_NOEXT="${FILENAME%.*}"
# ...and the opposite of filename - get the full path of the directory the file is in, but not the filename itself
DIRNAME=`dirname "${FILE}"`
OUTPUT_FILENAME="${FILENAME}.mpg"
i=1
while [ -f "${OUTPUT_DIR}/${OUTPUT_FILENAME}" ]; do
debugall "Output file ${OUTPUT_FILENAME} already exists; will create a new file : ${FILENAME}_${i}.avi"
OUTPUT_FILENAME="${FILENAME}_${i}.avi"
i=$((i+1))
done;
if [ $DRYRUN -eq 0 ]; then
ffmpeg -i "${FILE}" -target dvd "${OUTPUT_DIR}/${OUTPUT_FILENAME}" &>"/tmp/${FILENAME_NOEXT}_pass.log"
EXITCODE=$?
else
printf "Pass 1 call :\n"
EXITCODE=$?
fi
if [ $DRYRUN -eq 0 ]; then
if [ $EXITCODE -eq 0 ]; then
debug "-Pass 1 OK. Moving inbound file to processed directory."
mv -f "${FILE}" "${SUCCESS_DIR}"
rm -f "/tmp/${FILENAME_NOEXT}_pass.log"
else
debug "-Pass 1 failed. This file will be skipped and moved to failed directory."
debug "Check ${FILENAME_NOEXT}_pass.log for more information"
mv -f "/tmp/${FILENAME_NOEXT}_pass.log" "${FAILED_DIR}"
rm -f "${OUTPUT_DIR}/${OUTPUT_FILENAME}"
mv -f "${FILE}" "${FAILED_DIR}"
fi
fi
debug "Tidying up"
rm -f "/tmp/${FILENAME_NOEXT}.log"
rm -f "/tmp/${FILENAME}_details.txt"
else
debug "Skipping ${FILE} - not a file, or zero bytes in size"
fi
done
debugall "Finished ======================================================"
exit 0
When I run it, I'll get the following from the first loop which verifies to me that it is indeed seeing all the files in the folder, and that the loop is iterating over them as I would expect :-
So, as you can see, there's 4 files to process. The problem is, it then enters the second (identical) loop, processes the first file, then that's it - it never continues around.
The debug log file doesn't show anything untoward either :-
Code:
24/01/2009 14:04:37 =================================================================================
24/01/2009 14:04:37 Starting batch job - convert to MPEG2
24/01/2009 14:04:37 inbound directory : /home/video/batch/in/tompeg2
24/01/2009 14:04:37 outbound directory : /home/video/batch/out/mpeg2
24/01/2009 14:04:37 Skipping /home/video/batch/in/tompeg2 - not a file, or zero bytes in size
24/01/2009 14:04:37 Processing file /home/video/batch/in/tompeg2/rotm06-001_1.vob
24/01/2009 14:05:54 -Pass 1 OK. Moving inbound file to processed directory.
24/01/2009 14:05:54 Tidying up
24/01/2009 14:05:54 Finished ======================================================
This has got to be something so silly and stupid that I just can't see the wood for the trees! I suspect one of the operations in the 2nd loop is 'fiddling' with the FILE var or how 'read' pulls in the list of files from the find command. I've tried fiddling with alternative IFS settings but to no luck. Anyone put me out of my misery, because I'm just not seeing it ?!
Ok, thanks for that, I've given it a whirl with some quick files with spaces in and it seems fine, which kerfuddles me somewhat as I'm sure the whole reason I went that route with 'find' was down to filename handling! I've got some other scripts that exhibit the same problem and do more complicated stuff, so I'll change those and give them a whirl and see how we get on.
Simple expansion may work if you always enclose the variable in double quotes latter on:
for file in *.vob; do echo "$file"; done
abc def.vob
ab c.vob
bc d.vob
In this line:
if [ -f "${FILE}" ] && [ -s "${FILE}" ]; then
You might consider [ -f "${FILE}" -a -s "${FILE}" ]. && is normally used to run a second command depending on the success of the first command.
In this line: find $INPUT_DIR -maxdepth 1 -print0 -type f -name '*' | while read -d $'\0' FILE;
Could you explain the use of "$'\0'. I hadn't seen it before but the read command's "-d" argument seems to need it.
Running this test, there is an unrelated wrinkle:
find $INPUT_DIR -maxdepth 1 -print0 -type f -name '*' | while read -d $'\0' FILE; do echo file: "$FILE"; done
file: testdir
file: testdir/abc def.vob
file: testdir/bc d.vob
file: testdir/ab c.vob
$INPUT_DIR/. is returned. Put `-type f' before `-print0'.
Using capital letters for FILE is confusing to me because as I read it I have to remember that it isn't a constant. The convention is to use capital letters for constants like INPUT_DIR and small or mixed case letters for variables.
It's one of those things where this script has laid around, organically growing over the past couple of years and, despite my commenting, I have clearly not put much history around why that find command is laid out the way it is. In truth, I've tried lots of things over time because quite clearly I just didn't know how it worked, so I googled here and there, saw how others had done it, and tried it myself. Clearly at some point I decided to use NUL as the field separator coming out of find, but why I didn't stick with LF I don't know. So I've put it back to a plain -print on the find command, and put read back to the default of just using LF too. I've kicked it off but it still does the same. I tried the above suggestion too, but still the same thing. Maybe this is something to do with ffmpeg maybe....
Code:
TMPFILE=/tmp/$$.tmp
ls $INPUT_DIR >${TMPFILE}
while read FILE
do
if [ -f "${INPUT_DIR}/${FILE}" ] && [ -s "${INPUT_DIR}/${FILE}" ]; then
echo "[${FILE}]"
fi
done <${TMPFILE}
while read FILE
do
snip ...same stuff as original... snip
done <${TMPFILE}
rm -f ${TMPFILE}
debugall "Finished ======================================================"
exit 0
Results in first loop saying "Yep, 5 files, no problem!" then the 2nd identical loop saying "I'll do the first one but that's it, buddy!".
Code:
[howell@server1 tompeg2]$ batchjob_ffmpegToMPEG2-2.sh
27/01/2009 19:12:13 =================================================================================
27/01/2009 19:12:13 Starting batch job - convert to MPEG2
27/01/2009 19:12:13 inbound directory : /home/video/batch/in/tompeg2
27/01/2009 19:12:13 outbound directory : /home/video/batch/out/mpeg2
[psooc2.avi]
[psooc3.avi]
[psooc and another space & stuff.avi]
[psooc.avi]
[psooc with space.avi]
27/01/2009 19:12:13 Processing file psooc2.avi
27/01/2009 19:14:30 Finished ======================================================
I'll maybe try reducing it down and systematically add the lines into the second loop one by one to see which bit breaks it.... but later, when I get some time. Maybe you guys can help me from there then.... back with more later, but a few jobs to get out of the way first
You may have misunderstood my post. It was the location of -print0 that could be a problem. It needs to be after the `-type f' argument. Otherwise, you are printing results before restricting the results to -type f. So, you will get the "." directory and any other directories as results which isn't what you want.
I checked in the abs-guide.pdf and found a usage of the $'\0nnn' construct. I just was curious if you knew about it. I've used `tr '\n' '\0' in the past to convert returns to nulls, but hadn't seen $'\0' before. It's the dollar sign that caught my eye, and I was hoping you knew why -d $'\0' works and -d '\0' doesn't.
A construct you may find useful is:
command1 && command2 || command3
for a pseudo code example:
convert "$file1" "$file1.vob" && mv "${file1.vob" $DONE_DIR/ || mv "$file1" $ERR_DIR/
It the first command is executed, perform the second, otherwise the third.
You may have misunderstood my post. It was the location of -print0 that could be a problem. It needs to be after the `-type f' argument. Otherwise, you are printing results before restricting the results to -type f. So, you will get the "." directory and any other directories as results which isn't what you want.
Nope, I got you; I made the change. I wasn't aware order was important, but in some other versions of the script I purposely checked if the actioned list item was a file so I kinda fudged around it, but your solution is much neater - thanks.
Quote:
I checked in the abs-guide.pdf and found a usage of the $'\0nnn' construct. I just was curious if you knew about it. I've used `tr '\n' '\0' in the past to convert returns to nulls, but hadn't seen $'\0' before. It's the dollar sign that caught my eye, and I was hoping you knew why -d $'\0' works and -d '\0' doesn't.
I suspect I copied someone else without understanding why, but irrespective of that, I've put it back to just LF now but it still exhibits the same behaviour. I'll try the IFS set to linefeed as suggested above next....
I checked in the abs-guide.pdf and found a usage of the $'\0nnn' construct. I just was curious if you knew about it. I've used `tr '\n' '\0' in the past to convert returns to nulls, but hadn't seen $'\0' before. It's the dollar sign that caught my eye, and I was hoping you knew why -d $'\0' works and -d '\0' doesn't.
The $'\0' is expanded (to the empty string, since bash strings are NUL-terminated) by bash, whereas in tr '\n' '\0', tr interprets the escape sequences.
From man bash
Code:
QUOTING
...
Words of the form $'string' are treated specially. The word expands to string, with backslash-
escaped characters replaced as specified by the ANSI C standard. Backslash escape sequences,
if present, are decoded as follows:
\a alert (bell)
\b backspace
\e an escape character
\f form feed
\n new line
\r carriage return
\t horizontal tab
\v vertical tab
\\ backslash
\' single quote
\nnn the eight-bit character whose value is the octal value nnn (one to three digits)
\xHH the eight-bit character whose value is the hexadecimal value HH (one or two hex
digits)
\cx a control-x character
EDIT
I prefixed all invocations of mv, rm, and ffmpeg with echo and the script ran as expected, which suggests that some command inside the loop is doing something funny. Does ffmpeg ever read from standard input? Does changing that line to
Code:
ffmpeg -i "${FILE}" -target dvd "${OUTPUT_DIR}/${OUTPUT_FILENAME}" &> "/tmp/${FILENAME_NOEXT}_pass.log" < /dev/null
I'm working with the "while read FILE" version for now, as that exhibits the same behaviour as the original. If we can fix that one then I'll try the same process with the original script, and my other ones I have based on it, and see how it goes.
I agree, playing around here it does seem that ffmpeg is doing something 'funny'. Trying your suggestion now...
EDIT: Bingo, that's it
ffmpeg doesn't sit held anywhere asking for input, but it does offer an abort keypress - could it be this?
I'll go back and try it on my original scripts and see if that address the problem there too - I suspect it will
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
?!
