LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Programming (http://www.linuxquestions.org/questions/programming-9/)
-   -   bash script : another set of eyes, please? (http://www.linuxquestions.org/questions/programming-9/bash-script-another-set-of-eyes-please-699580/)

brumster 01-24-2009 08:10 AM

bash script : another set of eyes, please?
 
Ok, I'm at my wits end with a rather simple little script and no matter how many incantations of IFS/read fiddling I do, I can't figure it out.

It's basically a simple batch script that invokes ffmpeg on numerous files in a set location, converting them to another format. Problem is since some of them have come from Windows machines, there's every likelihood they could have spaces in them. I was lead to believe that using the -exec feature of find is the safest way to go about this, and it's what I've done in the past with success.

Script in it's entirety is :-

Code:

#!/bin/bash

# Where are the files to process (input files)?
INPUT_DIR=/home/video/batch/in/tompeg2

# Where do you want to create the final files (output files)?
OUTPUT_DIR=/home/video/batch/out/mpeg2

# If a file fails to convert, where shall we move the original input file so it's not attempted again, but not deleted?
FAILED_DIR=/home/video/batch/in/failed

# If a file converts ok, where shall we move the original input file to afterwards?
SUCCESS_DIR=/home/video/batch/in/processed

# Initialise the dryrun flag; don't change this here (it's a parameter)
DRYRUN=0

FILE=""

# Where should we log all our activity?
LOG=/home/video/batch/mpeg2batch.log

# THAT'S IT FOR CONFIGURABLE VARIABLES - DON'T EDIT BELOW UNLESS YOU KNOW WHAT YOU'RE DOING!

function debug {
  echo "`(date +"%d/%m/%Y %H:%M:%S")` $1" >> $LOG
}

function debugall {
  echo "`(date +"%d/%m/%Y %H:%M:%S")` $1"
  echo "`(date +"%d/%m/%Y %H:%M:%S")` $1" >> $LOG
}

# Parse parameters
for i in $@;
do
 case "$i" in
  "-dryrun" | "-dry")
  printf "\ndryrun : no files will be created, changed, moved or deleted\n"
  DRYRUN=1
  ;;
  "-help" | "-h" | "-?")
  printf "\nParameters are :-\n -dryrun      Perform dryrun only - do not create or change any files (non-destructive run)\n\n"
  exit;
  ;;
 esac
done

debugall "================================================================================="
debugall "Starting batch job - convert to MPEG2"
debugall "  inbound directory      : ${INPUT_DIR}"
debugall "  outbound directory      : ${OUTPUT_DIR}"

find $INPUT_DIR -maxdepth 1 -print0 -type f -name '*' | while read -d $'\0' FILE;
do
 if [ -f "${FILE}" ] && [ -s "${FILE}" ]; then
        echo "[${FILE}]"
 fi
done

find $INPUT_DIR -maxdepth 1 -print0 -type f -name '*' | while read -d $'\0' FILE;
do

 # Let's just make sure the file isn't empty (zero byte length), and also make sure we ignore directories too
 if [ -f "${FILE}" ] && [ -s "${FILE}" ]; then

        debugall "Processing file ${FILE}"

        # FILENAME is the name of the file itself, rather than the full path and filename
        FILENAME=`basename "${FILE}"`
        # ...and the same, but without whatever extension is on the end (.dvr-ms, .mpeg, etc)
        FILENAME_NOEXT="${FILENAME%.*}"
        # ...and the opposite of filename - get the full path of the directory the file is in, but not the filename itself
        DIRNAME=`dirname "${FILE}"`

          OUTPUT_FILENAME="${FILENAME}.mpg"
          i=1
          while [ -f "${OUTPUT_DIR}/${OUTPUT_FILENAME}" ]; do
            debugall "Output file ${OUTPUT_FILENAME} already exists; will create a new file : ${FILENAME}_${i}.avi"
            OUTPUT_FILENAME="${FILENAME}_${i}.avi"
            i=$((i+1))
          done;

          if [ $DRYRUN -eq 0 ]; then
                ffmpeg -i "${FILE}" -target dvd "${OUTPUT_DIR}/${OUTPUT_FILENAME}" &>"/tmp/${FILENAME_NOEXT}_pass.log"
                  EXITCODE=$?
          else
          printf "Pass 1 call :\n"
          EXITCODE=$?
          fi

          if [ $DRYRUN -eq 0 ]; then
            if [ $EXITCODE -eq 0 ]; then
              debug "-Pass 1 OK. Moving inbound file to processed directory."
              mv -f "${FILE}" "${SUCCESS_DIR}"
              rm -f "/tmp/${FILENAME_NOEXT}_pass.log"
            else
              debug "-Pass 1 failed. This file will be skipped and moved to failed directory."
              debug "Check ${FILENAME_NOEXT}_pass.log for more information"
              mv -f "/tmp/${FILENAME_NOEXT}_pass.log" "${FAILED_DIR}"
              rm -f "${OUTPUT_DIR}/${OUTPUT_FILENAME}"
              mv -f "${FILE}" "${FAILED_DIR}"
            fi
        fi

        debug "Tidying up"
        rm -f "/tmp/${FILENAME_NOEXT}.log"
        rm -f "/tmp/${FILENAME}_details.txt"
 else
  debug "Skipping ${FILE} -  not a file, or zero bytes in size"
 fi

done

debugall "Finished ======================================================"
exit 0

When I run it, I'll get the following from the first loop which verifies to me that it is indeed seeing all the files in the folder, and that the loop is iterating over them as I would expect :-

Code:

[howell@server1 bin]$ ./batchjob_ffmpegToMPEG2.sh
24/01/2009 14:04:37 =================================================================================
24/01/2009 14:04:37 Starting batch job - convert to MPEG2
24/01/2009 14:04:37  inbound directory      : /home/video/batch/in/tompeg2
24/01/2009 14:04:37  outbound directory      : /home/video/batch/out/mpeg2
[/home/video/batch/in/tompeg2/rotm06-001_1.vob]
[/home/video/batch/in/tompeg2/rotm06-001_2.vob]
[/home/video/batch/in/tompeg2/rotm06-002.vob]
[/home/video/batch/in/tompeg2/rotm06-003.vob]

So, as you can see, there's 4 files to process. The problem is, it then enters the second (identical) loop, processes the first file, then that's it - it never continues around.

Code:

24/01/2009 14:04:37 Processing file /home/video/batch/in/tompeg2/rotm06-001_1.vob
24/01/2009 14:05:54 Finished ======================================================

The debug log file doesn't show anything untoward either :-

Code:

24/01/2009 14:04:37 =================================================================================
24/01/2009 14:04:37 Starting batch job - convert to MPEG2
24/01/2009 14:04:37  inbound directory      : /home/video/batch/in/tompeg2
24/01/2009 14:04:37  outbound directory      : /home/video/batch/out/mpeg2
24/01/2009 14:04:37 Skipping /home/video/batch/in/tompeg2 -  not a file, or zero bytes in size
24/01/2009 14:04:37 Processing file /home/video/batch/in/tompeg2/rotm06-001_1.vob
24/01/2009 14:05:54 -Pass 1 OK. Moving inbound file to processed directory.
24/01/2009 14:05:54 Tidying up
24/01/2009 14:05:54 Finished ======================================================

This has got to be something so silly and stupid that I just can't see the wood for the trees! I suspect one of the operations in the 2nd loop is 'fiddling' with the FILE var or how 'read' pulls in the list of files from the find command. I've tried fiddling with alternative IFS settings but to no luck. Anyone put me out of my misery, because I'm just not seeing it :) ?!

itz2000 01-24-2009 09:04 AM

can you please run it with "#!/bin/bash -x" instead of "#!/bin/bash" so we can see where this stuck?

brumster 01-24-2009 11:41 AM

Quote:

Originally Posted by itz2000 (Post 3419700)
can you please run it with "#!/bin/bash -x" instead of "#!/bin/bash" so we can see where this stuck?

Sure thing... below.... not sure how to read this...

Code:

+ INPUT_DIR=/home/video/batch/in/tompeg2     
+ OUTPUT_DIR=/home/video/batch/out/mpeg2     
+ FAILED_DIR=/home/video/batch/in/failed     
+ SUCCESS_DIR=/home/video/batch/in/processed 
+ DRYRUN=0                                   
+ FILE=                                     
+ LOG=/home/video/batch/mpeg2batch.log       
+ debugall =================================================================================                                                                   
++ date '+%d/%m/%Y %H:%M:%S'                                                   
+ echo '24/01/2009 15:47:15 ================================================================================='                                                 
24/01/2009 15:47:15 =================================================================================                                                         
++ date '+%d/%m/%Y %H:%M:%S'                                                   
+ echo '24/01/2009 15:47:15 ================================================================================='                                                 
+ debugall 'Starting batch job - convert to MPEG2'                             
++ date '+%d/%m/%Y %H:%M:%S'                                                   
+ echo '24/01/2009 15:47:15 Starting batch job - convert to MPEG2'             
24/01/2009 15:47:15 Starting batch job - convert to MPEG2                     
++ date '+%d/%m/%Y %H:%M:%S'                                                   
+ echo '24/01/2009 15:47:15 Starting batch job - convert to MPEG2'             
+ debugall '  inbound directory      : /home/video/batch/in/tompeg2'         
++ date '+%d/%m/%Y %H:%M:%S'                                                   
+ echo '24/01/2009 15:47:15  inbound directory      : /home/video/batch/in/tompeg2'                                                                         
24/01/2009 15:47:15  inbound directory      : /home/video/batch/in/tompeg2   
++ date '+%d/%m/%Y %H:%M:%S'                                                   
+ echo '24/01/2009 15:47:15  inbound directory      : /home/video/batch/in/tompeg2'                                                                         
+ debugall '  outbound directory      : /home/video/batch/out/mpeg2'           
++ date '+%d/%m/%Y %H:%M:%S'                                                   
+ echo '24/01/2009 15:47:15  outbound directory      : /home/video/batch/out/mpeg2'                                                                           
24/01/2009 15:47:15  outbound directory      : /home/video/batch/out/mpeg2   
++ date '+%d/%m/%Y %H:%M:%S'                                                   
+ echo '24/01/2009 15:47:15  outbound directory      : /home/video/batch/out/mpeg2'                                                                           
+ find /home/video/batch/in/tompeg2 -maxdepth 1 -print0 -type f -name '*'     
+ read -d '' FILE                                                             
+ '[' -f /home/video/batch/in/tompeg2 ']'                                     
+ read -d '' FILE                                                             
+ '[' -f /home/video/batch/in/tompeg2/vts_01VTS_01_1.VOB ']'                   
+ '[' -s /home/video/batch/in/tompeg2/vts_01VTS_01_1.VOB ']'                   
+ echo '[/home/video/batch/in/tompeg2/vts_01VTS_01_1.VOB]'                     
[/home/video/batch/in/tompeg2/vts_01VTS_01_1.VOB]                             
+ read -d '' FILE                                                             
+ '[' -f /home/video/batch/in/tompeg2/vts_01VTS_01_2.VOB ']'                   
+ '[' -s /home/video/batch/in/tompeg2/vts_01VTS_01_2.VOB ']'                   
+ echo '[/home/video/batch/in/tompeg2/vts_01VTS_01_2.VOB]'                     
[/home/video/batch/in/tompeg2/vts_01VTS_01_2.VOB]                             
+ read -d '' FILE                                                             
+ '[' -f /home/video/batch/in/tompeg2/vts_01VTS_01_3.VOB ']'                   
+ '[' -s /home/video/batch/in/tompeg2/vts_01VTS_01_3.VOB ']'                   
+ echo '[/home/video/batch/in/tompeg2/vts_01VTS_01_3.VOB]'                     
[/home/video/batch/in/tompeg2/vts_01VTS_01_3.VOB]                             
+ read -d '' FILE                                                             
+ '[' -f /home/video/batch/in/tompeg2/vts_01VTS_01_4.VOB ']'                   
+ '[' -s /home/video/batch/in/tompeg2/vts_01VTS_01_4.VOB ']'                   
+ echo '[/home/video/batch/in/tompeg2/vts_01VTS_01_4.VOB]'                     
[/home/video/batch/in/tompeg2/vts_01VTS_01_4.VOB]                             
+ read -d '' FILE                                                             
+ find /home/video/batch/in/tompeg2 -maxdepth 1 -print0 -type f -name '*'     
+ read -d '' FILE                                                             
+ '[' -f /home/video/batch/in/tompeg2 ']'                                     
+ debug 'Skipping /home/video/batch/in/tompeg2 -  not a file, or zero bytes in size'                                                                           
++ date '+%d/%m/%Y %H:%M:%S'                                                   
+ echo '24/01/2009 15:47:15 Skipping /home/video/batch/in/tompeg2 -  not a file, or zero bytes in size'                                                       
+ read -d '' FILE                                                             
+ '[' -f /home/video/batch/in/tompeg2/vts_01VTS_01_1.VOB ']'                   
+ '[' -s /home/video/batch/in/tompeg2/vts_01VTS_01_1.VOB ']'                   
+ debugall 'Processing file /home/video/batch/in/tompeg2/vts_01VTS_01_1.VOB'   
++ date '+%d/%m/%Y %H:%M:%S'                                                   
+ echo '24/01/2009 15:47:15 Processing file /home/video/batch/in/tompeg2/vts_01VTS_01_1.VOB'                                                                   
24/01/2009 15:47:15 Processing file /home/video/batch/in/tompeg2/vts_01VTS_01_1.VOB                                                                           
++ date '+%d/%m/%Y %H:%M:%S'                                                   
+ echo '24/01/2009 15:47:15 Processing file /home/video/batch/in/tompeg2/vts_01VTS_01_1.VOB'                                                                   
++ basename /home/video/batch/in/tompeg2/vts_01VTS_01_1.VOB                   
+ FILENAME=vts_01VTS_01_1.VOB
+ FILENAME_NOEXT=vts_01VTS_01_1
++ dirname /home/video/batch/in/tompeg2/vts_01VTS_01_1.VOB
+ DIRNAME=/home/video/batch/in/tompeg2
+ OUTPUT_FILENAME=vts_01VTS_01_1.VOB.mpg
+ i=1
+ '[' -f /home/video/batch/out/mpeg2/vts_01VTS_01_1.VOB.mpg ']'
+ '[' 0 -eq 0 ']'
+ ffmpeg -i /home/video/batch/in/tompeg2/vts_01VTS_01_1.VOB -target dvd /home/video/batch/out/mpeg2/vts_01VTS_01_1.VOB.mpg
+ EXITCODE=0
+ '[' 0 -eq 0 ']'
+ '[' 0 -eq 0 ']'
+ debug '-Pass 1 OK. Moving inbound file to processed directory.'
++ date '+%d/%m/%Y %H:%M:%S'
+ echo '24/01/2009 15:52:27 -Pass 1 OK. Moving inbound file to processed directory.'
+ mv -f /home/video/batch/in/tompeg2/vts_01VTS_01_1.VOB /home/video/batch/in/processed
+ rm -f /tmp/vts_01VTS_01_1_pass.log
+ debug 'Tidying up'
++ date '+%d/%m/%Y %H:%M:%S'
+ echo '24/01/2009 15:52:27 Tidying up'
+ rm -f /tmp/vts_01VTS_01_1.log
+ rm -f /tmp/vts_01VTS_01_1.VOB_details.txt
+ read -d '' FILE
+ debugall 'Finished ======================================================'
++ date '+%d/%m/%Y %H:%M:%S'
+ echo '24/01/2009 15:52:27 Finished ======================================================'
24/01/2009 15:52:27 Finished ======================================================
++ date '+%d/%m/%Y %H:%M:%S'
+ echo '24/01/2009 15:52:27 Finished ======================================================'
+ exit 0


brumster 01-26-2009 02:51 AM

Ok, an alternative, if someone can show me another way of running through a loop of files that's safe with spaces in the filenames, then I'm all ears.

cicorino 01-26-2009 09:12 AM

Quote:

Originally Posted by brumster (Post 3421371)
Ok, an alternative, if someone can show me another way of running through a loop of files that's safe with spaces in the filenames, then I'm all ears.

I can't help more, but for this simple task I always do something like:

Code:

TMPFILE=`tempfile -s.deleteme`
ls /path/to/somepath >${TMPFILE}
while read E
do
 echo "do what I want on ${E}"
done <${TMPFILE}
rm -f ${TMPFILE}


brumster 01-27-2009 03:56 AM

Ok, thanks for that, I've given it a whirl with some quick files with spaces in and it seems fine, which kerfuddles me somewhat as I'm sure the whole reason I went that route with 'find' was down to filename handling! I've got some other scripts that exhibit the same problem and do more complicated stuff, so I'll change those and give them a whirl and see how we get on.

Thanks again

jschiwal 01-27-2009 05:13 AM

Simple expansion may work if you always enclose the variable in double quotes latter on:
for file in *.vob; do echo "$file"; done
abc def.vob
ab c.vob
bc d.vob

In this line:
if [ -f "${FILE}" ] && [ -s "${FILE}" ]; then

You might consider [ -f "${FILE}" -a -s "${FILE}" ]. && is normally used to run a second command depending on the success of the first command.

In this line: find $INPUT_DIR -maxdepth 1 -print0 -type f -name '*' | while read -d $'\0' FILE;
Could you explain the use of "$'\0'. I hadn't seen it before but the read command's "-d" argument seems to need it.

Running this test, there is an unrelated wrinkle:
find $INPUT_DIR -maxdepth 1 -print0 -type f -name '*' | while read -d $'\0' FILE; do echo file: "$FILE"; done
file: testdir
file: testdir/abc def.vob
file: testdir/bc d.vob
file: testdir/ab c.vob

$INPUT_DIR/. is returned. Put `-type f' before `-print0'.

Using capital letters for FILE is confusing to me because as I read it I have to remember that it isn't a constant. The convention is to use capital letters for constants like INPUT_DIR and small or mixed case letters for variables.

Sorry I didn't get further into your script.

brumster 01-27-2009 01:28 PM

It's one of those things where this script has laid around, organically growing over the past couple of years and, despite my commenting, I have clearly not put much history around why that find command is laid out the way it is. In truth, I've tried lots of things over time because quite clearly I just didn't know how it worked, so I googled here and there, saw how others had done it, and tried it myself. Clearly at some point I decided to use NUL as the field separator coming out of find, but why I didn't stick with LF I don't know. So I've put it back to a plain -print on the find command, and put read back to the default of just using LF too. I've kicked it off but it still does the same. I tried the above suggestion too, but still the same thing. Maybe this is something to do with ffmpeg maybe....

Code:

TMPFILE=/tmp/$$.tmp
ls $INPUT_DIR >${TMPFILE}
while read FILE
do
 if [ -f "${INPUT_DIR}/${FILE}" ] && [ -s "${INPUT_DIR}/${FILE}" ]; then
        echo "[${FILE}]"
 fi
done <${TMPFILE}

while read FILE
do

  snip ...same stuff as original... snip

done <${TMPFILE}
rm -f ${TMPFILE}

debugall "Finished ======================================================"
exit 0

Results in first loop saying "Yep, 5 files, no problem!" then the 2nd identical loop saying "I'll do the first one but that's it, buddy!".

Code:

[howell@server1 tompeg2]$ batchjob_ffmpegToMPEG2-2.sh
27/01/2009 19:12:13 =================================================================================
27/01/2009 19:12:13 Starting batch job - convert to MPEG2
27/01/2009 19:12:13  inbound directory      : /home/video/batch/in/tompeg2
27/01/2009 19:12:13  outbound directory      : /home/video/batch/out/mpeg2
[psooc2.avi]
[psooc3.avi]
[psooc and another space & stuff.avi]
[psooc.avi]
[psooc with space.avi]
27/01/2009 19:12:13 Processing file psooc2.avi
27/01/2009 19:14:30 Finished ======================================================

I'll maybe try reducing it down and systematically add the lines into the second loop one by one to see which bit breaks it.... but later, when I get some time. Maybe you guys can help me from there then.... back with more later, but a few jobs to get out of the way first :(

David the H. 01-27-2009 01:52 PM

Quote:

Originally Posted by brumster (Post 3421371)
Ok, an alternative, if someone can show me another way of running through a loop of files that's safe with spaces in the filenames, then I'm all ears.

I don't know if you've tried this yet, but I've always had success setting IFS as a newline at the beginning of the script.
Code:

IFS="
"

This should make it ignore spaces as field separators.

jschiwal 01-27-2009 11:13 PM

You may have misunderstood my post. It was the location of -print0 that could be a problem. It needs to be after the `-type f' argument. Otherwise, you are printing results before restricting the results to -type f. So, you will get the "." directory and any other directories as results which isn't what you want.

I checked in the abs-guide.pdf and found a usage of the $'\0nnn' construct. I just was curious if you knew about it. I've used `tr '\n' '\0' in the past to convert returns to nulls, but hadn't seen $'\0' before. It's the dollar sign that caught my eye, and I was hoping you knew why -d $'\0' works and -d '\0' doesn't.

A construct you may find useful is:
command1 && command2 || command3

for a pseudo code example:
convert "$file1" "$file1.vob" && mv "${file1.vob" $DONE_DIR/ || mv "$file1" $ERR_DIR/
It the first command is executed, perform the second, otherwise the third.

Good Luck.

brumster 01-28-2009 11:11 AM

Quote:

Originally Posted by jschiwal (Post 3423616)
You may have misunderstood my post. It was the location of -print0 that could be a problem. It needs to be after the `-type f' argument. Otherwise, you are printing results before restricting the results to -type f. So, you will get the "." directory and any other directories as results which isn't what you want.

Nope, I got you; I made the change. I wasn't aware order was important, but in some other versions of the script I purposely checked if the actioned list item was a file so I kinda fudged around it, but your solution is much neater - thanks.

Quote:

I checked in the abs-guide.pdf and found a usage of the $'\0nnn' construct. I just was curious if you knew about it. I've used `tr '\n' '\0' in the past to convert returns to nulls, but hadn't seen $'\0' before. It's the dollar sign that caught my eye, and I was hoping you knew why -d $'\0' works and -d '\0' doesn't.
I suspect I copied someone else without understanding why, but irrespective of that, I've put it back to just LF now but it still exhibits the same behaviour. I'll try the IFS set to linefeed as suggested above next....

ntubski 01-28-2009 06:56 PM

Quote:

Originally Posted by jschiwal (Post 3423616)
I checked in the abs-guide.pdf and found a usage of the $'\0nnn' construct. I just was curious if you knew about it. I've used `tr '\n' '\0' in the past to convert returns to nulls, but hadn't seen $'\0' before. It's the dollar sign that caught my eye, and I was hoping you knew why -d $'\0' works and -d '\0' doesn't.

The $'\0' is expanded (to the empty string, since bash strings are NUL-terminated) by bash, whereas in tr '\n' '\0', tr interprets the escape sequences.

From man bash
Code:

QUOTING
...
      Words of the form $'string' are treated specially.  The word expands to string, with backslash-
      escaped  characters  replaced as specified by the ANSI C standard.  Backslash escape sequences,
      if present, are decoded as follows:
              \a    alert (bell)
              \b    backspace
              \e    an escape character
              \f    form feed
              \n    new line
              \r    carriage return
              \t    horizontal tab
              \v    vertical tab
              \\    backslash
              \'    single quote
              \nnn  the eight-bit character whose value is the octal value nnn (one to three digits)
              \xHH  the eight-bit character whose value is the hexadecimal value HH (one or  two  hex
                    digits)
              \cx    a control-x character

EDIT

I prefixed all invocations of mv, rm, and ffmpeg with echo and the script ran as expected, which suggests that some command inside the loop is doing something funny. Does ffmpeg ever read from standard input? Does changing that line to
Code:

ffmpeg -i "${FILE}" -target dvd "${OUTPUT_DIR}/${OUTPUT_FILENAME}" &> "/tmp/${FILENAME_NOEXT}_pass.log" < /dev/null
help?

brumster 01-29-2009 03:50 AM

Bear with me; I'll give it a run ;)

I'm working with the "while read FILE" version for now, as that exhibits the same behaviour as the original. If we can fix that one then I'll try the same process with the original script, and my other ones I have based on it, and see how it goes.

I agree, playing around here it does seem that ffmpeg is doing something 'funny'. Trying your suggestion now...

EDIT: Bingo, that's it :D

ffmpeg doesn't sit held anywhere asking for input, but it does offer an abort keypress - could it be this?

I'll go back and try it on my original scripts and see if that address the problem there too - I suspect it will ;)

jschiwal 01-29-2009 06:27 PM

Thanks. I should have figured it out by trying "-d ''" instead of trying -d '\0'.


All times are GMT -5. The time now is 01:29 PM.