[SOLVED] Scope of unnamed pipes in BASH script

propofol · 06-13-2009, 12:09 PM

I would appreciate help with a script which seems to run into problems because of nested pipes. I am trying to convert a series of cardiac ultrasound videos into flash with identifying info removed.

Code:

#!/bin/bash
#Convert all avi files found in $1 folder into flash video

find "$1" -name "*.avi" -type f -print | while read file
        do
        name="${file##*/}"
        base="${name%.*}"
        echo "Converting $file -->  ${base}.flv"

        ( ffmpeg -i "$file"  -pix_fmt yuv444p -f yuv4mpegpipe - | ffmpeg -i - -an -croptop 16 -padtop 16 -padcolor 0 -aspect 4:3 -vcodec flv -b 1000000 -pass 1 -y "${base}.flv" )
        ( ffmpeg -i "$file"  -pix_fmt yuv444p -f yuv4mpegpipe - | ffmpeg -i - -an -croptop 16 -padtop 16 -padcolor 0 -aspect 4:3 -vcodec flv -b 1000000 -pass 2 -y "${base}.flv" )

        done

The ffmpeg command works well on its own for two pass encoding however as soon as I add the "find ... | while read file" pipe the script fails (BTW there are to many files so command line expansion is not an option)

Regards,
Stefan

antegallya · 06-13-2009, 01:54 PM

Hello,
how does it exactly fails ? Your script seems correct.

propofol · 06-13-2009, 07:44 PM

The script only processes the first file correctly then it seems to read in only a part of the file name into ${file} variable for the next files:

ie

Code:

mkflv.sh ~/test/

Converting /home/user/test/Image-1.avi -->  Image-1.flv
.... ffmpeg output ....
Converting /test/Image-2.avi -->  Image-2.flv
..... ffmpeg error: /test/Image-2.avi: no such file or directory ...

Regards,
Stefan

gnashley · 06-14-2009, 02:03 AM

Try using redirection instead of the pipe. It may also help to not run the ffmpeg commands in parallel, as shown.

Code:

#!/bin/bash
#Convert all avi files found in $1 folder into flash video

while read file
        do
        name="${file##*/}"
        base="${name%.*}"
        echo "Converting $file -->  ${base}.flv"

        ffmpeg -i "$file"  -pix_fmt yuv444p -f yuv4mpegpipe - | ffmpeg -i - -an -croptop 16 -padtop 16 -padcolor 0 -aspect 4:3 -vcodec flv -b 1000000 -pass 1 -y "${base}.flv"
        ffmpeg -i "$file"  -pix_fmt yuv444p -f yuv4mpegpipe - | ffmpeg -i - -an -croptop 16 -padtop 16 -padcolor 0 -aspect 4:3 -vcodec flv -b 1000000 -pass 2 -y "${base}.flv"

        done<$(find "$1" -name "*.avi" -type f -print)

jlinkels · 06-14-2009, 08:49 AM

No spaces in the file names?

Run your script with sh -x /path/to/your/script

It shows the value of all variables and you might get insight on why that is.

I recommend to avoid pipes in loops as the plague. In your original script code seems to be fine and this piping is allowed. However, in most other cases mistakes are easily made, not taking into account the piped process runs in its own subshell. Errors can be very hard to trace.

jlinkels

propofol · 06-14-2009, 10:48 PM

Thanks for the replies.

I unfortunately had to use the pipes with ffmpeg because the "croptop" option gave me an error about wrong format. I have also been unable to find info on how to use libavfilter (which apparently replaced vhook) to mask out a certain part of a video.

Quote:

done<$(find "$1" -name "*.avi" -type f -print)

This redirect gives me the following error:

Code:

line 13: $(find "$1" -name "*.avi" -type f -print): ambiguous redirect

Regards,
Stefan

Kenhelm · 06-14-2009, 11:20 PM

Try

Code:

done < <(find "$1" -name "*.avi" -type f -print)   # bash 2.x & 3.x
# or
done <<< "$(find "$1" -name "*.avi" -type f -print)"   # bash 3.x

http://linuxshellaccount.blogspot.co...output-to.html

propofol · 06-15-2009, 12:09 AM

The "bash -x script.sh" seems very useful and I was also unaware of the "< <(find ...)" construct. Strangely enough the same error happens ie when the loop repeats, the variable read into $file is truncated ie /home/user/test/video2.avi becomes /test/video2.avi.

I have solved the problem by making two scripts - maybe not very elegant but less pipes:

Code:

#!/bin/bash
#Convert avi to flash video
find "$1" -name "*.avi" -type f  -print0 | xargs -0I {} ~/flv.sh '{}'

flv.sh:

Code:

#!/bin/bash
#Convert into flash video

file=$1
new="${file##*/}"
base="${new%.*}"
echo "$file -->  ${base}.flv"

ffmpeg -i "$file"  -pix_fmt yuv444p -f yuv4mpegpipe - | ffmpeg -i - -an -croptop 16 -padtop 16 -padcolor 0 -aspect 4:3 -vcodec flv -b 1000000 -pass 1 /dev/null
ffmpeg -i "$file"  -pix_fmt yuv444p -f yuv4mpegpipe - | ffmpeg -i - -an -croptop 16 -padtop 16 -padcolor 0 -aspect 4:3 -vcodec flv -b 1000000 -pass 2 -y "${base}.flv"

Regards,
Stefan

antegallya · 06-15-2009, 03:12 AM

Hello,
fine you got it working.
But having to do such things to get that working seems really strange. The original script works here, does anyone get the same problem as profofol ?
Profofol, wich version of bash are you using ?
Is it always the substring "/home/user" that is removed regardless of the location of the files and of the directory of invocation ? Even if you put the files in /tmp/home/user/test/ ?
And if you remove the ffmpeg lines, I presume the $file variable is still truncated, isn't it ?
Do you use a special set of options at bash invocation ?

Does anyone know where the problem may come from or have any idea ?

propofol · 06-15-2009, 11:01 PM

My version of bash is: GNU bash, version 3.2.48(1)-release (i486-pc-linux-gnu)

When I do the same script but with

Code:

echo "abcdef" | cat -

in the place of ffmpeg statements, it works fine. Also, using variable expansion ie

Code:

for file in $(find "$1" -name "*.avi" -type f  -print )
do
...
done

works as I would expect.

Also when I use:

The result of moving it to /data/test is:

Code:

 ~/newflv.sh /data/test/
Converting /data/test/Image-1.avi -->  Image-1.flv
.... ffmpeg output ...
Converting avi -->  avi.flv
// This should have been /data/test/Image-2.avi -->  Image-2.flv
.... ffmpeg error - avi: no such file or directory ...
Converting /data/test/Image-3.avi -->  Image-3.flv 
.... ffmpeg ...
// execution stops however it should have continued to with one more avi

The file list in /data/test/ is:
Image-1.avi
Image-2.avi
Image-3.avi
Image-5.avi

antegallya · 06-16-2009, 03:28 AM

Oww, that ffmpeg may be reading from stdin may explain that behavior.
I'll have to do some tests.

But why does it work here ? Maybe we'll have to look more at ffmpeg.

antegallya · 06-16-2009, 04:00 AM

Oh yeah got it (I think).

Only suppositions :
For some reason, in

Quote:

ffmpeg -i "$file" -pix_fmt yuv444p -f yuv4mpegpipe - | ffmpeg -i - -an -croptop 16 -padtop 16 -padcolor 0 -aspect 4:3 -vcodec flv -b 1000000 -pass 1 /dev/null

the second ffmpeg continue to read stdin even when the first one doesn't produce any data (missing end of file or something ?) and then, since stdin is inherited from the parent shell, the second ffmpeg reads a part of a file name (why does it stops then, I don't know, maybe does ffmpeg read n bytes of data after some marker he read before or something ?).

So, redirecting stdin to /dev/null in the subshell should do the trick :

Quote:

( exec <"/dev/null"; ffmpeg -i "$file" -pix_fmt yuv444p -f yuv4mpegpipe - | ffmpeg -i - -an -croptop 16 -padtop 16 -padcolor 0 -aspect 4:3 -vcodec flv -b 1000000 -pass 1 -y "${base}.flv" )
( exec <"/dev/null"; ffmpeg -i "$file" -pix_fmt yuv444p -f yuv4mpegpipe - | ffmpeg -i - -an -croptop 16 -padtop 16 -padcolor 0 -aspect 4:3 -vcodec flv -b 1000000 -pass 2 -y "${base}.flv" )

propofol · 06-16-2009, 10:47 PM

Amazing - worked 1st time. I am still trying to get my head around the 'exec <"/dev/null"' part.

Regards,
Stefan

PS This is an example of what the output is: Image-1.flv

antegallya · 06-17-2009, 12:54 AM

Nice!
'exec <"/dev/null"' is part of bash's I/O redirections, it's explained there :
http://tldp.org/LDP/abs/html/io-redirection.html
It is a compact form form

Quote:

exec 0 < "/dev/null"

that is : replace the stdin file descriptor by the "/dev/null" file descriptor in the entire shell.

Regards,
Antegallya