LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 06-13-2009, 12:09 PM   #1
propofol
Member
 
Registered: Nov 2007
Location: Seattle
Distribution: Debian Wheezy & Jessie; Ubuntu
Posts: 260

Rep: Reputation: 50
Scope of unnamed pipes in BASH script


I would appreciate help with a script which seems to run into problems because of nested pipes. I am trying to convert a series of cardiac ultrasound videos into flash with identifying info removed.

Code:
#!/bin/bash
#Convert all avi files found in $1 folder into flash video

find "$1" -name "*.avi" -type f -print | while read file
        do
        name="${file##*/}"
        base="${name%.*}"
        echo "Converting $file -->  ${base}.flv"

        ( ffmpeg -i "$file"  -pix_fmt yuv444p -f yuv4mpegpipe - | ffmpeg -i - -an -croptop 16 -padtop 16 -padcolor 0 -aspect 4:3 -vcodec flv -b 1000000 -pass 1 -y "${base}.flv" )
        ( ffmpeg -i "$file"  -pix_fmt yuv444p -f yuv4mpegpipe - | ffmpeg -i - -an -croptop 16 -padtop 16 -padcolor 0 -aspect 4:3 -vcodec flv -b 1000000 -pass 2 -y "${base}.flv" )

        done
The ffmpeg command works well on its own for two pass encoding however as soon as I add the "find ... | while read file" pipe the script fails (BTW there are to many files so command line expansion is not an option)

Regards,
Stefan
 
Old 06-13-2009, 01:54 PM   #2
antegallya
Member
 
Registered: Jun 2008
Location: Belgium
Distribution: Debian
Posts: 109

Rep: Reputation: 42
Hello,
how does it exactly fails ? Your script seems correct.
 
Old 06-13-2009, 07:44 PM   #3
propofol
Member
 
Registered: Nov 2007
Location: Seattle
Distribution: Debian Wheezy & Jessie; Ubuntu
Posts: 260

Original Poster
Rep: Reputation: 50
The script only processes the first file correctly then it seems to read in only a part of the file name into ${file} variable for the next files:


ie
Code:
mkflv.sh ~/test/

Converting /home/user/test/Image-1.avi -->  Image-1.flv
.... ffmpeg output ....
Converting /test/Image-2.avi -->  Image-2.flv
..... ffmpeg error: /test/Image-2.avi: no such file or directory ...
Regards,
Stefan

Last edited by propofol; 06-13-2009 at 07:45 PM.
 
Old 06-14-2009, 02:03 AM   #4
gnashley
Amigo developer
 
Registered: Dec 2003
Location: Germany
Distribution: Slackware
Posts: 4,749

Rep: Reputation: 461Reputation: 461Reputation: 461Reputation: 461Reputation: 461
Try using redirection instead of the pipe. It may also help to not run the ffmpeg commands in parallel, as shown.
Code:
#!/bin/bash
#Convert all avi files found in $1 folder into flash video

while read file
        do
        name="${file##*/}"
        base="${name%.*}"
        echo "Converting $file -->  ${base}.flv"

        ffmpeg -i "$file"  -pix_fmt yuv444p -f yuv4mpegpipe - | ffmpeg -i - -an -croptop 16 -padtop 16 -padcolor 0 -aspect 4:3 -vcodec flv -b 1000000 -pass 1 -y "${base}.flv"
        ffmpeg -i "$file"  -pix_fmt yuv444p -f yuv4mpegpipe - | ffmpeg -i - -an -croptop 16 -padtop 16 -padcolor 0 -aspect 4:3 -vcodec flv -b 1000000 -pass 2 -y "${base}.flv"

        done<$(find "$1" -name "*.avi" -type f -print)
 
Old 06-14-2009, 08:49 AM   #5
jlinkels
Senior Member
 
Registered: Oct 2003
Location: Bonaire
Distribution: Debian Lenny/Squeeze/Wheezy/Sid
Posts: 4,053

Rep: Reputation: 484Reputation: 484Reputation: 484Reputation: 484Reputation: 484
No spaces in the file names?

Run your script with sh -x /path/to/your/script

It shows the value of all variables and you might get insight on why that is.

I recommend to avoid pipes in loops as the plague. In your original script code seems to be fine and this piping is allowed. However, in most other cases mistakes are easily made, not taking into account the piped process runs in its own subshell. Errors can be very hard to trace.

jlinkels
 
Old 06-14-2009, 10:48 PM   #6
propofol
Member
 
Registered: Nov 2007
Location: Seattle
Distribution: Debian Wheezy & Jessie; Ubuntu
Posts: 260

Original Poster
Rep: Reputation: 50
Thanks for the replies.

I unfortunately had to use the pipes with ffmpeg because the "croptop" option gave me an error about wrong format. I have also been unable to find info on how to use libavfilter (which apparently replaced vhook) to mask out a certain part of a video.

Quote:
done<$(find "$1" -name "*.avi" -type f -print)
This redirect gives me the following error:
Code:
line 13: $(find "$1" -name "*.avi" -type f -print): ambiguous redirect
Regards,
Stefan
 
Old 06-14-2009, 11:20 PM   #7
Kenhelm
Member
 
Registered: Mar 2008
Location: N. W. England
Distribution: Mandriva
Posts: 329

Rep: Reputation: 140Reputation: 140
Try
Code:
done < <(find "$1" -name "*.avi" -type f -print)   # bash 2.x & 3.x
# or
done <<< "$(find "$1" -name "*.avi" -type f -print)"   # bash 3.x
http://linuxshellaccount.blogspot.co...output-to.html

Last edited by Kenhelm; 06-14-2009 at 11:54 PM. Reason: Put double quotes in <<< "$( .... )"
 
Old 06-15-2009, 12:09 AM   #8
propofol
Member
 
Registered: Nov 2007
Location: Seattle
Distribution: Debian Wheezy & Jessie; Ubuntu
Posts: 260

Original Poster
Rep: Reputation: 50
The "bash -x script.sh" seems very useful and I was also unaware of the "< <(find ...)" construct. Strangely enough the same error happens ie when the loop repeats, the variable read into $file is truncated ie /home/user/test/video2.avi becomes /test/video2.avi.

I have solved the problem by making two scripts - maybe not very elegant but less pipes:

Code:
#!/bin/bash
#Convert avi to flash video
find "$1" -name "*.avi" -type f  -print0 | xargs -0I {} ~/flv.sh '{}'
flv.sh:
Code:
#!/bin/bash
#Convert into flash video

file=$1
new="${file##*/}"
base="${new%.*}"
echo "$file -->  ${base}.flv"

ffmpeg -i "$file"  -pix_fmt yuv444p -f yuv4mpegpipe - | ffmpeg -i - -an -croptop 16 -padtop 16 -padcolor 0 -aspect 4:3 -vcodec flv -b 1000000 -pass 1 /dev/null
ffmpeg -i "$file"  -pix_fmt yuv444p -f yuv4mpegpipe - | ffmpeg -i - -an -croptop 16 -padtop 16 -padcolor 0 -aspect 4:3 -vcodec flv -b 1000000 -pass 2 -y "${base}.flv"
Regards,
Stefan

Last edited by propofol; 06-15-2009 at 12:11 AM.
 
Old 06-15-2009, 03:12 AM   #9
antegallya
Member
 
Registered: Jun 2008
Location: Belgium
Distribution: Debian
Posts: 109

Rep: Reputation: 42
Hello,
fine you got it working.
But having to do such things to get that working seems really strange. The original script works here, does anyone get the same problem as profofol ?
Profofol, wich version of bash are you using ?
Is it always the substring "/home/user" that is removed regardless of the location of the files and of the directory of invocation ? Even if you put the files in /tmp/home/user/test/ ?
And if you remove the ffmpeg lines, I presume the $file variable is still truncated, isn't it ?
Do you use a special set of options at bash invocation ?

Does anyone know where the problem may come from or have any idea ?
 
Old 06-15-2009, 11:01 PM   #10
propofol
Member
 
Registered: Nov 2007
Location: Seattle
Distribution: Debian Wheezy & Jessie; Ubuntu
Posts: 260

Original Poster
Rep: Reputation: 50
My version of bash is: GNU bash, version 3.2.48(1)-release (i486-pc-linux-gnu)

When I do the same script but with
Code:
echo "abcdef" | cat -
in the place of ffmpeg statements, it works fine. Also, using variable expansion ie
Code:
for file in $(find "$1" -name "*.avi" -type f  -print )
do
...
done
works as I would expect.

Also when I use:


The result of moving it to /data/test is:
Code:
 ~/newflv.sh /data/test/
Converting /data/test/Image-1.avi -->  Image-1.flv
.... ffmpeg output ...
Converting avi -->  avi.flv
// This should have been /data/test/Image-2.avi -->  Image-2.flv
.... ffmpeg error - avi: no such file or directory ...
Converting /data/test/Image-3.avi -->  Image-3.flv 
.... ffmpeg ...
// execution stops however it should have continued to with one more avi
The file list in /data/test/ is:
Image-1.avi
Image-2.avi
Image-3.avi
Image-5.avi
 
Old 06-16-2009, 03:28 AM   #11
antegallya
Member
 
Registered: Jun 2008
Location: Belgium
Distribution: Debian
Posts: 109

Rep: Reputation: 42
Oww, that ffmpeg may be reading from stdin may explain that behavior.
I'll have to do some tests.

But why does it work here ? Maybe we'll have to look more at ffmpeg.

Last edited by antegallya; 06-16-2009 at 03:29 AM.
 
Old 06-16-2009, 04:00 AM   #12
antegallya
Member
 
Registered: Jun 2008
Location: Belgium
Distribution: Debian
Posts: 109

Rep: Reputation: 42
Oh yeah got it (I think).

Only suppositions :
For some reason, in
Quote:
ffmpeg -i "$file" -pix_fmt yuv444p -f yuv4mpegpipe - | ffmpeg -i - -an -croptop 16 -padtop 16 -padcolor 0 -aspect 4:3 -vcodec flv -b 1000000 -pass 1 /dev/null
the second ffmpeg continue to read stdin even when the first one doesn't produce any data (missing end of file or something ?) and then, since stdin is inherited from the parent shell, the second ffmpeg reads a part of a file name (why does it stops then, I don't know, maybe does ffmpeg read n bytes of data after some marker he read before or something ?).

So, redirecting stdin to /dev/null in the subshell should do the trick :
Quote:
( exec <"/dev/null"; ffmpeg -i "$file" -pix_fmt yuv444p -f yuv4mpegpipe - | ffmpeg -i - -an -croptop 16 -padtop 16 -padcolor 0 -aspect 4:3 -vcodec flv -b 1000000 -pass 1 -y "${base}.flv" )
( exec <"/dev/null"; ffmpeg -i "$file" -pix_fmt yuv444p -f yuv4mpegpipe - | ffmpeg -i - -an -croptop 16 -padtop 16 -padcolor 0 -aspect 4:3 -vcodec flv -b 1000000 -pass 2 -y "${base}.flv" )
 
1 members found this post helpful.
Old 06-16-2009, 10:47 PM   #13
propofol
Member
 
Registered: Nov 2007
Location: Seattle
Distribution: Debian Wheezy & Jessie; Ubuntu
Posts: 260

Original Poster
Rep: Reputation: 50
Amazing - worked 1st time. I am still trying to get my head around the 'exec <"/dev/null"' part.

Regards,
Stefan

PS This is an example of what the output is: Image-1.flv
 
Old 06-17-2009, 12:54 AM   #14
antegallya
Member
 
Registered: Jun 2008
Location: Belgium
Distribution: Debian
Posts: 109

Rep: Reputation: 42
Nice!
'exec <"/dev/null"' is part of bash's I/O redirections, it's explained there :
http://tldp.org/LDP/abs/html/io-redirection.html
It is a compact form form
Quote:
exec 0 < "/dev/null"
that is : replace the stdin file descriptor by the "/dev/null" file descriptor in the entire shell.

Regards,
Antegallya
 
1 members found this post helpful.
  


Reply

Tags
bash, ffmpeg, pipes, script


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: Using Named Pipes (FIFOs) with Bash LXer Syndicated Linux News 0 03-28-2009 04:42 AM
Bash - named pipes - mathew42 insight please PAix Programming 3 12-11-2007 09:09 AM
Bash, Env, Set, Export, Pipes....Scripting dtdionne Programming 6 09-06-2007 01:04 AM
Easy Bash Loop Scope Issue? scsi.king Programming 7 04-25-2007 05:41 AM
pipes and bash scripts twistedpair Linux - General 0 06-03-2004 09:59 PM


All times are GMT -5. The time now is 03:16 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration