LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 09-09-2018, 04:25 AM   #1
frater
Member
 
Registered: Jul 2008
Posts: 120

Rep: Reputation: 23
bash script goes off rails when using ffmpeg, I have no clue


I have a series of wav-files and I want to remove all the trailing and preceding silent space. The procedure itself is not the problem, although I may want to revisit that later.

My problem is with the bash script with which I'm doing it.
I've written many bash scripts and I think I have covered many of the pitfalls, but here I don't know what's happening.

It's a simple loop and the ffmpeg command is somehow influencing the variable in my bash script. I've never seen this before.

I the current folder I have a list of wav-files and they consist of 4 numbers and then "dot wav"

0000.wav
0001.wav
0002.wav
0003.wav

I collect these wav-files using the command find and put these in a file in a temporary folder.
If I cat this file it shows up nice...

./0000.wav
./0001.wav
./0002.wav

If I then use it in a "while read SOUND ; do" loop it runs fine too.

Only when I use the ffmpeg command in it and use "SOUND" as a parameter, something strange happens.

Here's the script and the output.
If I comment the remaining "ffmpeg line" it loops nice.
I'm hoping someone will give me an "ahaa! moment"

cat /usr/local/sbin/remsilence
Code:
#!/bin/bash

EXT=wav
TMPDIR=`mktemp -t -d ${0//*\/}.XXXXXXXXXX`
find -maxdepth 1 -type f -name \*.${EXT} | grep "${EXT}$" | sort  >${TMPDIR}/sounds
rm -r OUTPUT 2>/dev/null
mkdir OUTPUT

cutfile()
{
    echo "${SOUND}"
    ffmpeg -loglevel fatal -y -i ${SOUND} -af silenceremove=1:0:-96dB  ${TMPDIR}/begincut.${EXT}

    #ffmpeg -loglevel fatal -y -i ${TMPDIR}/begincut.${EXT} -af areverse  ${TMPDIR}/reversed.${EXT} || return 1
    #ffmpeg -loglevel fatal -y -i ${TMPDIR}/reversed.${EXT} -af silenceremove=1:0:-96dB  ${TMPDIR}/revcut.${EXT} || return 1
    #ffmpeg -loglevel fatal -y -i ${TMPDIR}/revcut.${EXT}   -af areverse  ${TMPDIR}/finished.${EXT} || return 1
    #mv "${TMPDIR}/finished.${EXT}" "OUTPUT/`filename ${SOUND}`"
}

head -n5 ${TMPDIR}/sounds

while read SOUND ; do
  cutfile

done<${TMPDIR}/sounds
rm -r ${TMPDIR}
remsilence
Code:
./0000.wav
./0001.wav
./0002.wav
./0003.wav
./0004.wav
./0000.wav
/0001.wav
./0002.wav
/0003.wav
./0004.wav
/0005.wav
./0006.wav
/0007.wav
./0008.wav
/0009.wav
./0010.wav
Why it is deleting the preceding period?????
 
Old 09-09-2018, 04:46 AM   #2
ondoho
LQ Addict
 
Registered: Dec 2013
Posts: 13,390
Blog Entries: 9

Rep: Reputation: 3652Reputation: 3652Reputation: 3652Reputation: 3652Reputation: 3652Reputation: 3652Reputation: 3652Reputation: 3652Reputation: 3652Reputation: 3652Reputation: 3652
i found that read can read all sorts of things, very unexpected.
i also found that ffmpeg is not so precise with using stdout and stderr etc.
shortly, all sorts of weird stuff can happen.

also, shouldn't you call cutfile with the file it is going to cut as "$1"?

and why don't you just loop over the files directly instead of saving them to a file first?

so instead of:
Code:
while read SOUND ; do
  cutfile

done<${TMPDIR}/sounds
why don't you just use:
Code:
for SOUND in ????.wav; do
  cutfile "$SOUND"
done
???
 
Old 09-09-2018, 05:02 AM   #3
frater
Member
 
Registered: Jul 2008
Posts: 120

Original Poster
Rep: Reputation: 23
Hi...
Thanks for thinking with me...

The solution is somewhere else, although I haven't yet figured why.
BTW.... without reading your answer I tried cutfile "${SOUND}", but that behaved the same
The problem is that bash is treating it as numbers.
But I still don't understand why it's not doing that immediately wrong.

I changed
Code:
find -maxdepth 1 -type f -name \*.${EXT}
into
Code:
find `pwd`-maxdepth 1 -type f -name \*.${EXT}
which immediately changed a lot

EDIT

No... it cuts 5 characters away

Last edited by frater; 09-09-2018 at 05:06 AM.
 
Old 09-09-2018, 05:09 AM   #4
ondoho
LQ Addict
 
Registered: Dec 2013
Posts: 13,390
Blog Entries: 9

Rep: Reputation: 3652Reputation: 3652Reputation: 3652Reputation: 3652Reputation: 3652Reputation: 3652Reputation: 3652Reputation: 3652Reputation: 3652Reputation: 3652Reputation: 3652
Quote:
Originally Posted by frater View Post
The problem is that bash is treating it as numbers.
sorry but you're going to have to clarify that.
as it stands, it's wrong - bash treats everything as strings unless you tell it explicitely to use arithmetics.
 
Old 09-09-2018, 05:19 AM   #5
frater
Member
 
Registered: Jul 2008
Posts: 120

Original Poster
Rep: Reputation: 23
Hi.... I already corrected that....
I was just trying to find a reason for this unexpected behaviour....
 
Old 09-09-2018, 05:24 AM   #6
frater
Member
 
Registered: Jul 2008
Posts: 120

Original Poster
Rep: Reputation: 23
Maybe I should just try another tool for this.
Never had this before. It's just as if I don't know anything about bash anymore...
 
Old 09-09-2018, 05:33 AM   #7
frater
Member
 
Registered: Jul 2008
Posts: 120

Original Poster
Rep: Reputation: 23
cat /usr/local/sbin/remsilence
Code:
#!/bin/bash

EXT=wav
TMPDIR=`mktemp -t -d ${0//*\/}.XXXXXXXXXX`
mkdir ${TMPDIR}/LIST
find `pwd` -maxdepth 1 -type f -name \*.${EXT} | grep "${EXT}$" | sort  >${TMPDIR}/LIST/sounds
rm -r OUTPUT 2>/dev/null
mkdir OUTPUT

cutfile()
{
    echo "${1}"
    ffmpeg -loglevel fatal -y -i ${1} -af silenceremove=1:0:-96dB  ${TMPDIR}/begincut.${EXT} || return 1

    ffmpeg -loglevel fatal -y -i ${TMPDIR}/begincut.${EXT} -af areverse  ${TMPDIR}/reversed.${EXT}
    ffmpeg -loglevel fatal -y -i ${TMPDIR}/reversed.${EXT} -af silenceremove=1:0:-96dB  ${TMPDIR}/revcut.${EXT}
    ffmpeg -loglevel fatal -y -i ${TMPDIR}/revcut.${EXT}   -af areverse  ${TMPDIR}/finished.${EXT}
    mv "${TMPDIR}/finished.${EXT}" "OUTPUT/`filename ${1}`"
    rm -f ${TMPDIR}/*.${EXT}
}

head -n5 ${TMPDIR}/LIST/sounds

while read SOUND ; do
  cutfile "${SOUND}"

done<${TMPDIR}/LIST/sounds
rm -r ${TMPDIR}
remsilence
Code:
/WORKDIR/0000.wav
/WORKDIR/0001.wav
/WORKDIR/0002.wav
/WORKDIR/0003.wav
/WORKDIR/0004.wav
/WORKDIR/0000.wav
KDIR/0001.wav
/WORKDIR/0002.wav
KDIR/0003.wav
/WORKDIR/0004.wav
KDIR/0005.wav
/WORKDIR/0006.wav
KDIR/0007.wav
/WORKDIR/0008.wav
KDIR/0009.wav
 
Old 09-09-2018, 05:42 AM   #8
frater
Member
 
Registered: Jul 2008
Posts: 120

Original Poster
Rep: Reputation: 23
Even when I accept that ffmpeg is doing strange things.
Why is it doing this only for the even tries.
 
Old 09-09-2018, 06:24 AM   #9
lougavulin
Member
 
Registered: Jul 2018
Distribution: Slackware,x86_64,current
Posts: 262

Rep: Reputation: 94
There is two ways to solve this.

Add -nostdin option to your ffmepg commands. From ffmepg's man :
Code:
       -stdin
           Enable interaction on standard input. On by default unless standard input is used as an input. To explicitly disable
           interaction you need to specify "-nostdin".

           Disabling interaction on standard input is useful, for example, if ffmpeg is in the background process group. Roughly
           the same result can be achieved with "ffmpeg ... < /dev/null" but it requires a shell.
Code:
ffmpeg -nostdin -loglevel fatal -y -i ${1} -af silenceremove=1:0:-96dB  ${TMPDIR}/begincut.${EXT} || return 1
[...]
Or replace while loop by for loop :
Code:
for SOUND in $(cat ${TMPDIR}/LIST/sounds); do
  cutfile "${SOUND}"
done
 
1 members found this post helpful.
Old 09-09-2018, 07:14 AM   #10
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 13,576

Rep: Reputation: 4341Reputation: 4341Reputation: 4341Reputation: 4341Reputation: 4341Reputation: 4341Reputation: 4341Reputation: 4341Reputation: 4341Reputation: 4341Reputation: 4341
you might want to insert set -xv at the beginning of your script to see what's happening. Probably that will give you some additional info.
Replacing that while cycle to for is not really suggested, using while is the preferred solution.
Code:
find `pwd` -maxdepth 1 -type f -name \*.${EXT} | grep "${EXT}$" | sort  >${TMPDIR}/LIST/sounds
can be probably replaced:
Code:
ls -1 *.${EXT} >${TMPDIR}/LIST/sounds
 
Old 09-09-2018, 11:38 AM   #11
frater
Member
 
Registered: Jul 2008
Posts: 120

Original Poster
Rep: Reputation: 23
Thanks all for thinking with me.
lougavulin gave me the key to the solution by telling me I should use the -nostdin with ffmpeg.
The script then started to behave like I would expect it to.

This is the first time that some binary is messing up my bash-script.
Does this happen more?
I would think it does.

I would really like to hear of some more pitfalls

Some things that I did to find some solutions that didn't work I have reverted in this script

cat /usr/local/sbin/remsilence
Code:
#!/bin/bash

EXT=wav
TMPDIR=`mktemp -t -d ${0//*\/}.XXXXXXXXXX`
find -maxdepth 1 -type f -name \*.${EXT} | grep "${EXT}$" | cut -b3- | sort  >${TMPDIR}/sounds
rm -r OUTPUT 2>/dev/null
mkdir OUTPUT

cutfile()
{
    echo "${SOUND}"
    ffmpeg -nostdin -loglevel fatal -y -i ${SOUND} -af silenceremove=1:0:-96dB  ${TMPDIR}/begincut.${EXT} || return 1

    ffmpeg -nostdin -loglevel fatal -y -i ${TMPDIR}/begincut.${EXT} -af areverse  ${TMPDIR}/reversed.${EXT}
    ffmpeg -nostdin -loglevel fatal -y -i ${TMPDIR}/reversed.${EXT} -af silenceremove=1:0:-96dB  ${TMPDIR}/revcut.${EXT}
    ffmpeg -nostdin -loglevel fatal -y -i ${TMPDIR}/revcut.${EXT}   -af areverse  ${TMPDIR}/finished.${EXT}
    mv "${TMPDIR}/finished.${EXT}" "OUTPUT/${SOUND}"
    rm -f ${TMPDIR}/*.${EXT}
}

head -n5 ${TMPDIR}/sounds

while read SOUND ; do
  cutfile
done<${TMPDIR}/sounds
rm -r ${TMPDIR}

Last edited by frater; 09-09-2018 at 11:42 AM.
 
Old 09-09-2018, 11:42 AM   #12
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 13,576

Rep: Reputation: 4341Reputation: 4341Reputation: 4341Reputation: 4341Reputation: 4341Reputation: 4341Reputation: 4341Reputation: 4341Reputation: 4341Reputation: 4341Reputation: 4341
this is not "some binaries", but there were two apps reading the same stdin stream (fed by <${TMPDIR}/LIST/sounds):
1. obviously the read command
2. now it was the ffmpeg tool.
 
1 members found this post helpful.
Old 09-09-2018, 01:04 PM   #13
frater
Member
 
Registered: Jul 2008
Posts: 120

Original Poster
Rep: Reputation: 23
After a while you start thinking of bash as a programming language and don't think how they achieve the things you actually want them to do.
At least I did....
 
Old 09-10-2018, 08:23 AM   #14
frater
Member
 
Registered: Jul 2008
Posts: 120

Original Poster
Rep: Reputation: 23
I now have a way to trim the silence of all wavs in the current folder.
But I'm not satisfied yet.
Each ffmpeg action results in a loss. I don't want this.
I'm already working with uncompressed WAVs and I see no reason why I should except any loss.

That's why I tried to use a different approach.
At first I tried "silencedetect", but that gave me unreliable results.

Then I thought of a bit "clumsy" way by using the results of the "silenceremove" process and use it to just cut the begin and end of the original WAV-file.
That worked.
But I still wasn't satisfied. According to the ffmpeg manual there is still some loss unless I use "-c:a copy".
So I added that to the command, but then I noticed it sometimes cut into the good part.

It seems ffmpeg uses frames when it is used with the "-c:a copy" command.
I then thought of shifting the parameters with 15ms so it has a better chance to cut at the right place.

Maybe I should ffprobe and extract "pkt_duration_time" and divide that by 2 and use that as offset?

Any comments are welcome.



cat /usr/local/sbin/remsilence
Code:
cat /usr/local/sbin/remsilence
#!/bin/bash

EXT=wav
THRESHOLD=70
TMPDIR=`mktemp -t -d ${0//*\/}.XXXXXXXXXX`
find -maxdepth 1 -type f -name \*.${EXT} | grep "${EXT}$" | cut -b3- | sort  >${TMPDIR}/sounds
rm -r OUTPUT 2>/dev/null
rm -r LOSSLESS 2>/dev/null
rm -r LESSLOSS 2>/dev/null
mkdir OUTPUT
mkdir LESSLOSS
mkdir LOSSLESS

cutsox()
{
    echo "${SOUND}"
    sox ${SOUND} ${TMPDIR}/begincut.${EXT} silence 1 0 -${THRESHOLD}d || return 1
    sox ${TMPDIR}/begincut.${EXT} ${TMPDIR}/reversed.${EXT} reverse
    sox ${TMPDIR}/reversed.${EXT} ${TMPDIR}/revcut.${EXT} silence 1 0 -${THRESHOLD}d
    sox ${TMPDIR}/revcut.${EXT}   ${TMPDIR}/finished.${EXT} reverse
}

cutffmpeg()
{
    echo "${SOUND}"
    ffmpeg -nostdin -loglevel fatal -y -i ${SOUND} -af silenceremove=1:0:-${THRESHOLD}dB ${TMPDIR}/begincut.${EXT} || return 1

    ffmpeg -nostdin -loglevel fatal -y -i ${TMPDIR}/begincut.${EXT} -af areverse ${TMPDIR}/reversed.${EXT}
    ffmpeg -nostdin -loglevel fatal -y -i ${TMPDIR}/reversed.${EXT} -af silenceremove=1:0:-${THRESHOLD}dB ${TMPDIR}/revcut.${EXT}
    ffmpeg -nostdin -loglevel fatal -y -i ${TMPDIR}/revcut.${EXT}   -af areverse ${TMPDIR}/finished.${EXT}
}

while read SOUND ; do
  cutffmpeg
  #cutsox

  cp "${TMPDIR}/finished.${EXT}" "OUTPUT/${SOUND}"

  ORGLENG=`ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 ${SOUND}`
  BEGINCT=`ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 ${TMPDIR}/begincut.${EXT}`
  FINLENG=`ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 ${TMPDIR}/finished.${EXT}`
  STARTSILENCE=`echo "${ORGLENG} ${BEGINCT}" | awk '{printf $1 - $2}'`
  ENDSILENCE=`echo "${BEGINCT} ${FINLENG}" | awk '{printf $1 - $2}'`

  echo "ORGLENG=${ORGLENG}"
  echo "BEGINCT=${BEGINCT}"
  echo "FINLENG=${FINLENG}"
  echo "STARTSILENCE=${STARTSILENCE}"
  echo "ENDSILENCE=${ENDSILENCE}"

  echo "${STARTSILENCE}" | grep -q 'e' && STARTSILENCE="0.0"

  ffmpeg -nostdin -loglevel fatal -y -i ${SOUND} -ss ${STARTSILENCE} -t ${FINLENG} LESSLOSS/${SOUND}

  # extract framelength in seconds 
  OFFSET=`ffprobe ${SOUND} -show_frames 2>/dev/null | grep pkt_duration_time | head -n1 | awk -F= '{print $2}'`
  # distract frame rate to make sure it is cut in the silence
  
  STARTSILENCE=`echo "${STARTSILENCE} ${OFFSET}" | awk '{printf $1 - $2}'`
  # if it is negative, make it 0
  echo "${STARTSILENCE}" | grep -q '-' && STARTSILENCE="0.0"
  # add the distracted size to the length
  FINLENG=`echo "${FINLENG} ${OFFSET}" | awk '{printf $1 + $2}'`

  echo "${SOUND} = ${ORGLENG} seconds with ${STARTSILENCE}s silence at start and ${ENDSILENCE}s at end. Result is ${FINLENG}s"
  echo "Offset = ${OFFSET}"

  ffmpeg -nostdin -loglevel fatal -y -i ${SOUND} -ss ${STARTSILENCE} -t ${FINLENG} -c:a copy LOSSLESS/${SOUND}


  rm -f ${TMPDIR}/*.${EXT}

done<${TMPDIR}/sounds
rm -r ${TMPDIR}

Last edited by frater; 09-10-2018 at 10:17 AM.
 
Old 09-10-2018, 10:15 AM   #15
frater
Member
 
Registered: Jul 2008
Posts: 120

Original Poster
Rep: Reputation: 23
In the end I just went with the "LESSLOSS" option.
It gives predictable results regarding the places it cuts the file and it's "one-pass" with probably no audible difference.
I'm using it for a set of WAVs that are generated by a TTS-engine and used for OpenTX (a radio for Remote Controlled Airplanes).

Having a predictable silence is more important and I don't need to check all files to make sure that none are "cut in the flesh".
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: Rails::API strips the fat off Ruby on Rails LXer Syndicated Linux News 0 11-23-2012 10:50 PM
[SOLVED] Why is ffmpeg stopping after the first encode when using my bash script? slinkoff Programming 6 07-06-2011 05:27 AM
[SOLVED] stuck at bash script to convert files with ffmpeg panseluta Linux - Newbie 6 03-10-2011 10:20 AM
A Personal BASH FFmpeg Batch Script (WIP) kDest Programming 9 04-24-2009 11:11 AM
Foreach script to copy files gone wrong... clue? xiaodown Programming 2 11-03-2004 12:00 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 03:37 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration