[SOLVED] bash script goes off rails when using ffmpeg, I have no clue

frater · 09-09-2018, 03:25 AM

I have a series of wav-files and I want to remove all the trailing and preceding silent space. The procedure itself is not the problem, although I may want to revisit that later.

My problem is with the bash script with which I'm doing it.
I've written many bash scripts and I think I have covered many of the pitfalls, but here I don't know what's happening.

It's a simple loop and the ffmpeg command is somehow influencing the variable in my bash script. I've never seen this before.

I the current folder I have a list of wav-files and they consist of 4 numbers and then "dot wav"

0000.wav
0001.wav
0002.wav
0003.wav

I collect these wav-files using the command find and put these in a file in a temporary folder.
If I cat this file it shows up nice...

./0000.wav
./0001.wav
./0002.wav

If I then use it in a "while read SOUND ; do" loop it runs fine too.

Only when I use the ffmpeg command in it and use "SOUND" as a parameter, something strange happens.

Here's the script and the output.
If I comment the remaining "ffmpeg line" it loops nice.
I'm hoping someone will give me an "ahaa! moment"

cat /usr/local/sbin/remsilence

Code:

#!/bin/bash

EXT=wav
TMPDIR=`mktemp -t -d ${0//*\/}.XXXXXXXXXX`
find -maxdepth 1 -type f -name \*.${EXT} | grep "${EXT}$" | sort  >${TMPDIR}/sounds
rm -r OUTPUT 2>/dev/null
mkdir OUTPUT

cutfile()
{
    echo "${SOUND}"
    ffmpeg -loglevel fatal -y -i ${SOUND} -af silenceremove=1:0:-96dB  ${TMPDIR}/begincut.${EXT}

    #ffmpeg -loglevel fatal -y -i ${TMPDIR}/begincut.${EXT} -af areverse  ${TMPDIR}/reversed.${EXT} || return 1
    #ffmpeg -loglevel fatal -y -i ${TMPDIR}/reversed.${EXT} -af silenceremove=1:0:-96dB  ${TMPDIR}/revcut.${EXT} || return 1
    #ffmpeg -loglevel fatal -y -i ${TMPDIR}/revcut.${EXT}   -af areverse  ${TMPDIR}/finished.${EXT} || return 1
    #mv "${TMPDIR}/finished.${EXT}" "OUTPUT/`filename ${SOUND}`"
}

head -n5 ${TMPDIR}/sounds

while read SOUND ; do
  cutfile

done<${TMPDIR}/sounds
rm -r ${TMPDIR}

remsilence

Code:

./0000.wav
./0001.wav
./0002.wav
./0003.wav
./0004.wav
./0000.wav
/0001.wav
./0002.wav
/0003.wav
./0004.wav
/0005.wav
./0006.wav
/0007.wav
./0008.wav
/0009.wav
./0010.wav

Why it is deleting the preceding period?????

ondoho · 09-09-2018, 03:46 AM

i found that read can read all sorts of things, very unexpected.
i also found that ffmpeg is not so precise with using stdout and stderr etc.
shortly, all sorts of weird stuff can happen.

also, shouldn't you call cutfile with the file it is going to cut as "$1"?

and why don't you just loop over the files directly instead of saving them to a file first?

so instead of:

Code:

while read SOUND ; do
  cutfile

done<${TMPDIR}/sounds

why don't you just use:

Code:

for SOUND in ????.wav; do
  cutfile "$SOUND"
done

???

frater · 09-09-2018, 04:02 AM

Hi...
Thanks for thinking with me...

The solution is somewhere else, although I haven't yet figured why.
BTW.... without reading your answer I tried cutfile "${SOUND}", but that behaved the same
The problem is that bash is treating it as numbers.
But I still don't understand why it's not doing that immediately wrong.

I changed

Code:

find -maxdepth 1 -type f -name \*.${EXT}

into

Code:

find `pwd`-maxdepth 1 -type f -name \*.${EXT}

which immediately changed a lot

EDIT

No... it cuts 5 characters away

ondoho · 09-09-2018, 04:09 AM

Quote:

Originally Posted by frater

The problem is that bash is treating it as numbers.

sorry but you're going to have to clarify that.
as it stands, it's wrong - bash treats everything as strings unless you tell it explicitely to use arithmetics.

frater · 09-09-2018, 04:19 AM

Hi.... I already corrected that....
I was just trying to find a reason for this unexpected behaviour....

frater · 09-09-2018, 04:24 AM

Maybe I should just try another tool for this.
Never had this before. It's just as if I don't know anything about bash anymore...

frater · 09-09-2018, 04:33 AM

cat /usr/local/sbin/remsilence

Code:

#!/bin/bash

EXT=wav
TMPDIR=`mktemp -t -d ${0//*\/}.XXXXXXXXXX`
mkdir ${TMPDIR}/LIST
find `pwd` -maxdepth 1 -type f -name \*.${EXT} | grep "${EXT}$" | sort  >${TMPDIR}/LIST/sounds
rm -r OUTPUT 2>/dev/null
mkdir OUTPUT

cutfile()
{
    echo "${1}"
    ffmpeg -loglevel fatal -y -i ${1} -af silenceremove=1:0:-96dB  ${TMPDIR}/begincut.${EXT} || return 1

    ffmpeg -loglevel fatal -y -i ${TMPDIR}/begincut.${EXT} -af areverse  ${TMPDIR}/reversed.${EXT}
    ffmpeg -loglevel fatal -y -i ${TMPDIR}/reversed.${EXT} -af silenceremove=1:0:-96dB  ${TMPDIR}/revcut.${EXT}
    ffmpeg -loglevel fatal -y -i ${TMPDIR}/revcut.${EXT}   -af areverse  ${TMPDIR}/finished.${EXT}
    mv "${TMPDIR}/finished.${EXT}" "OUTPUT/`filename ${1}`"
    rm -f ${TMPDIR}/*.${EXT}
}

head -n5 ${TMPDIR}/LIST/sounds

while read SOUND ; do
  cutfile "${SOUND}"

done<${TMPDIR}/LIST/sounds
rm -r ${TMPDIR}

remsilence

Code:

/WORKDIR/0000.wav
/WORKDIR/0001.wav
/WORKDIR/0002.wav
/WORKDIR/0003.wav
/WORKDIR/0004.wav
/WORKDIR/0000.wav
KDIR/0001.wav
/WORKDIR/0002.wav
KDIR/0003.wav
/WORKDIR/0004.wav
KDIR/0005.wav
/WORKDIR/0006.wav
KDIR/0007.wav
/WORKDIR/0008.wav
KDIR/0009.wav

frater · 09-09-2018, 04:42 AM

Even when I accept that ffmpeg is doing strange things.
Why is it doing this only for the even tries.

lougavulin · 09-09-2018, 05:24 AM

There is two ways to solve this.

Add -nostdin option to your ffmepg commands. From ffmepg's man :

Code:

       -stdin
           Enable interaction on standard input. On by default unless standard input is used as an input. To explicitly disable
           interaction you need to specify "-nostdin".

           Disabling interaction on standard input is useful, for example, if ffmpeg is in the background process group. Roughly
           the same result can be achieved with "ffmpeg ... < /dev/null" but it requires a shell.

Code:

ffmpeg -nostdin -loglevel fatal -y -i ${1} -af silenceremove=1:0:-96dB  ${TMPDIR}/begincut.${EXT} || return 1
[...]

Or replace while loop by for loop :

Code:

for SOUND in $(cat ${TMPDIR}/LIST/sounds); do
  cutfile "${SOUND}"
done

pan64 · 09-09-2018, 06:14 AM

you might want to insert set -xv at the beginning of your script to see what's happening. Probably that will give you some additional info.
Replacing that while cycle to for is not really suggested, using while is the preferred solution.

Code:

find `pwd` -maxdepth 1 -type f -name \*.${EXT} | grep "${EXT}$" | sort  >${TMPDIR}/LIST/sounds

can be probably replaced:

Code:

ls -1 *.${EXT} >${TMPDIR}/LIST/sounds

frater · 09-09-2018, 10:38 AM

Thanks all for thinking with me.
lougavulin gave me the key to the solution by telling me I should use the -nostdin with ffmpeg.
The script then started to behave like I would expect it to.

This is the first time that some binary is messing up my bash-script.
Does this happen more?
I would think it does.

I would really like to hear of some more pitfalls

Some things that I did to find some solutions that didn't work I have reverted in this script

cat /usr/local/sbin/remsilence

Code:

#!/bin/bash

EXT=wav
TMPDIR=`mktemp -t -d ${0//*\/}.XXXXXXXXXX`
find -maxdepth 1 -type f -name \*.${EXT} | grep "${EXT}$" | cut -b3- | sort  >${TMPDIR}/sounds
rm -r OUTPUT 2>/dev/null
mkdir OUTPUT

cutfile()
{
    echo "${SOUND}"
    ffmpeg -nostdin -loglevel fatal -y -i ${SOUND} -af silenceremove=1:0:-96dB  ${TMPDIR}/begincut.${EXT} || return 1

    ffmpeg -nostdin -loglevel fatal -y -i ${TMPDIR}/begincut.${EXT} -af areverse  ${TMPDIR}/reversed.${EXT}
    ffmpeg -nostdin -loglevel fatal -y -i ${TMPDIR}/reversed.${EXT} -af silenceremove=1:0:-96dB  ${TMPDIR}/revcut.${EXT}
    ffmpeg -nostdin -loglevel fatal -y -i ${TMPDIR}/revcut.${EXT}   -af areverse  ${TMPDIR}/finished.${EXT}
    mv "${TMPDIR}/finished.${EXT}" "OUTPUT/${SOUND}"
    rm -f ${TMPDIR}/*.${EXT}
}

head -n5 ${TMPDIR}/sounds

while read SOUND ; do
  cutfile
done<${TMPDIR}/sounds
rm -r ${TMPDIR}

pan64 · 09-09-2018, 10:42 AM

this is not "some binaries", but there were two apps reading the same stdin stream (fed by <${TMPDIR}/LIST/sounds):
1. obviously the read command
2. now it was the ffmpeg tool.

frater · 09-09-2018, 12:04 PM

After a while you start thinking of bash as a programming language and don't think how they achieve the things you actually want them to do.
At least I did....

frater · 09-10-2018, 07:23 AM

I now have a way to trim the silence of all wavs in the current folder.
But I'm not satisfied yet.
Each ffmpeg action results in a loss. I don't want this.
I'm already working with uncompressed WAVs and I see no reason why I should except any loss.

That's why I tried to use a different approach.
At first I tried "silencedetect", but that gave me unreliable results.

Then I thought of a bit "clumsy" way by using the results of the "silenceremove" process and use it to just cut the begin and end of the original WAV-file.
That worked.
But I still wasn't satisfied. According to the ffmpeg manual there is still some loss unless I use "-c:a copy".
So I added that to the command, but then I noticed it sometimes cut into the good part.

It seems ffmpeg uses frames when it is used with the "-c:a copy" command.
I then thought of shifting the parameters with 15ms so it has a better chance to cut at the right place.

Maybe I should ffprobe and extract "pkt_duration_time" and divide that by 2 and use that as offset?

Any comments are welcome.

cat /usr/local/sbin/remsilence

Code:

cat /usr/local/sbin/remsilence
#!/bin/bash

EXT=wav
THRESHOLD=70
TMPDIR=`mktemp -t -d ${0//*\/}.XXXXXXXXXX`
find -maxdepth 1 -type f -name \*.${EXT} | grep "${EXT}$" | cut -b3- | sort  >${TMPDIR}/sounds
rm -r OUTPUT 2>/dev/null
rm -r LOSSLESS 2>/dev/null
rm -r LESSLOSS 2>/dev/null
mkdir OUTPUT
mkdir LESSLOSS
mkdir LOSSLESS

cutsox()
{
    echo "${SOUND}"
    sox ${SOUND} ${TMPDIR}/begincut.${EXT} silence 1 0 -${THRESHOLD}d || return 1
    sox ${TMPDIR}/begincut.${EXT} ${TMPDIR}/reversed.${EXT} reverse
    sox ${TMPDIR}/reversed.${EXT} ${TMPDIR}/revcut.${EXT} silence 1 0 -${THRESHOLD}d
    sox ${TMPDIR}/revcut.${EXT}   ${TMPDIR}/finished.${EXT} reverse
}

cutffmpeg()
{
    echo "${SOUND}"
    ffmpeg -nostdin -loglevel fatal -y -i ${SOUND} -af silenceremove=1:0:-${THRESHOLD}dB ${TMPDIR}/begincut.${EXT} || return 1

    ffmpeg -nostdin -loglevel fatal -y -i ${TMPDIR}/begincut.${EXT} -af areverse ${TMPDIR}/reversed.${EXT}
    ffmpeg -nostdin -loglevel fatal -y -i ${TMPDIR}/reversed.${EXT} -af silenceremove=1:0:-${THRESHOLD}dB ${TMPDIR}/revcut.${EXT}
    ffmpeg -nostdin -loglevel fatal -y -i ${TMPDIR}/revcut.${EXT}   -af areverse ${TMPDIR}/finished.${EXT}
}

while read SOUND ; do
  cutffmpeg
  #cutsox

  cp "${TMPDIR}/finished.${EXT}" "OUTPUT/${SOUND}"

  ORGLENG=`ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 ${SOUND}`
  BEGINCT=`ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 ${TMPDIR}/begincut.${EXT}`
  FINLENG=`ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 ${TMPDIR}/finished.${EXT}`
  STARTSILENCE=`echo "${ORGLENG} ${BEGINCT}" | awk '{printf $1 - $2}'`
  ENDSILENCE=`echo "${BEGINCT} ${FINLENG}" | awk '{printf $1 - $2}'`

  echo "ORGLENG=${ORGLENG}"
  echo "BEGINCT=${BEGINCT}"
  echo "FINLENG=${FINLENG}"
  echo "STARTSILENCE=${STARTSILENCE}"
  echo "ENDSILENCE=${ENDSILENCE}"

  echo "${STARTSILENCE}" | grep -q 'e' && STARTSILENCE="0.0"

  ffmpeg -nostdin -loglevel fatal -y -i ${SOUND} -ss ${STARTSILENCE} -t ${FINLENG} LESSLOSS/${SOUND}

  # extract framelength in seconds 
  OFFSET=`ffprobe ${SOUND} -show_frames 2>/dev/null | grep pkt_duration_time | head -n1 | awk -F= '{print $2}'`
  # distract frame rate to make sure it is cut in the silence
  
  STARTSILENCE=`echo "${STARTSILENCE} ${OFFSET}" | awk '{printf $1 - $2}'`
  # if it is negative, make it 0
  echo "${STARTSILENCE}" | grep -q '-' && STARTSILENCE="0.0"
  # add the distracted size to the length
  FINLENG=`echo "${FINLENG} ${OFFSET}" | awk '{printf $1 + $2}'`

  echo "${SOUND} = ${ORGLENG} seconds with ${STARTSILENCE}s silence at start and ${ENDSILENCE}s at end. Result is ${FINLENG}s"
  echo "Offset = ${OFFSET}"

  ffmpeg -nostdin -loglevel fatal -y -i ${SOUND} -ss ${STARTSILENCE} -t ${FINLENG} -c:a copy LOSSLESS/${SOUND}


  rm -f ${TMPDIR}/*.${EXT}

done<${TMPDIR}/sounds
rm -r ${TMPDIR}

frater · 09-10-2018, 09:15 AM

In the end I just went with the "LESSLOSS" option.
It gives predictable results regarding the places it cuts the file and it's "one-pass" with probably no audible difference.
I'm using it for a set of WAVs that are generated by a TTS-engine and used for OpenTX (a radio for Remote Controlled Airplanes).

Having a predictable silence is more important and I don't need to check all files to make sure that none are "cut in the flesh".