a very, very strange problem I am unable to solve

marozsas · 11-17-2016, 11:40 AM

Hi there, nice to be here again after a long time.

I am writing a script to take snapshots of a directory full of movies when I found this strange behavior: looks like ffmpeg corrupts the script execution in some way.

The script bellow is not the original code. It is a reduced code that still have the problem.

Run it on a large folder of sub folders and files. I choose to run it on a /usr/share.

You should run it twice. On the first time, comment the ffmeg command; on the second, uncomment it. Notice how the filename is displayed differently between the runs. Obviously, in the second run the file names are corrupted at the very begin; for me, the "./" is missing. On a different set of files, more characters vanish at begin of string. At the end of set it miss even more characters (this is why I suggested to run in a large folder of files)

The input and output file for ffmpeg does not matter. Here, they are 2 fixed files. The input file is a movie (anyone you may have you should copy to /tmp as input.mp4). The output, a PNG snapshot, is always the same, overwritten each pass.
They are both on /tmp folder, just to show you the location is irrelevant and it has nothing to do with the folder where you run the script.

A run with ffmpeg line commented.

Code:

miguel@38Y46W1-x:~$ cd /usr/share
miguel@38Y46W1-x:/usr/share$ ~/bin/media_test.sh 
f: ./a11y-profile-manager/profiles/blindness/profile.gsettings
f: ./a11y-profile-manager/profiles/blindness/profile.manifest
f: ./a11y-profile-manager/profiles/braille/profile.gsettings
f: ./a11y-profile-manager/profiles/braille/profile.manifest
... long lines removed ....
f: ./zsh/vendor-completions/_timedatectl
f: ./zsh/vendor-completions/_udevadm
f: ./zsh/vendor-completions/_vlc
miguel@38Y46W1-x:/usr/share$

Now, a run with ffmpef uncommented. Note the corrupted filename:

Code:

miguel@38Y46W1-x:/usr/share$ media_test.sh 
f: ./a11y-profile-manager/profiles/blindness/profile.gsettings
f: /a11y-profile-manager/profiles/blindness/profile.manifest
f: /a11y-profile-manager/profiles/braille/profile.gsettings
f: /a11y-profile-manager/profiles/braille/profile.manifest
f: /a11y-profile-manager/profiles/high-contrast/profile.gsettings
f: /a11y-profile-manager/profiles/high-contrast/profile.manifest

... long lines removed ....

f: dor-completions/_timedate
f: dor-completions/_udev
f: dor-completions/_

WTF ????

I appreciate any ideas...and alternatives.
best regards,

The code:
copy any movie/trailer to /tmp/input.mp4 and uncomment ffmpeg to see the difference.

Code:

#!/bin/bash

while read f; do
  echo "f: ${f}"
  #ffmpeg -v 8 -y -ss 12 -i "/tmp/input.mp4" -vframes 1 -q:v 2 "/tmp/out.png"
done < <(find . -type f|sort)

I am running Ubuntu 16.04, up to date.

I already tried to do:

Code:

#!/bin/bash

find . -type f |sort | while read f; do
  echo "f: ${f}"
  #ffmpeg -v 8 -y -ss 12 -i "/tmp/input.mp4" -vframes 1 -q:v 2 "/tmp/out.png"
done

and changed the IFS to newline to deal with filenames with spaces, but this didn't changed the results.
I have tried to call ffmpeg as a bash function and even as a separate script. same results.

szboardstretcher · 11-17-2016, 11:57 AM

How about:

Code:

#!/bin/bash

for line in $(find . -type f); do 
  echo "f: $line"
  ffmpeg -v 8 -y -ss 12 -i "/tmp/input.mp4" -vframes 1 -q:v 2 "/tmp/out.png"
done

marozsas · 11-17-2016, 01:32 PM

Hi szboardstretcher,

Sort of.... It worked on /usr/share folder, but not on a folder with tons of mp3 files. Look:

Code:

miguel@38Y46W1-x:/media/miguel/MUSICA-J3S$ IFS="
"
miguel@38Y46W1-x:/media/miguel/MUSICA-J3S$ for line in $(find . -type f); do    echo "f: $line";   ffmpeg -v 8 -y -ss 12 -i "/tmp/input.mp4" -vframes 1 -q:v 2 "/tmp/out.png"; done
f: ./1_-_The_barber_of_Seville.mp3
f: /2_-_Willian_Tell.mp3
f: /3_-_La_gazza_ladra,_overture.mp3
f: /4_-_L_italiana_in_Algeri.mp3
f: /4-tissimo Guitar Quartet plays Tico Tico no Fubá.mp3
f: /5_-_Semiramide.mp3
f: /6_-_La_danza.mp3
f: /7_-_La_cenerentola.mp3
...

And it goes that until the end. It is missing the initial dot. At least the corruption is limited to the first char. Of course the rest of script fails because a relative path becomes an absolute one....
Anyway, why the code using "ffmpeg" has that behaviour ? Were you able to reproduce the same result ?
If yes, what do you think is the explanation ? A bug in BASH or in ffmpeg ?

cheers,

PS: And again, if I remove the ffmpeg command, there are no corruption on file names at all !

szboardstretcher · 11-17-2016, 01:38 PM

What are you doing with that IFS declaration? In a brand new session, without any further changes, does the script provided still give that output in the mp3 directory?

marozsas · 11-17-2016, 02:41 PM

To test in the music folder, I have to set IFS to newline to deal with spaces on file names.

unSpawn · 11-17-2016, 02:59 PM

Quote:

Originally Posted by marozsas

Hi there, nice to be here again after a long time.

Indeed! Welcome back.

"for" loops aren't what you want, "while" loops are. This should deal with spaces OK:

Code:

find $PWD -type f -iname \*.mp4| while read ITEM; do ffmpeg -v 8 -y -ss 12 -i "${ITEM}" -vframes 1 -q:v 2 "/tmp/$(basename "${ITEM}").png"; done

if unsure just use "echo" and escape quotes to see:

Code:

find $PWD -type f -iname \*.mp4| while read ITEM; do echo "ffmpeg -v 8 -y -ss 12 -i \"${ITEM}\" -vframes 1 -q:v 2 \"/tmp/$(basename "${ITEM}").png\""; done

szboardstretcher · 11-17-2016, 03:00 PM

Ah. Very well. Add some input/output redirection at the end of the ffmpeg line like this:

Code:

ffmpeg -v 8 -y -ss 12 -i "/tmp/input.mp4" -vframes 1 -q:v 2 "/tmp/out.png" < /dev/null > /dev/null 2>&1

szboardstretcher · 11-17-2016, 03:09 PM

Quote:

Originally Posted by unSpawn

Indeed! Welcome back.

"for" loops aren't what you want, "while" loops are. This should deal with spaces OK

Explanation of this difference in bash between for and while loops and how they handle whitespace:

http://unix.stackexchange.com/questi...h-for-vs-while

But to re-iterate what is being discussed here. He is looking for an explanation of why the ffmpeg line in his script will mess with the directory listing.

IE, the script without ffmpeg outputs:

Code:

f: ./1_-_The_barber_of_Seville.mp3
f: ./2_-_Willian_Tell.mp3

Whereas with the ffmpeg line outputs:

Code:

f: ./1_-_The_barber_of_Seville.mp3
f: /2_-_Willian_Tell.mp3

I think some input/output redirection, as i mentioned above, might be the answer - because the good line with the ./ in front happens before ffmpeg is ever run. After the first iteration, ie after ffmpeg is run, the output does not have the . in front.

marozsas · 11-18-2016, 03:04 PM

Quote:

Originally Posted by unSpawn

Indeed! Welcome back.

"for" loops aren't what you want, "while" loops are. This should deal with spaces OK:

Code:

find $PWD -type f -iname \*.mp4| while read ITEM; do ffmpeg -v 8 -y -ss 12 -i "${ITEM}" -vframes 1 -q:v 2 "/tmp/$(basename "${ITEM}").png"; done

if unsure just use "echo" and escape quotes to see:

Code:

find $PWD -type f -iname \*.mp4| while read ITEM; do echo "ffmpeg -v 8 -y -ss 12 -i \"${ITEM}\" -vframes 1 -q:v 2 \"/tmp/$(basename "${ITEM}").png\""; done

Hi unSpawn, nice to see you here after so long !

No. Same behavior: It works for the first file and fail for the next. The initial dot is missing.

Code:

miguel@38Y46W1-x:~/Música$ find $PWD -type f | while read ITEM; do echo "file: ${ITEM}"; ffmpeg -v -8 -y -ss 12 -i "/tmp/input.mp4" -vframes 1 -q:v 2 "/tmp/$(basename "${ITEM}").png"  ; done
file: /home/miguel/Música/Lovefool - Vintage Jazz Cardigans Cover ft. Haley Reinhart.mp3
file: home/miguel/Música/Hair- Aquarius.mp3
file: home/miguel/Música/Focus - Ray Charles Style Ariana Grande Cover ft. LaVance Colley.mp3
file: home/miguel/Música/Maps - Vintage 1970s Soul Maroon 5 Cover ft. Morgan James.mp3
...

Please, note I used a fixed input.mp4 file to test over a folder with mp3 files.
And I just notice this happens even on folders with just a handful of files.
So, the number of files and sub-directories is not relevant anyway....

Code:

miguel@38Y46W1-x:/usr/local/include/ntfs-3g$ find $PWD -type f | while read ITEM; do echo "file: ${ITEM}"; ffmpeg -v -8 -y -ss 12 -i "/tmp/input.mp4" -vframes 1 -q:v 2 "/tmp/$(basename "${ITEM}").png"  ; done
file: /usr/local/include/ntfs-3g/ea.h
file: usr/local/include/ntfs-3g/mst.h
file: usr/local/include/ntfs-3g/volume.h
file: usr/local/include/ntfs-3g/compat.h
...

And szboardstretcher was right ! The redirection of input, output and stderr solves the problem ! Thanks man ! Amazing !

Code:

miguel@38Y46W1-x:/usr/local/include/ntfs-3g$ find $PWD -type f | while read ITEM; do echo "file: ${ITEM}"; ffmpeg -v -8 -y -ss 12 -i "/tmp/input.mp4" -vframes 1 -q:v 2 "/tmp/$(basename "${ITEM}").png" < /dev/null > /dev/null 2>&1 ; done
file: /usr/local/include/ntfs-3g/ea.h
file: /usr/local/include/ntfs-3g/mst.h
file: /usr/local/include/ntfs-3g/volume.h
file: /usr/local/include/ntfs-3g/compat.h
file: /usr/local/include/ntfs-3g/dir.h
file: /usr/local/include/ntfs-3g/unistr.h
...

Nevertheless, the mystery continues.
Why the output of ffmpeg corrupts the list of input files ????

Anyway, I can continue with my script. Thanks for the several suggestions and the solution szboardstretcher!

Just curious, this is just me, or you guys manage to get the same results ?
Who is to blame ? Bash or ffmpeg ?

MadeInGermany · 11-19-2016, 07:34 AM

A while loop that reads from stdin can be harmed by programs that also read from stdin. ssh is known to cause this problem.
A fix is

Code:

</dev/null ssh ...

Or use the ssh option

Code:

ssh -n ...

Maybe the ffmpeg has a similar option that prevents it from reading stdin?
Or maybe you can feed its input file from stdin?

Code:

</tmp/input.mp4 ffmpeg ...

The last method is to let the loop read from another descriptor (not stdin that is descriptor 1)

Code:

while read f <&3; do
  echo "f: ${f}"
  ffmpeg ...
done 3< <(find . -type f|sort)