bash scripting and arrays
 

While I'm not new to Linux, I'm lost at scripting. Feel free to move this to a more appropriate place. I learned BASIC many years ago, so I think I can lay it out in a logical fashion, but most definitely need instruction here, preferably with explanations.

I have a lot of video files, but the output volumes are all over the place. I'll have to crank the volume on one video, and the next one will be deafening so I have been attempting to write a bash script to "level out" the files using ffmpeg's af volumedetect. (No, I don't mean normalize)

Using ffmpeg -i *.mkv -af volumedetect -f null /dev/null on one particular file yielded the following output:

frame=120550 fps=2457 q=0.0 Lsize=N/A time=01:23:48.09 bitrate=N/A
video:11302kB audio:2828304kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
[Parsed_volumedetect_0 @ 0x2532840] n_samples: 1448091648
[Parsed_volumedetect_0 @ 0x2532840] mean_volume: -25.2 dB
[Parsed_volumedetect_0 @ 0x2532840] max_volume: -1.3 dB
[Parsed_volumedetect_0 @ 0x2532840] histogram_1db: 138
[Parsed_volumedetect_0 @ 0x2532840] histogram_2db: 4022
[Parsed_volumedetect_0 @ 0x2532840] histogram_3db: 33737
[Parsed_volumedetect_0 @ 0x2532840] histogram_4db: 135497
[Parsed_volumedetect_0 @ 0x2532840] histogram_5db: 385067
[Parsed_volumedetect_0 @ 0x2532840] histogram_6db: 794773
[Parsed_volumedetect_0 @ 0x2532840] histogram_7db: 1318793

What I have been doing is running another instance of ffmpeg to encode it to mkv using, along with other parameters, taking the absolute value (a positive intger), in this case -vf "volume=1.3" (with quotes,) to bring the max_volume in line with that of the other files, but am at a loss how to automate this.

I have:

!# /bin/bash

mkdir ./encoded
for IFS in *.mkv; do
max_volume=`ffmpeg -i "${IFS}" -af volumedetect -f null /dev/null

As you can see I have been attempting to create an array thinking it was the way to go, if not, please advise me.

Now, I can understand "-f null" means a virtual file instead of an actual one, but from there....

I have tried to put the output through /dev/null but could not figure out how to obtain the max_volume value, change the negative to a positive or vice versa (*-1?) and then input that value into a second ffmpeg instance where the volume will be adjusted, video scaled, etc. I have the encoding parameters I want, the problem is inputting the value for "volume = "

Would it be better to output to a text file, or maybe stdout and then pick up the max_volume from stdin, and if so, how would I accomplish this?

I would appreciate some help resolving this. Thanks.

No need to go to a file, you should be able to pipe directly to something like sed. Using the above list, this should get you what you wantYou need to supply "some ffmpeg command" and the vertical bar symbol (pipe) pipes all the stdout into the sed. It is specific so as to ensure just the right data is picked up. Run it and see what it produces for you. You can assign that to a variable and use that in the following command.

excuse me if I seem a little dense, but I have no experience using sed, am I'm not sure exactly how to assign a variable to it. Here is what I have so far:

#! /bin/bash
mkdir ./encode
for IFS in *.mkv; do
ffmpeg -i "${IFS}" -af volumedetect | sed -nr '/max_volume:/ s/.*[[:space:]]-?([[:digit:]\.]+)[[:space:]]+dB$/-vf "volume=\1"/p'

This is where I get lost.

ffmpeg -i "$"IFS" bunch of encode stuff -af "volume=variable" ./encode/"$IFS";
done

Am I on the right track? I have the encode parameters I want, its a matter of inserting the correct volume setting taken fron max_volume above. I had one file, believe it or not, with a max_volume of -13.7 db. You can see why I would like to level off the volumes accross the different files.

Try it like this

Still don't seem to have it quite right. I ran:

#! /bin/bash
mkdir ./encode
for IFS in *.mkv; do
ffmpeg -i "${IFS}" -af volumedetect | sed -nr '/max_volume:/ s/.*[[:space:]]-?([[:digit:]\.]+)[[:space:]]+dB$/-vf "volume=\1"/p'
echo $vol;
done


And got:terry@TP500LA:~/Videos/test$ sh ./setvolume.sh
mkdir: cannot create directory ./encode: File exists
ffmpeg version 2.5.7-0ubuntu0.15.04.1 Copyright (c) 2000-2015 the FFmpeg developers
built with gcc 4.9.2 (Ubuntu 4.9.2-10ubuntu13)
configuration: --prefix=/usr --extra-version=0ubuntu0.15.04.1 --build-suffix=-ffmpeg --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --shlibdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --enable-shared --disable-stripping --enable-avresample --enable-avisynth --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libschroedinger --enable-libshine --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libwavpack --enable-libwebp --enable-libxvid --enable-opengl --enable-x11grab --enable-libdc1394 --enable-libiec61883 --enable-libzvbi --enable-libzmq --enable-frei0r --enable-libvpx --enable-libx264 --enable-libsoxr --enable-gnutls --enable-openal --enable-libopencv --enable-librtmp --enable-libx265
libavutil 54. 15.100 / 54. 15.100
libavcodec 56. 13.100 / 56. 13.100
libavformat 56. 15.102 / 56. 15.102
libavdevice 56. 3.100 / 56. 3.100
libavfilter 5. 2.103 / 5. 2.103
libavresample 2. 1. 0 / 2. 1. 0
libswscale 3. 1.101 / 3. 1.101
libswresample 1. 1.100 / 1. 1.100
libpostproc 53. 3.100 / 53. 3.100
Trailing options were found on the commandline.
Input #0, matroska,webm, from 'xxxx.mkv':
Metadata:
COMPATIBLE_BRANDS: isomiso2avc1mp41
MAJOR_BRAND : isom
MINOR_VERSION : 512
ENCODER : Lavf56.1.0
Duration: 01:33:46.19, start: 0.000000, bitrate: 1360 kb/s
Stream #0:0(und): Video: h264 (High), yuv420p, 704x384 [SAR 32:33 DAR 16:9], SAR 31:32 DAR 341:192, 30.30 fps, 29.97 tbr, 1k tbn, 59.94 tbc (default)
Stream #0:1(und): Audio: aac (LC), 48000 Hz, stereo, fltp (default)
At least one output file must be specified

Should I put in the second instance now?

Sorry, ran:

#! /bin/bash
mkdir ./encode
for IFS in *.mkv; do
vfvol=$(ffmpeg -i "${IFS}" -af volumedetect | sed -nr '/max_volume:/ s/.*[[:space:]]-?([[:digit:]\.]+)[[:space:]]+dB$/-vf "volume=\1"/p')
echo ${vfvol}
done

All together, with the second ffmpeg instance included I just ran:

#! /bin/bash
mkdir ./encode
for IFS in *.mkv; do
vfvol=$(ffmpeg -i "${IFS}" -af volumedetect | sed -nr '/max_volume:/ s/.*[[:space:]]-?([[:digit:]\.]+)[[:space:]]+dB$/-vf "volume=\1"/p')
ffmpeg -i ""${IFS} -f matroska -maxrate 1M -b:v 1500k -r 29.97 -vcodec libxvid -filter:v scale=704:384 -aspect 16:9 -qmin 3 -qmax 5 -bufsize 4096 -mbd 2 -bf 2 -trellis 1 -flags +aic -cmp 2 -subcmp 2 -g 300 -acodec ac3 -strict -2 -b:a 128k -ac 6 -af "volume="${vfvol}"" ./encode/"${IFS}";
done


and the result was:

terry@TP500LA:~/Videos/test$ sh ./setvolume.sh
mkdir: cannot create directory ./encode: File exists
ffmpeg version 2.5.7-0ubuntu0.15.04.1 Copyright (c) 2000-2015 the FFmpeg developers
built with gcc 4.9.2 (Ubuntu 4.9.2-10ubuntu13)
configuration: --prefix=/usr --extra-version=0ubuntu0.15.04.1 --build-suffix=-ffmpeg --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --shlibdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --enable-shared --disable-stripping --enable-avresample --enable-avisynth --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libschroedinger --enable-libshine --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libwavpack --enable-libwebp --enable-libxvid --enable-opengl --enable-x11grab --enable-libdc1394 --enable-libiec61883 --enable-libzvbi --enable-libzmq --enable-frei0r --enable-libvpx --enable-libx264 --enable-libsoxr --enable-gnutls --enable-openal --enable-libopencv --enable-librtmp --enable-libx265
libavutil 54. 15.100 / 54. 15.100
libavcodec 56. 13.100 / 56. 13.100
libavformat 56. 15.102 / 56. 15.102
libavdevice 56. 3.100 / 56. 3.100
libavfilter 5. 2.103 / 5. 2.103
libavresample 2. 1. 0 / 2. 1. 0
libswscale 3. 1.101 / 3. 1.101
libswresample 1. 1.100 / 1. 1.100
libpostproc 53. 3.100 / 53. 3.100
Trailing options were found on the commandline.
Input #0, matroska,webm, from 'xxxxx.mkv':
Metadata:
COMPATIBLE_BRANDS: isomiso2avc1mp41
MAJOR_BRAND : isom
MINOR_VERSION : 512
ENCODER : Lavf56.1.0
Duration: 01:33:46.19, start: 0.000000, bitrate: 1360 kb/s
Stream #0:0(und): Video: h264 (High), yuv420p, 704x384 [SAR 32:33 DAR 16:9], SAR 31:32 DAR 341:192, 30.30 fps, 29.97 tbr, 1k tbn, 59.94 tbc (default)
Stream #0:1(und): Audio: aac (LC), 48000 Hz, stereo, fltp (default)
At least one output file must be specified
ffmpeg version 2.5.7-0ubuntu0.15.04.1 Copyright (c) 2000-2015 the FFmpeg developers
built with gcc 4.9.2 (Ubuntu 4.9.2-10ubuntu13)
configuration: --prefix=/usr --extra-version=0ubuntu0.15.04.1 --build-suffix=-ffmpeg --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --shlibdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --enable-shared --disable-stripping --enable-avresample --enable-avisynth --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libschroedinger --enable-libshine --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libwavpack --enable-libwebp --enable-libxvid --enable-opengl --enable-x11grab --enable-libdc1394 --enable-libiec61883 --enable-libzvbi --enable-libzmq --enable-frei0r --enable-libvpx --enable-libx264 --enable-libsoxr --enable-gnutls --enable-openal --enable-libopencv --enable-librtmp --enable-libx265
libavutil 54. 15.100 / 54. 15.100
libavcodec 56. 13.100 / 56. 13.100
libavformat 56. 15.102 / 56. 15.102
libavdevice 56. 3.100 / 56. 3.100
libavfilter 5. 2.103 / 5. 2.103
libavresample 2. 1. 0 / 2. 1. 0
libswscale 3. 1.101 / 3. 1.101
libswresample 1. 1.100 / 1. 1.100
libpostproc 53. 3.100 / 53. 3.100
: No such file or directory
terry@TP500LA:~/Videos/test$


Still willing to give it a go?

I just noticed a nasty syntax error that was entirely my fault. I used ""${IFS} when it should have been "${IFS}"

I have corrected my mistake and the script is running on two test files right now and will give you the results when it is finished.

I would like to thank you for all your valuable help. Of course should this be successful I will mark it as solved.

I would just like to point out that using IFS as the variable could be problematic. IFS is a builtin variable used by bash to determine word splitting and the like. I am not 100 sure it is causing problems here but I would change it nonetheless.

I saw you wish to mark this as solved, but you do seem to have some errors appearing in your output, such as ': No such file or directory'.


Did you check the output of the first script with the echo?
Because your output does not show the relevant output anywhere. To add to this, your second ffmpeg call would be incorrect:
The data stored in the variable vfvol would be :- -vf "volume=\1", where '\1' is replaced with what was gleaned from the output. So in your code once the variable was expanded it would look like:
As you can see, this is not what you were expecting.

I am afraid that my tiny, underdeveloped brain has reached its stress limit, and too much stress leaves me horizontal for hours, sometimes days. (I am unable to work due to major depression.)

Here is what I have:

#! /bin/bash
mkdir ./encode
for IFS in *.mkv; do
vfvol=$(ffmpeg -i "${IFS}" -af volumedetect | sed -nr '/max_volume:/ s/.*[[:space:]]-?([[:digit:]\.]+)[[:space:]]+dB$/-vf "volume=\1"/p')
ffmpeg -i "${IFS}" -f matroska -maxrate 1M -b:v 1500k -r 29.97 -vcodec libxvid -filter:v scale=704:384 -aspect 16:9 -qmin 3 -qmax 5 -bufsize 4096 -mbd 2 -bf 2 -trellis 1 -flags +aic -cmp 2 -subcmp 2 -g 300 -acodec ac3 -strict -2 -b:a 128k -ac 6 -af "volume="${vfvol}"" ./encode/"${IFS}";
done

and the resulting file has the same volume level as the original.

I must ask that if you wish to help that you give me a complete solution rather than just a section or hint. I may have to set this aside for a bit while my brain resets itself. Thank you for your help and patience. I'll try again in a few hours, maybe a day or two. I won't give up.

Grail: I was under the impression that IFS should be used whenever there was the chance of a <space> in the variable, but then I make a lot of mistakes.

Also, I find that I am experiencing a little too much stress trying to apply hints and suggestions, which can trigger yet another major depressive episode (and explains why I am so bad at scripting. Would it be possible to give me a more complete solution. Take your suggestion of using

[code]echo ${vfvol}[/code

as example. I tried adding it after the fitst part and commenting out the second, but it said something about needing to output to a file. My head is so muddled right now that I can't seem to get it right. Do I need quotes around the variable, or something else?
Would you be able to do a little copy and paste to give me the entire sequence to get the value to echo on the screen? This is all so frustrating for me. I used to have a working brain before the Depression set it, and now I have to struggle.

Thanks

syg00

Still have not succeeded, but thanks for the help so far.

syg00 I ran what you suggested and not get any output. Here is what it gave me:

terry@TP500LA:~/Videos/media$ sh ./setvolume2.sh
./setvolume2.sh: 1: ./setvolume2.sh: #!: not found
mkdir: cannot create directory ./encode: File exists
ffmpeg version 2.5.7-0ubuntu0.15.04.1 Copyright (c) 2000-2015 the FFmpeg developers
built with gcc 4.9.2 (Ubuntu 4.9.2-10ubuntu13)
configuration: --prefix=/usr --extra-version=0ubuntu0.15.04.1 --build-suffix=-ffmpeg --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --shlibdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --enable-shared --disable-stripping --enable-avresample --enable-avisynth --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libschroedinger --enable-libshine --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libwavpack --enable-libwebp --enable-libxvid --enable-opengl --enable-x11grab --enable-libdc1394 --enable-libiec61883 --enable-libzvbi --enable-libzmq --enable-frei0r --enable-libvpx --enable-libx264 --enable-libsoxr --enable-gnutls --enable-openal --enable-libopencv --enable-librtmp --enable-libx265
libavutil 54. 15.100 / 54. 15.100
libavcodec 56. 13.100 / 56. 13.100
libavformat 56. 15.102 / 56. 15.102
libavdevice 56. 3.100 / 56. 3.100
libavfilter 5. 2.103 / 5. 2.103
libavresample 2. 1. 0 / 2. 1. 0
libswscale 3. 1.101 / 3. 1.101
libswresample 1. 1.100 / 1. 1.100
libpostproc 53. 3.100 / 53. 3.100
Trailing options were found on the commandline.
Input #0, matroska,webm, from 'xxxxx.mkv':
Metadata:
COMPATIBLE_BRANDS: isomiso2avc1mp41
MAJOR_BRAND : isom
MINOR_VERSION : 512
ENCODER : Lavf56.1.0
Duration: 02:00:01.23, start: 0.000000, bitrate: 344 kb/s
Stream #0:0(und): Video: h264 (High), yuv420p, 576x268 [SAR 335:336 DAR 15:7], SAR 254:255 DAR 12192:5695, 30.30 fps, 29.97 tbr, 1k tbn, 59.94 tbc (default)
Stream #0:1(und): Audio: aac (LC), 44100 Hz, stereo, fltp (default)
At least one output file must be specified

terry@TP500LA:~/Videos/media$


I waited for a while for the fog to lift from my brain, and have been doing some RTFM and trying many experiments that all failed in a most glorious fashion. I've even tried, armed with a sed tutorial to parse your sed command, but I am a mere Padewan and am having trouble interpretting it. I understand that it is going through space by space, but if there is a mistake there, as signified by the trailing options warning, I am unable to find it.

The regex I gave is for the specific data you listed.
Do not try to debug in a loop where you don't know the data being processed. Pick a file or two, and run your command against that. If the output includes a line like the "max_volume" in your original post, my regex should pull out the numbers you're looking for. When all that works on a select file or two, assign it to a variable, then echo it. Do it one command at a time by hand so you can see what is happening.
If your commands have errors, or not all ffmpeg commands produce that output, you will not get good results. Only you can resolve all those issues.

As for the regex, it is complex (maybe more so than needed), but was designed to make sure only the output line you offered was processed. Regex is notoriously difficult to get right. Here is a quick breakdown:

I've been parsing your sed in an attempt to learn about it and have slowly come to see the logic behind it and why you made it the way you did. I can match it up against the max_volume line and can appreciate your skill with scripting.

Some how I have mangled the second ffmpeg command so that it is not utilizing the generated $vfvol. Nor can I get echo to display the value on screen. Care to parse that for me? It is all complete, excepting, of course, the file names provided by $IFS.


#! /bin/bash
mkdir ./encode
for IFS in *.mkv; do
vfvol=$(ffmpeg -i "${IFS}" -af volumedetect | sed -nr '/max_volume:/ s/.*[[:space:]]-?([[:digit:]\.]+)[[:space:]]+dB$/-vf "volume=\1"/p') echo ${vfvol}
#ffmpeg -i "${IFS}" -f matroska -maxrate 1M -b:v 1500k -r 29.97 -vcodec libxvid -filter:v scale=704:384 -aspect 16:9 -qmin 3 -qmax 5 -bufsize 4096 -mbd 2 -bf 2 -trellis 1 -flags +aic -cmp 2 -subcmp 2 -g 300 -acodec ac3 -strict -2 -b:a 128k -ac 6 -af "volume="${vfvol}"" ./encode/"${IFS}";
done


I just had a thought. Is it at all possible that the audio is being encoded before it tries to apply the filter? What if I moved it to a spot before the codec. I know position doesn't usually matter, but there are times when it does.

Any thoughts?