[SOLVED] Bash Script Automation for new Subdirectories.

Valpskott · 11-06-2011, 01:49 PM

I'm producing game-related videos, and I've made 2 scripts to make my work much easier. Lets call them Script A and Script B.

Script A run several copies of Script B (one in each subdirectory). And every time I have new directories, I manually copy Script B into them, and then edit Script A so that the pathways for the new Script Bs are correct.

Sooo, I'd like help in remaking Script A so that it doesn't need editing each time new subdirectories are added.

Right now Script A looks something like this.

Code:

#!/bin/bash
cd Directory-01
./Script-B.sh
cd ..

cd Directory-02
./Script-B.sh
cd ..

cd Directory-03
./Script-B.sh
cd ..

#etc...

It's a simple script that just changes directory, run Script-B.sh in that directory, then goes to the parent directory and up the next subdirectory and runs Script-B.sh in that directory, and so on. Worth mentioning, the actual directories have more unique names, and each time I run the script the amount of subdirectories varies.

So what I need for my new Script A, is that it copies Script-B.sh into all the current subdirectories(I normally do this manually), then executes Script-B.sh in each of these subdirectories.

I'd also like help with Script B, right now, it outputs a standardised name as "Final-Video.mov" in the subdirectory the script is ran from. I'd really appreciate it if I could end the script with something like

Code:

mv Final-Video.mov ../$1.mov
rm -r -f $1

where $1 is the name of the current subdirectory, essentially naming the file after the Subdirectory it's in, moving it out of the subdirectory and then removing the subdirectory once the script is done. But how to get $1 to hold the name of the current child-directory?

corp769 · 11-06-2011, 01:51 PM

As far as all of your subdirectories, you can use a for loop, and loop through all file (*), and then do what you need to do. As far as "script B," what exactly does it do? Maybe you can share it with us, and possible find a completely better way to tackle what you are trying to accomplish.

Cheers,

Josh

Nominal Animal · 11-06-2011, 03:10 PM

Quote:

Originally Posted by Valpskott

But how to get $1 to hold the name of the current child-directory?

pwd will tell you the path to the current working directory. Therefore, basename `pwd` will tell you the name of the current directory, and dirname `pwd` the path to the parent directory (logically the same as .. ).

Quote:

Originally Posted by Valpskott

Script A run several copies of Script B (one in each subdirectory). And every time I have new directories, I manually copy Script B into them, and then edit Script A so that the pathways for the new Script Bs are correct.

Don't do that. Replace them with Script C, that takes one or more (target) directory names as parameters. Then you can do

Code:

find tree-containing-directories -mindepth 1 -depth -type d -print0 | xargs -r0 path-to-Script-C

or, if you have a very powerful machine, and want to do all directories in parallel,

Code:

find tree-containing-directories -mindepth 1 -depth -type d -print0 | bash -c 'while read -d "" DIR ; do path-to-Script-C "$DIR" & ; done ; wait'

No need to do any copying or setup by hand. Note that the -depth parameter above tells find to supply the deepest directories first; this way you'll always get all child directories before their parents, and you can freely nest the directories.

The basic structure of Script C could be something like the following:

Code:

#!/bin/bash

while [ $# -gt 0 ]; do
    dir="$1"
    name=$(basename "$1")
    shift 1

    if (
        cd "$dir" || exit $?
 
        # Current working directory is the one to work on.
        # "$name" is its name (not path).
        # Use 'exit N' to cancel work.

        # Does the final movie exist yet?
        if [ ! -e "final.ogg" ]; then

            # ... Work ...

        fi

        # Verify we now have "final.ogg" in the current directory.
        [ -e "final.ogg" ] || exit 1

        # Make sure nobody changed the directory name from under us.
        [ "$dir" -ef . ] || exit 1

        # Copy the final movie to parent directory, using the
        # name of the current directory, but with .ogg suffix.
        mv -f "final.ogg" "../$name.ogg" || exit $?

        # Remove this child directory.
        cd ..
        rm -rf "$name"
    ) ; then
        echo "$dir: Replaced with '$name.ogg' successfully." >&2
    else
        echo "$dir: Processing aborted due to an error." >&2
        exit 1
    fi
done

with # ... Work ... obviously replaced with the code you use to convert your input files into the final movie.

If you change a directory name while the script is still working, it will abort. You can then safely rerun the command. If the final movie already exists, the script will not regenerate it, just move (and rename) it.

Valpskott · 11-07-2011, 12:09 AM

Quote:

Originally Posted by corp769

As far as all of your subdirectories, you can use a for loop, and loop through all file (*), and then do what you need to do. As far as "script B," what exactly does it do? Maybe you can share it with us, and possible find a completely better way to tackle what you are trying to accomplish.

Cheers,

Josh

Currently, script B looks like this.

Code:

#!/bin/bash

#Merge videos
mencoder -oac copy -ovc copy -o 01-Merged.mov *.avi

#Extract Sound
ffmpeg -i 01-Merged.mov -vn -ar 48000 -ac 2 -ab 192000 -f flac 01-Audio-Unfixed.flac

#Noise Reduction, Compressor, Normalize
sox 01-Audio-Unfixed.flac 02-Audio-Fixed.flac noisered noiseprofil.sox 0.35 gain -l 9 compand 0.0,0.3 -60,-60,-30,-15,-20,-12,-4,-8,-2,-7 -2 norm -0.1

#Join Sound with Video and force video to 1280x720
ffmpeg -i 01-Merged.mov -i 02-Audio-Fixed.flac -map 0:0 -map 1:0 -f mp4 -vcodec mpeg4 -qscale 1 -s 1280x720 -qmin 1 -intra -threads 2 -acodec libmp3lame -ab 160k -ar 44100 -ac 2 02-Merged-Audiofix.mov

#Join Intro with Video
mencoder -oac copy -ovc copy -o 03-compress-me.mov ../intro.mov 02-Merged-Audiofix.mov

#Compress For Youtube
ffmpeg -i 03-compress-me.mov -f mp4 -acodec copy -pix_fmt yuv420p -vcodec libx264 -minrate 0 -b 5000k -b_strategy 1 -subcmp 2 -cmp 2 -coder 1 -flags +loop -flags2 dct8x8 -qmax 51 -subq 7 -qmin 10 -qcomp 0.6 -qdiff 4 -trellis 1 -aspect 16:9 Final-Video.mov

#Remove all files except original (in case of error) and final.
rm 01-Audio-Unfixed.flac
rm 01-Merged.mov
rm 02-Audio-Fixed.flac
rm 02-Merged-Audiofix.mov
rm 03-compress-me.mov
rm noiseprofil.sox
rm Script-B.sh

So, when script B is done... the Subdirectory is still there, and it contains the original videofiles + Final-Video.mov. So, if I have like 10 subdirectories with videos, I will have 10 videos with the name Final-Video.mov (one in each Subdirectory). The name of the Subdirectory is descriptive of what the video inside it, so thats why I'd like to name "Final-Video.mov" after the directory it's in.

Valpskott · 11-07-2011, 12:27 AM

Quote:

Originally Posted by Nominal Animal

pwd will tell you the path to the current working directory. Therefore, basename `pwd` will tell you the name of the current directory, and dirname `pwd` the path to the parent directory (logically the same as .. ).

Don't do that. Replace them with Script C, that takes one or more (target) directory names as parameters. Then you can do

Code:

find tree-containing-directories -mindepth 1 -depth -type d -print0 | xargs -r0 path-to-Script-C

or, if you have a very powerful machine, and want to do all directories in parallel,

Code:

find tree-containing-directories -mindepth 1 -depth -type d -print0 | bash -c 'while read -d "" DIR ; do path-to-Script-C "$DIR" & ; done ; wait'

No need to do any copying or setup by hand. Note that the -depth parameter above tells find to supply the deepest directories first; this way you'll always get all child directories before their parents, and you can freely nest the directories.

The basic structure of Script C could be something like the following:

Code:

#!/bin/bash

while [ $# -gt 0 ]; do
    dir="$1"
    name=$(basename "$1")
    shift 1

    if (
        cd "$dir" || exit $?
 
        # Current working directory is the one to work on.
        # "$name" is its name (not path).
        # Use 'exit N' to cancel work.

        # Does the final movie exist yet?
        if [ ! -e "final.ogg" ]; then

            # ... Work ...

        fi

        # Verify we now have "final.ogg" in the current directory.
        [ -e "final.ogg" ] || exit 1

        # Make sure nobody changed the directory name from under us.
        [ "$dir" -ef . ] || exit 1

        # Copy the final movie to parent directory, using the
        # name of the current directory, but with .ogg suffix.
        mv -f "final.ogg" "../$name.ogg" || exit $?

        # Remove this child directory.
        cd ..
        rm -rf "$name"
    ) ; then
        echo "$dir: Replaced with '$name.ogg' successfully." >&2
    else
        echo "$dir: Processing aborted due to an error." >&2
        exit 1
    fi
done

with # ... Work ... obviously replaced with the code you use to convert your input files into the final movie.

If you change a directory name while the script is still working, it will abort. You can then safely rerun the command. If the final movie already exists, the script will not regenerate it, just move (and rename) it.

I found that "basename `pwd`" didn't handle spaces too well.

Code:

filer@nellie:~/Video Edit/custom$ basename `pwd`
Video

Also, will the following line run Script C from within the Subdirectories, even if the path to Script C is in the parent? (no need to copy it to subdirectory?)

Code:

find tree-containing-directories -mindepth 1 -depth -type d -print0 | xargs -r0 path-to-Script-C

catkin · 11-07-2011, 02:34 AM

Quote:

Originally Posted by Valpskott

I found that "basename `pwd`" didn't handle spaces too well.

Use ...

Code:

basename "`pwd`"

... instead to prevent the shell from tokenising the pwd output into words where there is whitespace (space, tab, newline).

Or you may prefer

Code:

basename "$( pwd )"

for reasons explained here.

David the H. · 11-07-2011, 01:55 PM

Instead of basename+pwd, you can use bash's already supplied PWD variable and parameter expansion.

Code:

path="${PWD%/*}"
name="${PWD##*/}"

Valpskott · 11-08-2011, 01:53 AM

Nominal Animal, you are bash script crazy!
Tried your solution and it worked. Thank you very much!

Also, thank you all for taking your time to assist me.

I'll mark this thread as solved, but I'm still curious of how the PWD thing works and how variables work, and what the correct syntax is between the two.

Would this be correct

Code:

$mydir=basename "`pwd`"
echo $mydir

Would that print the name of the directory I'm in? Or what is a way to load variable $mydir with the name of the current dir(not complete path) I'm in?

David the H. · 11-08-2011, 09:06 AM

Quote:

Originally Posted by Valpskott

Would this be correct

Code:

$mydir=basename "`pwd`"
echo $mydir

Would that print the name of the directory I'm in? Or what is a way to load variable $mydir with the name of the current dir(not complete path) I'm in?

No, that's incorrect.

Let's go step by step, starting with basename. And as an example we'll even use a filename with, not one, but three spaces in it.

Code:

$ basename /home/david/test/my   file.txt
my

Hmm, didn't work. What happened is that the shell broke the filename up into two arguments "/home/david/test/my" and "file.txt". Since basename only accepts one argument, the second part was dropped.

So we quote the filename:

Code:

$ basename "/home/david/test/my   file.txt"
my   file.txt

This forces the shell to treat the entire string as a single unit, rather than breaking it up at whitespace. See the links below for details.

Now, we want to create a variable to store that value in. First of all, the syntax for setting a variable is very simply name=value. You don't put a $ in front of the name when setting it, only when retrieving the value. And there are no spaces around the equals sign either.

Let's try to set it directly first:

Code:

$ name=my    file.txt
bash: file.txt: command not found

Whoops! We had the same word-splitting problem again, and it thinks "file.txt" is meant to be a second command. And of course the solution is again to quote the string.

Code:

$ name="my    file.txt"

Now if we want to insert the output of one command into another command we use command substitution. This is either `..` or $(..), with the latter being much preferred.

In this case we insert the output of basename into the variable setting.

Code:

$ name="$( basename "/home/david/test/my   file.txt" )"

Notice how I put a second level of quotes around the "$(..)". This ensures that the output is treated as a single unit, just as with the literal string.

(Actually in this specific situation quoting isn't needed, as substitutions get copied over without word-breaking, but it doesn't hurt, and they are necessary elsewhere).

Now let's echo back the variable to check its value.

Code:

$ echo $name
my file.txt

What happened to the extra spaces? Well, without quotes the shell again broke the output up into separate arguments, and echo got passed two separate strings, "my" and "file.txt", which it printed separated by a single space.

Code:

$ echo "$name"
my   file.txt

That's better.

In fact, let's do something funky and try passing the output of basename directly to echo.

Code:

$ echo $( basename "/home/david/test/my   file.txt" )
my file.txt

$ echo "$( basename "/home/david/test/my   file.txt" )"
my   file.txt

Just as with the variable, the quotes protect the string formatting.

So as you can see, it's vital to understand how the shell handles arguments and whitespace. Read and absorb these three links.

http://mywiki.wooledge.org/Arguments
http://mywiki.wooledge.org/WordSplitting
http://mywiki.wooledge.org/Quotes

And don't forget the suggestion in my last post. $PWD is a variable automatically set by the shell that holds the full path to the current directory, in the same format as the pwd command.

Code:

$ pwd
/home/david/test
$ echo "$PWD"
/home/david/test

And built-in parameter substitution rules can take the place of the external basename and pathname commands.

Code:

$ name="/home/david/test/my   file.txt"
$ echo "$name"
/home/david/test/my   file.txt

$ echo "${name%/*}"	# strips off everything from the end to the last "/"
/home/david/test

$ echo "${name##*/}"	# strips off everything from the beginning to the last "/"
my   file.txt