BASH - determine when file is no longer written to

marwal · 03-27-2007, 02:43 PM

Im writing a script that rips a CD and encodes the WAV-files to MP3. Thats kind of basic.
What I wanted to do was to have several instances of encoders running parallell with the ripping. That went kind-of alright.

Im running icedax (cdda2wav) like this :

Code:

icedax --bulk -q dev=$DEVICE --cddb 1 && eject $DEVICE &

It then starts to rip all tracks. Another script is on a look-out for WAV-files and assignes them to the encoder (making sure there is no more than 3 encoders running at the same time).

My problem is to determine when a WAV-file is completly ripped. I was trying to use 'lsof' but it reports 'in use by icedax' for all WAV-files until the ripping process is completed.
Is there a way to make sure the WAV-file I assign to an encoder is complete?

I know I could use icedax and ripp one track at-a-time but I want to beat the application Grip and rip-encode faster.

I'm also intrerested in all kinds of ideas regarding my scripting. Im kind of new at BASH and need all tips i can get

.

Here are my scripts:

rip

Code:

#DEVICE=/dev/scd0
DEVICE=/dev/hdc
function what_time() {
  TIME=$(date +%H:%M:%S)
}

what_time; echo "$TIME : rip is starting"
# start ripping
icedax --bulk -q dev=$DEVICE --cddb 1 && eject $DEVICE &

#start encoding-agent (give it 10 seconds)
sleep 10
what_time; echo "$TIME : mwrip is calling encode"
./encode &

encode

Code:

function what_time() {
  TIME=$(date +%H:%M:%S)
}

function send_agent() {
  # is there job in stack
  if [ "${STACK[$POINTER]}" != "" ]; then
    NO_JOBS=$(jobs | wc -l)
    # if current jobs are fewer than 3...
    if [ $NO_JOBS -lt 3 ]; then
      what_time; echo "$TIME : found ${STACK[$POINTER]} and a free slot..."
      ./agent ${STACK[$POINTER]} &
      let "POINTER=$POINTER+1"
    fi
  fi
}


TRACKS=$(cat audio.cddb | grep TTITLE | wc -l)

STACK=[]
POINTER=1
COUNTER=1

while (($POINTER <= $TRACKS)); do
  while (($COUNTER <= $TRACKS)); do
    NO=$(printf "%02i" $COUNTER)
    FILE="audio_$NO.wav"
    if [ -f $FILE ]; then
      STACK[$COUNTER]=$FILE
      #echo ${STACK[$COUNTER]}
      let "COUNTER=$COUNTER+1"
    fi
    send_agent
  done
  send_agent
done

agent

Code:

WAV=$1
MP3=${WAV%.wav}.mp3
function what_time() {
  TIME=$(date +%H:%M:%S)
}

what_time; echo "$TIME : agent ($$) recoding $WAV to $MP3..."

lame -h -b 128 --quiet $WAV $MP3

what_time; echo "$TIME : agent ($$) is done."
exit 0

anomie · 03-27-2007, 03:19 PM

Let me suggest an idea on a conceptual level; take it or leave it.

1. Script enters a loop where it rips to wav files, one at a time.
2. After each wav file is ripped (within the loop), a bg job is submitted to encode that particular wav to mp3.

Make sense? This will ensure that each wav is ripped before you start encoding it. It also means that the ripper can continue while encoding jobs run.

macemoneta · 03-27-2007, 03:22 PM

When you start the process, save its process ID. For example:

Code:

somecommand &
somecommand_pid=$!

You can then later wait for that process (or those processes) to be complete with:

Code:

wait $somecommand_pid $someOthercommand_pid

theNbomr · 03-27-2007, 03:28 PM

Since ripping is likely to be an IO-bound process, you may find that there is no improvement in overall performance, and there may actually be a net loss, due to cache flushing and/or additional seeking. I don't know that bash has fine enough control over process monitoring to do what you are trying to accomplish. Using something slightly more sophisticated such as perl, which would allow you you fork an arbitrary number of processes, and then do waitpid() on each of them would be a more refined method. My opinion.

--- rod.

EDIT: macemoneta has just pointed out that Bash can indeed do what I suggested as a good reason to use perl.

marwal · 03-27-2007, 03:59 PM

Thank you all for your responses.
anomie) I know I could rip one track at the time. That would give me control over when that track is ripped and start encoding. It is slower though to rip track 1,2,3,4... than it is to rip --bulk. With bulk-ripping i rip one cd in about 2 minutes.

macemoneta) About PIDs. Since a run icedax with the --bulk parameter there is no sence in waiting for that job to finish. It would be the same as to first rip and then encode. My idea was to encode WAV-files on the fly while the ripping was still in progress.

theNbomr) Yes, ripping in parallell is no god idea. But encoding is and that is what i am doing. I keep three instances of lame-encoders running i parallell. I find (on my machine) that this increases encoding-speed. If I would run, like ten instances of encoders it would encode the pool of WAV-files slower.

If there is no way to make sure icedax --bulk is finnished with, say audio_01.wav - i would have to do as anomie says: rip one track, start encoding it while i rip track2 (and so on).
Possibly I can use filesize from the info-files icedax produces to determine if a WAV-file is done but I would prefer a system-call like "hello file, are you beeing written to? No!? Ok, i will now encode you and then kill you!"

macemoneta · 03-27-2007, 04:25 PM

If you just want the status of a background process, you can use jobs and grep the the PID. For example:

Code:

somecommand &
somecommand_pid=$!
.
.
.
running=`jobs -l | grep $somecommand_pid`
if [ "$running" != "" ]
then
   echo "encode still in progress"
fi

If you just want to rip and encode music in parallel, you know you can just use a package like grip, right?

marwal · 03-27-2007, 04:55 PM

Thank you macemoneta for your reply, although it indicates to me that you havent read the first, or even the second post i made in this thread.

macemoneta · 03-27-2007, 05:11 PM

I read all your posts, as well as the posts of everyone else. I understand them all, but I suspect you do not. Either that, or you are not communicating what you want clearly.

Grip is simply scheduling and dispatching highly optimized standalone rippers and encoders (which you can configure as you please). The idea that you will somehow "beat" this makes the basic premise weak.

I chose to assume you want to learn how to schedule and dispatch processes in your own code; but according to you, that's not what you want. So I now have to assume that you don't really know what you want. Can you clarify?

marwal · 03-27-2007, 05:37 PM

I like Grip. I use it. I'm aware of it (as i mention in my first posting).
It was the inspiration for me to try and optimize the rip-encoding process.

Even if i run my scripts rip- and encode parts in sequence i beat grip. That's because
1) i rip bulked
2) i have three encoders working (not 1 as grip).

The only thing missing now is to establish WHEN (in bulk ripping, running as ONE process) each WAV-file is completed so i can start encoding and NOT WAIT for the process to finish.

English is not my natural language it's challanging for me to express myself as well as i can in my native tongue. I'm sorry if this is making it hard for you (and others) to understand me.
I also realize my previous post might have sounded rude. I won't blame that on language-problems. It was more of a babelesque frustration-problem.
I know there is a solution for what im after and ill keep trying and report back here when i find it.
cheers.

macemoneta · 03-27-2007, 06:20 PM

Grip is somewhat better than just bulk ripping, in that you can exclude tracks you don't want, and the tracks are automatically labeled from freedb.org (or your favorite track database site).

To run multiple encoders on grip:

Config->Encode->Options and increase the number of CPUs. From the grip documentation at the site I linked to above: "This value specifies the number of simultaneous encode processes to run." In your case, set number of CPUs to 3 for three concurrent encoders.

marwal · 03-27-2007, 06:51 PM

macemoneta: I wasn't aware of that option in grip. That took away most of the entusiasm i had about his. And just as I realized the solution to my problem -> Since ripping is done in sequence, if audio_02.wav exists audio_01.wav must be done. So all i had to do was NOT to add the most recent WAV-file to the que until that file was the FINAL one.

Well... i think I'll program it anyway (you know... this was just a mind-exercise to begin with, like most of my small projects).