Help with converting PNG to JPG (every 0.5 seconds)

andy2008 · 05-11-2011, 04:21 AM

Hello everyone!

I'm trying to figure out how to solve the following problem:

I'm generating a PNG every 0.5 seconds (this PNG always overwrites itself)
Because a PNG file is too large, I need to convert it to JPG.

I made a script:

Code:

#!/bin/sh
while [ true ]; do
 /usr/bin/convert /home/file.png /var/www/html/website/file.jpg
 uleep 500
done

This script doesn't work well, because many times the conversion happens before the png file was completely written (as I said, this PNG is generated every 500 ms), so the jpg file will often result in a damaged picture, half a picture, etc.
Is there any way I can tell the "convert" command to do the conversion ONLY when the PNG was completely written on the HDD, and not before?

jadrevenge · 05-11-2011, 05:07 AM

#!/bin/sh
while [ true ]; do
while ! ( /usr/bin/convert -quiet -regard-warnings /home/file.png /tmp/$$.jpg 2>/dev/null ) ; do
sleep 0.02
done
cp /tmp/$$.jpg /var/www/html/website/file.jpg
sleep 0.5
done

neonsignal · 05-11-2011, 05:15 AM

You could make use of the inotify-tools package, which allows you to script inotify events:

Code:

#!/bin/sh
while [ true ] ; do
   inotifywait -e close_write file.png
   /usr/bin/convert file.png file.jpg
   echo "converted"
done

You cannot just check whether the file has been closed (eg using 'lsof file'), because you will still have a race condition.

The above script also has a race condition, but hopefully 500ms is long enough to do the conversion before the next write to the png. An improvement would be to do the conversion to a temporary file, and test the return value from the convert command; if it gives an error, then don't copy the temporary file to the public one.

Incidentally, is the 'usleep' command you were using a 500 millisecond sleep or a 500 microsecond sleep?

andy2008 · 05-11-2011, 05:59 AM

Thanks a lot to both of you for your answers.

@neonsignal: You are right, I was using microseconds.

Here's the script I'm using now:

Code:

#!/bin/sh
while [ true ]; do
inotifywait -e close_write /home/file.png
while ! ( /usr/bin/convert -quiet -regard-warnings /home/file.png /tmp/$$.jpg 2>/dev/null ) ; do
sleep 0.02
done
cp /tmp/$$.jpg /var/www/html/website/file.jpg
sleep 0.5
done

Even though it works MUCH better now, every 20 seconds or so, one or two of the converted JPGs is still corrupted. I have no idea why...

Later edit: inotify also shows these:

Code:

Watches established.
/home/file.png IGNORED
Setting up watches.
Watches established.
/home/file.png IGNORED
Setting up watches.
Watches established.
/home/file.png IGNORED
Setting up watches.
Watches established.
/home/file.png IGNORED
Setting up watches.
Watches established.
/home/file.png IGNORED

neonsignal · 05-11-2011, 06:50 AM

This probably means that the generating program is not just rewriting the file, but recreating it (which means the inotify returns with an error, because it loses the node). So this following script instead waits on a notify on the directory; however, it also has to test which file was written to (on the assumption that there might be other files in the same directory).

Code:

#!/bin/sh
while [ true ] ; do
   inotifywait -e close_write /home | grep 'file.png'
   if [ $? -eq 0 ] ; then
      /usr/bin/convert file.png file.jpg
   fi
done

If the inotify is working correctly, you do not need the half second delay.

andy2008 · 05-11-2011, 07:21 AM

Well, now the output has changed:

Quote:

/home/ CLOSE_WRITE,CLOSE file.png.swp
Setting up watches.
Watches established.
/home/ CLOSE_WRITE,CLOSE file.png.swp
Setting up watches.
Watches established.
/home/ CLOSE_WRITE,CLOSE file.png.swp
Setting up watches.
Watches established.
/home/ CLOSE_WRITE,CLOSE file.png.swp
Setting up watches.
Watches established.
/home/ CLOSE_WRITE,CLOSE file.png.swp

So, it looks like there is actually a renaming going on... but it still doesn't work.
Using your latest script, brings us back to the beginning... corrupted JPGs every 1-2 seconds.
I don't understand why file.png gets ignored, even though it's constantly replaced...

jadrevenge · 05-11-2011, 03:00 PM

you could try it with a dollar after the filename:

Code:

#!/bin/sh
while [ true ] ; do
   inotifywait -e close_write /home | grep 'file.png$'
   if [ $? -eq 0 ] ; then
      /usr/bin/convert file.png file.jpg
   fi
done

neonsignal · 05-11-2011, 05:42 PM

We may need to know more about the image generation sequence. As you say, the originating program may be creating the temporary (.png.swp) and then renaming it, in which case you would have to wait for a 'move-to' event rather than a 'close-write' event, eg:

Code:

#!/bin/sh
while [ true ] ; do
   inotifywait -e move_to /home | grep 'file.png$'
   if [ $? -eq 0 ] ; then
      /usr/bin/convert file.png file.jpg
   fi
done

(jadrevenge has added the '$' to prevent it matching on 'file.png.swp', which is triggering the convert too soon.)

andy2008 · 05-12-2011, 12:14 AM

Again, thanks for all the help, guys.

Using "moved_to" for the inotifywait event seems to do the trick, it matches the file correctly:

Code:

/ramcache/ MOVED_TO file.png
Setting up watches.
Watches established.
/ramcache/ MOVED_TO file.png
Setting up watches.
Watches established.
/ramcache/ MOVED_TO file.png
Setting up watches.
Watches established.

I'm now using this script, which seems to work best, but still not 100% OK:

Code:

#!/bin/sh
while [ true ] ; do
   inotifywait -e moved_to /ramcache | grep 'file.png$'
   if [ $? -eq 0 ] ; then
      /usr/bin/convert /ramcache/file.png /ramcache/$$.jpg 2>/dev/null
      usleep 1000
      cp /ramcache/$$.jpg /var/www/html/website/file.jpg
   fi
done

With this script the conversion process works much better, but every 30 seconds or so, I still get corrupted JPGs.
I even created a ramdisk to speed up caching, but it doesn't seem to help a lot.

andy2008 · 05-13-2011, 02:04 AM

Solved it !!
The problem was : cp /ramcache/$$.jpg /var/www/html/website/file.jpg
I changed it to: mv /ramcache/$$.jpg /var/www/html/website/file.jpg

Now the corrupted JPGs have disappeared.

Now, another problem:
Is there some way to send the whole " inotifywait -e moved_to /ramcache | grep 'file.png$' " to /dev/null?
I tried various ways but it doesn't seem to work.

Later edit: Never mind, " inotifywait -e moved_to /ramcache 2>/dev/null | grep 'file.png$' >/dev/null " seems to do the trick. Thank you, neonsignal and jadrevenge for all the help !

arizonagroovejet · 05-13-2011, 07:16 AM

Did you look at whether it's possible to make whatever it is that's producing the PNG file send it's output to stdout instead of a file? Because if you can do that you could pipe it's output to convert to produce a JPEG and save yourself all this hassle with ionotify.

andy2008 · 05-14-2011, 12:49 AM

No, unfortunately it can only save the PNGs in a specified folder.