I had the same problem with Musepack audio files.
Editing the appropriate four gnome-files according to the gnome administrator's guide didn't change too much (too make these modifications see the chapter about MIME types in the administrator's guide). The trick for me was to add also an entry for the new mime-type to the file /usr/share/mime/packages/freedesktop.org.xml as root and then run
After the next start nautilus treated and started my files like a charm
This seems to be a problem due to different locations to store the mime data in. Gnome has one place, the package shared-mime-info from freedesktop.org has another.
P.S.: The content sniffing seems to be turned off in gnome 2.6, so i had not to deal with this one.