Didn't realize it was your mission critical app. Thought it was just that one last stubborn app before you could switch altogether. I looked filemaker 5.5 up on wine and it doesn't look like it's working, but it has a linux native client, so it wouldn't get as much effort. I understand you wouldn't want to gamble with it, but setting up a test server wouldn't be much of a gamble, just to see how things go. I don't think the version of linux matters much as far as whether or not it'll run, what matters is whether or not the dependencies are met. It will matter for support, if you still have support, that it hasn't been certified on debian, or another flavor of linux. But if you are 100% of the "support", that would be your call.
I'm personally a big fan of debian on anything server or mission critical. Have you looked at kexi ? Here is a link.
http://packages.debian.org/stable/kde/kexi
Debian does run on the latest and greatest hardware too. Might be worth it to set up a test machine, see if it is something that might work for you, especially if the mac/windows version you are currently running is having some issues. If it doesn't, if it just humms along on it's own just dandy, I'm not so sure I would take a risk to fix something that isn't broken, when it's my ass on the line if it doesn't work. Which is exactly why Red Hat makes the big bucks, if you buy support from Red Hat, and something breaks, YOU won't be getting fired.
Then again, if it was my company and my money on the line, I for certain wouldn't want my database gold hung out on an older constantly vuneralble windows box. (By older windows I mean anything not XP, it would be crazy to use vista for anything important, and win2000 support is ending soon, if it hasn't already.) There is a reason NASA doesn't use anything windows on the space shuttle. It's because M$ themselves tell them not to use it anywhere mission critical. If my retirement was depending on it, I know which gamble I would prefer to take.
You can set up linux to look just like windows, which might make a few people in your office feel better. There are XP themes for iceWM for instance, and I just read about a slackware based desktop that aims for a windows clone sort of feel. STX linux I think it is. Here, found a link, it uses the Equinox Desktop Environment, which feels somewhat like windows.
http://distrocenter.linux.com/articl.../05/22/1828207
http://ede.sourceforge.net/page/
I haven't tried it myself, as I use mostly newer hardware, and I'm a big Kubuntu fan, but I have used slackware in the past, and I liked it. Slackware also has a fantastic reputation, as long as you know what you are doing, or are capable of looking a few things up. To just run a client or two, any one used to windows should feel just fine with it. (With the right desktop.)
Good luck.....let us know what you decided on. Might help someone else make similar choices someday, and I'm curious how it'll go.
David
P.S. Looks like their site (stx-linux) is down for a redesign at the moment.
P.P.S. My vote would go to debian, if my ass was on the line.