I have not had any problems with festival on Slackware. I installed the speech_tools and festival packages from
www.slacky.it and then added some MBROLA voices manually.
When you use it under KDE, are you doing it independently of the KTTSMgr? With the K Text-to-Speech Manager (which uses festival as a backend) you can choose to route the audio through aRts or ALSA, so you should be able to get something working there. And then see if it still gives you problems with voice switching through the kttsmgr.