i'm not sure if you are building a system or just toying with kernel and app develpement.
if you really want a hard realtime process that process has to actually be the kernel itself and the linux kernel has to be run by your non preepmtable realtime core process at a lower priority in what we would call user space. thats what i would try anyway.
if you are just setting up a music system i can say what i do to the kernel and the rest of my configuration for audio processing but as others have pointed out i don't think this has anything to do with hard realtime. hardrealtime i think is not even be possible on the linux kernel.
i might be wrong on that one.
this works as a desktop machine just fine by the way.
first is the kernel configuration for soft realtime
apply this patch
then in the config choose
Default I/O scheduler (CFQ)
and "Complete Preemption" as the preemption model
i use 1000 nanoseconds on the high res timer and i use a tickless system.
(tickless most likely doesn't effect realtime preemption) can't really remember why i did that.
make sure you choose Extended RTC operation
and of course Use RTC as default sequencer timer
make sure you have your hard disk set up in the fastest possible way.
OK you are also going to want PAM authentication that allows realtime access for normal users but for now you can do realtime as root till you figure that out.
next there is a utility "chrt"
my soundcard is on IRQ 11 so i set at boot
chrt -f -p 82 `pidof "IRQ 11"`
for the life of me i can't remember the exact reasoning behind this but i had it well reasoned out when i did it ;-}
next adjust the reatime priorities of the RTC mine is on IRQ 8
chrt -f -p 98 `pidof "IRQ 8"`
next i set the latency on the soundcard a little higher than it was.
higher latency means larger bursts acrocc the bus
setpci -v -s '04:08.0' latency_timer=80
my soundcard is 04:08.0
this latency value you will have to experiment with to find the best settings for your machine.
ok and last but not least is to turn up the niceness of xorg
performance of xorg really sucks but it cant be so nice you don't have a fluid mouse curser so it's kind of something you have to play with as well. i use something like
renice +2 -p `ps -C Xorg -o pid=`
lastly you have to adjust your alsa and audio server buffers or for whatever app you are using and make them as small as possible for the lowest latency possible.
if you have a very fast soundcrd like m-audio or something you can go with very small buffers
without getting overruns.
if you have a slow audio card you will need to use larger ones but make them as small as possible.