You could use top to see how much cpu your application is using. If it is already using 100% then efforts to give it higher priority are futile. See:
"My application reads one byte at a time from each serial port, using read()."
You could greatly reduce your cpu overhead if you could find a way to read more than one byte at a time. Could you read an entire frame in one read?
You could also greatly increase efficiency if you wrote a device driver for the device. You could set up the device driver to accept reads for a byte, a frame, or a record (if the concept of record has meaning for this device). Here is some information on Linux device drivers:
"Is there a commonly available version of Linux that has less system overhead than RH 8.0? Is Blue Cat or Lynx any better?"
You are dealing at the level of kernel efficiency. You might conceivably get better efficiency by compiling a different kernel configuration but changing distributions is probably irrelevent.
"Is there an easy way to "strip down" the RH Linux session so there are fewer processes running?"
If you do manage to do this then you will end up with an application that will work in your development environment but will fail in the real world. But anyway, the way to strip RH down is to remove the startup scripts that start processes that you deem superflous.
Be prepared. Create a LifeBoat CD.