I think the select() part of your program would be more interesting for investigations about this problem. Why do you use signal driven i/o at all?
i tend to agree with this approach.
you can create a thread that blocks on select indefinately or until woken/interrupted without registering the IO signal.
However, before doing that, i recommend commenting out the part in your code that handles the signal and see if it has any affect on the touchscreen.
if it has, return the signal handler code, and comment out all other code that relates to serial access (select, read, write etc...) to make sure that it's the signal capturing that causes the problem.
that way you can be sure that you know what the problem is and you won't spend hours/days (depending on system complexity) making adjustments that eventually won't advance you at all towards a complete stable system.