X doesn't have to run a window manager. You can start it with any application, like a kiosk system. This is obviously a limitation for common desktop users, but in your case it would happen on purpose. Your application would be the only program to display anything. Though you have to deal with user input and event loops yourself.
To span multiple screens you could maintain a large offscreen area, and copy parts of that image to the displays that are involved. Double-buffering is anyway good for smooth visual effects.