I am contemplating building a desktop environment that works with head mounted displays like the Oculus Rift. I want to create something that can display existing programs as 2d planes in 3d space. I also want future programs to be able to provide true 3d content.

That being said, I am not sure exactly which layer of linux to try and replace, or even what all the layers are. It seems like the relevant terms include: Window Manager, X Window Server, Xlib.

What is the name of the component that I should replace?

This might be a gross oversimplification but lets say I started with Debian as the core (not its desktop environment, just the command line) what would I need to do to get a standard gui program like gedit to display and work with mouse and keyboard input? A "Hello, World" desktop if you will?

I reason that if I can get something like that going, that I should be able to move the resulting data to textures in OpenGL and render them where I want.

I am floundering trying to find the right questions to ask, so any clarity you can provide would be appreciated.

this is a very ambitious project

I would suggest you start with learning something about 3d space and openGL
if you have done this already, than start writing your desktop

startx takes an application as parameter which will be executed, this can be your desktop, or just a xterm

so start in runlevel 3, and then startx mydesktop

you can test this with some other app, eg xterm

that's it, so no need to replace anything

If I write a c++ app and start it using startx how do I start another app like gedit and display its graphic output on the screen?

You need to create a compositor.
You can find a simple version I created:
https://bitbucket.org/ruapotato/vrsuse.git

(all this does is display half the screen side by side)

This issue that needs solved is barrel distortion. Without applying barrel distortion to both sides the screen will have "pincushion distortion" on the screen.

This might be fixable with hardware (a convex mirror creates barrel distortion)

Anyway I would love to help with your project. I think VR + Linux is the next big step.

heh.
reminds me of eaglemode:
http://eaglemode.sourceforge.net/