I'm using RHEL 5.2 server in this instance.

I installed XFree86-libs-4.1.0-50.i386 as well as openmotif-2.2.3-2.i386. After rebooting, right as the system would normally enter the GUI part of boot-up, the screen goes black, and never recovers. All I see is the backlight of the monitor flashing on and off. I can enter single user mode however. I removed motif, which didn't fix the problem. Removed XFree86, that fixed it. Nothing in any error logs or anything, it just stopped working. Am I using the wrong versions or something? I don't understand how libraries can do this...

Redhat EL5 : You cannot use random packages, meant for another OS.
Redhat EL5 is from 2007, and mostly based on Fedora 6.

'XFree86-libs-4.1.0-50.i386' is a Redhat 7 package, from 2003.
'openmotif-2.2.3-2.i386' is a Fedora 4 package, 2004.

Redhat EL5 is commercial, not very usable with no paid account.
Not paying : Don't use it.
CentOS 5.5 is the free identical version. http://www.centos.org/

Some CentOS packages may be used. http://mirror.centos.org/centos/5.5/
But not quite without problems, if you never updated anything
on that old RH EL 5.2 ( Current is "Update 5.5" ).
..