Not exactly sure what you're asking, but this is what I think I understand.
You have a machine running some other OS with a monitor resolution of 800x600. You also have a Linux box that's running 1024x768 resolution on its monitor.
Screen resolution shouldn't be a big issue unless it's too large for the monitor to produce. If you have an old monitor, that could be the problem because it might not even be capable of displaying resolutions that you'd want to use. Basically though, any monitor you plug in should work as long as Red Hat can detect it and recognize its specs. Any help?