I think this could be caused by any number of reasons. Your description seems quite vague to me. It could be a hardware problem or a problem with XFree or both. You'll have to see if you can't narrow down your problem more before I can help. Try [physically] removing or disabling all of your cards, such as network, sound. Everything but your video card. If you still have the problem, try swapping our your video card with another. If you have a lot of devices built right onto your motherboard you may have to disable those modules from loading into your kernel.
If you can't find any problems with your hardware, make sure that your XF86Config file has all of the proper configurations (you'll probably want to check this before your start pulling out cards from your box). Specifically, make sure your mouse, keyboard, monitor, and your video card are all setup correctly.
Does this happen ever single time, or just most of the time? Perhaps if you list your setup, such as what video card and motherboard and any other hardward you're using. Also, are you using a usb mouse/keyboard?