I would try editing your xorg.conf file. I think maybe xorg-server's auto detect is taking unusually long.
Try adding these lines at the beginning of your /etc/X11/xorg.conf
Option "AutoAddDevices" "off"
See if this helps, I still think its your xorg.conf file...
Also you can go to
System > Preferences > Sessions
and uncheck all that isnt needed, like login sounds, visual assistance, tracker, beagle anything that takes resources to start. You can always add delays to these startups if needed with a