Well no, that wouldn't turn SSH into a temperamental monster. If its a power saving reason from the information you've given I'll buy you a beer.
I'll explain; the power settings could only be a possible reason if you didn't have to jiggle the mouse about after boot, too. The fact that you have to do this straight after boot to get it to work suggests a different reason.
Now, perhaps you'd like to try and connect a few times to replicate the error, then walk over to the Fedora box and give me the output I asked for. Also, could you try connecting from another machine?