I don't think there are any downsides. In fact, those processors are built to scale.
Although I can see one quirk in your choice of governors. On AC power in some cases you could overheat (or to be more precise, reach critical temperature) your processor after long time of 100% usage. If your notebook has a fast processor and inadequate cooling system, as performance governor always runs the CPU at maximum available frequency. I personally prefer conservative governor, which scales frequency one step at a time although some say, that ondemand is better (
http://www.bughost.org/pipermail/pow...ay/000071.html) as far as energy efficiency goes.