Sometimes you just hit a server that's having a few problems. I downloaded Scientific Linux once, and it took 7 hours — at one stage it was threatening 8 days! The usual thing to do is to stop and try a different mirror.
You also need to do your downloads when traffic is light and the ISP won't be "managing" it, as they so politely put it. Different types of traffic get different treatment: things that must run at a certain speed (VOIP, streamed TV) get full speed and downloads and torrents are run slow. Saturday or Sunday mornings, or after midnight is best.