Have you checked the /usr/lib/codecs/ to see if you have the latest versions of things?
Some of the players like VLC enable filters by default. Disabling them might boost performance to a tolerable level. ffplay is good in running multi-threaded that which can be run multi-threaded. mpeg2video is multi-threaded. msmpeg4v2 is not. Most things h.264 are becoming multi-threaded. Make sure you enable threading (pthreads?) where appropriate. Plus all that other stuff. Your card on that system should be more than capable IMO. I got an aftermarket card, just to get HDMI output. If you're in the market for a new card, you might check out some of that Cuda stuff.