If you are not familiar with compiling the kernel/drivers in linux this is probably not a very good place to start. You are right about driver support. Some manufacturers make the situation worse by not releasing hardware specs and interfaces. Until recently VIA was one of these.
In fact framebuffer mode is good for watching movies. If you see the penguin as the system boots then you are using the framebuffer. KDE has the console as one of the login options, and you can try mplayer or whatever from there. I found I could watch an MPEG4 full-screen with an M10000 and just standard drivers.
Otherwise there is supposed to be a special version of Xine supplied by VIA which makes better use of the hardware, but I haven't tried it.
There are a couple of specialist EPIA distros around (
www.freevix.org/forum/ and
www.freepia.org) but they look like they are in a pretty rough state.