In old days when X Windows was released by Xfree86.org it was advised to look up source codes of it's included video card drivers.
These days Linux kernel ships with basic 2D KMS video card drivers. So it's source can be used to learn.
Others are svgalib drivers, linux kernel frame buffer drivers.
You should also see a section on video card at
wiki.osdev.org