There is a really thorough book called "Essential Linux Device Drivers" by Venkateswaran (Prentice Hall). Covers many things from how to build the kernel, to interrupts, dma, a/v, debugging, and many other subjects. I started to get into it, but set it aside when I lost interest in C, for more interesting languages like Haskell, Lisp, and Prolog.
Some how I doubt I'll ever get back into Linux kernel development; however, there is an interesting paper related to the House project (a Haskell-based OS), which describes a Haskell approach to low-level hardware interaction:
http://ogi.altocumulus.org/~hallgren/ICFP2005/
Microcontroller programming is also rather interesting... not because of the languages involved, but because it is rather cool to be able to program a full-fledged computer contained inside a chip that is only slightly larger than your thumbnail.