I recommend Computer Organization and Design by Patterson and Hennessy. It is a pretty indepth book on... well, the Organization and Design of computers
I know it has been used in college architecture courses for years and years. Focus' on the MIPS architecture (because it is very simple) at first, and then talks about what is different on more modern architectures. My only warning... this book isn't exactly light reading.