Yep, I have misread the original output. It is E5620 CPU, which is 4 core with 8 threads and three memory channels. Two CPUs give me 8 physical cores. I guess my original confusion was how three memory channels can be divided between 4 cores (or 8 threads), but it looks like my assumption that is divided equally is incorrect.
Anyway, from this discussion I understand that I just need to test each particular code with or without hyper-threading to see what gives better performance.