The binary is ready to go, so to speak. You'll be up and running in a short time. If you install the source, you'll end up with a faster machine (if you use the right compile flags), but it'll take forever to get installed. Ask anyone who has installed Gentoo or LFS. Depends on what you want the source for. If you want to read it and improve upon it, I'd just download the soueces that you need to look at, one at a time, to a working directory on your machine and develop from there.
My suggestion is to download the binaries and then recompile your kernel for your specific machine (there are some good Debian how-tos on that). You'll end up with the same result and it'll be up and running faster.