I've managed to install debian on an armv5 device(namely the zipit2). I've read that compiling will allow me to have a binary that's specific to my processor. Does that mean I can take ANY source code and as long as it's compiled on my zipit it will work on my zipit? for example fenec?

As long as it is portable enough to be compiled for ARM, and all dependencies are met, yeah.

Although you wouldn't compile on the ARM machine, you would cross-compile it from a more powerful machine.

what do you mean as "portable enough"? you mean the code has to be optimized for all those different processors?

Up to a point/yes, and more: it is possible for the code to be written with assumptions about the display, the way interrupts work on the particular processor, the organisation and access to memory that may not be consistent with the hardware that you have. And it is not just 'the program' but the program, plus all of the dependencies.

And there is speed; while you might think that this is just an issue of how fast the final program runs, it is quite possible that iof you run the code on a platform that is, eg, an order of magnitude slower, race hazards could show up that were never visible on the original platform.

So while, in principle, compiling for your platform could, theoretically, get anything working, in practice you might have a few interesting hurdles to overcome, and, if you are not good at debugging and have good documentation, you might find some of those hurdles a bit tough.