Funnily enough I've had to do the same thing at work recently. Usually when you get your hands on a device like this it should come with compilers specific to the architecture. The docs in mine simply require the path to include the toolchain you are compiling against.
However, I did have to make sure that any code outside of what was shipped with the device pointed to the correct cross-compiler ie.
CC=<location of gcc cross compiler>
[EDIT:] I'm doing this on a host machine - not on the device itself. After successful compile I copy across. Add -g to your CCFLAGS when compiling and use gdb if you run into trouble executing on the ARM device.