Only if you built an 'arm-linux-gcc' or can install it as a package on your distro.
Otherwise I'd suggest going to the Linux From Scratch site and following the link to the CLFS project; read the section on setting up the cross-compiler. Note that CLFS uses the 'uClibc' library rather than 'glibc', so don't follow the instructions exactly unless you're also building for uClibc.
The steps go something like this:
1. Build a cross-compiler:
- BUILD is the architecture of the machine running the compiler to build the other compiler
- HOST is the machine which will ultimately run the program being built (the compiler)
- TARGET is the machine which you want the compiler (which is being built here) to build for - ARM (check for typical 'triplets')
Confusing enough yet? I hope I didn't mix up 'HOST' and 'TARGET' ... you'll find out soon enough I guess ...
2. Build the C library 'glibc' or 'uClibc' are both popular; uClibc if space is important to you; glibc is enormous. See the CLFS docs for tips on how to do it. You need system headers from the linux kernel - this is just the public API, not the kernel's private headers.
3. Rebuild the cross-compiler; now you can rebuild the tools to use only the specially built C library. There is some hacking involved so that, for example, the linker doesn't attempt to link to /lib/libc.so.
And there you have it - oh - just to keep things safe, it's useful to have a user account exclusively for cross-compiling; that way the environment can be set up to avoid a few annoying traps (mostly related to linking or accidentally running the wrong compiler).
Since building the cross-compiler is such a big job the first time you do it, keep plenty notes - better still, write a script as you go along so it can all be rebuilt at the push of a button. The same goes for building the libs - and for that matter, any other software you may build with the cross-compiler.