Yes, in fact, that is/was sort of what I'm doing right now.
you can get a nice pre-built binary for all the gcc stuff you'll need from:
To build a binary that doesn't include any standard libs, use the -ffreestanding gcc option.
Also, if you're building for an embedded system, you'll need a linker script that will put everything in memory where you want it.
If you download FreeRTOS, it includes several demos including one for the atmel at91sam7x-ek evaluation board with GCC stuff and a linker script for you.
Of course, if you really don't want to use an OS at all, you'll end up doing a lot of extra work depending on what you want it to do.
For testing your arm binary, you can use the free skyeye emulator:
Lesseee, what else..Oh, the binutils might not be included in the gnuarm package, so you might have to grab the source and build them yourself from gnu.org. Don't forget to use the -tarm-elf target option.
There, that should get you up and running.