I found 'alot' of diskspace used in /usr/src by a custom kernel I built:

src]# du --max-depth=1 -h
1.2G ./linux-2.6.6

linux-2.6.6]# du --max-depth=1 -h ./
435M ./drivers
3.0M ./.tmp_versions
205M ./fs
42M ./include
7.0M ./security
1.9M ./init
4.4M ./lib
6.6M ./crypto
7.3M ./Documentation
99M ./sound
64K ./usr
9.7M ./kernel
69M ./arch
1.9M ./ipc
7.9M ./mm
222M ./net
1.8M ./scripts
1.2G ./

browsing the dirs I see many object files left in there. can I remove these and other binary files (how best?) and have a /usr/src/linux that works for most new app installs?

make clean

should clean up the object files.

nowdays most packages no longer look in /usr/src/linux
the link to the source directory is a link "build"
in /lib/modules/kernelVersion

it is absolutely imperative to keep the build directory of your currently running kernel
and if you have a link /usr/src/linux it is absolutely imperative to update it so it points to your currently running kernel source or delete it altogether..one or the other

there are headers in /usr/include/linux that take care of the user space programs

some of the stuff generated during the build is needed but most likely not the object files
like you said....
I think but am not sure that "make clean" will wipe out some configuration things you will need
including your .config so you won't even be able to rebuild without starting from scratch.

make mrproper wipes all... not sure about make clean. I thought it was okay, but worth checking. You can always make a backup of your .config anyway.