If you compile and install a kernel "manually" and you want to remove it, you have to do that manually, too. (Just make sure that you still have another kernel on your hd that you can use when you boot the next time).
Fortunately, it is quite simple to identify the files related to a specific kernel.
Files that belong to a kernel 2.X.X will have that number in their file name as well.
First there is the kernel itself: vmlinuz-2.X.X
Second, a symbol table named: System.map-2.X.X
Most of the time a copy of the config file that has being used for compiling the kernel is being copied to /boot, too: config-2.X:X
The modules produced during compilation are stored in /lib/modules/2.X.X
You can simply delete them all and the kernel is gone.
DON'T FORGET to reconfigure the the boot manager (lilo / grub) to use another kernel and test it before you remove anything.