No...there IS a noticable difference with speed regarding different kernel sizes.
A great place to check is is with your stock 2.4- check out lsmod, and find what modules are being loaded and, more importantly, used. If you never use some stuff (firewire, for example), don't put it in there.
Personally, i've gotten my kernel to about 1.5M or so, and am pretty satifsied. Sure, there are still some things I could probably get out of it, but it is much slimmer than the stock 2.4. Just play around...eventually you'll get it. Then just save your old .config file and you'll be good for the 2.6 series

.
Hope this helps
Andrew