The speed difference you would win with placing your kernel in sector 0 instead of the bootloader is so minimal you wouldn't even notice the difference. If you have set the bootloader to no delay then it will instantly start loading the kernel. I'd say on a reasonable computer that would take maybe a few microseconds, if not nanoseconds.
You would win a lot more speed by doing what trickykid suggested. If it's a kernel just for playing mp3's I'm sure you could get the kernel smaller then 400k. You might also want to hack into all your bootscripts since a standard distribution will do a bunch of extra checks on bootup which you might not really need.
You might want to read the following howto
It explains the whole bootprocess which might help you to find out which parts you could skip.