xmms also comes with IRman remote control. I don't know what hardware you need to connect to the computer, but using xmms with remote control would be great. If you are doing your own ripping, check into using ogg format instead of MP3. It is smaller, sounds better, is open, and there are a LOT of linux tools for ripping to ogg. oggripper, for one, if it took you more than 6 seconds to understand how to use it, well, you have had too much wine, time to call it a night. :-)
I would think, for a fully optimized VectorLinux install, no print services, stripped down to nothing bu the Alsa sound server and cron, etc (the bare minimum), you could be looking at about an 8 second boot to login, then another 6 to load the lightest windows manager, piped to a TV out card to your TV.
One perk for xmms, is expanded stereo. It is a standard plugin that makes music come alive, imho. I connect my computer (via 30' of RCA cable), because of the sound quality.
I helped someone figure out how to use xmms -e as a MIME type to enque that song to the end of the playlist, or xmms -E for enque as next song, and xmms -Ef to enque as next song, then skip forward to it. Would be great to have that added as remote control function. :-)
I am very interested in this kind of thing, and yes, USB 2.0 is fully available in Linux.
Try downloading some 'Live CDs', knoppix, vector, etc, and experimenting. No HD install, until you want to, and you can see how it all works, or if it meets your needs. Note: on a 266, wine would probably be a bit of a dog. See if you can find useful native stuff here.