I'm currently experimenting with luks/cryptsetup too. Personally I wouldn't kill the password slot just yet. In case you manage to mess something up you'll no longer have access to the partition, if you wipe the pw slot.
Although I haven't managed to setup my system yet, I've read a number of docs for cryptsetup/crypttab. Accordingly you need to setup a keyscript in the crypttab to read a keyfile from a usb thumb drive. At least on debian-based systems you would modify the crypttab with proper parameters and then invoke "update-initramfs -u" as root.
The keyscripts on my system (debian wheezy) are located in /lib/cryptsetup/scripts. I think there was a script that waits for a specific device to appear, then read something from it and pass that to cryptsetup to unlock the partition.
Here's an example of a doc I found on google:
Although this uses a custom keyscript I'd try to reuse existing scripts.
Be careful though, if you lose your thumb drive, you lose all access (make backups). Not to mention whoever finds the device will have full access to all of your devices secured in this manner.