Yes. There are two ways to do that. One is to copy the entire raw USB drive into a file, and start the virtual machine with that file as the drive to boot from. Changes would have to be copied back to the USB drive if you want them there. The other (easier) way is to attach the USB drive to the virtual machine, either as a drive, or as a USB device.
I have done this with SD cards (because good SD cards that are faster than USB memory sticks are available).