As per schneidz...
If you make a live USB you can boot from it without disturbing your system as the BIOS, (providing it can handle USB and has the boot list configured correctly), will boot the USB before looking for a hard disk.
Just adding a USB stick as storage and installing to it is NOT
the same thing. The hard disk's MBR will be changed to point at Grub which is installed on the stick during Ubuntu installation. Removing the stick and booting will therefore try to access a non-existant file; the Grub or Grub2 boot loader which is on the stick and would be used to redirect the boot sequence back to Windows on your hard drive. (With Ubuntu as first choice/default)