Backup your data - and reinstall ubuntu while the pendrive is plugged in.
One of the partitioning options is to use whole disk encryption, and put the /boot partition on the pendrive. Ubuntu installer does the rest.
However - unlike some proprietary offerings, you are quite safe just encrypting the root partition (seperate boot partition). /boot contains only static files and nothing in going to get written to this. you can just keep your keyring on the - ahem - key.
Bear in mind that it is very difficult to protect against someone with physical access to the machine.