Never mind, I figured it out - thanks mainly to this post:
Basically with the problem like mine the first thing needed is to install lvm2:
sudo apt-get install lvm2
Then I plug in USB drive and in desktop environment I'm prompted for a passphrase for the just attached encrypted volume. So I type in a passphrase. Nothing happens as this is LVM volume and it needs to be mounted differently.
So I do:
Under VG column I find a name of the volume group I need.
sudo lvdisplay <volume_group_name>
Result gives me to volumes - one named 'swap' and the other - 'root'. Clearly it's the 'root' that I need to recover my data.
So the last thing I do:
sudo mount /dev/<volume_group_name>/<volume(in my case 'root')> /mnt
And there you go - I have my data back!