If you want "lean and mean", it's often better to install such a distro---rather than tweaking a "full-featured" one like Ubuntu.
Examples include Arch, DSL, Slackware and others.
This said, you can start by removing unwanted stuff using the package manager. It will keep track of dependencies and warn you about the side-effects of removing a particular package. Beyond this, just deleting files is risky.
It's much easier to start lean and then grow fatter with time. Computers and Humans have some commonality here....
