Like any other operating system, linux speed is dependent on a number of different things.
First and foremost, linux depends on what services are running on your machine. If you check the output of ps aux you should be able to see all the things running. You should disable anything you don't need.
Additionally, depending on the type of hard disk, you might need to tune the performance of the disk to increase throughput using hdparm.
Also, if you're trying to run the latest and greatest graphics effects like Compiz or KDE4 composting, your graphics card may not be able to handle it without 256MB GPU RAM.