man quota
OR
Create a partition, the size you want to limit this directory to, then mount your new partition on that mount point.
For example, you want /home to not exceed 4GB, so create/resize your root partition using something like:
http://www.gnu.org/software/parted/parted.html
And then after the new partition is created:
mount -t ext3 /dev/hdx /home
You will of course need to copy everything over first, so I suggest:
mount -t ext3 /dev/hdx /mnt/temp
Then:
cp -a /home /mnt/temp
And finally:
umount /mnt/temp
telinit 1
rm -r /home (be ultra sure you want this, and be VERY VERY careful... I don't suggest this way FYI)
mount -t ext3 /dev/hdx /home
If it works, make changes to /etc/fstab:
/dev/hdx /home ext3 defaults 1 1
And finally:
telinit 3 (or 5 depending on your needs, again, if you understand this, go ahead, but BE very careful, if you don't understand it, don't do it

use quotes instead as it's more 'proper' for your needs most likely).
Cool