(1) It's normal for memory usage to steadily increase, since Linux does quite a lot of caching to improve performance. if a new app needs memory, the OS willrelease some of the cache space.
(2) I'm not sure exactly, but you can enable filesystem quotas (make sure you have the quota package installed, add usrquota to the appropriate filesystem in /etc/fstab, and remount) and use them. You may need to enable something in sendmail to respect the quotas. There's an excellent quota howto at
http://www.tldp.org/HOWTO/Quota.html if you need more details.
