As far as I know there is no package that comes with RHEL4 to quota mail specifically. You can however use the Linux user quota system compiled into the kernel.
Postfix (like sendmail) stores new mail in the /var/spool/mail/ directory. Once the user has read the message, this is transferred either to the $HOME/mbox file or the $HOME/Mail/ directory. So you would get the best results by keeping /home on a separate partition and implementing your quota there. These are the steps to setup a quota (assuming /dev/hda2 is your /home pratition):
i) vi /etc/fstab and add the 'usrquota' option to the /home partition's line:
/dev/hda2 /home ext3 usrquota,default 0 2
ii) quotacheck -c /home - creates the quota database for the /home partition.
iii) edquota -u <username> - set the quota sizes for individual users. In your case you would probably want to set a 'hard' quota on block sizes. Sizes are specified in kb by default.
iv) quotaon /home - activates the quota
The next time you reboot your server the quota will be activated automatically.
'What will I do if the "home" partition is filled up with email' - keep your /home partition on an LVM so that you will be able to expand its size easily later on. You may also consider using the 'tmpwatch' command do automatically delete old mail after a given interval.
Alternatively you may want to visit
http://www.sourceforge.net or
http://www.freshmeat.net for applications specific for mail quotas.
Hope this helps.