I recently noticed that my Postfix email server has a few people who never clean up their mailboxes. I have IMAP only so all email remains on the server in their home directory via Maildir style.

/home/$user/Maildir

My question is how does Postfix handle email delivery if I set a user quota and that quota has been exceeded. Obviously Postfix can't deliver any messages to your home directory so what is the process from there? Does Postfix reject the message back to the sender as undeliverable or does it re-attempt at a later time?

I doubt it makes a difference but I set each users home directory to 2GB.

It will probably not work with quota if exceeded, be careful. You should set quota with your imap server, not with your smtp server. Try cyrus imap, which has excellent email quota capabilities that work with web email, outlook express and others.

I was setting disk quotas with the Linux file system - not the MTA. What is the difference between setting up a MDA and a Linux system wide quota?

If postfix cannot save to the directory because of quota then it will crash or not deliver the message. Postfix is an SMTP Server and SMTP doesn't care about quotas. All it cares about is delivering messages from one SMTP server to another - that's it. Your MDA is what controls the quota; it should still get the message, warn the user and prevent the user from sending any more messages until he/she reduces the quota. Cyrus IMAP does all that very well. Dovecot was trying to implement quotas but it is not so good. Linux file system quotas are only for file systems, not email systems.