Postfix is a mail transfer agent (MTA) and you need an MTA of some sort, and since you have postfix configured, you may as well stick with it.
Mailman will do what you suggest. You can change the sender to be the poster (see general options - make replies to Poster). You can restrict members' ability to use the list by requiring all but wanted senders to be moderated, and discarding moderated posts.
Not sure about the error message, since we don't know how you've set postfix up. If you aren't using virtual users, do you have a user "test" on your system?