hi there.
you probably want to read through
http://l7-filter.sourceforge.net/
these people have built application layer filtering (thus l7).
they support regexp-patterns and guides for writing/modifying them.
afaik, they provide support for 55 protocols.
perhaps you should have a look at their project-page. its quite clear how l7-filtering works after reading through their documentation.
they are currently discussing with netfilter to be included within iptables and the linux stock kernel.