Not sure what you are looking for here.
Are you looking for security or routing?
There are things like vyatta:
http://www.brocade.com/forms/jsp/vya...8qHRoCj7bw_wcB
(Recently purchased by Brocade, still free but you have to deal with sales calls and other BS from them)
IPcop:
http://www.ipcop.org/
You can also use something like FireStarter to create a GUI for your IPtables service:
http://www.tecmint.com/firestarter-a...linux-systems/
And others.
Take a look at:
http://www.tecmint.com/open-source-s...linux-systems/
Also, I am curious why you are looking at application layer solutions for firewalls? Most companies implement network security lower on the OSI model by using hardware based firewalls, which would be my recommendation.
mod_security is NOT an effective firewall solution aside from HTTP