I am trying to do something that in theory should be extremely easy but doesn't seem to work for me. I want to take a port that has mirrored traffic (used for my web-filtering product) and eliminate all of the traffic from networks that are not filtered. I want to do this by using Citrix XenServer and the iptables built into the server and DROP the packets that do not match what I want to accept. I have attached a basic design.

Simple explanation:

Traffic comes across span, hits the physical server, iptables filters off traffic, remaining traffic is visible to Windows Virtual servers.

Network Design

Looks like you're trying to set up an IDS, can you post the rules you have so far ?

# Generated by iptables-save v1.3.5 on Tue Dec 8 14:24:06 2009
*raw
:PREROUTING ACCEPT [776:101474]
:OUTPUT ACCEPT [424:50996]
#:Webfilter_IDS - [0:0]
#-A PREROUTING -s 172.24.0.0/255.255.0.0 -i eth5 -j Webfilter_IDS
#-A Webfitler_IDS -s 172.24.253.0/255.255.255.0 -i eth5 -j DROP
COMMIT
# Completed on Tue Dec 8 14:24:06 2009
# Generated by iptables-save v1.3.5 on Tue Dec 8 14:24:06 2009
*mangle
:PREROUTING ACCEPT [2127:303772]
:INPUT ACCEPT [1681:217868]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [1156:171954]
:POSTROUTING ACCEPT [1156:171954]
#:Webfilter_IDS - [0:0]
#-A PREROUTING -s 172.24.0.0/255.255.0.0 -i eth5 -j Webfilter_IDS
#-A Webfilter_IDS -s 172.24.253.0/255.255.255.0 -i eth5 -j DROP
COMMIT
# Completed on Tue Dec 8 14:24:06 2009
# Generated by iptables-save v1.3.5 on Tue Dec 8 14:24:06 2009
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [1251:184611]
:RH-Firewall-1-INPUT - [0:0]
#:Webfilter_IDS - [0:0]
-A INPUT -j RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT
#-A RH-Firewall-1-INPUT -s 172.24.0.0/255.255.0.0 -i eth5 -j Webfilter_IDS
-A RH-Firewall-1-INPUT -s 172.24.0.0/16 -i eth5 -j DROP
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
-A RH-Firewall-1-INPUT -p icmp -m icmp --icmp-type any -j ACCEPT
-A RH-Firewall-1-INPUT -p esp -j ACCEPT
-A RH-Firewall-1-INPUT -p ah -j ACCEPT
-A RH-Firewall-1-INPUT -d 224.0.0.251 -p udp -m udp --dport 5353 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp -m udp --dport 631 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m tcp --dport 631 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A RH-Firewall-1-INPUT -p udp -m state --state NEW -m udp --dport 694 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 443 -j ACCEPT
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited
#-A Webfilter_IDS -s 172.24.253.0/255.255.255.0 -i eth5 -j DROP
COMMIT
# Completed on Tue Dec 8 14:24:06 2009

On a span port you probably don't want to reject any traffic as you would be interfering with the traffic you're mirroring, ( the source may get confused if it sees an ACK from the original destination as well as an ICMP host-prohibited from your XenServer for packets it sends).
Try changing the last REJECT to a DROP and see how that goes, if you could give an example of traffic that is getting through that you've denied that would be great

cheers