--syn means a SYN packet (--tcp-flags SYN,ACK,RST SYN)
You know SYNs create new TCP connections...
--state NEW may apply to non-SYN packets...
ie, non-SYN packets may create NEW entries in netfilter's state table
So you must make sure that only SYN's create NEW entries in the state table
Some scanners use SYN/FIN's for portscanning and OS fingerprinting.
These 3 rules alone protect against all invalid combinations:
iptables -A tcp-in -p tcp --tcp-flags SYN,FIN SYN,FIN -j DROP
iptables -A tcp-in -p tcp -m state --state NEW ! --syn -j DROP
iptables -A tcp-in -p tcp -m state --state INVALID -j DROP
Note: We SYN/FIN rule before the check for --syn. There was a bug in the iptables manpage that stated that --syn "only match TCP packets with the SYN bit set and the ACK and FIN bits cleared", which isn't true.