Actually using tethereal but the filters are the same format. I'm try to setup a
filter that only me gives packets with syn and fin without ack flags from foreign
host. Here's what I've come up with:

tethereal -f "src host not localhost and tcp[tcpflags] & (tcp-syn|tcp-fin) != 0 and
tcp[tcpflags] & (tcp-ack) != 1"

This doesn't seem to work however, I'm getting packets captured on various
conditions other than expected condition I tried to set. Looking for a little help.

Thanks

This post is very old, but still actual, so I am going to answer it.

The problem in the specified filter is that the expression

tcp[tcpflags] & (tcp-ack) != 1

is always true, since tcp[tcpflags] & (tcp-ack) can only be 0 or 16. This is because the value of the tcp-ack flag is 16. So, to skip the packets including ACK, the filter should be this:

src host not localhost and tcp[tcpflags] & (tcp-syn|tcp-fin) != 0 and tcp[tcpflags] & (tcp-ack) != 16

The values of the different flags are: UAPRSF
URG: 32
ACK: 16
PSH: 8
RST: 4
SYN: 2
FIN: 1

Also to do that in tcpdump you have to use the following filter expression (include the quotes):

'src host not localhost and tcp[13]&3!=0 and tcp[13]&16!=16'