I have two interfaces in my box:

eth0 Link encap:Ethernet HWaddr 00:10:4B:9E:1F:E7
inet addr:172.16.0.8 Bcast:172.16.0.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

eth1 Link encap:Ethernet HWaddr 00:50:dA:66:0d:d5
inet addr:192.168.0.1 Bcast:192.168.0.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

The second interface is used to monitor the network with tcpdump, I want it to receive any traffic sent to it, but not send any packets out.

When one of my machines on the network tried to connect to smb server running on 172.16.0.8, the first thing the client does is send arp request to learn the MAC of 172.16.0.8. Why the heck does ot get response through both interfaces? It should only hear back from 00:10:4B:9E:1F:E7

15:08:02.855604 0:50:4:4:22:5 Broadcast arp 60: arp who-has 172.16.0.8 tell 172.16.0.3
15:08:02.855691 0:50:da:66:d:d5 0:50:4:4:22:5 arp 42: arp reply 172.16.0.8 is-at 0:50:da:66:d:d5
15:08:02.855679 0:10:4b:9e:1f:e7 0:50:4:4:22:5 arp 60: arp reply 172.16.0.8 is-at 0:10:4b:9e:1f:e7
15:08:02.855722 0:50:da:66:d:d5 0:50:4:4:22:5 arp 60: arp reply 172.16.0.8 is-at 0:50:da:66:d:d5

/proc/sys/net/ipv4/ip_forward is 0, there's nothing in my iptables indication that I want eth1 to respond to arps for 172.16.0.0/24

How do I stop eth1 from responding to arps for network it does not belong to?

If you're only running tcpdump on the second interface, don't give it an IP.

Tried that, but then tcpdump complains, that the interface has not been configured. Is there a way to bring up an interface without giving it an IP?

Sure, you should be able to just
# ifconfig eth1 up

If you already configured it with an IP, and ifconfig eth1 down didn't erase it (i.e. next time you "up" it, it goes back to the previous IP), you will need to use the ifconfig man page to determine what the correct option is to delete an IP from an interface. You should also make sure that the broadcast address was removed as well (which should happen automatically when you remove the IP).