are both of those IPs on the same NIC?? or do you have two NICs??
cuz my best guess is that sshd just binds to the interface on which the given IP is... if that's the case, and you only have one NIC, then that might explain why you are getting this behavior... you could always forget about the "ListenAddress" thing and instead filter the packets using iptables...
just my

...