You have to write own scripts that do what you want.
Just put in netfilter on table nat, POSTROUTING rules that match your packets with desired IPs and input interface.
After that, make a bash script that at every 5 minutes it run:
# iptables -nvL POSTROUTING -t nat | grep IP and process the output
(type iptables -nvL POSTROUTING -t nat to see how if the output formed - 1st and 2nd field is what do you need).
After that, when the value reached 600MBytes, block his IP with iptables.