I realize this thread is a bit stale and the problem is already solved, but I had this same (at least, similar) problem with my homemade router, and I came upon this thread.
I too thought it was a mis-configured iptables that was somehow not letting the right packets through for Google Hangouts. And it was just on my side... I could receive audio and video just fine from others, but they only got audio from me. Strangely enough, this same effect occurs with Skype and Facetime. All of my Steam-based games could not connect to any remote server whatsoever. I had used iptable configurations from https://help.ubuntu.com/community/Ro...d_Masquerading
and from https://wiki.archlinux.org/index.php...single_machine
but still no luck.
Finally I broke down and used Wireshark. I captured packets when connected to a "good" router that was able to connect to a Steam server, and then I captured packets when connected to my wonky router, and compared the results.
It turns out that my wonky router was getting several ICMP "Time-to-live exceeded (Fragment reassembly time exceeded)" packets returned whenever it tried to connect. Following that lead, I ran this command on my router to find out the info on my network adapters:
Of the adapters listed, it was my WAN-facing adapter that was the problem: It's MTU was set to 576 somehow. It should be 1500, like the rest of the Internet. I changed it to what it should be with:
ifconfig eth0 mtu 1500 up
Then I crossed my fingers and tried connecting to a Steam-server using the wonky router... It worked! So did Google Hangouts and Facetime! (I haven't tried Skype yet though but it is promising.) The above command won't keep the MTU at 1500 between reboots, so I had to add an entry to my config (searching for linux permanently change MTU does the trick.)
So, as far as I could tell, my router was seeing several "large" packets come through, so it broke them up into smaller chunks and caused all sorts of commotion.