Okay, I have converted a netbook into an SSH tunneling box. I setup port forwarding on my router and I have already verified that I can SSH into the netbook/server with another netbook(main) from just about any access point, and the tunneling works great. Now for my question: The netbook/server has built-in speakers and a mic. Is it possible to stream audio in both directions over SSH, so I can talk to people in my house using the encrypted tunnel? If I can't do it in both directions, can I at least turn the netbook/server into a listening device?

You can use pulseaudio for that purpose. You can actually configure pulseaudio in such manner, that you would be able to connect to it remotely from different computer. It is easily achievable by using paprefs.

No luck with pulseaudio, but I did get something out this:

Doing this "links" my microphone to the speakers on the server, but there's 1 or 2 seconds of lag. I tried to switch arecord and aplay in the above command, thinking it would turn the netbook into a listening device, but the terminal window just freezes and nothing happens. I tried arecord, and then played the file locally from an sshfs mounted directory, but it's choppy. maybe I'm going about it the wrong way. Is there any voice chat program designed to run from the console, like teamspeak? I don't want to load my server up with a whole bunch of stuff.

Maybe try NAS.
ANd just simple switching won't do what you want to achieve. You turn it upside-down: ssh to remote computer, ssh from remote computer to local computer and do exactly same as you stated above, then it work.

P.S. pulseaudio would work, if you would setup ssh-tunnel and use it for communication to-from pulseaudio. You just need make it accessible remotely and allow connecting to streams from remote computer.

Okay, I'll try the pulseaudio thing again. Does the linux distro on the server matter? It's running Arch off of a 4GB flash drive.

It doesn't matter at all, as long pulseaudio is able to another pulseaudio. I don't think they would change protocol to often.

Unfortunately, I ran into a bug with SSH on Arch Linux and I have to start from square 1 again with the server project. You guys can go ahead and close this thread (or remove it). The bug I'm talking about is where SSH freezes up for no reason after a few minutes After a lot of testing (and connecting my netbook directly to the server) I have verified that the problem is Arch. My desktop system runs Linux Mint (using the same sshd_config file!), and I can do anything through SSH (scp, tunneling, mounting, etc.) for as long as I want. So until I find another suitable distro, I don't have a server to play with