I have played with a bunch of different distros in the past month and I have noticed differences in response in the terminal emulator programs. In particular when trying to ssh into my server. At one point I was running Fedora 20 on my desktop and with the default terminal emulator it seemed like when I ssh I would get prompted for password almost immediately and then be logged in as soon as I typed the credentials.

Now I am running Debian Wheezy with xfce desktop and using the stock Terminal Emulator. When I ssh now it seems like it takes nearly 30 seconds to make the connection before I get prompted for password and then takes some more waiting.

What causes the variance in performance I'm seeing? Is it the distro, the desktop environment, the terminal emulator or a combination?

Probably none of the above; if you're running ssh, I suspect it's network related. When you make a ssh connection to a server, the server will try to do a reverse DNS lookup on the client's IP address. It will wait until this reverse lookup succeeds (or times out) before presenting an authentication prompt. Are you changing your IP address as you're trying all of these different distros? If so, have you made sure reverse look-ups work correctly.

You can use one or more -v (for verbose) flags as part of your ssh command to get a better idea of what's going on. You can also check the ssh server logs.

I have noticed slowdowns in XFCE Terminal as well. I just switched to another terminal and everything is better.

The one I use, with similar features:
http://lilyterm.luna.com.tw/html/download.html

Interesting, what are the specs of your computer? Some terminal emulators and desktop environments are much more resource-intensive than others. That could be part of the issue if you have an older machine.

The word for it is bloated.

@Ryanms3030
Get a lighter terminal something like rxvt-unicode-lite or evilvte, here a nice trick to run rxvt as daemon:
then run the client urxvtc, it uses a lot less memory and is much faster.

Cheers

Thanks everyone. I'm testing lilyterm now and it's still slow connecting to ssh. I guess it's network related. The things that have changed are that I set up the router to reserve my server's ip (using dhcp on the server) and I also set up port forwarding for ssh so I can ssh from outside the firewall.

I have been noticing some other weirdness lately. I will have to restart my iptables and sshd service for apparently no reason to be able to ssh in sometimes which kind of defeats the purpose since I have to do that locally on the server since it prevents me from ssh.

Usually nameserver related. Most of the times i had this, the server could not find the client on the ns for reverse lookup. Maybe fedora handles your dhcp differently?