When you say directly connect, do you mean that you are using a cross-over ethernet cable and trying to ssh from the laptop in to the server?
Ensure first that your cable is a cross-over. The laptop and server will need IP address on the same subnet. Below is a link on how to configure:
Just in case SSH isn't installed by default you can install it by running 'sudo apt-get install openssh-server'. The catch-22 being ofcourse you will need to be connected to the internet before this will work. Hope this provided some steps to get started, if I'm way off in left field, please elaborate a bit on how you are needing to connect and what you are doing.