Depending on the amount of RAM you have it's quite common that it fills up. Only on machines with very large amounts of RAM (> 1 GB) the kernel has enough space not to need the swap area. Using the swap on your harddrive does affect performance, but not significantly. Dunno what RAM you have, but it might be a good idea to add another bank

Use the ps and top commands to find resource-hungry processes.