I'm doing a small install, no X etc
Since you will not be installing X, you may do without swap.
My system uses swap on regular intervals, but I use X and a lot of graphical applications which consume memory.
Remember, graphical environments are memory hogs while command line programs are lightweigh.
But if you plan on only using the command line and having very few active services, large amounts of SWAP will go to waste.
But you may want a safety net just in case.
If possible, create a small swap... 32 MB for example.