Even though the community always rants about not using ls as input (which, in general I agree with), it does have a few nice options that can be used to your advantage.
Have a look at this:
rm -rf $(ls -1dAtr --quoting-style=shell --group-directories-first .[a-zA-Z0-9]* | head -1)
I've used A instead of a to get rid of the . and .. (just in case), the l option is replaced by 1 (no need for the long listing, 1 column works).
The quoting-style= options can has a few ways of quoting text, I'm using shell to put the output in single quotes (do have a look at the ls manual page). No need to worry about spaces and other "weird" characters.
The group-directories-first makes sure directories are listed first, this together with the r option makes sure that the first dot directory shown is the oldest.
The d option is needed to keep ls from outputting contents of the directories itself.
EDIT: My initial version didn't work, the above is tested (minimal) and seems to do the job.