ls sorts in alphebetical order, if you want to sort by modification time you need to use the "-t" flag.
Will give you the name of the oldest file, you can use xargs to pipe it to rm for removal
ls -rt | tail -1 | xargs rm -i
Just beware of files with spaces in them, this approach won't handle them properly. You'd need find with the "-print0" flag and the "-0" flag to xargs to handle files with spaces in them properly.