Does the directory name have any special characters/symbols or spaces. If so, you need to escape the special character with a back slash or quotes. For example if the directory is called My Music and is empty, you can delete it this way:
#The back slash escapes the space character in My Music
#The quotes here also escapes the space character
Have you tried running the sync command after umounting the device? The sync command updates the filesystem.
Have you tried deleting the directory from the file manager?
Is the directory immutable? If a file or directory has the immutable flag set, not even root can delete or modify it unless the root user removes the immutable flag.