I suspected that would be the problem as soon as I saw the title of your question.
Incidentally you can also use "unalias ls" to unset it for that terminal. Of course it will only be temporary unless you remove or alter the line in your bash startup scripts that sets it in the first place. But I generally find it easier to just backslash-escape the command when I need to than to alter or unset it and have to reset it again later.
P.S: Another major program that has optional color support is grep.