Turns out that the button on that mouse was just squirrely. I tried several
settings, but none made it any better.
It was a noname mouse, that works fine in the windows world.
I had an extra Intellimouse that connected and it works fine.
Weird... when I launched gnome-mouse-porperties I get an audio_alsa error. But, I'm not having any sound problems.