first off, you can try the protocol on "auto", but that's probably not the issue.
the problem is most likely that your xorg.conf file is pointing to the wrong device.
There should be a line somthine like
Option "Device" "/dev/input/mouse0"
If you type 'cat /dev/input/mouse0'
replacing that of course with what your xorg.conf file reads and move your mouse you should see a bunch of jiberish on the screen.
Now press Ctrl+C
If your console is permanently stuck with funny characters, type 'reset' and press enter. You won't actually see reset when you type it but it'll still work
If you don't see any jiberish you've got the wrong device node and need to try another one, maybe /dev/mouse or /dev/input/mice
If you do see jiberish, then i don't know what the problem could be