There are two ways to delete, "Backspace" and "Delete". Unfortunately, there is really no agreement on which is the "real" delete, and so some programs are set up so that "backspace" erases a character and moves the cursor left, while the other just spits out "^H"; but then other programs have it the other way around. Some programs try to help by automatically swapping the two, but it only confuses the issue. :P If you think this is dumb, you're not alone.
Anyway, there are other ways to get the backspace that you want. Try Shift-Bksp, or Ctrl-Bksp, or using the Delete key or whatever else might be available.
As for how to set it so that all programs are consistent --well, good luck.