However in vi, it behaves as a left arrow - it goes to the left without erasing anything.
Are you asking about vi insert mode or command mode ?
In command mode, it looks to me the expected backspace behaviour.
In insert mode, it erases the character when they have just been inserted previously in, but only show they have been deleted when insert mode is left with <Escape>.
In cat and my own shell scripts, Backspace just keeps writing ^Hs on the screen. It is the same at the login screen.
^H is backspace character visual representation when not catched by the driver.
As you already found,
stty erase <Backspace>
is the way to have the driver interpret it as an instruction to erase the previous character.
As of changing system key mapping, should you really want this odd situation commonly found with linux where the backspace key send delete and delete send backspace, or whatever, it can be done by playing with loadkeys.
About the visual bell issue, I do not experience it.
What is your TERM variable value when in console ?
What shell are your running ?