Found it!
For those of you that ever have this problem, here is the solution. Apparenly xxd -r does interesting things when inside vi and not in binary mode. To avoid this, put the following in your .vimrc, and open the file in binary mode: vim -b
" vim -b : edit binary using xxd-format!
augroup Binary
au!
au BufReadPre *.bin let &bin=1
au BufReadPost *.bin if &bin | %!xxd
au BufReadPost *.bin set ft=xxd | endif
au BufWritePre *.bin if &bin | %!xxd -r
au BufWritePre *.bin endif
au BufWritePost *.bin if &bin | %!xxd
au BufWritePost *.bin set nomod | endif
augroup END
Using this, Vim can be used as a pretty cool hex editor. Happy editing!!!