Hmm. You might want to check the permissions of the global config file (/etc/vimrc or whatever it is on your system).
I have it set up like this - I have a ~/.vimrc that is sourced by ~/.gvimrc so that I keep all the common options in ~/.vimrc and the ones specific to the graphical interface in ~/.gvimrc. This way vim behaves like I expect it to whether it's in graphical mode or not.
My current ~/.vimrc on my laptop:
let g:miniBufExplMapWindowNavVim = 1
let g:miniBufExplMapWindowNavArrows = 1
let g:miniBufExplMapCTabSwitchBuffs = 1
let g:miniBufExplModSelTarget = 1
...and my simple ~/.gvimrc:
set guifont=ProFont\ 11