I'm just guessing from memory:
Does gvim have a menu to change the font? If so you could note the command it issues when the menu is used and try it in vim.
Another possibility is the font X uses can be changed (?by using an xset command in .xsession or .xinitrc?) then vim might use that font inside X.
Hope some help.