Try XF86config or xorgconfig (case sensitive, not sure i got the case right) from a virtual terminal (ctrl+alt+f1) as root.
This will launch a configuration program which asks you about your mouse, keyboard, video, etc. there is an option is the config for the amount of video ram