512MB RAM should be more than enough.
Two things to look at:
1. See if you've got a header with a lot of in-line functions (they can be a real pig).. If so, see if you can reduce the #/in-line functions (or at least segregate them to a separate module)
2. Consider rebuilding your GCC with "--with-low-memory":
'Hope that helps .. PSM