X definitely uses more resources than console mode, since there is all that graphical stuff it has to do (not to mention the X server that has to run, since X is designed with networking in mind). There are varying degrees of memory and CPU-intensiveness with X, though, depending on what window manager or desktop environment you use. Fluxbox or Blackbox, for example, will use less resources and will in general be faster than Gnome or KDE. There are window managers for just about every need. So obviously, if you have an older machine that has trouble with Gnome, you could try a more light-weight window manager to speed things up a bit (and use fewer resources).
As for stability - I don't think there's really that much difference. Of course, the more programs you have running, the greater the chance that something will crash, but the nice thing about the way Linux and X are designed, if X crashes, it's not going to take your whole machine down with it. So even if you're running the most unstable alpha-versioned window manager out there, Linux itself should still be perfectly stable, since the kernel is independent of the window environment (unlike the OSes distributed by some other vendors
)
In short - use X and a window environment if you need or want one, but it's not necessary. For example, if you have a machine that's acting as a firewall, or server, and it isn't doing anything else, then there's not really any need to use X on it.