You'll have to be more specific. In Linux, there are 5 ways to do graphics, as far as I know:
- ASCII art on the console,
- framebuffer graphics on the console,
- SVGAlib on the console (deprecated AFAIK),
- more or less experimental alternatives to X11.
For serious work with good graphics, I only see two of those as valid: framebuffer and X11. I don't know framebuffer, nor do I know if Java can interact with it (there's always JNI, though). So I (wrongly?) assumed that you would use X11.
X11 can be very small if you really want, so system resources are not really a problem. The smallest X11 probably is kdrive (not to be confused with KDE application "kdrive"...), formerly known as tinyX. But I'm not sure you'll get enough performance out of it: it all depends on how much you depend on hardware acceleration.
If you need hardware acceleration, or other fancy things, you're probably better off with X.Org or XFree86. It is not that much bigger, actually. I run XFree86 on a P150MMX with 32MB RAM. You probably have better than that
Just don't forget to optimize its configuration for best performance:
- use a font server... or not;
- don't load modules that you don't need;
Now the window manager. Indeed you don't need a window manager. But some applications can't be told to open at the wanted location with the wanted size (eg: 800x600+0+0); in such case, Matchbox is usefull, as it is lightweight and opens all windows in fullscreen mode (provided you don't use side-panels).
If you adapt /etc/inittab as told on my site, last line could look like that:
X:5:respawn:/bin/su - UTIL -c '/usr/local/bin/startx /usr/local/bin/java+app.sh'
with "java+app.sh" containing the script that I wrote above.
All this actually depends on what exactly it is you want to do.