Happy to hear you use Debian!
Are you familiar with dselect, the package management system?
By using it, you can select the package you want (ie: mozilla) and it will resolve 'dependancy' issues for you by suggesting additional required packages.
In your case, you could select just the packages you need for a base x-windows system without the xserver itself.
dselect can be intimidating for new users so I'd suggest taking a look at dselect Documentation for Beginners
WHILE you have dselect open.