You need to choose a distribution that can guaranty you to not install any graphics library as dependencies.
Unfortunatly most of the distribution do not make a clean split between cli and gui packages.
At least one does it: NuTyX, it's even one of it's target concept. Thanks to the concept of collection which in this case can guaranty you that not one single lib from xorg (for example) will be install. I suggest you to take a look at the documentation for more info (or you can register on our NuTyX forum as well)
Last edited by tnut; 01-11-2017 at 03:55 PM.
Reason: syntax correx