It depends on which "speed" you mean:
dnetc or 3D games will certainly not be faster; however, if you are running several servers with hundreds of threads, you might probably see an increase in performance.
And of course, like XavierP already said, an operating system that doesn't waste dozens of MiB RAM for the GUI and significant amounts of CPU time for stupid menu animations that no one needs
will be faster
