If I recompile Xorg , disabling its network support , will it improve performance significantly ?

Has anyone tried this ?

According to wikipedia article, it was a vnc-like method in the old days. Now that we have ssh , realvnc etc I would like to remove it.

You can't, networking is core functionality, even if you run X locally you are still using loopback network device to connect.

You can disable active listening by how you start it.

$ startx -- :0 -depth 24 -nolisten tcp

Which is the default for many distros anyway. Of course this breaks things like Xdmx, and X forwarding. Wayland is supposed to make the X model less network based. I'm not sure what mir brings to the table. Not that any lightweight window managers currently play well with such things.

As far as performance, I've found that cwm works for me. Or something like dwm. You don't really double performance, but an extra 5% is something that gamers pay thousands for. One other trick is to launch resource hogs with a high nice level from the command line.

$ nice -n 10 firefox

By default everything that launches under X runs with the same priority as X (priority/nice level 0). Which can have some performance contentions IMO. Running buggy browsers with a lower priority lets you switch to other things with a higher priority to kill the buggy browser. And helps to get you back to business quicker.

It's not about the performance, but its sure nice to have an extra boost.

Its more like removing what is not needed, trimming down. If X doesnt really need networking support, I dont see any reason to keep it for my purpose.

Also if I am correct , X just talks to the hardware and draws the windows etc via the window-manager ?(or whatever that you use)

There is no question about it needing loopback or whatever in that case , as long its simple objective is to draw stuff on screen.