Linux tend to be worldwide adopted. Thus you will find a lot of fonts for as many languages as possible, including non-European ones, which IMHO is a very good thing.
But may be I didn't get you well. Do you think that disabling fonts that you don't use will speed up your system ? I don't believe that you will see any improvement in doing so.
Would it be the case, all you can do is try to comment out some FontPaths entries in the "Files" section of your /etc/X11/xorg.conf but I doubt it will noticeably speed up X starting.