I solved the initial zlib problem by looking in the config.log, of course after posting though
Apparently it was trying to find zlib.h, but for some reason mine was in include/linux instead of X11RS/... So I just made a link to it. Hopefully that won't cause any issues.
But now more problems ...
Apparently I'm missing confdefs.h. So now I get to find it, then put it somewhere.
Is there a catch-all development package I should look at getting?