Java is closed source, right, but there are licensees who are entitled to use and modify it and to redistribute the result --- as a binary, as you say. Karl Blackdown and his team regularly start with a given version from Sun and add improvements especially for Linux. In fact, Blackdown was the first Java ever for Linux, quite a while before Sun supported our favourite OS.
BTW, there are other vendors and VMs. Eg, IBM and BEA are licensees having the right to produce and distribute or sell their very own Java VMs.
There are also clean-room implementations, that were created without any support from Sun, thus avoiding the need of acquiring a licence. Kaffe is among the most advanced clean-room VMs, to my knowledge. In fact, there are a two or three clean-room implementations that are also open source, but none of these is up the VM from Sun in features. Some of them, however, are much better suited than Sun's big ship for specialised applications like embedded software.
So you are right, the Java is closed source, but there is something like open source Java, and Sun isn't the only vendor, although they do all the specification work, with support from many others in the Java Community Process (JCP).
Coming back to the original subject of the thread: Not sure if Sun or Blackdown or IBM is the right source, but one of them has binaries compiled with more than one glibc version (probably using GCC). It is vital that you get the VM that matches your libraries, otherwise you will experience all kinds of strange effects.