I'm currently working on an HPC cluster for arbitrary reasons, but I'm having trouble deciding on what distributed filesystem to use.

There are so many options it's a bit overwhelming (ceph, glusterfs, lustre, beegfs, etc.), and it's hard to find conclusive comparisons between them, especially for HPC environments as most seem to focus on HA.

In my setup, I plan to use the filesystem in parallel with each node acting as both a host (contributing to the storage pool) and client (connecting to the pool). I need whatever I choose to run on Ubuntu's current LTS release, and preferably to support client side caching (RDMA support is also a welcome inclusion).

Thank you in advance for your input.

Size of data? Performance requirements? Number of nodes? Reliability requirements?

The size and performance requirements are going to be somewhat arbitrary.
I don't have a specific use in mind, so the needs may vary. Currently I'm mostly thinking on AI, so likely it'll see a lot of use in storing large sets of various data for sorting, or for training neural networks, but that's just one use-case.
Obviously I plan to store as much of my datasets as possible locally on each machine, and I know that, practically speaking, a network filesystem, distributed or otherwise, is going to be slow (which is part of why having decent client-side caching would be nice).
Honestly, I'm not going to get to picky about performance in this case. If it can even partially scale up when I fully move over to InfiniBand, then I'll be satisfied for now. I can switch to dedicated I/O severs in a future revision. So, I guess, whatever gives the best general performance across a wide variety of scenarios.

As for the number of nodes, currently I have four in total including the head and workers (there was going to be five, but I had some bad hardware and had to make due). I do plan to expand in the future, perhaps to as many as a dozen or more workers, so being able to scale for that will likely be important (Though at that point, I'll likely already be using dedicated I/O servers).

And for reliability...
Well, that's not too big of a concern, especially with only four working nodes. Having the data still be accessible if one of them fails (or simply decides not to turn on when it should) would be nice though. I also imagine that, as I scale up, that will likely become more useful. So again, having the option would be good, but I'd likely give up some reliability and availability in exchange for a significant performance boost.
I have plenty of places to store important data. This is really just to act as a simple way to distribute and hold large data, anything critical can go, or be copied, elsewhere.

If it can even partially scale up when I fully move over to InfiniBand, then I'll be satisfied for now. I can switch to dedicated I/O severs in mcdvoice survey a future revision. So, I guess, whatever gives the best general performance across a wide variety of scenarios.