You haven't really provided enough information, but I would guess you have a 32-bit program that is trying to use more memory than is possible for a 32-bit program.
Could there be something wrong with the inputs you gave the program that cause it to use excess memory?
Do you have source code for the program and access to a 64-bit Linux, so you could let the program use more memory?
Edit: I see you previously asked the same question in a bioinformatics forum and provided a little more info and got the answer:
Pay lots of $$$ for the 64 bit version of USEARCH or split the input and reference data sets.
So you thought asking the same question in a less on-topic forum and providing less information would get you a better answer !?
But there is a small chance I do have a better answer. I expect you are using that 32-bit program on a 32-bit Linux. Can you switch to 64-bit Linux? Even if you can't get the 64-bit version of the program, the 32-bit program will have 30% more capacity when running on 64-bit Linux. Probably the problem you have given it is more than 30% bigger than the capacity it has now, so with 30% extra, it will still fail. But there is a small but significant chance the problem size is in between the limit for a 32-bit program on 32-bit Linux and the larger limit for a 32-bit program on 64-bit Linux.
Edit2: The publisher of the program provided instructions for estimating and reducing the memory requirements of your problem:
Those don't make any sense to me because I have no clue what this program even does. If those suggestions make sense to you try them. If they don't make sense to you, perhaps you are attempting a project within a topic you don't know enough about.
I suspect the free 32-bit version of the program may be a toy unsuitable for real problems. I don't see any place the publisher clearly says this, but I get the idea that real problems require extreme amounts of memory even for 64-bit systems.