You've got tons of choices:
However, you might be better off finding an open source audio environment you like (rather than limiting your search to libraries that happen to support C++ - which isn't necessarily the best choice for a lot of things you might like to do). For example, maybe you'll find a sound editing program ... that's best used with its own scripting language. Or Perl, or Python (instead of C or C++).
Similarly, I get this impression your "algorithm" might be tied to C/C++ implementation details (for example, when you say "arrays", I'm guessing you might literally mean "c/C++ arrays of data"). You might well find that one or another library pretty much forces you to use it's own data structures - and you might also find this works out *better* for your end-goal (implementing your ideas in code, and listening to the results)!
Anyway - there are *lots* of choices. Experiment, keep an open mind ... and see what works best for you!
IMHO .. PSM