I have 3 systems which are showing following behavior

1) A regularly scheduled process goes into un-interruptible sleep.
2) Subsequent launches of same process stay in "R" state but are not doing anything.
3) Load average starts growing.
4) System becomes unresponsive and needs to be rebooted.

I need help in trying to find what the resource is the process going into uninterruptible sleep on.

I cannot attach debugger, run strace or kill it. So I don't have visibility what the process is doing. Logs don't indicate any crashes or pending operations.

Any help is much appreciated.

So you've deliberately not given us any information that might assist us in helping you (what product/process, what distro, loadavg when things slow down ...) , and you want unpaid help ?.
Hmmm.
Is it a commercial product ?.

If you are unable (not allowed ?) to use diagnostic tooling, how are we to help ?.
Uninterruptable sleep is a scheduler state imposed on a process by some other entity - interrupt handler/driver/parent or the like. Should conceptually only be done by code in interrupt context, but things like Apache and Oracle have abused this for their own end. And screwed it up royally by (not so) occasionally forgetting about their comatose children. There is no necessary correlation between a "D" state and a resource wait - the generally accepted case is disk I/O wait, but the code that imposes the sleep gets to decide what/when it awakes the sleeper. No-one else gets a say in things.
How the later instances get queued is for the vendor/programmer to answer.