You need to get keychain in order to manage ssh-manager and make sure that every ssh process can pick up the proper ssh-manager session.
http://www.gentoo.org/proj/en/keychain/
Please note that keychain may want to put your session credentials in a subdirectory, then look for them in the main directory. I handled this with a symlink.
I am doing the exact same thing you are doing, remotely managing/reconfiguring linux machines via a scriptfrom a server. I question that you can't have scripts on your client machine and run them; seems to me that if you have SSH access you certainly can put a script on the client, run it, then delete it. This is what I do.
By doing it that way, I eliminate the problem you are having, of multiple password entries. I upload the script, run it sudo exactly once via SSH, and it gathers all data/does all work. I enter the password once. When I am all done, I delete the script.