I have a server, call it Host, which communicates to a device under development. That device runs linux and has its own ip address; when it boots, Host uses "ssh-add" to add a provided id_rsa, then it can scp a directory to the device without being prompted for a password. This is great for setting up automated scripts on Host to boot Device, scp executables to Device, and then run tests which synchronize via ssh.

CI <--> Host <--> Device

I have a server running a Continuous Integration service, call it CI, and I'd like to have the CI tool run those automated tests. I've set up rsa keys such that, when I'm logged into CI, I can ssh and scp to Host without being prompted for a password. However, when I'm logged into CI and ssh into Host, and then try to invoke those same scripts which work fine when I've logged directly onto Host, I'm prompted for the password to the device.

If, from CI, I try to invoke Host's script with "ssh Host script.sh", I get "Permission denied, please try again." (twice) and "Permission denied (publickey, password).

I'd like to understand why CI ssh'ing to Host cannot then scp to Device, while Host can scp to Device without problems. Even better, I want to know how to fix it so the continuous integration server can remotely run those scripts which scp to Device.

I figured CI needs Device's rsa key. I tried to scp that provided id_rsa to server CI and add it to CI, but "ssh-add id_rsa" gets "Could not open a connection to your authentication agent."

Many thanks.

I don't usually use ssh-agent/ssh-add, you could try 'ssh-copy-id' - this will deploy the keys to the target host. You would run it once from [CI] -> [host], then from [host] -> [device]

hth

Questions:

- The "device under development" contains two authorized keys - one from the host and a second from the CI? Then it might be necessary to add "-oForwardAgent=yes" to use your local ssh-agent on the CI.

- If you want to use an already running ssh-agent on the host instead, it must be told to the login session to use it by setting the appropriate SSH_AUTH_SOCK of the already running ssh-agent on the host.

One reference I really like about agent forwarding.

The Device contains one key: its own private key. Host has Device's public key, and the test can do everything it needs to do from a script on Host with that setup.

Yes, this is what I want to do. The ssh session between Host and Device is left up and running.

Thank you for that link; the visuals help.

I've solved it with some more help. Here's what works:
1) On the Host server, create a file in my user's home directory containing:

Then from CI, invoke
I may not be precisely correct, but I believe "eval" says to run through the command line evaluation process for ssh-agent, thus re-evaluating the rsa keys held in the agent on Host, and at the same time adds the key to that agent. The ssh command from CI evaluates that file before executing my test-kickoff script. If you have any clarification to add, please do. Thank you!

eval will interpret the output of ssh-agent as it would have been typed directly on the command line. In the essence it will set two environment variables which will point to the used socket in /tmp and the ssh-agent's pid (just run ssh-agent it without eval and you could copy & paste its output).

Then the key is added by ssh-add to this running agent.


When I look into the script, it might be the case that you will have many ssh-agents running at the same time as they are never stopped again, one for each login. You can check with:If this is the case, we can look into limiting it to one, which is always running.

You are quite right. How do I limit it to one agent? That sounds cleaner than figuring out which agent to kill at the end.

We need a short helper script:When you look it up with Google by some of the variable names used there, there are plenty variations of such a script. This one originated from one of them but I extended it to allow ssh-agents of more than one user on a machine (like on a server). Therefore I check whether there is already an ssh-agent of this particular user.

This script could be saved in ~/.ssh/ssh-login and needs to be sourced during login by adding one line to the ~/.bash_profile, ~/.bash_login or ~/.profile (I don't know which one is used in your distributino):The leftover daemons you can just kill.

After implementing what you suggest, I executed my scripts from the continuous integration server on CI and got one ssh-agent; good. I repeated the process and got a second ssh-agent on Host, which isn't an improvement. Then I remembered that the continuous integration server runs a non-interactive non-login shell, so I added "source /home/user/.ssh/ssh-login" to the continuous integration job. It's Hudson, BTW, running on RHEL, and the host is FC 12.

The script failed ("hostname" replaces real host name):

I tried `ps --no-heading -p7184 -o uid` (using a PID of an active child process) directly on the command line while VNC'd to Host and got nothing. I can't find "--no-heading" in the man page for ps. Then, I tried `ps -pPID -o uid --no-header` and that seemed to work. I then modified ssh-login accordingly, but it still fails:

I don't know where to go next.

This is odd, the man-page of ps on Debian says:Maybe it's a Debian extension, but on openSUSE 11.3 it's also working, despite the fact that it's not in the man page in this case. Anyhow, you solved it.

So, we have the issue that the ssh-agent from the last login is still running and the file with its settings is also there, but it's not recognized due to the output of the ps command in the script returns nothing, like on the command line. This would imply that the ssh-agent isn't running any longer and so a new one is started.

Can you check, what PIDs the still running ssh-agent have in ps, as you mention they are still running. The one recorded in /home/hudson/.ssh/env-hostname is not among them?

env-$HOSTNAME has this:
SSH_AUTH_SOCK=/tmp/ssh-lOyBVF7182/agent.7182; export SSH_AUTH_SOCK;
SSH_AGENT_PID=7184; export SSH_AGENT_PID;

That lines up with what your script reports.

If I want to stop now, I could invoke the Host scripts from the CI tool *without* sourcing myfile or .bash_profile, and it will work without leaving a zombie process. Until the system reboots, right? So that's not a good option. Another option could be for me to put "killall -9 ssh-agent" at the end of my Hudson job.

What is not a god option? That you have to enter the passphrase once after a reboot? Or that it's running all the time?

The first; I would prefer to be able to run these tests non-interactively at any time, even after a reboot.

When I use "ps -pPID -o uid --no-header" on the ssh-agent's PID, at the command line, it gives me my correct UID. But doesn't the result I posted about at 4:10 imply when that same line is used inside your script, nothing is returned? ("FOUND_UID= ")

When you just want to connect to the device, and don't want to use the ssh-key for other purpose, you could also remove the passphrase by using As the passphrase protects only the private part of the key, nothing needs to be changed on the device under test.

You are right, that it should also output something useful inside the script. Maybe the script you use is absorbing any output for any reason (you mentioned that you put it inside a script). An option could be to put it in ~/.bashrc which should be used for a non-interactive backup.