[SOLVED] How to define variable in `ssh`

Ben Wang · 04-23-2015, 05:17 PM

I need to use `ssh` to send command to different computers to run programs. Can I define a variable of `Directory` and `program` in front of this shell script? Then, I do not need to write them every time? The following is my shell script.

`host_list=("c15-0330-10.ad.mtu.edu" "c15-0330-11.ad.mtu.edu" "c15-0330-12.ad.mtu.edu")`
`# I have multiple programs`
`# program=c("L_1","L_2","L_3")`
`ssh -f "${host_list[0]}" 'set Directory="/home/campus27/zwang10/Desktop/AWRR/program/power/vmodel_1/nprot/K_10"; cd $Directory && nohup Rscript L_1.R> L_1_sh.txt;echo "The job L_1 is finished" |mutt "zwang10@mtu.edu" -s "The job L_1 is finished"';
ssh -f "${host_list[1]}" 'set Directory="/home/campus27/zwang10/Desktop/AWRR/program/power/vmodel_1/nprot/K_10"; cd $Directory && nohup Rscript L_2.R> L_2_sh.txt;echo "The job L_2 is finished" |mutt "zwang10@mtu.edu" -s "The job L_2 is finished"';
ssh -f "${host_list[2]}" 'set Directory="/home/campus27/zwang10/Desktop/AWRR/program/power/vmodel_1/nprot/K_10"; cd $Directory && nohup Rscript L_3.R> L_3_sh.txt;echo "The job L_3 is finished" |mutt "zwang10@mtu.edu" -s "The job L_3 is finished"';`

millgates · 04-23-2015, 05:30 PM

Yes, you can. But they will not expand within single quotes. If you want to use variables, use double quotes instead of single quotes
If you need to quote within double quotes, just escape the double quotes. Please note that literal strings don't need to be quoted unless they contain spaces/special characters. If you do need to quote within the quotes, you may escape the inner quotes, or concatenate double quoted and single quoted strings by placing them immediately next to each other

Code:

"double quoted $variable "
"double quoted \"double quoted text inside double quoted\" double quoted"
'single quoted'"$variable double quoted"'single quoted'

Ben Wang · 04-23-2015, 05:44 PM

Quote:

Originally Posted by millgates

Yes, you can. But they will not expand within single quotes. If you want to use variables, use double quotes instead of single quotes
If you need to quote within double quotes, just escape the double quotes. Please note that literal strings don't need to be quoted unless they contain spaces/special characters. If you do need to quote within the quotes, you may escape the inner quotes, or concatenate double quoted and single quoted strings by placing them immediately next to each other

Code:

"double quoted $variable "
"double quoted \"double quoted text inside double quoted\" double quoted"
'single quoted'"$variable double quoted"'single quoted'

I tried following code, but it does not work.

Code:

host_list=("c15-0330-10.ad.mtu.edu" "c15-0330-11.ad.mtu.edu" "c15-0330-12.ad.mtu.edu")
# I have multiple progrmas
#program=c("L_1","L_2","L_3")
address="/home/campus27/zwang10/Desktop/AWRR/program/power/vmodel_1/nprot/K_10"
ssh -f "${host_list[0]}" 'cd' "$address" 'nohup Rscript L_1.R> L_1_sh.txt;echo "The job L_1 is finished" |mutt "zwang10@mtu.edu" -s "The job L_1 is finished"';

It shows

Code:

cd: Too many arguments.

millgates · 04-23-2015, 05:51 PM

Quote:

Originally Posted by Ben Wang

ssh -f "${host_list[0]}" 'cd' "$address" 'nohup Rscript L_1.R> L_1_sh.txt;echo "The job L_1 is finished" |mutt "zwang10@mtu.edu" -s "The job L_1 is finished"';[/CODE]
It shows

Code:

cd: Too many arguments.

You're missing some kind of operator after 'cd' "$address", such as a semicolon or &&.

Ben Wang · 04-23-2015, 06:01 PM

Quote:

Originally Posted by millgates

You're missing some kind of operator after 'cd' "$address", such as a semicolon or &&.

I tried following command

Code:

host_list=("c15-0330-10.ad.mtu.edu" "c15-0330-11.ad.mtu.edu" "c15-0330-12.ad.mtu.edu")
# I have multiple progrmas
#program=c("L_1","L_2","L_3")
address="/home/campus27/zwang10/Desktop/AWRR/program/power/vmodel_1/nprot/K_10"
ssh -f "${host_list[0]}" 'cd' "$address" '&& nohup Rscript L_1.R> L_1_sh.txt';

And now it works now.

millgates · 04-23-2015, 06:13 PM

The problem is that if you leave the && unquoted, it will be interpreted on the local machine, so it's actually

Code:

{ssh -f "${host_list[0]}" 'cd' "$address"} && {'nohup Rscript L_1.R> L_1_sh.txt'}

rather than

Code:

ssh -f "${host_list[0]}" {'cd' "$address" '&& nohup Rscript L_1.R> L_1_sh.txt'}

The quoting in shell may seem a little confusing at first, especially when the commands get interpreted twice, but you'll get used to it, eventually.

Ben Wang · 04-23-2015, 10:05 PM

Quote:

Originally Posted by millgates

The problem is that if you leave the && unquoted, it will be interpreted on the local machine, so it's actually

Code:

{ssh -f "${host_list[0]}" 'cd' "$address"} && {'nohup Rscript L_1.R> L_1_sh.txt'}

rather than

Code:

ssh -f "${host_list[0]}" {'cd' "$address" '&& nohup Rscript L_1.R> L_1_sh.txt'}

The quoting in shell may seem a little confusing at first, especially when the commands get interpreted twice, but you'll get used to it, eventually.

I want to set programs as variable. I tried following code

Code:

host_list=("c15-0330-01.ad.mtu.edu" "c15-0330-02.ad.mtu.edu" "c15-0330-03.ad.mtu.edu" "c15-0330-04.ad.mtu.edu")
program=("L_1" "L_2" "L_3" "L_4")
address="/home/campus27/zwang10/Desktop/AWRR/program/power/vmodel_1/nprot/K_10"
ssh -f "${host_list[0]}" 'cd' "$address" && 'nohup Rscript' "${program[0]}.R" '>' "${program[0]}_sh.txt" ; 'echo' "The job "${program[0]}" is finished" | 'mutt "zwang10@mtu.edu" -s' "${program[0]} is finished";

But it says

Code:

./test_2.sh: line 4: nohup Rscript: command not found
./test_2.sh: line 4: mutt "zwang10@mtu.edu" -s: command not found

millgates · 04-24-2015, 03:40 AM

Now the && is not within quotes, again. Also, the semicolon and the pipe later on are not within quotes, either. That said, your quoting is a little confused. It doesn't have to be so chaotic.
Keep in mind that the string after the ssh command will be interpreted twice. Once on the local machine, once on the remote machine. So let's start from the top without quotes:

Code:

ssh -f ${host_list[0]} cd $address && nohup Rscript ${program[0]}.R > ${program[0]}_sh.txt ; echo The job ${program[0]} is finished | mutt zwang10@mtu.edu -s ${program[0]} is finished;

so, we actually want to pass three arguments to ssh: -f, the hostname, and the code to execute on the remote host -- there's really no reason for the command to be split into multiple arguments.
Now, -f is a literal, no need to quote it.
The hostname is a variable, and while in this particular case it doesn't contain spaces, it's always a good idea to quote your variables. Single quotes would prevent expansion of the variable which we do not want, so it's definitely double quotes for this one, as you correctly used in your examples.
Last, our command contains variables which we want expanded on the local host (because that's where they are defined -- if the variables were defined on the remote host, you would want single quotes to have them expand there), so why don't we put the entire command within double quotes to keep the local shell from splitting it into multiple arguments and interpretting the semicolons, &&'s, pipes, etc:

Code:

ssh -f "${host_list[0]}" "cd $address && nohup Rscript ${program[0]}.R > ${program[0]}_sh.txt ; echo The job ${program[0]} is finished | mutt zwang10@mtu.edu -s ${program[0]} is finished;"

What will happen now? the local shell will split the line into four tokens, ssh, -f, ${host_list[0]}, and the command. It will remove the double quotes and also expand all variables that are not within single quotes. So, the command sent to the remote shell is:

Code:

cd /home/campus27/zwang10/Desktop/AWRR/program/power/vmodel_1/nprot/K_10 && nohup Rscript L_1.R > L_1_sh.txt ; echo The job L_1 is finished | mutt zwang10@mtu.edu -s L_1 is finished;

now, the command arriving to the remote shell does not contain any quotes, because they have been removed by the local shell. In this particular case most of the arguments (except for the subject for mutt) contain anything that needs to be quoted. However, we were using some variables, which didn't, but could have in general case contained spaces. Keep in mind that some day in the future you may decide to change the values of the variables in the script into something that happens to contain spaces. Then your script will stop working and you will have to read through it and put quotes wherever they are needed. So it's considered a good practice to treat variables as something that contains spaces even if they don't right now, to keep your code maintainable.
But there is a problem: the double quotes do not nest and they get removed by the local shell, so we need to protect them so they can survive their way to the remote shell. The most straight-forward way to do that is to escape them:

Code:

ssh -f "${host_list[0]}" "cd \"$address\" && nohup Rscript \"${program[0]}.R\" > \"${program[0]}_sh.txt\" ; echo \"The job ${program[0]} is finished\" | mutt zwang10@mtu.edu -s \"${program[0]} is finished\";"

Looks a little ugly, but should get the job done. What happens now? The outter quotes will be removed just like before, but the escaped quotes will not be interpreted. Instead, each occurence of a backslash followed by a double quote will be replaced by a literal double quote. So, now the string that gets to the remote shell is:

Code:

cd "/home/campus27/zwang10/Desktop/AWRR/program/power/vmodel_1/nprot/K_10" && nohup Rscript "L_1" > "L_1_sh.txt" ; echo "The job L_1 is finished" | mutt zwang10@mtu.edu -s "L_1 is finished";

Instead of double quotes, we could have just as well used single quotes for the inner quoting, because all the variables have already been expanded (We would need double quotes here if we had variables which we actually wanted to be expanded on the remote host). Even better -- because single quotes lose their special meaning within double quotes -- we don't even need to quote them:

Code:

ssh -f "${host_list[0]}" "cd '$address' && nohup Rscript '${program[0]}.R' > '${program[0]}_sh.txt' ; echo 'The job ${program[0]} is finished' | mutt zwang10@mtu.edu -s '${program[0]} is finished';"

So the command executed by the remote shell is

Code:

cd '/home/campus27/zwang10/Desktop/AWRR/program/power/vmodel_1/nprot/K_10' && nohup Rscript 'L_1.R' > 'L_1_sh.txt' ; echo 'The job L_1 is finished' | mutt zwang10@mtu.edu -s 'L_1 is finished';

Ben Wang · 04-24-2015, 08:56 PM

You illustration is very clear and very helpful. Now, I want the email report the time it starts and it ends. I am not sure I understand it right or not. Here is my understanding:
1. in the localhost, shell will remove the double quotes of the command.
2. Since the variable in the command is not quoted by double quotes, it can be expanded.
3. Localhost will send the command which is removed of double quotes.
4. Remote host will run the command, in which the variable is already expanded.
According what you said, I put

Code:

END TIME = `date +"r"`

out of single quote. But the email always shows the START TIME is exact as END TIME. Here is my shell script.

Code:

host_list=("c15-0330-01.ad.mtu.edu" "c15-0330-02.ad.mtu.edu" "c15-0330-03.ad.mtu.edu" "c15-0330-04.ad.mtu.edu")
program=("L_1" "L_4" "L_3" "L_4")
subject="The job is finished"
START=$(date +"%r")
address="/home/campus27/zwang10/Desktop/AWRR/program/power/vmodel_1/nprot/K_10"
ssh -f "${host_list[0]}" "cd '$address' && nohup Rscript '${program[0]}.R' > '${program[0]}_sh.txt';echo 'The job\n $address\n${program[0]} is finished\nSTART TIME = $START\n' END TIME =`date +"%r"` | mutt zwang10@mtu.edu -s '${host_list[0]} - Job ${program[0]}.R finished' -a '$address/${program[0]}_sh.txt';"

millgates · 04-25-2015, 08:10 AM

Ok, now there's another shell feature coming into play -- the command substitution.
command substitution, just like parameter substitution, happens within double quotes. That means the command date +"r" will be executed and its output substituted into the command on the localhost before it is sent over through ssh. So, you have to protect it from expanding on the local host if you want it to be executed on the remote host.
So, the possibilities are:
1/ escape the backticks (or, in the case of the $() syntax, the $ character

Code:

 ssh ... " ... \`date +"%r"\` ..."

2/ put the command substitution within single quotes, but not within double quotes, where single quotes lose their special meaning.

Code:

ssh ... " ... "'`date +"%r"`'" ... "

Some advices:
a/ You seem to use both backticks and the $() construct for command substitution. While both do almost the same thing, the latter is preferred (it nests more conveniently) if your shell supports it.
b/ As the command grows larger and more complex, consider if it wouldn't be more practical to place it in a script on the remote host and just do 'ssh -f hostname "/path/to/script"' instead.