I've got an expect script working when I set the variable for the password inside the script, but I can't figure out how to provide the password as an argument. Here is the boiled down problem:

Script:

#!/usr/bin/expect
set pass [lindex $argv 0]

send "$pass\r"

Usage:

./myScript.sh testPassword

Output:

- it does nothing and simply returns the prompt. This is to highlight the issue, it also fails when used in the actual context of - ;"password:" {send "$pass\r";exp_continue};}

However, if I change the send command to be this:

send "$pass"

it correctly repeats the password to the prompt. It seems to dislike the "\r" escape sequence for some reason. I've googled many examples of this in the past few days and I've tried many combinations of things but can't seem to find what simple and obvious thing I am missing. Would really appreciate any help you can offer!

Also - why can't I just use #!/usr/bin/expect AND #!/bin/bash in the same script? Is expect also a shell and that conflicts with bash or something? I've only been through one scripting book so far but it really didn't cover the # operator very well.

There's not problem with your usage--not technically anyway.

It's that you don't understand what '\r' represents. When your script executes:
The value of the variable 'pass' is written to standard output. The '\r' acts to send an Enter/Return key press. Specifically, the effect is to move the output cursor to the beginning of the current line.

Because the cursor has been moved to the beginning of the line, when the script ends, your bash prompt is written on top of the value of 'pass'.

The reason the command works without the '\r' is because the bash prompt is written immediately after the value of 'pass' -- the cursor is not moved.

You misunderstand. The '#' is not an operator. It truly just means that the text following the '#' is a comment (in most scripting languages). The shebang line (e.g. '#!/bin/bash') is a special form of comment that must be placed (if present) at the beginning of the script file to identify what program should be used to interpret all the commands in the file.

You can have a bash script launch an expect script. You can have an expect script launch a bash script. Just write the pieces of the task(s) in separate files. Launch the pieces as necessary.

As an advanced approach--'advanced' not meaning 'better' but simply not normally used--can embed expect scripts within a bash script using 'here' documents. For instance:
The above uses the fact that the cat command will echo everything between the cat command and the EOF, then pipes all the text to /usr/bin/expect, which then interprets the text as an expect script. This allows you to use bash variables directly in the expect script without dealing with command line options. However, there can be some tricky syntax situations where bash does some substitutions in the cat text before sending it to expect.

I don't understand why a carriage return/line feed, or as you say "Enter/Return" key press would move the cursor to the beginning of the current line. It should move it to the beginning of the next line?

I'm sorry, I thought that smaller example would make the question easier to answer - looks like I was wrong there haha Ok, here's my full script and output. Hopefully this will be able to convey the full picture better:

********************
SCRIPT:


#!/usr/bin/expect
set pass [lindex $argv 0]
set user "myUsername"
set src "mySourceFile.name"
set dest "/myRemoteDirectory/myDestFile.name"

spawn scp -o StrictHostKeyChecking=no $src $user@myhost:$dest
expect {
;"password:" {send "$pass\r";exp_continue};}

********************
OUTPUT:
[user@localHost ~]$ ./copyFile.sh myPassword
spawn scp -o StrictHostKeyChecking=no .bashrc user@remoteHost:/myRemoteDirectory/myDestFile.name
user@remoteHost's password:
Permission denied, please try again.
user@remoteHost's password:
Permission denied, please try again.
user@remoteHost's password:
Permission denied (publickey,gssapi-with-mic,password).
lost connection

********************

Now, when I change this line:

- set pass [lindex $argv 0]

to this:

- set pass actualPassword

It works perfectly fine and I get the following output. Any ideas?

********************
[user@localHost ~]$ ./copyFile.sh
spawn scp -o StrictHostKeyChecking=no .bashrc user@remoteHost:/myRemoteDirectory/myDestFile.name
user@remoteHost's password:
.mySourceFile.name

No. The '\r' is somewhat of a historical carryover from the days where typewriters were prevalent. You read '\r' as 'carriage return' -- 'carriage' being the mechanism that moved from left to right as letters were typed and 'return' to move the carriage back to the initial left position. It was common to want the carriage to return to the left (for corrections) without advancing to the next line.

There is another character '\n' which is read as 'new line' or 'newline'. This character does what you are probably thinking. In Linux, the newline not only scrolls to the next line but also performs a 'carriage return' by placing the cursor at the beginning of that new line. On Windows/DOS systems, the newline simply advances the line, without the carriage return. The effect on Windows/DOS for using '\n' only (and not '\r\n') is text that looks like:
To verify, try it yourself:
As long as your bash prompt is less than 50 characters long, you should still see some of the number printed after your bash prompt returns.

Compared with:
Which should show the full range of numbers with your bash prompt on the next line.

There are reasons why the distinction is still maintained. For instance, sometimes you want to re-write a line with updated information once every second. In such a case, a '\r' is needed instead of '\n'. You see this kind of thing all the time with console-based progress bars.

As for your script, I'm not an expect expert, but I would guess that '$argv 0' does not contain what you expect. It may hold the newline/carriage return from when you entered the command. I'll have to experiment a little to find out.

EDIT:
As a follow-up, does your password contain any special characters? For instance an asterisk '*'? If it uses special characters, you need to make sure that bash is not doing any wildcard substitution/matching. If your password does contain special wildcard characters, I would suggest enclosing your password in double quotes at the command line:

Nice explanation by Dark_Helmet; see also the ASCII table codes 10 & 13 (decimal) http://www.asciitable.com/

I've been using lf/cr 10/13 in code for years without realizing what they were actually doing. Great explanation indeed, thank you for that

About the script - after running the first script (just the output) again I could see that it was the special characters. Double quotes didn't do it, but single quotes did. Great catch! Thank you so much for your help, I have a lot to learn about Linux

Glad I could help. And if I may be so bold, from what you said, it sounds as though your password contains a '$' somewhere. The '$' would be interpreted by bash as the beginning of a shell variable reference. So bash would attempt to substitute the value of the variable for the reference--all before invoking expect. More than likely, the accidental variable reference evaluated to an empty string. Empty string or not, it mucked up what expect saw.

If you want to use a literal $ on the command line, you can do what you discovered: enclose the text in single quotes (which prevents bash from doing any substitutions in the text), or you can escape the $ with a backslash. For instance: