Expect called from bash; Unexpected EOF !

Honest Abe · 04-15-2024, 02:28 AM

I have few legacy machines which are about to be decommissioned. There is no scope of adding them to puppet/ansible etc. There are no ssh keys either as the user management is done via a third party tool and password is rotated weekly.

So, if I want to execute a script from a centralized jump box on these bunch of servers, expect is the only thing I can use to automate my tasks! I am only trying to fetch information from about ~20 servers.

These servers authenticate with a common username, but passwords are different for all.

Code sample-

Code:

#!/bin/bash
serverlist=/path/to/map.txt
#username is common and password from file
user=admin
##login to each Server
#for i in $(cat $serverlist)
while IFS= read  line
        do
        echo $line
        set timeout 100
        server= echo $line | awk -F ' ' '{print $1}'   # first field is servername
        password= echo $line | awk -F ' ' '{print $2}' # second field is password
        #echo $server
        #echo $password
        /bin/expect << THIS
        exp_internal 1
        spawn ssh -q -o StrictHostKeyChecking=no -t "$user@$server"
        expect ":"
        send -- "$password\r"
        expect "$ "
        send "sudo -i\r"
        expect ":"
        send -- "$password\r"
        expect "# "
        send "hostname --fqdn;uname -a; who -rb; uptime \r"
        expect "# "
        send "exit \r"
        THIS
        done < $serverlist

The bash part of the script is very basic as I am merely setting up variables and extracting hostname and password from maps.txt (text file, fields separated by a single space). This part works without issues.
The problem starts with the EXPECT bit; specifically with the heredoc invocation.

Execution -

Code:

$: bash -x test.sh
+ serverlist=/Path/to/map.txt
+ user=admin
test.sh: line 29: warning: here-document at line 15 delimited by end-of-file (wanted `THIS')
test.sh: line 30: syntax error: unexpected end of file

What I have done -
a. exp_internal 1 (adding or removing it does not give additional debugging info)
b. made sure my 'map.txt' is a unix file (verified with 'cat -A' and dos2unix to make sure there are no weird characters)
c. If I disable the expect block, the rest of the script does not have any errors.

What am I missing ?

pan64 · 04-15-2024, 02:35 AM

this is wrong:

Code:

server= echo $line | awk -F ' ' '{print $1}'

this is better:

Code:

server=$(echo $line | awk -F ' ' '{print $1}')

Honest Abe · 04-15-2024, 02:54 AM

Thanks for your attention, @pan64. And so good to see you in the forum.

You are correct. I indeed wrapped them into $(), but must have removed them when I was trying to debug. As the error message is not very helpful, I must have been adjusting this or that.

Confirming that wrapping these values in $() does not take away the errors on the expect block. The errors persist.

pan64 · 04-15-2024, 03:16 AM

I suggest to use shellcheck to analyze your script.

michaelk · 04-15-2024, 04:46 AM

Heredocs are not normal, you need a - i.e. <<- EOF. If I remember correctly.

Honest Abe · 04-15-2024, 05:11 AM

That is a great lead. Looks like I should not have indented the HEREDOC

Code:

#!/bin/bash
serverlist=/path/to/map.txt
#username is common and password from file
user=admin
##login to each Server
#for i in $(cat $serverlist)
while IFS= read -r line
        do
        echo "$line"
        server=$(echo "$line" | awk -F ' ' '{print $1}')   # first field is servername
        password=$(echo "$line" | awk -F ' ' '{print $2}') # second field is password
        #echo $server
        #echo $password
        /bin/expect <<THIS
        set timeout 100
        exp_internal 1
        spawn ssh -q -o StrictHostKeyChecking=no -t "$user@$server"
        expect ":"
        send -- "$password\r"
        expect "~]$ "
        send "sudo -i \r"
        expect ":"
        send -- "$password\r"
        expect "# "
        send "hostname --fqdn;uname -a; who -rb; uptime \r"
        expect "# "
        send "exit \r"
THIS
        done < "$serverlist"

The script works under normal condition.

The only condition it breaks is when the password contain a '[' character, where it gives this error message -

Code:

spawn ssh -q -o StrictHostKeyChecking=no -t admin@server2.domain.com
parent: waiting for sync byte
parent: telling child to go ahead
parent: now unsynchronized from child
spawn: returns {669298}

expect: does "" (spawn_id exp4) match glob pattern ":"? no
Password:
expect: does "\rPassword: " (spawn_id exp4) match glob pattern ":"? yes
expect: set expect_out(0,string) ":"
expect: set expect_out(spawn_id) "exp4"
expect: set expect_out(buffer) "\rPassword:"
missing close-bracket
    while executing
"send -- "AbCd5EF*/r[H\r"   # not the real password, but close 
        expect "~]$ "
        send "sudo -i \r"
        expect ":"
        send -- "AbCd5EF*/r["

I did find out why it breaks.
From: https://tcl.tk/man/tcl8.6/TclCmd/Tcl.htm#M11
If a word contains an open bracket (“[”) then Tcl performs command substitution. To do this it invokes the Tcl interpreter recursively to process the characters following the open bracket as a Tcl script. The script may contain any number of commands and must be terminated by a close bracket (“]”). The result of the script (i.e. the result of its last command) is substituted into the word in place of the brackets and all of the characters between them. There may be any number of command substitutions in a single word. Command substitution is not performed on words enclosed in braces.

so how do I tell expect to treat any '[' is $password as only a string and not as a special character ?

As soon as I change to this -

Code:

        send -- {$password\r}
        expect "~]$ "
        send "sudo -i \r"
        expect ":"
        send -- {$password\r}
        expect "# "

The code stops executing. There are no bash errors, but expect is stuck !

Code:

 bash -x test.sh
+ serverlist=/Path/To/map.txt
+ user=admin
+ IFS=
+ read -r line
+ echo 'server1.domain.com A6bc~deFG0Hi'
server1.domain.com A6bc~deFG0Hi
++ echo 'server1.domain.com A6bc~deFG0Hi'
++ awk -F ' ' '{print $1}'
+ server=server1.domain.com
++ echo 'server1.domain.com A6bc~deFG0Hi'
++ awk -F ' ' '{print $2}'
+ password=A6bc~deFG0Hi
+ /bin/expect
spawn ssh -q -o StrictHostKeyChecking=no -t admin@server1.domain.com
parent: waiting for sync byte
parent: telling child to go ahead
parent: now unsynchronized from child
spawn: returns {672245}

expect: does "" (spawn_id exp4) match glob pattern ":"? no

expect: does "\r" (spawn_id exp4) match glob pattern ":"? no
Password:
expect: does "\rPassword: " (spawn_id exp4) match glob pattern ":"? yes
expect: set expect_out(0,string) ":"
expect: set expect_out(spawn_id) "exp4"
expect: set expect_out(buffer) "\rPassword:"
send: sending "A6bc~deFG0Hi\r" to { exp4 }

expect: does " " (spawn_id exp4) match glob pattern "~]$ "? no

I suspected \r to be the issue, but removing it did not change the behaviour.

Any pointers ?

pan64 · 04-15-2024, 05:53 AM

here you can find some ideas to easily test your expect script and also probably you can improve it too: https://phoenixnap.com/kb/linux-expect
also you may try:

Code:

while IFS= read -r server password
do
/bin/expect .....
done

Honest Abe · 04-15-2024, 08:07 AM

I tried as you suggested and modified my while loop thus -

Code:

while IFS=' ' read -r server password

Allows me to get rid of 2x AWK commands, but the password string gives the same error "missing close-bracket" when it sees a '['.

so how do I tell expect to treat any '[' is $password as only a string and not as a special character ?

pan64 · 04-15-2024, 08:23 AM

it looks quite simple: https://unix.stackexchange.com/quest...-expect-script
export password and use $env{password}

Honest Abe · 04-15-2024, 08:29 AM

tried escaping the "[" in the maps.txt with {} but didnt work.
But tried with a simple \, and now it behaves fine.

What I do not understand is, bash is reading the file, encounters the escape character and thus treats '[' normally. But '[' is not a special character in bash ( [] is or [[]] is !). And then when the strings go from bash to the expect block, should it not complain again about the rogue [ ?

ntubski · 04-15-2024, 05:04 PM

Quote:

Originally Posted by Honest Abe

What I do not understand is, bash is reading the file, encounters the escape character and thus treats '[' normally. But '[' is not a special character in bash ( [] is or [[]] is !). And then when the strings go from bash to the expect block, should it not complain again about the rogue [ ?

You have to remember there are two layers of evaluation, bash and then TCL/expect. Perhaps the main part you missed is that bash HERE docs do $variable expansion, so when you have

Code:

        password=xyz[
        /bin/expect <<THIS
        send -- "$password\r"
THIS

That is entirely equivalent from expect's point of view to this:

Code:

        /bin/expect <<THIS
        send -- "xyz[\r"
THIS

Bash has no problem expanding the variable containing "[", but then expect is seeing the unbalanced [ as just literal text, so it flags an error.

(I'm not that familiar with TCL, but based on the doc page you linked, I believe the problem with using {} to escape is that it only takes effect if the { is the start of the word, but you have the double quote character first.)

https://www.gnu.org/software/bash/ma...Here-Documents

pan64 · 04-16-2024, 12:34 AM

As an experience you can add set -xv to your script (let's say before while) and you will see all commands (line by line) as they are evaluated by the shell and as they are really look like when executed. You can't do that with expect/tcl.
As I mentioned in post #9 using variables will eliminate this issue, because they will not be preprocessed at all, their content will remain intact.

murugesandins · 04-22-2024, 10:33 PM

Quote:

Originally Posted by Honest Abe

tried escaping the "[" in the maps.txt with {} but didnt work.
?

I am using following commands using CYGWIN_NT at windows:
password="$(
server="$(
...
send "\$env(password)\n"

Due to your query I have changed my localhost (windows) password to xyz[ since I am admin

I tried your script at windows using CYGWIN_NT and sshd server listening at 127.0.0.1
Due to your query I made following changes:
01)
Make a backup of /etc/sshd_config before modification if not taken earlier.
File: /etc/sshd_config
Replace:
#PermitRootLogin prohibit-password
...
#PubkeyAuthentication yes
...
#GSSAPIAuthentication no
...
#UsePAM no
With:
#PermitRootLogin prohibit-password
PermitRootLogin no
...
#PubkeyAuthentication yes
PubkeyAuthentication yes
...
#GSSAPIAuthentication no
GSSAPIAuthentication yes
...
#UsePAM no
UsePAM yes
Add:
PasswordAuthentication yes
ChallengeResponseAuthentication no
Make a backup of updated file /etc/sshd_config including approved comment from supervisor(Hill Jason I like this technical person a lot from 2005 to 2013 and I have never seen such a technical person throughout my life ever who awarded me during 2008)
02)
restart the sshd service at windows
/cygdrive/c/WINDOWS/system32/sc.exe stop sshd
/cygdrive/c/WINDOWS/system32/sc.exe start sshd
My test result:

Quote:

murugesandins@127.0.0.1 /home/murugesandins [ 0 ]
$ /usr/bin/vim ./post6497800.sh;./post6497800.sh
line :127.0.0.1 xyz[: line
server 127.0.0.1
password xyz[
spawn /usr/bin/ssh -q -o StrictHostKeyChecking=no -t murugesandins@127.0.0.1
murugesandins@127.0.0.1's password:
Last login: Tue Apr 23 08:57:18 2024 from 127.0.0.1
Last logon Tue 23-Apr-2024 08:57 AM
TODAY DUST BIN DATE Tue 23-Apr-2024 10:35 AM
murugesandins@127.0.0.1 /home/murugesandins [ 0 ]
$ /usr/bin/pwd;
/home/murugesandins
murugesandins@127.0.0.1 /home/murugesandins [ 0 ]
$ /usr/bin/hostname --fqdn;/usr/bin/uname -s | /usr/bin/sed "s/-[0-9]*.[0-9]*.-[0-9]*$//;";/usr/bin/uname -r;/usr/bin/uname -m;/usr/bin/who -a;/usr/bin/uptime;
murugesan-openssl
CYGWIN_NT
3.5.3-1.x86_64
x86_64
pty0 2024-04-21 10:08 7475 id=y0 term=0 exit=0
murugesandins - pty1 2024-04-23 06:32 . 62707 (127.0.0.1)
murugesandins - pty2 2024-04-23 08:08 . 2940 (127.0.0.1)
pty3 2024-04-23 06:34 63724 id=y3 term=0 exit=0
murugesandins - pty4 2024-04-23 08:57 . 17145 (127.0.0.1)
08:57:38 up 4 days, 21:17, 3 users, load average: 0.00, 0.00, 0.00
murugesandins@127.0.0.1 /home/murugesandins [ 0 ]
$
murugesandins@127.0.0.1 /home/murugesandins [ 0 ]
$

Code:

#!/bin/bash
user=$LOGNAME   # I am using murugesandins. You can change this to admin at your environment.
serverlist=/path/to/map.txt
if [[ -f $serverlist ]]
then
        while IFS= read -r line
        do
                echo "line :$line: line"
                export server="$(echo "$line" | awk -F ' ' '{print $1}')"   # first field is servername
                export password="$(echo "$line" | awk -F ' ' '{print $2}')" # second field is password
                echo "server   $server"
                echo "password $password"
        /bin/expect <<THIS
        spawn /usr/bin/ssh -q -o StrictHostKeyChecking=no -t "$user@$server"
        expect "$user@$server's password:"
        send "\$env(password)\n"
        expect "$ " {send "/usr/bin/pwd;\n"}
        expect "$ " {send "/usr/bin/hostname --fqdn;\
/usr/bin/uname -s | /usr/bin/sed \"s/-\[0-9\]*\\.\[0-9\]*\\.-\[0-9\]*\$//;\";\
/usr/bin/uname -r;\
/usr/bin/uname -m;\
/usr/bin/who -a;\
/usr/bin/uptime;\
\n"}
        expect "$ " {send "exit;\n"}
THIS
        done < "$serverlist"
        echo
else
        echo "ls $serverlist"
        /usr/bin/ls $serverlist
fi

After completing related test I have changed my localhost password.
No sudo at cygwin
Hence I have removed sudo.