[SOLVED] Backup with rsnapshot and ssh has passphraseless public key authentication failure

j0sh-linux · 03-06-2012, 01:04 AM

I am trying to setup rsnapshot to take backups of a remote server using public key authentication without passphrase and as root user. I think the public key authentication fails as I am asked for the root user password when I run "rsnapshot hourly" .

Here is the console output,

Code:

require Lchown
Lchown module loaded successfully
Setting locale to POSIX "C"
echo 16391 > /var/run/rsnapshot.pid
mv /.snapshots/hourly.5/ /.snapshots/_delete.16391/
mv /.snapshots/hourly.4/ /.snapshots/hourly.5/
mv /.snapshots/hourly.3/ /.snapshots/hourly.4/
mv /.snapshots/hourly.2/ /.snapshots/hourly.3/
mv /.snapshots/hourly.1/ /.snapshots/hourly.2/
mv /.snapshots/hourly.0/ /.snapshots/hourly.1/
mkdir -m 0755 -p /.snapshots/hourly.0/ZW-JOSH/
/usr/bin/rsync -avvv --delete --rsh="/usr/bin/ssh -vvv" \
    root@zw-josh.local.josh.com:/etc/ /.snapshots/hourly.0/ZW-JOSH/etc/
opening connection using: /usr/bin/ssh -vvv -l root zw-josh.local.josh.com rsync --server --sender -vvvlogDtpre.is . /etc/
OpenSSH_4.3p2, OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008
debug1: Reading configuration data /root/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to zw-josh.local.josh.com [10.71.68.112] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/identity type -1
debug1: identity file /root/.ssh/id_rsa type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: loaded 3 keys
debug1: Remote protocol version 2.0, remote software version OpenSSH_4.3
debug1: match: OpenSSH_4.3 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_4.3
debug2: fd 3 setting O_NONBLOCK
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_init: found hmac-md5
debug1: kex: server->client aes128-ctr hmac-md5 none
debug2: mac_init: found hmac-md5
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug2: dh_gen_key: priv key bits set: 135/256
debug2: bits set: 493/1024
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug3: check_host_in_hostfile: filename /root/.ssh/known_hosts
debug3: check_host_in_hostfile: match line 3
debug3: check_host_in_hostfile: filename /root/.ssh/known_hosts
debug3: check_host_in_hostfile: match line 3
debug1: Host 'zw-josh.local.josh.com' is known and matches the RSA host key.
debug1: Found key in /root/.ssh/known_hosts:3
debug2: bits set: 501/1024
debug1: ssh_rsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /root/.ssh/identity ((nil))
debug2: key: /root/.ssh/id_rsa ((nil))
debug2: key: /root/.ssh/id_dsa ((nil))
debug1: Authentications that can continue: publickey,gssapi-with-mic,password
debug3: start over, passed a different list publickey,gssapi-with-mic,password
debug3: preferred gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup gssapi-with-mic
debug3: remaining preferred: publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-with-mic
debug1: Next authentication method: gssapi-with-mic
debug3: Trying to reverse map address 10.71.68.112.
debug1: Unspecified GSS failure.  Minor code may provide more information
No credentials cache found

debug1: Unspecified GSS failure.  Minor code may provide more information
No credentials cache found

debug1: Unspecified GSS failure.  Minor code may provide more information
No credentials cache found

debug2: we did not send a packet, disable method
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /root/.ssh/identity
debug3: no such identity: /root/.ssh/identity
debug1: Trying private key: /root/.ssh/id_rsa
debug3: no such identity: /root/.ssh/id_rsa
debug1: Trying private key: /root/.ssh/id_dsa
debug3: no such identity: /root/.ssh/id_dsa
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
root@zw-josh.local.josh.com's password:

The rsnapshot config file is as follows,

Code:

# 
snapshot_root   /.snapshots/

cmd_cp          /bin/cp
cmd_rm          /bin/rm
cmd_rsync       /usr/bin/rsync
cmd_ssh /usr/bin/ssh
cmd_logger      /bin/logger
cmd_du          /usr/bin/du
cmd_rsnapshot_diff      /usr/local/bin/rsnapshot-diff
interval        hourly  6
interval        daily   7
interval        weekly  4
interval        monthly 3
verbose         5
loglevel        5
logfile /var/log/rsnapshot/rsnapshot.log
lockfile        /var/run/rsnapshot.pid
rsync_short_args        -avv
rsync_long_args --delete        --numeric-ids   --relative      --delete-excluded
ssh_args        -vvv    -o      BatchMode=yes
du_args -csh
one_fs  0

#include        ???
#include        ???
#exclude        ???
#exclude        ???

#include_file   /path/to/include/file
#exclude_file   /path/to/exclude/file

link_dest       1
sync_first      0
use_lazy_deletes        1
rsync_numtries  1

backup  root@zw-josh.local.josh.com:/etc/       ZW-JOSH/etc/

On the remote host I have configured the sshd to PermitRootLogins=forced-commands-only . And also the public key generated was copied to the authorized_keys2 file and a symlink authorized_keys was created that links to the aforementioned file.

The private key on the rsnapshot server is in the /root/cron directory, and there is a config file in /root/.ssh/ that has the details as below,

Code:

Host    root
Hostname        zw-josh.local.josh.com
IdentityFile    /root/cron/localhost-rsnapshot-key

There is no id_dsa (or id_rsa , or identity) file inside /root/.ssh . And I am using ssh protocol 2.

Does anyone have any idea why public key authentication is not working? And also, if possible, does anyone know how what arguements I can give to ssh to only try public key authentication ? Thanks.

EricTRA · 03-06-2012, 01:35 AM

Hello and welcome to LinuxQuestions,

Why would you create an authorized_keys2 file and symlink to it? Do you have any particular reason for that? SSH is very strict about permissions on the files. What are the permissions on your identity file? I'd check permissions on both files, remove the symlink and use what's to be used (authorized_keys).

Looking forward to your participation in the forums. Have fun with Linux.

Kind regards,

Eric

j0sh-linux · 03-06-2012, 02:55 AM

Quote:

Originally Posted by EricTRA

Why would you create an authorized_keys2 file and symlink to it? Do you have any particular reason for that?

While searching the www for solutions, I had come across this -> A possible solution

Anyway, here are the permissions of the files on the remote host

Code:

drwx------ 2 root root     4096 Mar  4 15:04 .ssh

lrwxrwxrwx 1 root root  16 Mar  4 15:04 authorized_keys -> authorized_keys2
-rw------- 1 root root 677 Feb 29 16:42 authorized_keys2
-rw-r--r-- 1 root root 394 Apr  5  2011 known_hosts

On the rsnapshot server, some of the permissions are as follows,

Code:

drwx------  2 root root     4096 Mar  5 18:01 .ssh

# ls -l .ssh/
total 8
-rw------- 1 root root   90 Mar  5 18:02 config
-rw-r--r-- 1 root root 1231 Feb 28 17:25 known_hosts

The identity file is /root/cron/localhost-rsnapshot-key , here are it's permissions,

Code:

/root/cron
drwxr-xr-x  2 root root     4096 Feb 28 14:59 cron

/root/cron/localhost-rsnapshot-key
-rw------- 1 root root 668 Feb 28 14:59 localhost-rsnapshot-key

I implemented your suggestion and tried it out, but I got the same result, and exactly the same output on console as in the previous case. Shall I bring back the authorized_keys2 file and the symlink, or should I leave it as it is with just the authorized_keys file?

As a side note, I checked whether setting PermitRootLogin=yes works, and it did work perfectly.

EricTRA · 03-06-2012, 07:40 AM

Hi,

What distro and version are you using?

Kind regards,

Eric

j0sh-linux · 03-06-2012, 12:08 PM

It's Red Hat 5.7

EricTRA · 03-06-2012, 12:18 PM

Hi,

Do you have by any chance SELinux enabled?

Kind regards,

Eric

j0sh-linux · 03-06-2012, 02:16 PM

Hi Eric,

It's installed , but not running

Reuti · 03-07-2012, 09:53 AM

Did you also specify a ForceCommand for the root user? I would assume that it won’t work this way with rsnapshot, as the command line is assembled on the fly and can’t be defined beforehand (unless you use some kind of wrapper to get the original command line options). It might work with PermitRootLogin=without-password setting.

lithos · 03-07-2012, 10:58 AM

Hi,

I would try to set the public-key to a ssh connection where:

Code:

opening connection using: /usr/bin/ssh -vvv -l root zw-josh.local.josh.com rsync --server --sender -vvvlogDtpre.is . /etc/

I would modify to:
opening connection using: /usr/bin/ssh -vvv -l root -i rsa_key zw-josh.local.josh.com rsync --server --sender -vvvlogDtpre.is . /etc/

j0sh-linux · 03-08-2012, 02:00 AM

Here is the sshd_config file parameters on the remote host

Code:

#
#Port 22
#Protocol 2,1
Protocol 2
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::

# HostKey for protocol version 1
#HostKey /etc/ssh/ssh_host_key
# HostKeys for protocol version 2
#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_dsa_key


#KeyRegenerationInterval 1h
#ServerKeyBits 768


#SyslogFacility AUTH
SyslogFacility AUTHPRIV
#LogLevel INFO


#LoginGraceTime 2m
#PermitRootLogin=yes
PermitRootLogin forced-commands-only
#StrictModes yes
#MaxAuthTries 6

#RSAAuthentication yes
#PubkeyAuthentication yes
#AuthorizedKeysFile     .ssh/authorized_keys


#RhostsRSAAuthentication no
# similar for protocol version 2
#HostbasedAuthentication no

#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
#IgnoreRhosts yes


#PasswordAuthentication yes
#PermitEmptyPasswords no
PasswordAuthentication yes


#ChallengeResponseAuthentication yes
ChallengeResponseAuthentication no


#KerberosAuthentication no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
#KerberosGetAFSToken no


#GSSAPIAuthentication no
GSSAPIAuthentication yes
#GSSAPICleanupCredentials yes
GSSAPICleanupCredentials yes


# ChallengeResponseAuthentication=no
#UsePAM no
UsePAM yes

AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL
#AllowTcpForwarding yes
#GatewayPorts no
#X11Forwarding no
X11Forwarding yes
#X11DisplayOffset 10
#X11UseLocalhost yes
#PrintMotd yes
#PrintLastLog yes
#TCPKeepAlive yes
#UseLogin no
#UsePrivilegeSeparation yes
#PermitUserEnvironment no
#Compression delayed
#ClientAliveInterval 0
#ClientAliveCountMax 3
#ShowPatchLevel no
#UseDNS yes
#PidFile /var/run/sshd.pid
#MaxStartups 10
#PermitTunnel no
#ChrootDirectory none

#Banner /some/path

Subsystem       sftp    /usr/libexec/openssh/sftp-server

lithos · 03-08-2012, 02:38 AM

Hi,

A quick look shows that you don't have enabled pubkey auth.
Uncomment the lines to enable and restart SSHD.

Quote:

Originally Posted by j0sh-linux

Here is the sshd_config file parameters on the remote host

Code:

#RSAAuthentication yes
#PubkeyAuthentication yes
#AuthorizedKeysFile     .ssh/authorized_keys


#RhostsRSAAuthentication no
# similar for protocol version 2
#HostbasedAuthentication no

#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
#IgnoreRhosts yes

Of course you will need to copy the pubkeys (LQ guide) to this server into user's .ssh/ directory to make them work. (skip the process generating ssh keys as you already have a key)

good luck

njlinuxmike · 03-09-2012, 12:31 PM

Just a tidbit on permissions... In all my setups I have needed to have the "authorized_keys" file set to perms "600"

Cheers

Mike

j0sh-linux · 03-12-2012, 03:45 AM

Quote:

Originally Posted by lithos

Hi,

A quick look shows that you don't have enabled pubkey auth.
Uncomment the lines to enable and restart SSHD.

Of course you will need to copy the pubkeys (LQ guide) to this server into user's .ssh/ directory to make them work. (skip the process generating ssh keys as you already have a key)

good luck

Tried it and it still did not work. Now I suspect the problem may be somewhere else. But first, I would have to explain the complete picture which I probably should have done before.

The rsnapshot server will be using cron to do automated logins and take backup. And then when the authentication process takes place, I have PermitRootLogins=forced-commands-only. So on the remote host in the authorized_keys file, I have the following before the public key data,

"from="192.xx.xx.xx",command="/root/cron/validate-rsync" ssh-dss......"

So if only the IP address of the rsnapshot server is recognized, then the "validate-rsync" script will be run. See here for the script --> validate-sync

I suspect after looking at this Ubuntuforum topic that cron is having issues using ssh.

j0sh-linux · 03-12-2012, 04:06 AM

Here some more data regarding permissions and ownership,

The private key is /root/cron/localhost-rsnapshot-key

Code:

-rw------- 1 root root 668 Mar  8 16:36 /root/cron/localhost-rsnapshot-key

drwxr-xr-x  2 root root     4096 Mar  8 16:36 cron

And ssh will read the location of the private key from /root/.ssh/config

Code:

drwx------  2 root root     4096 Mar  5 18:01 .ssh

-rw------- 1 root root 90 Mar  5 18:02 /root/.ssh/config

And the content of /root/.ssh/config

Code:

Host    root
Hostname        rsnap.local.josh.com
IdentityFile    /root/cron/localhost-rsnapshot-key

j0sh-linux · 03-15-2012, 07:37 AM

Finally figured out what the problem was, with some help from the author of Using Rsnapshot and SSH. Judging from the logs below, it seems like ssh was not able to find the correct private key file.

Code:

...
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /root/.ssh/identity
debug3: no such identity: /root/.ssh/identity
debug1: Trying private key: /root/.ssh/id_rsa
debug3: no such identity: /root/.ssh/id_rsa
debug1: Trying private key: /root/.ssh/id_dsa
debug3: no such identity: /root/.ssh/id_dsa
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
...

So I had to correct the /etc/.ssh/config file which had the info about where the private key was, so earlier it was,

Code:

Host    root
Hostname        rsnap.local.josh.com
IdentityFile    /root/cron/localhost-rsnapshot-key

but then I changed it to,

Code:

Host    *.local.josh.com
User    root
IdentityFile    /root/cron/localhost-rsnapshot-key

This advice helped as well,

Quote:

Originally Posted by lithos

Hi,

A quick look shows that you don't have enabled pubkey auth.
Uncomment the lines to enable and restart SSHD.

Quote:

Originally Posted by j0sh-linux

Here is the sshd_config file parameters on the remote host

Code:
#RSAAuthentication yes
#PubkeyAuthentication yes
#AuthorizedKeysFile .ssh/authorized_keys

#RhostsRSAAuthentication no
# similar for protocol version 2
#HostbasedAuthentication no

#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
#IgnoreRhosts yes

After this, the public key was found and used successfully. But I had another problem where the authentication was failing. Turns out there was a "newline" in the /root/.ssh/authorized_keys file on the remote server. The trick is to look into the log file of sshd on the remote server (/var/log/secure) for the error,

Code:

error: buffer_get_ret: trying to get more bytes 4 than in buffer 0

Please refer to this website for more info on this matter. It seems that the newline is a default action because of using ssh-copy-id. This newline is not visible in editor like nano . Found it with the vi editor. Basically there needs to be just spaces between fields, so my /root/.ssh/authorized_keys file starts with,

Code:

from="192.168.50.4",command="/root/cron/validate-rsync" ssh-dss JSGGEHK....

And after this, my backup problem has been solved.