rsync fails because public key/known host not on target

cookedheads · 02-01-2013, 12:27 PM

We're decom'ing a box that had been used to hold off site backups for our production server. The public key was set up between the old box and production so a cron job could rsync everything nightly, but I need to get the new box and the production box talking to each other without screwing up the key structure on production.

Here's the error from the cron log:

Permission denied, please try again.
Permission denied, please try again.
Permission denied (publickey,gssapi-with-mic,password).
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(359)

help?

I'm a dba who inherited the linux admin job, so be gentle...

lleb · 02-01-2013, 06:06 PM

just create a new key for the new box and then add it to the backup box.

i wrote this a while back, this should help you.

Code:

###### DIRECTIONS FOR CREATING RSA KEY################

	Directions for creating the rsa key and making the two
 servers talk to each other without password.

	1st change directory into .ssh and check what files are there.

 		[user@user ~]$ cd .ssh
		[user@user .ssh]$ ls -l
		total 4
		-rw-r--r-- 1 user group 2980 Jun 13 12:02 known_hosts

	2nd create the rsa key.

		[user@user .ssh]$ ssh-keygen -t rsa -b 4096
		Generating public/private rsa key pair.
		Enter file in which to save the key (/usr/user/.ssh/id_rsa):
		Enter passphrase (empty for no passphrase):
		Enter same passphrase again:
		Your identification has been saved in /usr/user/.ssh/id_rsa.
		Your public key has been saved in /usr/user/.ssh/id_rsa.pub.
		The key fingerprint is:
		cb:b0:40:c6:e9:f4:9e:f5:71:fc:c3:00:c0:f7:c6:75 user@user.localdomain

	3rd check that there are two new files with the following permissions

		[user@user .ssh]$ ls -l
		total 12
		-rw------- 1 user group 3243 Jun 22 15:50 id_rsa
		-rw-r--r-- 1 user group  743 Jun 22 15:50 id_rsa.pub
		-rw-r--r-- 1 user group 2980 Jun 13 12:02 known_hosts
	
	4th change directory back to the users $HOME

		[user@user .ssh]$ cd

	5th copy the key to the remote server

		[user@user ~]$ ssh-copy-id -i ~/.ssh/id_rsa.pub user@XXX.XXX.XXX.XXX
		25
		user@XXX.XXX.XXX.XXX's password:
		Now try logging into the machine, with "ssh 'user@XXX.XXX.XXX.XXX'", and check in:

		.ssh/authorized_keys

		to make sure we haven't added extra keys that you weren't expecting.

	6th, follow directions on the screen.

		[user@user ~]$ ssh user@XXX.XXX.XXX.XXX
		Last login: Fri Jun 22 14:12:08 2012 from 10.10.4.77
		[user@user ~]$ exit
		logout
		Connection to XXX.XXX.XXX.XXX closed.

Now the rsync.sh script will work without interaction.  THIS IS A MUST DO.

cookedheads · 02-02-2013, 08:55 AM

First, thank you for the prompt reply and for going into so much detail. Just to clarify:

I do all this on the new box?

I ask because 1) Obviously, there's already a key on production and 2) I'm trying to learn and dont' understand how creating a key on a new box will be allowed to talk to the existing production box with its own existing key. If it's too complex to type out, maybe even link to an easily understood document would help. I obviously tried to find this answer myself first, but public keys et al are all so far over my head at this point that my eyes glazed over.

Thanks,
Tracy

d3vrandom · 02-02-2013, 09:28 AM

Basic idea behind public key authentication:

- Each user has a pair of keys public and private. Stuff encrypted with the public key can only be decrypted with the private key and vice versa.

- If you want to login from machine A to machine B you have to install your public key in the authorized_keys file on machine B. That way when you attempt to login to B it will encrypt data using your public key. Since only you have the private key you are the only one that can decrypt the data thus proving your credentials to machine B.

Now to answer your question on where to generate the new key and where to install the public key:

- If you are pulling backups from the production box into the backup box then you have to install the backup box's public key in the production box. In this instance it is the backup box that is initiating the transaction - backup box is connecting to production box.

- If you are pushing backups from the production box to the backup box you have to install the production box's public key in the backup box. In this case it is the production box that is connecting to the backup box.

So it depends on how you have it setup really. The recommended way is to pull backups from the production box. The reason is that if you are pushing backups and the production box is compromised the attacker could also compromise your backups.

cookedheads · 02-02-2013, 10:56 AM

you guys are just unbelievably awesome. I looked forever for an explanation this clear for WEEKS...

okay, so just to make sure I don't break anything, my intention is to use rsync to get a staged drive on prod to be replicated on to staged drive on the new backup, replicating what was done with the old backup with an existing cron job on the prod box. Is that pushing or pulling?

and if it's pushing, can I "start over" and change from push to pull if keys are already in place in production?

lleb · 02-02-2013, 09:29 PM

yes. rsync can go either way, its all in the syntax:

rsync -<options> source/ destination/

so a push would look something like this:

Code:

rsync -aviS /home/foo/ user@backup_server:/home/foo2/

a pull would be very simular:

Code:

rsync -aviS user@remote_server:/path/to/files /path/to/local/files/

also note that if you are pushing then you will issue the ssk-keygen on the server pushing the data, if you are pulling, well then you issue it from there. so in your case if you are still pushing from old production server to new backup server, then just issue the copy-ssh-id from there to the new backup server and poof you are finished.

on the other hand if you are pulling data from the old production server to the new backup server, then you will have to perform all of the steps above.

FYI it will NOT hurt anything to perform this on both servers so you can go either way without issue. I typically setup bi-directional keys as it just makes life simpler in the long run.

d3vrandom · 02-03-2013, 06:56 AM

Quote:

Originally Posted by lleb

FYI it will NOT hurt anything to perform this on both servers so you can go either way without issue. I typically setup bi-directional keys as it just makes life simpler in the long run.

Well this is not correct. If you allow users on the production box to log into the backup box you open up the backup box to attack should the production box be compromised. So not a good idea.

d3vrandom · 02-03-2013, 07:13 AM

Quote:

Originally Posted by cookedheads

you guys are just unbelievably awesome. I looked forever for an explanation this clear for WEEKS...

okay, so just to make sure I don't break anything, my intention is to use rsync to get a staged drive on prod to be replicated on to staged drive on the new backup, replicating what was done with the old backup with an existing cron job on the prod box. Is that pushing or pulling?

and if it's pushing, can I "start over" and change from push to pull if keys are already in place in production?

That sounds like pushing if the production box is initiating the transaction.

What you should do is:

1) log into the backup box

2) Run ssh-keygen - it will generate the two keys private (.ssh/id_rsa) and public (.ssh/id_rsa.pub).

3) Use ssh-copy-id to install the public key onto the production box. Alternatively you can manually copy the contents of .ssh/id_rsa.pub into the .ssh/authorized_keys file on the production box.

4) Now you should be able to log into the production box from the backup box without using a password.

5) Setup a cron job to rsync from the production box to the backup box:

rsync -az user@productionbox:/path/to/directory /local/directory/

If you want to maintain incremental backups I recommend you look into rsnapshot.

chrism01 · 02-03-2013, 06:12 PM

Take what lleb said in post #2 and do it on the initiating/src box, sending the pub key to the target.
So, if pushing from prod to the new_backup, perform procedure on prod and send pub key to new_backup.
If pulling from new_backup, run procedure on there and send pub key to prod.
Its probably going to be easiest to use the same direction as the current/old system.

lleb · 02-03-2013, 10:33 PM

Quote:

Originally Posted by d3vrandom

Well this is not correct. If you allow users on the production box to log into the backup box you open up the backup box to attack should the production box be compromised. So not a good idea.

if you are foolish enough to setup USERS on the back up box, then you would be correct, I never said anything about USERS, i was talking about the backup script that would be run by, one would hope, someone with the proper credentials who already has full access to both servers. again i stand 100% behind my statement.

do not take a quantifiable simple fact and twist it around. we are talking about someone who is supposed to be "In Charge" of this process, thus this individual will have proper credentials on BOTH systems. to setup ssh keys going both ways for that individual will not be a problem.

d3vrandom · 02-04-2013, 06:58 AM

Quote:

Originally Posted by lleb

if you are foolish enough to setup USERS on the back up box, then you would be correct, I never said anything about USERS, i was talking about the backup script that would be run by, one would hope, someone with the proper credentials who already has full access to both servers. again i stand 100% behind my statement.

do not take a quantifiable simple fact and twist it around. we are talking about someone who is supposed to be "In Charge" of this process, thus this individual will have proper credentials on BOTH systems. to setup ssh keys going both ways for that individual will not be a problem.

By users I meant user account and I was not against a person having the ability to log into both systems.

What I don't think should happen is that a script running on the production box can log into the backup box automatically. If root is compromised on the production box the attacker could also attack the backup box. You loose both production and your backups.

Instead you want to set it up the other way around and only allow the backup box to connect to the production box. Since the backup box does backups only and does not run any servers it is less likely to be compromised.

cookedheads · 02-04-2013, 08:10 AM

it sounds like it would be much easier to maintain the old method, which I'm sorely tempted to and may still do but it sounds like "best practices" would be pulling from the backup box so I think I'm going to try to start all over from the new backup box with the first set of step. Either way, after doing a find . -name "ssh-copy-id", that command isn't on my RHEL4 box.

The closest thing I've got ( at least under usr/bin ) is ssh-gen, ssh-add, ssh-keyscan, and ssh-agent.

any thoughts? magic beans? words of wisdom? given that I've been handed the upgrade to RHEL 6, I see chocolate chip cookies ( and a lot more questions ) in all your futures

chrism01 · 02-05-2013, 12:57 AM

You can see the official support timelines here https://access.redhat.com/support/po...pdates/errata/. Note that Extended Life support (where RHEL4 is now) is an extra payment.

Its possible that RHEL4 is so old it doesn't have ssh-copy-id, in which case see the manual http://www.linuxtopia.org/online_boo...nt-config.html.
Basically scp the keys to the server & add into the file.

If your current system is setup for prod to copy to backup might easier (less surprises) to keep that method for now.
When you upgrade, you can consider reversing it.
Of course it really depends on which box you consider most secure

Also, you should always have off-line backups...

PS you should not upgrade across major versions eg v4 => v5; its not supported by RH
https://access.redhat.com/knowledge/...ted_Notes.html
Minor version updates eg v6.1 => v6.2 are supported...

lleb · 02-05-2013, 08:37 PM

if RH4 does not have ssh-copy-id, its no big deal.

1. create key as above, nothing changes here.
1a. might be a good idea to rename the id_rsa.pub file if there already is one on the backup server.
2. scp id_rsa.pub (MAKE SURE IT IS THE .pub AND NOTHING ELSE) to the backup server from the production box, or vis a vis depending on how you need the script to connect.
2a. scp id_rsa.pub backup_user@backup_server:/path/to/b.../home/dir/.ssh/
3. ssh into the backup server as the user who will be receiving the backup data.
4. cd ~/.ssh/
5. cat id_rsa.pub >> authorized_keys
6. verify permissions of the .ssh/ directory. should look something like below:

Code:

ssmas-imac:~ ssma$ d ~/.ssh
total 88
drwx------  11 ssma  staff         374 Jan  5 21:28 ./
drwxrwxr-x+ 93 ssma  _lpoperator  3162 Feb  5 16:23 ../
-rw-------   1 ssma  staff        4424 Jan  5 21:17 authorized_keys
-rw-r--r--   1 ssma  staff         175 Jan  5 21:28 config
-r--------   1 ssma  staff        3239 Jul 21  2012 id_rsa
-rw-r--r--   1 ssma  staff         741 Jan  5 21:17 id_rsa.cent.pub
-rw-r--r--   1 ssma  staff         752 Jul 21  2012 id_rsa.pub
-rw-r--r--   1 ssma  staff         752 Dec  8 00:33 id_rsa.pub.imac
-rw-r--r--   1 ssma  staff         752 Dec  4 16:10 id_rsa.pub.ssma
-rw-r--r--   1 ssma  staff         730 Dec  8 14:37 id_rsa.pub.web
-rw-r--r--   1 ssma  staff        5427 Jan 31 16:57 known_hosts

NOTE d on my server is an alias for ls -laF

once that is complete you should be able to ssh from the production box into the live server or vis a vis as required to the proper user.

as for the root ssh keys, that is a bit more complicated here is a good write up on how to handle that better:

http://www.jms1.net/code/rsync-backup.shtml

John Simpson really knows his stuff. there is a small script that ONLY allows this specific root ssh key to be used to rsync and nothing else. all other types of connections are terminated if they use that specific key.

FYI the paths are going to be different then yours as IIRC this was written either for Whitebox (no clue what vs) or CentOS 5.x (that could be any RHE from v3 - v5)

as chrism01 stated above it would be a great idea to upgrade those RH4 boxes ASAP. there are no further security patches or updates of any kind for that old of a system. Ideally you would perform a fresh install with either RHE6.x or use the free CentOS6.x (same OS without the copy protected name and logo) for performance and security.