How can I transfer 12GB data to new server

landysaccount · 01-07-2013, 12:23 PM

Hello.

I'm in the middle of changing our server. We bought server space at a different location and need to transfer all the content from our current server to the new one. I need to move 12GB over to the new server. What would be the most convenient, fastest, and easier way to transfer all that data to the new server... Doing a "physical" transfer is not an option.

I know is going to be a loooong procedure. I'm testing with some data and is taking a long long time.

Any ideas?

Thanks in advanced for your time and input.

pan64 · 01-07-2013, 12:48 PM

it depends on the type of the data. Several (huge amount of) small files can be easily transferred with rsync. Text files can be compressed, so you can lower the amount of data to push over the line.

jefro · 01-07-2013, 01:23 PM

Might be worth it to decide on best compression method that can be used. It may depend more on what tools you have access to. Normal way is to ftp it all from one host to another.

landysaccount · 01-07-2013, 09:00 PM

I'm actually hosting about 10 websites on our local server so, I would like to move all that data to the new server... I do a backup of all the important files: /var/www, /home/, /var/lib/mysql, and others with tar czf and it results into a 12GB tarball.

I don't have a big upload pipe so, I need to do a reliable transfer during off peak hours, 1am to 6am...

Thanks.

landysaccount · 01-07-2013, 09:46 PM

Hello.

Just found an option on how to do this and hope it can help others.

Use rsync:

rsync -avzrhP localpath usrname@remote:remotepath

I'm testing this now.

Thanks.

TenTenths · 01-08-2013, 05:37 AM

You should also consider doing a mysqldump of the databases and compressing/transfering that and re-loading it in to the new MySQL instance.

Consider a phased migration, doing one site at a time rather than an "all-in-one" switch.

sneakyimp · 01-10-2013, 02:50 AM

rsync rules -- and scp is perhaps a bit simpler. Make sure you login to the new server first and then use that to login to the old server -- otherwise there might be some vestige of your new server's password(s) left in memory on the old server.

chrism01 · 01-10-2013, 02:57 AM

If its going across a public link, I'd use scp.
imho, rsync is overkill, because its main claim to fame is sending diffs, but for a one-off as here, that's redundant, not to mention the time taken to calc the diffs...

lleb · 01-10-2013, 05:58 PM

Quote:

Originally Posted by chrism01

If its going across a public link, I'd use scp.
imho, rsync is overkill, because its main claim to fame is sending diffs, but for a one-off as here, that's redundant, not to mention the time taken to calc the diffs...

ive used rsync and scp and find them to be both about the same speed. negligible difference when you are talking the amount of data he is transferring, not to mention at least with rsync if there is an interruption you can easily resume. you can not do that with scp.

really other then rsync the only other option i could think of is wget -m as it works the same as rysnc for recovery, but does not have the overhead of rsync.

sneakyimp · 01-10-2013, 06:11 PM

Are rsync and wget encrypted? Unless I'm mistaken, wget only supports HTTP/HTTPS and FTP. The former would require you to move the file your web root and transfer via apache. The latter would require insecure transmission via FTP.

pan64 · 01-11-2013, 12:14 AM

rsync can be executed over an ssh tunnel, if you want to do so. https is encrypted too.
and also rsync has a compression algorithm to minimize the traffic.

lleb · 01-12-2013, 11:32 AM

Quote:

Originally Posted by sneakyimp

Are rsync and wget encrypted? Unless I'm mistaken, wget only supports HTTP/HTTPS and FTP. The former would require you to move the file your web root and transfer via apache. The latter would require insecure transmission via FTP.

typically when you rsync between two systems you setup ssh keys and use that for the transfer security.

Code:

###### DIRECTIONS FOR CREATING RSA KEY################

	Directions for creating the rsa key and making the two
 servers talk to each other without password.

	1st change directory into .ssh and check what files are there.

 		[user@user ~]$ cd .ssh
		[user@user .ssh]$ ls -l
		total 4
		-rw-r--r-- 1 user group 2980 Jun 13 12:02 known_hosts

	2nd create the rsa key.

		[user@user .ssh]$ ssh-keygen -t rsa -b 4096
		Generating public/private rsa key pair.
		Enter file in which to save the key (/usr/user/.ssh/id_rsa):
		Enter passphrase (empty for no passphrase):
		Enter same passphrase again:
		Your identification has been saved in /usr/user/.ssh/id_rsa.
		Your public key has been saved in /usr/user/.ssh/id_rsa.pub.
		The key fingerprint is:
		cb:b0:40:c6:e9:f4:9e:f5:71:fc:c3:00:c0:f7:c6:75 user@user.localdomain

	3rd check that there are two new files with the following permissions

		[user@user .ssh]$ ls -l
		total 12
		-rw------- 1 user group 3243 Jun 22 15:50 id_rsa
		-rw-r--r-- 1 user group  743 Jun 22 15:50 id_rsa.pub
		-rw-r--r-- 1 user group 2980 Jun 13 12:02 known_hosts
	
	4th change directory back to the users $HOME

		[user@user .ssh]$ cd

	5th copy the key to the remote server

		[user@user ~]$ ssh-copy-id -i ~/.ssh/id_rsa.pub user@XXX.XXX.XXX.XXX
		25
		user@XXX.XXX.XXX.XXX's password:
		Now try logging into the machine, with "ssh 'user@XXX.XXX.XXX.XXX'", and check in:

		.ssh/authorized_keys

		to make sure we haven't added extra keys that you weren't expecting.

	6th, follow directions on the screen.

		[user@user ~]$ ssh user@XXX.XXX.XXX.XXX
		Last login: Fri Jun 22 14:12:08 2012 from 10.10.4.77
		[user@user ~]$ exit
		logout
		Connection to XXX.XXX.XXX.XXX closed.

then when you run your rsync it will look like the following:

Code:

rsync -aviS /source_files/ -e ssh user@remote_IP:/destination/

the -e tells rsync to execute the following command, in this case ssh and as you already setup the ssh keys it will be able to perform the handshake without user interaction thus allowing it to be scripted for cron jobs while maintaining security.