LinuxQuestions.org
Register a domain and help support LQ
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices

Reply
 
Search this Thread
Old 01-07-2013, 12:23 PM   #1
landysaccount
Member
 
Registered: Sep 2008
Location: Dominican Republic
Distribution: Debian Squeeze
Posts: 177

Rep: Reputation: 17
How can I transfer 12GB data to new server


Hello.

I'm in the middle of changing our server. We bought server space at a different location and need to transfer all the content from our current server to the new one. I need to move 12GB over to the new server. What would be the most convenient, fastest, and easier way to transfer all that data to the new server... Doing a "physical" transfer is not an option.

I know is going to be a loooong procedure. I'm testing with some data and is taking a long long time.

Any ideas?

Thanks in advanced for your time and input.

Last edited by landysaccount; 01-07-2013 at 12:24 PM.
 
Old 01-07-2013, 12:48 PM   #2
pan64
Senior Member
 
Registered: Mar 2012
Location: Hungary
Distribution: debian i686 (solaris)
Posts: 4,929

Rep: Reputation: 1306Reputation: 1306Reputation: 1306Reputation: 1306Reputation: 1306Reputation: 1306Reputation: 1306Reputation: 1306Reputation: 1306Reputation: 1306
it depends on the type of the data. Several (huge amount of) small files can be easily transferred with rsync. Text files can be compressed, so you can lower the amount of data to push over the line.
 
1 members found this post helpful.
Old 01-07-2013, 01:23 PM   #3
jefro
Guru
 
Registered: Mar 2008
Posts: 11,755

Rep: Reputation: 1446Reputation: 1446Reputation: 1446Reputation: 1446Reputation: 1446Reputation: 1446Reputation: 1446Reputation: 1446Reputation: 1446Reputation: 1446
Might be worth it to decide on best compression method that can be used. It may depend more on what tools you have access to. Normal way is to ftp it all from one host to another.
 
Old 01-07-2013, 09:00 PM   #4
landysaccount
Member
 
Registered: Sep 2008
Location: Dominican Republic
Distribution: Debian Squeeze
Posts: 177

Original Poster
Rep: Reputation: 17
I'm actually hosting about 10 websites on our local server so, I would like to move all that data to the new server... I do a backup of all the important files: /var/www, /home/, /var/lib/mysql, and others with tar czf and it results into a 12GB tarball.

I don't have a big upload pipe so, I need to do a reliable transfer during off peak hours, 1am to 6am...

Thanks.
 
Old 01-07-2013, 09:46 PM   #5
landysaccount
Member
 
Registered: Sep 2008
Location: Dominican Republic
Distribution: Debian Squeeze
Posts: 177

Original Poster
Rep: Reputation: 17
Hello.

Just found an option on how to do this and hope it can help others.

Use rsync:

rsync -avzrhP localpath usrname@remote:remotepath

I'm testing this now.

Thanks.
 
1 members found this post helpful.
Old 01-08-2013, 05:37 AM   #6
TenTenths
Senior Member
 
Registered: Aug 2011
Location: Dublin
Distribution: Centos 5 / 6
Posts: 1,529

Rep: Reputation: 472Reputation: 472Reputation: 472Reputation: 472Reputation: 472
You should also consider doing a mysqldump of the databases and compressing/transfering that and re-loading it in to the new MySQL instance.

Consider a phased migration, doing one site at a time rather than an "all-in-one" switch.
 
Old 01-10-2013, 02:50 AM   #7
sneakyimp
Member
 
Registered: Dec 2004
Posts: 795

Rep: Reputation: 50
rsync rules -- and scp is perhaps a bit simpler. Make sure you login to the new server first and then use that to login to the old server -- otherwise there might be some vestige of your new server's password(s) left in memory on the old server.
 
Old 01-10-2013, 02:57 AM   #8
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.5, Centos 5.10
Posts: 16,289

Rep: Reputation: 2034Reputation: 2034Reputation: 2034Reputation: 2034Reputation: 2034Reputation: 2034Reputation: 2034Reputation: 2034Reputation: 2034Reputation: 2034Reputation: 2034
If its going across a public link, I'd use scp.
imho, rsync is overkill, because its main claim to fame is sending diffs, but for a one-off as here, that's redundant, not to mention the time taken to calc the diffs...
 
Old 01-10-2013, 05:58 PM   #9
lleb
Senior Member
 
Registered: Dec 2005
Location: Florida
Distribution: CentOS/Fedora
Posts: 2,556

Rep: Reputation: 463Reputation: 463Reputation: 463Reputation: 463Reputation: 463
Quote:
Originally Posted by chrism01 View Post
If its going across a public link, I'd use scp.
imho, rsync is overkill, because its main claim to fame is sending diffs, but for a one-off as here, that's redundant, not to mention the time taken to calc the diffs...
ive used rsync and scp and find them to be both about the same speed. negligible difference when you are talking the amount of data he is transferring, not to mention at least with rsync if there is an interruption you can easily resume. you can not do that with scp.

really other then rsync the only other option i could think of is wget -m as it works the same as rysnc for recovery, but does not have the overhead of rsync.
 
Old 01-10-2013, 06:11 PM   #10
sneakyimp
Member
 
Registered: Dec 2004
Posts: 795

Rep: Reputation: 50
Are rsync and wget encrypted? Unless I'm mistaken, wget only supports HTTP/HTTPS and FTP. The former would require you to move the file your web root and transfer via apache. The latter would require insecure transmission via FTP.
 
1 members found this post helpful.
Old 01-11-2013, 12:14 AM   #11
pan64
Senior Member
 
Registered: Mar 2012
Location: Hungary
Distribution: debian i686 (solaris)
Posts: 4,929

Rep: Reputation: 1306Reputation: 1306Reputation: 1306Reputation: 1306Reputation: 1306Reputation: 1306Reputation: 1306Reputation: 1306Reputation: 1306Reputation: 1306
rsync can be executed over an ssh tunnel, if you want to do so. https is encrypted too.
and also rsync has a compression algorithm to minimize the traffic.

Last edited by pan64; 01-11-2013 at 12:20 AM.
 
1 members found this post helpful.
Old 01-12-2013, 11:32 AM   #12
lleb
Senior Member
 
Registered: Dec 2005
Location: Florida
Distribution: CentOS/Fedora
Posts: 2,556

Rep: Reputation: 463Reputation: 463Reputation: 463Reputation: 463Reputation: 463
Quote:
Originally Posted by sneakyimp View Post
Are rsync and wget encrypted? Unless I'm mistaken, wget only supports HTTP/HTTPS and FTP. The former would require you to move the file your web root and transfer via apache. The latter would require insecure transmission via FTP.

typically when you rsync between two systems you setup ssh keys and use that for the transfer security.

Code:
###### DIRECTIONS FOR CREATING RSA KEY################

	Directions for creating the rsa key and making the two
 servers talk to each other without password.

	1st change directory into .ssh and check what files are there.

 		[user@user ~]$ cd .ssh
		[user@user .ssh]$ ls -l
		total 4
		-rw-r--r-- 1 user group 2980 Jun 13 12:02 known_hosts

	2nd create the rsa key.

		[user@user .ssh]$ ssh-keygen -t rsa -b 4096
		Generating public/private rsa key pair.
		Enter file in which to save the key (/usr/user/.ssh/id_rsa):
		Enter passphrase (empty for no passphrase):
		Enter same passphrase again:
		Your identification has been saved in /usr/user/.ssh/id_rsa.
		Your public key has been saved in /usr/user/.ssh/id_rsa.pub.
		The key fingerprint is:
		cb:b0:40:c6:e9:f4:9e:f5:71:fc:c3:00:c0:f7:c6:75 user@user.localdomain

	3rd check that there are two new files with the following permissions

		[user@user .ssh]$ ls -l
		total 12
		-rw------- 1 user group 3243 Jun 22 15:50 id_rsa
		-rw-r--r-- 1 user group  743 Jun 22 15:50 id_rsa.pub
		-rw-r--r-- 1 user group 2980 Jun 13 12:02 known_hosts
	
	4th change directory back to the users $HOME

		[user@user .ssh]$ cd

	5th copy the key to the remote server

		[user@user ~]$ ssh-copy-id -i ~/.ssh/id_rsa.pub user@XXX.XXX.XXX.XXX
		25
		user@XXX.XXX.XXX.XXX's password:
		Now try logging into the machine, with "ssh 'user@XXX.XXX.XXX.XXX'", and check in:

		.ssh/authorized_keys

		to make sure we haven't added extra keys that you weren't expecting.

	6th, follow directions on the screen.

		[user@user ~]$ ssh user@XXX.XXX.XXX.XXX
		Last login: Fri Jun 22 14:12:08 2012 from 10.10.4.77
		[user@user ~]$ exit
		logout
		Connection to XXX.XXX.XXX.XXX closed.
then when you run your rsync it will look like the following:

Code:
rsync -aviS /source_files/ -e ssh user@remote_IP:/destination/
the -e tells rsync to execute the following command, in this case ssh and as you already setup the ssh keys it will be able to perform the handshake without user interaction thus allowing it to be scripted for cron jobs while maintaining security.
 
1 members found this post helpful.
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
How to transfer hexadecimal data from client to server(socket programming)..? vktafs Programming 1 11-05-2012 05:50 PM
LXer: MySQL Data Transfer using Sql Server Integration Services LXer Syndicated Linux News 0 09-02-2009 06:40 AM
memcpy_toio transfers data in 4 byte chunks, but I need to transfer data in one lump. jbreaka4lyfe Linux - Embedded & Single-board computer 2 06-02-2008 11:25 AM
data transfer from windows server abhattacharya Linux - Software 1 05-15-2008 04:58 AM


All times are GMT -5. The time now is 12:10 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration