LinuxQuestions.org
Review your favorite Linux distribution.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices


Reply
  Search this Thread
Old 03-18-2014, 04:20 AM   #1
Iyyappan
Member
 
Registered: Dec 2008
Location: Chennai, India
Distribution: CentOS 5, SLES 11
Posts: 245

Rep: Reputation: 4
rsync and tar files in remote server


Hi,
Currently have a script to rsync all the files from local to remote

rsync -zvr --exclude-from='/usr/local/sysadm/exclude-list.txt' /usr/local/apache24/ test@192.168.1.1:/tmp/Backup/$DATE

But the disk space usage is high in the remote server. To reduce space, I need to zip or tar all the files....

I need to rsync the files first and then remotes zip or tar all the folders....

I am using the below command to zip the files remotely

ssh test@192.168.1.1 gzip /tmp/Backup/$DATE/*

but I do not know how to zip the folders remotely. Also the zip or tar should remove the existing folders and only the zip or tar must be present in the remote server

Last edited by Iyyappan; 03-18-2014 at 04:32 AM.
 
Old 03-18-2014, 04:51 AM   #2
pan64
LQ Addict
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 21,842

Rep: Reputation: 7309Reputation: 7309Reputation: 7309Reputation: 7309Reputation: 7309Reputation: 7309Reputation: 7309Reputation: 7309Reputation: 7309Reputation: 7309Reputation: 7309
none of these tools will remove files during operation. That means after a successful tar/gzip/whatever can be the source files deleted.
if you want to rsync gzipped files you need to use a version of gzip with the option --rsyncable.
 
Old 03-18-2014, 04:53 AM   #3
kirukan
Senior Member
 
Registered: Jun 2008
Location: Eelam
Distribution: Redhat, Solaris, Suse
Posts: 1,278

Rep: Reputation: 148Reputation: 148
Why you execute the zip from remote. Instead, add a cronjob (find and zip all the files)
 
Old 03-18-2014, 05:13 AM   #4
Iyyappan
Member
 
Registered: Dec 2008
Location: Chennai, India
Distribution: CentOS 5, SLES 11
Posts: 245

Original Poster
Rep: Reputation: 4
I do not need all the folders to be zipped. I need only selected folders to be zipped/tar.

If I try to tar locally by excluding some folders, I would be left behind with the tar file, for this I need to put a separate cron to remove them, which I don't want. So I thought of zipping them remotely in the same script....
 
Old 03-18-2014, 06:30 AM   #5
linosaurusroot
Member
 
Registered: Oct 2012
Distribution: OpenSuSE,RHEL,Fedora,OpenBSD
Posts: 982
Blog Entries: 2

Rep: Reputation: 244Reputation: 244Reputation: 244
What about rsnapshot which will avoid duplication between the backups taken on different days?
 
Old 04-18-2014, 06:13 AM   #6
ruario
Senior Member
 
Registered: Jan 2011
Location: Oslo, Norway
Distribution: Slackware
Posts: 2,557

Rep: Reputation: 1762Reputation: 1762Reputation: 1762Reputation: 1762Reputation: 1762Reputation: 1762Reputation: 1762Reputation: 1762Reputation: 1762Reputation: 1762Reputation: 1762
Quote:
Originally Posted by Iyyappan View Post
ssh test@192.168.1.1 gzip /tmp/Backup/$DATE/*

but I do not know how to zip the folders remotely. Also the zip or tar should remove the existing folders and only the zip or tar must be present in the remote server
If you use zip rather than gzip, you could have it remove everything, e.g.

Code:
ssh test@192.168.1.1 "zip -r --move /tmp/Backup/${DATE}.zip /tmp/Backup/$DATE"
The zip archive /tmp/Backup/${DATE}.zip would contain everything that was in /tmp/Backup/$DATE previously. After the zip command is finished /tmp/Backup/$DATE and its subdirectories and files would be removed.

Last edited by ruario; 04-18-2014 at 06:21 AM.
 
Old 04-18-2014, 06:19 AM   #7
ruario
Senior Member
 
Registered: Jan 2011
Location: Oslo, Norway
Distribution: Slackware
Posts: 2,557

Rep: Reputation: 1762Reputation: 1762Reputation: 1762Reputation: 1762Reputation: 1762Reputation: 1762Reputation: 1762Reputation: 1762Reputation: 1762Reputation: 1762Reputation: 1762
Alternatively if you just want to use gzip and do not care about everything being in a single archive, you could:

Code:
ssh test@192.168.1.1 "gzip -r /tmp/Backup/$DATE"
For better space saving use xz instead:

Code:
ssh test@192.168.1.1 "find /tmp/Backup/$DATE -type f -exec xz {} \;"
EDIT: Actually, given this is a backup I would use bzip2 rather than gzip or xz. Although the result will not be as small as xz compression, bzip2 has better recovery tools (in the form of bzip2recover), should any of the files get damaged or corrupted in the future.

Code:
ssh test@192.168.1.1 "find /tmp/Backup/$DATE -type f -exec bzip2 {} \;"
P.S. I'd also recommend lbzip2 instead of bzip2 for a quicker result.

Last edited by ruario; 04-18-2014 at 06:53 AM.
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
rsync files recursively but place in one remote directory wreckedred Linux - Newbie 3 01-30-2011 07:17 AM
Could I run rsync to download files from a server without rsync daemon? Richard.Yang Linux - Software 1 09-18-2009 04:08 AM
Rsync connecting to a remote server Rudy Linux - Software 8 08-28-2009 12:09 PM
rsync not delete non existing files on remote server proNick Linux - Software 2 09-10-2008 03:47 AM
LXer: Hands on: Back up files using the tar command and rsync LXer Syndicated Linux News 0 07-04-2006 03:54 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Server

All times are GMT -5. The time now is 10:11 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration