LinuxQuestions.org
Register a domain and help support LQ
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices

Reply
 
Search this Thread
Old 04-24-2013, 12:52 AM   #1
enkrypted
LQ Newbie
 
Registered: Apr 2013
Posts: 22

Rep: Reputation: Disabled
Archive a directory without using additional disk space?


I have a VPS with 10 GB disk space, and 8.5 GB is occupied by multiple files, that I want to transfer to my computer and then delete from the vps.
It takes a lot of time to download the files one at a time, and then rearranging them into the directory/subdirectory structure they originally had.
So I want to be able to make an archive of these files(all of them are within a single directory), without using any additional disk space. Like maybe simply convert the directory into an archive, or maybe archive a file then delete it, and so on.
Is that possible? compression does not matter that much to me.
Thanks in advance
 
Old 04-24-2013, 01:08 AM   #2
evo2
Guru
 
Registered: Jan 2009
Location: Japan
Distribution: Mostly Debian and Scientific Linux
Posts: 5,447

Rep: Reputation: 1204Reputation: 1204Reputation: 1204Reputation: 1204Reputation: 1204Reputation: 1204Reputation: 1204Reputation: 1204Reputation: 1204
Hi,

why do you need to archive them to get them off the vps? Have you considered using something like rsync?
Code:
rsync -auzv -e ssh myvps:/path/to/files/to/archive .
The above command would be run on the target machine, "myvps" is the hostname or ip of your vps, and the directory structure of /path/to/files/to/archive on the vps would be synced to the directory "archive" in current working directory. The -z options means the data is compressed before transfer (and decompressed at the destination) and the "-e ssh" (which is probably default anyway) means the authentication etc is all done using ssh. Once you've got what you want from the vps you can delete the originals.

Evo2.
 
Old 04-24-2013, 01:12 AM   #3
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,492
Blog Entries: 54

Rep: Reputation: 2904Reputation: 2904Reputation: 2904Reputation: 2904Reputation: 2904Reputation: 2904Reputation: 2904Reputation: 2904Reputation: 2904Reputation: 2904Reputation: 2904
Quote:
Originally Posted by enkrypted View Post
So I want to be able to make an archive of these files(all of them are within a single directory), without using any additional disk space. Like maybe simply convert the directory into an archive, or maybe archive a file then delete it, and so on. Is that possible? compression does not matter that much to me.
While you can use tar w/o compression and use the --delete-files switch this does not make sense as the archive file will take up as much space as the original. Besides the smaller the archive the less D/L time it will take. So compression does matter. What you could do is try and find out which files benefit from compression the most (plain ASCII files) and see what you gain from compressing those first. One other thing that may come in handy is a tmpfs scratch disk, tho at 8G you'll still better archive large files separately to avoid it halting on lack of scratch space.


Quote:
Originally Posted by enkrypted View Post
It takes a lot of time to download the files one at a time, and then rearranging them into the directory/subdirectory structure they originally had.
...but rsync will accomplish all of that. Just start it and fetch some $beverage ...

Last edited by unSpawn; 04-24-2013 at 01:13 AM.
 
1 members found this post helpful.
Old 04-24-2013, 01:21 AM   #4
eklavya
Member
 
Registered: Mar 2013
Posts: 574

Rep: Reputation: 117Reputation: 117
If you want to copy the directory structure of your archive directory, not the files
Code:
rsync -av --include '*/' --exclude '*' /path/of/the/source/folder /path/of/the/destination/folder
If you want to copy the files only, not the directory structure
Code:
find /path/of/the/source/folder -type f -exec cp {} /path/of/the/destination/folder \;
 
Old 04-24-2013, 02:52 AM   #5
enkrypted
LQ Newbie
 
Registered: Apr 2013
Posts: 22

Original Poster
Rep: Reputation: Disabled
I have a relatively slow internet connection at home. My maximum download speed is ~400 kbps.
Also, is rsync possible on Windows? And do I need to install it on the vps as well?
I want to convert it into an archive because that way I would be able to transfer it in one go. I like to use IDM, as that gives me the highest speed.
Also, these files are media files. Videos to be specific.
Thanks for the help. I'll try both the methods.
 
Old 04-24-2013, 03:59 AM   #6
pan64
Senior Member
 
Registered: Mar 2012
Location: Hungary
Distribution: debian i686 (solaris)
Posts: 4,950

Rep: Reputation: 1309Reputation: 1309Reputation: 1309Reputation: 1309Reputation: 1309Reputation: 1309Reputation: 1309Reputation: 1309Reputation: 1309Reputation: 1309
rsync is definitely usable on slow networks, available on windows too. it should not be installed at all, because it consists of 2 or 3 files altogether and you can put them anywhere you want (but of course you can install it).
rsync is able to transfer only differences.
 
Old 04-24-2013, 04:06 AM   #7
enkrypted
LQ Newbie
 
Registered: Apr 2013
Posts: 22

Original Poster
Rep: Reputation: Disabled
So would I be able to get my maximum speed using rsync?
 
Old 04-24-2013, 04:08 AM   #8
pan64
Senior Member
 
Registered: Mar 2012
Location: Hungary
Distribution: debian i686 (solaris)
Posts: 4,950

Rep: Reputation: 1309Reputation: 1309Reputation: 1309Reputation: 1309Reputation: 1309Reputation: 1309Reputation: 1309Reputation: 1309Reputation: 1309Reputation: 1309
I have no idea about your config, but probably yes.
 
Old 04-24-2013, 04:09 AM   #9
enkrypted
LQ Newbie
 
Registered: Apr 2013
Posts: 22

Original Poster
Rep: Reputation: Disabled
Ok, I'll try it soon
 
Old 04-24-2013, 05:09 PM   #10
evo2
Guru
 
Registered: Jan 2009
Location: Japan
Distribution: Mostly Debian and Scientific Linux
Posts: 5,447

Rep: Reputation: 1204Reputation: 1204Reputation: 1204Reputation: 1204Reputation: 1204Reputation: 1204Reputation: 1204Reputation: 1204Reputation: 1204
Hi,

Quote:
Originally Posted by enkrypted View Post
I want to convert it into an archive because that way I would be able to transfer it in one go.
They don't need to be in an archive to do that. Additionally, with rsync, if the transfer is interrupted for some reason you can pick up where you left off.

Quote:
I like to use IDM, as that gives me the highest speed.
What is IDM ("apt-cache search idm" doesn't yield anything relevant).

Quote:
Also, these files are media files. Videos to be specific.
Ok, in that case you probably won't gain anything by trying to compress the transfer (ie don't use the -z option)

Cheers,

Evo2.
 
Old 04-25-2013, 04:28 AM   #11
enkrypted
LQ Newbie
 
Registered: Apr 2013
Posts: 22

Original Poster
Rep: Reputation: Disabled
I want to archive the files, not necessarily compress them.Internet Download Manager(IDM) is a Windows software for boosting downloads.
 
Old 04-25-2013, 04:52 AM   #12
pan64
Senior Member
 
Registered: Mar 2012
Location: Hungary
Distribution: debian i686 (solaris)
Posts: 4,950

Rep: Reputation: 1309Reputation: 1309Reputation: 1309Reputation: 1309Reputation: 1309Reputation: 1309Reputation: 1309Reputation: 1309Reputation: 1309Reputation: 1309
have you tried rsync? it can really do that job.
 
Old 04-25-2013, 05:09 AM   #13
enkrypted
LQ Newbie
 
Registered: Apr 2013
Posts: 22

Original Poster
Rep: Reputation: Disabled
I will try it tomorrow, when I have more files.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Dual booting with little disk space. Two distros sharing home directory?[TEMP SOLVED] searching_for_answers Linux - Hardware 14 10-13-2009 12:25 PM
Check disk space used in a directory cli_man Linux - General 5 03-13-2009 03:28 PM
Cedega "Disk space required for the installation exceeds available disk space" Solved Spewdemon LinuxQuestions.org Member Success Stories 1 10-18-2007 06:19 PM
Root (/) partition resized, but I can't used the additional space. dickgregory Linux - General 2 03-26-2006 09:24 AM
3Gb of disk space lost! Disk space problem or mother board conflicts with HDD Mistreated Linux - Hardware 4 12-06-2004 03:58 PM


All times are GMT -5. The time now is 11:28 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration