Linux - GeneralThis Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
(Hopefully) use tar to create an archive of size x. If the size of the actual file (y) is bigger than x, then split the archive into parts.
The tricky part of this, though, is being able to extract files from an archive without having to have every part there. So basically if I make a back up of a folder that has 3 files, "bob.avi" (2 MB), "alice.mpeg" (5 MB) and "tony.avi" (50 MB), and the archive was split into parts, I could still at least run tar -tf on archive1.tar and see if a file is there.
I know you can split archives in tar by using -M & -L (i.e.: tar -M -L 102400 -cf archive.tar videos/*). But if I run tar -tf on any of the generated archives, it says unexpected EOF. Which, I know why it does, but I'm not just not sure if there's a way around this.
Secondly, I know I could write a script and stat each file it's trying to archive, but stat calls can be intensive on huge backups and isn't very clean in a way.
Tar command |split =size | mkisofs stuff |cdrecord stuff
or whatever your choice is.
Home Movies? BTW I've seen people put away terabytes of home movies, but I've never seen them open the archives again and spend days/weeks looking at them. My son in law sent a gigabyte of useless footage on his new daughter :-/. Am I going to watch it a second time?
A better approach might be to sort things by date, shrink file size (mpeg instead of avi) and categorise them. "Victor's wedding 2001 where the cake blew up"
Tar command |split =size | mkisofs stuff |cdrecord stuff
or whatever your choice is.
Home Movies? BTW I've seen people put away terabytes of home movies, but I've never seen them open the archives again and spend days/weeks looking at them. My son in law sent a gigabyte of useless footage on his new daughter :-/. Am I going to watch it a second time?
A better approach might be to sort things by date, shrink file size (mpeg instead of avi) and categorise them. "Victor's wedding 2001 where the cake blew up"
Actually this is for system backups, not movies, I was just using that as an example.
I don't want to use split as it's no different than using tar with the -M & -L switches, you need the entire archive to get a file from one of the parts. This is what I'm trying to avoid.
To quote the Kerryman when asked for directions by a tourist:
"If I was you, I wouldn't start from here at all!"
What size is the archive you're creating? Why must it be one archive, and not many?
Once you put number out, solutions will suggest themselves.
Size would ideally be 10GB max, since disk space on the provided hardware isn't much (~50 GB VPSes). Archives because it's already implemented that way in the backup script. I could do it otherwise but it would be a very big hassle (worse than this I'm sure) to do so, but it is hard to explain unless you've used ObjectStorage (from Soft Layer) before, as I'm using that for the storage.
10GB is one hell of an archive.
The only guy I knew handling that sort of data used no archives and complete hard disks. It was cheapest. He had boxes of 500MB ide and 1TB sata (Probably 3 or 4 TB now), and they were his archives. Each one was backed up on another disk, needless to say. His output was disks full of tiff files(for integrity) and jpegs (for everyday use).
He had the knack of hot swapping drives (in windows), a trick I never was tempted to imitate, although I did do it once to prove I had the technique correct. You can probably pull it in linux too, if you mount -noatime or unmount.
10GB is one hell of an archive.
The only guy I knew handling that sort of data used no archives and complete hard disks. It was cheapest. He had boxes of 500MB ide and 1TB sata (Probably 3 or 4 TB now), and they were his archives. Each one was backed up on another disk, needless to say. His output was disks full of tiff files(for integrity) and jpegs (for everyday use).
He had the knack of hot swapping drives (in windows), a trick I never was tempted to imitate, although I did do it once to prove I had the technique correct. You can probably pull it in linux too, if you mount -noatime or unmount.
Well I was using 10GB as an example. I really just want to know if there a solution to this, and if so, what it is.
There is, but it's not an archive. Any archive format I am aware of has a part without which the whole archive is knackered. There is zipfix and equivalents, but that's messing
The solution is called copying. Archiving is also cpu intensive. look at the output of
There is, but it's not an archive. Any archive format I am aware of has a part without which the whole archive is knackered. There is zipfix and equivalents, but that's messing
The solution is called copying. Archiving is also cpu intensive. look at the output of
time bzip2 /pat/to/10G
While it might be intensive, the thing is if it's done during downtime then it wouldn't be as much (if at all) a problem.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.