Does a command, or a combination of commands exist that would move a bunch of files into a new directory and once that directory is full ...
 

Hey LQ,

So basically I've got a directory with like 30k .gz files in it. These are back up files.

I want to burn these to DISC media. What I normally do is browse the directory, keep selecting archive files until the file manager reports to me that I am nearing 4.5GB, then I CUT them, create a directory and name the directory with two dates, the first date is the date of the oldest archive I currently have CUT, the second date is the date of the newest archive I have CUT.

Naming of the directories is simple for me to do, I don't mind doing that manually, but is there a command that would do this cutting and creation of directories for me once I specify how big I want the directories to be?

I presume this is more in the realm of shell scripting rather than piping and chaining up commands together. I am prepared to do this as well if need be, but I recently learned about zgrep and was surprised someone had built a utility just for that purpose, so I figured I'd inquire.

Any tips in the right direction greatly appreciated as usual.

Cheers all.

even tar can split the created tar file into pieces.
Otherwise you can use du (for example) to check the size of a directory.

Great find!

 

I don't think datapacker can be helpful here - it doesn't have any date functionality, and appears to consider file size not disk usage, which matters when you're aggregating files to burn to DVD.

Wouldn't surprise me if a proper tool existed, but also wouldn't surprise me if most people just use a script.

Possibly start with something like:
This outputs a list of filenames (no directories) with their disk usage and last modified time (which the sort orders by), and this can then be looped through in Awk or Bash, summing and compiling the filenames until the threshold, logging the end/start dates, then repeating until all files are batched, which I'd then review before performing the operation.

When dealing with cases like this I often don't do file modification directly - instead I'll generate a script that does it, so I can manually review it before running - in case any oddities stick out. (An example in the above would be if there are any filenames with newlines in; the first half will deal with them, but the second half needs additional params if they are a possibility; such a situation can/should be guarded against in the script, but if that were overlooked, may manifest itself in an command that looks odd.)

 

On a separate note, title of this thread is not a good one: titles should be a concise description of what a thread is about.

The first half of this thread's title does not contribute anything to that description, but readers don't know that until they've got a dozen words in - some will have given up by then.

Like a ladder, it is best when they are long enough to get from one place to another, but not so long that it takes a prolonged crawl to get there.

Simplification is complicated

That's as many characters as the title field would let me put in there. Now that you've read the post, what would you have titled it as?

title 1 : How to efficiently burn lots of files into multiple DVDs
title 2 : How to move files to directories based on their total disk size

Again, it's complicated to simplify...

'Read before rant' maybe the solution here

Thanks for all the responses. I ended up writing a PHP script to do it all for me (I know PHP, don't know Bash at all haha).

Cheers.

That's ... not surprising. You should have taken that as a hint!

Again, titles should be concise, direct, succinct - not verbose.


As I already said, the entire first half of it is redundant - everything upto and including the word "would" could be removed and it would improve the title, but even that second part could use half the words and still convey the same detail: "move files into directories until full".

An even better one would communicate the overall problem, e.g. "batching files into DVD-sized groups".

Perfect opportunity for you to begin learning bash. :study:

Bash is after all the commonly used shell and everything you do on the command line (including launching php & probably writing your php script) begins with using bash.