Find, compress and move to new location but maintain folder structure

LDouglasLJr · 08-23-2019, 11:15 AM

CRON
L

Leichter Jr, Leo D. (CONTR) <leo.leichter@netl.doe.gov>
Fri 8/23/2019 12:12 PM
Inbox
To:
dleichter@hotmail.com;
Find, compress and move to new location but maintain folder structure

Was presented with something and I've kind of got it working but not completely and needing some help on the final part.

I have a folder structure /some/folder/structure/
Now after /structure/ there are more 12 sub-folders. Now each of these sub-folders contains in some instances many more sub-folders.

These are storing log files that are being collected from other systems.

Now I want to compress these log files after a number of days and then move them to an archive directory but I want to maintain this folder structure.

So I have a new archive area /new/archive/area/ and I have replicated the existing folder structure from /some/folder/structure/ to /new/archive/area/

I have a cron job that runs nightly to handle this compression/move.

Currently I have

echo "Beginning log compressions 'date +%Y-%m-%d:%H:%M:%S'"

cd /some/folder/structure/
find . / -type f -name "*.log" -mtime +5 -print -exe gzip -f {} \: -exec mv {}.gz /new/archive/area/\;

echo "End log compressions 'date +%Y-%m-%d:%H:%M:%S'"

But this is doing exactly what I feared it would and it's only creating these zipped files at the root of /new/archive/area/ instead of the adding these newly compressed files to the folder structure of each sub-folder.

Any help would be appreciated, thanks in advance.

Skaperen · 08-23-2019, 09:08 PM

try to get it to copy keeping the folder structure, not compressing, first. that will be the hard part. once you get that working where you are copying one file at a time, changing that to do compression will be easier.

rnturn · 08-25-2019, 10:43 AM

Quote:

Originally Posted by LDouglasLJr

<snip>

Find, compress and move to new location but maintain folder structure

<snip>

Now after /structure/ there are more 12 sub-folders. Now each of these sub-folders contains in some instances many more sub-folders.

These are storing log files that are being collected from other systems.

Now I want to compress these log files after a number of days and then move them to an archive directory but I want to maintain this folder structure.

<snip>

Currently I have

echo "Beginning log compressions 'date +%Y-%m-%d:%H:%M:%S'"

cd /some/folder/structure/
find . / -type f -name "*.log" -mtime +5 -print -exe gzip -f {} \: -exec mv {}.gz /new/archive/area/\;

echo "End log compressions 'date +%Y-%m-%d:%H:%M:%S'"

IMHO, this cries out for a process that performs each step rather than try to bend find(1) into doing it as a "one-liner". How do you do incremental testing on a complex one-liner? How does the next person maintain it? You know what you need to do for an arbitrary log file---just think about the steps involved in processing that one log file and implement those steps. Then, work out how you'll deal with the directory tree. Personally, I'd "cd" to "source-tree" and do a find at the top of the tree searching for directories. These'll be relative to "source-tree" which is perfect to append to "target-dir". Pass the relative directory path off to a script or shell function that does the compression and movement.

It may not be as impressive as bending find(1) into doing it with single line but it's a solution that you -- or that "next person" -- will be able to come back to in the future and understand how it works.

Good luck...

Firerat · 08-25-2019, 12:11 PM

untested

Code:

#!/bin/bash
echo "Beginning log compressions 'date +%Y-%m-%d:%H:%M:%S'"
Orig=/some/folder/structure/
New=/new/archive/area/

while read log;do
    # remove the leading dir structure
    log=${log#${Orig}}

    # if the new structure does not exist create it
    [[ -d ${New}${log%/*} ]] || mkdir -p ${New}${log%/*}
    # /!\ the above only checks if a dir exists
    # if we happen to have a file there strange things may happen

    # compress and move
    gzip ${Orig}${log} \
        && mv ${Orig}${log}.gz ${New}${log%/*}/${log}.gz \
        || echo "should do something about that fail"
done < <(find ${Orig} -type f -name "*.log" -mtime +5)

echo "End log compressions 'date +%Y-%m-%d:%H:%M:%S'"

far from perfect
I have already hinted at one problem.
You should check the exit status of mkdir before going on to the gzip and move. If you don't you will end up with compressed files in the original location ( at least that is the plan, I havn't tested the script )

good luck