LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices


Reply
  Search this Thread
Old 08-23-2019, 11:15 AM   #1
LDouglasLJr
LQ Newbie
 
Registered: Oct 2011
Posts: 4

Rep: Reputation: Disabled
Find, compress and move to new location but maintain folder structure


CRON
L

Leichter Jr, Leo D. (CONTR) <leo.leichter@netl.doe.gov>
Fri 8/23/2019 12:12 PM
Inbox
To:
dleichter@hotmail.com;
Find, compress and move to new location but maintain folder structure

Was presented with something and I've kind of got it working but not completely and needing some help on the final part.

I have a folder structure /some/folder/structure/
Now after /structure/ there are more 12 sub-folders. Now each of these sub-folders contains in some instances many more sub-folders.

These are storing log files that are being collected from other systems.

Now I want to compress these log files after a number of days and then move them to an archive directory but I want to maintain this folder structure.

So I have a new archive area /new/archive/area/ and I have replicated the existing folder structure from /some/folder/structure/ to /new/archive/area/

I have a cron job that runs nightly to handle this compression/move.

Currently I have

echo "Beginning log compressions 'date +%Y-%m-%d:%H:%M:%S'"

cd /some/folder/structure/
find . / -type f -name "*.log" -mtime +5 -print -exe gzip -f {} \: -exec mv {}.gz /new/archive/area/\;

echo "End log compressions 'date +%Y-%m-%d:%H:%M:%S'"

But this is doing exactly what I feared it would and it's only creating these zipped files at the root of /new/archive/area/ instead of the adding these newly compressed files to the folder structure of each sub-folder.

Any help would be appreciated, thanks in advance.
 
Old 08-23-2019, 09:08 PM   #2
Skaperen
Senior Member
 
Registered: May 2009
Location: WV, USA
Distribution: Xubuntu, Slackware, Amazon Linux
Posts: 1,903
Blog Entries: 21

Rep: Reputation: 126Reputation: 126
try to get it to copy keeping the folder structure, not compressing, first. that will be the hard part. once you get that working where you are copying one file at a time, changing that to do compression will be easier.
 
Old 08-25-2019, 10:43 AM   #3
rnturn
Senior Member
 
Registered: Jan 2003
Location: Illinois (Chicago area)
Distribution: CentOS, MacOS, [Open]SuSE, Raspian, Red Hat, Slackware, Solaris, Tru64
Posts: 1,504

Rep: Reputation: 167Reputation: 167
Quote:
Originally Posted by LDouglasLJr View Post
<snip>

Find, compress and move to new location but maintain folder structure

<snip>

Now after /structure/ there are more 12 sub-folders. Now each of these sub-folders contains in some instances many more sub-folders.

These are storing log files that are being collected from other systems.

Now I want to compress these log files after a number of days and then move them to an archive directory but I want to maintain this folder structure.

<snip>

Currently I have

echo "Beginning log compressions 'date +%Y-%m-%d:%H:%M:%S'"

cd /some/folder/structure/
find . / -type f -name "*.log" -mtime +5 -print -exe gzip -f {} \: -exec mv {}.gz /new/archive/area/\;

echo "End log compressions 'date +%Y-%m-%d:%H:%M:%S'"
IMHO, this cries out for a process that performs each step rather than try to bend find(1) into doing it as a "one-liner". How do you do incremental testing on a complex one-liner? How does the next person maintain it? You know what you need to do for an arbitrary log file---just think about the steps involved in processing that one log file and implement those steps. Then, work out how you'll deal with the directory tree. Personally, I'd "cd" to "source-tree" and do a find at the top of the tree searching for directories. These'll be relative to "source-tree" which is perfect to append to "target-dir". Pass the relative directory path off to a script or shell function that does the compression and movement.

It may not be as impressive as bending find(1) into doing it with single line but it's a solution that you -- or that "next person" -- will be able to come back to in the future and understand how it works.

Good luck...
 
Old 08-25-2019, 12:11 PM   #4
Firerat
Senior Member
 
Registered: Oct 2008
Distribution: Debian sid
Posts: 2,391

Rep: Reputation: 653Reputation: 653Reputation: 653Reputation: 653Reputation: 653Reputation: 653
untested

Code:
#!/bin/bash
echo "Beginning log compressions 'date +%Y-%m-%d:%H:%M:%S'"
Orig=/some/folder/structure/
New=/new/archive/area/

while read log;do
    # remove the leading dir structure
    log=${log#${Orig}}

    # if the new structure does not exist create it
    [[ -d ${New}${log%/*} ]] || mkdir -p ${New}${log%/*}
    # /!\ the above only checks if a dir exists
    # if we happen to have a file there strange things may happen

    # compress and move
    gzip ${Orig}${log} \
        && mv ${Orig}${log}.gz ${New}${log%/*}/${log}.gz \
        || echo "should do something about that fail"
done < <(find ${Orig} -type f -name "*.log" -mtime +5)

echo "End log compressions 'date +%Y-%m-%d:%H:%M:%S'"
far from perfect
I have already hinted at one problem.
You should check the exit status of mkdir before going on to the gzip and move. If you don't you will end up with compressed files in the original location ( at least that is the plan, I havn't tested the script )

good luck
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: Google I/O Android News: Location, Location, Location (Plus Cloud Messaging and Bluetooth) LXer Syndicated Linux News 0 06-05-2013 01:00 PM
Compress several files without directory structure srg46srg Linux - Newbie 1 12-04-2012 07:01 AM
How to compress and move log older than 30 days from one folder to another LittleMaster Linux - Newbie 6 09-20-2012 12:30 AM
Home Jail Folder Structure like Gobolinux Directory Structure luispt Linux - General 3 07-26-2008 06:46 PM
location, location, location! mermxx LQ Suggestions & Feedback 9 09-25-2004 03:08 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - General

All times are GMT -5. The time now is 04:46 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration