LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices


Reply
  Search this Thread
Old 11-02-2009, 02:40 PM   #1
Cerephim
Member
 
Registered: Mar 2005
Location: Dayton, Ohio
Distribution: CentOS
Posts: 49

Rep: Reputation: 0
Archiving old files from /home/* directories to less expensive disk storage.


Hi Folks,
Please help me with this scripting issue.

We've asked and asked the users to archive their old files in their home directory. They don't do it.

I need a script that will find any non hidden files that have not been accessed in 60 days or more, and write them to an existing large LUN (SATA disks) archiving file system.

I'd like to have the script create a directory on the archive file system which matches each users home directory the first time an old file is to be moved from the home directory. That way, when a user asks to have a file restored to his home directory, I know where to look for it.

Later, if files in the archive haven't been access for 90 more days, we'll move them to tape using NetBackup.

Thanks!
 
Old 11-02-2009, 07:25 PM   #2
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Rocky 9.2
Posts: 18,359

Rep: Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751
What have you got so far? You'll need the find cmd for the time matching (try mtime). You would start by listing the dirs under /home and for each one, check if there is an existing dir on the target server. Personally, I'd just enforce creating one for every user (permanently), even if he has nothing to backup. Simplifies the code and sooner or later he'll have a file to backup.
 
Old 11-03-2009, 09:50 AM   #3
Cerephim
Member
 
Registered: Mar 2005
Location: Dayton, Ohio
Distribution: CentOS
Posts: 49

Original Poster
Rep: Reputation: 0
Finding and weeding out the files is no problem. The difficulty I'm having is creating directories and writing the files to the archive filesystem.
Here's what I have so far:
find /home/ -mtime +60 > /tmp/FilesMoreThan60DaysOld.txt
egrep -v ".kde|.zshrc|.emacs|.bash|.zshrc|.gtkrc" /tmp/FilesMoreThan60DaysOld.txt > /tmp/FilesMoreThan60DaysOld1.txt

This gives me a nice list of files targeted to be moved. Now, how do I create the directory structure necessary to move the files? Or would it be easier to just do a loop?

BTW, my shell scripting SUCKS!

foreach FILE in /tmp/FilesMoreThan60DaysOld1.txt
mv $FILE /archive
end

Something like that. As I've mentioned, my scripting leaves a lot to be desired.

Any help is really appreciated.
 
Old 11-03-2009, 10:24 AM   #4
kschmitt
Member
 
Registered: Jul 2009
Location: Chicago Suburbs
Distribution: Crux, CentOS, RHEL, Ubuntu
Posts: 96

Rep: Reputation: 23
Quote:
Originally Posted by Cerephim View Post
Finding and weeding out the files is no problem. The difficulty I'm having is creating directories and writing the files to the archive filesystem.
Here's what I have so far:
find /home/ -mtime +60 > /tmp/FilesMoreThan60DaysOld.txt
egrep -v ".kde|.zshrc|.emacs|.bash|.zshrc|.gtkrc" /tmp/FilesMoreThan60DaysOld.txt > /tmp/FilesMoreThan60DaysOld1.txt
You're close! Investigate the -exec command in find perhaps?

One (simplistic) way would be

find /home -type f -mtime +60 -exec tar -rf /tmp/oldfiles.tar {} \; -exec rm -f {} \;

To break that down:
find /home -type f
#Find only files (not directories) in /home
-mtime +60
#the files must be 60 days or older
-exec tar -rf /tmp/oldfiles.tar {} \;
#append the found file to /tmp/oldfiles.tar, creating the tar file if necessary
#NOTE, you can't use in-line compressions here (j and z flags)
-exec rm -f {} \;
#remove the found file
 
Old 11-05-2009, 02:06 PM   #5
Cerephim
Member
 
Registered: Mar 2005
Location: Dayton, Ohio
Distribution: CentOS
Posts: 49

Original Poster
Rep: Reputation: 0
Okay, I think I understand what you're doing...

However, the output of the find command needs to be edited via grep for hidden files that shouldn't be removed like:
.bashrc
.bash_profile
etc.

How can I feed the output of the grep command into the tar command? Everytime I try, it fails.
 
Old 11-06-2009, 11:24 AM   #6
kschmitt
Member
 
Registered: Jul 2009
Location: Chicago Suburbs
Distribution: Crux, CentOS, RHEL, Ubuntu
Posts: 96

Rep: Reputation: 23
Quote:
Originally Posted by Cerephim View Post
Okay, I think I understand what you're doing...

However, the output of the find command needs to be edited via grep for hidden files that shouldn't be removed like:
.bashrc
.bash_profile
etc.

How can I feed the output of the grep command into the tar command? Everytime I try, it fails.

Hey, I said it was simplistic

But I think what you want is to go more complex...

Allright, lets try this: you can chain rules in find, and inside each rule, you can run execs!

Lets start with this:
(Note, this only works if you give it the full path, because I'm too tired to craft a more clever first-expression here)

find /home/ \( -type d -name ".*" -prune \) -o \( -type f ! -name ".*" \)

find /home
#List things in home

\( -type d -name ".*" -prune \)
#Prune (don't look in) directories that start with "."

-o
#or

\( -type f ! -name ".*" \)
#look only at files that don't start with "."


Now, if that made sense... we add two execs
find /home/ \( -type d -name ".*" -prune \) -o \( -type f ! -name ".*" -exec tar -rvf /path/to/my/archive.tar {} \; rm -f {} \; \)

To break that last expression down:
-type f
#Look at files
! -name ".*"
#not starting with "."
-exec tar -rvf /path/to/my/archive.tar {} \;
#append the files to the tar file /path/to/my/archive.tar
rm -f {} \;
#remove the file


The exec _only_ works on the files, not the directories since it's enclosed in the parenthesis \(\), so there may be empty directories laying around afterwards.

Does that help at all?



PS: Thanks for asking the question, it forced me to remember some find kung-foo I hadn't done in years!
 
Old 11-06-2009, 11:25 AM   #7
kschmitt
Member
 
Registered: Jul 2009
Location: Chicago Suburbs
Distribution: Crux, CentOS, RHEL, Ubuntu
Posts: 96

Rep: Reputation: 23
Quote:
Originally Posted by kschmitt View Post
Hey, I said it was simplistic

find /home/ \( -type d -name ".*" -prune \) -o \( -type f ! -name ".*" \)
For the love of god run this part a few times before you run the part with the rm!!

Good luck
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
/dev/md1 nearly full - how do I work out which files/directories are on that disk? trumpet900 Linux - Newbie 1 07-26-2007 05:56 AM
copying files from my home directory to my USB Flash Disk neo_phyte Red Hat 9 04-24-2006 12:32 AM
.htaccess files doesn't work in home directories rehtorisi Linux - Software 3 08-08-2005 08:02 AM
multi-DVD archiving of enormous directories DraconianTimes Linux - Software 2 05-10-2004 04:30 PM
Disk Archiving/Imaging dtaliafe Linux - General 1 01-11-2002 09:37 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Server

All times are GMT -5. The time now is 08:27 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration