LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 05-15-2015, 03:22 PM   #1
Ranko Kohime
LQ Newbie
 
Registered: Nov 2013
Posts: 18

Rep: Reputation: 0
File Meta Data backup solution?


As part of my backup solution, I treat a certain part of my disk space as "expendable", such as my Downloads folder, where nearly everything in it can be re-downloaded from the source if need be, but I need a list of the files to do that.

My current solution involves running the following command once daily through cron, however it has some shortcomings.
Code:
#!/bin/bash
set -x
cd /home/ranko
{ echo `date`; find /home/ranko \( -path /home/ranko/.steam -o -path /home/ranko/mnt \) -prune -o  -exec ls -lihd {} \; ; find /var/ranko   -exec ls -lihd {} \; ; find /ranko -exec ls -lihd {} \; ; } | gzip -c9 > ~/Documents/Complete-File-Listing-`date +%F`.txt.gz
echo -e "File listing is Running"
First is that it presents a flat text file, which is certainly grep-able, but not browse-able. If for example I'm not certain of the name I'm looking for, it requires significant effort to browse the specific directory I'm looking for that file in.

Second is that it requires manual effort to prune versions, at present, and stores redundant information.

Third is that I haven't figured out how to store CRC information, without duplication of filenames, and the subsequent search that would have to be done for the CRC after locating the file itself.

I'd like to have a simple, file-browser-like interface that allows me to see when a file was created, modified, deleted, perhaps with CRC's each time that a file is changed.

Does a program like this exist already?
 
Old 06-14-2015, 07:16 PM   #2
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
First of all what precisely is your backup solution? Secondly I don't see you treat your Downloads directory as expendable, as evidenced by your cron job? Because if you did they you would just say 'rsync --exclude=~/Downloads', then dedupe ~/Downloads and finally 'md5deep -r ~/Downloads' to get a complete list with names and hashes?..
 
Old 06-17-2015, 01:38 AM   #3
Ranko Kohime
LQ Newbie
 
Registered: Nov 2013
Posts: 18

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by unSpawn View Post
First of all what precisely is your backup solution? Secondly I don't see you treat your Downloads directory as expendable, as evidenced by your cron job? Because if you did they you would just say 'rsync --exclude=~/Downloads', then dedupe ~/Downloads and finally 'md5deep -r ~/Downloads' to get a complete list with names and hashes?..
My backup solution involves knowing EVERYTHING that is on my system, except for completely unneeded files (therefore .steam, (because cloud) and mnt, where I don't want the script tying up any thumbdrives or sshfs I may have mounted), even knowing what I had in my Downloads folder shortly before a crash. The resulting text file is dumped into a directory that is on a scheduled backup. (Currently Crashplan is the scheduled backup, but interested in moving from that to something else, and a second backup is done to a local thumbdrive whenever I remember to do that.)

This isn't rsync'ing, BTW, this is merely ls piped to a text file. As for an MD5 sum, that would work, partially, but I want to store file dates and sizes, which MD5 does not do in it's hash files.
 
Old 06-17-2015, 11:51 AM   #4
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
I think you may be better off building a list once (and I'm not saying you should use flat files: you could easily use a sqlite3 database) then either use 0) inotify-driven scripts to store detailed entry modifications / additions / deletions or 1) use a HIDS that stores hashes and details (like say Samhain). As for the UI question I don't know of any such thing. It's usually commercial backup solutions that come with such drill down interfaces.
 
Old 09-04-2015, 12:52 AM   #5
Ranko Kohime
LQ Newbie
 
Registered: Nov 2013
Posts: 18

Original Poster
Rep: Reputation: 0
I completely forgot that there was a piece of software I used many moons ago, (when I last used Windows) in the heyday of CD-Rs. It was called Where Is It?, and an automatable version of that is basically what I am seeking. Does such a thing exist in open source form?
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Trying to setup backup solution on server... having troubles with file systems Aunnix Linux - Newbie 19 11-05-2012 12:57 PM
Open Source File Replication/backup solution needed jeveretts Linux - Newbie 5 06-06-2010 11:19 AM
Backup Solution for Sles environment - need scripts to migrate data hannesve Linux - Newbie 1 07-09-2008 07:54 AM
MP3 ID3 Tag/Meta Data heri0n Linux - Software 2 08-28-2006 10:45 AM
data backup solution suggestions disorderly Linux - General 5 08-31-2004 07:41 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 11:25 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration