LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 05-14-2013, 07:05 PM   #1
DaemonHunter
LQ Newbie
 
Registered: Mar 2013
Posts: 13

Rep: Reputation: Disabled
Array linking; is it possible?


Let me start by saying that I really appreciate any responses people want to offer. Additionally I am more than interested in alternative programs/applications that do what my script is designed to do, however, what I am really looking for are answers that relate specifically to what I am trying to do with this script as I intend on completing the script for academic purposes as opposed to simply opting for someone else's program.

What I am doing is making a bash script that is designed to sync a directory in my home directory with network storage. The program works recursively to copy files that exist only in either the home or network directory to the appropriate location. Up to this point everything has been tested and works fine; the final stage is the most difficult. I want the program to update files in both the home directory and the network directory based on ctime (modify time). meaning if a file in my home directory is being synced with the network directory and has a "higher" or more up to date modify time, the file will clobber the network file and vieca versa. to make things simple I will use the home dir as my example as the semantics behind talking about the network array is not important. My idea thus far has been to create two arrays for the home dir and the network dir.


I use this to cut the ctime - HomeCtime=(`ls -lt | cut --complement -d " " -f 1-4 | cut --complement -c 1-4 | cut --complement -c 18-1000000`)


I use this for the file names, what I have here is a quick script whipped up to test and make sure that the arrays correlated properly but essentially the home array is all you should be concerned with.

HomeArray=(`ls -RA ~/.Teir1/cheese/ | egrep -v ^d`)


echo "HOME ARRAY = ${HomeArray[$1]}"



NetArray=(`ls -RA /media/1A63-8831/cheese/ | egrep -v ^d`)

echo "NET ARRAY = ${NetArray[$1]}"



NOW! Finally I am going to compare ${HomeCtime} and ${NetCtime} to find the most up to date files in; however I need to be able to call the HomeArray (filename) equivalent to the HomeCtime array entry; assuming that we are still for these purposes concerned only with the home array.


Once I can get this initial problem sorted out I will post subsequent threads to sort through the upcoming issues but am limiting this post to this issue of array linking.

To sum up, I need to be able to store ;lets say the index number; from the ctime array in a variable and call the same number form the Homearray (file name array)

anything you can offer will be appreciated. I apologise for the wordiness and complexity of this thread.
 
Old 05-14-2013, 08:27 PM   #2
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.8, Centos 5.10
Posts: 17,240

Rep: Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324
I hope I've understood
In any case I do have a few comments

1. *nix files have 3 times: atime (access), mtime (modify content/data), ctime (alter metadata/inode info eg ownerships, perms).
I'm pretty sure you want mtime, not ctime.
http://www.unix.com/tips-tutorials/2...ime-atime.html

2. there are good tools for doing dir tree syncing:
a) rsync - unidirectional, although you can run it in either direction
http://linux.die.net/man/1/rsync
b) unison - bi-directional
http://www.cyberciti.biz/faq/unison-...hronizer-tool/

Would be worth you reading up on those, even if you do decide to write your own code

4. When assigning the results of cmds to vars, you don't need (`...`). either use just backquotes `..` or use just $(...)
The latter is preferred for clarity and eg you can nest it (although that's rare)

5. Your cmd
Code:
HomeCtime=(`ls -lt | cut --complement -d " " -f 1-4 | cut --complement -c 1-4 | cut --complement -c 18-1000000`)

# gives me this
 echo $HomeCtime
May

# or
HomeCtime=`ls -lt | cut --complement -d " " -f 1-4 | cut --complement -c 1-4 | cut --complement -c 18-1000000`

# result
 echo $HomeCtime
May 10 17:54 t2. May 10 16:39 t.s May 9 16:08 cis May 9 16:06 t.t May 9 15:11 t1. May 3 17:06 sed May 3 11:09 t1. May 3 10:57 tmp Apr 26 23:35 tes Apr 26 17:45 dat Apr 26 16:26 fil Apr 26 16:24 sho Apr 22 15:24 get Apr 19 14:34 gre Apr 10 17:12 mr. Feb 27 14:48 cou Feb 27 14:38 cou Feb 27 14:35 cou Feb 27 14:21 ave Feb 27 14:14 awk

# which has cut off the filenames 

# FYI
bash --version
GNU bash, version 4.1.2(1)-release (x86_64-redhat-linux-gnu)
Consider using the stat cmd
Code:
stat t.t
  File: `t.t'
  Size: 426       	Blocks: 8          IO Block: 4096   regular file
Device: 812h/2066d	Inode: 1311081     Links: 1
Access: (0664/-rw-rw-r--)  Uid: (  500/   chris)   Gid: (  500/   chris)
Access: 2013-05-09 16:06:40.342869154 +1000
Modify: 2013-05-09 16:06:37.936975388 +1000
Change: 2013-05-09 16:06:37.936975388 +1000

# epoch secs
 stat -c '%X %Y %Z' t.t
1368079600 1368079597 1368079597
http://linux.die.net/man/1/stat
See also awk
Code:
ls -l|awk '{print $6,$7,$8,$9}'
   
Feb 27 14:21 average.awk
Feb 27 14:14 awk1.awk
May 9 16:08 ciscofw2csv.pl
Feb 27 14:48 count_space_by_ug.awk
Feb 27 14:35 count_users0.awk
Feb 27 14:38 count_users2.awk
Apr 26 17:45 date_calc.sh
Apr 26 16:26 filelist_double-star.sh
Apr 22 15:24 get_field.awk
Apr 19 14:34 grep_examples.sh
Apr 10 17:12 mr.pl
May 3 17:06 sed_examples.sh
Apr 26 16:24 show_params.bash
May 3 11:09 t1.sh
May 9 15:11 t1.t
May 10 17:54 t2.sh
Apr 26 23:35 test.pl
May 3 10:57 tmp
May 10 16:39 t.sh
May 9 16:06 t.t
You're going to need to reduce datestamps to epoch seconds for ease of comparison if you use them.


Anyway, hope that lot helps somewhat

Last edited by chrism01; 05-14-2013 at 08:33 PM.
 
1 members found this post helpful.
Old 05-16-2013, 05:58 PM   #3
DaemonHunter
LQ Newbie
 
Registered: Mar 2013
Posts: 13

Original Poster
Rep: Reputation: Disabled
Great! thanks for your post, and I totally misspoke, I did mean Mtime not ctime, I always get those confused. I realised my issue, I was over complicating things, what I need can be summed up in a simple for or while loop. after talking with a freind about the issue it became apparent that arrays are not needed.

here is some sudo code whipped up to help anyone looking at this post.


who=$(whoami)
basenamehome=/home/$who/
basenamenet=/network/dev/
targetdirectory=somedir for syncing
findfilenames () { cd $basenamehome/targetdirectory
find .
}


for filenames in findfilenames;
do
stathome=(stat $filenames) #when I actually use this code stat will have
statnetwork=(stat $filenames) #switches used to limit the output to mtime

if [ stathome -gt statnetwork ]; then
cp $basenamehome$filenames $basenamenet$filenames

elif [ statnetwork -gt stathome]; then
cp $basenamenet$filenames $basenameshome$filenames

done
exit 0


while I am sure that there are some errors in this code it is just meant to represent a general idea of how to accomplish the goal I had in mind in the begin post. Additionally there will be locks in place to make sure that the file output of the find function is equivalent to the find output where it to be preformed on the network dev.
 
Old 05-16-2013, 07:52 PM   #4
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.8, Centos 5.10
Posts: 17,240

Rep: Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324
I'd recommend using ${..} when embedding vars or concatenating them.
Also [[ ]] instead of [ ]
Code:
 cp ${basenamehome}$filenames ${basenamenet}$filenames
http://tldp.org/LDP/abs/html/testcon...ml#DBLBRACKETS
 
Old 05-17-2013, 03:14 PM   #5
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian sid + kde 3.5 & 4.4
Posts: 6,823

Rep: Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957
To start with, please use ***[code][/code]*** tags around your code and data, to preserve the original formatting and to improve readability. Do not use quote tags, bolding, colors, "start/end" lines, or other creative techniques.


chrism01 has already given some good points, particularly about using stat instead of parsing ls. I also endorse the use of unison or a similar tool for backups. No need to re-invent the wheel.

But let's see if I can give a few more pointers, using the script in post #3.

Code:
who=$(whoami)
basenamehome=/home/$who/
bash has a bunch of preset variables available for your use, including $USER and $HOME. You shouldn't need these two lines.

Code:
findfilenames () { cd $basenamehome/targetdirectory
find .
}
Kudos on using a function. But you need to consider how the output of find will be processed. Since filenames can contain just about any character, the only safe delimiter is the ascii null value. This can be done with the -print0 action, but all subsequent commands need to take it into account too. See below for more.

Code:
for filenames in findfilenames;
Since the function name is a command, you can't just use it directly here. Nor should you Read Lines With For.

You have two choices here to correct this. First is to use a while+read loop, again set to handle null separators. The second is to have your function store the find results in an array first, and then you can use a for loop with that. You'll still need to use a while loop inside the function to set the array, however.

How can I find and deal with file names containing newlines, spaces or both?
http://mywiki.wooledge.org/BashFAQ/020

Code:
if [ stathome -gt statnetwork ]; then
cp $basenamehome$filenames $basenamenet$filenames
1. When using advanced shells like bash or ksh, it's recommended to use [[..]] for string/file tests, and ((..)) for numerical tests. Avoid using the old [..] test unless you specifically need POSIX-style portability.

http://mywiki.wooledge.org/BashFAQ/031
http://mywiki.wooledge.org/ArithmeticExpression

2. QUOTE ALL OF YOUR VARIABLE EXPANSIONS. You should never leave the quotes off a parameter expansion unless you explicitly want the resulting string to be word-split by the shell and any possible globbing patterns expanded. This is a vitally important concept in scripting, so train yourself to do it correctly now. You can learn about the exceptions when you need them.

http://mywiki.wooledge.org/Arguments
http://mywiki.wooledge.org/WordSplitting
http://mywiki.wooledge.org/Quotes

3. Your logic appears to be wrong. You're running a loop that processes each individual file separately, so don't run a command that moves all the files at once. If you want to save processing (which is a good idea), then add each revised filename to an output array instead, then use that with a single cp command after the loop is finished.

4. I don't agree with chrism01 about using the full ${..} bracketed variable form, however. They are only needed with parameter substitution and when the variable name is directly followed by other alphanumeric characters ([a-zA-Z0-9_]) that the shell could mistake for part of the name. In all other cases it just adds clutter to the script, and clutter leads to an increased chance of errors.
 
Old 05-17-2013, 05:56 PM   #6
DaemonHunter
LQ Newbie
 
Registered: Mar 2013
Posts: 13

Original Poster
Rep: Reputation: Disabled
Thanks for the advice, I did mention that it was really hastily whipped up just to give an idea of what I realised was the most efficient way to get done what I need.

"Your logic appears to be wrong. You're running a loop that processes each individual file separately, so don't run a command that moves all the files at once. If you want to save processing (which is a good idea), then add each revised filename to an output array instead, then use that with a single cp command after the loop is finished."

Thanks for the heads up I just popped this idea off really quickly, it will deffinately be best to cp as the iteration occurs. I will post the final code once it is ready but I am working on other parts of the script at the moment and have yet to test this section.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] mdadm: only give one device per ARRAY line: /dev/md/:raid and array laughing_man77 Linux - Hardware 4 03-23-2012 05:05 PM
[SOLVED] shell script help: copying directory list into an array and then accessing the array richman1234 Linux - Newbie 6 07-26-2010 12:19 AM
Static linking vs. dynamic linking? posix_memalign Programming 13 04-18-2010 03:27 PM
Problem in using both load time linking and runtime linking durgaprasad_j Linux - General 0 08-01-2006 04:49 AM
(linking?) problem compiling very simple array class btb Programming 3 08-02-2005 04:16 PM


All times are GMT -5. The time now is 10:08 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration