Bash copy script by times
 

Hi, what is the most effective bash script according to performance to find and copy files from one LINUX server to another using specifications described below.

I need to get a bash script which finds only new files created in server A in directories with name "Z" between interval from 0 to 10 minutes ago. Then transfer them to server B. I think it can be done by formatting a query and executing it for each founded new file "scp /X/Y.../Z/file root@hostname:/X/Y.../Z/" If script finds no such remote path on server B it will continue copying second file which directory exists. File should be copied with permissions, group, owner and creation time.

X/Y... are various directories path. I want setup a cron job to execute this script every 10 minutes. So the performance is very important in this case.

Thank you.

Ok...show us what you've written so far, and describe the environment, and we can help. There's no way of knowing how to tune a script, without knowing the parameters. How many files? Size of the files? Speed of the link between the two systems? How often are files created? File locking in place on the network share?

Also....this sounds VERY much like a homework question. What's the goal you're trying to reach?? If it was me, I'd use rsync...that's exactly what it was designed for.

Actually I haven't done any homeworks yet, because I don't know how to do them :) So I just have an empty paper :)
I know that I could use Rsync and Find commands, but I'm not able to write complete script. I will provide as
much information as I know.
Server A has about 20 000 directories and 200 000 files. Server B has about 20 000 directories and 2 000 000 files.
These servers are in two different locations, and the speed between them is ~30Mbps
In period of 10 minutes about 5-10 small files (one file ~10KB) could be added.
File locking? I am not using NFS.

I am thinking about new idea. Maybe it isn't worth to hasle with "time" because if my server A will go out of control (BIOS, NTP errors)
and all files will be copied not paying attention to creation time. Maybe I need a script which could find files in all directories named Z using path /home/users/*/personal/Z/ and add prefix "1_" to all files in Z folders. For example file name "aaaa" will become "1_aaaa" Then use Rsync to copy (one way sync from A server to B) all new files exept those who has prefixes "1_". I think I will notice performance delay comparing to first variant, but maybe this solution is more stable.

You're missing the point of homework. YOU need to learn...we will be glad to HELP, but you have to show effort on your part. All you've done so far is (basically), ask us to provide you a script.
If you know that you can use rsync and find commands, then start there. Do it...try it..experiment. That's called 'learning'. Break the task down into pieces, and figure out each one, the same way you would with ANY problem. Read the man pages for rsync and find. Once rsync is set up, it's entire purpose is to keep two directory structures synchronized...which is what you're asking.

However, since this is homework, presumably the idea isn't about how to set up basic Linux utilities, but to teach you about commands and scripting. So again, show us your efforts and where you're stuck, and we can help.

Ok, I decided that script fired up in cron every 10 minutes should:
1. Copy all files using Rsync from serverA directories Z to serverB at the same path directories Z, except those files who has prefix "1_"
2. Add prefix "1_" to every file in directories Z, which was added 10 minutes ago.

So for 1. step this should help "rsync -avz --exclude='1_*' -e ssh remoteuser@remotehost:/remote/*/dir /this/*/dir/"
Does wildcard symbol (*) is recognized and Rsync will understand that it needs to copy remoteuser@remotehost:/remote/AAA/dir to /this/AAA/dir
In this case directory is AAA, but I have them and BBB, CCC and so on, about 20 000 folders in total.


For step nr. 2 I find out that if want to search created files 10 minutes ago and my directory structure is quite the same the commmand "find /home/users/*/personal/Z -type f -amin -10" works great. I tested in on my server. Now I need combine XARGS or EXEC. Some people recommends replace FIND with script like this "for file in *; do mv $file 1_$file; done" But here I stuck...

This is it.. I give up or I need to read entire bash programming book :) Could someone write a full script combining 1 and 2 steps?
Thank you.

So AGAIN...POST WHAT YOU HAVE WRITTEN. Where are you 'stuck'?? What have you TRIED? What message(s)/result(s) do you get?
Then you need to start reading. Again, no one here is going to do your homework for you. This is YOUR homework..do it. Tricks like this have been tried MANY times here..."Oh, I do xxx and yyy right?? Can someone just 'help' by writing these in a script for me??". Sorry, but if you know the commands, then you also know how to script them...all a script is, is a set of commands in one file.

There are almost 1 MILLION hits for "linux bash scripting tutorial" you can find on Google. They have examples...why is it hard for you to find them, or read your textbook?
http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO.html
http://tldp.org/LDP/abs/html/

I don't know what is wrong with you men... Are you teacher of informatics at school or did you have a terrible childhood?
As I said I'm not a programmer, and I have looked all day in Google and I read a lot, but didn't find an answer. I am not going to learn all batch programming language because of one script.
I would better ask help, and next time provide my help to people here. If you are not able to write two rows of script to someone because of your ego, you should work more not with command line
but with your personality.

This is your homework, not ours. YOU need to learn, but don't seem to want to. Again, we are not going to do your homework for you. And if it's just two lines, why can't you write it? Why would you need to "learn all batch programming language", for a two-line script??

If you really looked all day in Google and can't figure out a two-line script...you should consider dropping whatever class you're taking, or let the teacher know what problems you're having.

Sometimes you need 2 weeks or more learning if you want to do professionaly 20 minutes job. I need a good and stable solution which will be used on real production servers.
Thats why I'm not willing to experiment but better to ask experts in this area. This conversation is useless. You have your opinion I have mine. But I bet that you don't have many friends.

From your second post, #3 in this thread:
So sorry..you've already admitted this is homework. And if you are an administrator, then YOU need to know how to do a professional job.
Yes, it is useless...because you seem to think that it's ok to try to get others to do your homework for you, while most people say it's not a good idea. You've spent more time arguing and complaining about how no one would write a script for you, than you would have spent taking some VERY easily-found examples and making them work yourself. pan64 already did your homework for you in your other thread...but you won't even put the effort into putting one string (that they wrote), into a script (that they ALSO wrote), to make it work for you. If you're an administrator at a company now, let them know your level of skill, so they can attempt to train you. If you're a student, do your own work.

Thread reported to moderators.

Absolutely not. Despite you didn't clear it up, your original question appears to be a real homework posted almost verbatim. The LQ policy is clear about this point, as you should have read in the LQ rules upon your subscription:

Likewise, let's accept for a moment that it may be not homework: how do you think someone would be willing to help you when you state I am not going to learn all batch programming language because of one script or Sometimes you need 2 weeks or more learning if you want to do professionaly 20 minutes job. I need a good and stable solution... that's why I'm not willing to experiment?

Try to show some effort instead: maybe you might receive a better and suitable response. Last but not least, please refrain from posting immature statements addressed to other members, as those ones between the lines above. We are not in school, here!

Since the other thread has been locked, I need to post my reply to it here.

The OP there and many of the replies following from it suffer from the same common syntax problem:

Don't Read Lines With For!


Indeed, any form of command or variable substitution that attempts to extract strings containing spaces suffers from the same problem. The output can only be read as a single string (with quotes), or as individual words (without quotes). You can't ever completely guarantee safe processing this way. Always use a while+read loop with null separators if you need to handle arbitrary lists, or another technique like find with -exec.

How can I read a file (data stream, variable) line-by-line (and/or field-by-field)?
http://mywiki.wooledge.org/BashFAQ/001

Here are a couple of good links about using find:
http://mywiki.wooledge.org/UsingFind
http://www.grymoire.com/Unix/Find.html