LinuxQuestions.org
Register a domain and help support LQ
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 09-18-2012, 10:01 AM   #1
driftux
LQ Newbie
 
Registered: Sep 2012
Posts: 13

Rep: Reputation: Disabled
Bash copy script by times


Hi, what is the most effective bash script according to performance to find and copy files from one LINUX server to another using specifications described below.

I need to get a bash script which finds only new files created in server A in directories with name "Z" between interval from 0 to 10 minutes ago. Then transfer them to server B. I think it can be done by formatting a query and executing it for each founded new file "scp /X/Y.../Z/file root@hostname:/X/Y.../Z/" If script finds no such remote path on server B it will continue copying second file which directory exists. File should be copied with permissions, group, owner and creation time.

X/Y... are various directories path. I want setup a cron job to execute this script every 10 minutes. So the performance is very important in this case.

Thank you.
 
Old 09-18-2012, 10:34 AM   #2
TB0ne
Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 14,136

Rep: Reputation: 2456Reputation: 2456Reputation: 2456Reputation: 2456Reputation: 2456Reputation: 2456Reputation: 2456Reputation: 2456Reputation: 2456Reputation: 2456Reputation: 2456
Quote:
Originally Posted by driftux View Post
Hi, what is the most effective bash script according to performance to find and copy files from one LINUX server to another using specifications described below.

I need to get a bash script which finds only new files created in server A in directories with name "Z" between interval from 0 to 10 minutes ago. Then transfer them to server B. I think it can be done by formatting a query and executing it for each founded new file "scp /X/Y.../Z/file root@hostname:/X/Y.../Z/" If script finds no such remote path on server B it will continue copying second file which directory exists. File should be copied with permissions, group, owner and creation time.

X/Y... are various directories path. I want setup a cron job to execute this script every 10 minutes. So the performance is very important in this case.
Ok...show us what you've written so far, and describe the environment, and we can help. There's no way of knowing how to tune a script, without knowing the parameters. How many files? Size of the files? Speed of the link between the two systems? How often are files created? File locking in place on the network share?

Also....this sounds VERY much like a homework question. What's the goal you're trying to reach?? If it was me, I'd use rsync...that's exactly what it was designed for.
 
Old 09-18-2012, 01:19 PM   #3
driftux
LQ Newbie
 
Registered: Sep 2012
Posts: 13

Original Poster
Rep: Reputation: Disabled
Actually I haven't done any homeworks yet, because I don't know how to do them So I just have an empty paper
I know that I could use Rsync and Find commands, but I'm not able to write complete script. I will provide as
much information as I know.
Server A has about 20 000 directories and 200 000 files. Server B has about 20 000 directories and 2 000 000 files.
These servers are in two different locations, and the speed between them is ~30Mbps
In period of 10 minutes about 5-10 small files (one file ~10KB) could be added.
File locking? I am not using NFS.

I am thinking about new idea. Maybe it isn't worth to hasle with "time" because if my server A will go out of control (BIOS, NTP errors)
and all files will be copied not paying attention to creation time. Maybe I need a script which could find files in all directories named Z using path /home/users/*/personal/Z/ and add prefix "1_" to all files in Z folders. For example file name "aaaa" will become "1_aaaa" Then use Rsync to copy (one way sync from A server to B) all new files exept those who has prefixes "1_". I think I will notice performance delay comparing to first variant, but maybe this solution is more stable.

Last edited by driftux; 09-18-2012 at 01:22 PM.
 
Old 09-18-2012, 01:53 PM   #4
TB0ne
Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 14,136

Rep: Reputation: 2456Reputation: 2456Reputation: 2456Reputation: 2456Reputation: 2456Reputation: 2456Reputation: 2456Reputation: 2456Reputation: 2456Reputation: 2456Reputation: 2456
Quote:
Originally Posted by driftux View Post
Actually I haven't done any homeworks yet, because I don't know how to do them So I just have an empty paper
You're missing the point of homework. YOU need to learn...we will be glad to HELP, but you have to show effort on your part. All you've done so far is (basically), ask us to provide you a script.
Quote:
I know that I could use Rsync and Find commands, but I'm not able to write complete script. I will provide as much information as I know.
Server A has about 20 000 directories and 200 000 files. Server B has about 20 000 directories and 2 000 000 files.
These servers are in two different locations, and the speed between them is ~30Mbps/ In period of 10 minutes about 5-10 small files (one file ~10KB) could be added. File locking? I am not using NFS.

I am thinking about new idea. Maybe it isn't worth to hasle with "time" because if my server A will go out of control (BIOS, NTP errors)
and all files will be copied not paying attention to creation time. Maybe I need a script which could find files in all directories named Z using path /home/users/*/personal/Z/ and add prefix "1_" to all files in Z folders. For example file name "aaaa" will become "1_aaaa" Then use Rsync to copy (one way sync from A server to B) all new files exept those who has prefixes "1_". I think I will notice performance delay comparing to first variant, but maybe this solution is more stable.
If you know that you can use rsync and find commands, then start there. Do it...try it..experiment. That's called 'learning'. Break the task down into pieces, and figure out each one, the same way you would with ANY problem. Read the man pages for rsync and find. Once rsync is set up, it's entire purpose is to keep two directory structures synchronized...which is what you're asking.

However, since this is homework, presumably the idea isn't about how to set up basic Linux utilities, but to teach you about commands and scripting. So again, show us your efforts and where you're stuck, and we can help.
 
2 members found this post helpful.
Old 09-18-2012, 04:27 PM   #5
driftux
LQ Newbie
 
Registered: Sep 2012
Posts: 13

Original Poster
Rep: Reputation: Disabled
Ok, I decided that script fired up in cron every 10 minutes should:
1. Copy all files using Rsync from serverA directories Z to serverB at the same path directories Z, except those files who has prefix "1_"
2. Add prefix "1_" to every file in directories Z, which was added 10 minutes ago.

So for 1. step this should help "rsync -avz --exclude='1_*' -e ssh remoteuser@remotehost:/remote/*/dir /this/*/dir/"
Does wildcard symbol (*) is recognized and Rsync will understand that it needs to copy remoteuser@remotehost:/remote/AAA/dir to /this/AAA/dir
In this case directory is AAA, but I have them and BBB, CCC and so on, about 20 000 folders in total.


For step nr. 2 I find out that if want to search created files 10 minutes ago and my directory structure is quite the same the commmand "find /home/users/*/personal/Z -type f -amin -10" works great. I tested in on my server. Now I need combine XARGS or EXEC. Some people recommends replace FIND with script like this "for file in *; do mv $file 1_$file; done" But here I stuck...

This is it.. I give up or I need to read entire bash programming book Could someone write a full script combining 1 and 2 steps?
Thank you.
 
Old 09-18-2012, 08:40 PM   #6
TB0ne
Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 14,136

Rep: Reputation: 2456Reputation: 2456Reputation: 2456Reputation: 2456Reputation: 2456Reputation: 2456Reputation: 2456Reputation: 2456Reputation: 2456Reputation: 2456Reputation: 2456
Quote:
Originally Posted by driftux View Post
Ok, I decided that script fired up in cron every 10 minutes should:
1. Copy all files using Rsync from serverA directories Z to serverB at the same path directories Z, except those files who has prefix "1_"
2. Add prefix "1_" to every file in directories Z, which was added 10 minutes ago.

So for 1. step this should help "rsync -avz --exclude='1_*' -e ssh remoteuser@remotehost:/remote/*/dir /this/*/dir/"
Does wildcard symbol (*) is recognized and Rsync will understand that it needs to copy remoteuser@remotehost:/remote/AAA/dir to /this/AAA/dir
In this case directory is AAA, but I have them and BBB, CCC and so on, about 20 000 folders in total.


For step nr. 2 I find out that if want to search created files 10 minutes ago and my directory structure is quite the same the commmand "find /home/users/*/personal/Z -type f -amin -10" works great. I tested in on my server. Now I need combine XARGS or EXEC. Some people recommends replace FIND with script like this "for file in *; do mv $file 1_$file; done" But here I stuck...
So AGAIN...POST WHAT YOU HAVE WRITTEN. Where are you 'stuck'?? What have you TRIED? What message(s)/result(s) do you get?
Quote:
This is it.. I give up or I need to read entire bash programming book Could someone write a full script combining 1 and 2 steps?
Thank you.
Then you need to start reading. Again, no one here is going to do your homework for you. This is YOUR homework..do it. Tricks like this have been tried MANY times here..."Oh, I do xxx and yyy right?? Can someone just 'help' by writing these in a script for me??". Sorry, but if you know the commands, then you also know how to script them...all a script is, is a set of commands in one file.

There are almost 1 MILLION hits for "linux bash scripting tutorial" you can find on Google. They have examples...why is it hard for you to find them, or read your textbook?
http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO.html
http://tldp.org/LDP/abs/html/

Last edited by TB0ne; 09-18-2012 at 08:42 PM.
 
Old 09-19-2012, 10:56 AM   #7
driftux
LQ Newbie
 
Registered: Sep 2012
Posts: 13

Original Poster
Rep: Reputation: Disabled
I don't know what is wrong with you men... Are you teacher of informatics at school or did you have a terrible childhood?
As I said I'm not a programmer, and I have looked all day in Google and I read a lot, but didn't find an answer. I am not going to learn all batch programming language because of one script.
I would better ask help, and next time provide my help to people here. If you are not able to write two rows of script to someone because of your ego, you should work more not with command line
but with your personality.
 
Old 09-19-2012, 11:09 AM   #8
TB0ne
Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 14,136

Rep: Reputation: 2456Reputation: 2456Reputation: 2456Reputation: 2456Reputation: 2456Reputation: 2456Reputation: 2456Reputation: 2456Reputation: 2456Reputation: 2456Reputation: 2456
Quote:
Originally Posted by driftux View Post
I don't know what is wrong with you men... Are you teacher of informatics at school or did you have a terrible childhood?
As I said I'm not a programmer, and I have looked all day in Google and I read a lot, but didn't find an answer. I am not going to learn all batch programming language because of one script.
I would better ask help, and next time provide my help to people here. If you are not able to write two rows of script to someone because of your ego, you should work more not with command line but with your personality.
This is your homework, not ours. YOU need to learn, but don't seem to want to. Again, we are not going to do your homework for you. And if it's just two lines, why can't you write it? Why would you need to "learn all batch programming language", for a two-line script??

If you really looked all day in Google and can't figure out a two-line script...you should consider dropping whatever class you're taking, or let the teacher know what problems you're having.
 
Old 09-20-2012, 02:57 AM   #9
driftux
LQ Newbie
 
Registered: Sep 2012
Posts: 13

Original Poster
Rep: Reputation: Disabled
Sometimes you need 2 weeks or more learning if you want to do professionaly 20 minutes job. I need a good and stable solution which will be used on real production servers.
Thats why I'm not willing to experiment but better to ask experts in this area. This conversation is useless. You have your opinion I have mine. But I bet that you don't have many friends.
 
Old 09-20-2012, 12:34 PM   #10
TB0ne
Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 14,136

Rep: Reputation: 2456Reputation: 2456Reputation: 2456Reputation: 2456Reputation: 2456Reputation: 2456Reputation: 2456Reputation: 2456Reputation: 2456Reputation: 2456Reputation: 2456
Quote:
Originally Posted by driftux View Post
Sometimes you need 2 weeks or more learning if you want to do professionaly 20 minutes job. I need a good and stable solution which will be used on real production servers.
From your second post, #3 in this thread:
Quote:
Originally Posted by driftux
Actually I haven't done any homeworks yet, because I don't know how to do them So I just have an empty paper
So sorry..you've already admitted this is homework. And if you are an administrator, then YOU need to know how to do a professional job.
Quote:
Thats why I'm not willing to experiment but better to ask experts in this area. This conversation is useless. You have your opinion I have mine. But I bet that you don't have many friends.
Yes, it is useless...because you seem to think that it's ok to try to get others to do your homework for you, while most people say it's not a good idea. You've spent more time arguing and complaining about how no one would write a script for you, than you would have spent taking some VERY easily-found examples and making them work yourself. pan64 already did your homework for you in your other thread...but you won't even put the effort into putting one string (that they wrote), into a script (that they ALSO wrote), to make it work for you. If you're an administrator at a company now, let them know your level of skill, so they can attempt to train you. If you're a student, do your own work.

Thread reported to moderators.

Last edited by TB0ne; 09-20-2012 at 12:37 PM.
 
Old 09-21-2012, 05:08 PM   #11
colucix
Moderator
 
Registered: Sep 2003
Location: Bologna
Distribution: CentOS 6.5 OpenSuSE 12.3
Posts: 10,453

Rep: Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941
Quote:
Originally Posted by driftux View Post
Could someone write a full script combining 1 and 2 steps?
Absolutely not. Despite you didn't clear it up, your original question appears to be a real homework posted almost verbatim. The LQ policy is clear about this point, as you should have read in the LQ rules upon your subscription:
Quote:
Do not post homework assignments verbatim. We're happy to assist if you have specific questions or have hit a stumbling point, however. Let us know what you've already tried and what references you have used (including class notes, books, and searches) and we'll do our best to help. Keep in mind that your instructor might also be an LQ member.

Likewise, let's accept for a moment that it may be not homework: how do you think someone would be willing to help you when you state I am not going to learn all batch programming language because of one script or Sometimes you need 2 weeks or more learning if you want to do professionaly 20 minutes job. I need a good and stable solution... that's why I'm not willing to experiment?

Try to show some effort instead: maybe you might receive a better and suitable response. Last but not least, please refrain from posting immature statements addressed to other members, as those ones between the lines above. We are not in school, here!
 
1 members found this post helpful.
Old 09-23-2012, 01:36 AM   #12
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian sid + kde 3.5 & 4.4
Posts: 6,823

Rep: Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946
Since the other thread has been locked, I need to post my reply to it here.

The OP there and many of the replies following from it suffer from the same common syntax problem:

Don't Read Lines With For!


Indeed, any form of command or variable substitution that attempts to extract strings containing spaces suffers from the same problem. The output can only be read as a single string (with quotes), or as individual words (without quotes). You can't ever completely guarantee safe processing this way. Always use a while+read loop with null separators if you need to handle arbitrary lists, or another technique like find with -exec.

How can I read a file (data stream, variable) line-by-line (and/or field-by-field)?
http://mywiki.wooledge.org/BashFAQ/001

Here are a couple of good links about using find:
http://mywiki.wooledge.org/UsingFind
http://www.grymoire.com/Unix/Find.html
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Script to Hit 'Enter' key multiple times in bash script. nithinmp Ubuntu 7 07-19-2012 07:01 AM
Bash script to copy without overwriting .. Help grandeinter Programming 5 10-22-2010 07:13 PM
bash: run a script at most 3 times simultanously paterijk Programming 4 08-18-2005 02:49 PM
Bash script to alert by email 3 times then stop. pmpc00 Linux - General 2 11-04-2004 07:23 AM
Running two copy's of the same bash script AndiOliver Programming 2 10-06-2004 03:46 AM


All times are GMT -5. The time now is 11:23 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration