shell script to compare filese b/w local and remote and delete files from local serve
Linux - NetworkingThis forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
shell script to compare filese b/w local and remote and delete files from local serve
Hi,
I just got this script to write from my boss, but I thought I should
discuss my problem here too.
Ive got this situation where:
1) There are 2 server's ( A & B) located at quite a distance from each
other..
2) Only B can access A using the shell. A cannot access B. The access is possible using ssh
3) There is a perl script on B which runs every hour. It downloads
certain files from another remote server using ftp. When these files
have been downloaded comepletely they are 'scp'd to A using a shell
script.
Now I have been told to:
1) Make a shell script in B, to compare, whether the downloaded files
in B have been completely 'scp'd to A.
2) When the comparing has been done, giving a result that the files are
same on both the servers, the script then should delete the downloaded
files from B
Frankly speaking, I am confused as to how to go about comparing the
files...
unspawn:
[..How about simply using MD5sum or SHA1sum?..]
I had thought of using the md5sum or sha1sum, but i was confused as to how to implement them in the script..
[..So how did he get the impression you are capable of writing such a script?..]
I dont know what impression he has of me..Surely, it would not seem proper to go and ask him..
Ill have to do some serious reading to understand the script
zoony23:
[..Simply, you can scp with '-u' option. It will delete source files in A after copying them to B. See man scp for more...]
I do not have to scp the source files...Just have to compare the local and remote files..As I had written there is already another script doing the scp
Ill have to do some serious reading to understand the script
I'll un-fsck it, HTH:
Code:
#!/bin/sh
# Set debug and exit-on-error flag. Good for testing purposes.
# Also note there's an "echo rm" safeguard instead of a plain "rm".
set -xe
# Functions are nice for keeping a group of commands executed often together
function getSum() {
# fill array with the output of the command
sum=($(md5sum "$1"))
# if the first item of the array isn't empty then echo the contents
if [ -n "${sum[0]}" ]; then
echo ${sum[0]}
else
# else the first item of the array is empty and some error occurred.
echo "error in $FUNCNAME"
fi
}
compSum() {
# "expr" match is just another way of saying
# "if string_content_a matches string_content_b then doSomething"
# If the exit status of the command is zero we echo rm (else we don't do zilch)
expr match "$(getSum "$1")" "$(getSum "$2")" >/dev/null && echo "rm -f $1"
}
# Find files (type f) in path "/some/dirA" (placeholder, obviously), then
# read output of find into variable "A", then
find /some/dirA -type f | while read A; do
# see if we can find a similarly named file in placeholder directoryname /some/dirB/
B=$(basename "${A}"); B="/some/dirB/${B}"
if [ -f "${B}" ]; then
# if we find it then compare sums,
compSum "${A}" "${B}"
else
# if we don't find a similarly named file echo that.
echo "no ${B}"
fi
# exit for "find"'s "while" loop
done
# Always return zero for normal script execution finish
exit 0
Best way is to test this on your own dev box. Set up two directories with similar contents, change one file, run the script and see what's happening.
I dont know what impression he has of me.
Apparently a good one so let's keep it that way. Take care though: passing off other people's work as your own work can be risky (think general support, troubleshooting or extending it) if you know zilch about shell scripting. It's an excellent reason to go learn scripting RSN I think ;-p
Im going through the script u wrote..Please excuse me..as Im still a novice as regards to serious shell scripting..it may take some time for me to understand it...
Ill be going through ur advise of testing the script on my linux box..Though, I did speak to my boss regarding the use of check sums on the files, but he does not want that...
I did speak to my boss regarding the use of check sums on the files, but he does not want that.
Any particular and compelling reason why?
BTW, did you really have to make a whole new thread for this? I mean, we could have handled your script here perfectly since it's in line with your original post.
Any particular and compelling reason why?
He thinks that my code is a bit confusing.
Yes,
I did not need to have a whole new thread, but actually I had really got confused and my boss was pressing on me for the result of the script.
Im posting a block of code suggested by him. This really got me confused as to what he really wants. According to me, this code just checks whether the files are present..
array1=ssh 192.168.10.98
cd /home/username/scp
FILES=`ls -1`
for FILE in $FILES
do
> "$FILE.copy"
done
According to my boss, using the above code, the *.copy files will be created which will confirm the presence of the original files. Once that is done, then the files can be deleted from localhost.
But, according to me this in no way does confirm that the original files have been created with the proper size
He thinks that my code is a bit confusing.
What's important is that "confusing" it is NOT a constructive comment to help you improve your scripting-fu with (meaning he probably doesn't have the right development and management skills) and it is NOT a qualitative argument to reject a script on. Next to that the script is well-commented (OK, from my POV) just to be acceptable, doesn't do anything disasterous (echo rm instead of rm) and has been tested by me.
Yes,I did not need to have a whole new thread, but actually I had really got confused and my boss was pressing on me for the result of the script.
OK, fair enough. Too much pressure ain't good. How much time left?
According to my boss, using the above code, the *.copy files will be created which will confirm the presence of the original files.
That may be so but he makes a few assumptions:
- the script lists files in the current directory (no WD assigned), and
- it doesn't remove any $FILE.copy names first, and
- it doesn't check for like $FILE.copy.copy, and
- touching files for the sake of testing is not performant (it's a daft kludge).
Any removal based on this will be flawed.
But, according to me this in no way does confirm that the original files have been created with the proper size
You're right. A file may exist with size zero.
BTW, did you test my script as you said before?
Here's a way to check for files and match size. I will make it readable even for the fsckin four year old your boss is.
Code:
#!/bin/sh -
# Set exit-on-error flag. Good for testing purposes.
set -e
# Find files (type f) in path "/some/dirA" (placeholder, obviously), then
# read output of find into variable "A", then
find /some/dirA -type f | while read A; do
# see if we can find a similarly named file in placeholder directoryname /some/dirB/
B=$(basename "${A}"); B="/some/dirB/${B}"
if [ -f "${B}" ]; then
# if we find it then we compare size,
sizeOfA=$(stat -c %s "${A}")
sizeOfB=$(stat -c %s "${B}")
if [ "$sizeOfA" -eq "$sizeOfB" ]; then
# size matches: echo to stdout
echo "${A} ($sizeOfA) = ${B} ($sizeOfB)."
else
# size doesn't match: echo to stderr
echo "OMG TEH FILEZ MATCH0RZ NOT ${A} ($sizeOfA) = ${B} ($sizeOfB)." > /dev/stderr
fi
else
# if we don't find a similarly named file echo that.
echo "no ${B}"
fi
# exit for "find"'s "while" loop
done
# Always return zero for normal script execution finish
exit 0
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.