LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Networking
User Name
Password
Linux - Networking This forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.

Notices


Reply
  Search this Thread
Old 08-18-2006, 04:29 AM   #1
dsids
Member
 
Registered: Mar 2006
Distribution: FC4
Posts: 184

Rep: Reputation: 31
shell script to compare filese b/w local and remote and delete files from local serve


Hi,
I just got this script to write from my boss, but I thought I should
discuss my problem here too.

Ive got this situation where:
1) There are 2 server's ( A & B) located at quite a distance from each
other..

2) Only B can access A using the shell. A cannot access B. The access is possible using ssh

3) There is a perl script on B which runs every hour. It downloads
certain files from another remote server using ftp. When these files
have been downloaded comepletely they are 'scp'd to A using a shell
script.

Now I have been told to:
1) Make a shell script in B, to compare, whether the downloaded files
in B have been completely 'scp'd to A.
2) When the comparing has been done, giving a result that the files are
same on both the servers, the script then should delete the downloaded
files from B

Frankly speaking, I am confused as to how to go about comparing the
files...

Thanks
Danish
 
Old 08-18-2006, 04:36 AM   #2
w3bd3vil
Senior Member
 
Registered: Jun 2006
Location: Hyderabad, India
Distribution: Fedora
Posts: 1,191

Rep: Reputation: 49
I think the diff command would come in use.
man diff
 
Old 08-18-2006, 05:53 AM   #3
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
1) Make a shell script in B, to compare, whether the downloaded files in B have been completely 'scp'd to A.
Use the exit code of the scp process?


I am confused as to how to go about comparing the files
How about simply using MD5sum or SHA1sum?
Code:
#!/bin/sh
set -xe
getSum() { sum=($(md5sum "$1")); [ -n "${sum[0]}" ] && echo ${sum[0]} || echo "error in $FUNCNAME"; }
compSum() { expr match "$(getSum "$1")" "$(getSum "$2")" >/dev/null && echo "rm -f $1"; }
find /some/dirA -type f|while read A; do B=$(basename "${A}"); B="/some/dirB/${B}"
 if [ -f "${B}" ]; then compSum "${A}" "${B}"; else echo "no ${B}"; fi; done; exit 0

I just got this script to write from my boss
So how did he get the impression you are capable of writing such a script?
 
Old 08-18-2006, 06:06 AM   #4
zoony23
LQ Newbie
 
Registered: Jul 2006
Location: USA, South Korea
Distribution: Fedora
Posts: 12

Rep: Reputation: 0
Simply, you can scp with '-u' option. It will delete source files in A after copying them to B. See man scp for more.
 
Old 08-18-2006, 06:32 AM   #5
dsids
Member
 
Registered: Mar 2006
Distribution: FC4
Posts: 184

Original Poster
Rep: Reputation: 31
unspawn:
[..How about simply using MD5sum or SHA1sum?..]

I had thought of using the md5sum or sha1sum, but i was confused as to how to implement them in the script..


[..So how did he get the impression you are capable of writing such a script?..]

I dont know what impression he has of me..Surely, it would not seem proper to go and ask him..

Ill have to do some serious reading to understand the script


zoony23:
[..Simply, you can scp with '-u' option. It will delete source files in A after copying them to B. See man scp for more...]

I do not have to scp the source files...Just have to compare the local and remote files..As I had written there is already another script doing the scp


Thanks
Danish
 
Old 08-18-2006, 07:02 AM   #6
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
Ill have to do some serious reading to understand the script
I'll un-fsck it, HTH:
Code:
#!/bin/sh
# Set debug and exit-on-error flag. Good for testing purposes.
# Also note there's an "echo rm" safeguard instead of a plain "rm".
set -xe
# Functions are nice for keeping a group of commands executed often together
function getSum() { 
 # fill array with the output of the command
 sum=($(md5sum "$1"))
 # if the first item of the array isn't empty then echo the contents
 if [ -n "${sum[0]}" ]; then
  echo ${sum[0]}
 else
  # else the first item of the array is empty and some error occurred.
  echo "error in $FUNCNAME"
 fi
 }

compSum() { 
 # "expr" match is just another way of saying
 # "if string_content_a matches string_content_b then doSomething"
 # If the exit status of the command is zero we echo rm (else we don't do zilch)
 expr match "$(getSum "$1")" "$(getSum "$2")" >/dev/null && echo "rm -f $1"
 }

# Find files (type f) in path "/some/dirA" (placeholder, obviously), then
# read output of find into variable "A", then
find /some/dirA -type f | while read A; do
 # see if we can find a similarly named file in placeholder directoryname /some/dirB/
 B=$(basename "${A}"); B="/some/dirB/${B}"
 if [ -f "${B}" ]; then
  # if we find it then compare sums, 
  compSum "${A}" "${B}"
 else
  # if we don't find a similarly named file echo that.
  echo "no ${B}"
 fi
# exit for "find"'s "while" loop
done
# Always return zero for normal script execution finish
exit 0
Best way is to test this on your own dev box. Set up two directories with similar contents, change one file, run the script and see what's happening.


I dont know what impression he has of me.
Apparently a good one so let's keep it that way. Take care though: passing off other people's work as your own work can be risky (think general support, troubleshooting or extending it) if you know zilch about shell scripting. It's an excellent reason to go learn scripting RSN I think ;-p

Last edited by unSpawn; 08-18-2006 at 07:03 AM.
 
Old 08-18-2006, 08:21 AM   #7
dsids
Member
 
Registered: Mar 2006
Distribution: FC4
Posts: 184

Original Poster
Rep: Reputation: 31
unspawn:
I really appreciate ur help..

Im going through the script u wrote..Please excuse me..as Im still a novice as regards to serious shell scripting..it may take some time for me to understand it...

Ill be going through ur advise of testing the script on my linux box..Though, I did speak to my boss regarding the use of check sums on the files, but he does not want that...

Ill have to think of some other way..

Thanks
Danish
 
Old 08-19-2006, 05:30 AM   #8
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
I did speak to my boss regarding the use of check sums on the files, but he does not want that.
Any particular and compelling reason why?


BTW, did you really have to make a whole new thread for this? I mean, we could have handled your script here perfectly since it's in line with your original post.
 
Old 08-23-2006, 12:59 AM   #9
dsids
Member
 
Registered: Mar 2006
Distribution: FC4
Posts: 184

Original Poster
Rep: Reputation: 31
Any particular and compelling reason why?
He thinks that my code is a bit confusing.

Yes,
I did not need to have a whole new thread, but actually I had really got confused and my boss was pressing on me for the result of the script.

Im posting a block of code suggested by him. This really got me confused as to what he really wants. According to me, this code just checks whether the files are present..

array1=ssh 192.168.10.98

cd /home/username/scp
FILES=`ls -1`
for FILE in $FILES
do
> "$FILE.copy"
done


According to my boss, using the above code, the *.copy files will be created which will confirm the presence of the original files. Once that is done, then the files can be deleted from localhost.
But, according to me this in no way does confirm that the original files have been created with the proper size

Thanks
Danish
 
Old 08-23-2006, 07:20 AM   #10
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
He thinks that my code is a bit confusing.
What's important is that "confusing" it is NOT a constructive comment to help you improve your scripting-fu with (meaning he probably doesn't have the right development and management skills) and it is NOT a qualitative argument to reject a script on. Next to that the script is well-commented (OK, from my POV) just to be acceptable, doesn't do anything disasterous (echo rm instead of rm) and has been tested by me.


Yes,I did not need to have a whole new thread, but actually I had really got confused and my boss was pressing on me for the result of the script.
OK, fair enough. Too much pressure ain't good. How much time left?


According to my boss, using the above code, the *.copy files will be created which will confirm the presence of the original files.
That may be so but he makes a few assumptions:
- the script lists files in the current directory (no WD assigned), and
- it doesn't remove any $FILE.copy names first, and
- it doesn't check for like $FILE.copy.copy, and
- touching files for the sake of testing is not performant (it's a daft kludge).
Any removal based on this will be flawed.


But, according to me this in no way does confirm that the original files have been created with the proper size
You're right. A file may exist with size zero.


BTW, did you test my script as you said before?


Here's a way to check for files and match size. I will make it readable even for the fsckin four year old your boss is.
Code:
#!/bin/sh -
# Set exit-on-error flag. Good for testing purposes.
set -e
# Find files (type f) in path "/some/dirA" (placeholder, obviously), then
# read output of find into variable "A", then
find /some/dirA -type f | while read A; do
 # see if we can find a similarly named file in placeholder directoryname /some/dirB/
 B=$(basename "${A}"); B="/some/dirB/${B}"
 if [ -f "${B}" ]; then
  # if we find it then we compare size, 
  sizeOfA=$(stat -c %s "${A}")
  sizeOfB=$(stat -c %s "${B}")
  if [ "$sizeOfA" -eq "$sizeOfB" ]; then
   # size matches: echo to stdout
   echo "${A} ($sizeOfA) = ${B} ($sizeOfB)."
  else
   # size doesn't match: echo to stderr
   echo "OMG TEH FILEZ MATCH0RZ NOT ${A} ($sizeOfA) = ${B} ($sizeOfB)." > /dev/stderr
  fi
 else
  # if we don't find a similarly named file echo that.
  echo "no ${B}"
 fi
# exit for "find"'s "while" loop
done
# Always return zero for normal script execution finish
exit 0
HTH
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Detect remote or local connection from shell. unreal128 Linux - General 2 07-29-2006 12:13 PM
Synchronizing local and remote files jrdioko Linux - Software 10 11-15-2005 01:13 PM
local rsync shell script eradrix Programming 13 07-09-2005 12:57 AM
mounting local hard drive in a remote shell (SSH) noir911 Linux - General 1 06-24-2005 05:55 AM
determining if shell is local connection or remote SaxyWeed Linux - General 3 01-26-2004 11:17 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Networking

All times are GMT -5. The time now is 05:51 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration