Differences between one file and another
Good afternoon chaps,
I hope you're all well. I've been reading man page after man page, and article after article to try my best not to ask for help on this one, but I think it best that I resort to the assistance of those more knowledgable to aid in my learning on this one. I am still working on my SFTP Scripts and now am facing another hurdle. I need to download certain file based on whether they already exist locally, and therefore just download new ones. I have come up with a theory that if I connect to the SFTP Server, and do ls to get a list, and store that list.. then do an ls on the local directory, then compare the two.. output the differences, pass that file into a loop and download the filenames that are different. (There's probably an easier way, but this was (I thought) a good idea, so I ran with it). I hit some hurdles on the way, but the following is what I have: Code:
/usr/bin/expect <<! > $FTPLIST The problem is with Comm or Diff.. As you can see I've been experimenting with 'Sort' as Comm apparently needs lists to be sorted first. But I've tried using diff, and I don't get the results I need. I would like to be able to get a list of all files that are in the remote location, which are not in the local location, and then download them. I was figuring I could generate this list using either comm or diff and then store that somewhere and run a loop on the lines within the file for FTP Download like: Code:
for line in $NEW; do The results of the test, using the cat $var in the above script displays the following: Code:
Remote Thanks Jon |
maybe these programs will help:
md5sum sha256sum sha1sum sha384sum sha224sum sha512sum instead of ls maybe you should use find. |
Would using Rsync work for you?
|
..is there a reason not to use rsync for this?
|
Hi,
Thanks for your response. I tried out find in place of LS but it's not a valid SFTP command, so assuming you meant in place of the local LS command? Would that offer any benefits to the ls in the way it is? I'll research the other commands you've posted to see if they will achieve the goal. Thanks Jon ---------- Post added 07-03-14 at 08:48 AM ---------- Rsync could be an option. But would I still need the list of filenames to download? |
^ i would log into the remote server using ssh and run something like:
Code:
find /whatever/floats/your/boat -type f -exec md5sum '{}' \; > local.md5 |
I have no SSH Access to the Server, SFTP Only...
|
i guess you have to do things the hard way.
try doing ls -1 on both remote and local and diff the results ? |
Ahh now we're talking :)
I've been playing around at home on the mac... as I don't want to faff about with the VPN to play around with the actual script and I have got the following to work: Code:
#!/bin/bash I'll try the above philosophy in the live script tomorrow :) Thanks for you help |
Update:
Although it worked nicely at home on the mac, with the above test... it doesn't work on the Red Hat box at the office... I think it's something to do with the way I'm getting the ls data in the first place on the remote location.. The only way I could think of getting the ls data was to capture the entire output of the Expect Script and therefore SFTP Session, and then grep the file names out... Code:
/usr/bin/expect <<! > $FTPLIST So the upshot is, it doesn't work and still displays the same.. Anyone have any other ideas? :) |
The following:
Code:
grep 'BAT_.*\.jpeg\.pgp' $FTPLIST > $RLS Code:
# ./img.sh |
I'm coming to the conclusion that there is something wrong with the DATA in the files. Maybe it doesn't like the _'s? Because everything I try just displays the contents of one file or the other.
I've tried, Comm, Diff, AWK, SED, and some crazy loops, nothing will work :( |
maybe if you supply a small example of both $LLS and $RLS someone would be able to hax something together.
|
Hi,
Thanks for your response... The contents are above: $RLS Code:
BAT_123456.jpeg.pgp Code:
BAT_123456.jpeg.pgp |
Code:
[schneidz@hyper jonnybinthemix]$ uname -a -m -p Code:
[schneidz@hyper jonnybinthemix]$ cat rls | while read jpg |
I have some progress...
After reading the entire man page for diff, I experimented using diff -b which ignores white space.. This does work better.. however I just need to experiment a bit more and try to get ride of the grep 'BAT_.*\.jpeg\.pgp' $FTPLIST > $RLS from displaying the actual ls command, because the ls command is ls -1 *.jpeg.pgp and therefore it includes that in the grep... and therefore finds the missing file but also finds the ls command as a difference also. |
All times are GMT -5. The time now is 03:54 PM. |