FTP::Recursive and MD5

innuendo_98 · 03-29-2005, 06:32 AM

If someone could give some pointers on FTP transfer, a few days ago I was dealing with file transfer for which I use the NET::FTP::Recursive, everything ok until I realized that somewhere along the process files got corrupted. Is there a way to spot file corruption 'on the fly' because let's say you're copying 1gb of data will be a mess to restart such a process, I've heard that md5 can do the trick there but got no idea from where to start.

TheLinuxDuck · 03-29-2005, 09:19 AM

It almost sounds to me like there is another problem at hand, if you're having file corruption on an FTP transfer. Can you post the relevant parts of your code? The first idea that occurred to me was to have the system, on which the files live, create textfiles which simply holds the md5 sum of the files that are to be transferred..

In two bash scripts, the first "finder.sh":

Code:

#!/bin/bash
# finder.sh

md5er=./md5check.sh
path=./

find $path -type f -exec $md5er {} \;

and the second, md5check.sh:

Code:

#!/bin/bash
# md5check.sh

if [ -z "$1" ]; then
  echo "$0: No filename provided"
  exit
fi

if [ ! -e "$1" ]; then
  echo "$0: Not a legitimate filename"
  exit
fi

if [ -e "$1.md5" ]; then
  existing=`cat "$1.md5"`
  new=`md5sum "$1" | awk '{print $1;}'`
  if [ "$existing" = "$new" ]; then
    # they match, no need to redo it
    exit
  fi
fi

md5sum "$1" | awk '{print $1;}' > "$1.md5"

Basically these two scripts just recurse a dir tree (given in finder.sh), and see if md5 sum files of the files exist, and if so makes sure they are up to date..

Then, your script can retrieve the files and the md5 files via FTP, and use the sum in the .md5 files to check against the files tranferred.

Like I said, this was my first thought.. there might be a better/another way to go about it.. (=

innuendo_98 · 03-29-2005, 10:12 AM

You're code is very helpful, though the way I'm handling this is at the directory level, once the directory is found it will send the entire directory structure to the ftp destination, any error produced during the transfer should go to the STDERR but apparently it didn't go that way.
This would be the code that does the transfer ...

my(@listofdirs);
if(opendir DIR, "$inpath") {
@listofdirs = grep { -d "$inpath/$_" && !/^\./ } readdir DIR;
foreach $CLIENT (@listofdirs){
if ($client eq $CLIENT){

chdir $inpath."/".$client."dataZ";
#The Following Process will recursively send the local current working directory and its contents to the ftp object's current working directory
$ftp = Net::FTP::Recursive->new($config{'host'}, Debug => 0); #debug will send error to STDERR ( $ftplog )
$ftp->login( $config{'user'}, $config{'pass'});
$ftp->cwd( $config{'trgt'}."/".$client);
$ftp->rput(); #Copy files
$ftp->quit;

open(ERRLOG,">>$ftplog") or
die("Unable to open LogFile: $!\n"); #Log errors from FTP process
carpout(\*ERRLOG);
close(ERRLOG);

TheLinuxDuck · 03-29-2005, 12:17 PM

Well, the very first thing that I notice is in the line:

Code:

 $ftp = Net::FTP::Recursive->new($config{'host'}, Debug => 0);

According to the Net::Cmd core of Net::FTP:

Quote:

Different packages may implement different levels of debug but a non-zero value results in copies of all commands and responses also being sent to STDERR.

And since a debug value of 0 typically means "No Debug info", I bet that this iis why you're not getting any debugging info to STDERR. It should probably be 1.

I'd change that and then see what kind of output you're getting.

Also, in order for carpout to work, it needs to be called prior to the any STDERR output. From the CGI::Carp doc page:

Quote:

The carpout() function requires one argument, which should be a reference to an open filehandle for writing errors. It should be called in a BEGIN block at the top of the CGI application so that compiler errors will be caught. Example:

Code:

   BEGIN {
     use CGI::Carp qw(carpout);
     open(LOG, ">>/usr/local/cgi-logs/mycgi-log") or
       die("Unable to open mycgi-log: $!\n");
     carpout(LOG);
   }