perl md5sum big file segfaults

#!/usr/bin/perl

strict;
use Digest::MD5 qw(md5_hex);
use Digest::SHA1 qw(sha1_hex);

my $content = "";

open(my $fh, '<:raw', "/tmp/a.tgz" ) or die "cannot open file $filename";
{
    local $/;
    $content = <$fh>;
}
close($fh);

print length($content) . "\n";

# Check that sha1 and md5 are correct
my $md5Value=md5_hex($content);
print "md5sum is ok\n" if $debug;
my $sha1Value=sha1_hex($content);
print "sha1sum is ok\n" if $debug;

this code segfaults if /tmp/a.tgz is big enough (3 GB).
Do you know how can I handle it?

Is your Perl 64-bit? How much memory does your computer have?

this was occurred on several different hosts with huge amount of ram (like 256 GB).
This is perl 5, version 16, subversion 3 (v5.16.3) built for x86_64-linux-thread-multi

Originally Posted by pan64 #!/usr/bin/perl strict; use Digest::MD5 qw(md5_hex); use Digest::SHA1 qw(sha1_hex); my $content = ""; open(my $fh, '<:raw', "/tmp/a.tgz" ) or die "cannot open file $filename"; { local $/; $content = <$fh>; } close($fh); print length($content) . "\n"; # Check that sha1 and md5 are correct my $md5Value=md5_hex($content); print "md5sum is ok\n" if $debug; my $sha1Value=sha1_hex($content); print "sha1sum is ok\n" if $debug; this code segfaults if /tmp/a.tgz is big enough (3 GB). Do you know how can I handle it?

That would be an expensive (memory wise) thing to do, reading the entire huge file handle at once. I'd personally read it in chunks.

my $MD5SUM = do {
    open(my $fh, '<:raw', "/tmp/a.tgz" ) or die "cannot open file $filename";
    my $MDChunk = Digest::MD5->new;
    local $/ = \131072; # Define a 128k chunk
    while (<$fh>) {
        $MDChunk->add($_);
    }
    $MDChunk->hexdigest;
};
<...rest of code....>

..would process it one 128k chunk at a time. Totally untested. Just a thought.

yes, just the file is downloaded from the net using something like this:

        my $ua = LWP::UserAgent->new(
            ssl_opts => { SSL_verify_mode => 'SSL_VERIFY_NONE'},
            );
        my $req = HTTP::Request->new( GET => "$url" );
        $req->authorization_basic( "$User", "$Password" );
        my $response = $ua->request($req);
        $content = $response->content;

So I have no really chance to do that.
Downloading works.

Hi

I really suck at Perl, but there is a way to set up a callback with HTTP::Request, so you can process the data "chunk by chunk".

https://stackoverflow.com/questions/...-http-resource

If you can use shell commands, its a lot easier. Something like this will pipe the downloaded data to md5sum, and uses a lot less memory:

wget -q -O - http://example.org | md5sum

yes, you are almost right. There are workarounds, now I'm forced to "find another way".
I need to download a file and check the md5sum/sha1sum and the perl script can do that (not only this, but even more - like find out the name and location of the file and other special things). It works perfectly, the only problem is when the file itself is too big.

The current workaround is to save the file and run checksum calculation on file.

But this thread is about the segfault of *_hex calls which looks like a bug.

You don't have to process every data in one step; example:

use Digest::MD5;

sub DoOne {
    my ($filename, $fh, $block, $blocklen, $ctx, $digest);
    $filename=$_[0];

    open $fh, '<', $filename or return '';
    binmode $fh;

    $ctx = Digest::MD5->new;

    do {
	$blocklen= read $fh, $block, 1024;
	if ($blocklen>0) {
	    $ctx->add ($block);
	}
    } while ($blocklen>0);
    close $fh;

    $digest = $ctx->hexdigest;

    return $digest;
}