LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 06-30-2004, 07:39 PM   #1
Apostasy
Member
 
Registered: Jun 2004
Location: Wisconsin
Distribution: Ubuntu
Posts: 31

Rep: Reputation: 15
generate crc value for file(s) ?


i'm tired of googling... is there any !#&#@&$ way to generate a crc32 value for a file??? maybe like
$ md5sum file.ext
but only
$ crc32 file.ext
???

i don't know if there's a shell command that can do this or not... can't find one.. and i can't seem to find any hashing programs for linux
 
Old 06-30-2004, 07:42 PM   #2
Dark_Helmet
Senior Member
 
Registered: Jan 2003
Posts: 2,786

Rep: Reputation: 370Reputation: 370Reputation: 370Reputation: 370
I believe the command you're looking for is cksum
 
Old 06-30-2004, 07:49 PM   #3
Apostasy
Member
 
Registered: Jun 2004
Location: Wisconsin
Distribution: Ubuntu
Posts: 31

Original Poster
Rep: Reputation: 15
a crc32 value is 8 digits long... cksum gave me two values.. a 10 digit value and a 9 digit value.. niether one came close to the crc32 value of a known file i checked

upon cksum --help execution.. i get
or: cksum [OPTION]
Print CRC checksum and byte counts of each FILE.

--help display this help and exit
--version output version information and exit
.. if this a crc checking option.. why am i not getting an 8 digit number? i'm confused :/
educate this ignorant newbie

Last edited by Apostasy; 06-30-2004 at 07:58 PM.
 
Old 06-30-2004, 08:03 PM   #4
Dark_Helmet
Senior Member
 
Registered: Jan 2003
Posts: 2,786

Rep: Reputation: 370Reputation: 370Reputation: 370Reputation: 370
Sorry about that. I thought I remembered that the last byte of a crc was the parity or some other type of property. That's why I suggested it (10 hex digits = 5 bytes - 1 byte = 4 bytes = 32 bits)

I'm also a bit fuzzy about calculating CRC. I believe it's possible to calculate it in a number of different ways. For the files you have, did the website you downloaded them from provide information on how/what was used to calculate them? Or if you got it from another individual, you might be able to ask them?

The only checksum tools I'm aware of are md5sum, cksum, and plain old sum. I checked the man pages for md5sum, and it doesn't seem like there are options to "downshift" it to 32 bits. And I think sum reports a 16-bit CRC.

<edit>
When talking about CRC and saying things like crc16, crc32, and crc64, the number represents the size of each "block" of data used in calculating the crc. In other words, for crc16, the routine will split the file up into 16-bit chunks to work with. Similarly, it will be 32-bits for crc32, and so on.
</edt>

Last edited by Dark_Helmet; 06-30-2004 at 08:09 PM.
 
Old 07-02-2004, 03:41 AM   #5
Dark_Helmet
Senior Member
 
Registered: Jan 2003
Posts: 2,786

Rep: Reputation: 370Reputation: 370Reputation: 370Reputation: 370
I don't know if you've found a solution yet, but I did some more searching.

Here is a website that describes CRC error detection "painlessly": http://www.repairfaq.org/filipg/LINK/F_crc_v3.html

I also did some other checking, and noticed that the library zlib has a crc32 calculator in it. So I wrote a quickie program to use it. You can try it if you like, but i make no guarantees. If you read through the website above, it will explain how calculating a CRC can differ from one implementation to the next (specifically the polynomial chosen). Using the routines in zlib gives a little reassurance that the polynomial wasn't just plucked out of the air.

If you want to give my program a try, type this up (or copy-&-paste) it into a file. Save it as something like simple_crc.c. Actually, you can name it whatever you like, just make sure to use ".c" for the end of the filename.
Code:
#include <stdio.h>
#include <zlib.h>
#include <unistd.h>

int main( int argc, char *argv[] )
{
  FILE *input_file;
  char *file_buffer;
  unsigned long file_size;
  unsigned long bytes_read;
  unsigned long crc;
  unsigned long buffer_index;

  if( argc < 2 )
  {
    printf( "I need a filename to calculate crc on\n" );
    return 1;
  }

  input_file = fopen( argv[1], "r" );
  if( input_file == NULL )
  {
    printf( "Could not open %s for reading.\n", argv[1] );
    return 2;
  }

  fseek( input_file, 0, SEEK_END );
  file_size = ftell( input_file );
  fseek( input_file, 0, SEEK_SET );

  if( file_size == 0 )
  {
    printf( "File size is calculated to be 0. Cannot calculate CRC\n" );
    return 3;
  }

  file_buffer = (char *)malloc( file_size );
  if( file_buffer == NULL )
  {
    printf( "Unable to reserve enough memory for file buffer.\n" );
    return 4;
  }

  bytes_read = read( (int)input_file, file_buffer, file_size );

  if( bytes_read < file_size )
  {
    printf( "Did not read entire contents of the file into buffer.\n" );
    printf( "  number of bytes read: %d\n", bytes_read );
    free( file_buffer);
    return 5;
  }

  crc = crc32( 0L, Z_NULL, 0 );

  crc = crc32( crc, file_buffer, file_size );

  printf("Calculated CRC: %0X\n", crc );

  free( file_buffer );

  return 0;
}
At a command line, issue this command: gcc -o simple_crc -lz simple_crc.c

If everything went smoothly, you have a new file simply names "simple_crc" in the current directory. To use it: ./simple_crc filename

If the program spits out a message starting with anything other than "Calculated CRC:", then there was a problem. Since this is a quickie program, all you can really do is try again or verify the filename you gave is correct. In other words, it's not very robust when it comes to handling unexpected conditions.
 
Old 07-02-2004, 08:01 AM   #6
Apostasy
Member
 
Registered: Jun 2004
Location: Wisconsin
Distribution: Ubuntu
Posts: 31

Original Poster
Rep: Reputation: 15
thanks for sticking with me this long and no, i haven't found a solution to my problem yet...
i've tried the program you gave me however i'm getting different results for all my files... eg:
_____________________________________________________________________________________
[apostasy@localhost hack--DUSK]$ dir
(B-A)Hack_Legend_of_the_Twilight_-_01_(E48F18B2).mkv
(B-A)Hack_Legend_of_the_Twilight_-_02_(5023D2C9).mkv
(B-A)Hack_Legend_of_the_Twilight_-_03_(0875FB91).mkv
(B-A)Hack_Legend_of_the_Twilight_-_04_(CE26F317).mkv
[apostasy@localhost hack--DUSK]$ simple_crc \(B-A\)Hack_Legend_of_the_Twilight_- _01_\(E48F18B2\).mkv
Calculated CRC: 785D3BFE
[apostasy@localhost hack--DUSK]$ simple_crc \(B-A\)Hack_Legend_of_the_Twilight_- _02_\(5023D2C9\).mkv
Calculated CRC: E71E2733
[apostasy@localhost hack--DUSK]$ simple_crc \(B-A\)Hack_Legend_of_the_Twilight_- _03_\(0875FB91\).mkv
Calculated CRC: 87AD85D5
[apostasy@localhost hack--DUSK]$ simple_crc \(B-A\)Hack_Legend_of_the_Twilight_- _04_\(CE26F317\).mkv
Calculated CRC: 41A82D34
[apostasy@localhost hack--DUSK]$
_____________________________________________________________________________________
i have many files like this (with crc32 in their name) and unless they are all corrupted, i am not getting what i'm looking for.
the reason i need to get matching crc values is that i have many files which i do not know the value of... and need to know for database purposes.
I will ask around to see how these people are getting these values (what tools they are using)
btw simple_crc is awesome.. if this all gets sorted out i will be using it often
 
Old 07-08-2004, 02:26 AM   #7
elektron
LQ Newbie
 
Registered: Jul 2004
Location: Cave
Distribution: slack, gentoo, debian, roothat
Posts: 8

Rep: Reputation: 0
yea i had the same problems with that code.. didn't seem to work, could be a couple reasons.. first of all we're CRCin files that are at least a couple hundred megs, no need to load the whole thing into memory :-/..

anyway i'm werking on a better version, that might actually work ? not sure
 
Old 07-08-2004, 02:44 AM   #8
elektron
LQ Newbie
 
Registered: Jul 2004
Location: Cave
Distribution: slack, gentoo, debian, roothat
Posts: 8

Rep: Reputation: 0
crc32.c

here code that werks for me ( on a rcent gentoo linux) ==

Code:
gw tmp # gcc -o crc32 -lz crc32.c
gw tmp # ./crc32 \[S\^M\]\ InuYasha\ 138\ \[F712D55E\].avi
crc32.c by elektron
File Name: [S^M] InuYasha 138 [F712D55E].avi
File Size: 202235904
Calculated CRC: F712D55E
Code:
#include <errno.h>
#include <stdio.h>
#include <zlib.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>

int main( int argc, char *argv[] )
{
  FILE *input_file;
  char *file_buffer;
  unsigned long file_size;
  unsigned long chunk_size=1024;
  unsigned long bytes_read;
  unsigned long crc=0L;
  unsigned long buffer_index;
  int input_fd;

  printf( "crc32.c by elektron\n");
  if( argc < 2 )
  {
    printf( "I need a filename to calculate crc on\n" );
    return 1;
  }

  printf( "File Name: %s\n", argv[1] );
  input_file = fopen( argv[1], "r" );
  if( input_file == NULL )
  {
    printf( "Could not open %s for reading.\n", argv[1] );
    return 2;
  }

  fseek( input_file, 0, SEEK_END );
  file_size = ftell( input_file );
  rewind( input_file);

  printf( "File Size: %09d\n", file_size );

  if( file_size == 0 )
  {
    printf( "File size is calculated to be 0. Cannot calculate CRC\n" );
    return 3;
  }
  file_buffer = (char *)malloc( chunk_size+1 );
  if( file_buffer == NULL )
  {
    printf( "Unable to reserve enough memory for file buffer.\n" );
    return 4;
  }
  crc = crc32( crc, Z_NULL, 0 );

  input_fd = open( argv[1], O_RDONLY );
  if( input_fd == -1 )
  {
    printf( "Could not open %s for reading.\n", argv[1] );
    return 2;
  }
 while( 1 ){

 bytes_read = read( input_fd, file_buffer, chunk_size );

 if (bytes_read == 0 ) {
        break;
 }
 if (bytes_read == -1 ) {
        perror("read");
        break;
 }

  crc = crc32( crc, file_buffer, chunk_size );

}
  printf("Calculated CRC: %0X\n", crc );

  free( file_buffer );

  return 0;
}
happy verification :-P
 
Old 07-08-2004, 05:39 PM   #9
Apostasy
Member
 
Registered: Jun 2004
Location: Wisconsin
Distribution: Ubuntu
Posts: 31

Original Poster
Rep: Reputation: 15
Code:
[apostasy@localhost Kiddy Grade]$ crc32 \(B-A\)Kiddy_Grade_-_01_\(892D29A1\).mkv
crc32.c by elektron
File Name: (B-A)Kiddy_Grade_-_01_(892D29A1).mkv
File Size: 145790583
Calculated CRC: E4C83678
i'm getting different results for all my files
<EDIT>after thinking about it some i decided to try hasing some .avi files... your program works like its supposed to... but once i start hashing 'container' formats like .ogm or .mkv; that's where i am running into problems.. for some reason the mentioned below app does not give me the same problem</EDIT>

A friend pointed me to http://calcchecksum.sourceforge.net and i'm using this app for now, yet i'm still searching for a command-line oriented tool. I've emailed the author of calcchecksum and he told me that there are plans to implement command-line functions into this, yet he is currently not actively working on the project.
Code:
[apostasy@localhost Puni Puni Poemy]$ crc32 Puni.Puni.Poemy.01.Poemi.Is.In.A.Bad.Mood.\[AXP.DVDRip\].\[Dual.Audio\].\[7A9F8C2A\].ogm
crc32.c by elektron
File Name: Puni.Puni.Poemy.01.Poemi.Is.In.A.Bad.Mood.[AXP.DVDRip].[Dual.Audio].[7A9F8C2A].ogm
File Size: 366681173
Calculated CRC: D9D18910
Code:
[apostasy@localhost DayDream]$ crc32 \[A-Keep\]_DayDream_OVA_01_\[F978C89B\].avi
crc32.c by elektron
File Name: [A-Keep]_DayDream_OVA_01_[F978C89B].avi
File Size: 241700864
Calculated CRC: F978C89B

Last edited by Apostasy; 07-08-2004 at 05:53 PM.
 
Old 07-08-2004, 11:30 PM   #10
elektron
LQ Newbie
 
Registered: Jul 2004
Location: Cave
Distribution: slack, gentoo, debian, roothat
Posts: 8

Rep: Reputation: 0
wierd... not familiar with 'container' formats like .ogm or .mkv, but the source should werk for computing correct crc32 values for any file. maybe there's a more obscure bug. according to http://www.afterdawn.com/glossary/terms/container.cfm avi is also a 'container' format. /me looks into this
 
Old 07-08-2004, 11:37 PM   #11
Tinkster
Moderator
 
Registered: Apr 2002
Location: in a fallen world
Distribution: slackware by choice, others too :} ... android.
Posts: 23,067
Blog Entries: 11

Rep: Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910
You could try md5sum ... works for me :)


Cheers,
Tink
 
Old 07-08-2004, 11:52 PM   #12
elektron
LQ Newbie
 
Registered: Jul 2004
Location: Cave
Distribution: slack, gentoo, debian, roothat
Posts: 8

Rep: Reputation: 0
bug fixed :

found what appears to be a bug, also fixed option parsing.

from now on, you can grab newest version here : http://kremlor.net/projects/crc32/index.html

here's the latest code as of now :

Code:
/*******************************************************************************
  crc32.c by elektron
           
  Compile: 
           
    # gcc -lz -o crc32 crc32.c
           
  Usage:   
    # ./crc32 -h
           
    crc32 by elektron
      crc32 [-h] file1 [fileN...]
           
           
  Notes:   
    This is a hack from someone elses code :
     http://www.linuxquestions.org/questi...hreadid=199640   
           
    Find the most current version of this application at:
         http://kremlor.net/projects/crc32
           
           
*******************************************************************************/



#include <errno.h>
#include <stdio.h>
#include <zlib.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <libgen.h>


#define _GNU_SOURCE
#include <getopt.h>

int usage( char *progname )
{ 
  printf( "%s by elektron\n", progname);
  printf( "  %s [-h] file1 [fileN...]\n\n", progname );
}

unsigned long getFileSize( char *filename )
{
  FILE *input_file;
  unsigned long file_size;
  
  input_file = fopen( filename, "r" );
  if( input_file == NULL )
  { 
    printf( "Could not open %s for reading.\n", filename );
    return 0;
  }
  
  fseek( input_file, 0, SEEK_END );
  file_size = ftell( input_file );
  rewind( input_file);
  
  fclose( input_file );
  return file_size;

}

unsigned long computeCRC32( char *filename )
{ 
  int input_fd;
  char *file_buffer;
  unsigned long file_size;
  unsigned long chunk_size=1024;
  unsigned long bytes_read;
  unsigned long crc=0L;
  
  file_size = getFileSize( filename );
  
  if( file_size == 0 )
  { 
    printf( "File size is calculated to be 0. Cannot calculate CRC\n" );
    return 0;
  }
  
  file_buffer = (char *)malloc( chunk_size );
  if( file_buffer == NULL )
  { 
    printf( "Unable to reserve enough memory for file buffer.\n" );
    return 0;
  }
  crc = crc32( 0L, Z_NULL, 0 );
  
  input_fd = open( filename, O_RDONLY );
  if( input_fd == -1 )
  { 
    printf( "Could not open %s for reading.\n", filename );
    return 0;
  }
  while( 1 ){

   bytes_read = read( input_fd, file_buffer, chunk_size );

   if (bytes_read == 0 ) {
        break;
   }
   if (bytes_read == -1 ) {
        perror("read");
        break;
   }
   
   crc = crc32( crc, file_buffer, bytes_read );
  
  }
  
  free( file_buffer );
  
  close( input_fd );
  
  return crc;
}

int main( int argc, char *argv[] )
{ 
  unsigned long crc=0L;
  extern char *optarg;
  extern int optind, opterr, optopt;
  int i;
  
  while ( getopt( argc, argv, "h" ) != -1 ) {
  //    printf("optarg = %s, optind = %d, opterr = %d, optopt = %d\n",
  //            optarg, optind, opterr, optopt);
        if ( strncmp( argv[optind-1], "-h", 2 ) == 0 ) {
                usage( basename(argv[0]) );
                return 0;
        }
  }
 //printf("optarg = %s, optind = %d, opterr = %d, optopt = %d, argc = %d\n",
 //     optarg, optind, opterr, optopt, argc);
  
  
  if( argc < optind+1 )
  { 
    printf( "I need at least one filename to calculate crc on\n" );
    return 1;
  }
  for ( i=optind; i<argc; i++){
    crc = computeCRC32(argv[i] );

    printf("%s 0x%08X\n",argv[i], crc );
  }
}
/me appreciates bug reports..
 
Old 07-08-2004, 11:55 PM   #13
elektron
LQ Newbie
 
Registered: Jul 2004
Location: Cave
Distribution: slack, gentoo, debian, roothat
Posts: 8

Rep: Reputation: 0
Quote:
Originally posted by Tinkster
You could try md5sum ... works for me
we're talking about crc32 values here, not md5's
 
Old 07-09-2004, 12:02 AM   #14
elektron
LQ Newbie
 
Registered: Jul 2004
Location: Cave
Distribution: slack, gentoo, debian, roothat
Posts: 8

Rep: Reputation: 0
Code:
gw crc32 # nice crc32 /arc/pub/anime/00_Completed_Series_2/Lain/* 
/arc/pub/anime/00_Completed_Series_2/Lain/[KAA]_Serial_Experiments_Lain_01.DVD[F348D976].avi 0xF348D976
/arc/pub/anime/00_Completed_Series_2/Lain/[KAA]_Serial_Experiments_Lain_02.DVD[83D466C5].avi 0x83D466C5
gw crc32 # crc32 -h 
crc32 by elektron
  crc32 [-h] file1 [fileN...]
 
Old 07-09-2004, 02:23 AM   #15
Dark_Helmet
Senior Member
 
Registered: Jan 2003
Posts: 2,786

Rep: Reputation: 370Reputation: 370Reputation: 370Reputation: 370
Tinkster:
You and I think alike I already checked md5sum for its available options, but couldn't find anything telling it to downshift and calculate a 32-bit CRC. It spits out a full 128-bit hash which which is a bit more than Apostasy needs

elektron:
Take that code and run with it
Just a pointer: I noticed you swapped some things around a bit; All those "return X" statements for unexpected error conditions were to give a non-zero exit result for the program (just to follow traditional *nix guidelines). In your computeCRC32 function, you return 0 on those same error conditions. The only problem I see is that value gets assigned to the crc value. In theory, 0x00000000 is a valid CRC 32 value. It might be rare (specifically a 1 in about 4 billion chance), but in such a case, you wouldn't be able to distinguish whether it's a legitimate CRC or an error.

Also, you have a slight memory leak... Here's the sequence:
1. The malloc() in generateCRC32() goes off without a hitch
2. Suppose the open() call fails (returns -1)
In that scenario, you return from the generateCRC32() without freeing the memory for file_buffer. If the user supplied more than one filename on the command line, the possibility exists that you could chew through memory quickly.

Apostasy:
Did you ever find out what program the people used to calculate their CRC value? I trust the guys that wrote the zlib library, and that means the key portion of elektron's program is solid. If the value it spits out does not match what you have, then[list=1][*]There is a different polynomial used between the original CRC calculation and the one in zlib[*]The files actually are corrupt[*]The CRC does not apply to the "container" file[/list=1]
For #3, I'm suggesting something along the lines of a single zipped file. Say you had a gargantuan text file. You could calculate the CRC on the text file, then compress it. The CRC of the compressed file would not match the CRC of the uncompressed file. This would be a very, very odd way of distributing a CRC value, but it is possible.

As a side note, from the perspective of CRC, it should not matter whether a file is a "container", text, movie, or whatever. A CRC calculator simply looks at a file as long string of bytes; it doesn't know and doesn't care what those bytes are used for. If there is a CRC calculator that changes the calculation based on the type of file, then that's a crappy and completely untrustworthy program.

<edit>
One other thing to consider. These container formats you mentioned, are they anything like the relationship between wav files and mp3s? What I'm getting at is this (along the lines of #3 again). Say you downloaded a wav file with the crc embedded in the filename ("Sir_Mix_A_Lot-Baby_Got_Back(12AB34CD).wav"). If you encode that wav file to mp3, usually, only the extension changes ("Sir_Mix_A_Lot-Baby_Got_Back(12AB34CD).mp3"). However, the CRC value for the mp3 will not be the same as the CRC for the wav file; meaning the filename is giving misleading information. So, if the container files you have went through some sort of similar process, that can explain why you get different CRC values.
</edit>

Last edited by Dark_Helmet; 07-09-2004 at 02:39 AM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[N00B] Generate filenames from a file! Dee-ehn Programming 12 12-28-2004 11:06 AM
How may I tell gcc to generate an assembly file? LongName Programming 4 08-30-2004 03:31 AM
crc Straterra Slackware - Installation 2 03-05-2004 05:06 PM
Perl Script Reading a txt file and generate html to be published! kofi Linux - Software 1 09-22-2003 06:12 PM
How can I find/generate a kernel .config file? David Reid Linux - Newbie 5 08-08-2002 06:30 AM


All times are GMT -5. The time now is 08:42 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration