randomize shuffling of a file with many lines

papori · 04-14-2014, 10:51 AM

Hi all,
I have a file with many lines (600M),and i want to create permutation of each line, random permutation.
input for example:
first line:
abcdef
second line:
abcdef

and so on..

Possible output:
first line:
efdacb
second line:
afebcd

any linux/perl/python solution will be helpfull!
Thanks,
Pap

schneidz · 04-14-2014, 11:07 AM

not really following but maybe this will help:

Code:

expr  600000 % $RANDOM

metaschima · 04-14-2014, 11:35 AM

If you wanted to just mix up lines in a file, 'shuf' would work, but it seems you want to mix up individual characters within a line.

Do you accept C code for an answer ? With that many lines you probably need it to be fast.

Guttorm · 04-14-2014, 12:02 PM

Hi

PHP has a function called str_shuffle that does the job, but only for a line. Something like this should do it:

PHP Code:



$fp = fopen("filename.txt","r");
while ( ($line = fgets($fp)) !== false ) {
  $line = rtrim($line);
  echo str_shuffle($line),"\n";
}
fclose($fp);

metaschima · 04-14-2014, 08:41 PM

Alright, here it is in C, it was a bit of practice for me:

Code:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#define STRING_MAX 256

int main (int argc, char * argv[])
{
	if (2 != argc)
	{
		printf ("Usage: %s infile\n", argv[0]);
		return 1;
	}

	char instring[STRING_MAX];
	unsigned int rseed;
	unsigned int i;

	char outstring[STRING_MAX] = {};
	char * strptr;

	FILE * infile = fopen (argv[1], "rb");
	if (NULL == infile)
	{
		fprintf (stderr, "ERROR: Cannot open %s\n", argv[1]);
		return 1;
	}

	FILE * urandom = fopen ("/dev/urandom", "rb");
	if (NULL == urandom)
	{
		fprintf (stderr, "ERROR: Cannot open /dev/urandom\n");
		return 1;
	}
	fread (&rseed, sizeof (rseed), 1, urandom);
	fclose (urandom);
	srand (rseed);

	while (fgets (instring, STRING_MAX, infile))
	{
		instring[strlen (instring) - 1] = 0;
		strptr = outstring;
		while (0 != strlen (instring))
		{
			unsigned int rnum = rand() % strlen (instring);
			*strptr = instring[rnum];
			strptr++;
			for (i = rnum; i < strlen (instring); i++)
			{
				instring[i] = instring[i+1];
			}
		}
		printf ("%s\n", outstring);
	}

	fclose (infile);
	return 0;
}

Compile it with 'gcc charshuf.c -Wall -o charshuf'.

Input:

Code:

abcdef
ghijkl
mnopqr

Output:

Code:

fdbace
hlgkji
nrpmqo

papori · 04-15-2014, 12:11 PM

Thanks metaschima!!
I gave it a shoot, and it looks great!!

metaschima · 04-15-2014, 12:19 PM

You may want to adjust the maximum string size STRING_MAX as needed for long lines.