LinuxQuestions.org
Latest LQ Deal: Linux Power User Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 04-14-2014, 11:51 AM   #1
papori
LQ Newbie
 
Registered: Feb 2011
Posts: 23

Rep: Reputation: 0
randomize shuffling of a file with many lines


Hi all,
I have a file with many lines (600M),and i want to create permutation of each line, random permutation.
input for example:
first line:
abcdef
second line:
abcdef

and so on..

Possible output:
first line:
efdacb
second line:
afebcd

any linux/perl/python solution will be helpfull!
Thanks,
Pap
 
Old 04-14-2014, 12:07 PM   #2
schneidz
LQ Guru
 
Registered: May 2005
Location: boston, usa
Distribution: fc-15/ fc-20-live-usb/ aix
Posts: 5,167

Rep: Reputation: 890Reputation: 890Reputation: 890Reputation: 890Reputation: 890Reputation: 890Reputation: 890
not really following but maybe this will help:
Code:
expr  600000 % $RANDOM
 
Old 04-14-2014, 12:35 PM   #3
metaschima
Senior Member
 
Registered: Dec 2013
Distribution: Slackware
Posts: 1,982

Rep: Reputation: 491Reputation: 491Reputation: 491Reputation: 491Reputation: 491
If you wanted to just mix up lines in a file, 'shuf' would work, but it seems you want to mix up individual characters within a line.

Do you accept C code for an answer ? With that many lines you probably need it to be fast.
 
Old 04-14-2014, 01:02 PM   #4
Guttorm
Senior Member
 
Registered: Dec 2003
Location: Trondheim, Norway
Distribution: Debian and Ubuntu
Posts: 1,295

Rep: Reputation: 335Reputation: 335Reputation: 335Reputation: 335
Hi

PHP has a function called str_shuffle that does the job, but only for a line. Something like this should do it:
PHP Code:
$fp fopen("filename.txt","r");
while ( (
$line fgets($fp)) !== false ) {
  
$line rtrim($line);
  echo 
str_shuffle($line),"\n";
}
fclose($fp); 

Last edited by Guttorm; 04-14-2014 at 01:05 PM.
 
Old 04-14-2014, 09:41 PM   #5
metaschima
Senior Member
 
Registered: Dec 2013
Distribution: Slackware
Posts: 1,982

Rep: Reputation: 491Reputation: 491Reputation: 491Reputation: 491Reputation: 491
Alright, here it is in C, it was a bit of practice for me:

Code:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#define STRING_MAX 256

int main (int argc, char * argv[])
{
	if (2 != argc)
	{
		printf ("Usage: %s infile\n", argv[0]);
		return 1;
	}

	char instring[STRING_MAX];
	unsigned int rseed;
	unsigned int i;

	char outstring[STRING_MAX] = {};
	char * strptr;

	FILE * infile = fopen (argv[1], "rb");
	if (NULL == infile)
	{
		fprintf (stderr, "ERROR: Cannot open %s\n", argv[1]);
		return 1;
	}

	FILE * urandom = fopen ("/dev/urandom", "rb");
	if (NULL == urandom)
	{
		fprintf (stderr, "ERROR: Cannot open /dev/urandom\n");
		return 1;
	}
	fread (&rseed, sizeof (rseed), 1, urandom);
	fclose (urandom);
	srand (rseed);

	while (fgets (instring, STRING_MAX, infile))
	{
		instring[strlen (instring) - 1] = 0;
		strptr = outstring;
		while (0 != strlen (instring))
		{
			unsigned int rnum = rand() % strlen (instring);
			*strptr = instring[rnum];
			strptr++;
			for (i = rnum; i < strlen (instring); i++)
			{
				instring[i] = instring[i+1];
			}
		}
		printf ("%s\n", outstring);
	}

	fclose (infile);
	return 0;
}
Compile it with 'gcc charshuf.c -Wall -o charshuf'.

Input:
Code:
abcdef
ghijkl
mnopqr
Output:
Code:
fdbace
hlgkji
nrpmqo
 
Old 04-15-2014, 01:11 PM   #6
papori
LQ Newbie
 
Registered: Feb 2011
Posts: 23

Original Poster
Rep: Reputation: 0
Thanks metaschima!!
I gave it a shoot, and it looks great!!
 
Old 04-15-2014, 01:19 PM   #7
metaschima
Senior Member
 
Registered: Dec 2013
Distribution: Slackware
Posts: 1,982

Rep: Reputation: 491Reputation: 491Reputation: 491Reputation: 491Reputation: 491
You may want to adjust the maximum string size STRING_MAX as needed for long lines.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Perl: how to replace blank lines in a file with given lines from another karamaz0v Programming 8 04-19-2012 07:48 AM
how to copy some lines in a file and delete these lines after gartura Linux - General 1 07-20-2010 09:55 AM
Delete Duplicate Lines in a file, leaving only the unique lines left xmrkite Linux - Software 6 01-14-2010 07:18 PM
replace several lines in a file with other lines in another file if condition yara Linux - General 12 10-27-2009 04:46 PM
Substitute specific lines with lines from another file rahmathullakm Programming 4 01-10-2009 06:47 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 08:16 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration