LinuxQuestions.org - [SOLVED] how to group data in linux

Page 1 of 2

Show 50 post(s) from this thread on one page

- Linux - Newbie (https://www.linuxquestions.org/questions/linux-newbie-8/)

- - how to group data in linux (https://www.linuxquestions.org/questions/linux-newbie-8/how-to-group-data-in-linux-4175608313/)

Asoo	06-21-2017 08:38 AM

how to group data in linux

Greetings!

My file looks like this

Code:

ID        Num1        Num2

1        1        2

1        2        4

1        3        6

2        4        8

2        5        10

3        6        12

3        7        14

3        8        16

3        9        18

4        10        20

5        11        22

5        12        24

I want to group data in such a way that generation of each file should not contain more than 3 ID's. I have wrote a python script and it's working but the file is too huge, so it is taking a lot of time.

I have tried writing some awk commands but cannot figure it out how to put conditions in the command. (I am trying to use the command in loop for the creation of multiple files).

In python code looks like this:

Code:

infile=open("Input.txt","r")

outfile=open("Output1.txt","w")



list_value=[] # Storing the values of 1st column

file=2 # Naming the file

value="" # 1st column value

for line in infile:

    value=line.split()[0] 

    if value in list_value:

        outfile.write(line)

    else:

        list_value.append(value)

        if (len(list_value)) < 4:

            outfile.write(line)

        elif (len(list_value))==4:

            outfile=open("Output"+str(file)+".txt","w")

            outfile.write(line)

            file=file+1

            list_value=[]

            list_value.append(value)

infile.close()

outfile.close()

Thanks!

pan64

06-21-2017 09:23 AM

I would say awk will not be faster, so better to try to speed up using python.
You never close outfile which may cause problems.
can you tell us something about the size of this file and the running time?

BW-userx

06-21-2017 09:26 AM

the bigger a plate of food the longer it will take to eat all of it.

jeremy

06-21-2017 01:23 PM

BW-userx, you seem to have formed a pattern of posting off-topic or non-constructive comments. Please refrain from this moving forward. If you have any questions, feel free to contact a mod or myself directly.

--jeremy

Turbocapitalist

06-21-2017 01:43 PM

Quote:

Originally Posted by Asoo (Post 5725172)

I want to group data in such a way that generation of each file should not contain more than 3 ID's.

Can you post a quick sample of what you want the output to look like based o n the input data you have shown?

BW-userx

06-21-2017 01:44 PM

Quote:

Originally Posted by jeremy (Post 5725277)

the bigger a plate of food the longer it will take to eat all of it.
is a metaphoric explanation as to why it takes longer to process bigger files then smaller ones. if you need me to PM that to you I will.

in reference to this in post #1
"it's working but the file is too huge, so it is taking a lot of time."
and this in post #2
I would say awk will not be faster, so better to try to speed up using python.

Asoo	06-22-2017 02:47 AM

Quote:

Originally Posted by Turbocapitalist (Post 5725282)

Can you post a quick sample of what you want the output to look like based o n the input data you have shown?

Thank you so much for the reply.

1st file should look like this:

Code:

Second file should look like this:

Code:

Asoo	06-22-2017 02:49 AM

Quote:

Originally Posted by pan64 (Post 5725190)

Yeah, closing the outfiles should be done, I missed that line and edited my question again.

The file is almost 600 GB.

pan64

06-22-2017 02:58 AM

I would say you can try:

Code:

time cat file > newname

so you can check how long does it take. And you can check also your python script, theoretically that should be almost the same. You can only improve your script if that was definitely longer.

Asoo	06-22-2017 03:18 AM

Quote:

Originally Posted by pan64 (Post 5725503)

I would say you can try:

Code:

time cat file > newname

Okay, I will try this command and will update you with time. I'll also see how to improve script also.

Thank You!

JJJCR

06-22-2017 04:51 AM

600GB is quite a massive file for a text file.

I think the best way to approach it is to split into multiple files, into multiple outputs, then combine the multiple output to a manageable file and come up with a desired output.

I'm just curious what text editor do you use to open the 600GB file?

JJJCR

06-22-2017 04:55 AM

Quote:

Originally Posted by Asoo (Post 5725499)

Thank you so much for the reply.

1st file should look like this:

Code:

Second file should look like this:

Code:

Quote:

I want to group data in such a way that generation of each file should not contain more than 3 ID's.

In post #1 you mentioned that generation of each file should not contain more than 3 ID's, but in your desired output there are ID's more than 3. I supposed first column is the ID based on post #1.

Asoo	06-22-2017 05:00 AM

Quote:

Originally Posted by JJJCR (Post 5725557)

This file is a result of sequence processing. I want to use this file for further processes. That's why I need grouping and chunking of this file into multiple files each of almost 50 GB for the further process. After that, I can combine the output.

I am using vi as a text editor.

Asoo	06-22-2017 05:01 AM

Quote:

Originally Posted by JJJCR (Post 5725559)

In post #1 you mentioned that generation of each file should not contain more than 3 ID's, but in your desired output there are ID's more than 3. I supposed first column is the ID based on post #1.

The ID's are only 3 in the first file (1,2 and 3) but multiple values for each ID (which should be included).

pan64

06-22-2017 05:56 AM

probably you need to generate several pieces instead of that one big file.

All times are GMT -5. The time now is 06:40 PM.

Page 1 of 2

Show 50 post(s) from this thread on one page