[SOLVED] how to group data in linux

Asoo · 06-21-2017, 08:38 AM

Greetings!

My file looks like this

Code:

ID	Num1	Num2
1	1	2
1	2	4
1	3	6
2	4	8
2	5	10
3	6	12
3	7	14
3	8	16
3	9	18
4	10	20
5	11	22
5	12	24

I want to group data in such a way that generation of each file should not contain more than 3 ID's. I have wrote a python script and it's working but the file is too huge, so it is taking a lot of time.

I have tried writing some awk commands but cannot figure it out how to put conditions in the command. (I am trying to use the command in loop for the creation of multiple files).

In python code looks like this:

Code:

infile=open("Input.txt","r")
outfile=open("Output1.txt","w")

list_value=[] # Storing the values of 1st column
file=2 # Naming the file
value="" # 1st column value
for line in infile:
    value=line.split()[0] 
    if value in list_value:
        outfile.write(line)
    else:
        list_value.append(value)
        if (len(list_value)) < 4:
            outfile.write(line)
        elif (len(list_value))==4:
            outfile=open("Output"+str(file)+".txt","w")
            outfile.write(line)
            file=file+1
            list_value=[]
            list_value.append(value)
infile.close()
outfile.close()

Thanks!

pan64 · 06-21-2017, 09:23 AM

I would say awk will not be faster, so better to try to speed up using python.
You never close outfile which may cause problems.
can you tell us something about the size of this file and the running time?

BW-userx · 06-21-2017, 09:26 AM

the bigger a plate of food the longer it will take to eat all of it.

jeremy · 06-21-2017, 01:23 PM

BW-userx, you seem to have formed a pattern of posting off-topic or non-constructive comments. Please refrain from this moving forward. If you have any questions, feel free to contact a mod or myself directly.

--jeremy

Turbocapitalist · 06-21-2017, 01:43 PM

Quote:

Originally Posted by Asoo

I want to group data in such a way that generation of each file should not contain more than 3 ID's.

Can you post a quick sample of what you want the output to look like based o n the input data you have shown?

BW-userx · 06-21-2017, 01:44 PM

Quote:

Originally Posted by jeremy

BW-userx, you seem to have formed a pattern of posting off-topic or non-constructive comments. Please refrain from this moving forward. If you have any questions, feel free to contact a mod or myself directly.

--jeremy

the bigger a plate of food the longer it will take to eat all of it.
is a metaphoric explanation as to why it takes longer to process bigger files then smaller ones. if you need me to PM that to you I will.

in reference to this in post #1
"it's working but the file is too huge, so it is taking a lot of time."
and this in post #2
I would say awk will not be faster, so better to try to speed up using python.

Asoo · 06-22-2017, 02:47 AM

Quote:

Originally Posted by Turbocapitalist

Can you post a quick sample of what you want the output to look like based o n the input data you have shown?

Thank you so much for the reply.

1st file should look like this:

Code:

Second file should look like this:

Code:

4   10  20
5   11  22
5   12  24

Asoo · 06-22-2017, 02:49 AM

Quote:

Originally Posted by pan64

I would say awk will not be faster, so better to try to speed up using python.
You never close outfile which may cause problems.
can you tell us something about the size of this file and the running time?

Yeah, closing the outfiles should be done, I missed that line and edited my question again.

The file is almost 600 GB.

pan64 · 06-22-2017, 02:58 AM

I would say you can try:

Code:

time cat file > newname

so you can check how long does it take. And you can check also your python script, theoretically that should be almost the same. You can only improve your script if that was definitely longer.

Asoo · 06-22-2017, 03:18 AM

Quote:

Originally Posted by pan64

I would say you can try:

Code:

time cat file > newname

Okay, I will try this command and will update you with time. I'll also see how to improve script also.

Thank You!

JJJCR · 06-22-2017, 04:51 AM

600GB is quite a massive file for a text file.

I think the best way to approach it is to split into multiple files, into multiple outputs, then combine the multiple output to a manageable file and come up with a desired output.

I'm just curious what text editor do you use to open the 600GB file?

JJJCR · 06-22-2017, 04:55 AM

Quote:

Originally Posted by Asoo

Thank you so much for the reply.

1st file should look like this:

Code:

Second file should look like this:

Code:

4   10  20
5   11  22
5   12  24

Quote:

I want to group data in such a way that generation of each file should not contain more than 3 ID's.

In post #1 you mentioned that generation of each file should not contain more than 3 ID's, but in your desired output there are ID's more than 3. I supposed first column is the ID based on post #1.

Asoo · 06-22-2017, 05:00 AM

Quote:

Originally Posted by JJJCR

600GB is quite a massive file for a text file.

I think the best way to approach it is to split into multiple files, into multiple outputs, then combine the multiple output to a manageable file and come up with a desired output.

I'm just curious what text editor do you use to open the 600GB file?

This file is a result of sequence processing. I want to use this file for further processes. That's why I need grouping and chunking of this file into multiple files each of almost 50 GB for the further process. After that, I can combine the output.

I am using vi as a text editor.

Asoo · 06-22-2017, 05:01 AM

Quote:

Originally Posted by JJJCR

In post #1 you mentioned that generation of each file should not contain more than 3 ID's, but in your desired output there are ID's more than 3. I supposed first column is the ID based on post #1.

The ID's are only 3 in the first file (1,2 and 3) but multiple values for each ID (which should be included).

pan64 · 06-22-2017, 05:56 AM

probably you need to generate several pieces instead of that one big file.