Multiple column sort
 

Hi,

I have some data in the following format:
My goal is to get the maximum value from the right column for each unique value in the left column. For instance, after processing the following 4 lines:
I would like to get just the last line,
since "5.952" is the largest value for "1298501997.943".

Similarly, for the following lines:
I would like to get:
And for:
simply:
and so on...

I tried searching for some hints in awk/uniq/etc., but not sure even how to formulate the query.
I could write a Python script, but it feels that proceeding with awk or some other standard tools would be more efficient (especially since I have a lot of data - millions/tens of millions of lines).

PS: Is there any Python module for text processing scenarios like that?

Thank you

Hi,

Is this what you are looking for: Hope this helps.

Yes, exactly what I was looking for! Thank you very much!

You're welcome :)

For the exercise, here is a Ruby solution:

Druuna's answer is excellent. It is the portable, simple solution.

Since you expressed an interest in awk programming, here is an alternate solution, with explanations:
The (NF>1) rule means that the following action is only done for records (lines) that contain more than one field (word). If the first field is a key in the associative array data and the second field is larger than the existing value, the value is updated. If the first field is not a key, then the value is added to the array.

This condition is needed because awk considers nonexistent equal to zero. Without the condition, it would ignore negative numbers! This way, it will accept anything as the first value (second field), even a string, and update it if it sees anything it considers "larger". This works for numbers as well as strings.

While this streams the data (in linear time with respect to input size), it does keep the data in memory until the end. For each key (first field), only the thus far seen largest value is kept, so typically it should use less memory than presorting it with e.g. sort command. For normal data sets this does not matter, but if you have a humongous data set, with a lot of duplicate keys, this method may be faster.

The END rule is only run once after everything else. It will loop over all keys in the data array (first fields from the input), and output the value for each one. Note that I use %s for strings; the fields are only interpreted when comparing, never converted from/to numbers, so you are assured to get the original values. (Note that Druuna's solution does the same, it also always yields the original data, without conversions.)

The output order is random by default. If you use mawk, you can add PROCINFO["sorted_in"]="@ind_num_asc" at the beginning of the END rule to get the output sorted. If you use gawk, you can modify the END rule, to
where the END rule first sort the keys (first fields) into array key, then output the values (second fields) in that order.

Unfortunately POSIX does not specify any portable sort functions. While you can obviously code one yourself, it would be counter-productive; the sort command does it much more efficiently. If you have few duplications, I'd recommend Druuna's solution. If you have lots and lots of duplicate keys, using this version and sorting the results may be a bit faster or use less RAM.

And a slightly condensed version of druuna's:
And an alternate ruby:

Cool,good solution!