[SOLVED] merging *.csv files to remove duplicate lines

Originally Posted by shruggy Interestingly, recently there was another thread about these data.

GOOD CATCH. Automating the download process using lynx is a great addition, gets me outta the loop. Thanks.

Rather than lynx, I'd do it with cURL. See my post #6 there.

Originally Posted by shruggy No, I would do it not with lynx, but rather with cURL. See my post #6 there.

Roger that, will look at it. Thanks

After some testing I ended up with: sort -g -u file1 file2 -o file3
The -g resolves issues with sorting numerically on the .csv. I tried -n but sometimes the second field in the .csv would confuse things. So far the -g is working.
Thanks again, esp, wpeckham.

You are very welcome, and I am very glad that worked well for you!

Originally Posted by rdx After some testing I ended up with: sort -g -u file1 file2 -o file3 The -g resolves issues with sorting numerically on the .csv. I tried -n but sometimes the second field in the .csv would confuse things. So far the -g is working. Thanks again, esp, wpeckham.

I was wrong, -g and -u are conflicting somehow. I don't know what the deal is but be warned, these two "-g -u" don't play well together.

That makes sense. Sort -u only works if the data (files) are already sorted, -g changes the sort order.