[SOLVED] Sort with BUG?

perfection · 02-13-2019, 12:22 PM

I use the command several times to sort files and remove duplicates, but doing so with a file of 192,300 Lines it left many duplicate lines! However running this command on small files of type 1000 lines this never happened!

Code:

sort -u List -o List-End

I also tried these other commands but it always happens the same

Code:

sort List | uniq > List2.txt

awk '!i[$0]++' < List > List2

Anyone know to give some explanation?
Or any idea how I can remove the duplicates from my file?

Turbocapitalist · 02-13-2019, 12:39 PM

Have you checked the duplicates for extraneous white space? That gets counted in both sort and uniq.

Edit:

Code:

sort list | sed -r 's/[[:space:]]+$//' | uniq -c > list2

pan64 · 02-13-2019, 12:47 PM

yes, if you can't post duplicated lines you need to compare them yourself and find the differences. Probably that is whitespace or some other tricky char.
You can remove almost 99% of that big file just keep some lines which look similar (after a sort - with or without -u).

perfection · 02-13-2019, 04:14 PM

I noticed that sort sorts the files differently than the Atom Editor
I noticed that by ordering the file by Atom Editor the command below also works.

Code:

uniq List > List-uniq

Using this command below informed by @Turbocapitalist
I have verified that the final line qtd is the same as the work done by Atom Editor or the previous command that I reported

Code:

sort List | sed -r 's/[[:space:]]+$//' | uniq > List2

However when comparing this file in Kompare it says they are different files, it sees all the lines as a single line, making it difficult to understand the difference between the 2 generated files

I did not understand this sed command. If you can explain? I did not understand what he does.

Code:

sed -r 's/[[:space:]]+$//'

Sorting Sort, Example:
###A9AdsMiddleBoxTop
###A9AdsOutOfStockWidgetTop
###A9AdsServicesWidgetTop
###AD-300x250
###AD-300x250-1
###AD-300x250-2
###AD-300x250-3
###AD-HOME-LEFT

Sorting Atom Editor, Example:
_.gif?ref=
_.gif?t=
_03100x07696.$image,domain=pcwelt.de
_100_ad.
_100x480_
_115x220.
_120_60.
_120_600_

Images to facilitate

Turbocapitalist · 02-14-2019, 12:19 AM

Quote:

Originally Posted by perfection

I did not understand this sed command. If you can explain? I did not understand what he does.

Code:

sed -r 's/[[:space:]]+$//'

The -r tells it to use Extended Regular Expressions, which are a bit easier to read and have more choices.

The s/ / / is a substitution command.

The [[:something:]] is a character class. The one above covers all white space, including tabs, and the plus means check for one or more of them. Since it is immediately followed by a dollar sign $, which stands for the end of the line, it means seach for one or more white spaces at the end of the line. So it will zap spaces at the end of the line but leave them alone elsehwere in the line.

See the following for the authoritative answer.

Code:

man sed
man 7 regex

l0f4r0 · 02-14-2019, 04:15 AM

@perfection: so are the following your 2 remaining issues now?

sort in Atom Editor gives different output than GNU sort.
>> What command did you use exactly in Atom editor? Because depending on the packages installed, it seems that there can be different kind of sortings (https://github.com/atom/sort-lines)
Kompare tells you files are different.
>> What about vimdiff or diff?