I have a file of lines that I want to keep in a non-sorted (as far as any Unix app can tell) order but remove duplicates.

Yes, there's lots of ways: https://duckduckgo.com/?q=linux+remove+duplicate+lines+without+sorting

3 members found this post helpful.

Sounds like you are looking for a "uniq" command that does not require source file to be sorted and is not interactive.
I do not think there is such a command.
However; in the past I dabbled with using this Windows based program running under Wine: WinMerge

--ET

although the reverse logic in @boughtonp link is cute

1 members found this post helpful.

You could do it with AWK and an associative array keyed on the contents of each line. I'm not sure how well that would scale though. How large a text file are you considering?

Most resources suggest the ulta-short
More explicit is
You can append a file name, otherwise it reads stdin.

1 members found this post helpful.

While I will not now write the "necessary one-liner" for you, the algorithm essentially is this:

• Produce a version of the input file which contains a "record number" field to the left of the record's contents.
• Sort the resulting file by the second field: the actual content
• Now that the "duplicate values" are adjacent, considering only the second field remove all but the first occurrence. This is trivial now, because you need only consider the current value against its immediate predecessor.
• Re-sort the resulting file by the first ("record number") field.
• Remove the "record number" field to produce the final result.

Fifty years ago, they did this with magnetic tapes. It may well be that they did it earlier using punched cards.

If you dont mind the file being sorted use