text file manipulation

crowzie · 08-13-2015, 08:38 AM

#first transformed all spaces into a new line:

sed 'se[[:space:]]e\neg' -i textfile.txt

#Removed all empty lines:

sed '/^$/d' -i textfile.txt

#Then sorted alphabetically and removed duplicates:

cat textfile.txt | sort | uniq > textfile_sorted_uniq.txt

#Text is now in single column format

#Im still left with the entries that had duplicates when all I need is the entries that never had duplicates to begin with.

pan64 · 08-13-2015, 09:13 AM

Code:

#first transformed all spaces into a new line:
sed 'se[[:space:]]e\neg' -i textfile.txt

#Removed all empty lines:
sed '/^$/d' -i textfile.txt

# then comes what I already wrote:
sort textfile.txt | uniq -d > duplicates
grep -vFxf duplicates textfile.txt > result

(do not use cat file | sort | uniq, but sort -u file instead)

dugan · 08-13-2015, 02:46 PM

Quote:

Originally Posted by syg00

@dugan, is that guaranteed to print in the order of input ?.

Yes.