Create words file from a book

freeroute · 11-26-2016, 03:36 AM

Hello,

I have some books in format plain text (extension .txt) I would like to create a word file from these books (put every words into new row). What methods do you recommend?

Part of the book:
"I.

IN HET DORP.

Ik ben een vondeling.

Maar tot mijn achtste jaar geloofde ik, evenals alle andere kinderen, ook eene moeder te hebben, want als ik weende, was er eene vrouw die mij in hare armen nam en mij tegen haar boezem drukte totdat mijne tranen ophielden te vloeien."

I use this method:

1. Arrange every words into new row:

Code:

tr ' \t' '\n' <Alleen_op_de_Wereld.txt >book.txt

Do you have a better command for this step?
2. sort, uniq commands book.txt
3. remove special characters with commands

Code:

tr -d ':' <inputfile>outputfile

Code:

tr -d ';' <inputfile>outputfile

etc..
4. at last

Code:

cat book1.txt book2.txt etc.>words_of_books.txt

Thank you in advance.

kaz2100 · 11-26-2016, 03:44 AM

It should not be difficult, I guess.

It all depends what you want. So, it is a kind of hard for us to give suggestions without knowing your goal.

freeroute · 11-26-2016, 03:49 AM

Quote:

Originally Posted by kaz2100

It should not be difficult, I guess.

It all depends what you want. So, it is a kind of hard for us to give suggestions without knowing your goal.

These are dutch books. My goal to create a uniq, lowercase dictionary plain text file.

pan64 · 11-26-2016, 03:50 AM

I would say it looks like a homework (or similar) and also you can find several solutions on the net, you just need to google....

Jjanel · 11-26-2016, 03:59 AM

How familiar are you with "regex"? sed/vi/perl?
Random bits: (globally)
s/[^a-Z0-9]/ / (turn all NON-alphanumerics to spaces)

Code:

s/  / /

(change any 2 or more spaces into one: needed code tags for 2spaces)
tr... uppercase lowercase
(now, only 'alpha-numeric' single-space-seperated words left)
then turn all the spaces into \n
then your sort&uniq

freeroute · 11-26-2016, 04:01 AM

Quote:

Originally Posted by Jjanel

How familiar are you with "regex"? sed/vi/perl?
Random bits: (globally)
s/[^a-Z0-9]/ /
s/ / /
(now, only 'alpha-numeric' single-space-seperated words left)
then turn all the spaces into \n
then your sort&uniq

Thank you very much.

kakistocrat · 11-26-2016, 04:08 AM

It can be done in "awk", too, as a one-liner.

Though with "tr" you have to convert case, then change white space to new lines, then sort, then select unique instances.

Jjanel · 11-26-2016, 04:17 AM

Thanks for the polite "Thank you very much."! And posting 'trys' in #1! (important @LQ)
One more friendly&supportive reminder: post your solution when you mark Thread [SOLVED]

pan64 · 11-26-2016, 04:49 AM

https://is.gd/53by1d
http://unix.stackexchange.com/questi...-output-sorted

grail · 11-26-2016, 06:16 AM

Ditto to awk / perl / ruby / python being able to perform the task rather easily.

tronayne · 11-26-2016, 09:58 AM

This gets real easy with awk.

Here's a sample program (from Alfred V. Aho, Brian W. Kernighan, Peter J. Weinberger, The AWK Programming Language:

Code:

cat wordfreq
# wordfreq - print number of occurrences of each word
#   input:  text
#   output: number-word pairs sorted by number

    { gsub(/[.,:;!?(){}]/, "")    # remove punctuation
      for (i = 1; i <= NF; i++)
          count[$i]++
    }
END { for (w in count)
          print count[w], w | "sort -rn"
    }

You execute it:

Code:

awk -f wordfreq filename.txt > list.txt

Which, as indicated, print a unique list of words with the number of uses of those words in the input text (that's kind of handy to know in some instances).

Now, what you really want is a unique list of words in your text document.

So, you can peel the number off with sed or simply change the print directive in the program to not print the number (and modify the sort directive while you're at it -- you're not sorting by numeric).

I suspect you're doing this for creating an index of the document? Once you have the list of subjects (which you will need to edit out by hand (or use sed to remove the words you don't want) then use the indexing capability of LibreOffice or OpenOffice (you'll need to read up on how to do that but the main thing you need is the unique list of words).

So, give it a go and see what you get; the language is not relevant, the words are simply unique patterns of text.

Hope this helps some.

keefaz · 11-26-2016, 11:56 AM

Same kind of script with perl one liner

Code:

perl -lne 'map {$s{lc $_}++ if $_} split /\W+/; END{print for sort keys %s}' input.txt > output.txt

- split /\W+/ : split lines by non word characters
- $s{lc $_}++ : convert to lower case, store word as key in %s hash, increment its value (word count)
- END{print for sort keys %s} : print each key in %s (sorted alphabetically)

grail · 11-26-2016, 01:47 PM

A quick sed / sort for the same:

Code:

sed -r -e 's/\W\B|\s/\n/g;s/.*/\L&/' file | sort -u

I added a little to the splitting option as my file had words like "won't" which left me with "won" and "t" using just '\W'

The awk is a bit longer and convoluted:

Code:

awk 'NF && !_[$0]++{print tolower($0)}' RS="\\\W\\\B|\\\s" file | sort

And some ruby:

Code:

ruby -F'\W\B|\s' -lane 'a||=[];a << $F.map(&:downcase).reject{|x| x.empty?};END{puts a.flatten.uniq.sort}' file

keefaz · 11-27-2016, 09:17 AM

Quote:

Originally Posted by grail

I added a little to the splitting option as my file had words like "won't" which left me with "won" and "t" using just '\W'

Yes good catch!