Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question?
If it is not in the man pages or the how-to's this is the place! |
Notices |
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
Are you new to LinuxQuestions.org? Visit the following links:
Site Howto |
Site FAQ |
Sitemap |
Register Now
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
|
 |
11-26-2016, 03:36 AM
|
#1
|
Member
Registered: Jul 2016
Location: Hungary
Distribution: Debian
Posts: 69
Rep: 
|
Create words file from a book
Hello,
I have some books in format plain text (extension .txt) I would like to create a word file from these books (put every words into new row). What methods do you recommend?
Part of the book:
"I.
IN HET DORP.
Ik ben een vondeling.
Maar tot mijn achtste jaar geloofde ik, evenals alle andere kinderen, ook eene moeder te hebben, want als ik weende, was er eene vrouw die mij in hare armen nam en mij tegen haar boezem drukte totdat mijne tranen ophielden te vloeien."
I use this method:
1. Arrange every words into new row:
Code:
tr ' \t' '\n' <Alleen_op_de_Wereld.txt >book.txt
Do you have a better command for this step?
2. sort, uniq commands book.txt
3. remove special characters with commands
Code:
tr -d ':' <inputfile>outputfile
Code:
tr -d ';' <inputfile>outputfile
etc..
4. at last
Code:
cat book1.txt book2.txt etc.>words_of_books.txt
Thank you in advance.
|
|
|
11-26-2016, 03:44 AM
|
#2
|
Senior Member
Registered: Apr 2005
Location: Penguin land, with apple, no gates
Distribution: SlackWare > Debian testing woody(32) sarge etch lenny squeeze(+64) wheezy .. bullseye bookworm
Posts: 1,849
Rep: 
|
It should not be difficult, I guess.
It all depends what you want. So, it is a kind of hard for us to give suggestions without knowing your goal.
|
|
|
11-26-2016, 03:49 AM
|
#3
|
Member
Registered: Jul 2016
Location: Hungary
Distribution: Debian
Posts: 69
Original Poster
Rep: 
|
Quote:
Originally Posted by kaz2100
It should not be difficult, I guess.
It all depends what you want. So, it is a kind of hard for us to give suggestions without knowing your goal.
|
These are dutch books. My goal to create a uniq, lowercase dictionary plain text file.
|
|
|
11-26-2016, 03:50 AM
|
#4
|
LQ Addict
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 24,678
|
I would say it looks like a homework (or similar) and also you can find several solutions on the net, you just need to google....
|
|
|
11-26-2016, 03:59 AM
|
#5
|
Member
Registered: Jun 2016
Distribution: any&all, in VBox; Ol'UnixCLI; NO GUI resources
Posts: 999
|
How familiar are you with "regex"? sed/vi/perl?
Random bits: (globally)
s/[^a-Z0-9]/ / (turn all NON-alphanumerics to spaces)
(change any 2 or more spaces into one: needed code tags for 2spaces)
tr... uppercase lowercase
(now, only 'alpha-numeric' single-space-seperated words left)
then turn all the spaces into \n
then your sort&uniq
Last edited by Jjanel; 11-26-2016 at 04:46 AM.
Reason: fix the TWO-spaces with CODE tag
|
|
1 members found this post helpful.
|
11-26-2016, 04:01 AM
|
#6
|
Member
Registered: Jul 2016
Location: Hungary
Distribution: Debian
Posts: 69
Original Poster
Rep: 
|
Quote:
Originally Posted by Jjanel
How familiar are you with "regex"? sed/vi/perl?
Random bits: (globally)
s/[^a-Z0-9]/ /
s/ / /
(now, only 'alpha-numeric' single-space-seperated words left)
then turn all the spaces into \n
then your sort&uniq
|
Thank you very much.
|
|
|
11-26-2016, 04:08 AM
|
#7
|
LQ Guru
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 7,756
|
It can be done in "awk", too, as a one-liner.
Though with "tr" you have to convert case, then change white space to new lines, then sort, then select unique instances.
|
|
1 members found this post helpful.
|
11-26-2016, 04:17 AM
|
#8
|
Member
Registered: Jun 2016
Distribution: any&all, in VBox; Ol'UnixCLI; NO GUI resources
Posts: 999
|
Thanks for the polite "Thank you very much."! And posting 'trys' in #1! (important @LQ)
One more friendly&supportive reminder: post your solution when you mark Thread [SOLVED] 
|
|
1 members found this post helpful.
|
11-26-2016, 04:49 AM
|
#9
|
LQ Addict
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 24,678
|
|
|
1 members found this post helpful.
|
11-26-2016, 06:16 AM
|
#10
|
LQ Guru
Registered: Sep 2009
Location: Perth
Distribution: Arch
Posts: 10,042
|
Ditto to awk / perl / ruby / python being able to perform the task rather easily.
|
|
1 members found this post helpful.
|
11-26-2016, 09:58 AM
|
#11
|
Senior Member
Registered: Oct 2003
Location: Northeastern Michigan, where Carhartt is a Designer Label
Distribution: Slackware 32- & 64-bit Stable
Posts: 3,541
|
This gets real easy with awk.
Here's a sample program (from Alfred V. Aho, Brian W. Kernighan, Peter J. Weinberger, The AWK Programming Language:
Code:
cat wordfreq
# wordfreq - print number of occurrences of each word
# input: text
# output: number-word pairs sorted by number
{ gsub(/[.,:;!?(){}]/, "") # remove punctuation
for (i = 1; i <= NF; i++)
count[$i]++
}
END { for (w in count)
print count[w], w | "sort -rn"
}
You execute it:
Code:
awk -f wordfreq filename.txt > list.txt
Which, as indicated, print a unique list of words with the number of uses of those words in the input text (that's kind of handy to know in some instances).
Now, what you really want is a unique list of words in your text document.
So, you can peel the number off with sed or simply change the print directive in the program to not print the number (and modify the sort directive while you're at it -- you're not sorting by numeric).
I suspect you're doing this for creating an index of the document? Once you have the list of subjects (which you will need to edit out by hand (or use sed to remove the words you don't want) then use the indexing capability of LibreOffice or OpenOffice (you'll need to read up on how to do that but the main thing you need is the unique list of words).
So, give it a go and see what you get; the language is not relevant, the words are simply unique patterns of text.
Hope this helps some.
Last edited by tronayne; 11-26-2016 at 10:01 AM.
Reason: Forgot something.
|
|
1 members found this post helpful.
|
11-26-2016, 11:56 AM
|
#12
|
LQ Guru
Registered: Mar 2004
Distribution: Slackware
Posts: 6,858
|
Same kind of script with perl one liner
Code:
perl -lne 'map {$s{lc $_}++ if $_} split /\W+/; END{print for sort keys %s}' input.txt > output.txt
- split /\W+/ : split lines by non word characters
- $s{lc $_}++ : convert to lower case, store word as key in %s hash, increment its value (word count)
- END{print for sort keys %s} : print each key in %s (sorted alphabetically)
|
|
1 members found this post helpful.
|
11-26-2016, 01:47 PM
|
#13
|
LQ Guru
Registered: Sep 2009
Location: Perth
Distribution: Arch
Posts: 10,042
|
A quick sed / sort for the same:
Code:
sed -r -e 's/\W\B|\s/\n/g;s/.*/\L&/' file | sort -u
I added a little to the splitting option as my file had words like "won't" which left me with "won" and "t" using just '\W'
The awk is a bit longer and convoluted:
Code:
awk 'NF && !_[$0]++{print tolower($0)}' RS="\\\W\\\B|\\\s" file | sort
And some ruby:
Code:
ruby -F'\W\B|\s' -lane 'a||=[];a << $F.map(&:downcase).reject{|x| x.empty?};END{puts a.flatten.uniq.sort}' file
|
|
2 members found this post helpful.
|
11-27-2016, 09:17 AM
|
#14
|
LQ Guru
Registered: Mar 2004
Distribution: Slackware
Posts: 6,858
|
Quote:
Originally Posted by grail
I added a little to the splitting option as my file had words like "won't" which left me with "won" and "t" using just '\W'
|
Yes good catch! 
|
|
1 members found this post helpful.
|
All times are GMT -5. The time now is 10:54 AM.
|
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
|
Latest Threads
LQ News
|
|