Formating a file with awk

tifu · 03-15-2005, 12:56 PM

Using KSH (and awk), can I get some help with reading file01 with the content below

================================
This is a sample text file
that contains letters and numbers,
some in UPPERCASE and some in lowercase.

-list 1
list 234
some words are not ForMATed correctly
================================

..... and outputing it to a new file (file02) with the format below? (each word on a new line, formated to one column, and numeric characters removed). If possible, the words should be in lowercase only, but that's not critical.

This
is
a
sample
text
file
that
contains
letters
and
numbers
some
in
UPPERCASE
and
some
in
lowercase
-list
list
some
words
are
not
ForMATed
correctly

Thanks

Tifu

whittycat · 03-15-2005, 01:33 PM

Forget awk. Sed will do this quite nicely with a little help from tr.

Put your text in a file, say qug, then

tr ' ' '\n' <qug | sed -e's/\.//;s/\,//;s/[0-9]*//;/^$/ d' >qug1

If that isn't exactly right it's got to be close.

Tony Sumner

tifu · 03-15-2005, 01:47 PM

Tony,

I appreciate the help. That's exactly what I needed.

Tifu

six6 · 03-15-2005, 03:07 PM

And if you needed an operating system independent method:

perl -e 'while (<>) {tr/A-Za-z\ \n-//cd; s/^\s+//; s/\s+/\n/g; s/(\w)/\L$1/g; print;}' < file1 > file2

Notice that does uppercase->lowercase too.

tifu · 03-16-2005, 11:12 PM

six6,

Wonderful! As an extra bonus (apart from changing the case for all to lowercase) the perl one-liner also removes all non-alpha characters like # $ % & etc.

What I intended (and did not add from the beginning) was to strip all but a-z characters, but keep [ - ] dash as in "e-mail", and [ ' ] appostrophe as in "ain't".

Could the perl solution you provided be modified easily to do that?

Thanks again

Tifu

six6 · 03-16-2005, 11:35 PM

@tifu

Unfortunately, you can't do this as a "one-liner" from the shell, because of the shell's weakness: it won't allow single quotes (apostrophes) inside other single quotes, even if preceded by a backslash!

But, you can easily overcome this. Just change the "one-liner" to it's own perl script!

Code:

#!/usr/bin/perl
open FILE, "file1" or die "Couldn't open file1 because: $!\n";
while (<FILE>) {
 tr/A-Za-z\ \n\-\\'//cd; # Deletes characters we didn't specify
 s/^\s+//; # Delete leading whitespace
 s/\s+/\n/g; # Changes all spaces to one new line
 s/(\w)/\L$1/g; # Changes letters to lowercase
 print;
}

And run it from your shell.

If you want to allow other things, like @ symbols for example, just change the line
tr/A-Za-z\ \n\-\\'//cd;
to
tr/\'\@A-Za-z\ \n\-\\'\@//cd;

Get the idea? Behold the power of perl!

Edit: Oops, should have put a "\" in front of the "-" originally...