POSIX Character Classes

brgr88 · 04-20-2013, 11:11 AM

Is there a way to print the member characters of the various POSIX character classes?
I'm essentially looking for a portable way to list the characters in the classes for use in character sets other than ASCII.

For instance, something like:
echo [:graph:] > file

to give me all printables except space, and so forth.

konsolebox · 04-21-2013, 11:28 PM

I'm not sure but perhaps tput could help you. If not I guess you could examine grep's source code.

David the H. · 04-23-2013, 09:35 AM

That's an interesting question, and I've wondered about it myself in the past. Unfortunately I don't think there's anything like a simple command available. You'll probably just have to do some testing with the encodings you're interested in.

After a bit of internet snooping I came across this page describing the unicode collation algorithm:

http://www.unicode.org/reports/tr10/

I've only just glanced through it myself, but I think it's safe to say that the answer isn't as simple as pulling up a chart listing the correct order. In particular section 1.8 points out that collation is a function of the language setting, not the character set or encoding used. So if there's an answer anywhere, it's likely buried somewhere in the various language definition files.

danielbmartin · 04-23-2013, 06:59 PM

Quote:

Originally Posted by brgr88

Is there a way to print the member characters of the various POSIX character classes?

Perhaps I don't understand the nuances of the problem. Here's an approach which might be useful. I don't know enough bash to write a character string containing every character and have used this ...

Code:

"0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"

... to represent such a string. See what can be done with it. This code ...

Code:

echo; echo "This is the character class [:digit:]"
printf "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"  \
|sed 's/./&\n/g'  \
|grep [[:digit:]] \
|paste -s -d"\0"

echo; echo "This is the character class [:upper:]"
printf "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"  \
|sed 's/./&\n/g'  \
|grep [[:upper:]] \
|paste -s -d"\0"

echo; echo "This is the character class [:lower:]"
printf "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"  \
|sed 's/./&\n/g'  \
|grep [[:lower:]] \
|paste -s -d"\0"

... produces this result ...

Code:

This is the character class [:digit:]
0123456789

This is the character class [:upper:]
ABCDEFGHIJKLMNOPQRSTUVWXYZ

This is the character class [:lower:]
abcdefghijklmnopqrstuvwxyz

These three similar code snippets may be combined ..

Code:

for j in digit upper lower
  do
    echo; echo "This is the character class [:$j:]"
    printf "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"  \
    |sed 's/./&\n/g'  \
    |grep [[:$j:]] \
    |paste -s -d"\0"
  done

Daniel B. Martin

chrism01 · 04-24-2013, 02:31 AM

@Daniel, have a read of this http://www.regular-expressions.info/posixbrackets.html and consider different locales as well ...

grail · 04-24-2013, 06:48 AM

So here is something a little more encompassing as far as which characters are being looked at. Obviously to have more than those in the ascii table you need to look at unicode.

Code:

#!/bin/bash

for class
do

	reg="[[:$class:]]"

	for i in {001..377}
	do
		[[ $i =~ [89] ]] && continue

		[[ $(echo -e "\0$i") =~ $reg ]] && echo -ne "\0$i "
	done

	echo
done

Called with the class(es) you wish to look at as parameters:

Code:

./class.sh digit alnum

ntubski · 04-26-2013, 10:09 PM

Code:

seq 1 $((0x10ffff)) | # all unicode code points
xargs printf '%08X' | # convert to hex string
xxd -r -p           | # convert to raw binary
iconv -f UCS4       | # convert to current text encoding
grep -o '[[:graph:]]' # print only the characters in the class