[SOLVED] filtering out displayable characters in bash

blumenwesen · 03-26-2023, 05:59 AM

How can I omit all unreadable or incorrectly displayed characters?

GNU Awk 5.1.0, API: 3.0 (GNU MPFR 4.1.0, GNU MP 6.2.1)

Code:

$ a="֎"
$ printf %x "\""$a
58e(base) # correct

$ echo "֎" | nawk '{print | eval "printf \"%x\" \"\\\""$0"\""}'
d6(base) # wrong

$ echo -e "֎" | awk '{ cmd="printf %x \\\"" $0; cmd | getline hex; close(cmd); print hex }'
d6
(base) # wrong

$ echo -e "֎" | awk '{ system("/usr/bin/printf \"%x\" \\\""$0"") }'
/usr/bin/printf: warning: �: character(s) following character constant have been ignored
d6(base) # wrong


# complete script
$ echo -e "\n\n\n"  | awk '{ split($0, z, ""); for(y=0; y<length(z); y++){ if(system("/usr/bin/fc-list :charset=$(/usr/bin/printf \"%x\" \\\"\""z[y]"\")")){ print z[y] } } }'
/usr/bin/printf: '"': expected a numeric value
/usr/share/fonts/truetype/lyx/wasy10.ttf: wasy10:style=LyX
/usr/share/fonts/truetype/lyx/stmary10.ttf: stmary10:style=LyX
/usr/bin/printf: warning: ��: character(s) following character constant have been ignored
/usr/share/fonts/truetype/lato/Lato-Medium.ttf: Lato,Lato Medium:style=Medium,Regular
# ...  wrong

# How can it be the same result like that?
$ for z in $(echo -e "\n\n\n"); do
$ [[ $(fc-list :charset=$(printf %x "\"$z")) ]] && echo "$z"
$ done

 # correct

Thanks for the help.

MadeInGermany · 03-26-2023, 08:13 AM

Perhaps you can use the [:print:] character class?

How to delete characters that are not in [:print:]

Bash:

Code:

string='text with bad characters'
newstring=${string//[^[:print:]]/}

stdin to stdout with tr:

Code:

tr -dc '[:print:]'

awk stdin:

Code:

{ gsub(/[^[:print:]]/,"") }

You might need to set LC_ALL=C (LC_CTYPE at least) to enforce an ASCII character set.

NevemTeve · 03-26-2023, 08:32 AM

Perhaps this:

Code:

echo '֎Szűrő' | iconv -f UTF-8 -t ASCII//IGNORE 2>/dev/null
Szr

Or perhaps:

Code:

echo '֎Szűrő' | iconv -f UTF-8 -t ISO-8859-2//IGNORE 2>/dev/null | iconv -f ISO-8859-2 -t UTF-8
Szűrő

blumenwesen · 03-26-2023, 09:38 AM

The first one is displayed normally as a man in a circle, the second one as a heart with sinus rhythm as well, so it should also appear in the issue.
The third is a bordered box with F21F, the fourth is a bordered box with F220, both should not be displayed.

Code:

$ a=""; [[ $(fc-list :charset=$(printf %x "\"$a")) ]] && echo "$a"
# result: 

$ a=""; [[ $(fc-list :charset=$(printf %x "\"$a")) ]] && echo "$a"
# result: 

$ a=""; [[ $(fc-list :charset=$(printf %x "\"$a")) ]] && echo "$a"
# result: nothing

$ a=""; [[ $(fc-list :charset=$(printf %x "\"$a")) ]] && echo "$a"
# result: nothing

string=''
echo ${string//[^[:print:]]/} # displays all

echo -e "" | iconv -f UTF-8 -t ISO-8859-2//IGNORE 2>/dev/null | iconv -f ISO-8859-2 -t UTF-8 # displays nothing

NevemTeve · 03-26-2023, 12:18 PM

What do you actually wish to accomplish? Check a character's presence in a font-file?

blumenwesen · 03-26-2023, 01:30 PM

Check a file for characters that cannot be represented symbolically, i.e. filter out the boxes surrounded by hexadecimal numbers.
I wanted to use awk because the lines for the start and end are predefined, and further adjustments are needed.

NevemTeve · 03-26-2023, 11:24 PM

That depends on the used font-file, I guess.

pan64 · 03-27-2023, 01:13 AM

yes, probably the tool (fc-list) uses a different locale, not "compatible" with the actual font. You cannot handle it [easily] in awk.

blumenwesen · 03-27-2023, 07:50 AM

Ok, then I have to try without, thanks anyway. =)

MadeInGermany · 03-27-2023, 08:39 AM

The

Code:

$ printf "%x\n" \"$a
58e

seems to be special to the bash-builtin printf.
The external printf command fails:

Code:

$ /usr/bin/printf "%x\n" \"$a
/usr/bin/printf: warning: Ž: character(s) following character constant have been ignored
d6

And another bash-builtin fails:

Code:

$ echo \"$a
"֎

Avoid system() in awk!
In bash you can try an explicit while loop

Code:

echo -e "\n\n\n" |
while IFS= read -r z
do
  [[ $(fc-list :charset=$(printf "%x" "\"$z")) ]] &&
  echo "$z"
done

The pipe forces the loop into a sub shell. If you want it in the main shell (and give variables back) then use a process substitution

Code:

while IFS= read -r z
do
  [[ $(fc-list :charset=$(printf "%x" "\"$z")) ]] &&
  echo "$z"
done < <( echo -e "\n\n\n" )

Read reads a (newline-separated) line. Missing a split function you might need another loop to iterate over the individual characters.

boughtonp · 03-27-2023, 08:50 AM

Quote:

Originally Posted by blumenwesen

Check a file for characters that cannot be represented symbolically, i.e. filter out the boxes surrounded by hexadecimal numbers.

Awk does not know how your font will choose to render the characters (i.e. bytes / hex numbers) that it will receive.

The characters you've used are F21D, F21E, F21F, F220 - all four of those are within the E000–F8FF Unicode Private Use Area which seems to mean there is no standardized symbol for any of them, and so you need to determine what defines the first two as valid characters in your context.

After that, well, it depends if/how well Awk deals with multi-byte characters - since Gawk only has single-byte escape sequences, you may be better to use (e.g.) Perl or Python that have explicit support.

blumenwesen · 03-27-2023, 02:08 PM

It seems that it can only work with awk if there is an alternative for the hex conversion of printf inside, but I don't know any.

Now I'm going to do it like this, I just thought awk might be faster.

Code:

FILE=$(echo -e "\n\n\n\n\n\n\n" | tail -n +3 | head -n -2)
for z in $FILE; do
[[ $(fc-list :charset=$(printf %x "\"$z")) ]] && echo "$z"
done
# result:

