[SOLVED] Remove all lines containing extended characters

hattori.hanzo · 11-07-2010, 04:59 PM

I am using 'sed -e /foo/d' to match lines which I want to delete from a file.

I discovered I have some lines which contain random (extended?) characters like 'è¨á»§äµäµ' which I would also like to delete.

The lines in the file should only contain alpha numeric characters.

Thanks & Regards

GrapefruiTgirl · 11-07-2010, 05:19 PM

Perhaps:

Code:

sed '/[^a-z0-9A-Z ]/d'
# or:
sed '/[^[:alnum:] ]/d'

?

Note I left the "space" character there too - remove it if you don't want spaces either.

hattori.hanzo · 11-07-2010, 08:15 PM

Thanks for the response. Probably I didnt make it clear in the first post. Here is a sample:

Code:

2010-11-07 13:03:56,2347985439,27437985441,SOA,com,domain,_sites,j,_tcp,
2010-11-07 13:03:56,2347985439,27437985441,A,com,domain,host4,,,
2010-11-07 13:03:57,2787984329,27437985441,SOA,com,domain,_sites,è¨á»§äµäµ                       _tcp,
2010-11-07 13:03:57,2787444439,27437985441,A,com,domain,host2,,,                     ï¼host,
2010-11-07 13:03:57,2780005439,27437985441,A,com,host9,ab2,,,,com,ï¬´à¡«èá»§âºïª·Å
2010-11-07 13:04:01,2787843905,27437985441,A,com,host,us,host6,,

Thanks & Regards,

GrapefruiTgirl · 11-07-2010, 08:35 PM

OK, so the only difference is that you may have punctuation in lines that you wish to keep?

Code:

sed '/[^[:alnum:][:punct:] ]/d'

This should be one way of working around it - so if a line contains stuff other than alphanumeric, spaces, or punctuation, it gets removed.

Does this work?

hattori.hanzo · 11-09-2010, 12:38 AM

Hi, tested it but unfortunately not...

Code:

[root@host1 tmp]$ cat test.txt
2010-11-07 13:03:56,2347985439,27437985441,SOA,com,domain,_sites,j,_tcp,
2010-11-07 13:03:56,2347985439,27437985441,A,com,domain,host4,,,
2010-11-07 13:03:57,2787984329,27437985441,SOA,com,domain,_sites,Ã¨Â»Â§Ã¤Âµ
_tcp,
2010-11-07 13:03:57,2787444439,27437985441,A,com,domain,host2,,,Ã¯ost,
2010-11-07 13:03:57,2780005439,27437985441,A,com,host9,ab2,,,,com,ï¬´à¡«Ã¨Â§Ã¢ÂªÂ·Ã
2010-11-07
13:04:01,2
[root@host1 tmp]$ sed '/[^[:alnum:][:punct:] ]/d' test.txt
2010-11-07 13:03:56,2347985439,27437985441,SOA,com,domain,_sites,j,_tcp,
2010-11-07 13:03:56,2347985439,27437985441,A,com,domain,host4,,,
2010-11-07 13:03:57,2787984329,27437985441,SOA,com,domain,_sites,Ã¨Â»Â§Ã¤Âµ
_tcp,
2010-11-07 13:03:57,2787444439,27437985441,A,com,domain,host2,,,Ã¯ost,
2010-11-07 13:03:57,2780005439,27437985441,A,com,host9,ab2,,,,com,ï¬´à¡«Ã¨Â§Ã¢ÂªÂ·Ã
2010-11-07
13:04:01,2

Thanks & Regards

crts · 11-09-2010, 05:00 AM

Hi,

unfortunately I do not see an easy way to accomplish this task with sed.
The Problem is that sed does not "see" the weird characters. Instead it sees their octal representation. The problem is, I do not know if it is possible to define "octal" character ranges in sed. Of what I've tried so far nothing has worked. So you will have to check for the octal representation of the "funny" characters and remove them like:

Code:

sed -r '/\o302/ d' file

If one line has "funny-chars" whose octal value is not 302, then those lines probably won't get removed. You will have to remove them like

Code:

sed -r '/\o302|\o203|\o303/ d' file

Maybe you should look for an alternative. One that lets you you specify octal ranges.

colucix · 11-09-2010, 05:41 AM

To me the problem is that in POSIX standard the character classes [:alnum:], [:punct:] and so on, include these special symbols and the reason is that they are valid characters in some languages. To restrict their meaning to the ASCII table, you can try to set the current locale to C, just for the execution of the sed command, e.g.

Code:

LC_ALL=C sed 's/[^[:alnum:][:punct:][:space:]]//g' file

syg00 · 11-09-2010, 06:05 AM

Once again a fine explanation by colucix - one assume the OP wants to delete the lines as per the previous responses rather than just the characters.

hattori.hanzo · 11-09-2010, 06:24 AM

Thanks colucix for the suggestion and explanation. That worked perfectly.