Linux - GeneralThis Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
Due to an application neglecting Unicode (it was abcde) I now have a bunch of music files with wrong character encoding. Depending on the terminal I either see a normal '?' or a black '?' on a white background (in a black terminal) in the place of previously not recognized non-ASCII characters. (The shells themselves know Unicode, I can type in what I want.) I wanted to do
Code:
$ ls -laR audio/|grep ?
in order to see the wrong filenames and then correct them manually. However, grep doesn't find anything as it's not really a '?', somehow the shell seems to decide to show a '?' as the character is not recognized. How can I find the erroneous filenames now?
Last edited by elektronaut; 11-30-2006 at 12:12 AM.
You could always just grep for the offending character (or set of characters)?
How about:
ls -laR audio/ > list.txt
... and see if the highlighted question-marks occur there?
Can you provide an example of a filename which gives this weird result?
Have you tried changing the font used in the terminal?
Do I understand you correctly - you can type non-ascii characters in the terminal and they display correctly? How about if you name a file using the offending characters (in cli) then use ls to display them?
Last edited by Simon Bridge; 11-30-2006 at 12:45 AM.
You could just find files with a nonprintable character in their name.
Code:
find -name "*[^ -~]*" | ...
The printable characters ( man ascii) are space .. tilde, i.e., 32 .. 126. Caret complements the set, [^ -~] matches one non-printable character.
Quote:
Originally Posted by elektronaut
grep doesn't find anything as it's not really a '?', somehow the shell seems to decide to show a '?' as the character is not recognized.
I think it's ls, not the shell, that makes that decision.
Quote:
Originally Posted by man ls
-b, --escape
print octal escapes for nongraphic characters
-q, --hide-control-chars
print ? instead of non graphic characters
--show-control-chars
show non graphic characters as-is (default unless program is `ls' and output is a terminal)
So when you just run ls, output is a terminal, and it prints "?". When you run ls | grep, ls outputs the nonprintable characters as-is. For illustration, create a file with a name consisting of 3 characters, f, C-o, o:
Code:
$ touch f^Ootype: f C-v C-o o
$ ls
f?o
$ ls -qdefault on a terminal
f?o
$ ls -b
f\017o
$ ls -b | grep \\\\alternative solution to find
f\017o
$ ls --show-control-chars
fo C-o is invisible on terminal
$ ls --show-control-chars | cat -v
f^Oo
$ find -name "*[^ -~]*"
./f?o
$ find -name "*[^ -~]*" | cat -v
./f^Oo
Apparently find also prints differently to a terminal -- hence my hint "| ..." above. Writing to a file, as Simon Bridge suggested, is a good idea; then you can take it step by step.
It depends on the filesystem where the files are, on your locale, on your terminal and on the font that your using.
You can probably convert all the names to utf8 using convmv
like if they are in iso-8859-1, and you want them in utf8
First of all, I want to thank you all for your instantaneous help! It's great to have people like you around here !!!
Quote:
Originally Posted by Simon Bridge
You could always just grep for the offending character (or set of characters)?
How about:
ls -laR audio/ > list.txt
... and see if the highlighted question-marks occur there?
Can you provide an example of a filename which gives this weird result?
There are no question marks in the list. Here are some examples, viewed with different viewers/editors:
Code:
gedit (what you see here is different from what I see in gedit.
There are more '' in the first two names. The rest is the same. Strange.):
Donna Regina - Let Slow.mp3
Farben - Don .mp3
Georges_Brassens-La_mauvaise_réputation_(CD_2).flac.m3u
George_Brassens-George_Brassens_et_sa_guitare-02-AuprÚs_de_mon_arbre.flac
Sportfreunde_Stiller-Burli-02-Lauth_Anhören.flac
less:
Donna Regina - Let<U+4C249A14> Slow.mp3
Farben - Don<U+29323120><U+53202D20><U+754D206F><U+4C206863><U+A65766F><U+2C4BB400><U+433BB204>.mp3
Georges_Brassens-La_mauvaise_rputation_(CD_2).ogg.m3u
George_Brassens-George_Brassens_et_sa_guitare-02-Auprs_de_mon_arbre.flac
Sportfreunde_Stiller-Burli-02-Lauth_Anhren.flac
vim:
Donna Regina - Let<8c><89><89><94> Slow.mp3
Farben - Don<8c><84> <93><88><82> <93><92><81><8c><88><86><8a><99><97><99><92><90><80><83><8e><88><84>.mp3
Georges_Brassens-La_mauvaise_réputation_(CD_2).flac.m3u
George_Brassens-George_Brassens_et_sa_guitare-02-Auprès_de_mon_arbre.flac
Sportfreunde_Stiller-Burli-02-Lauth_Anhören.flac
As you can see, I already corrected some names manually (the last three), but only less shows them correctly. I get the feeling there is still some work which needs to be done regarding Unicode support...
Quote:
Originally Posted by Simon Bridge
Have you tried changing the font used in the terminal?
I got the same results with less in the bash and in a (real) terminal, so I don't think it has anything to do with the font used in the terminal.
Quote:
Originally Posted by Simon Bridge
Do I understand you correctly - you can type non-ascii characters in the terminal and they display correctly? How about if you name a file using the offending characters (in cli) then use ls to display them?
Code:
$ touch
$ ls | grep
$ touch Let Slow.mp3
$ ls|grep Let
Let
Really looks like this
$ touch Let<U+4C249A14> Slow.mp3
bash: U+4C249A14: No such file or directory
Quote:
Originally Posted by Quigi
You could just find files with a nonprintable character in their name.
Code:
find -name "*[^ -~]*" | ...
The printable characters ( man ascii) are space .. tilde, i.e., 32 .. 126. Caret complements the set, [^ -~] matches one non-printable character.
In this list I can find file names like the ones I already mentioned, but also plain ASCII filenames, like this one: './mp3/MotorFM/motorfmpodcast16.mp3'.[/code]
Quote:
Originally Posted by Quigi
So when you just run ls, output is a terminal, and it prints "?". When you run ls | grep, ls outputs the nonprintable characters as-is. For illustration, create a file with a name consisting of 3 characters, f, C-o, o:
Code:
see above
Wow, this taught me quite something!
Quote:
Originally Posted by nx5000
It depends on the filesystem where the files are, on your locale, on your terminal and on the font that your using.
You can probably convert all the names to utf8 using convmv
like if they are in iso-8859-1, and you want them in utf8
Code:
convmv -f iso-8859-1 -t utf8 -r /mp3
I'm not sure, but I don't think this is the reason if I look at the result of 'touch ' mentioned above.
Quote:
Originally Posted by archtoad6
Would:
Code:
$ ls -laR audio/|grep \?
work better?
No, '?' is not regarded as a metacharacter in this context:
Code:
$ touch exa?mple
$ ls|grep ?
exa?mple
------------------------------------
So, in conclusion, this seems to be it:
Code:
$ ls -laRq|grep ? > incorrect-filenames.txt
The results in the file seem to be o.k. I'm still a bit puzzled about the different ways viewers/editors display filenames if -q is not used, respectively the way of displaying filenames I already corrected manually - only 'less' did it correctly. If someone has some helpful links to enlighten me about the background of this whole story - Unicode & terminals, viewers, editors, etc. - go ahead, I'd be grateful if you post them here.
Also, does anybody know if there is a possibility of using 'abcde' to rip audio cd's to disk and preserving the original accented characters? I'm not sure which program that is used by 'abcde' (as it's a script using other programs) is responsible for this faulty behaviour.
You were right, it wasn't a real question mark. Apparently it doesn't matter if you use '?' or '\?' with grep in this case:
Code:
$ ls -laRq ~/audio/|grep ? > one
$ ls -laRq ~/audio/|grep \? > two
$ diff one two
$
(The empty prompt shows that there are no differences between those two files.)
No, it is a "real" question mark. With -q you make ls output one (instead of a "non-printable" character) even when the output is not to a terminal.
Your second example is simple: When you type [b]grep \?[b] the shell passes a "?" to grep.
Without the backslash, the shell expands the question mark into all single-character file names in the current directory. If there is no such file name, it depends on the shell. E.g., bash will silently pass the "?" literally to grep (just like in your second example), hence the same result. But csh/tcsh will report the error "grep: No match."
The question mark is not only a special character for the shell, but also for grep. It means zero or one occurrence of the preceding item (e.g., character). As there is nothing preceding, grep shows lenience and assumes that you meant "\?", i.e., literal question mark.
The "correct" way to enter the command would be [b]grep \\\?[b] -- the shell will interpret \\ as one literal backslash and \? as one literal backslash, so grep gets \?, and that backslash will again escape the question mark. By providing less question marks, you're relying on lenience of bash and grep.
To get a feel of what's going on, try "echo ?" etc. to learn what the shell thinks you mean. Then "touch x" and try again. Also run grep (without "ls |) and type some lines containing a question mark (they get repeated back to you) and some without.
I'm not sure, but I don't think this is the reason if I look at the result of 'touch ' mentioned above.
You are right, I was just saying this as a general remark. If you want a correct utf8 support, you need these conditions I mentionned; this seems(*) to be the case for you, its only that you have used an application that was creating files with a wrong encoding in their names.
(*) can you verify your setting by issuing this in a shell
Code:
printf '\xc3\xa9\n'
if your environment is ok (utf8), you will get the letter
As you are on a debian like, I would advice you some nice readings.. prepare several hours and some coffee If you don't have them, you have to download them.
man charsets
man unicode
/usr/share/doc/HOWTO/en-html/Unicode-HOWTO.html http://www.ietf.org/rfc/rfc2279.txt
man convmv
man iconv
I'm pretty sure the tool you need is convmv, its done for your problem, no need to do it manually! The options to give to it may be different depending on the input(8859) and output encoding you want.
On a unicode environment, if you try to display a non-ascii caracter, you will get the special interrogation mark:
convmv is really great. It recursively changed all the filenames correctly. Now I'm trying to find a way to get iconv to convert all the wrong playlists in the same fashion. I wrote a little shell script (good exercise for a like me) that hands over all playlists in a directory to iconv, but unfortunately iconv is not as smart as convmv. It rigourously converts all files, regardless of their encoding, which leads to funny results if it converts unicode to unicode, thinking that it would be iso8859-1 in the beginning. I guess I have to add a line in the script which peeks into the files and checks for wrong encoding. I'll post this later if I succeed, so other travellers will gain from this.
The shell gave me the same echos you described, but I didn't find the time yet to learn about all that voodoo behind the stage.
I'm also trying to find same bad mp3 filenames resulting of changing the locale from en_US to en_US.utf8.
I find that char � in the bad filenames. The solution I find was to make
Code:
ls -lR1 > list.txt
and open the list.txt with kwrite. The kwrite show a message about the encoding and with the find function I could find the � bad filenames.
There is any way to make a list of all files with bad filename result of change of enconding?
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
man ascii) are space .. tilde, i.e., 32 .. 126. Caret complements the set, [^ -~] matches one non-printable character.
If you don't have them, you have to download them.
like me) that hands over all playlists in a directory to iconv, but unfortunately iconv is not as smart as convmv. It rigourously converts all files, regardless of their encoding, which leads to funny results if it converts unicode to unicode, thinking that it would be iso8859-1 in the beginning. I guess I have to add a line in the script which peeks into the files and checks for wrong encoding. I'll post this later if I succeed, so other travellers will gain from this.
