Latest LQ Deal: Complete CCNA, CCNP & Red Hat Certification Training Bundle
Go Back > Forums > Non-*NIX Forums > Programming
User Name
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.


  Search this Thread
Old 07-19-2010, 02:58 AM   #1
LQ Newbie
Registered: Jul 2010
Posts: 1

Rep: Reputation: 0
Renaming files with asian characters


I have a kind of strange problem that I haven't been able to resolve for a couple of days now so I thought I'd ask to see if anyone else had come across this.

I have a bunch of files that I need to rename, ordinarily this is pretty easy task. The problem here is that the file names have Chinese / Japanese characters (sorry for my ignorance I can't tell the difference).

ie [$$$$$$$$].SOMETHING BLAH BLAH.ext

Where all the "$$$$" are insert Chinese characters.

The problem is that sed or perl doesn't seem to handle the Chinese characters correctly so using a regular expression like this 's/^[*.]//' which would normally work doesn't.

From what I have read so far I believe these characters are double encoded UTF-8 (not 100% sure) which could be the problem.

So far I've tried numerous different regex's as well as playing around with convmv to see if I could convert the filenames to just single encoded characters but I've had no luck.

Has anyone else come across this? I don't really want to rename 100+ files by hand.

Old 07-19-2010, 07:09 AM   #2
David the H.
Bash Guru
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian + kde 4 / 5
Posts: 6,837

Rep: Reputation: 1985Reputation: 1985Reputation: 1985Reputation: 1985Reputation: 1985Reputation: 1985Reputation: 1985Reputation: 1985Reputation: 1985Reputation: 1985Reputation: 1985
Yes, it can be quite difficult to work with foreign language filenames on the command line. For CJK, the encodings are usually 2-4 bytes long, and unless you have the appropriate IM and knowledge, almost impossible to work with directly.

First of all, sed does work with unicode. However the example pattern you gave above is all wrong. The "wildcard" is ".*" (not "*."), and they shouldn't go inside brackets unless you're trying to match literal characters. So...
...should strip all the characters in front of "SOMETHING".

Second, try using globbing to grab the file by the part of the name you can type.
mv "*SOMETHING BLAH BLAH.ext" "new_file_name.txt"
Now a more complicated method. The uniname command (available in the uniutils package) will show you the multi-byte encodings of the individual characters, which you can then use in the shell, with a bit of trickery.
$ ls *file1.txt

$ ls *file1.txt |uniname -bnpu
character  encoded as     glyph
        0   E6 97 A5       日
        1   E6 9C AC       本
        2   E8 AA 9E       語
        3   66             f
        4   69             i
        5   6C             l
        6   65             e
        7   31             1
        8   2E             .
        9   74             t
       10   78             x
       11   74             t
       12   0A

$ mv $'\xE6\x97\xA5\xE6\x9C\xAC\xE8\xAA\x9Efile1.txt' new_file_name.txt
$'' is a bash shell pattern which expands various escape sequences into their actual characters. Octal byte-codes like the ones above can be used in it in the pattern "\xNN".

This is cumbersome though, and only really good for scripting purposes. I'm only including it here for completeness.

One option I recommend is to use the qmv command from the renameutils package. qmv will load all the filenames given to it into a text editor, which you can then edit by hand. When you save the file, it will automatically rename all the files at once.

The following uses my personal set-up for qmv, which places the from and to names on sequential lines. You can set it up as an alias if you want. The editor used is determined by your $EDITOR environmental variable, or by the -e option.

$ ls *

$ qmv -v -f sc -o separate,indicator1='f|',indicator2='t|' *

#outputs to nano:




Unfortunately though, it doesn't seem to want to send them to the editor as pure unicode strings, so all you get are bytecodes. Notice that in this case the bytes are displayed in decimal form instead of octal, but the basic concept is the same. Just rename the "t" lines (but don't touch the "f" lines) and save and close the editor.

Finally, there are other bulk renamers out there (mostly gui), such as krename and pyrenamer.


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Similar Threads
Thread Thread Starter Forum Replies Last Post
renaming files with spaces and special characters. bowens44 Linux - Newbie 8 06-29-2009 07:52 PM
cannot copy files with asian character file name. ufmale Linux - Newbie 2 05-31-2008 06:03 AM
Asian files hosted on windows machine not displaing TRaven Ubuntu 7 10-26-2007 03:56 PM
Files using Asian characters can't be read Mega Man X Ubuntu 0 04-29-2007 05:20 PM
How to modify the names of files and replace characters with other characters or symb peter88 Linux - General 2 12-10-2006 04:05 AM > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 06:21 PM.

Main Menu
Write for LQ is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration