LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 03-18-2006, 11:40 AM   #1
Schreiberling
Member
 
Registered: Jan 2006
Location: Bremen, Germany
Distribution: Linux Mint 16 KDE
Posts: 40

Rep: Reputation: 15
Converting lots of text files to Unicode


That's not the first time I have expressed all relevant information about my question in the title. Therefore, I'm finishing here in awaitness for your answers.
 
Old 03-18-2006, 11:51 AM   #2
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
Check out iconv

See reply title.
 
Old 03-19-2006, 07:45 AM   #3
Schreiberling
Member
 
Registered: Jan 2006
Location: Bremen, Germany
Distribution: Linux Mint 16 KDE
Posts: 40

Original Poster
Rep: Reputation: 15
Gui

I'm sorry for overhastily posing that question, mostly my questions are such specific that I can't find a suitable solution. I have reviewed the similar threads, unfortunately there was no GUI program mentioned. Is there a program that is capable of converting for example ISO-8859-1 to UTF-8? Just like as in OpenOffice with your .doc files.
Thanx.
 
Old 03-19-2006, 08:55 AM   #4
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
unfortunately there was no GUI program mentioned
You didn't mention the necessity of a GUI.


Is there a program that is capable of converting for example ISO-8859-1 to UTF-8?
iconv!


Oh, well. Here's an example of a lame GUI for iconv.
It depends on default shell, iconv and a package called "zenity" which supercedes Xdialog.
Save as "iconv_gui" somewhere in your path (say /usr/local/bin), chown as necessary, chmod 0750, and run.
Files (copy) are saved to dir "TMP/iconv_out".
YMMV(VM).
Code:
#!/bin/sh
# iconv_gui for www.linuxquestions.org 20060319
if [ -z "${TMP}" -o  ! -d "${TMP}" -o ! -w "${TMP}" ]; then exit 127; fi
if [ -d "${TMP}" -a ! -d "${TMP}/iconv_out" ]; then mkdir "${TMP}/iconv_out"; fi
convList=($(iconv --list | tr -d ",\/\/"|egrep -v "(^(The.f|not.n|the.F|listed.w)|[[:blank:]])"))
if [ -x "${#convList[@]}" ]; then exit 1; fi
convFrom=$(zenity --width=400 --title="Iconv-GUI: select \"From\" charset:" --list --column=" ${convList[@]}" --separator=" " 2>/dev/null)
if [ -x "${convFrom}" ]; then exit 1; fi
convTo=$(zenity --width=400 --title="Iconv-GUI: select \"to\" charset:" --list --column=" ${convList[@]}" --separator=" " 2>/dev/null)
if [ -x "${convTo}" ]; then exit 1; fi
zenity --width=600 --title="Iconv-GUI: select files:" --file-selection --multiple --separator="\n" | while read l; do
if [ -f "${l}" ]; then convOut="${TMP}/iconv_out/$(basename "${l}")"; if [ ! -f "${convOut}" ]; then 
iconv -f ${convFrom} -t ${convTo} "${l}" > "${convOut}"; fi; fi; done; exit 0
 
Old 07-13-2007, 03:01 PM   #5
mauran
LQ Newbie
 
Registered: Dec 2005
Location: Sri Lanka
Distribution: unbuntu 7.04
Posts: 17

Rep: Reputation: 0
see this thread

http://www.linuxquestions.org/questi...d.php?t=568708
 
Old 06-08-2013, 07:48 AM   #6
dreamtraveler
LQ Newbie
 
Registered: Jun 2013
Posts: 4

Rep: Reputation: Disabled
Quote:
Originally Posted by unSpawn View Post
unfortunately there was no GUI program mentioned
You didn't mention the necessity of a GUI.


Is there a program that is capable of converting for example ISO-8859-1 to UTF-8?
iconv!


Oh, well. Here's an example of a lame GUI for iconv.
It depends on default shell, iconv and a package called "zenity" which supercedes Xdialog.
Save as "" somewhere in your path (say /usr/local/bin), chown as necessary, chmod 0750, and run.
Files (copy) are saved to dir "TMP/iconv_out".
YMMV(VM).
Code:
#!/bin/sh
# iconv_gui for  20060319
if [ -z "${TMP}" -o  ! -d "${TMP}" -o ! -w "${TMP}" ]; then exit 127; fi
if [ -d "${TMP}" -a ! -d "${TMP}/iconv_out" ]; then mkdir "${TMP}/iconv_out"; fi
convList=($(iconv --list | tr -d ",\/\/"|egrep -v "(^(The.f|not.n|the.F|listed.w)|[[:blank:]])"))
if [ -x "${#convList[@]}" ]; then exit 1; fi
convFrom=$(zenity --width=400 --title="Iconv-GUI: select \"From\" charset:" --list --column=" ${convList[@]}" --separator=" " 2>/dev/null)
if [ -x "${convFrom}" ]; then exit 1; fi
convTo=$(zenity --width=400 --title="Iconv-GUI: select \"to\" charset:" --list --column=" ${convList[@]}" --separator=" " 2>/dev/null)
if [ -x "${convTo}" ]; then exit 1; fi
zenity --width=600 --title="Iconv-GUI: select files:" --file-selection --multiple --separator="\n" | while read l; do
if [ -f "${l}" ]; then convOut="${TMP}/iconv_out/$(basename "${l}")"; if [ ! -f "${convOut}" ]; then 
iconv -f ${convFrom} -t ${convTo} "${l}" > "${convOut}"; fi; fi; done; exit 0
i know this is a bit old post but can you please be a little more specific on how to run this script, like does it needs to be placed in the same folder as the .srt files do i simply run it or run it via terminal finding the path to iconv_gui ?
 
Old 06-09-2013, 07:51 AM   #7
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Arch + Xfce
Posts: 6,852

Rep: Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037
The script itself doesn't need to be in the working directory, as long as the shell knows where to read it from. Just make it executable and either put it in a directory listed in your $PATH, or run it with an absolute pathname: "/path/to/myscript.sh".

The particular design of this script doesn't depend on it being run in any specific location either; the zenity file dialog will let you select files from anywhere on your system.

Finally, I've taken the liberty of re-writing UnSpawn's old code into a nice, fresh, bash script, complete with comments explaining what it's doing, along with a couple of additions of my own.

Code:
#!/bin/bash

# Read the list of iconv's supported encodings into an array.
# iconv nicely reconfigures its output when its not going to a terminal, so this is easy
# mapfile requires bash v.4+, otherwise use a while+read loop.
# the second line resets the array without the trailing slashes on each entry
mapfile -t convList < <( iconv --list )
convList=( "${convList[@]%%//}" )

[[ -z ${convList[@]} ]] && echo 'No iconv encodings detected. Exiting.' && exit 1

# Pop up a zenity dialog requesting the input encoding, and save the selected value.
convFrom=$( zenity --width=400 --title='Iconv-GUI: select the "From" charset:' --list --column="${convList[@]}" --separator=' ' 2>/dev/null )

[[ -z $convFrom ]] && echo 'No input encoding selected. Exiting.' && exit 1

# Do the same for the output encoding.
convTo=$( zenity --width=400 --title='Iconv-GUI: select the "to" charset:' --list --column="${convList[@]}" --separator=' ' 2>/dev/null )

[[ -z $convTo ]] && echo 'No output encoding selected. Exiting.' && exit 1

# Request the location of the output directory.
outDir=$( zenity --width=600 --title='Iconv-GUI: select output directory:' --file-selection --directory )

[[ ! -w $outDir ]] && echo "$outDir: Directory isn't writable.  Exiting." && exit 1

# Confirm the settings on the command line.
echo "All selected files will be converted from $convFrom to $convTo."
echo "The converted files will be placed in $outDir."

# Pop up a file selector dialog, and loop over the files selected,
# ( Filenames are delimited with newlines; zenity
# ( apparently doesn't support null separators )
echo 'Select files to process.'

while IFS='' read -r fname; do

    if [[ -r $fname ]]; then

        convOut="$outDir/${fname##*/}"

        if [[ -f $convOut ]]; then
            echo "Output file $convOut already exists.  Skipping."
            continue
        fi

        # Do some simple checking for non-text files.  This is far from perfect.
        if [[ $( file -b "$fname" ) != *text* ]]; then
 
            zenity --width=600 --title='Iconv-GUI: WARNING:' \
            --question --text "[WARNING] $file: May be a binary file.  Convert it anyway? [WARNING]"

            (( $? = 1 )) && echo "$fname: Skipping this file." && continue

        fi

        # if the above tests pass, we can now convert the file
        # with //TRANSLIT it will attempt to substitute unsupported characters
        # with similar ones that are supported.

        iconv -f "$convFrom" -t "$convTo//TRANSLIT" "$fname" > "$convOut"

        if (( $? > 0 )); then
            echo "$fname: [WARNING!] Conversion unsuccessful. [WARNING!]"
        else
            echo "$fname: Conversion successful."
        fi

    else

        echo "$fname: Not a readable file.  Skipping."

    fi

done < <( zenity --width=600 --title='Iconv-GUI: select files:' --file-selection --multiple --separator='\n' )

exit 0
It's untested though, so there may be errors.

IMO, though, this kind of half-gui-half-cli thing is really more trouble than its worth. It would probably be better to use the ncurses based dialog instead, if you had to have some kind of "graphical" interface.

Last edited by David the H.; 06-11-2013 at 03:41 PM. Reason: cleaned up some quoting problems
 
Old 06-11-2013, 07:30 AM   #8
dreamtraveler
LQ Newbie
 
Registered: Jun 2013
Posts: 4

Rep: Reputation: Disabled
i see thanks will give it a shot... i am going to use it to encode .srt for vlc
 
Old 06-11-2013, 08:56 AM   #9
dreamtraveler
LQ Newbie
 
Registered: Jun 2013
Posts: 4

Rep: Reputation: Disabled
the first error i got was http://dpaste.com/1228888/ and i replaced it with [[ "${#convList[@]}" -eq 0 ]]
then second error was http://dpaste.com/1228936/ asked at #bash and i got a few replies like http://dpaste.com/hold/1229015/
then i run the script through shellcheck.net and got errors at lines 25, 75 and 79 http://dpaste.com/hold/1229030/
though i did saw the gui xD

Last edited by dreamtraveler; 06-11-2013 at 08:58 AM.
 
Old 06-11-2013, 11:33 AM   #10
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Arch + Xfce
Posts: 6,852

Rep: Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037
The first error is just an extra unbalanced quote-mark that snuck in there. Just remove it and that part should start working. And since quoting errors tend to propagate throughout a script, that may be also be the cause of some of the others as well.

It would be easier for us if you'd just post your code and error messages here directly, instead of using an external pastebin site. That's why we have [code][/code] tags available.
 
Old 06-11-2013, 03:12 PM   #11
dreamtraveler
LQ Newbie
 
Registered: Jun 2013
Posts: 4

Rep: Reputation: Disabled
Quote:
Originally Posted by David the H. View Post
The first error is just an extra unbalanced quote-mark that snuck in there. Just remove it and that part should start working. And since quoting errors tend to propagate throughout a script, that may be also be the cause of some of the others as well.

It would be easier for us if you'd just post your code and error messages here directly, instead of using an external pastebin site. That's why we have [code][/code] tags available.
Yes you are right the first error was the extra-quote mark deleted it and it works fine the next error occurred when i click option 850 from gui with errors

Code:
/home/dreamtraveler/Desktop/Subs/1: line 75: unexpected EOF while looking for matching `"'
     /home/dreamtraveler/Desktop/Subs/1: line 78: syntax error: unexpected end of file
shellcheck.net mentions there is something off at line 25

Code:
[[ ! -w $outDir ]] && echo "$outDir: Directory isn't writable.  Exiting." && exit 1
                                                        ^––This apostrophe terminated the single quoted string!
 
Old 06-11-2013, 03:24 PM   #12
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Arch + Xfce
Posts: 6,852

Rep: Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037
Understand that the line number the script gives you in its error messages is the point at which the shell could no longer parse the command, which is not necessarily where the error actually lies. Quoting mistakes in particular are hard to track down because it will just keep on looking for the next one, and then the next pair, until it reaches the end of the script and the last unmatched mark (usually giving you an EOF error).

You have to backtrack from the error line upwards until you find the actual mistake.


On it's own, that last error message shouldn't have caused any problems. Since the single quote is inside double quotes, it's escaped.

This line above it, though, has another quoting error.
Code:
convTo=$( zenity --width=400 --title='Iconv-GUI: select the "to" charset:" --list --column=" ${convList[@]}" --separator=" " 2>/dev/null )
I used a single quote to start the title string, but a double one to close it.


I was certainly unusually sloppy when I wrote that script. It was the end of a long day. I'm going to do a quick run-through and update my post to correct whatever mistakes I find.
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Convert DOS text files to UNIX text files ta0kira Linux - Software 7 03-15-2011 11:42 AM
converting Windows text to Linux text joshknape Linux - Software 3 09-11-2005 12:52 PM
How do I enter Unicode text in X11? uman Linux - Software 0 12-07-2004 10:14 PM
Networking&Text files; A string or two, and LOTS of int's quentusrex Programming 9 10-27-2004 05:19 AM
grep on Unicode files overbored Linux - Software 2 07-18-2004 01:12 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 06:49 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration