Replace the Euro-sign (€) using sed-

Bartonsen · 11-21-2018, 01:19 PM

"file.csv" is converted from excel, and contains a lot of the euro-sign €.
How can I remove this character from the file using bash script?

sed -i 's/\€//g' file.csv
works ok if run from commandline, but running from script, it's not working. Why?

Tried the following:
#!/usr/bin/sh and #!/bin/bash

sed -i "s/\&euro//g" file.csv
sed -i "s/\x80//g" file.csv
sed -i 's/\<80>//g' file.csv
sed -i 's/\€//g' file.csv

RandomTroll · 11-21-2018, 01:29 PM

Have you tried

Code:

 tr -d '€'

ondoho · 11-21-2018, 01:44 PM

Quote:

Originally Posted by Bartonsen

sed -i 's/\€//g' file.csv
works ok if run from commandline, but running from script, it's not working. Why?

i see no reason why it shouldn't work, other than that your bash script file isn't utf-8 encoded?
are you using the exact same command?
so you alwys make a copy of the original file.csv before attempting it?
what does the euro sign look like inside the .csv file?

Bartonsen · 11-21-2018, 02:01 PM

tr -d '€' works from commandline, but does nothing when running from script.

utf-8 could be an issue, but don't know how to check that.

For the script, I enter the € sign, but when doing vi on the file, € looks like <80>

scasey · 11-21-2018, 02:03 PM

Quote:

Originally Posted by ondoho

what does the euro sign look like inside the .csv file?

Yes. cat or more the csv file, copy the character and paste it into the sed command.
or
Remove it in Excel before converting to csv.

BW-userx · 11-21-2018, 02:03 PM

maybe use the hex number thingy.

Bartonsen · 11-21-2018, 02:24 PM

Hmmm... doing cat or vi on the csv file, the symbol looks like €.

Opening the csv file in excel, the euro-symbol is displayed as â‚¬

I should add that this is a csv file converted from excel using: libreoffice --headless --convert-to csv file.xlsx --infilter=CSV:59,34,UTF8

Mayber the infilter is wrong?
I chosed 59 because I think that means semicolon as separator, but I don't really know what I'm doing.
It works as expected on other excel files though, but ran into problems now when the excel sheet contains €...

Bartonsen · 11-21-2018, 03:09 PM

I'm probably doing something wrong in the infilter for the libreoffice command, but I found this to work ok for my script:

Code:

tr -cd '\11\12\15\40-\176' < file-with-binary-chars > clean-file

scasey · 11-21-2018, 03:11 PM

If more displays it as you expect, then

Code:

sed -i 's/€//g' file.csv

should work.
Note that you shouldn't need to escape the €

As stated, you are backing up file.csv before each attempt so it can be restored to try again...or, for testing don't do the inplace: When

Code:

sed 's/€//g' file.csv | grep €

returns nothing, you'll have it.

mina86 · 11-21-2018, 06:07 PM

‘tr -d €’ is not a correct solution. Observe how it corrupts characters other then euro sign:

Code:

$ echo '<żółw €>' | tr -d €
<żóÅw >

This is because tr does not understand Unicode and operates on bytes and € are two bytes.

To delete all non-ASCII characters, you need ‘tr -d '\200-\377' as in:

Code:

$ echo '<żółw €>' | tr -d '\200-\377'
<w >

To delete just the euro sign use sed as has been mentioned by others. If your script doesn’t work, ‘hexdump -C’ can help you find out differences in encoding that may cause that. You can also explicitly encode the character:

Code:

$ echo '<żółw €>' | sed $'s/\342\202\254/EURO/'
<żółw EURO>

though (notice this uses $'…' which is bash extension and) I don’t think you should be needing to do that since just using € should work if you save your files in the same encoding.

Lastly, you can play around with iconv to do various things to your input file, e.g.:

Code:

$ echo '<żółw €>' | iconv -f UTF-8 -t ASCII//TRANSLIT
<zolw EUR>
$ echo '<żółw €>' | iconv -f UTF-8 -t ASCII//IGNORE
<w >

NevemTeve · 11-21-2018, 10:13 PM

Guess your datafile is in windows-125x encoding, while your script is in UTF-8.

tr(1) works on bytes (encoding-independent) so this should work:

Code:

tr -d $'\x80' file.csv >file1.csv

Or, save your script as ISO-8859-1, and add this line at the beginning:

Code:

export LC_ALL=en_US.ISO-8859-1

After it sed might work:

Code:

sed -i 's/\x80//g' file.csv

l0f4r0 · 11-22-2018, 03:26 AM

Maybe

Code:

file -i file.csv
file -i <your_script>

could help to know first which encoding is being used...

tofino_surfer · 12-11-2018, 12:18 PM

Quote:

Tried the following:
#!/usr/bin/sh and #!/bin/bash

On many Linux systems such as RH/Fedora these are the same. sh just links to bash. The old /usr/bin/sh doesn't exist anymore.

Code:

$ type sh
sh is /usr/bin/sh
$ ll /usr/bin/sh
lrwxrwxrwx 1 root root 4 May 18  2018 /usr/bin/sh -> bash

Try this on your system.

RandomTroll · 12-11-2018, 12:28 PM

If you're trying to convert from UTF to ISO-8859 try utf8trans.

tofino_surfer · 12-11-2018, 02:02 PM

The Unicode for euro symbol is 20AC.

http://eurosymbol.eu/ascii-code

Code:

$ echo -e '\u20AC'
€
$ echo -ne '\u20AC' | hexdump -C
00000000  e2 82 ac                                          |...|
00000003
$ echo -e "\xe2\x82\xac"
€
$ echo -e '\xe2\x82\xac'
€

Try the following. I don't know if single or double quotes are needed or if it matters at all which you use. I changed the delimiter to # for readability.

Code:

sed -i 's#\xe2\x82\xac##g' file.csv
sed -i "s#\xe2\x82\xac##g" file.csv