Download your favorite Linux distribution at LQ ISO.
Go Back > Forums > Linux Forums > Linux - Newbie
User Name
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!


  Search this Thread
Old 05-07-2010, 02:22 PM   #1
LQ Newbie
Registered: Aug 2008
Location: Mauritius
Distribution: Mandriva 2008 Powerpack
Posts: 12

Rep: Reputation: 0
Text manipulation using sed

I have a number of text files (26 per database x 100+ databases) which need 'correcting' in order to import into postgresql.

I think that I have identified all the problem characters and I need to automate the process as much as possible.

I have a script to convert the characters and I do them one by one (not effecient but easier to understand).

What I cannot understand is why some conversions work in the script :

eg : sed 's/|/","/g' < > file.out
sed 's/$/");/' < > file2.out

but others will not :

eg sed 's/"/-/g

If I vi the individual data file, I can use the following :

eg :%s/"/-/g or
:%s'/'-'g and these work.

My problem characters are
1. ' to - (:%s/'/-/g)
2. " to - (:%s/"/-/g)
3. / to - (:%s'/'-'g)
4. \ to (:%s'\' 'g)
5. \\ to -(:%s'/\\'-'g)
6. ` to - (:%s/`/-/g)
7. & to a (:%s/&/a/g)
8. @ to a (:%s/@/a/g)
Old 05-07-2010, 02:36 PM   #2
LQ Guru
Registered: Sep 2003
Location: Bologna
Distribution: CentOS 6.5 OpenSuSE 12.3
Posts: 10,509

Rep: Reputation: 1976Reputation: 1976Reputation: 1976Reputation: 1976Reputation: 1976Reputation: 1976Reputation: 1976Reputation: 1976Reputation: 1976Reputation: 1976Reputation: 1976
Maybe it depends from the order by which it makes substitutions. For example you have to change \\ before \ otherwise the former will become -\. Regarding the syntax I checked and the following works for me:
sed "s/'/-/"
sed 's/"/-/'
sed 's/\//-/'
sed 's/\\\\/-/'
sed 's/\\/-/'
sed 's/`/-/'
sed 's/&/a/'
sed 's/@/a/'
Eventually, can you post some lines of an input file where you encounter problems?
Old 05-08-2010, 01:17 AM   #3
LQ Guru
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,421

Rep: Reputation: 2815Reputation: 2815Reputation: 2815Reputation: 2815Reputation: 2815Reputation: 2815Reputation: 2815Reputation: 2815Reputation: 2815Reputation: 2815Reputation: 2815
How about placing each group into a variable and then character class:

sed -e "s@[$change_dash]@-@g" -e 's@\\\\@-@g'
sed "s/[$change_empty]//g" $1
sed "s/[$change_a]/a/g" $1


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Similar Threads
Thread Thread Starter Forum Replies Last Post
Challenging text manipulation using using sed or alternative phugoid Linux - Software 1 03-24-2009 02:41 AM
CSV | Text manipulation lmedland Programming 26 07-31-2008 11:44 AM
Easy string/text manipulation/indentation for restructured text brianmcgee Linux - Software 1 04-22-2008 09:27 PM
need help with text manipulation pcorajr Programming 12 12-15-2006 08:33 AM
More text manipulation ice_hockey Linux - General 2 05-28-2005 02:43 AM

All times are GMT -5. The time now is 09:27 PM.

Main Menu
Write for LQ is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration