LinuxQuestions.org
Go Job Hunting at the LQ Job Marketplace
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices



Reply
 
Search this Thread
Old 10-28-2002, 10:02 AM   #1
dong@aproposret
LQ Newbie
 
Registered: Oct 2002
Location: Chapel Hill, NC
Posts: 7

Rep: Reputation: 0
noeol help needed


I've received a .txt file of customer data, & it is set to 'noeol' - thought there are thousands of records,the file shows only 1 huge line. I can see the the carriage return (^M) throughout the file, but I do not know the syntax to use in vi (or vim) to change the carriage return to an end-of-line that my vi editor will recognize. I can see it properly if I head the file, but outputting the result to a new file doesn't help. Does anyone know either the syntax to turn off the 'noeol' set mode, or a vi substitution command that will give me a new record after each ^M?
 
Old 10-29-2002, 03:59 AM   #2
Mik
Senior Member
 
Registered: Dec 2001
Location: The Netherlands
Distribution: Ubuntu
Posts: 1,316

Rep: Reputation: 46
A substitution command in vi for removing ^M's would look like this:

:%s/^M//g

To be able to enter the ^M as a control character. Keep the ctrl button pressed and then type V and then M.

You can also do the same with sed:

sed -e "s/^M//" filename > newfile

The ^M should be entered in the same way as for vi.

Last edited by Mik; 10-29-2002 at 04:00 AM.
 
Old 10-29-2002, 07:31 AM   #3
dong@aproposret
LQ Newbie
 
Registered: Oct 2002
Location: Chapel Hill, NC
Posts: 7

Original Poster
Rep: Reputation: 0
The problem wasn't to remove the carraige return characters. What I needed to do was to insert an actual newline at the point where the carriage return characters were present. I don't know how to do it with vi, & was having trouble with the correct syntax in sed - but if finally hit on the correct syntax:
s/^M/\
/g
The reason you are seeing the command on two lines is I inserted a newline by hitting the <Enter> key (after escaping it's special meaning).
Here is what I started with - one line where I needed it to be 143,238 lines(records):
08900004|Richard|Adams|1106 W Briggs Ave||Fairfield|IA|52556-2523|no^M044010002|Linda|Porter|2500 Woodscrest Ave||Lincoln|NE|68502-4055|no^M044010011|Jim|Coyne|
4900 Osprey Dr||St Petersburg|FL|78465|no^M044010028|Helenjon|Olerich|2722 Wyoming St||Omaha|NE|68112-2838|no^M044010029|Barry|Haltom|222SETTLER Way Blvd|# 832|Sugar Land|TX|77478|no^M044010038|Shelia|Raglin|159 Hutton St|Apt 23|Jersey City|NJ|07307-3750|no^M044010039|Philip|Caper|79 Matin St||Cambridge|MA|02138|no^M044010051|Daniel|Baylson|172 Amber Dr||San Francisco|CA|94131-1642|no^M044010052|D
ebra|Brown|15612Decator||Omaha|NE|68118|no^M044010058|Bill|Crumbaugh|600 Main|| Aurelia|IA|51005|no^M044010061|Jos|Burger|BURGERMEE|STER CEULENSTR.68|MAASTRICHT|NL|6212 CT|no^M044010062|Ginia|Cleeves|4172 Piedra Ct||Boulder|CO|80301-1632|no

This is what I ended up with:
08900004|RICHARD|ADAMS|1106 W BRIGGS AVE||FAIRFIELD|IA|52556-2523|NO|
044010002|LINDA|PORTER|2500 WOODSCREST AVE||LINCOLN|NE|68502-4055|NO|
044010011|JIM|COYNE|4900 OSPREY DR||ST PETERSBURG|FL|78465|NO|
044010028|HELENJON|OLERICH|2722 WYOMING ST||OMAHA|NE|68112-2838|NO|
044010029|BARRY|HALTOM|222SETTLER WAY BLVD|# 832|SUGAR LAND|TX|77478|NO|
044010038|SHELIA|RAGLIN|159 HUTTON ST|APT 23|JERSEY CITY|NJ|07307-3750|NO|
 
Old 10-29-2002, 09:34 AM   #4
Mik
Senior Member
 
Registered: Dec 2001
Location: The Netherlands
Distribution: Ubuntu
Posts: 1,316

Rep: Reputation: 46
I guess I must have read your post wrong, sorry about that. But just in case you want to still know how to do it in vi, here is one way:

:%s/^M/^M/g

The ^M you are replacing it with will be converted to the proper newline character so you will get the right newline characters.
 
Old 10-29-2002, 10:10 AM   #5
dong@aproposret
LQ Newbie
 
Registered: Oct 2002
Location: Chapel Hill, NC
Posts: 7

Original Poster
Rep: Reputation: 0
Thank you! I just tried it & it worked.
The thought never crossed my mind to attempt to replace something with itself to accomplish this. And all this time, I foolishly thought I was a vi pro...
 
Old 10-29-2002, 10:24 AM   #6
dong@aproposret
LQ Newbie
 
Registered: Oct 2002
Location: Chapel Hill, NC
Posts: 7

Original Poster
Rep: Reputation: 0
Mik,
BTW, when I used the sed script, it ran in 5 seconds, but when i did it in vi, it took over 11 minutes! The source file is over 10MG & has 148,376 records (changes to be made) rolled up into one line.
 
Old 05-22-2008, 05:14 AM   #7
dreamcarrior
Member
 
Registered: Nov 2005
Location: Waco, TX
Distribution: Fedora and DSL
Posts: 49

Rep: Reputation: 15
how about ^@? I got many files containing ^@ in the files and would like to remove all ^@ characters. It will be convenient to remove all ^@ characters in batch so that I don't have to go through thousands of them. Can anyone help?
 
Old 05-22-2008, 09:37 PM   #8
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.6, Centos 5.10
Posts: 16,324

Rep: Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041
The solns in post #2 should work
 
Old 05-23-2008, 11:43 AM   #9
dreamcarrior
Member
 
Registered: Nov 2005
Location: Waco, TX
Distribution: Fedora and DSL
Posts: 49

Rep: Reputation: 15
Yes, the solution in number 2 does work to edit the files one by one. However, I have thousands of them, it is tedious to work on those files one by one. It is much better to have scripts converting all of them automatically.

I finally used the following script to keep all characters and numbers I am interested in and get rid of everything else.

sed -e '/wave/ d' -e 's/[^0-9,-]//g'
 
Old 05-25-2008, 08:32 PM   #10
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.6, Centos 5.10
Posts: 16,324

Rep: Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041
Indeed, but the 2nd soln in post #2 is at the cmd line, so you could script it
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
C Help Needed Please. InvisibleSniper Programming 8 10-17-2005 10:05 AM
help needed sajid anwar Linux - General 1 11-19-2004 08:20 AM
Help needed! =) kith Linux - General 14 02-10-2004 01:44 PM
C++ help needed sabeel Programming 6 11-27-2003 01:19 PM
help needed here... c12ayon Programming 2 10-29-2003 11:59 AM


All times are GMT -5. The time now is 02:04 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration