Linux - GeneralThis Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
I've received a .txt file of customer data, & it is set to 'noeol' - thought there are thousands of records,the file shows only 1 huge line. I can see the the carriage return (^M) throughout the file, but I do not know the syntax to use in vi (or vim) to change the carriage return to an end-of-line that my vi editor will recognize. I can see it properly if I head the file, but outputting the result to a new file doesn't help. Does anyone know either the syntax to turn off the 'noeol' set mode, or a vi substitution command that will give me a new record after each ^M?
The problem wasn't to remove the carraige return characters. What I needed to do was to insert an actual newline at the point where the carriage return characters were present. I don't know how to do it with vi, & was having trouble with the correct syntax in sed - but if finally hit on the correct syntax:
s/^M/\
/g
The reason you are seeing the command on two lines is I inserted a newline by hitting the <Enter> key (after escaping it's special meaning).
Here is what I started with - one line where I needed it to be 143,238 lines(records):
08900004|Richard|Adams|1106 W Briggs Ave||Fairfield|IA|52556-2523|no^M044010002|Linda|Porter|2500 Woodscrest Ave||Lincoln|NE|68502-4055|no^M044010011|Jim|Coyne|
4900 Osprey Dr||St Petersburg|FL|78465|no^M044010028|Helenjon|Olerich|2722 Wyoming St||Omaha|NE|68112-2838|no^M044010029|Barry|Haltom|222SETTLER Way Blvd|# 832|Sugar Land|TX|77478|no^M044010038|Shelia|Raglin|159 Hutton St|Apt 23|Jersey City|NJ|07307-3750|no^M044010039|Philip|Caper|79 Matin St||Cambridge|MA|02138|no^M044010051|Daniel|Baylson|172 Amber Dr||San Francisco|CA|94131-1642|no^M044010052|D
ebra|Brown|15612Decator||Omaha|NE|68118|no^M044010058|Bill|Crumbaugh|600 Main|| Aurelia|IA|51005|no^M044010061|Jos|Burger|BURGERMEE|STER CEULENSTR.68|MAASTRICHT|NL|6212 CT|no^M044010062|Ginia|Cleeves|4172 Piedra Ct||Boulder|CO|80301-1632|no
This is what I ended up with:
08900004|RICHARD|ADAMS|1106 W BRIGGS AVE||FAIRFIELD|IA|52556-2523|NO|
044010002|LINDA|PORTER|2500 WOODSCREST AVE||LINCOLN|NE|68502-4055|NO|
044010011|JIM|COYNE|4900 OSPREY DR||ST PETERSBURG|FL|78465|NO|
044010028|HELENJON|OLERICH|2722 WYOMING ST||OMAHA|NE|68112-2838|NO|
044010029|BARRY|HALTOM|222SETTLER WAY BLVD|# 832|SUGAR LAND|TX|77478|NO|
044010038|SHELIA|RAGLIN|159 HUTTON ST|APT 23|JERSEY CITY|NJ|07307-3750|NO|
Thank you! I just tried it & it worked.
The thought never crossed my mind to attempt to replace something with itself to accomplish this. And all this time, I foolishly thought I was a vi pro...
Mik,
BTW, when I used the sed script, it ran in 5 seconds, but when i did it in vi, it took over 11 minutes! The source file is over 10MG & has 148,376 records (changes to be made) rolled up into one line.
how about ^@? I got many files containing ^@ in the files and would like to remove all ^@ characters. It will be convenient to remove all ^@ characters in batch so that I don't have to go through thousands of them. Can anyone help?
Yes, the solution in number 2 does work to edit the files one by one. However, I have thousands of them, it is tedious to work on those files one by one. It is much better to have scripts converting all of them automatically.
I finally used the following script to keep all characters and numbers I am interested in and get rid of everything else.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.