LinuxQuestions.org
Did you know LQ has a Linux Hardware Compatibility List?
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices

Reply
 
Search this Thread
Old 05-09-2008, 04:12 PM   #1
lourencojunior
LQ Newbie
 
Registered: May 2008
Posts: 8

Rep: Reputation: 0
Merge lines from a file


Hallo to everyone!

I would like to merge lines of a huge file into a formatted one. Be the following file (it has 1.9GB) called `input.txt':
Code:
2
3
5
7
11
13
17
19
23
29
...
3999999797
3999999799
3999999869
3999999881
3999999901
3999999911
3999999919
3999999937
3999999979
I would like to get (output.txt):
Code:
2, 3, 5, 7, 11, 13, 17, 19, 23, 29, (...) 3999999797, 3999999799, 3999999869, 3999999881, 3999999901, 3999999911, 3999999919, 3999999937, 
3999999979.
Well, I have already tried the following commands:
Code:
$ sed -e :a -e N -e 's/\n/, /' -e ta input.txt > output.txt
It fails because there is no memory enough to complete the operation since the sed buffer uses the memory.

Code:
$ paste -s -d "," input.txt > output.txt
It almost works, but it misses the space required for formatting.

Code:
$ paste -s -d "," input.txt | sed "s/,/, /g" > output.txt
Same problem that first option tried.

Code:
$ tr -t "," ", " < input.txt > output.txt
It does not substitute one character `,' by two ", "


- Does anyone have one or more solution for this problem?

Thanks in advance.
[]s
 
Old 05-09-2008, 04:21 PM   #2
druuna
LQ Veteran
 
Registered: Sep 2003
Posts: 10,532
Blog Entries: 7

Rep: Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374
Hi,

Don't know if this will work with 4 billion numbers, but give it a try:

awk 'BEGIN { ORS=", " } { print } ' infile > outfile

or

awk '{ printf("%d, ", $1) } ' infile > outfile

There will be an extra ,<space> at the end of the line in the output file.

Hope this helps.

Last edited by druuna; 05-09-2008 at 04:27 PM. Reason: Added second awk solution
 
Old 05-09-2008, 04:47 PM   #3
radoulov
Member
 
Registered: Apr 2007
Location: Milano, Italia/Варна, България
Distribution: Ubuntu, Open SUSE
Posts: 212

Rep: Reputation: 35
Code:
perl -00 -ple's/\n/, /g' file
 
Old 05-09-2008, 05:17 PM   #4
vadkutya
Member
 
Registered: Apr 2008
Distribution: slackware 10.2
Posts: 117

Rep: Reputation: 16
this sounds like homework, isn't it? the file is explicitly large so you can't use editor macros. you have to do the job with sed or awk...

but druuna helped you already

vadkutya
 
Old 05-09-2008, 08:33 PM   #5
rodrigosetti
LQ Newbie
 
Registered: May 2008
Posts: 1

Rep: Reputation: 0
shell solution

I think this one will fail because of memory too:

paste -s -d, < input.txt | sed 's/,/, /g' > output.txt
 
Old 05-12-2008, 01:54 PM   #6
lourencojunior
LQ Newbie
 
Registered: May 2008
Posts: 8

Original Poster
Rep: Reputation: 0
Thanks for all and sorry for delay in replying!!

I've used druuna solution: awk 'BEGIN { ORS=", " } { print } ' infile > outfile

Well, now I have another problem to solve. My file has one line and this finish with ", " and I need to substitute by `.'

Again, sed does not work properly because the memory issues.

Any idea?
 
  


Reply

Tags
lines, merge, paste, sed, tr


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Merge lines in a file using sed arobic Programming 8 01-20-2012 02:11 PM
LXer: KHTML Vs Webkit: To Merge or Not To Merge LXer Syndicated Linux News 0 10-27-2007 06:41 AM
File merge frustration! the-yikes Slackware 2 04-28-2007 09:01 PM
merge avi with subtitle file ? norah_cobain Linux - Software 28 05-25-2006 05:42 PM
merge multiple lines of a single file into one line groverrajiv Linux - Newbie 4 05-26-2004 02:38 AM


All times are GMT -5. The time now is 03:25 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration