Merge lines from a file

lourencojunior · 05-09-2008, 04:12 PM

Hallo to everyone!

I would like to merge lines of a huge file into a formatted one. Be the following file (it has 1.9GB) called `input.txt':

Code:

I would like to get (output.txt):

Code:

2, 3, 5, 7, 11, 13, 17, 19, 23, 29, (...) 3999999797, 3999999799, 3999999869, 3999999881, 3999999901, 3999999911, 3999999919, 3999999937, 
3999999979.

Well, I have already tried the following commands:

Code:

$ sed -e :a -e N -e 's/\n/, /' -e ta input.txt > output.txt

It fails because there is no memory enough to complete the operation since the sed buffer uses the memory.

Code:

$ paste -s -d "," input.txt > output.txt

It almost works, but it misses the space required for formatting.

Code:

$ paste -s -d "," input.txt | sed "s/,/, /g" > output.txt

Same problem that first option tried.

Code:

$ tr -t "," ", " < input.txt > output.txt

It does not substitute one character `,' by two ", "

- Does anyone have one or more solution for this problem?

Thanks in advance.
[]s

druuna · 05-09-2008, 04:21 PM

Hi,

Don't know if this will work with 4 billion numbers, but give it a try:

awk 'BEGIN { ORS=", " } { print } ' infile > outfile

or

awk '{ printf("%d, ", $1) } ' infile > outfile

There will be an extra ,<space> at the end of the line in the output file.

Hope this helps.

radoulov · 05-09-2008, 04:47 PM

Code:

perl -00 -ple's/\n/, /g' file

vadkutya · 05-09-2008, 05:17 PM

this sounds like homework, isn't it? the file is explicitly large so you can't use editor macros. you have to do the job with sed or awk...

but druuna helped you already

vadkutya

rodrigosetti · 05-09-2008, 08:33 PM

I think this one will fail because of memory too:

paste -s -d, < input.txt | sed 's/,/, /g' > output.txt

lourencojunior · 05-12-2008, 01:54 PM

Thanks for all and sorry for delay in replying!!

I've used druuna solution: awk 'BEGIN { ORS=", " } { print } ' infile > outfile

Well, now I have another problem to solve. My file has one line and this finish with ", " and I need to substitute by `.'

Again, sed does not work properly because the memory issues.

Any idea?