[SOLVED] Split large .lvm file when column number changes

CLD422 · 03-02-2012, 06:17 PM

I have several large .lvm files which contain 2 to 5 columns. The problem is that they can start with 4 columns and midway through become 2 columns. Then back to 4. I need to split these files at each point where the column number changes. The two files I am working with currently contain ~99,900,00 lines each. Below is an exmple of what the data looks like when it switches. Any help would be much appreciated.

--I have tried 'split' and looked at 'csplit' - it looks like 'awk' may have the ability to do this, but I don't have enough linux know-how.

0.761719 0.000000
0.750000 0.000000
0.754395 0.000000
0.748535 0.000000
0.751465 0.000000
0.747070 0.000000
0.745605 0.000000
0.747070 0.000000
0.742676 0.000000
0.741211 0.000000
0.752930 0.000000
0.748535 0.000000
0.747070 0.000000
0.747070 0.000000
0.748535 0.000000
0.752930 0.000000
0.763184 0.000000
0.763184 0.000000
0.757324 0.000000
0.754395 0.000000
0.757324 0.000000
0.754395 0.000000
0.755859 0.000000
0.760254 0.000000
0.750000 0.000000
0.751465 0.000000
0.751465 0.000000
0.755859 0.000000
0.747070 0.000000
0.750000 0.000000
0.000000 0.000000 0.000000 0.741211
0.000000 0.000000 0.000000 0.750000
0.000000 0.000000 0.000000 0.741211
0.000000 0.000000 0.000000 0.745605
0.000000 0.000000 0.000000 0.736816
0.000000 0.000000 0.000000 0.747070
0.000000 0.000000 0.000000 0.747070
0.000000 0.000000 0.000000 0.751465
0.000000 0.000000 0.000000 0.751465
0.000000 0.000000 0.000000 0.748535
0.000000 0.000000 0.000000 0.744141
0.000000 0.000000 0.000000 0.745605
0.000000 0.000000 0.000000 0.741211
0.000000 0.000000 0.000000 0.748535
0.000000 0.000000 0.000000 0.752930
0.000000 0.000000 0.000000 0.764648
0.000000 0.000000 0.000000 0.761719

grail · 03-02-2012, 11:00 PM

Maybe something like:

Code:

awk 'x != NF{x = NF}{print > "file"x}' file

This will place all similar numbered columns into files based on number of columns.

schneidz · 03-03-2012, 09:15 AM

[untested]

Code:

cat file.txt | while read line
do
 if [`echo $line | wc -w` -eq 2 ]
 then
  echo $line >> file.2-columns
 fi
 if [`echo $line | wc -w` -eq 4 ]
 then
  echo $line >> file.4-columns
 fi
done

CLD422 · 03-05-2012, 04:51 PM

grail, Thank you very much! That is exactly what I needed.