[SOLVED] how to add blank lines whenever the value of a given column alters in a data file?

vcmota · 01-07-2011, 11:47 AM

Please, somebody help me, I am dying here... All I want is a command that reads one data file with several columns and prints it in another one. However, whenever the value in one specific column alters, it prints one empty line in the new file. For example, consider the file

2.54000000 2.54000000
2.54000000 2.54000000
2.54000000 2.54000000
2.54000000 2.56000000
2.54000000 2.56000000
2.54000000 2.56000000
2.54000000 2.58000000
2.54000000 2.58000000
2.54000000 2.58000000
2.54000000 2.60000000
2.54000000 2.60000000
2.54000000 2.60000000
2.56000000 2.56000000
2.56000000 2.56000000
2.56000000 2.56000000
2.56000000 2.58000000
2.56000000 2.58000000
2.56000000 2.58000000
2.56000000 2.60000000
2.56000000 2.60000000
2.56000000 2.60000000

what I need is a command that return the file

2.54000000 2.54000000
2.54000000 2.54000000
2.54000000 2.54000000

2.54000000 2.56000000
2.54000000 2.56000000
2.54000000 2.56000000

2.54000000 2.58000000
2.54000000 2.58000000
2.54000000 2.58000000

2.54000000 2.60000000
2.54000000 2.60000000
2.54000000 2.60000000

2.56000000 2.56000000
2.56000000 2.56000000
2.56000000 2.56000000

2.56000000 2.58000000
2.56000000 2.58000000
2.56000000 2.58000000

2.56000000 2.60000000
2.56000000 2.60000000
2.56000000 2.60000000

Kills me is that this problem seems to have a simple solution, a simple bash command or a simple combination of bash commands rather than an elaborated script, which I can't see... Thank you all very much!

David the H. · 01-07-2011, 12:32 PM

It could certainly be done easily in awk as well, but here's my first go at a bash-only solution:

Code:

#!/bin/bash

while read line; do

     if [[ ! ${line#* } = ${prev:-${line#* }} ]] ; then
          echo -e "\n$line"
     else
          echo "$line"
     fi

prev="${line#* }"

done <infile.txt >outfile.txt

exit 0

Edit: Here's a slightly modified version that allows you to specify whichever column you want, simply by changing the command used in the test variable.

Code:

#!/bin/bash

while read line; do

test="$( cut -d" " -f3 <<<"$line" )" # using cut, for example

     if [[ ! $test = ${prev:-$test} ]] ; then
          echo -e "\n$line"
     else
          echo "$line"
     fi

prev="$test"

done <infile.txt >outfile.txt

exit 0

grail · 01-08-2011, 01:26 AM

As the man said, awk can do it too:

Code:

awk 'BEGIN{printf "Enter column number: ";getline col < "-"}NR != 1 && $col != old{print ""}{old = $col}1' file

This includes the request for a column number. If you are not interested in being requested, delete the BEGIN and prior to first quote put the following:

Code:

-vcol=<number>

Where number is again the column you wish to change on.

David the H. · 01-08-2011, 08:20 PM

Grail, that's a very interesting solution. I didn't know you could have awk prompt for user input. Thanks for teaching me something new.

vcmota · 01-10-2011, 07:52 AM

Grail and David, thank you both very much for your help. I gave up of looking for a simple command after sometime and decided to wrote an script using gawk. It is not so simple and elegant as the solution that Grail proposes (I am a beginner in the world of programming...), but it works fine. I put it in my bin folder, and now I have a simple command after all. If you guys are interested I can post it here. Thank you very much again!

grail · 01-10-2011, 06:25 PM

Always interested to see a solution

Please mark question as SOLVED once you have one.

vcmota · 01-11-2011, 07:07 AM

Ok Grail, this is my solution:

awk -v column=$1 -v spaces=$2 '!/^$/{ ++i
if(i == 1) {
lfield[i]=$0
field[i]=$column
getline
++i
lfield[i]=$0
field[i]=$column}
else{
lfield[i]=$0
field[i]=$column}
if(field[i-1] != field[i]){
print lfield[i-1]
j=0
while (j <= spaces-1){
print ""
++j}
}
else{
print lfield[i-1]}
}
END {print lfield[i]}' $3

It reads the column and allow you to chose the number of spaces you want. This is useful for pipe, and also if for example you relies on gnuplot for plotting data (my case), since 2 spaces are interpreted as separation between two data blocks and 1 space is the requested separation in order to do 3 dimensional grid points plots. As I said, it is not elegant and simple as your solution (I am a newbie in awk), but it works. By the way, could you indicate me any good manual for awk? All the manuals I could find in google are after all exactly the same (http://www.cs.unibo.it/~renzo/doc/awk/nawkA4.pdf) and many details of the language are not too clear (the use of getline for example).

grail · 01-11-2011, 07:51 AM

The following is my goto page when I am stuck / forget something:

http://www.gnu.org/manual/gawk/html_node/index.html

I would probably point out that your opening test seems a bit meaningless unless your data has blank lines?

As you are still learning, here is all you needed in your code whilst keeping most of the same constructs (I did change while to for as you can probably see it is the same process):

Code:

awk -v column=$1 -v spaces=$2 '!/^$/{
    lfield[++i]=$0
    field[i]=$column
    if(i == 1) {
        next
    }

    print lfield[i-1]

    if(field[i-1] != field[i])
        for(j=0; j <= spaces-1; j++)
            print ""
}
END {print lfield[i]}' $3

vcmota · 01-11-2011, 10:41 AM

Grail, thank you again for the help. I appreciate the changes you made and I am going to use them. Yes, I need the opening because the file will certainly have blank lines in case of pipe. That is, in case I need to give spaces regarding two columns, after the first usage of the command I will have blank lines, which must be ignored in executing the command for the second time, like this:

myscript 2 2 file.dat | myscript 1 2

grail · 01-11-2011, 08:47 PM

A perhaps simpler way then is to use NF. Just put those to letters in place of - !/^$/

This works as everything in front of curly braces is evaluated as an expression where zero is false and all else is true.
NF = 0 on lines with no fields
NF > 0 on all other lines