need help processing large data files

lothario · 05-25-2008, 02:48 AM

I have to process some very large text data files.
Each is over 100 Mb.

Here is a sample:

Quote:

system A 2008_05_24:23: ...data...
system A 2008_05_24:23: ...data...
system A 2008_05_24:23: ...data...
system A 2008_05_24:22: ...data...
system A 2008_05_24:22: ...data...
system B 2008_05_24:22: ...data...
system B 2008_05_24:22: ...data...
system A 2008_05_24:22: ...data...
system A 2008_05_24:21: ...data...
system A 2008_05_24:21: ...data...
system B 2008_05_24:21: ...data...

I want to insert a "-----" separator whenever there
is any difference in the first 22 characters of the
current and next line.

In other words, the above data file should look
like this:

Quote:

system A 2008_05_24:23: ...data...
system A 2008_05_24:23: ...data...
system A 2008_05_24:23: ...data...
-----
system A 2008_05_24:22: ...data...
system A 2008_05_24:22: ...data...
-----
system B 2008_05_24:22: ...data...
system B 2008_05_24:22: ...data...
-----
system A 2008_05_24:22: ...data...
-----
system A 2008_05_24:21: ...data...
system A 2008_05_24:21: ...data...
-----
system B 2008_05_24:21: ...data...

I am not bash script expert, I need some help.

Thanks.

druuna · 05-25-2008, 06:11 AM

Hi,

Would this do:

Code:

#!/bin/bash

awk 'BEGIN { prevPart = "" }
{ 
currPart = substr($0,1,23)
if ( prevPart != currPart && NR != "1" ) { print "-----" }
print $0
prevPart = currPart }
' infile

A testrun with the provided data:

Code:

$ ./seperate.sh
system A 2008_05_24:23: ...data...
system A 2008_05_24:23: ...data...
system A 2008_05_24:23: ...data...
-----
system A 2008_05_24:22: ...data...
system A 2008_05_24:22: ...data...
-----
system B 2008_05_24:22: ...data...
system B 2008_05_24:22: ...data...
-----
system A 2008_05_24:22: ...data...
-----
system A 2008_05_24:21: ...data...
system A 2008_05_24:21: ...data...
-----
system B 2008_05_24:21: ...data...
$

The && NR != "1" part is there to supress the first ----- line. If anything is unclear, just ask.

Hope this helps.

lothario · 05-27-2008, 10:16 PM

Thanks. That was exactly what I needed.