Split large file into smaller files

mikes88 · 03-21-2012, 10:30 AM

Hello again,

ive been bashing my head on this one trying to look at different AWK scripts to split my file into smaller files based on a specific line.

At the top of every file theres a line that reads:

Code:

/ info / info / servername / info / info / info / info

then a bunch of data under it reading:

Code:

Wed Feb 01 08:51:24 EST 2012,Wed Feb 01 08:51:44 EST 2012,1098.35,
Wed Feb 01 08:51:44 EST 2012,Wed Feb 01 08:52:04 EST 2012,915.25,
Wed Feb 01 08:52:04 EST 2012,Wed Feb 01 08:52:24 EST 2012,937.7,
Wed Feb 01 08:52:24 EST 2012,Wed Feb 01 08:52:44 EST 2012,957.25,
Wed Feb 01 08:52:44 EST 2012,Wed Feb 01 08:53:04 EST 2012,703.3,
Wed Feb 01 08:53:04 EST 2012,Wed Feb 01 08:53:24 EST 2012,757.75,
Wed Feb 01 08:53:24 EST 2012,Wed Feb 01 08:53:44 EST 2012,891.8,
Wed Feb 01 08:53:44 EST 2012,Wed Feb 01 08:54:04 EST 2012,951.75,
Wed Feb 01 08:54:04 EST 2012,Wed Feb 01 08:54:24 EST 2012,997.25,
Wed Feb 01 08:54:24 EST 2012,Wed Feb 01 08:54:44 EST 2012,1018.4,
Wed Feb 01 08:54:44 EST 2012,Wed Feb 01 08:55:04 EST 2012,1046.95,
Wed Feb 01 08:55:04 EST 2012,Wed Feb 01 08:55:24 EST 2012,838.3,
Wed Feb 01 08:55:24 EST 2012,Wed Feb 01 08:56:04 EST 2012,912.05,

What i need to do is read that first line then take every line and out put it to another file named the servername. There are multiple servers in the same file that start with that first line:

Code:

/ info / info / servername / info / info / info / info

acid_kewpie · 03-21-2012, 10:40 AM

keeping this within awk, somethign like this shoouuuuld work. Untested...

Code:

    awk -F"/" '{
      if { $4 != "" } {
         OUTFILE = $4
      } else {
         print $0 OUTFILE
      }
    }' filename

no validity checking whatsoever, but if your input is reliable, should be fine. Offhand, I'm not totally sure if the server name will be the 3rd or 4th field, as the line starts with a field seperator. I'd expect it to be the 4th though.

mikes88 · 03-21-2012, 10:51 AM

Quote:

Originally Posted by acid_kewpie

keeping this within awk, somethign like this shoouuuuld work. Untested...

Code:

    awk -F"/" '{
      if { $4 != "" } {
         OUTFILE = $4
      } else {
         print $0 OUTFILE
      }
    }' filename

no validity checking whatsoever, but if your input is reliable, should be fine. Offhand, I'm not totally sure if the server name will be the 3rd or 4th field, as the line starts with a field seperator. I'd expect it to be the 4th though.

I tried to execute the code but was getting syntax erors...

Code:

awk: cmd. line:1:       if { $7 != "" } {
awk: cmd. line:1:          ^ syntax error
awk: cmd. line:3:       } else {
awk: cmd. line:3:         ^ syntax error

acid_kewpie · 03-21-2012, 10:53 AM

think of it as an exercise for the reader. ;-) mixed up my TCL with my AWK...

druuna · 03-21-2012, 11:01 AM

Hi,

I do believe that the generated OUTFILE also has a nice surprise

colucix · 03-21-2012, 11:01 AM

Chris, may I correct the code?

Code:

    awk -F" / " '{
      if ( $3 != "" )
         OUTFILE = $3
      else 
         print > OUTFILE
    }' filename

Using spaces in the field separator avoids spaces in the output file name. Then basically you have to explicitly redirect the output of the print statement to the output file, using the ">" symbol as in bash (the only difference with bash redirection is that the output is appended by default).

mikes88 · 03-21-2012, 11:14 AM

Isnt working... Just outputs to the screen and not a file.

druuna · 03-21-2012, 11:19 AM

Hi,

I just checked colucix code and it works.

colucix · 03-21-2012, 11:19 AM

Quote:

Originally Posted by mikes88

Isnt working... Just outputs to the screen and not a file.

Please, can you post an example of a real header line?

druuna · 03-21-2012, 11:21 AM

Hi,

Are you sure the first line is exactly as given in post #1? Especially the spaces used.

Here's an alternative to tackling the separator issue:

Code:

#!/bin/bash

awk -F"/" '
/info/ { gsub(/[ ]+/,"",$0)    }
{
  if ( $4 != "" ) {
    OUTFILE = $4
  } else {
    print $0 > OUTFILE
  }
}' filename

Hope this helps.

mikes88 · 03-21-2012, 11:32 AM

Quote:

Originally Posted by druuna

Hi,

Are you sure the first line is exactly as given in post #1? Especially the spaces used.

Here's an alternative to tackling the separator issue:

Code:

#!/bin/bash

awk -F"/" '
/info/ { gsub(/[ ]+/,"",$0)    }
{
  if ( $4 != "" ) {
    OUTFILE = $4
  } else {
    print $0 > OUTFILE
  }
}' filename

Hope this helps.

The spaces are exact. / info info / info / servername / info / info / info / info
(sorry missed a word)

getting the following error:

Code:

awk: cmd. line:6: (FILENAME=testing-split.csv FNR=1) fatal: expression for `>' redirection has null string value

grail · 03-21-2012, 11:44 AM

This error would tell us that the if has not been entered and hence OUTFILE variable is empty.

colucix · 03-21-2012, 12:11 PM

Please, post your current code and a REAL header line.

mikes88 · 03-21-2012, 12:38 PM

Quote:

Originally Posted by colucix

Please, post your current code and a REAL header line.

Code:

awk -F"/" '
"/ Toronto MDS_GW2 /" { gsub(/[ ]+/,"",$0)    }
{
  if ( $3 != "" ) {
    OUTFILE = $3
  } else {
    print $0 > OUTFILE
  }
}' testing-split.csv

header

Code:

/ Toronto MDS_GW2 / 10_10_10_10_7036 / tsr2 / service(type=dist) / service / SELECTFEED / Update

druuna · 03-21-2012, 12:50 PM

Hi,

Have a look at this

Code:

#!/bin/bash

awk -F"/" '
/\// { gsub(/[ ]+/,"",$0) }
{
  if ( $4 != "" ) {
    OUTFILE = $4
  } else {
    print $0 > OUTFILE
  }
}' testing-split.csv

I'm assuming that Toronto MDS_GW2 is not unique, so I used /\// which looks for a forward slash (which needs to be escaped, it has special meaning in awk). I also believe $3 should be $4.