Split large file into smaller files

mikes88 · 03-21-2012, 01:01 PM

Quote:

Originally Posted by druuna

Hi,

Have a look at this

Code:

#!/bin/bash

awk -F"/" '
/\// { gsub(/[ ]+/,"",$0) }
{
  if ( $4 != "" ) {
    OUTFILE = $4
  } else {
    print $0 > OUTFILE
  }
}' testing-split.csv

I'm assuming that Toronto MDS_GW2 is not unique, so I used /\// which looks for a forward slash (which needs to be escaped, it has special meaning in awk). I also believe $3 should be $4.

when i tried to run that i got the following:

Code:

-bash-3.2$ ./split.sh
awk: cmd. line:6: (FILENAME=testing-split.csv FNR=1) fatal: expression for `>' redirection has null string value

-bash-3.2$ awk -f split.sh
awk: split.sh:3: awk -F"/" '
awk: split.sh:3:           ^ invalid char ''' in expression

druuna · 03-21-2012, 01:08 PM

Remove the double quotes around /\//

Code:

/\//  vs  "/\//"

They are not there in my code, but are in your code.....

EDIT: Or just copy/paste my code......

grail · 03-21-2012, 01:12 PM

So I notice that you are on an older version of bash, which version and derivative (ie. gawk, mawk, etc) of awk are you using?

mikes88 · 03-21-2012, 01:13 PM

Quote:

Originally Posted by druuna

Remove the double quotes around /\//

Code:

/\//  vs  "/\//"

They are not there in my code, but are in your code.....

EDIT: Or just copy/paste my code......

UI am using your code. i copied and pasted the code right from here...

mikes88 · 03-21-2012, 01:32 PM

Quote:

Originally Posted by grail

So I notice that you are on an older version of bash, which version and derivative (ie. gawk, mawk, etc) of awk are you using?

Awk 3.1.5

grail · 03-21-2012, 02:05 PM

hmmm ... seems to run just fine for me too

What happens if you run the following at the command line:

Code:

awk -F" */ *" '{if ( $4 != "" )OUTFILE = $4;else print > OUTFILE}' testing-split.csv

Also I assume there is more than just the header line in the file.

mikes88 · 03-21-2012, 02:12 PM

Quote:

Originally Posted by grail

hmmm ... seems to run just fine for me too

What happens if you run the following at the command line:

Code:

awk -F" */ *" '{if ( $4 != "" )OUTFILE = $4;else print > OUTFILE}' testing-split.csv

Also I assume there is more than just the header line in the file.

Not sure why it wont work for me. I even typed it out and also just pasted the 1 liner... Nothing

mikes88 · 03-21-2012, 02:14 PM

Quote:

Originally Posted by mikes88

Not sure why it wont work for me. I even typed it out and also just pasted the 1 liner... Nothing

WOW it works! i opened the file with the data and apparently theres a space at the top... so how do you get it to skip that space?

grail · 03-21-2012, 02:44 PM

??? more information please. You opened what file? the only way a space could be presented is if it were in your original file.

mikes88 · 03-21-2012, 03:03 PM

Quote:

Originally Posted by grail

??? more information please. You opened what file? the only way a space could be presented is if it were in your original file.

in the original data file testing-split.csv the dumb output program puts a space at the top oof the file so instead of teh data starting on line 1 it starts on line 2. That space was the issue and giving off that error. Just wondering how i can get awk to skip the first line or even just remove it before running.

grail · 03-21-2012, 03:13 PM

Code:

NR>1{...}

druuna · 03-22-2012, 02:36 AM

Hi,

Quote:

Originally Posted by mikes88

in the original data file testing-split.csv the dumb output program puts a space at the top oof the file so instead of teh data starting on line 1 it starts on line 2. That space was the issue and giving off that error. Just wondering how i can get awk to skip the first line or even just remove it before running.

Try this:

Code:

#!/bin/bash

awk -F"/" '/\// { gsub(/[ ]+/,"",$0) ; OUTFILE = $4 }
NR > 2 { print $0 > OUTFILE }
' testing-split.csv

Hope this helps.

mikes88 · 03-22-2012, 09:24 AM

Quote:

Originally Posted by druuna

Hi,
Try this:

Code:

#!/bin/bash

awk -F"/" '/\// { gsub(/[ ]+/,"",$0) ; OUTFILE = $4 }
NR > 2 { print $0 > OUTFILE }
' testing-split.csv

Hope this helps.

Replace

Code:

awk -F"/" '
/\// { gsub(/[ ]+/,"",$0) }
{
  if ( $4 != "" ) {
    OUTFILE = "cpu-"$4".csv"
  } else {
    print $0 > OUTFILE
  }
}' cpu.csv

with that new code then?

druuna · 03-22-2012, 10:07 AM

Hi,

Quote:

Originally Posted by mikes88

Replace [old] with that new code then?

Yes.

mikes88 · 03-22-2012, 10:14 AM

Quote:

Originally Posted by druuna

Hi,
Yes.

works like a charm! once again thank you Druuna.