Scripting: split file into 12 lines array
 

Hi,

I need to split a file into an array. The split is at every 12 line.
eg.

line 1
line 2
...
line 24


then the array will look something like

items[0] = line 1 ... line 12
items[1] = line 13 ... line 24

right now I am read line by line from the file and putting into an array. This is a little slow.

If there is a better way, please point me in the right direction.

Thanks,
Mike

tell us what exactly what problem you are solving.

I'm not sure how you can avoid doing this (in one way or another). You have to read the file to get the items at all, so to that degree you're I/O bound.

If the lines are uniform, you could theoretically do something involving bytes or size, but that strikes me as an unlikely possibility.

I don't know if it will run faster, but you could use something like this (untested code, just for concept)

Note that sed starts line numbering at 1, not at 0.

This will cause sed to read the entire file every time through the loop, not just the 12 lines requested.

Please see the thread below:
sed script to parse a file into smaller files with set # of lines

Kevin Barry

I did some testing a while back, and found perl was faster at subsetting a (huge) file than sed, even if both were stopped after the requisite lines (only) were found rather than continuing to read.
As usual, YMMV.

with 80million? lines of file, you can (for the last post in that thread)
1) lose the cat because its useless,
2) avoid using bash's while read loop to read big files.
3) and if bash solution is desired, no need to call external sed command. use bash's own string substitution.
4) or use awk

Please post an example, either here or in the other thread.
Kevin Barry

an example for which point ? 1,2,3 or 4?

Your solution to the problem taking into account all 4.
Kevin Barry

1) Instead of . use input redirection
or open/close the file
2) its well known that processing large files with bash's while read loop is slower (much slower) than using tools like awk. you can search some of my previous posts (way back) which i demonstrated this.

3) I am not sure what that sed line is doing ie s/./&/, care to explain?

4) Have already provided awk suggestion in that thread.

Ok, but I'm not sure how you direct to separate files with awk without running through the file more than once. That's my awk ignorance, though, which is why I was hoping you had an example.I'm not sure, either. It's something that OP had in his or her original script, and again, I didn't test my code. It was an example. This isn't the thread to argue about such things; therefore, it would be helpful if you'd show what you mean by "awk can do it better."
Kevin Barry

you mean this?
NR%4==1 means at every 4th line. eg
using this concept, OP can change it to NR%4000000 for his requirement. notice that count variable "c" is incremented at every 4th line. this variable "c" will be appended to file name.
the awk one liner above summarizes what OP did will those bunch of seds
the s/./&/w is just writing to the file (my guess), which is just a simple print with redirection ">" in awk.

whatever it is, there's no need to call sed. (echo or printf will do )

what i mean "better" is in the sense of speed performance on big files as compared to bash's while read loop.