split large text file

allan.sammar@vfemail.net · 09-02-2017, 10:47 PM

How can i split 1 GB text file [wordlists] into multiple smaller size text file?

frankbell · 09-02-2017, 11:09 PM

See man split.

Oh, and thanks for the question. I didn't know about the split command before.

pan64 · 09-03-2017, 03:52 AM

oh yes, and you can use awk too (or perl/python/whatever) if you have a not-so-trivial condition to split.
https://is.gd/j4MZnR

Sefyir · 09-03-2017, 08:47 AM

split was my immediate thought and it works but I'm going to assume the OP wants the integrity of each word saved.
Examining the file below you'll notice it splits on [ok] and [4]. Not good if you have a wordlist.

Code:

$ cat foobar
ok1
ok2
ok3
ok4
ok5
ok6
ok7

Code:

$ split -n2 foobar
$ cat xaa; echo -e '\n\n'; cat xab
...
ok3
ok


4
ok5
...

To make this work, examine these parts of the -n --number=CHUNKS command

Code:

       N      split into N files based on size of input

       K/N    output Kth of N to stdout

       l/N    split into N files without splitting lines/records

       l/K/N  output Kth of N to stdout without splitting lines/records

       r/N    like 'l' but use round robin distribution

       r/K/N  likewise but only output Kth of N to stdout

There's another part, separating on another character besides the newline, however I could not get that to work

Code:

       -t, --separator=SEP
              use  SEP instead of newline as the record separator; '\0' (zero)
              specifies the NUL character

Code:

split -t '\n' foobar 
split: multi-character separator ‘\\n’

I'm not sure why the above does not work, or more specifically, how to specify a newline (or tab, etc)

chrism01 · 09-03-2017, 11:27 PM

For really simple stuff eg just N lines, you could use head/tail, but split is more capable.
As above, for non-trivial rules/situations, use Perl (or similar).

allan.sammar@vfemail.net · 09-13-2017, 09:50 PM

Quote:

Originally Posted by Sefyir

split was my immediate thought and it works but I'm going to assume the OP wants the integrity of each word saved.
Examining the file below you'll notice it splits on [ok] and [4]. Not good if you have a wordlist.

Code:

$ cat foobar
ok1
ok2
ok3
ok4
ok5
ok6
ok7

Code:

$ split -n2 foobar
$ cat xaa; echo -e '\n\n'; cat xab
...
ok3
ok


4
ok5
...

To make this work, examine these parts of the -n --number=CHUNKS command

Code:

       N      split into N files based on size of input

       K/N    output Kth of N to stdout

       l/N    split into N files without splitting lines/records

       l/K/N  output Kth of N to stdout without splitting lines/records

       r/N    like 'l' but use round robin distribution

       r/K/N  likewise but only output Kth of N to stdout

There's another part, separating on another character besides the newline, however I could not get that to work

Code:

       -t, --separator=SEP
              use  SEP instead of newline as the record separator; '\0' (zero)
              specifies the NUL character

Code:

split -t '\n' foobar 
split: multi-character separator ‘\\n’

I'm not sure why the above does not work, or more specifically, how to specify a newline (or tab, etc)

thanks Sefyir that is help alot .. it does work fine ,, thanks again