How to Split a large file according to the same aggregated files

Haba2015 · 08-17-2015, 01:22 PM

Hi Everyone,

I have aggregated some files (>100) using:

cat *.txt >> bigfie.txt

Now I want to split this big file according to the original or individual filenames. I want my files back according to former filenames.

I have the file names in x*** when I do:

split bigfile.txt ~/path/newfiles/

Please help!!!

schneidz · 08-17-2015, 01:27 PM

what does bigfie.txt look like. is there any type of delimiter (or were the original files all the same length) ?

else there is no way to know where the original files begin (it'll be like getting pee out of a pool).

grail · 08-17-2015, 01:28 PM

Unless you have some way of identifying, within the bigfile.txt, where a file's contents begin and start I would think your pretty buggered. As far as I am aware cat will not
have placed any delimiters in the file.

Are you able to provide any more information on whether or not you have any way to tell where data starts and stops?

Haba2015 · 08-17-2015, 01:49 PM

Thanks Grail and schneidz

The files are based on 6hourly precipitation format. For example, they are in this format:

2015010106.txt
2015010112.txt
2015010118.txt
.
.
.
.
2015033018.txt

where first three files stand for 2015 Jan 1(2015/01/01), 6 UTC, 12 UTC, 18 UTC respectively. I did aggregate them together to perform some overall verification analysis. For space sake, I deleted individual files. Mistake!

Now I need to split this again, and later subset according to each UTC.

schneidz · 08-17-2015, 01:53 PM

are you saying the lines can be differentiated by the timestamp ?

then maybe loop thru the file and grep out based on timestamp.

Haba2015 · 08-17-2015, 02:03 PM

Thanks. I have tried to do that:

For example to extract 6UTC,

grep 2015????06 bigfile.txt>only6UTC.txt

did noting. Help would be appreciated.

syg00 · 08-17-2015, 07:47 PM

You are presuming the data has the format as the filename - or contains the (original) filename. Pretty big assumption. Lets see the first 20 lines - presuming common data format in the initial files.

Code:

head -n 20 bigfile.txt