We are using the combined log format, and each line has a destination and a referrer URL as per usual.

I need to go through the file and put all the requests for each specific domain, into specific files named after the domain.

So, if there are 2,000 requests to www.mysite.com, I want to pull all of those out and put them into www.mysite.com.log. If there are 4,000 requests to widgets.other.site.com, I want to pull all of those out and put them into widgets.other.site.com.log.

Ive currently got this psuedo-plan, to apply to each line of the MASSIVE log:

* grep for the first domain listed in each line
* write domain into a file (if not already exist in file)
* massage file for any weirdness

when thats all done parsing I should have a file with a list of domains, that I can put into a grep "for each" loop against the original MASSIVE file and output to a file named after the domain being searched for.

Sort that by time stamp and done.

Any suggestions on this? Pointers?

Is that something you need to do only once, or do you foresee a recurring need for this? Because in the latter case I would refrain from this dual-stage process that involves manual editing of the intermediate file.

What is the background of your need? On my sites I am using self-written logging scripts. (Basically a php script that I include on every page). It gives me a much more useful output because I can write out what I need, in the format that I need it to be, and cut all the stuff I don't need, which reduces log file sizes significantly

Herr Joe_2000!

Its a one time thing. 20G of data. I've already got it half-way there.

I'll post my history in case anyone ever has to do something like this again. I'd certainly love to see it done in php/python if anyone has the time!

use awk,

assuming field 4 to be www_somesite_com


you can get awk to do all the "massage file for any weirdness" before the print statement

We would need sample input and desired output to be of any more help to you.

Yes, agree, that's exactly what you should do. Although depending on how you want to sort the data (one file per domain vs. one file per page) it might become a bit more complex. But basically just recognize the domain from the current line and write to the corresponding log file in append mode, is what it comes down to...

yeah, well lets say $4 == "www.foo.bar/foo/page/files/file.html"

Edit probably don't need $0, just a habit I have