parsing out squid access log with awk and grep
 

I'm trying to recreate a simple script I wrote to parse out the access.log to get a rough idea of websites that users are going to on our corp network. The issue I'm having is I want to pull out any line from access.log that ends in .com/ .org/ .net/ or whatever to only see what the user entered into the address bar and drop pictures, js's and everything else and log only this.

so what I do is :
awk '{print $8} | grep -e '[cong]|[ore]|[mgtv][/]'$ and nothing happens.

I know there is an easier way to do this with awk alone, . . . anyone?

Thx

Didn't work. :( It doesn't output anything.

Here's the line that is being spit out from the access.log after I awk out the 7th field:

When I run the Ruby code, it prints out nothing to std out. I'm not familiar with Ruby, I assume it is acting like awk(?)

if the lines you want ends in a slash, eg .com/, then add "\/" to the regex

The problem in the grep command
is that the -e option does not use extended regular expressions, so that the pipe symbol is interpreted literally. Maybe you want the -E (uppercase e) option. Indeed using awk you don't really need to pipe the results into grep:
which is similar to the ruby code suggested by kurumi! ;)

So I am confused on 2 fronts here (easily done some times):

1. You start by referring to field 8 but then in post #3 you talk about the 7th field?

2. You state the following:
But again in post #3 your output shows:
Maybe you could show some of the log so we can ascertain exactly which field you are referring to and then which part of that field are you interested in?

Guys, I appreciate all of the help! I'm sorry this has been a bit of a flustercluck from the beginning. I have solved it, and with your help! The access.log from squid looks like this:
for example and in this part of my script, I'm parsing out the url, which through test awk considers field 8, by my count it's 7, but that was my confusion.So what I am doing is trying to get a rough listing of websites that the users have entered into the address bar and drop everything else that squid logs, ie images, scripts, and redirects. Because this is a corporate network, I want to ensure that the user doesn't purposefully head to sites that aren't allowed at work, ie warez, pornography, etc. I don't fault anyone for being accidentally redirected, and yes this can be subverted if a user goes to google to search out porn or whatever, but this is an attempt of a bit of "loose" security. I do have Dansguardian installed and I havent decided to sit down and tweak the thing until its perfect, I imagine its a long and drawn out process, and I'd rather not deal with the multiple calls of "Hey I went here and I got this message". So this script pulls this stuff so I can then pull the data out by IP and logrotate the results using cron daily.

I appreciate the help guys! I've been avoiding regex, awk, and sed for a while now, only using it minimally, and unfortunatly, I get confused.

Thanks again!

Those results were "post awk, pre-grep" so After I used awk to print out the desired field from the log, I was to grep out only things that ended in webadress.com/ and drop all webaddress.com/picone.jpg and webaddress.com/scriptone.js. Squid logs not only the site the user is going to, but everything that gets loaded as well. Thank you!

Well there is no need to use grep if you are also using awk as it has most of the functionality built in as well.
colucix's example is the one I would go for, if using awk, although it will give you the entire field and your explanation still has me asking
if you want it all or just up until .com, .net, etc, ie. up until the first slash after http://.
If the above is desired, you could easily use split:
This is based on your example above.