Need to read log file line to certain string then grep out 400

jaa1180 · 06-19-2013, 03:05 PM

On my web server I need to see the HTTP 400 requests. Or other bad requests.

Example: Log entry...

[18/Jun/2013:00:00:28 -0400] "GET /your-overthere/?/12072/order HTTP/1.1" 200 261335 "-" "Mozilla/5.0 (compatible; Blekkobot; ScoutJet; +http://blekko.com/about/blekkobot)"

How can I read up to the HTTP/1.1, then grep based off of the 200, 400, 404 or whatever is after.

I have tried:
egrep "400" access.log | sed 's/^.\{,45\}//' | grep "HTTP"

Of course it is still reading incorrectly.
Ideas?

Kustom42 · 06-19-2013, 03:54 PM

Ok well if I read your post right you are trying to grep for the "400" or other error codes and then get x amount of info before or after the result?

grep has this built in,

You can do a -A for lines after or -B for lines before. So egrep "400" /var/log/httpd/access.log -B 2 -A 2 would give you the two lines before and after the line it matches.

jaa1180 · 06-19-2013, 04:08 PM

Quote:

Originally Posted by Kustom42

Ok well if I read your post right you are trying to grep for the "400" or other error codes and then get x amount of info before or after the result?

grep has this built in,

You can do a -A for lines after or -B for lines before. So egrep "400" /var/log/httpd/access.log -B 2 -A 2 would give you the two lines before and after the line it matches.

Sort of. I am looking to grep or some other command to output only the lines with the 400 error or whatever error.
So if I could search for [HTTP/1.1" 400] and only show the lines in the log that match that, it would be great. I don't know how to get grep to do that.

Kustom42 · 06-19-2013, 04:31 PM

Quote:

Originally Posted by jaa1180

Sort of. I am looking to grep or some other command to output only the lines with the 400 error or whatever error.
So if I could search for [HTTP/1.1" 400] and only show the lines in the log that match that, it would be great. I don't know how to get grep to do that.

Ok so you are probably having issues with double/single quotes interpreting the backslash here.

If you use double quotes, ", / will be interpreted as a backslash and " would also be interpreted. Basically double quotes still allow for bash interpretation and expansion.

If you use single quotes, ', it will treat it as a literal string and ignore special characters. For example, if you did the following:

Code:

#!/bin/bash

MYVARIABLE="Hello World!"

echo "$MYVARIABLE" # This would print the string "Hello World!" to stdout.
echo '$MYVARIABLE' # Would print $MYVARIABLE

jaa1180 · 06-19-2013, 04:46 PM

Quote:

Originally Posted by Kustom42

Ok so you are probably having issues with double/single quotes interpreting the backslash here.

If you use double quotes, ", / will be interpreted as a backslash and " would also be interpreted. Basically double quotes still allow for bash interpretation and expansion.

If you use single quotes, ', it will treat it as a literal string and ignore special characters. For example, if you did the following:

Code:

#!/bin/bash

MYVARIABLE="Hello World!"

echo "$MYVARIABLE" # This would print the string "Hello World!" to stdout.
echo '$MYVARIABLE' # Would print $MYVARIABLE

Okay, forgot about the single quotes. I will see if that does the trick. Thank you sir!

Firerat · 06-19-2013, 04:47 PM

Quote:

Originally Posted by jaa1180

Sort of. I am looking to grep or some other command to output only the lines with the 400 error or whatever error.
So if I could search for [HTTP/1.1" 400] and only show the lines in the log that match that, it would be great. I don't know how to get grep to do that.

Code:

grep "HTTP/1\.1\" 4.." access.log

I'm sure I'm missing something

Kustom42 · 06-19-2013, 05:38 PM

Quote:

Originally Posted by Firerat

Code:

grep "HTTP/1\.1\" 4.." access.log

I'm sure I'm missing something

Although this will work it would be nice to explain why. Putting a backslash infront of a special character "escapes" the character. However, in this case there really is no need to do it and

Code:

grep "HTTP/1\.1\" 4.." access.log

and

Code:

grep 'HTTP/1.1" 4' access.log

Would accomplish the same thing.

jaa1180 · 06-19-2013, 05:43 PM

Cool, thanks Kustom42.

Firerat · 06-19-2013, 06:16 PM

true

and

Code:

grep 'HTTP/1\.1" 4[0-9][0-9] ' access.log

would be more exact, not that it makes much difference with this type of log

jaa1180 · 06-21-2013, 08:57 AM

Code:

grep 'HTTP/1.1" 404'

That worked. It is getting me the lines I need. Thanks all!!