Shell grep issue
List,
I have an ever-growing firewall logfile on my FreeBSD, and I want to perform some actions on the events of the last 15 minutes. Here is a sample: May _1 22:19:55 10.11.12.13 May 01 2004 20:20:29: %PIX-4-106023: Deny udp src inside:10.161.6.63/123 dst outside:131.107.1.10/123 May _1 22:19:55 10.11.12.13 May 01 2004 20:20:29: %PIX-4-106023: Deny udp src inside:10.130.71.149/1814 dst outside:193.100.2.1/22200 May _1 22:19:55 10.11.12.13 May 01 2004 20:20:29: %PIX-3-106011: Deny inbound (No xlate) icmp src inside:10.31.255.34 dst inside:10.134.9.130 May _1 22:19:55 10.11.12.13 May 01 2004 20:20:29: %PIX-4-106023: Deny tcp src inside:10.150.143.62/3719 dst outside:206.173.193.10/80 May _1 22:19:55 10.11.12.13 May 01 2004 20:20:29: %PIX-4-106023: Deny udp src inside:10.140.155.27/1652 dst outside:192.43.244.18/123 Is there a way that I can make my box only show the last 15 minutes? I tried various ways, but I think I'm making it overly complicated :( Any suggestions? Thx, Phil |
A lot depends on what you're doing with the data from the last 15 minutes.
A simple way to extract the last bits from a file (not necessarily the last 15 minutes but...) is to use tail. You can ask for the last n lines (where n is the number of lines to select), you can 'follow' the output of a log file, etc... Check the fine man page or info for details. EX: tail /var/log.boot.log You can select as many lines as you might think will get the job done. Trying to get lines that match the last 15 minutes would be work, and it's do-able, but not off the top of my head. I'd also need mroe detail as to the specifics of wht you're doing. |
You can grep the log file for the time about 15 minutes ago, probably leaving off the seconds. Use grep -m 1 -n to get the line number of the first hit. Then parse out the line number, subtract that from the total wc of the file and do a tail on that number.
|
I have been thinking about tailing # number of lines, but that is practically not doable; sometimes this spans 15 minutes, sometimes 2 hours. I would need it more accurate than 'last 10.000 lines' or so.
WC -l on the total file is not an option either since that file is 10+ gigs and if I'd have to grep AND wc the file, that would double the execution time. Isn't there a way to compare 2 dates? |
Not really. You usually have to write code.
Here is some C that kind of does what you want. We use it for file times, so you'll have to play with it - Code:
/* t_diff */ |
wow. thats the longest peice of code i've ever seen posted lol!! really, i'll bet that awk has an answer for you. i just started it in school, but if FreeBSD has awk (i'm new so i only assume it does), then awk would be a much better bet. we just learned in class today about how to do operations on fields, and i can give you a conceptual answer in awk, and i do hope this helps somewhat. i have not had an opportunity to try this myself just yet and i am by no means experienced with awk. but here goes. i know you can get the system date, and i know awk can parse fields within file. it can even parse fields within fields. so like if i had
May _1 22:19:55 10.11.12.13 May 01 2004 20:20:29: in a file, i could take out just the minute portion of the system date, subract 15 and save it. then i could compare it with all the minute feilds and echo all the ones less than 15 to wherever. ill bet that a much much much simpler solution lies in awk. i'll put up a link to an online book about awk, and maybe someone more experienced can offer such a solution. awk book sorry i couldnt be of more assistance. |
jim:
I didn't have time to look into the long code yet, doesn't seem to be the most convenient solution. But thanks for the suggestion, if I run out of other options/suggestions I will have a look :) sphynx: FreeBSD indeed has several AWK flavors, I tried to play with it as well. The problem with your suggestion is that my logfile spans multiple days. So I need to grep on the current date first, then on time and so on. Hmmm, might need to play a bit longer to find something useful... |
Hi,
My solution is a simple shell script (for now), which can be optimized later, I suppose. It's very slow (as you might imagine). But, it works. Also, I didn't understand the logfile line. For example, this line: May _1 22:19:55 10.11.12.13 May 01 2004 20:20:29: %PIX-4-106023: Deny udp src inside:10.161.6.63/123 dst outside:131.107.1.10/123 I don't know what the fields: May _1 22:19:55 10.11.12.13 are. So, I've just ignored them. I'm using fields 5,6,7 and 8. Code:
#!/bin/ksh system), which could take, er, a little while to parse a 10GB file. So, here's the C version of it. It runs a little faster. It takes about a second to parse a file with 250000 lines and about 6 seconds to parse a 26MB file with about a million lines. Should be fast enough. Beware, it does not have a LOT of checks which should be added before using it. For example, what happens when the line does not conform to the logfile input? Also, I've (being lazy) used a 10000 character array. You should use malloc or something similar if you're going to be using this in production. Here it is: Code:
#include <stdio.h> |
i was pondering an awk script to do this as i mentioned before. i have had a little time to think about it, and when finals are over i will give it a serious go. it's an interesting problem at the very least, and will give me some experience with awk. i'll let you konw if/what i come up with. we did stuff that is conceptually identical to what you want in lecture today, but unfortunatly i have so much due, so quickly i won't have time to sit down and give it serious thought for a little while.
|
All times are GMT -5. The time now is 12:25 PM. |