I need a shell script, my End goal for script is to filter all the API’s responding above 3 seconds.

Currently I am using this script, but it is not a streamlined and create multiple file.

#!/bin/bash

cat /data/httpd/apiv2.vt.com-access_log | awk '{ print $15 }' | cut -d \? -f1 | sort | uniq -c | sort -n | tr -d '**' | tr -d '/' > /tmp/apiv2.vt.com-access_log-dec.txt

cat /tmp/apiv2.vt.com-access_log-dec.txt | awk '$2 >= "3000000" { print $2}' > /tmp/apiv2.vt.com-access_log-dec2.txt

for i in `cat /tmp/apiv2.vt.com-access_log-dec2.txt `;do

grep "$i" /data/httpd/apiv2.vt.com-access_log >> /tmp/apiv2.vt.com-access_log-dec3.txt

done

Welcome to LQ

Please use [code][/code] tags around your code to keep formatting and make it easier to read

As you are predominantly using awk and grep to do the hard yards, I would suggest re-writing into a single awk script.

There is no real reason for all the temp files and using a for loop is the wrong process for reading a file, which of course is nullified if you do it all in awk anyway

1 members found this post helpful.

can you restructured above script as per your suggested way?
that will really helpful for me.

Just click on the 'edit' button and you'll be able to do it.

1 members found this post helpful.

There is no real reason for all the temp files and using a for loop is the wrong process for reading a file, which of course is nullified if you do it all in awk anyway


I am talking about this.

cat and cut are redundant with awk. But in general, since you are using bash, you can probably eliminate the temprory files and use process substitution instead.

1 members found this post helpful.

akk!, it is difficult to know what your script does because we don't see the log files, and what the desired result is.

you don't need to convert it all to awk (and nobody is likely to do it for you), but show us something we can work with.
code tags, working script, example input, example output, and desired output.

1 members found this post helpful.

please use [code]here comes your script[/code] tags around your code to keep formatting and make it easier to read
We have no idea what is this good for, because we don't know the format of the input file and don't know what do you want to collect. But this chain can be definitely implemented in one single awk, which will make it much faster and probably more readable too.
The second line looks like processing the result of the first line, probably you can pipe the two lines into one and no need to save that file at all. cat is not required, therefore you only need to add the awk to your pipe chain, which - again - can be simplified - and use altogether one awk script.

the for/cat/grep loop will work on the result and will filter the original file, but the mentioned awk can do this too, without this loop structure.
But again, without knowing what is in the log file (exactly) and what do you want to filter out (by example) will be hard to go further.
So please give us more details...

1 members found this post helpful.

I don't see the point of last loop

grep apiv2.vt.com-access_log some things in apiv2.vt.com-access_log-dec2.txt that come from apiv2.vt.com-access_log...

1 members found this post helpful.

You have been given several suggestions as well as have been asked for clarification about what your script does, including a request for sample I/O.

LQ members are not a help desk and are not here to provide you with code. The purpose of the LQ site is that we are here to help you learn by you completing your work with our suggestions.

Please provide the feedback and information that members have requested.This is very difficult to tell because your description of the problem is incomplete and vague. Please try to explain your problem more clearly. Once again, suggest you illustrate sample input and output to clarify what you're asking about.

Information about how to ask a better question.