LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 12-27-2017, 10:03 PM   #1
akk!
LQ Newbie
 
Registered: Dec 2017
Posts: 3

Rep: Reputation: Disabled
Need a streamlined shell script


I need a shell script, my End goal for script is to filter all the APIís responding above 3 seconds.

Currently I am using this script, but it is not a streamlined and create multiple file.

#!/bin/bash

cat /data/httpd/apiv2.vt.com-access_log | awk '{ print $15 }' | cut -d \? -f1 | sort | uniq -c | sort -n | tr -d '**' | tr -d '/' > /tmp/apiv2.vt.com-access_log-dec.txt

cat /tmp/apiv2.vt.com-access_log-dec.txt | awk '$2 >= "3000000" { print $2}' > /tmp/apiv2.vt.com-access_log-dec2.txt

for i in `cat /tmp/apiv2.vt.com-access_log-dec2.txt `;do

grep "$i" /data/httpd/apiv2.vt.com-access_log >> /tmp/apiv2.vt.com-access_log-dec3.txt

done
 
Old 12-28-2017, 12:07 AM   #2
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,767

Rep: Reputation: 3050Reputation: 3050Reputation: 3050Reputation: 3050Reputation: 3050Reputation: 3050Reputation: 3050Reputation: 3050Reputation: 3050Reputation: 3050Reputation: 3050
Welcome to LQ

Please use [code][/code] tags around your code to keep formatting and make it easier to read

As you are predominantly using awk and grep to do the hard yards, I would suggest re-writing into a single awk script.

There is no real reason for all the temp files and using a for loop is the wrong process for reading a file, which of course is nullified if you do it all in awk anyway
 
1 members found this post helpful.
Old 12-28-2017, 02:07 AM   #3
akk!
LQ Newbie
 
Registered: Dec 2017
Posts: 3

Original Poster
Rep: Reputation: Disabled
further action

can you restructured above script as per your suggested way?
that will really helpful for me.
 
Old 12-28-2017, 02:57 AM   #4
Turbocapitalist
Senior Member
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 4,425
Blog Entries: 3

Rep: Reputation: 2203Reputation: 2203Reputation: 2203Reputation: 2203Reputation: 2203Reputation: 2203Reputation: 2203Reputation: 2203Reputation: 2203Reputation: 2203Reputation: 2203
Just click on the 'edit' button and you'll be able to do it.
 
1 members found this post helpful.
Old 12-28-2017, 03:39 AM   #5
akk!
LQ Newbie
 
Registered: Dec 2017
Posts: 3

Original Poster
Rep: Reputation: Disabled
further action

There is no real reason for all the temp files and using a for loop is the wrong process for reading a file, which of course is nullified if you do it all in awk anyway


I am talking about this.
 
Old 12-28-2017, 03:45 AM   #6
Turbocapitalist
Senior Member
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 4,425
Blog Entries: 3

Rep: Reputation: 2203Reputation: 2203Reputation: 2203Reputation: 2203Reputation: 2203Reputation: 2203Reputation: 2203Reputation: 2203Reputation: 2203Reputation: 2203Reputation: 2203
cat and cut are redundant with awk. But in general, since you are using bash, you can probably eliminate the temprory files and use process substitution instead.
 
1 members found this post helpful.
Old 12-28-2017, 04:42 AM   #7
ondoho
LQ Addict
 
Registered: Dec 2013
Posts: 13,390
Blog Entries: 9

Rep: Reputation: 3652Reputation: 3652Reputation: 3652Reputation: 3652Reputation: 3652Reputation: 3652Reputation: 3652Reputation: 3652Reputation: 3652Reputation: 3652Reputation: 3652
akk!, it is difficult to know what your script does because we don't see the log files, and what the desired result is.

you don't need to convert it all to awk (and nobody is likely to do it for you), but show us something we can work with.
code tags, working script, example input, example output, and desired output.
 
1 members found this post helpful.
Old 12-28-2017, 06:18 AM   #8
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 13,576

Rep: Reputation: 4341Reputation: 4341Reputation: 4341Reputation: 4341Reputation: 4341Reputation: 4341Reputation: 4341Reputation: 4341Reputation: 4341Reputation: 4341Reputation: 4341
please use [code]here comes your script[/code] tags around your code to keep formatting and make it easier to read
Code:
cat /data/httpd/apiv2.vt.com-access_log | awk '{ print $15 }' | cut -d \? -f1 | sort | uniq -c | sort -n | tr -d '**' | tr -d '/'
We have no idea what is this good for, because we don't know the format of the input file and don't know what do you want to collect. But this chain can be definitely implemented in one single awk, which will make it much faster and probably more readable too.
Code:
cat /tmp/apiv2.vt.com-access_log-dec.txt | awk '$2 >= "3000000" { print $2}' > /tmp/apiv2.vt.com-access_log-dec2.txt
The second line looks like processing the result of the first line, probably you can pipe the two lines into one and no need to save that file at all. cat is not required, therefore you only need to add the awk to your pipe chain, which - again - can be simplified - and use altogether one awk script.

the for/cat/grep loop will work on the result and will filter the original file, but the mentioned awk can do this too, without this loop structure.
But again, without knowing what is in the log file (exactly) and what do you want to filter out (by example) will be hard to go further.
So please give us more details...
 
1 members found this post helpful.
Old 12-28-2017, 06:25 AM   #9
keefaz
LQ Guru
 
Registered: Mar 2004
Distribution: Slackware
Posts: 6,230

Rep: Reputation: 724Reputation: 724Reputation: 724Reputation: 724Reputation: 724Reputation: 724Reputation: 724
I don't see the point of last loop

grep apiv2.vt.com-access_log some things in apiv2.vt.com-access_log-dec2.txt that come from apiv2.vt.com-access_log...
 
1 members found this post helpful.
Old 12-28-2017, 09:40 AM   #10
rtmistler
Moderator
 
Registered: Mar 2011
Location: USA
Distribution: MINT Debian, Angstrom, SUSE, Ubuntu, Debian
Posts: 8,416
Blog Entries: 13

Rep: Reputation: 3736Reputation: 3736Reputation: 3736Reputation: 3736Reputation: 3736Reputation: 3736Reputation: 3736Reputation: 3736Reputation: 3736Reputation: 3736Reputation: 3736
Quote:
Originally Posted by akk! View Post
can you restructured above script as per your suggested way?
that will really helpful for me.
You have been given several suggestions as well as have been asked for clarification about what your script does, including a request for sample I/O.

LQ members are not a help desk and are not here to provide you with code. The purpose of the LQ site is that we are here to help you learn by you completing your work with our suggestions.

Please provide the feedback and information that members have requested.
Quote:
Originally Posted by akk! View Post
There is no real reason for all the temp files and using a for loop is the wrong process for reading a file, which of course is nullified if you do it all in awk anyway


I am talking about this.
This is very difficult to tell because your description of the problem is incomplete and vague. Please try to explain your problem more clearly. Once again, suggest you illustrate sample input and output to clarify what you're asking about.

Information about how to ask a better question.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Shell script for run an shell script on server using ssh bloodstreetboy Linux - Server 5 01-12-2013 04:23 AM
How to pass command line arguments from one shell script to another shell script VijayaRaghavanLakshman Linux - Newbie 5 01-20-2012 10:12 PM
Executing a Shell script with 654 permissions inside another shell script. changusee2k Linux - Newbie 2 06-07-2011 08:58 PM
LXer: Streamlined Perl Number Matching Script For Unix Or Linux LXer Syndicated Linux News 0 09-18-2008 01:00 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 06:31 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration