LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 08-09-2017, 09:33 AM   #1
sakusri16
LQ Newbie
 
Registered: Aug 2017
Posts: 13

Rep: Reputation: Disabled
Wink Need a script to filter high cpu utilization process value in log file


We have created a script to find top 5 cpu usage process details and we stored that details in particular log file .we have added that script into Cron and scheduled every 5 mins . After one week it has lot of details .its take very long time to check above 50% process details in that particular file

Example log file lines :

www-data 2263 9.8 1.5 1069596 128300 ? S 09:37 0:02 php-fpm: pool www
www-data 2282 3.2 0.9 985808 74116 ? S 09:37 0:00 php-fpm: pool www

Bolded value is CPU usage :

We need a script to filter above 60% usage process and that script need to check every lines in that log file .Please help it will save our lot of time. thanks in advance
 
Old 08-09-2017, 09:41 AM   #2
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,690
Blog Entries: 3

Rep: Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920
Welcome.

awk can do that kind of comparison. If white space is the delimiter between data fields then you don't have to set it with -F or by changing the FS variable. The individual fields are available as $1, $2, $3, and so on. In this case it looks like you want to compare the third field, so here's something to get you started:

Code:
awk '$3 > 6.0' /var/log/some.data.log

awk '$3 > 6.0' /var/log/some.data.log | more
See:

And of course, the authoritative reference for the version of awk on your system will be found in the manual page:

Code:
man awk
It's a pretty easy programming language.
 
2 members found this post helpful.
Old 08-09-2017, 09:57 AM   #3
rtmistler
Moderator
 
Registered: Mar 2011
Location: USA
Distribution: MINT Debian, Angstrom, SUSE, Ubuntu, Debian
Posts: 9,357
Blog Entries: 13

Rep: Reputation: 4411Reputation: 4411Reputation: 4411Reputation: 4411Reputation: 4411Reputation: 4411Reputation: 4411Reputation: 4411Reputation: 4411Reputation: 4411Reputation: 4411
Quote:
Originally Posted by sakusri16 View Post
We have created a script to find top 5 cpu usage process details and we stored that details in particular log file .we have added that script into Cron and scheduled every 5 mins . After one week it has lot of details .its take very long time to check above 50% process details in that particular file

Example log file lines :

www-data 2263 9.8 1.5 1069596 128300 ? S 09:37 0:02 php-fpm: pool www
www-data 2282 3.2 0.9 985808 74116 ? S 09:37 0:00 php-fpm: pool www

Bolded value is CPU usage :

We need a script to filter above 60% usage process and that script need to check every lines in that log file .Please help it will save our lot of time. thanks in advance
You've written a script, and now you wish someone to write a script for you to do something you haven't yet attempted on your own.

Please make an attempt.

I fully agree with Turbocapitalist's suggestion about awk and what it can do for you. I feel you should put forth an attempt. The few reasons are (1) that you should versus ask others to do for you, and (2) you best can assess the flow of your requirements as you generate output and determine how you wish to filter it and how you wish to format this output.

Once you have something, please then share a clip of sample data as well as your updated script and people can then evaluate and help you to optimize what you have started with.
 
1 members found this post helpful.
Old 08-09-2017, 10:20 AM   #4
sakusri16
LQ Newbie
 
Registered: Aug 2017
Posts: 13

Original Poster
Rep: Reputation: Disabled
Smile Thank you so much

Thank you so much its working as expected @Turbocapitalist

that log file have date and time for above , i forgot to update , we have to print that details also

Example :

Sat Jul 15 09:38:10 IST 2017

www-data 2263 9.8 1.5 1069596 128300 ? S 09:37 0:02 php-fpm: pool www
www-data 2282 3.2 0.9 985808 74116 ? S 09:37 0:00 php-fpm: pool www


@rtmistler ->Thank you for your suggestions ,we have tried to create a script but we can't . we are new to linux scripting .we will follow in future requests. Thanks
 
Old 08-09-2017, 10:25 AM   #5
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,690
Blog Entries: 3

Rep: Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920
The scripting is the same for nearly all the recent operating systems, even OS X, not just GNU/Linux. So you will find it useful. However, nothing from Windows is relevant, it is the odd one out.

Quote:
Originally Posted by sakusri16 View Post
that log file have date and time for above , i forgot to update , we have to print that details also
Ok. We can guide you through that and help you find a way to get awk to print what you need. The first questions is, where are you getting that date + time from?
 
1 members found this post helpful.
Old 08-09-2017, 10:28 AM   #6
rtmistler
Moderator
 
Registered: Mar 2011
Location: USA
Distribution: MINT Debian, Angstrom, SUSE, Ubuntu, Debian
Posts: 9,357
Blog Entries: 13

Rep: Reputation: 4411Reputation: 4411Reputation: 4411Reputation: 4411Reputation: 4411Reputation: 4411Reputation: 4411Reputation: 4411Reputation: 4411Reputation: 4411Reputation: 4411
Quote:
Originally Posted by sakusri16 View Post
Thank you so much its working as expected @Turbocapitalist
Excellent progress. Glad to hear that this got you further.
Quote:
Originally Posted by sakusri16 View Post
@rtmistler ->Thank you for your suggestions ,we have tried to create a script but we can't . we are new to linux scripting .we will follow in future requests. Thanks
This is exactly why we ask to see code from posters and ask them to make their attempts. For us to understand your expertise with things.

If it is bash scripting you are doing, there are many references to help you get started, continue at your own pace, and also work though problems with scripts. Here are some references to consider:

Bash Beginner's Guide
Advance Bash Scripting Guide
My Bash Blog on Debugging Scripts

However if you are scripting with another language, then you may need different references.
 
1 members found this post helpful.
Old 08-09-2017, 10:43 AM   #7
sakusri16
LQ Newbie
 
Registered: Aug 2017
Posts: 13

Original Poster
Rep: Reputation: Disabled
Thanks for the reply

That date and time is Cron job running date and time

we have write a script with " while true; do date >>somedata.log; etc ..."

Thanks @rtmistler
 
Old 08-09-2017, 10:52 AM   #8
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,690
Blog Entries: 3

Rep: Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920
Quote:
Originally Posted by sakusri16 View Post
we have write a script with " while true; do date >>somedata.log; etc ..."
Please show a few lines from your log before and after such a date stamp. How familiar are you with regular expression (aka regex) pattern matching?
 
1 members found this post helpful.
Old 08-09-2017, 11:31 AM   #9
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 23,954

Rep: Reputation: 7029Reputation: 7029Reputation: 7029Reputation: 7029Reputation: 7029Reputation: 7029Reputation: 7029Reputation: 7029Reputation: 7029Reputation: 7029Reputation: 7029
Quote:
Originally Posted by sakusri16 View Post
Thank you so much its working as expected @Turbocapitalist that log file have date and time for above , i forgot to update , we have to print that details also

Example :
Sat Jul 15 09:38:10 IST 2017
www-data 2263 9.8 1.5 1069596 128300 ? S 09:37 0:02 php-fpm: pool www
www-data 2282 3.2 0.9 985808 74116 ? S 09:37 0:00 php-fpm: pool www

@rtmistler ->Thank you for your suggestions ,we have tried to create a script but we can't . we are new to linux scripting .we will follow in future requests. Thanks
Very confused here; from your first post, you said:
Quote:
Originally Posted by sakusri16
We have created a script....
...that indicates that you HAVE created a script. But you can't create a second script? rtmistler pointed you towards the bash scripting tutorials, which are very detailed and include examples. And since you've created a script before, you should have some experience doing so. You need to at least show your own efforts, and show us the first script you wrote, and what attempts you've made at the second.

We are always going to be happy to help, but you need to show your own efforts and tell us where you're stuck first.
 
Old 08-09-2017, 11:43 AM   #10
sakusri16
LQ Newbie
 
Registered: Aug 2017
Posts: 13

Original Poster
Rep: Reputation: Disabled
Hi


Sat Jul 15 09:49:26 IST 2017
www-data 2773 6.0 1.8 1083404 149224 ? S 09:45 0:14 php-fpm: pool www
www-data 2697 5.1 1.9 1618592 162940 ? S 09:44 0:16 php-fpm: pool www
www-data 2623 4.6 1.6 1613324 130784 ? S 09:43 0:18 php-fpm: pool www
www-data 2866 4.3 1.8 1545412 153800 ? S 09:46 0:08 php-fpm: pool www

Sat Jul 15 12:17:34 IST 2017
www-data 8984 6.5 1.3 1009988 109804 ? S 12:17 0:02 php-fpm: pool www
www-data 9020 4.7 1.1 983736 94324 ? S 12:17 0:00 php-fpm: pool www
www-data 9019 3.3 0.8 912588 66956 ? S 12:17 0:00 php-fpm: pool www
www-data 9021 1.3 0.8 912484 65808 ? S 12:17 0:00 php-fpm: pool www


I m not familar with regular expressions pattern

Thanks
Srini
 
Old 08-09-2017, 11:47 AM   #11
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,690
Blog Entries: 3

Rep: Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920
Thanks. Then I would recommend taking a quick look at the manual page:

Code:
man 7 regex
and then move on to a tutorial :

http://www.regular-expressions.info/quickstart.html

One pattern that might work would be any line that is 29 characters long and starting with a letter. How far can you get adding that to your exising awk script?

Last edited by Turbocapitalist; 08-09-2017 at 11:51 AM.
 
Old 08-09-2017, 12:36 PM   #12
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 23,954

Rep: Reputation: 7029Reputation: 7029Reputation: 7029Reputation: 7029Reputation: 7029Reputation: 7029Reputation: 7029Reputation: 7029Reputation: 7029Reputation: 7029Reputation: 7029
Quote:
Originally Posted by sakusri16 View Post
Hi
Sat Jul 15 09:49:26 IST 2017
www-data 2773 6.0 1.8 1083404 149224 ? S 09:45 0:14 php-fpm: pool www
www-data 2697 5.1 1.9 1618592 162940 ? S 09:44 0:16 php-fpm: pool www
www-data 2623 4.6 1.6 1613324 130784 ? S 09:43 0:18 php-fpm: pool www
www-data 2866 4.3 1.8 1545412 153800 ? S 09:46 0:08 php-fpm: pool www

Sat Jul 15 12:17:34 IST 2017
www-data 8984 6.5 1.3 1009988 109804 ? S 12:17 0:02 php-fpm: pool www
www-data 9020 4.7 1.1 983736 94324 ? S 12:17 0:00 php-fpm: pool www
www-data 9019 3.3 0.8 912588 66956 ? S 12:17 0:00 php-fpm: pool www
www-data 9021 1.3 0.8 912484 65808 ? S 12:17 0:00 php-fpm: pool www

I m not familar with regular expressions pattern
Again, can you post what YOU have done/tried/written to even attempt to do this?? We are happy to help, but you've not posted anything so far that shows any effort on your part. Again, can you post the first script you wrote that generates this log file?? Because just appending the date as you mentioned seems very inefficient, if it's part of what you want reported on. You could probably output the date first, followed by the rest of the data, leaving you one line for each record, each with a date like
Code:
Sat Jul 15 12:17:34 IST 2017 www-data  8984  6.5  1.3 1009988 109804 ?      S    12:17   0:02 php-fpm: pool www
...which would be far easier to parse/work with.

So again: post your script and show us your efforts.
 
Old 08-11-2017, 05:48 AM   #13
sakusri16
LQ Newbie
 
Registered: Aug 2017
Posts: 13

Original Poster
Rep: Reputation: Disabled
Original script : while true; do date >> somedata.log; ps aux | sort -nrk 3,3 | head -n 5 >> somedata.log; sleep 300; done

After executing above script we are getting below output .

Sat Jul 15 09:49:26 IST 2017
www-data 2773 6.0 1.8 1083404 149224 ? S 09:45 0:14 php-fpm: pool www
www-data 2697 5.1 1.9 1618592 162940 ? S 09:44 0:16 php-fpm: pool www
www-data 2623 4.6 1.6 1613324 130784 ? S 09:43 0:18 php-fpm: pool www
www-data 2866 4.3 1.8 1545412 153800 ? S 09:46 0:08 php-fpm: pool www


We need output like below

Sat Jul 15 12:17:34 IST 2017 www-data 8984 6.5 1.3 1009988 109804 ? S 12:17 0:02 php-fpm: pool www

We have tried to modify the script as below but we are not getting expected result . Please check and correct my mistake .

while true; do date| ps aux | sort -nrk 3,3 | head -n 5 >> somedata.log; sleep 300; done


Thanks
Srini
 
Old 08-11-2017, 06:15 AM   #14
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,690
Blog Entries: 3

Rep: Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920
This part is ok:

Code:
ps aux | sort -k 3,3nr | head -n 5
Remember that ps does not process any input so sending it input via a pipe will be of no use.

If you want to prepend a date onto each line, you'd have to use sed or awk. There are several ways to do either. If things remain simple then perhaps command subsitution is one option:

Code:
ps aux | sort -k 3,3nr | head -n 5 | sed -e "s/^/$(date +'%F %T %Z ')/"
sed is using the substitution command there s/// It looks for the beginnig of the line ^ and replaces it with the result of $(...) In effect that is prepending the output. For sed see:

Code:
man sed
man 7 regex
 
Old 08-11-2017, 06:18 AM   #15
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,690
Blog Entries: 3

Rep: Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920
Another way would be to store the date in a variable and let the shell interpret that variable:

Code:
A=$(date +'%F %T %Z'); ps aux | sort -k 3,3nr | head -n 5 | sed -e "s/^/$A /"
The trick to that is how double quotes allow the shell to interpret before it passes data onward to the program being called.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
High Run queue utilization although CPU utilization is low.. rajeprag Linux - Server 0 08-18-2013 09:44 AM
Need the script to calculate the CPU,process utilization mahbabu999 Linux - Server 3 01-18-2013 07:51 AM
monitor CPU and RAM High utilization through script azheruddin Linux - Newbie 1 12-09-2012 10:00 AM
[SOLVED] defunct bash process has high cpu utilization aedurkee Linux - Server 4 09-22-2012 07:11 AM
High CPU utilization on only one CPU out of 4 CPUs makam.sreekanth Linux - Kernel 2 09-03-2009 04:12 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 01:39 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration