LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices

Reply
 
Search this Thread
Old 09-02-2010, 08:47 PM   #1
koshihaku
Member
 
Registered: Aug 2010
Posts: 50

Rep: Reputation: 0
Question please give me some advice about my script


The following code is for monitoring the memory used by apache processes. But I got a problem that the data I got by this script is much larger than the physical memory. It was said that there are some libraries used simultaneously by many processes, so the data I got has some double counted part. Because apache has many httpd processes.

Anyone have an idea of getting the multi-processes memory used? Thanks a lot!


Code:
#!/bin/sh
#G.sh 20100813

USAGE="Usage: $0 processName"

if [ "$#" -ne 1 ]; then
  echo "$USAGE"
  exit 1
fi

# In case the monitored process has not yet started
# keep searching until its PID is found
PROCESS_PID=""
while :
do
  PROCESS_PID=`/bin/pidof "$1"`

  if [ "$PROCESS_PID.X" != ".X" ]; then
    break
  fi
done

LOG_FILE="memusage.csv"

echo "ElapsedTime,VmSize,VmRSS" > "$LOG_FILE"

ELAPSED_TIME=`date +%H:%M:%S:%N`
PERIOD=2        # seconds

while :
do
  TOTAL_VM_SIZE=0
  TOTAL_VM_RSS=0

  for PID in $PROCESS_PID; do
    if [ -d "/proc/$PID" ] ; then
      VM_SIZE=`awk '/VmSize/ {print $2}' < "/proc/$PID/status"`
      if [ "$VM_SIZE.X" = ".X" ]; then
        continue
      fi
      VM_RSS=`awk '/VmRSS/ {print $2}' < "/proc/$PID/status"`
      if [ "$VM_RSS.X" = ".X" ]; then
        continue
      fi
      let TOTAL_VM_SIZE+=VM_SIZE
      let TOTAL_VM_RSS+=VM_RSS
    else
      echo "$1 ($PID) is no longer a running process"
      exit 0
    fi
  done

  echo "$ELAPSED_TIME,$TOTAL_VM_SIZE,$TOTAL_VM_RSS" >> "$LOG_FILE"
  sleep "$PERIOD"
  ELAPSED_TIME=`date +%H:%M:%S:%N`
done
 
Old 09-02-2010, 11:57 PM   #2
kbp
Senior Member
 
Registered: Aug 2009
Posts: 3,758

Rep: Reputation: 643Reputation: 643Reputation: 643Reputation: 643Reputation: 643Reputation: 643
Maybe subtracting VmLib from each process will be more accurate...

cheers
 
Old 09-03-2010, 12:12 AM   #3
koshihaku
Member
 
Registered: Aug 2010
Posts: 50

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by kbp View Post
Maybe subtracting VmLib from each process will be more accurate...

cheers
but the VmLib is often larger than the VmRss e.g.
VmPeak: 67980 kB
VmSize: 67976 kB
VmLck: 0 kB
VmHWM: 5676 kB
VmRSS: 5672 kB
VmData: 4504 kB
VmStk: 84 kB
VmExe: 372 kB
VmLib: 15324 kB


And after I revising my script as the following
Code:
#!/bin/bash
#G.sh 20100813

USAGE="Usage: $0 processName"

if [ "$#" -ne 1 ]; then
  echo "$USAGE"
  exit 1
fi

# In case the monitored process has not yet started
# keep searching until its PID is found
PROCESS_PID=""
while :
do
  PROCESS_PID=`/bin/pidof "$1"`

  if [ "$PROCESS_PID.X" != ".X" ]; then
    break
  fi
done

LOG_FILE="memusage.csv"

echo "ElapsedTime,VmSize,VmRSS" > "$LOG_FILE"

ELAPSED_TIME=`date +%H:%M:%S:%N`
PERIOD=0.5      # seconds

while :
do
  TOTAL_VM_SIZE=0
  TOTAL_VM_RSS=0
  VmLib=0

  for PID in $PROCESS_PID; do
    if [ -d "/proc/$PID" ] ; then
      VM_SIZE=`awk '/VmSize/ {print $2}' < "/proc/$PID/status"`
      if [ "$VM_SIZE.X" = ".X" ]; then
        continue
      fi
      VM_RSS=`awk '/VmRSS/ {print $2}' < "/proc/$PID/status"`
      if [ "$VM_RSS.X" = ".X" ]; then
        continue
      fi
      VmLib=`awk '/VmLib/ {print $2}' < "/proc/$PID/status"`
      if [ "$VmLib.X" = ".X" ]; then
        continue
      fi
      let REAL_VM_SIZE=VM_SIZE-VmLib
      let TOTAL_VM_SIZE+=REAL_VM_SIZE
      let TOTAL_VM_RSS+=VM_RSS
    fi
  done

  echo "$ELAPSED_TIME,$TOTAL_VM_SIZE,$TOTAL_VM_RSS,$MemFree" >> "$LOG_FILE"
  sleep "$PERIOD"
  ELAPSED_TIME=`date +%H:%M:%S:%N`
done

I got the result : ElapsedTime VmSize VmRSS
14:08:14:487053684 31643556 3456272
14:08:28:750838950 31643556 3456272


but the physical memory is only 1G...
 
Old 09-03-2010, 02:03 AM   #4
kbp
Senior Member
 
Registered: Aug 2009
Posts: 3,758

Rep: Reputation: 643Reputation: 643Reputation: 643Reputation: 643Reputation: 643Reputation: 643
It seems what you really want is the sum of private RSS ... someone has already written a script to do what you are trying to do though if that helps .. http://www.pixelbeat.org/scripts/ps_mem.py

cheers
 
1 members found this post helpful.
Old 09-03-2010, 09:34 PM   #5
koshihaku
Member
 
Registered: Aug 2010
Posts: 50

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by kbp View Post
It seems what you really want is the sum of private RSS ... someone has already written a script to do what you are trying to do though if that helps .. http://www.pixelbeat.org/scripts/ps_mem.py

cheers
Thank you very much! I think it will help alot. I will try it now.
 
Old 09-03-2010, 10:04 PM   #6
koshihaku
Member
 
Registered: Aug 2010
Posts: 50

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by kbp View Post
It seems what you really want is the sum of private RSS ... someone has already written a script to do what you are trying to do though if that helps .. http://www.pixelbeat.org/scripts/ps_mem.py

cheers
It is good! But I need to record the output every several seconds in a log file. I don't know python. Is there a way to run it periodically can write the realtime output to file?
 
Old 09-03-2010, 10:21 PM   #7
kbp
Senior Member
 
Registered: Aug 2009
Posts: 3,758

Rep: Reputation: 643Reputation: 643Reputation: 643Reputation: 643Reputation: 643Reputation: 643
If you don't know python maybe you could wrap it in a shell script :

Code:
...
while :
do
  ./ps_mem.py httpd >> process.log
  sleep "$PERIOD"
done
...
 
1 members found this post helpful.
Old 09-03-2010, 11:21 PM   #8
koshihaku
Member
 
Registered: Aug 2010
Posts: 50

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by kbp View Post
If you don't know python maybe you could wrap it in a shell script :

Code:
...
while :
do
  ./ps_mem.py httpd >> process.log
  sleep "$PERIOD"
done
...
Thanks, I wraped the script into my shell script, I am trying it.

I think it will work, thanks a lot

Last edited by koshihaku; 09-10-2010 at 02:26 AM.
 
Old 09-03-2010, 11:29 PM   #9
koshihaku
Member
 
Registered: Aug 2010
Posts: 50

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by kbp View Post
If you don't know python maybe you could wrap it in a shell script :

Code:
...
while :
do
  ./ps_mem.py httpd >> process.log
  sleep "$PERIOD"
done
...


Oh, I > the output to output.csv

what is the difference between the .csv and .log ? It worked when I use .csv
 
Old 09-05-2010, 10:18 AM   #10
kbp
Senior Member
 
Registered: Aug 2009
Posts: 3,758

Rep: Reputation: 643Reputation: 643Reputation: 643Reputation: 643Reputation: 643Reputation: 643
Your script was writing output as csv, you may have to massage the data from this one if you want in it the same format
 
Old 09-09-2010, 02:00 AM   #11
koshihaku
Member
 
Registered: Aug 2010
Posts: 50

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by kbp View Post
Your script was writing output as csv, you may have to massage the data from this one if you want in it the same format
I run the script when benchmarking the server by httperf, but there is no changing of memory used in the output. Do you have any idea about this?
 
Old 09-09-2010, 09:22 AM   #12
kbp
Senior Member
 
Registered: Aug 2009
Posts: 3,758

Rep: Reputation: 643Reputation: 643Reputation: 643Reputation: 643Reputation: 643Reputation: 643
Maybe it's handling the load with the currently allocated resources .. ?
 
Old 09-09-2010, 11:28 PM   #13
koshihaku
Member
 
Registered: Aug 2010
Posts: 50

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by kbp View Post
Maybe it's handling the load with the currently allocated resources .. ?
Is there a way to run this script in a high priority? Because when the server is saturated, there is no output until the workload is low. I want the script run inspite of high workload.
 
Old 09-10-2010, 01:23 AM   #14
kbp
Senior Member
 
Registered: Aug 2009
Posts: 3,758

Rep: Reputation: 643Reputation: 643Reputation: 643Reputation: 643Reputation: 643Reputation: 643
'man nice'
 
1 members found this post helpful.
Old 09-10-2010, 01:26 AM   #15
koshihaku
Member
 
Registered: Aug 2010
Posts: 50

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by kbp View Post
'man nice'
I input:
# sudo bash processname &


and returned the pid of processname ,then

sudo nice -20 pid

Is there a way to get the pid of processname or bash without running with &?
I mean run the script and return the pid of it .
 
  


Reply

Tags
memory, rss, script


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Who can give me an advice? MondayGirl General 6 03-20-2007 02:27 PM
Please give advice ja21pu Linux - Hardware 2 07-21-2006 10:52 AM
Give me advice Getright for Linux xprometeus Linux - Software 3 11-13-2005 03:19 AM
Could someone Give Advice jandrews Linux - General 7 10-18-2003 11:16 PM
Please give your advice, tips, and comments. gsmonk Linux - Networking 1 04-05-2003 08:57 PM


All times are GMT -5. The time now is 07:24 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration