LinuxQuestions.org
Page 1 of 2 1 2
Show 50 post(s) from this thread on one page

LinuxQuestions.org (/questions/)
-   Linux - General (https://www.linuxquestions.org/questions/linux-general-1/)
-   -   please give me some advice about my script (https://www.linuxquestions.org/questions/linux-general-1/please-give-me-some-advice-about-my-script-830060/)

koshihaku 09-02-2010 08:47 PM
please give me some advice about my script
 
The following code is for monitoring the memory used by apache processes. But I got a problem that the data I got by this script is much larger than the physical memory. It was said that there are some libraries used simultaneously by many processes, so the data I got has some double counted part. Because apache has many httpd processes.

Anyone have an idea of getting the multi-processes memory used? Thanks a lot!


Code:
#!/bin/sh
#G.sh 20100813

USAGE="Usage: $0 processName"

if [ "$#" -ne 1 ]; then
  echo "$USAGE"
  exit 1
fi

# In case the monitored process has not yet started
# keep searching until its PID is found
PROCESS_PID=""
while :
do
  PROCESS_PID=`/bin/pidof "$1"`

  if [ "$PROCESS_PID.X" != ".X" ]; then
    break
  fi
done

LOG_FILE="memusage.csv"

echo "ElapsedTime,VmSize,VmRSS" > "$LOG_FILE"

ELAPSED_TIME=`date +%H:%M:%S:%N`
PERIOD=2        # seconds

while :
do
  TOTAL_VM_SIZE=0
  TOTAL_VM_RSS=0

  for PID in $PROCESS_PID; do
    if [ -d "/proc/$PID" ] ; then
      VM_SIZE=`awk '/VmSize/ {print $2}' < "/proc/$PID/status"`
      if [ "$VM_SIZE.X" = ".X" ]; then
        continue
      fi
      VM_RSS=`awk '/VmRSS/ {print $2}' < "/proc/$PID/status"`
      if [ "$VM_RSS.X" = ".X" ]; then
        continue
      fi
      let TOTAL_VM_SIZE+=VM_SIZE
      let TOTAL_VM_RSS+=VM_RSS
    else
      echo "$1 ($PID) is no longer a running process"
      exit 0
    fi
  done

  echo "$ELAPSED_TIME,$TOTAL_VM_SIZE,$TOTAL_VM_RSS" >> "$LOG_FILE"
  sleep "$PERIOD"
  ELAPSED_TIME=`date +%H:%M:%S:%N`
done

kbp 09-02-2010 11:57 PM
Maybe subtracting VmLib from each process will be more accurate...

cheers

koshihaku 09-03-2010 12:12 AM
Quote:
Originally Posted by kbp (Post 4086678)
Maybe subtracting VmLib from each process will be more accurate...

cheers
but the VmLib is often larger than the VmRss e.g.
VmPeak: 67980 kB
VmSize: 67976 kB
VmLck: 0 kB
VmHWM: 5676 kB
VmRSS: 5672 kB
VmData: 4504 kB
VmStk: 84 kB
VmExe: 372 kB
VmLib: 15324 kB


And after I revising my script as the following
Code:
#!/bin/bash
#G.sh 20100813

USAGE="Usage: $0 processName"

if [ "$#" -ne 1 ]; then
  echo "$USAGE"
  exit 1
fi

# In case the monitored process has not yet started
# keep searching until its PID is found
PROCESS_PID=""
while :
do
  PROCESS_PID=`/bin/pidof "$1"`

  if [ "$PROCESS_PID.X" != ".X" ]; then
    break
  fi
done

LOG_FILE="memusage.csv"

echo "ElapsedTime,VmSize,VmRSS" > "$LOG_FILE"

ELAPSED_TIME=`date +%H:%M:%S:%N`
PERIOD=0.5      # seconds

while :
do
  TOTAL_VM_SIZE=0
  TOTAL_VM_RSS=0
  VmLib=0

  for PID in $PROCESS_PID; do
    if [ -d "/proc/$PID" ] ; then
      VM_SIZE=`awk '/VmSize/ {print $2}' < "/proc/$PID/status"`
      if [ "$VM_SIZE.X" = ".X" ]; then
        continue
      fi
      VM_RSS=`awk '/VmRSS/ {print $2}' < "/proc/$PID/status"`
      if [ "$VM_RSS.X" = ".X" ]; then
        continue
      fi
      VmLib=`awk '/VmLib/ {print $2}' < "/proc/$PID/status"`
      if [ "$VmLib.X" = ".X" ]; then
        continue
      fi
      let REAL_VM_SIZE=VM_SIZE-VmLib
      let TOTAL_VM_SIZE+=REAL_VM_SIZE
      let TOTAL_VM_RSS+=VM_RSS
    fi
  done

  echo "$ELAPSED_TIME,$TOTAL_VM_SIZE,$TOTAL_VM_RSS,$MemFree" >> "$LOG_FILE"
  sleep "$PERIOD"
  ELAPSED_TIME=`date +%H:%M:%S:%N`
done

I got the result : ElapsedTime VmSize VmRSS
14:08:14:487053684 31643556 3456272
14:08:28:750838950 31643556 3456272


but the physical memory is only 1G...

kbp 09-03-2010 02:03 AM
It seems what you really want is the sum of private RSS ... someone has already written a script to do what you are trying to do though if that helps .. http://www.pixelbeat.org/scripts/ps_mem.py

cheers

koshihaku 09-03-2010 09:34 PM
Quote:
Originally Posted by kbp (Post 4086763)
It seems what you really want is the sum of private RSS ... someone has already written a script to do what you are trying to do though if that helps .. http://www.pixelbeat.org/scripts/ps_mem.py

cheers
Thank you very much! I think it will help alot. I will try it now.

koshihaku 09-03-2010 10:04 PM
Quote:
Originally Posted by kbp (Post 4086763)
It seems what you really want is the sum of private RSS ... someone has already written a script to do what you are trying to do though if that helps .. http://www.pixelbeat.org/scripts/ps_mem.py

cheers
It is good! But I need to record the output every several seconds in a log file. I don't know python. Is there a way to run it periodically can write the realtime output to file?

kbp 09-03-2010 10:21 PM
If you don't know python maybe you could wrap it in a shell script :

Code:
...
while :
do
  ./ps_mem.py httpd >> process.log
  sleep "$PERIOD"
done
...

koshihaku 09-03-2010 11:21 PM
Quote:
Originally Posted by kbp (Post 4087572)
If you don't know python maybe you could wrap it in a shell script :

Code:
...
while :
do
  ./ps_mem.py httpd >> process.log
  sleep "$PERIOD"
done
...
Thanks, I wraped the script into my shell script, I am trying it.

I think it will work, thanks a lot

koshihaku 09-03-2010 11:29 PM
Quote:
Originally Posted by kbp (Post 4087572)
If you don't know python maybe you could wrap it in a shell script :

Code:
...
while :
do
  ./ps_mem.py httpd >> process.log
  sleep "$PERIOD"
done
...


Oh, I > the output to output.csv

what is the difference between the .csv and .log ? It worked when I use .csv

kbp 09-05-2010 10:18 AM
Your script was writing output as csv, you may have to massage the data from this one if you want in it the same format

koshihaku 09-09-2010 02:00 AM
Quote:
Originally Posted by kbp (Post 4088653)
Your script was writing output as csv, you may have to massage the data from this one if you want in it the same format
I run the script when benchmarking the server by httperf, but there is no changing of memory used in the output. Do you have any idea about this?

kbp 09-09-2010 09:22 AM
Maybe it's handling the load with the currently allocated resources .. ?

koshihaku 09-09-2010 11:28 PM
Quote:
Originally Posted by kbp (Post 4092511)
Maybe it's handling the load with the currently allocated resources .. ?
Is there a way to run this script in a high priority? Because when the server is saturated, there is no output until the workload is low. I want the script run inspite of high workload.

kbp 09-10-2010 01:23 AM
'man nice'

koshihaku 09-10-2010 01:26 AM
Quote:
Originally Posted by kbp (Post 4093137)
'man nice'
I input:
# sudo bash processname &


and returned the pid of processname ,then

sudo nice -20 pid

Is there a way to get the pid of processname or bash without running with &?
I mean run the script and return the pid of it .


All times are GMT -5. The time now is 12:36 AM.
Page 1 of 2 1 2
Show 50 post(s) from this thread on one page