LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 11-23-2015, 06:08 PM   #1
macdonjo
LQ Newbie
 
Registered: Nov 2015
Distribution: Cent OS
Posts: 4

Rep: Reputation: Disabled
How to compare memory usage across Python scripts?


I have 7 Python scripts averaging 300 lines. They are crawlers that run infinitely.

"top" is said to not represent the numbers truthfully: http://serverfault.com/questions/736...g-memory-usage

How can I see which one is the most resource intensive so I can work on optimizing it?
 
Old 11-23-2015, 07:08 PM   #2
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 19,096

Rep: Reputation: 3325Reputation: 3325Reputation: 3325Reputation: 3325Reputation: 3325Reputation: 3325Reputation: 3325Reputation: 3325Reputation: 3325Reputation: 3325Reputation: 3325
Many things are said about Linux memory usage - most of them wrong, or hard to interpret correctly.
Go find ps_mem.py and use that. Being python you can read it directly - it has some excellent comments for background.
 
Old 11-24-2015, 07:06 AM   #3
macdonjo
LQ Newbie
 
Registered: Nov 2015
Distribution: Cent OS
Posts: 4

Original Poster
Rep: Reputation: Disabled
Does the program need to terminate to make that work? As I said, it runs infinitely.
 
Old 11-24-2015, 07:14 AM   #4
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 15,085

Rep: Reputation: 4957Reputation: 4957Reputation: 4957Reputation: 4957Reputation: 4957Reputation: 4957Reputation: 4957Reputation: 4957Reputation: 4957Reputation: 4957Reputation: 4957
https://docs.python.org/2/library/profile.html
http://stackoverflow.com/questions/5...-python-script

You may try to switch on/off (start/stop) profiler somehow and allow to run your process infinitely.
 
Old 11-24-2015, 07:17 AM   #5
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 19,096

Rep: Reputation: 3325Reputation: 3325Reputation: 3325Reputation: 3325Reputation: 3325Reputation: 3325Reputation: 3325Reputation: 3325Reputation: 3325Reputation: 3325Reputation: 3325
No, it's a snapshot. Once a task terminates, all the allocation data is lost - unless you have auditing and/or history monitor in place. The latter is a better long term option - collectl say.
Run ps_mem with a interval (20 secs, 5 mins, 30 mins, whatever) and send the output to a file. Easy to see any increase for a particular task.
 
Old 11-24-2015, 07:19 AM   #6
macdonjo
LQ Newbie
 
Registered: Nov 2015
Distribution: Cent OS
Posts: 4

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by syg00 View Post
No, it's a snapshot. Once a task terminates, all the allocation data is lost - unless you have auditing and/or history monitor in place. The latter is a better long term option - collectl say.
Run ps_mem with a interval (20 secs, 5 mins, 30 mins, whatever) and send the output to a file. Easy to see any increase for a particular task.
This is awesome! Thank-you!
 
Old 11-24-2015, 07:29 AM   #7
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 19,096

Rep: Reputation: 3325Reputation: 3325Reputation: 3325Reputation: 3325Reputation: 3325Reputation: 3325Reputation: 3325Reputation: 3325Reputation: 3325Reputation: 3325Reputation: 3325
You can pass in a list of pids, so you don't have a lot of unnecessary entries if you wish. The total line is also just for those tasks.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Shell script for CPU usage, memory usage, disk partition space and service status reetesh.amity Linux - Server 6 10-12-2015 07:51 PM
LXer: Python Scripts as a Replacement for Bash Utility Scripts LXer Syndicated Linux News 1 01-17-2013 08:08 AM
X hangs after prolonged usage ... memory usage goes upto 195MB duryodhan Slackware 18 05-10-2007 06:16 PM
getting realtime info on memory usage-cpu and harddrive usage steering Linux - Newbie 5 03-03-2005 08:43 PM
how to determine cpu usage, memory usage, I/O usage by a particular user logged on li rags2k Programming 4 08-21-2004 04:45 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 09:25 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration