[SOLVED] Memory profiling: a getlimits.py issue

budrz89 · 12-29-2021, 11:12 AM

Hi,

I wasn't sure which subforum to go to but I am using Artix linux, and I installed the latest updates today. My problem is related to profiling my Python code using the "memory_profiler" module. I'd place some @profile decorators around the functions that I'm interested in and whenever I run

Code:

python -m memory_profiler <my program>.py

I get this error:

Code:

Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/numpy/core/getlimits.py", line 516, in __init__
    self.dtype = numeric.dtype(int_type)
TypeError: 'numpy.dtype[bool_]' object is not callable

The error disappears when I remove the @profile and execute it normally. But I want to use it (or something like it) so I can get some information on what line in my code is consuming a lot of memory. I've looked around to see what are some good memory profilers and I found these that seem to match what I'm looking for:

https://stackoverflow.com/questions/...is-recommended

https://smira.ru/wp-content/uploads/2011/08/heapy.html

I've tried the guppy module and it seems nice (especially after going through the tutorial in the above link) but my head spins when I use it on my code and I don't know what I'm looking at (pointers to memory, for example, instead of a specific function). Then again, I am a noob when it comes to using it.

Anyway, I'm hoping anyone can help me figure out why I'm getting that error or maybe even offer some other ways in profiling my code for finding memory consumption.

teckk · 12-30-2021, 07:42 AM

Another code tree that isn't straight forward to build. I don't have numpy or guppy installed.

https://github.com/conda-forge/memor...iler-feedstock
https://conda-forge.org/docs/
https://github.com/conda-forge/memor...ild-locally.py

Here are archives if anyone wants to try and build it, and test it.
https://codeload.github.com/conda-fo...s/heads/master

https://pypi.io/packages/source/m/me...-0.60.0.tar.gz
sha256: 6a12869511d6cebcb29b71ba26985675a58e16e06b3c523b49f67c5497a33d1c

budrz89 · 12-30-2021, 08:24 AM

I'll check it out.

budrz89 · 01-01-2022, 11:12 AM

Sorry but what is the proper way to test memory_profiler-0.60.0?

I had to patch a few test files to include "from memory_profiler import profile" as it was needed in some of them. After patching them, I ran

Code:

pytest

and I get

Code:

platform linux -- Python 3.10.1, pytest-6.2.5, py-1.11.0, pluggy-0.13.1
rootdir: /home/budrz89/build/python-memory-profiler-feedstock/memory_profiler-0.60.0
plugins: datafixtures-1.0.0, mock-3.6.1
collected 29 items                                                             

test/test_attributes.py E                                                [  3%]
test/test_exit_code.py ...                                               [ 13%]
test/test_func.py E                                                      [ 17%]
test/test_gen.py ..                                                      [ 24%]
test/test_increment_display.py ...F                                      [ 37%]
test/test_loop.py ..                                                     [ 44%]
test/test_loop_decorated.py ..                                           [ 51%]
test/test_memory_usage.py ...                                            [ 62%]
test/test_mprof.py .                                                     [ 65%]
test/test_mprofile.py EEE                                                [ 75%]
test/test_nested.py .                                                    [ 79%]
test/test_precision_command_line.py ..                                   [ 86%]
test/test_precision_import.py ..                                         [ 93%]
test/test_stream_unicode.py E                                            [ 96%]
test/test_tracemalloc.py E                                               [100%]

==================================== ERRORS ====================================
_____________________ ERROR at setup of test_with_profile ______________________
file /home/budrz89/build/python-memory-profiler-feedstock/memory_profiler-0.60.0/test/test_attributes.py, line 4
  @profile
  def test_with_profile(arg1):
E       fixture 'arg1' not found
>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, class_mocker, datafix, datafix_dir, datafix_read, datafix_readbin, doctest_namespace, mocker, module_mocker, monkeypatch, package_mocker, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, session_mocker, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

/home/budrz89/build/python-memory-profiler-feedstock/memory_profiler-0.60.0/test/test_attributes.py:4
___________________________ ERROR at setup of test_1 ___________________________
file /home/budrz89/build/python-memory-profiler-feedstock/memory_profiler-0.60.0/test/test_func.py, line 3
  @profile
  def test_1(i):
E       fixture 'i' not found
>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, class_mocker, datafix, datafix_dir, datafix_read, datafix_readbin, doctest_namespace, mocker, module_mocker, monkeypatch, package_mocker, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, session_mocker, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

/home/budrz89/build/python-memory-profiler-feedstock/memory_profiler-0.60.0/test/test_func.py:3
___________________________ ERROR at setup of test1 ____________________________
file /home/budrz89/build/python-memory-profiler-feedstock/memory_profiler-0.60.0/test/test_mprofile.py, line 6
  @profile
  def test1(l):
E       fixture 'l' not found
>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, class_mocker, datafix, datafix_dir, datafix_read, datafix_readbin, doctest_namespace, mocker, module_mocker, monkeypatch, package_mocker, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, session_mocker, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

/home/budrz89/build/python-memory-profiler-feedstock/memory_profiler-0.60.0/test/test_mprofile.py:6
___________________________ ERROR at setup of test2 ____________________________
file /home/budrz89/build/python-memory-profiler-feedstock/memory_profiler-0.60.0/test/test_mprofile.py, line 13
  @profile
  def test2(l):
E       fixture 'l' not found
>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, class_mocker, datafix, datafix_dir, datafix_read, datafix_readbin, doctest_namespace, mocker, module_mocker, monkeypatch, package_mocker, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, session_mocker, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

/home/budrz89/build/python-memory-profiler-feedstock/memory_profiler-0.60.0/test/test_mprofile.py:13
___________________________ ERROR at setup of test3 ____________________________
file /home/budrz89/build/python-memory-profiler-feedstock/memory_profiler-0.60.0/test/test_mprofile.py, line 19
  def test3(l):
E       fixture 'l' not found
>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, class_mocker, datafix, datafix_dir, datafix_read, datafix_readbin, doctest_namespace, mocker, module_mocker, monkeypatch, package_mocker, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, session_mocker, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

/home/budrz89/build/python-memory-profiler-feedstock/memory_profiler-0.60.0/test/test_mprofile.py:19
________________________ ERROR at setup of test_unicode ________________________
file /home/budrz89/build/python-memory-profiler-feedstock/memory_profiler-0.60.0/test/test_stream_unicode.py, line 6
  @profile(stream=f)
  def test_unicode(txt):
E       fixture 'txt' not found
>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, class_mocker, datafix, datafix_dir, datafix_read, datafix_readbin, doctest_namespace, mocker, module_mocker, monkeypatch, package_mocker, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, session_mocker, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

/home/budrz89/build/python-memory-profiler-feedstock/memory_profiler-0.60.0/test/test_stream_unicode.py:6
____________________ ERROR at setup of test_memory_profiler ____________________
file /home/budrz89/build/python-memory-profiler-feedstock/memory_profiler-0.60.0/test/test_tracemalloc.py, line 18
  def test_memory_profiler(test_input, expected):
E       fixture 'test_input' not found
>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, class_mocker, datafix, datafix_dir, datafix_read, datafix_readbin, doctest_namespace, mocker, module_mocker, monkeypatch, package_mocker, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, session_mocker, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

/home/budrz89/build/python-memory-profiler-feedstock/memory_profiler-0.60.0/test/test_tracemalloc.py:18
=================================== FAILURES ===================================
____________________ TestIncrementDisplay.test_normal_incr _____________________

self = <test_increment_display.TestIncrementDisplay testMethod=test_normal_incr>

    def test_normal_incr(self):
    
        def normal_incr():
            use_some_memory = [1] * (10 ** 6)
    
        profiler = LineProfiler()
        wrapped = profiler(normal_incr)
        wrapped()
    
        show_results(profiler)
        results = list(list(profiler.code_map.values())[0].values())[-1]
    
>       self.assertGreater(results[0], 0)
E       AssertionError: 0.0 not greater than 0

test/test_increment_display.py:38: AssertionError
----------------------------- Captured stdout call -----------------------------
Filename: /home/budrz89/build/python-memory-profiler-feedstock/memory_profiler-0.60.0/test/test_increment_display.py

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
    28     99.9 MiB     99.9 MiB           1           def normal_incr():
    29     99.9 MiB      0.0 MiB           1               use_some_memory = [1] * (10 ** 6)


=============================== warnings summary ===============================
test/test_async.py:8
  /home/budrz89/build/python-memory-profiler-feedstock/memory_profiler-0.60.0/test/test_async.py:8: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
    def my_func():

test/memory_profiler.py:1177
  /home/budrz89/build/python-memory-profiler-feedstock/memory_profiler-0.60.0/test/memory_profiler.py:1177: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
    def wrapper(*args, **kwargs):

test/test_exit_code.py::TestExitCode::test_exit_code_fail
  /home/budrz89/build/python-memory-profiler-feedstock/memory_profiler-0.60.0/test/test_exit_code.py:30: DeprecationWarning: Please use assertRaisesRegex instead.
    self.assertRaisesRegexp(SystemExit, '1', self.run_action)

test/test_exit_code.py::TestExitCode::test_exit_code_success
  /home/budrz89/build/python-memory-profiler-feedstock/memory_profiler-0.60.0/test/test_exit_code.py:21: DeprecationWarning: Please use assertRaisesRegex instead.
    self.assertRaisesRegexp(SystemExit, '0', self.run_action)

-- Docs: https://docs.pytest.org/en/stable/warnings.html
=========================== short test summary info ============================
FAILED test/test_increment_display.py::TestIncrementDisplay::test_normal_incr
ERROR test/test_attributes.py::test_with_profile
ERROR test/test_func.py::test_1
ERROR test/test_mprofile.py::test1
ERROR test/test_mprofile.py::test2
ERROR test/test_mprofile.py::test3
ERROR test/test_stream_unicode.py::test_unicode
ERROR test/test_tracemalloc.py::test_memory_profiler
============= 1 failed, 21 passed, 4 warnings, 7 errors in 15.02s ==============

Are there some pytest plugins that I'm missing?

budrz89 · 01-01-2022, 03:38 PM

The only other thing that I can think of is first copy memory_profiler.py and mprof.py to the test directory and create a script:

Code:

#!/bin/bash

for f in *.py; do
  if [ -f $f ]; then
    if [[ $f != memory_profiler.py && $f != mprof.py ]]; then
      echo $f;
      if ! python -m memory_profiler $f; then
        echo failed
        echo
      else
        echo success
        echo
      fi
    fi
  fi
done

which seems to work ok, except that it may have missed a couple tests (some of the outputs were "Ran 0 tests in 0.000s"). I tee'd the output to a logfile (see attachment).

budrz89 · 01-02-2022, 02:12 PM

Looks like Gentoo devs seem to have done something similar (minus the loops and patches): https://gitweb.gentoo.org/repo/gento...er-0.60.ebuild. So, I tried something like that while still copying memory_profiler.py and mprof.py to the test directory (just in case):

Code:

  cp -v memory_profiler.py test/
  cp -v mprof.py test/
  python3 -m memory_profiler test/test_as.py || echo test_as failed
  python3 -m memory_profiler test/test_func.py || echo test_func failed
  python3 -m memory_profiler test/test_gen.py || echo test_gen failed
  python3 -m memory_profiler test/test_loop.py || echo test_loop failed
  python3 -m memory_profiler test/test_loop_decorated.py || echo test_loop_decorated failed
  python3 -m memory_profiler test/test_mprofile.py || echo test_mprofile failed
  python3 -m memory_profiler test/test_nested.py || echo test_nested failed
  python3 -m memory_profiler test/test_precision_command_line.py || echo test_precision_command_line failed
  python3 -m memory_profiler test/test_unicode.py || echo test_unicode failed

  python3 test/test_exception.py || echo test_exception failed
  python3 test/test_exit_code.py || echo test_exit_code failed
  python3 test/test_global.py || echo test_global failed
  python3 test/test_import.py || echo test_import failed
  python3 test/test_memory_usage.py || echo test_memory_usage failed
  python3 test/test_mprof.py || echo test_mprof failed
  python3 test/test_precision_import.py || echo test_precision_import failed
  python3 test/test_stream_unicode.py || echo test_stream_unicode failed
  python3 test/test_tracemalloc.py || echo test_tracemalloc failed

Attached is a logfile of the output.

budrz89 · 01-03-2022, 03:44 PM

Testing memory_profiler against the one on github (https://github.com/pythonprofilers/memory_profiler), I get a similar result:

Code:

python3 -m scalene prj06.py

Code:

Filename: prj06.py

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
     1   39.684 MiB   39.684 MiB           1   @profile
     2                                         def my_func():
     3   47.160 MiB    7.477 MiB           1       a = [1] * (10 ** 6)
     4  199.785 MiB  152.625 MiB           1       b = [2] * (2 * 10 ** 7)
     5   47.328 MiB -152.457 MiB           1       del b
     6   47.328 MiB    0.000 MiB           1       return a

I found a different profiler called scalene (https://github.com/plasma-umass/scalene) and running the same code with it:

Code:

scalene --outfile prj06-scalene.log prj06.py

Code:

                                             Memory usage: ▁█▁▁▁ (max: 161.32MB, growth rate:   1%)                                              
                                                  prj06.py: % of time =   0.00% out of   0.07s.                                                  
       ╷       ╷       ╷       ╷        ╷       ╷               ╷       ╷                                                                        
  Line │Time   │–––––– │–––––– │Memory  │–––––– │–––––––––––    │Copy   │                                                                        
       │Python │native │system │Python  │peak   │timeline/%     │(MB/s) │prj06.py                                                                
╺━━━━━━┿━━━━━━━┿━━━━━━━┿━━━━━━━┿━━━━━━━━┿━━━━━━━┿━━━━━━━━━━━━━━━┿━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸
     1 │       │       │       │        │       │               │       │#@profile                                                               
     2 │       │       │       │        │       │               │       │def my_func():                                                          
     3 │       │       │       │   5%   │  161M │▁█▁▁▁▁▁▁▁ 100% │       │    a = [1] * (10 ** 6)                                                 
     4 │       │       │       │        │       │               │       │    b = [2] * (2 * 10 ** 7)                                             
     5 │       │       │       │        │       │               │       │    del b                                                               
     6 │       │       │       │        │       │               │       │    return a                                                            
     7 │       │       │       │        │       │               │       │                                                                        
     8 │       │       │       │        │       │               │       │if __name__ == '__main__':                                              
     9 │       │       │       │        │       │               │       │    my_func()                                                           
    10 │       │       │       │        │       │               │       │                                                                        
       ╵       ╵       ╵       ╵        ╵       ╵               ╵       ╵                                                                        
Top PEAK memory consumption, by line:
(1)     3:   161 MB

Now I'm a little confused by these results. Why the dependency in line number and memory usage? I'd understand maybe one is more precise than the other but difference in line numbers?

budrz89 · 01-04-2022, 11:41 AM

I still got the same error in my first post using either scalene or memory_profiler so I investigated a little further into the full traceback (below is a snippet):

Code:

...
  File "mycode.py", line 662, in myfunc
    C = diags(C).toarray()
  File "/usr/lib/python3.10/site-packages/scipy/sparse/construct.py", line 186, in diags
    return dia_matrix((data_arr, offsets), shape=(m, n)).asformat(format)
  File "/usr/lib/python3.10/site-packages/scipy/sparse/dia.py", line 125, in __init__
    dtype=get_index_dtype(maxval=max(shape)),
  File "/usr/lib/python3.10/site-packages/scipy/sparse/sputils.py", line 153, in get_index_dtype
    int32min = np.iinfo(np.int32).min
  File "/usr/lib/python3.10/site-packages/numpy/core/getlimits.py", line 518, in __init__
    self.dtype = numeric.dtype(type(int_type))
TypeError: 'numpy.dtype[bool_]' object is not callable

I replaced

Code:

C = diags(C).toarray()

with

Code:

C = np.diagflat(C)

and it seem to have fixed that problem that I was having earlier. As a bonus, the amount of time to complete is cut in half! However, the memory consumed is still about the same so I would need to figure out what's going on there.

In either case, I'll mark this thread as "SOLVED". Thanks everyone for the suggestions!