[SOLVED] Embedding python problem on -current

montagdude · 06-15-2016, 07:47 PM

I'm hoping someone can shed light on this situation. I want to embed python in an application I am writing in order to do some real time plotting. I attempted to compile and run the example from the official python docs here:

https://docs.python.org/2/extending/...pure-embedding.

It builds fine but doesn't run. I would assume it's something that I'm doing wrong (probably is, anyway), but it works fine on my RHEL 6 computer at work and also on my Windows 10 VM using MinGW and Anaconda Python. So I'm confused about why it is not working here. I'll tell you exactly what I'm doing, and maybe someone can tell me what I'm doing wrong:

example.c

Code:

#include <Python.h>

int
main(int argc, char *argv[])
{
    PyObject *pName, *pModule, *pDict, *pFunc;
    PyObject *pArgs, *pValue;
    int i;

    if (argc < 3) {
        fprintf(stderr,"Usage: call pythonfile funcname [args]\n");
        return 1;
    }

    Py_Initialize();
    pName = PyString_FromString(argv[1]);
    /* Error checking of pName left out */

    pModule = PyImport_Import(pName);
    Py_DECREF(pName);

    if (pModule != NULL) {
        pFunc = PyObject_GetAttrString(pModule, argv[2]);
        /* pFunc is a new reference */

        if (pFunc && PyCallable_Check(pFunc)) {
            pArgs = PyTuple_New(argc - 3);
            for (i = 0; i < argc - 3; ++i) {
                pValue = PyInt_FromLong(atoi(argv[i + 3]));
                if (!pValue) {
                    Py_DECREF(pArgs);
                    Py_DECREF(pModule);
                    fprintf(stderr, "Cannot convert argument\n");
                    return 1;
                }
                /* pValue reference stolen here: */
                PyTuple_SetItem(pArgs, i, pValue);
            }
            pValue = PyObject_CallObject(pFunc, pArgs);
            Py_DECREF(pArgs);
            if (pValue != NULL) {
                printf("Result of call: %ld\n", PyInt_AsLong(pValue));
                Py_DECREF(pValue);
            }
            else {
                Py_DECREF(pFunc);
                Py_DECREF(pModule);
                PyErr_Print();
                fprintf(stderr,"Call failed\n");
                return 1;
            }
        }
        else {
            if (PyErr_Occurred())
                PyErr_Print();
            fprintf(stderr, "Cannot find function \"%s\"\n", argv[2]);
        }
        Py_XDECREF(pFunc);
        Py_DECREF(pModule);
    }
    else {
        PyErr_Print();
        fprintf(stderr, "Failed to load \"%s\"\n", argv[1]);
        return 1;
    }
    Py_Finalize();
    return 0;
}

multiply.py:

Code:

def multiply(a,b):
    print "Will compute", a, "times", b
    c = 0
    for i in range(0, a):
        c = c + b
    return c

Both the above files are in the working directory. I compile it like so:

Code:

gcc $(python-config --cflags) -o example $(python-config --ldflags) example.c

It compiles and links fine. I'm trying to run it like this:

Code:

./example multiply multiply 3 2

Finally, here's the error:

Code:

ImportError: No module named multiply
Failed to load "multiply"

FYI, I am running slackware64-current with Python 2.7.11, GCC 5.3.0, and multilib. If anyone can get it working and tell me what I'm doing wrong, I'd be grateful.

Richard Cranium · 06-15-2016, 08:42 PM

Slackware doesn't set PYTHONPATH to include the current directory.
Run...

Code:

export PYTHONPATH=.

...in the shell where you are running this and it will work.

montagdude · 06-15-2016, 10:02 PM

It's so obvious once you say it. Thanks. I had tried before to put multiply.py somewhere that would be in the system PYTHONPATH as a test, but I guess I failed at that. (I think I put it one level too high.)

montagdude · 06-16-2016, 12:02 AM

Okay, I know this is marked solved, but one thing is still bothering me. That is, it is apparently only necessary to set PYTHONPATH=. when running embedded python. When running pure python, (e.g., making a python script that imports multiply from the example in post #1), it will find modules in the working directory without having to set PYTHONPATH. Why is that? I take it that the environment must be somehow different when the interpreter is initialized from a C or C++ code compared to running it from the command line.

dwblas · 06-16-2016, 11:34 AM

Quote:

Originally Posted by montagdude

When running pure python, (e.g., making a python script that imports multiply from the example in post #1), it will find modules in the working directory without having to set PYTHONPATH. Why is that? I take it that the environment must be somehow different when the interpreter is initialized from a C or C++ code compared to running it from the command line.

That depends on where you are calling it from. Each IDE has it's own set of rules, as does BASH. Apparently the software you are using to run the program also checks the current directory.

montagdude · 06-16-2016, 04:07 PM

In my case, I'm always running it through BASH. No IDE or anything. Running the Python interpreter directly from the command line finds modules in the working directory, but it does not when running a C/C++ code that has embedded Python, also directly from the command line, unless I set PYTHONPATH to include the working directory. It's not that important, because my actual application will install the Python module in a location where it will definitely be found, but I'm still curious about this behavior.

Richard Cranium · 06-16-2016, 06:02 PM

Well, the code used by the python command line interpreter is different from the example above. You can go see it at the python mercurial repository or the github clone of the same. I took a quick glance to see if anything obvious would explain it, but I didn't look in any great depth.

notKlaatu · 06-16-2016, 06:12 PM

Running a Python interpreter is different than launching a Python script. These two things treat the PATH differently.

You can test it:

Code:

$ python
>>> import sys
>>> sys.path

versus:

Code:

$ echo "import sys" > mypath.py
$ echo "print(sys.path)" >> mypath.py
$ python ./mypath.py

montagdude · 06-16-2016, 08:00 PM

First method:

Code:

dan@Thinkpad-T430:~/VirtualBox_shared/Xoptfoil/src/embedded_python_test$ python
Python 2.7.11 (default, Mar  3 2016, 13:35:30) 
[GCC 5.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print (sys.path)
['', '/usr/lib64/python27.zip', '/usr/lib64/python2.7', '/usr/lib64/python2.7/plat-linux2', '/usr/lib64/python2.7/lib-tk', '/usr/lib64/python2.7/lib-old', '/usr/lib64/python2.7/lib-dynload', '/usr/lib64/python2.7/site-packages', '/usr/lib64/python2.7/site-packages/gtk-2.0']

Second method:

Code:

dan@Thinkpad-T430:~/VirtualBox_shared/Xoptfoil/src/embedded_python_test$ python plotting_test.py
['/home/dan/VirtualBox_shared/Xoptfoil/src/embedded_python_test', '/usr/lib64/python27.zip', '/usr/lib64/python2.7', '/usr/lib64/python2.7/plat-linux2', '/usr/lib64/python2.7/lib-tk', '/usr/lib64/python2.7/lib-old', '/usr/lib64/python2.7/lib-dynload', '/usr/lib64/python2.7/site-packages', '/usr/lib64/python2.7/site-packages/gtk-2.0']

And finally, running using plotting_test.py as an embedded module in C++, with plotting_test.py being moved to /usr/lib64/python2.7 and not setting PYTHONPATH from the shell:

Code:

dan@Thinkpad-T430:~/VirtualBox_shared/Xoptfoil/src/embedded_python_test$ ./embedded 
Calling python function from C++ ...
['/usr/lib64/python27.zip', '/usr/lib64/python2.7', '/usr/lib64/python2.7/plat-linux2', '/usr/lib64/python2.7/lib-tk', '/usr/lib64/python2.7/lib-old', '/usr/lib64/python2.7/lib-dynload', '/usr/lib64/python2.7/site-packages', '/usr/lib64/python2.7/site-packages/gtk-2.0']

So, what I've learned is that running a python script from BASH automatically appends the working directory to the path, while embedding it in C/C++ does not. (I guess that was obvious, anyway.) Running python interactively does not append the working directory either, but I guess it must just know to always look there first. This has been educational. If I want to dig deeper into why it works this way, I will certainly look into the resources that Richard mentioned. Thanks, guys.