[SOLVED] What on earth is $PYTHONPATH

chris_crunch · 03-07-2016, 05:45 AM

Sorry for all these questions guys...

But I've been trying to make sense of this statement for several days.

Quote:

Add the module directory to your $PYTHONPATH by export PYTHONPATH=/path/to/caffe/python:$PYTHONPATH or the like for import caffe.

What does '$PYTHONPATH' mean? I tried

Code:

chris@chris-crunch:~$ echo $PYTHONPATH

And nothing happened. This is just all so confusing. I am losing hope. I think my brain is just not designed to understand computers. God freakin damn it.

business_kid · 03-07-2016, 06:11 AM

$PYTHONPATH is an environment variable. you echo it to read it, or set it as you will.

There are things called python modules, python scripts which do stuff. Often they default install somewhere, and that may not be the default module directory so this problem arises. $PYTHONPATH lists all the nutty places you have your python modules.

The best thing to do is look for your python modules. Mine are in /usr/lib64/python-2.7/site-packages/ (Slackware).

pan64 · 03-07-2016, 06:15 AM

I think it was already discussed: http://www.linuxquestions.org/questi...th-4175573941/
Would be better to continue: what's happened, what is your problem? What did you exactly tried and what did you expect?

chris_crunch · 03-07-2016, 09:47 AM

Yeah, I mean, I just don't really get it. $PYHTONPATH is an environment variable. I don't know what that means, but when I use echo $PYTHONPATH to read it, I obtain nothing! It's just blank. Why?

DavidMcCann · 03-07-2016, 10:38 AM

To find out about $PYTHONPATH, see man python. I don't have it set either, probably because I don't program in Python.

pan64 · 03-07-2016, 10:45 AM

probably it was not set. I mean it has never got a value.
You need to understand: environment variables (especially the exported ones) are inherited, so the child process will get all those variables from its parent. Non-exported variables will not be inherited, so child process will not see that.
Child process means the process started by its parent. Parent process means obviously the process which started its children.
You can check it easily by:

Code:

echo $DFG
# will return nothing, this has not been defined
DFG=2345
echo $DFG
# will return 2345
bash   # this will start a new process, this new is the child and the original is the parent
echo $DFG
# nothing again, DFG was not exported
exit   # return to parent
echo $DFG
# you will see 2345 again
export $DFG   # make it exported, will be inherited
bash          # new child again
echo $DFG
# you should see 2345 here again

If you open two terminals (and within terminals two bash shells) they will not be parent/children to each other, they will not be able to inherit anything from each other.
This is valid for python too: when you run python you start a new child. When you execute commands inside python:
(in this case python is a child, and bash was its parent)

Code:

>>> import sys
>>> sys.path.append('home/chris/apps/caffe-master/python/caffe/pycaffe.py')
>>> import caffe

you will modify its own environment, but its parent and other non-parent shells will not see this change.
By the way you missed the / at the beginning of this string, you ought to write:

Code:

sys.path.append('/home/chris/apps/caffe-master/python/caffe')

I hope you understand it. If not, please ask.

suicidaleggroll · 03-07-2016, 11:19 AM

PYTHONPATH is an environment variable, like PWD, USER, HOME, etc. It contains a list of all of the directories python should search when you go to import a module. When you tried to echo it and got nothing, that simply means it was empty...it didn't exist yet, it's up to you to create it and put something useful there.

business_kid · 03-09-2016, 05:41 AM

Cut to the chase. What and where is the python module you are trying to use? Don't waste time on $PYTHONPATH (which I haven't set either).

You probably have to install whatever module the thing is groping for.

grail · 03-09-2016, 06:55 AM

I am surprised that you are reading a text / web page which mentions this variable but then apparently gives absolutely no information on why it is set and what happens if it is not set.

chris_crunch · 03-14-2016, 06:50 AM

Thanks for the guide pan64. I (think) understand what you have written.
So $PYTHONPATH is an environment variable, a bash variable, that affects the way python runs, and will be passed to Python once it's called. When you import a module into python, it will look here.
It's a variable, but does it have a location? i.e. is there a file somewhere on my computer that I could read with a text editor, telling me what the variable is set to? Surely there must be- how could it exist in the magic mist...?
By 'calling' the variable what does that mean?
And why is my $PYTHONPATH blank, if the interpreter uses the PYTHONPATH to search for modules to import- I have imported modules before, so I assume the interpreter has another way to import modules?
I did this:

Code:

export PYTHONPATH=home/chris/apps/caffe-master/python/caffe:$PYTHONPATH

So does this mean I've set it to also search for modules in this directory? If so, why would it not have searched this directory for a module anyway?

Code:

chris@chris-crunch:~/anaconda2/bin$ echo $PYTHONPATH 
home/chris/apps/caffe-master/python/caffe:

So it is indeed now not blank.

Then,

Code:

 chris@chris-crunch:~/anaconda2/bin$ python
Python 2.7.11 |Anaconda 2.5.0 (64-bit)| (default, Dec  6 2015, 18:08:32) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import sys
>>> sys.path.append('/home/chris/apps/caffe-master/python/caffe/pycaffe.py')
>>> import caffe
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named caffe
>>>

I'm not sure what this code is doing- is it again telling the interpreter where to search for these modules?
I don't know what the issue is at all here. Perhaps I should just uninstall and reinstall caffe entirely; although even if that fixed it I would like to know what I'm doing wrong.

pan64 · 03-14-2016, 07:13 AM

the variable contains the location (or several different directories) where the packages located. These directories are separated by :
You must not add the package itself but the directory where your package is stored. In your last post the original variable:

Code:

chris@chris-crunch:~/anaconda2/bin$ echo $PYTHONPATH 
home/chris/apps/caffe-master/python/caffe:

was wrong because the / at the beginning was missing, therefore it cannot be used (and probably that caffe at the end should be removed too)

Code:

sys.path.append('/home/chris/apps/caffe-master/python/caffe/pycaffe.py')

here you added the module itself to that variable which will not work again. You need to add /home/chris/apps/caffe-master/python either by specifying in a shell or by appending it inside python.

chris_crunch · 03-14-2016, 07:22 AM

Oops. Thanks! That's that problem solved I think... but now there's more errors, (sigh)

Code:

chris@chris-crunch:~$ export PYTHONPATH=/home/chris/apps/caffe-master/python:$PYTHONPATH
chris@chris-crunch:~$ python
Python 2.7.11 |Anaconda 2.5.0 (64-bit)| (default, Dec  6 2015, 18:08:32) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import sys
>>> sys.path.append('/home/chris/apps/caffe-master/python')
>>> import caffe
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/chris/apps/caffe-master/python/caffe/__init__.py", line 1, in <module>
    from .pycaffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, RMSPropSolver, AdaDeltaSolver, AdamSolver
  File "/home/chris/apps/caffe-master/python/caffe/pycaffe.py", line 11, in <module>
    import numpy as np
  File "/home/chris/anaconda2/lib/python2.7/site-packages/numpy/__init__.py", line 184, in <module>
    from . import add_newdocs
  File "/home/chris/anaconda2/lib/python2.7/site-packages/numpy/add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "/home/chris/anaconda2/lib/python2.7/site-packages/numpy/lib/__init__.py", line 8, in <module>
    from .type_check import *
  File "/home/chris/anaconda2/lib/python2.7/site-packages/numpy/lib/type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "/home/chris/anaconda2/lib/python2.7/site-packages/numpy/core/__init__.py", line 58, in <module>
    from numpy.testing import Tester
  File "/home/chris/anaconda2/lib/python2.7/site-packages/numpy/testing/__init__.py", line 14, in <module>
    from .utils import *
  File "/home/chris/anaconda2/lib/python2.7/site-packages/numpy/testing/utils.py", line 15, in <module>
    from tempfile import mkdtemp
  File "/home/chris/anaconda2/lib/python2.7/tempfile.py", line 32, in <module>
    import io as _io
  File "/home/chris/apps/caffe-master/python/caffe/io.py", line 2, in <module>
    import skimage.io
  File "/home/chris/anaconda2/lib/python2.7/site-packages/skimage/__init__.py", line 111, in <module>
    from .util.dtype import *
  File "/home/chris/anaconda2/lib/python2.7/site-packages/skimage/util/__init__.py", line 1, in <module>
    from .dtype import (img_as_float, img_as_int, img_as_uint, img_as_ubyte,
  File "/home/chris/anaconda2/lib/python2.7/site-packages/skimage/util/dtype.py", line 8, in <module>
    dtype_range = {np.bool_: (False, True),
AttributeError: 'module' object has no attribute 'bool_'
>>> import caffe
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/chris/apps/caffe-master/python/caffe/__init__.py", line 1, in <module>
    from .pycaffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, RMSPropSolver, AdaDeltaSolver, AdamSolver
  File "/home/chris/apps/caffe-master/python/caffe/pycaffe.py", line 11, in <module>
    import numpy as np
  File "/home/chris/anaconda2/lib/python2.7/site-packages/numpy/__init__.py", line 184, in <module>
    from . import add_newdocs
  File "/home/chris/anaconda2/lib/python2.7/site-packages/numpy/add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "/home/chris/anaconda2/lib/python2.7/site-packages/numpy/lib/__init__.py", line 8, in <module>
    from .type_check import *
  File "/home/chris/anaconda2/lib/python2.7/site-packages/numpy/lib/type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "/home/chris/anaconda2/lib/python2.7/site-packages/numpy/core/__init__.py", line 14, in <module>
    from . import multiarray
ImportError: cannot import name multiarray

chris_crunch · 03-14-2016, 07:57 AM

It seems numpy as broke too?!

Code:

chris@chris-crunch:~$ python
Python 2.7.11 |Anaconda 2.5.0 (64-bit)| (default, Dec  6 2015, 18:08:32) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import numpy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/chris/anaconda2/lib/python2.7/site-packages/numpy/__init__.py", line 184, in <module>
    from . import add_newdocs
  File "/home/chris/anaconda2/lib/python2.7/site-packages/numpy/add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "/home/chris/anaconda2/lib/python2.7/site-packages/numpy/lib/__init__.py", line 8, in <module>
    from .type_check import *
  File "/home/chris/anaconda2/lib/python2.7/site-packages/numpy/lib/type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "/home/chris/anaconda2/lib/python2.7/site-packages/numpy/core/__init__.py", line 58, in <module>
    from numpy.testing import Tester
  File "/home/chris/anaconda2/lib/python2.7/site-packages/numpy/testing/__init__.py", line 14, in <module>
    from .utils import *
  File "/home/chris/anaconda2/lib/python2.7/site-packages/numpy/testing/utils.py", line 15, in <module>
    from tempfile import mkdtemp
  File "/home/chris/anaconda2/lib/python2.7/tempfile.py", line 32, in <module>
    import io as _io
  File "/home/chris/apps/caffe-master/python/caffe/io.py", line 2, in <module>
    import skimage.io
  File "/home/chris/anaconda2/lib/python2.7/site-packages/skimage/__init__.py", line 111, in <module>
    from .util.dtype import *
  File "/home/chris/anaconda2/lib/python2.7/site-packages/skimage/util/__init__.py", line 1, in <module>
    from .dtype import (img_as_float, img_as_int, img_as_uint, img_as_ubyte,
  File "/home/chris/anaconda2/lib/python2.7/site-packages/skimage/util/dtype.py", line 8, in <module>
    dtype_range = {np.bool_: (False, True),
AttributeError: 'module' object has no attribute 'bool_'

pan64 · 03-14-2016, 08:30 AM

yes, either broken or incompatible.

chris_crunch · 03-14-2016, 08:55 AM

I closed the terminal and started it up again, trying only this:

Code:

chris@chris-crunch:~$ export PYTHONPATH=/home/chris/apps/caffe-master/python:$PYTHONPATH
chris@chris-crunch:~$ python
Python 2.7.11 |Anaconda 2.5.0 (64-bit)| (default, Dec  6 2015, 18:08:32) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import numpy
>>> import caffe
>>>

woop! It worked. Thanks for all you help. I'm still a bit confused as to why it worked.
-why did sys.path.append not work, but setting the PYTHONPATH did the job?
-Do I need to set this PYTHONPATH variable every time I start up a new terminal, or is there a way to do this permanently?
-If so, do you have a note or something on your computer for frequently used commands in the command line that you can efficiently copy and paste out of?