LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 11-17-2022, 10:41 AM   #1
sundialsvcs
LQ Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 10,007
Blog Entries: 4

Rep: Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607
Python can't find MySQL ... why not?


Okay, this is not strictly a "Linux" question, but it's close. (MacOS is Unix® ...)

All of the sudden, Python can no longer locate the MySQL interface libraries: it now thinks that they are in /usr/bin.

I can see from the traceback that it is using @rpath to find the location, and that it knows the name of the library that it should be looking for. But it's not looking in the right place anymore. (As I've said, "this used to work ...")

All of the web postings that I've so far found about "rpath" imply that in order to change it you must patch the binary. But that makes no sense to me. Obviously there should be something, somewhere, that informs Python where to look. But now it's broken – actually, "it broke" – and I really not enough of a Python God to understand why.

Relevant traceback:
Code:
  File "/Users/mike/.virtualenvs/djangoprod/lib/python3.11/site-packages/MySQLdb/__init__.py", line 18, in <module>
    from . import _mysql
ImportError: dlopen(/Users/mike/.virtualenvs/djangoprod/lib/python3.11/site-packages/MySQLdb/_mysql.cpython-311-darwin.so, 0x0002): Library not loaded: '@rpath/libmysqlclient.21.dylib'
  Referenced from: '/Users/mike/.virtualenvs/djangoprod/lib/python3.11/site-packages/MySQLdb/_mysql.cpython-311-darwin.so'
  Reason: tried: '/usr/lib/libmysqlclient.21.dylib' (no such file)
Right now I am considering nuking the present "virtual environment" and re-creating it from scratch ... because it sure looks like "everything that's now going very wrong" is inside it. But I'd right now really like to hear from anyone who is more familiar with this "magick" than I am.
 
Old 11-17-2022, 01:36 PM   #2
pan64
LQ Addict
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 19,248

Rep: Reputation: 6525Reputation: 6525Reputation: 6525Reputation: 6525Reputation: 6525Reputation: 6525Reputation: 6525Reputation: 6525Reputation: 6525Reputation: 6525Reputation: 6525
from my side I would try to create that venv again (save what you have first).
 
Old 11-17-2022, 04:36 PM   #3
sundialsvcs
LQ Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 10,007

Original Poster
Blog Entries: 4

Rep: Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607
I already did that. Renamed the old venv, created a new one, tested it out ... identical results.

Python simply does not actually know where to look for the library. And, I do not at all understand the mechanism by which it is supposed to be able to do so.
 
Old 11-18-2022, 01:50 AM   #4
pan64
LQ Addict
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 19,248

Rep: Reputation: 6525Reputation: 6525Reputation: 6525Reputation: 6525Reputation: 6525Reputation: 6525Reputation: 6525Reputation: 6525Reputation: 6525Reputation: 6525Reputation: 6525
that means you have something misconfigured in that system. Let's say you have an LD_LIBRARY_PATH set to something strange. Or a python related environment variable is set to something unusual.
Actually the dir site-packages inside the venv looks ok, so I don't really know. The other thing can be the lib itself which is incompatible with the system you have (that's why you would need to reinstall that python module from scratch).
 
Old 11-18-2022, 04:26 PM   #5
sundialsvcs
LQ Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 10,007

Original Poster
Blog Entries: 4

Rep: Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607
(P.S.: I re-posted this here instead of "Software" because it's a better location, but didn't know how to remove the old thread.)

Further investigation led me to man dyld on MacOS, which confirmed what environment variables are used, including DYLD_LIBRARY_PATH. Also, it finally explained to me what @RPATH is (on MacOS), and where it comes from.
Quote:
Dyld maintains a current stack of paths called the run path list. When @rpath is encountered it is substituted with each path in the run path list until a loadable dylib if found. The run path stack is built from the LC_RPATH load commands in the depencency chain that lead to the current dylib load. You can add an LC_RPATH load command to an image with the -rpath option to ld(1). You can even add a LC_RPATH load command path that starts with @loader_path/, and it will push a path on the run path stack that relative to the image containing the LC_RPATH. The use of @rpath is most useful when you have a complex directory structure of programs and dylibs which can be installed anywhere, but keep their relative positions. This scenario could be implemented using @loader_path, but every client of a dylib could need a different load path because its relative position in the file system is different. The use of @rpath introduces a level of indirection that simplies things. You pick a location in your directory structure as an anchor point. Each dylib then gets an install path that starts with @rpath and is the path to the dylib relative to the anchor point. Each main executable is linked with -rpath @loader_path/zzz, where zzz is the path from the executable to the anchor point. At runtime dyld sets it run path to be the anchor point, then each dylib is found relative to the anchor point.
I've also discovered a MacOS feature called "System Integrity Protection" which might be interfering with the passing of the DYLD_LIBRARY_PATH environment variable.

Last edited by sundialsvcs; 11-18-2022 at 05:33 PM.
 
Old 11-27-2022, 05:32 PM   #6
sundialsvcs
LQ Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 10,007

Original Poster
Blog Entries: 4

Rep: Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607
I have finally SOLVED this problem. It turns out that it was caused by a recently-introduced MacOS (OS/X) feature known as System Integrity Protection (SIP) which, among other things, conceals the DYLD_LIBRARY_PATH variable setting from child processes. Apparently the Python MySQL interface must use one. Turning off this feature solved the problem.

Mind you, it's anybody's guess why doing such a strange thing with environment variables or dynamic loading would "protect system integrity." So far as I can tell, it would just break a lot of things.

Last edited by sundialsvcs; 11-27-2022 at 05:35 PM.
 
2 members found this post helpful.
Old 11-27-2022, 05:50 PM   #7
astrogeek
Moderator
 
Registered: Oct 2008
Distribution: Slackware [64]-X.{0|1|2|37|-current} ::12<=X<=15, FreeBSD_12{.0|.1}
Posts: 5,973
Blog Entries: 23

Rep: Reputation: 3945Reputation: 3945Reputation: 3945Reputation: 3945Reputation: 3945Reputation: 3945Reputation: 3945Reputation: 3945Reputation: 3945Reputation: 3945Reputation: 3945
Quote:
Originally Posted by sundialsvcs View Post
(P.S.: I re-posted this here instead of "Software" because it's a better location, but didn't know how to remove the old thread.)
When you want to move a post to another forum just use the Report button and request the move, but no harm done and your other thread has been closed.

Quote:
Originally Posted by sundialsvcs View Post
I have finally SOLVED this problem. It turns out that it was caused by a recently-introduced MacOS (OS/X) feature known as System Integrity Protection (SIP) which, among other things, conceals the DYLD_LIBRARY_PATH variable setting from child processes.
Thanks for reporting the solution, I have been curious what the cause may be but am not a Mac user myself. Yet another helpful feature which prevents ordinary use and has prevented you from using all that time for more productive pursuits!
 
Old 11-28-2022, 09:13 AM   #8
sundialsvcs
LQ Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 10,007

Original Poster
Blog Entries: 4

Rep: Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607
Astrogeek, why don't you just remove the old thread altogether. Please. Just make it vanish.

Yes, I am puzzled about certain aspects of the design of this feature. It does have a lot of very good security-oriented ideas which I would like to keep ... although I protect myself by not running all the time as an "Administrator." It's most unfortunate that I can only "turn the whole thing 'on' or 'off.'"

In MacOS, "applications" are actually "packages," and they are typically self-contained. I really don't know the fine details of how they handle "dynamic load libraries" within themselves, because I've never actually looked. But, it really baffles me that they would say that DYLD_LIBRARY_PATH settings changes work for the main process but cannot be seen(!) by a child. What's the point of that? I have no idea.

Last edited by sundialsvcs; 11-28-2022 at 09:15 AM.
 
Old 11-28-2022, 11:51 AM   #9
astrogeek
Moderator
 
Registered: Oct 2008
Distribution: Slackware [64]-X.{0|1|2|37|-current} ::12<=X<=15, FreeBSD_12{.0|.1}
Posts: 5,973
Blog Entries: 23

Rep: Reputation: 3945Reputation: 3945Reputation: 3945Reputation: 3945Reputation: 3945Reputation: 3945Reputation: 3945Reputation: 3945Reputation: 3945Reputation: 3945Reputation: 3945
Quote:
Originally Posted by sundialsvcs View Post
Astrogeek, why don't you just remove the old thread altogether. Please. Just make it vanish.
LQ policy is averse to removal of any member content other than overt spam and abuse, and that generally extends to unreplied duplicate posts where the preference is for closure rather than removal.
 
Old 11-29-2022, 06:51 PM   #10
sundialsvcs
LQ Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 10,007

Original Poster
Blog Entries: 4

Rep: Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607Reputation: 3607
Quote:
Originally Posted by astrogeek View Post
LQ policy is averse to removal of any member content other than overt spam and abuse, and that generally extends to unreplied duplicate posts where the preference is for closure rather than removal.
I will of course leave this up to your "management discretion." Been there, did that. (Hated it.) Better you than me.

Never want to be "a Moderator" again ... god bless ye.

Last edited by sundialsvcs; 11-29-2022 at 06:55 PM.
 
1 members found this post helpful.
Old 11-30-2022, 01:43 AM   #11
pan64
LQ Addict
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 19,248

Rep: Reputation: 6525Reputation: 6525Reputation: 6525Reputation: 6525Reputation: 6525Reputation: 6525Reputation: 6525Reputation: 6525Reputation: 6525Reputation: 6525Reputation: 6525
Quote:
Originally Posted by sundialsvcs View Post
I have finally SOLVED this problem. It turns out that it was caused by a recently-introduced MacOS (OS/X) feature known as System Integrity Protection (SIP) which, among other things, conceals the DYLD_LIBRARY_PATH variable setting from child processes. Apparently the Python MySQL interface must use one. Turning off this feature solved the problem.

Mind you, it's anybody's guess why doing such a strange thing with environment variables or dynamic loading would "protect system integrity." So far as I can tell, it would just break a lot of things.
Good to know. Thanks.
From the other hand it's just a feature where we (humans as developers) fend us (humans as users) from humans (as hackers). We limit ourselves to protect ourselves.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: Python Python Python (aka Python 3) LXer Syndicated Linux News 0 08-05-2009 09:30 PM
Can MySQL log on via SSH/bash? mysql:x:27:101:MySQL Server:/var/lib/mysql:/bin/bash Ujjain Linux - Newbie 2 04-24-2009 03:21 PM
mysql error Can't connect to local MySQL server through socket '/var/lib/mysql/mysql. SpellChainz Linux - Newbie 1 06-23-2007 04:35 PM
mysql error Can't connect to local MySQL server through socket '/var/lib/mysql/mysql. Dannux Linux - Software 3 03-24-2006 09:44 AM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 06:29 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration