How to change (add to) library search path at runtime on Linux
 

I'm writing an application with a plugin architecture and would like modify (specifically add to) the dynamic library search path while the main executable is running.

The plugin paths are not known until the application is running so I can't set LD_LIBRARY_PATH ahead of time.

My understanding (although I haven't tested it) is that the executable will only parse the LD_LIBRARY_PATH once, early on, so modifying this environment variable at runtime will have no effect.

(Note - on Windows the solution is to modify the PATH environment variable)

I know it's possible to specify the full path to load a dynamic library, and this would work if the plugin(s) only had a single library to load, but some of them will have a bunch of libraries with their own inter-dependencies, so I'd like the plugin folder(s) to get added to the search path to pick up the dependant libs.

Is there a proper technique to accomplish this on Linux?
Any guidance is appreciated.

Q: How is modifying %PATH% on Windows any different from modifying $LD_LIBRARY_PATH on Linux?

Q: What's wrong with "wrapping" your executable in a script that sets LD_LIBRARY_PATH, then runs the binary?

Which is the way, say, FireFox runs and which is the way I am using building stuff from scratch - in the latter case the script is autogenerated.

Thanks for the responses thus far.

On Windows, the PATH variable will be reparsed just prior to each load library call. On Linux, (please correct me if I'm wrong) - my understanding is that LD_LIBRARY_PATH will only be parsed at the application startup, and changing it at runtime will have no effect. I'm also hoping to find the proper solution, perhaps various Unix flavors will have different behaviour with respect to reparsing LD_LIBRARY_PATH. I'd like to know if there is a sanctioned way to accomplish this goal.

Wrapping the application is a partial solution. It adds a fair amount of unwanted complexity as there is more logic to loading the plugins than just checking a folder for their existence. This logic is in the application and is already cross-platform. Having to pull that logic out into a pre-run script isn't ideal. In addition, a wrapper will prevent the ability to load additional plugins while the application is running without requiring a restart (eg. a user action causes a download of a new plugin).

You can use 'dlopen' (wrapped into GNU libtool functions if desired) - 'man 3 dlopen'. You can load library from whatever file using 'dlopen'.

add your search path in /etc/ld.so.conf
and then run the command ldconfig as root

If your application has per-user libraries or modules, they should be put in the application directory in the users home directory, $HOME/.myapp/lib/ for example. If there are ancillary libraries in strange places, run your application with $HOME/.myapp/lib/ appended to LD_LIBRARY_PATH, and let the user decide how to manage their libraries.

In Linux, handling shared libraries is quite well standardized. There's ldconfig utility managing /etc/ld.so* files to configure exactly where the libraries may reside, and of course LD_LIBRARY_PATH to add special cases. Dynamic libraries are versioned, with the default symlinked to the latest version. (Okay, some libraries have goofed in not bumping the version number at the proper time, but that's another matter.) Applications are expected to adhere to this information, and not go out of their way to do something else.

How does this work, then?

Assume the user has a special module in $HOME/.myapp/lib/, which requires a number of libraries scattered all over, none of them in the expected places. If your application was executed with $HOME/.myapp/lib/ in LD_LIBRARY_PATH, the user only needs to add symlinks in $HOME/.myapp/lib/ to point to the wayward libraries.

Your application does nothing special. It uses just the library filenames without paths in the dlopen() call.

And please, do not write an application which dynamically edits those symlinks; it's just weird and suspect behaviour. I'd personally never use such a badly behaving application.

You could, however, write a separate configurator tool, which uses e.g. ldd, file and readelf tools with e.g. find, the user can use to locate the libraries and manage the needed symlinks. But I don't believe the users will actually need one.

Nominal Animal

Thanks for all the help, I'm understanding the platform a little better.

I tested altering the LD_LIBRARY_PATH in the running executable, and as suspected that doesn't work. (Setting LD_LIBRARY_PATH ahead of time by hand does allow the libs to load ok).

Calling dlopen with a full path worked fine unless the library has a dependency that is not on the search path. For example, if I have a plugin with libs A and B, where A depends on B, and I try to dlopen /fullpath/A - the dlopen will fail even if I had already done a dlopen /fullpath/B. dlopen is only happy if B is on the search path.

I took a quick look into libtool, and noticed some promising functions in libltdl that may address this issue. I haven't had a chance to try it yet, but thanks for the pointer there.

The non-linux-standard plugin architecture is actually a result of the application not really being a single-user installed application at all. It is actually a distributed, cross-platform, Java based service. (With an additional end-user component to it). Some of the plugins are trivially cross-platform because they are pure Java. Some plugins (for performance and/or 3rd party library integration) have a native C/C++ component to them, and thus bundle native libraries in the plugin. We have Web-service based plugin repositories from which the service can download plugins. For simplicity, this is done the same on each platform, hence standard linux install techniques of the libraries are not used. Everything works well on all platforms (Linux, Windows, Mac, and Solaris) except for this issue of dependant libraries on the Unix platforms.

On UNIXish sytem there is a notion of TMPDIR - if needed, you can first find all the needed libraries, then symlink them to a directory under TMPDIR, then use the the full path with that directory.