/bin/ld: cannot find -lc++

hmarzooq · 12-14-2021, 08:59 AM

Hello!

I'm facing a problem where a make file is throwing below error:

Code:

/bin/ld: cannot find -lc++

The shared library in error (lc++) is part of libc++ which I've build and installed. However, the make file is not able to find this library. Therefore, I'm assuming this is an issue of a linker (ld) not able to search the lc++ lib in the search path.

Assuming the actual error i.e.

Quote:

/bin/ld: cannot find -lc++

means that linker (ld) cannot find lc++ in the search path, I did investigate below (RPATH issue) and fixed the "not found" issue and re-run make clean & make after removing/creating build dir.

Code:

ldd /usr/local/lib/x86_64-unknown-linux/libc++.so.1.0
        linux-vdso.so.1 (0x00007ffe49d33000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fec62477000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fec620cc000)
        libm.so.6 => /lib64/libm.so.6 (0x00007fec61d8c000)
        librt.so.1 => /lib64/librt.so.1 (0x00007fec61b84000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fec6196e000)
        libatomic.so.1 => /lib64/libatomic.so.1 (0x00007fec61766000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fec62966000)
        libc++abi.so.1 => not found

fixing it by export

Code:

LD_LIBRARY_PATH=/usr/local/lib/x86_64-unknown-linux/

Verifying:

Code:

ldd /usr/local/lib/x86_64-unknown-linux/libc++.so.1.0
        linux-vdso.so.1 (0x00007ffce078e000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fefd6ae0000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fefd6735000)
        libm.so.6 => /lib64/libm.so.6 (0x00007fefd63f5000)
        librt.so.1 => /lib64/librt.so.1 (0x00007fefd61ed000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fefd5fd7000)
        libatomic.so.1 => /lib64/libatomic.so.1 (0x00007fefd5dcf000)
        libc++abi.so.1 => /usr/local/lib/x86_64-unknown-linux/libc++abi.so.1 (0x00007fefd5b8e000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fefd6fcf000)

landed with same error

Code:

/bin/ld: cannot find -lc++

some more info:

Code:

# pwd
/usr/local/lib/x86_64-unknown-linux
[root@ip-1-1-0-1 x86_64-unknown-linux]# ls -ltrh
total 4.0M
-rw-r--r-- 1 root root 1.9M Dec 13 02:47 libc++.a
-rwxr-xr-x 1 root root 1.1M Dec 13 02:47 libc++.so.1.0
-rw-r--r-- 1 root root   28 Dec 13 02:47 libc++.so
-rw-r--r-- 1 root root  13K Dec 13 02:47 libc++experimental.a
-rwxr-xr-x 1 root root 358K Dec 13 02:47 libc++abi.so.1.0
-rw-r--r-- 1 root root 665K Dec 13 02:47 libc++abi.a
lrwxrwxrwx 1 root root   13 Dec 13 07:00 libc++.so.1 -> libc++.so.1.0
lrwxrwxrwx 1 root root   16 Dec 13 07:00 libc++abi.so.1 -> libc++abi.so.1.0
lrwxrwxrwx 1 root root   14 Dec 13 07:00 libc++abi.so -> libc++abi.so.1
[root@ip-10-3-0-165 x86_64-unknown-linux]#

Code:

# cat /etc/ld.so.conf
include ld.so.conf.d/*.conf

# pwd
/etc/ld.so.conf.d

# ls
bind-export-x86_64.conf  dyninst-x86_64.conf  llvm5.0-x86_64.conf  mariadb-x86_64.conf

 grep -rnw . -e "linux"
#

machine info: Amazon Linux 2

Code:

cat /proc/version
Linux version 4.14.252-195.483.amzn2.x86_64 (mockbuild@ip-1-0-1-1) (gcc version 7.3.1 20180712 (Red Hat 7.3.1-13) (GCC))

I've been struggling with this problem from last few days and kind of stuck at the moment. Therefore, any help or guidance is highly appreciated!

Thanks!

shruggy · 12-14-2021, 09:24 AM

Just wondering. Would adding

Code:

-L/usr/local/lib/x86_64-unknown-linux

to this line of the Makefile help?

Another thought. Are you sure the library in question cannot be installed from a repository rather than built from source? Amazon Linux 2 looks similar to RHEL 7, and the library is packaged for the latter by the EPEL project as libcxx (to link with it, you'll need libcxx-devel as well).

From the docs:

Quote:

For the Amazon Linux AMI, access to the Extra Packages for Enterprise Linux (EPEL) repository is configured, but it is not enabled by default. Amazon Linux 2 is not configured to use the EPEL repository. EPEL provides third-party packages in addition to those that are in the repositories. The third-party packages are not supported by AWS. You can enable the EPEL repository with the following commands:

For Amazon Linux 2:

Code:

[ec2-user ~]$ sudo yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

hmarzooq · 12-14-2021, 09:49 AM

Quote:

Originally Posted by shruggy

Just wondering. Would adding

Code:

-L/usr/local/lib/x86_64-unknown-linux

to this line of the Makefile help?

I've tried that with this line - but got same error. I can try again with the line you mentioned. So it should be?

Code:

SHLIB_LINK += -L$(AUTOV8_OUT) -L/usr/local/lib/x86_64-unknown-linux

Quote:

Another thought. Are you sure the library is not provided by Amazon Linux repositories? Amazon Linux 2 looks similar to CentOS 7, and the library is packaged for the latter by the EPEL project as libcxx.

From the docs:

I remember doing

Code:

sudo yum install epel-release
yum install libcxx

but if memory serves, then I got same error. Maybe I should try again by building and installing only libcxxabi from here and libcxx via yum. Note: Both libcxx & libcxxabi are required to build plv8 for Linux.

Thanks!

shruggy · 12-14-2021, 10:19 AM

Quote:

Originally Posted by hmarzooq

I've tried that with this line - but got same error. I can try again with the line you mentioned.

TBH, I now think this line would be better. You'll probably have to add

Code:

-I/usr/local/include

to the CCFLAGS line, too.

Quote:

I remember doing

Code:

sudo yum install epel-release
yum install libcxx

Don't forget

Code:

yum install libcxx-devel

as well.

Quote:

Maybe I should try again by building and installing only libcxxabi from here and libcxx via yum.

I guess libc++ and libc++abi must be in sync. You'll probably have to build the same version as the one in EPEL then.

hmarzooq · 12-14-2021, 09:09 PM

Quote:

Originally Posted by shruggy

TBH, I now think this line would be better. You'll probably have to add

Code:

-I/usr/local/include

to the CCFLAGS line, too.

tried and got same error

Code:

/bin/ld: cannot find -lc++

now trying again by adding directory "/usr/local/lib/x86_64-unknown-linux" to:

Code:

# cat /etc/ld.so.conf
include ld.so.conf.d/*.conf
# ls /etc/ld.so.conf.d/x86_64-unknown-linux.conf
/etc/ld.so.conf.d/x86_64-unknown-linux.conf
# cat /etc/ld.so.conf.d/x86_64-unknown-linux.conf
/usr/local/lib/x86_64-unknown-linux
# sudo ldconfig

Quote:

Originally Posted by shruggy

Don't forget

Code:

yum install libcxx-devel

as well.

I guess libc++ and libc++abi must be in sync. You'll probably have to build the same version as the one in EPEL then.

Yes, makes sense. But how to check versions for libcxx and libcxxabi? Tried something like below but it return command not found "libc++.so.1.0 --version"

Thanks!

hmarzooq · 12-14-2021, 09:40 PM

Quote:

Originally Posted by hmarzooq

tried and got same error

Code:

/bin/ld: cannot find -lc++

now trying again by adding directory "/usr/local/lib/x86_64-unknown-linux" to:

Code:

# cat /etc/ld.so.conf
include ld.so.conf.d/*.conf
# ls /etc/ld.so.conf.d/x86_64-unknown-linux.conf
/etc/ld.so.conf.d/x86_64-unknown-linux.conf
# cat /etc/ld.so.conf.d/x86_64-unknown-linux.conf
/usr/local/lib/x86_64-unknown-linux
# sudo ldconfig

Well, that dosen't help. Landed with same error

Code:

/bin/ld: cannot find -lc++
collect2: error: ld returned 1 exit status
make: *** [plv8-3.0.0.so] Error 1

shruggy · 12-15-2021, 02:08 AM

Quote:

Originally Posted by hmarzooq

tried and got same error

Code:

/bin/ld: cannot find -lc++

Check Using a custom built libc++ in the libc++ docs.

Quote:

But how to check versions for libcxx and libcxxabi?

Code:

$ rpm -q libcxx
libcxx-3.8.0-3.el7.x86_64

Looks like it is this tag.

hmarzooq · 12-15-2021, 02:30 AM

Quote:

Originally Posted by shruggy

Check Using a custom built libc++ in the libc++ docs.

Ok, will try this option as well.

Quote:

Originally Posted by shruggy

Code:

$ rpm -q libcxx
libcxx-3.8.0-3.el7.x86_64

Looks like it is this tag.

Code:

[root@ip-10-3-0-165 x86_64-unknown-linux-gnu]# rpm -q libcxx
package libcxx is not installed
[root@ip-10-3-0-165 x86_64-unknown-linux-gnu]# rpm -q libc++
package libc++ is not installed
[root@ip-10-3-0-165 x86_64-unknown-linux-gnu]#

On a separate note, I'm wondering why clang is not installed as part of Bootstrapping method mentioned here? I'm not able to find where clang is installed. I used locate to search for clang and it returns clang references within the llvm project. I tried which clang and it returned empty. Typing clang gives "-bash: clang: command not found"

hmarzooq · 12-15-2021, 06:41 AM

I've got some clue that ld is not able to search lc++ in search path. These libraries are located inside

Code:

 /usr/local/lib/x86_64-unknown-linux-gnu/

but below we've a path until "usr/local". Not sure if it needs full path i.e. "/usr/local/lib/x86_64-unknown-linux-gnu"

Code:

# ld --verbose | grep SEARCH_DIR | tr -s ' ;' \\012
SEARCH_DIR("=/usr/x86_64-redhat-linux/lib64")
SEARCH_DIR("=/usr/lib64")
SEARCH_DIR("=/usr/local/lib64")
SEARCH_DIR("=/lib64")
SEARCH_DIR("=/usr/x86_64-redhat-linux/lib")
SEARCH_DIR("=/usr/local/lib")
SEARCH_DIR("=/lib")
SEARCH_DIR("=/usr/lib")
#

If this is the case, then how I can add "/usr/local/lib/x86_64-unknown-linux-gnu" in ld config path so that it'll show in above listing?

I think LDFLAGS are the one which is used to set libraries required by ld but I don't see any such flag in make file or Makefile.shared

hmarzooq · 12-15-2021, 09:40 AM

Hello!

Error "/bin/ld: cannot find -lc++" is fixed by using three different approaches individually.

1: Appending "-L/usr/local/lib/x86_64-unknown-linux-gnu" to <https://github.com/plv8/plv8/blob/r3.1alpha/Makefile#L166>

2: Doing "make LDFLAGS=-L/usr/local/lib/x86_64-unknown-linux-gnu"

3: Copying files from "/usr/local/lib/x86_64-unknown-linux-gnu" to "/usr/lib" as "usr/local/lib" is in ld's search.

Is there a better way than above 3 (manual fixes) to add "/usr/local/lib/x86_64-unknown-linux-gnu" in ld's search path? I mean, where I don't need to do it manually and perhaps can be added as part of LLVM bootstrapping build?

pan64 · 12-15-2021, 10:03 AM

I think in your original post this line is not ok:

Code:

-rw-r--r-- 1 root root   28 Dec 13 02:47 libc++.so

The size of this file cannot be 28 (so corrupted). That should be a link to libc++.so.1.0 like the other one.

hmarzooq · 12-15-2021, 10:16 AM

Quote:

Originally Posted by pan64

I think in your original post this line is not ok:

Code:

-rw-r--r-- 1 root root   28 Dec 13 02:47 libc++.so

The size of this file cannot be 28 (so corrupted). That should be a link to libc++.so.1.0 like the other one.

That's true. I've created soft links as below after my original post and the new size is here.

Code:

ln -sf libc++.so.1.0 libc++.so ln -sf libc++.so.1.0 libc++.so.1

pan64 · 12-15-2021, 10:33 AM

Quote:

Originally Posted by hmarzooq

That's true. I've created soft links as below after my original post

And did it help?

hmarzooq · 12-16-2021, 03:04 AM

Quote:

Originally Posted by pan64

And did it help?

I did created the mentioned soft links, however, I've not tested whether it worked without it? But I believe its working more due to the 3 options I posted above.

hmarzooq · 12-16-2021, 03:19 AM

I'm now getting different error when creating plv8 extension. I think its again related to ldd search path. Therefore, I'm posting here in continuation of this thread.

I'm now getting error when creating extension:

Code:

# sudo -u postgres psql
could not change directory to "/root/plv8-3.0.0": Permission denied
psql (13.3)
Type "help" for help.

postgres=# CREATE EXTENSION plv8;
ERROR:  **could not load library "/usr/lib64/pgsql/plv8-3.0.0.so": libc++.so.1: cannot open shared object file: No such file or directory**
postgres=# exit

Code:

s -ltrh /usr/lib64/pgsql/plv8-3.0.0.so
-rwxr-xr-x 1 root root 153M Dec 16 07:54 /usr/lib64/pgsql/plv8-3.0.0.so

ls -ltrh /usr/local/lib/libc++.so.1

Code:

 ldd /usr/lib64/pgsql/plv8-3.0.0.so
        linux-vdso.so.1 (0x00007ffd03f21000)
        librt.so.1 => /lib64/librt.so.1 (0x00007f708ec34000)
        libc++.so.1 => not found
        libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f708e8b2000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f708e572000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f708e35c000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f708dfb1000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f7091178000)

Code:

ld --verbose | grep SEARCH_DIR | tr -s ' ;' \\012
SEARCH_DIR("=/usr/x86_64-redhat-linux/lib64")
SEARCH_DIR("=/usr/lib64")
SEARCH_DIR("=/usr/local/lib64")
SEARCH_DIR("=/lib64")
SEARCH_DIR("=/usr/x86_64-redhat-linux/lib")
SEARCH_DIR("=/usr/local/lib")
SEARCH_DIR("=/lib")
SEARCH_DIR("=/usr/lib")

I've tried copying plv8-3.0.0.so into /usr/local/lib as this dir is in ld's search path. But still getting same error

Code:

could not load library "/usr/lib64/pgsql/plv8-3.0.0.so": libc++.so.1: cannot open shared object file: No such file or directory

I've build using make and make install of plv8 and apparently there was no errors.

Any idea/guidance ? Please.

Thanks!