[SOLVED] MUMPS: A distributed direct solver... linking issues when building examples

ejspeiro · 08-23-2012, 02:20 PM

Hello guys,

So my research work has lead me to the consideration of MUMPS [1] to distributively solve linear systems of equations using direct methods.

I was capable of successfully building the library and getting the required archive files... but I am having issues in the linking stage when building the accompanying examples.

My guess is that my ignorance when it comes to how is the linking process actually being done, is preventing me from getting the linking command line just right for this.

I may be also having some issues regarding the MPI implementation I am using

I do not want to try to guess which information do you guys need to help, so I will take a couple of initial guesses and hopefully, you guys will require some extra info to help me.

This is the shell scenario arising when trying to compile the example I am interested in:

Code:

[ejspeiro@node01 examples]$ make c_example
mpicc -O2  -I/home/ejspeiro/libraries/scalapack-2.0.2/SRC -I/home/ejspeiro/libraries/BLACS/SRC/MPI -I/home/ejspeiro/libraries/MUMPS_4.10.0/include -I. -I../include -c c_example.c -o c_example.o
mpif90 -o c_example -O2 c_example.o -L../lib/libdmumps.a -L../lib/libmumps_common.a -L../lib/libdmumps.a  -L../PORD/lib/ -lpord  -L/home/ejspeiro/libraries/MUMPS_4.10.0/lib -lcmumps -ldmumps -lmumps_common -lpord -lsmumps -lzmumps -L/home/ejspeiro/libraries/scalapack-2.0.2/libscalapack.a -L/home/ejspeiro/libraries/BLACS/LIB/blacs_MPI-LINUX-0.a -L/home/ejspeiro/libraries/BLACS/LIB/blacsF77init_MPI-LINUX-0.a -L/home/ejspeiro/libraries/BLACS/LIB/blacs_MPI-LINUX-0.a -L/home/ejspeiro/libraries/BLAS -lblas -lpthread c_example.o -L../lib/libdmumps.a -L../lib/libmumps_common.a -L../lib/libdmumps.a
c_example.o: In function `main':
c_example.c:(.text+0x0): multiple definition of `main'
c_example.o:c_example.c:(.text+0x0): first defined here
/home/ejspeiro/libraries/MUMPS_4.10.0/lib/libdmumps.a(dmumps_c.o): In function `dmumps_c':
mumps_c.c:(.text+0x8bf): undefined reference to `DMUMPS_F77'
collect2: ld returned 1 exit status
make: *** [c_example] Error 1
[ejspeiro@node01 examples]$

This is how my mpicc and my mpif90 look like:

Code:

[ejspeiro@node01 examples]$ mpicc -show
gcc -I/opt/intel/impi/3.2.0.011/include64 -L/opt/intel/impi/3.2.0.011/lib64 -Xlinker --enable-new-dtags -Xlinker -rpath -Xlinker $libdir -Xlinker -rpath -Xlinker /opt/intel/mpi-rt/3.2 -lmpi -lmpigf -lmpigi -lrt -lpthread -ldl
[ejspeiro@node01 examples]$
[ejspeiro@node01 examples]$
[ejspeiro@node01 examples]$
[ejspeiro@node01 examples]$ mpif90 -show
gfortran -I/opt/intel/impi/3.2.0.011/include64/gfortran/4.1.0 -I/opt/intel/impi/3.2.0.011/include64 -L/opt/intel/impi/3.2.0.011/lib64 -Xlinker --enable-new-dtags -Xlinker -rpath -Xlinker $libdir -Xlinker -rpath -Xlinker /opt/intel/mpi-rt/3.2 -lmpi -lmpigf -lmpigi -lrt -lpthread -ldl
[ejspeiro@node01 examples]$

Any advice?!? Thanks in advanced!

lyle_s · 08-25-2012, 12:22 AM

I notice that you have "c_example.o" listed twice in the arguments to the mpif90 command that are likely causing the "multiple definition of `main'" error.

A similar situation in C produces a similar error message:

Code:

#include <limits.h>
#include <stdio.h>

int main()
{
        printf("%d\n", LINE_MAX);
}

lyle@bowman:~$ gcc -c test.c

lyle@bowman:~$ gcc test.o test.o -o test
test.o: In function `main':
test.c:(.text+0x0): multiple definition of `main'
test.o:test.c:(.text+0x0): first defined here
collect2: ld returned 1 exit status

I'm not sure about the "undefined reference to `DMUMPS_F77'"; maybe you're missing the object file (the .o file) that defines it?

Lyle.

ejspeiro · 08-25-2012, 02:04 AM

lyle_s: Thank you... you were right about the double .o being stated. However, I am still facing the only linking issue that is keeping me from actually building this example:

Code:

[ejspeiro@node01 examples]$ make c_example
mpif90 -o c_example -O2 -L../lib/libdmumps.a -L../lib/libmumps_common.a -L../lib/libdmumps.a  -L../PORD/lib/ -lpord  -L/home/ejspeiro/libraries/MUMPS_4.10.0/lib -lcmumps -ldmumps -lmumps_common -lpord -lsmumps -lzmumps -L/home/ejspeiro/libraries/scalapack-2.0.2/libscalapack.a -L/home/ejspeiro/libraries/BLACS/LIB/blacs_MPI-LINUX-0.a -L/home/ejspeiro/libraries/BLACS/LIB/blacsF77init_MPI-LINUX-0.a -L/home/ejspeiro/libraries/BLACS/LIB/blacs_MPI-LINUX-0.a -L/home/ejspeiro/libraries/BLAS -lblas -lpthread c_example.o -L../lib/libdmumps.a -L../lib/libmumps_common.a -L../lib/libdmumps.a
c_example.o: In function `main':
c_example.c:(.text+0xc2): undefined reference to `dmumps_c'
c_example.c:(.text+0x153): undefined reference to `dmumps_c'
c_example.c:(.text+0x163): undefined reference to `dmumps_c'
collect2: ld returned 1 exit status
make: *** [c_example] Error 1

My problem is that, the function that the linker is claiming to be missing (dmumps_c), is actually contained within the archives I am including in my linking command... how am I sure... well they should be within ../lib/libdmumps.a:

Code:

[ejspeiro@node01 lib]$ nm libdmumps.a | grep dmumps_c.o
dmumps_c.o:
[ejspeiro@node01 lib]$

HOWEVER, if I move the position of the .o I am trying to build, I get a different linking error...

Code:

[ejspeiro@node01 examples]$ make c_example
mpif90 -o c_example c_example.o -O2 -L../lib/libdmumps.a -L../lib/libmumps_common.a -L../lib/libdmumps.a  -L../PORD/lib/ -lpord  -L/home/ejspeiro/libraries/MUMPS_4.10.0/lib -lcmumps -ldmumps -lmumps_common -lpord -lsmumps -lzmumps -L/home/ejspeiro/libraries/scalapack-2.0.2/libscalapack.a -L/home/ejspeiro/libraries/BLACS/LIB/blacs_MPI-LINUX-0.a -L/home/ejspeiro/libraries/BLACS/LIB/blacsF77init_MPI-LINUX-0.a -L/home/ejspeiro/libraries/BLACS/LIB/blacs_MPI-LINUX-0.a -L/home/ejspeiro/libraries/BLAS -lblas -lpthread  -L../lib/libdmumps.a -L../lib/libmumps_common.a -L../lib/libdmumps.a
/home/ejspeiro/libraries/MUMPS_4.10.0/lib/libdmumps.a(dmumps_c.o): In function `dmumps_c':
mumps_c.c:(.text+0x8bf): undefined reference to `DMUMPS_F77'
collect2: ld returned 1 exit status
make: *** [c_example] Error 1
[ejspeiro@node01 examples]$

Notice how now, I get the original error (minus the double definition of the main function), by having moved the c_example.o from almost the end of the statement to the very end.

Again, my claim is that I DO NOT know how to link correctly

Any hints?

Thanks again!

lyle_s · 08-25-2012, 10:59 AM

The linker you're passing the arguments to is ld. You might want to review using ld's man page (using the command "man ld"), especially the -L and -l flags.

-L is used to add directories that the linker will look in to find your libraries.

-l is used to add the library itself.

You have several -L/path/to/some/something.a which isn't correct since -L is for directories.

I learned most of what I know about the general subject of linking from http://www.iecc.com/linker/. Possibly reading some of this will make the linking process a bit less mysterious to you.

I made a attempt to correct the problem; I don't have your setup, so it probably won't work, but hopefully it could be a start:

Code:

mpif90 -L../lib/ -L../PORD/lib/ -L/home/ejspeiro/libraries/MUMPS_4.10.0/lib -lpord -lcmumps -ldmumps -lmumps_common -lpord -lsmumps -lzmumps -L/home/ejspeiro/libraries/scalapack-2.0.2/ -lscalapack -L/home/ejspeiro/libraries/BLACS/LIB/ -l:blacs_MPI-LINUX-0.a -l:blacsF77init_MPI-LINUX-0.a -L/home/ejspeiro/libraries/BLAS -lblas -lpthread -O2 -o c_example c_example.o

Lyle.

ejspeiro · 08-25-2012, 03:08 PM

Lyle: Thanks for your reply!

I went back and created logic links to the BLACS libraries so that -l could do its job properly:

Code:

[ejspeiro@node01 LIB]$ pwd
/home/ejspeiro/libraries/BLACS/LIB
[ejspeiro@node01 LIB]$ ll
total 872
-rw-rw-r-- 1 ejspeiro ejspeiro   5092 Aug 14 10:35 blacsCinit_MPI-LINUX-0.a
-rw-rw-r-- 1 ejspeiro ejspeiro   4412 Aug 14 10:35 blacsF77init_MPI-LINUX-0.a
-rw-rw-r-- 1 ejspeiro ejspeiro 816116 Aug 14 10:35 blacs_MPI-LINUX-0.a
lrwxrwxrwx 1 ejspeiro ejspeiro     24 Aug 25 12:57 libblacsCinit_MPI-LINUX-0.a -> blacsCinit_MPI-LINUX-0.a
lrwxrwxrwx 1 ejspeiro ejspeiro     26 Aug 25 12:57 libblacsF77init_MPI-LINUX-0.a -> blacsF77init_MPI-LINUX-0.a
lrwxrwxrwx 1 ejspeiro ejspeiro     19 Aug 25 12:58 libblacs_MPI-LINUX-0.a -> blacs_MPI-LINUX-0.a
-rw-r--r-- 1 ejspeiro ejspeiro    226 Jan 18  2006 LIB.log
[ejspeiro@node01 LIB]$

Then, and I tried your line, but with the modifications necessary given the links I have created:

Code:

[ejspeiro@node01 examples]$ mpif90 -L../lib/ -L../PORD/lib/ -L/home/ejspeiro/libraries/MUMPS_4.10.0/lib -lpord -lcmumps -ldmumps -lmumps_common -lpord -lsmumps -lzmumps -L/home/ejspeiro/libraries/scalapack-2.0.2/ -lscalapack -L/home/ejspeiro/libraries/BLACS/LIB/ -lblacs_MPI-LINUX-0 -lblacsF77init_MPI-LINUX-0 -L/home/ejspeiro/libraries/BLAS -lblas -lpthread -O2 -o c_example c_example.o
c_example.o: In function `main':
c_example.c:(.text+0xc2): undefined reference to `dmumps_c'
c_example.c:(.text+0x153): undefined reference to `dmumps_c'
c_example.c:(.text+0x163): undefined reference to `dmumps_c'
collect2: ld returned 1 exit status

Again, when I move the c_example.o to the very beginning, I get the error originally mentioned:

Code:

[ejspeiro@node01 examples]$ mpif90 c_example.o -L../lib/ -L../PORD/lib/ -L/home/ejspeiro/libraries/MUMPS_4.10.0/lib -lpord -lcmumps -ldmumps -lmumps_common -lpord -lsmumps -lzmumps -L/home/ejspeiro/libraries/scalapack-2.0.2/ -lscalapack -L/home/ejspeiro/libraries/BLACS/LIB/ -lblacs_MPI-LINUX-0 -lblacsF77init_MPI-LINUX-0 -L/home/ejspeiro/libraries/BLAS -lblas -lpthread -O2 -o c_example
../lib//libdmumps.a(dmumps_c.o): In function `dmumps_c':
mumps_c.c:(.text+0x8bf): undefined reference to `DMUMPS_F77'
collect2: ld returned 1 exit status

It was of my understanding that you could use -L to get the particular .a file included in the linking... was I wrong? And another question... what is the purpose of the ':' as you propose on your link statement for the BLACS libraries?

Thanks!

ejspeiro · 08-29-2012, 01:50 PM

Dear lyle_s:

I have solved my issue! Thanks... I basically followed your tip and redid the linking statement:

Code:

mpicc -O2  -I/home/ejspeiro/libraries/scalapack-2.0.2/SRC -I/home/ejspeiro/libraries/BLACS/SRC/MPI -I/home/ejspeiro/libraries/MUMPS_4.10.0/include -I. -I../include -c c_example.c -o c_example.o
mpif90 -o c_example c_example.o -O2 -L../lib -ldmumps -L../lib -lmumps_common -L../lib -ldmumps  -L../PORD/lib/ -lpord  -L/home/ejspeiro/libraries/MUMPS_4.10.0/lib -lcmumps -ldmumps -lmumps_common -lpord -lsmumps -lzmumps -L/home/ejspeiro/libraries/scalapack-2.0.2 -lscalapack /home/ejspeiro/libraries/BLACS/LIB/blacs_MPI-LINUX-0.a /home/ejspeiro/libraries/BLACS/LIB/blacsF77init_MPI-LINUX-0.a /home/ejspeiro/libraries/BLACS/LIB/blacs_MPI-LINUX-0.a ../../lapack-3.4.1/liblapack.a -L/home/ejspeiro/libraries/BLAS -lblas -lpthread  -L../lib -ldmumps -L../lib -lmumps_common -L../lib -ldmumps

Thanks a lot... I am calling this one solved \m/

lyle_s · 08-29-2012, 07:04 PM

Awesome!

Lyle.