LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 06-28-2011, 02:25 PM   #1
ejspeiro
Member
 
Registered: Feb 2011
Distribution: Ubuntu 14.04 LTS (Trusty Tahr)
Posts: 203

Rep: Reputation: 26
Question Modular use of MPI


Hi there!

I was wondering... can one use MPI only in a module of a given sequential program? That is, if one have an already working sequential program, can one append to it a MPI-based module?

Say I have a working main. It calls module() and what this module does is to spread a group of processes to accomplish a given task.

Can that be done or the entire original sequential program must be rewritten?

Thanks in advanced!
 
Old 06-28-2011, 06:33 PM   #2
Nominal Animal
Senior Member
 
Registered: Dec 2010
Location: Finland
Distribution: Xubuntu, CentOS, LFS
Posts: 1,723
Blog Entries: 3

Rep: Reputation: 948Reputation: 948Reputation: 948Reputation: 948Reputation: 948Reputation: 948Reputation: 948Reputation: 948
No, for a couple of reasons.

The program itself does not "spread processes". You do it by running your program via the mpirun command (or equivalent for whatever MPI implementation you use), which uses the configured methods to start the desired number of processes, in parallel, either locally or remotely. The MPI functions your program calls simply contacts these other processes.

Although the MPI specs do not say what a program is allowed to do before initializing MPI or after finalizing MPI, for example the OpenMPI developers recommend as little as possible. Apparently changing program state such as opening files et cetera, is a problem.

If you add MPI_Init() or MPI_Init_thread() to the beginning of your program, and a matching MPI_Finalize() before the program exits, then you can do MPI stuff in one "module" in your program. Still, every single one of the processes started by mpirun or equivalent will execute the entire program, and not just your added "module". You could add conditional clauses to skip the standard parts if the process rank is greater than zero, but that is unlikely to work for any nontrivial program; usually there are memory structures and whatnot created elsewhere in your program, needed by your computational task.

There is also no reliable way to detect whether the program was run via mpirun or equivalent, or not. I've tried to find a way, for fun -- in order to make a helper program MPI-aware, but not requiring MPI -- but there apparently is no portable way. There might be for specific versions of specific MPI libraries, but I'm not aware of them either.

(You can, of course, have the module execute either mpirun or some other command to start a separate MPI-using process, and have that separate process do the task, but I assume that was not what you meant.)

Hope this helps.

Last edited by Nominal Animal; 06-28-2011 at 06:36 PM.
 
1 members found this post helpful.
Old 06-28-2011, 11:23 PM   #3
ejspeiro
Member
 
Registered: Feb 2011
Distribution: Ubuntu 14.04 LTS (Trusty Tahr)
Posts: 203

Original Poster
Rep: Reputation: 26
Oh yes! The dreaded answer I originally considered. The problem is that I ave an already sequential piece of software that simulates things (details on request XD) and I want to use MPI-based libraries to solve for the systems of equations, BUT because of this, I'm gonna have to implement a conditional compilation scheme so a PARALLEL version of the whole simulator can be written, so I can use this library.

I just though on asking first because you never know when someone might jump off with something awesome! XD

Thank you Nominal Animal!
 
Old 06-29-2011, 05:00 AM   #4
Nominal Animal
Senior Member
 
Registered: Dec 2010
Location: Finland
Distribution: Xubuntu, CentOS, LFS
Posts: 1,723
Blog Entries: 3

Rep: Reputation: 948Reputation: 948Reputation: 948Reputation: 948Reputation: 948Reputation: 948Reputation: 948Reputation: 948
Although there are no sane ways to avoid having to compile separate MPI and non-MPI versions of the program, I did find a way for OpenMPI version 1.4.

For my own sanity, the following example only includes support for the functions MPI_Init(), MPI_Finalize(), MPI_Comm_rank(), MPI_Comm_size(), and constants MPI_SUCCESS and MPI_COMM_WORLD; others can be added with a bit of work, based on the OpenMPI header files.

Code:
#include <stdio.h>
#include <stdlib.h>
#include <dlfcn.h>

/* <openmpi/mpi.h> replacement headers, definitions, and dummy replacements.
*/
#define   MPI_SUCCESS 0

typedef void *MPI_Comm;

static int dummy_mpi_init(int *argc_ptr, char ***argv_ptr)     { return MPI_SUCCESS; }
static int dummy_mpi_finalize(void)                            { return MPI_SUCCESS; }
static int dummy_mpi_comm_rank(MPI_Comm c, int *ptr) { *ptr = 0; return MPI_SUCCESS; }
static int dummy_mpi_comm_size(MPI_Comm c, int *ptr) { *ptr = 0; return MPI_SUCCESS; }

MPI_Comm MPI_COMM_WORLD               = NULL;

int (*MPI_Init)(int *, char ***)      = dummy_mpi_init;
int (*MPI_Finalize)(void)             = dummy_mpi_finalize;
int (*MPI_Comm_rank)(MPI_Comm, int *) = dummy_mpi_comm_rank;
int (*MPI_Comm_size)(MPI_Comm, int *) = dummy_mpi_comm_size;

/* Load OpenMPI if available, and OMPI_COMM_WORLD_SIZE is set
 * (only mpirun/orterun should set that environment variable).
 * Returns 1 if it thinks real OpenMPI is available.
*/
int load_openmpi(void)
{
	if (!getenv("OMPI_COMM_WORLD_SIZE"))
		return 0;

	void *const libmpi = dlopen("libmpi.so.0",      RTLD_NOW | RTLD_GLOBAL | RTLD_DEEPBIND);
	void *const librte = dlopen("libopen-rte.so.0", RTLD_NOW | RTLD_GLOBAL | RTLD_DEEPBIND);
	void *const libpal = dlopen("libopen-pal.so.0", RTLD_NOW | RTLD_GLOBAL | RTLD_DEEPBIND);

	if (!libmpi || !librte || !libpal) {
		if (libpal) dlclose(libpal);
		if (librte) dlclose(librte);
		if (libmpi) dlclose(libmpi);
		return 0;
	}

	MPI_COMM_WORLD = dlsym(libmpi, "ompi_mpi_comm_world");

	MPI_Init       = dlsym(libmpi, "MPI_Init");
	MPI_Finalize   = dlsym(libmpi, "MPI_Finalize");
	MPI_Comm_rank  = dlsym(libmpi, "MPI_Comm_rank");
	MPI_Comm_size  = dlsym(libmpi, "MPI_Comm_size");

	return 1;
}


int main(int argc, char **argv)
{
	int	result, process, processes;

	load_openmpi();

	result = MPI_Init(&argc, &argv);
	if (result != MPI_SUCCESS) {
		fprintf(stderr, "MPI_Init() failed (error %d).\n", result);
		return 1;
	}

	result = MPI_Comm_rank(MPI_COMM_WORLD, &process);
	if (result != MPI_SUCCESS) {
		MPI_Finalize();
		fprintf(stderr, "MPI_Comm_rank() failed (error %d).\n", result);
		return 1;
	}

	result = MPI_Comm_size(MPI_COMM_WORLD, &processes);
	if (result != MPI_SUCCESS) {
		MPI_Finalize();
		fprintf(stderr, "MPI_Comm_size() failed (error %d).\n", result);
		return 1;
	}

	if (processes > 0)
		printf("MPI process %d of %d\n", process + 1, processes);
	else
		printf("Non-MPI process\n");
	fflush(stdout);

	MPI_Finalize();

	return 0;
}
Compile the above sources using gcc, not mpicc. For example, if the above is in test-mpi.c use
Code:
gcc -Wall -O3 -fomit-frame-pointer -ftree-vectorize -ldl -o test-mpi test-mpi.c
When the program is run, the variable processes will be 0 if MPI is not available, and positive (number of parallel processes) if MPI is available.

Because the MPI_ functions above are actually function pointers, there is a small overhead to calling them, compared to normal OpenMPI programs. It is only measurable if you call these functions several hundred times a second. The function pointers default to safe dummy functions, so that you can always call them, whether MPI is available or not.

The load_openmpi() function will try to dynamically load the three libraries and look up the actual symbols, if and only if the environment variable OMPI_COMM_WORLD_SIZE is set. Hopefully mpirun/orterun will keep setting that.

The dummy functions are written based on the MPI spec, and are therefore not a problem. The dynamic library file names, internal types, and symbol names are very specific to OpenMPI (version 1.4 here), and are a bit of a pain: they have to be parsed from the OpenMPI header files down to the binary implementation level. For example, note how the MPI_Comm type is actually just a pointer, and MPI_COMM_WORLD "constant" is actually a pointer to ompi_mpi_comm_world symbol. On the other hand, this level must stay unchanged in future versions of the library too, otherwise all other previously compiled programs using OpenMPI will break too. (While compatibility issues can be fixed for other programs by recompiling against the new version of OpenMPI, such changes need to be addressed in the source code for programs like my example above -- not so trivial.)

The reason I posted this code is that a very similar mechanism can be used for dynamically loading plugins or other shared libraries. For plugins, you scan the plugin library, and dlopen() each one using the path and not just file name. If you make sure the binary level interface does not change, this should work quite well. It's just MPI that makes all of this difficult.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
MPI vs LAM/MPI vs OpenMPI vs MPICH2 manojg Linux - General 0 11-28-2010 12:37 PM
modular kernel ramjgn Linux - Enterprise 0 04-05-2009 06:24 AM
modular X11 specialized Slackware 5 06-08-2008 10:47 AM
Non-modular Kernel from scratch Help shotokan Linux From Scratch 6 09-18-2005 09:12 AM
Completely modular system Pelican Linux - General 2 07-01-2005 05:54 PM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 04:06 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration