LinuxQuestions.org
Visit Jeremy's Blog.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 02-16-2012, 04:33 PM   #1
bigearsbilly
Senior Member
 
Registered: Mar 2004
Location: england
Distribution: Mint, Armbian, NetBSD, Puppy, Raspbian
Posts: 3,515

Rep: Reputation: 239Reputation: 239Reputation: 239
runtime loadable modules


I want to be able to load functions into a process while it is running, like kernel modules can.

I have a lot of real-time financial data coming in 24x7 from the outside world. These are saved to disk in many files.

I have a service reading the data files as they are
writing to disk with the requirement to format the data so they are graphable into new files, keeping the originals.

What I would like is to have run-time loadable filter modules, like kernel modules or shared libraries so that
I can apply different filters without stopping and recompiling. C/C++ code.

Any ideas how this can be done?
 
Old 02-16-2012, 05:27 PM   #2
bigearsbilly
Senior Member
 
Registered: Mar 2004
Location: england
Distribution: Mint, Armbian, NetBSD, Puppy, Raspbian
Posts: 3,515

Original Poster
Rep: Reputation: 239Reputation: 239Reputation: 239
FYI

http://www.linuxjournal.com/article/3687
 
Old 02-16-2012, 05:30 PM   #3
sundialsvcs
LQ Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 10,672
Blog Entries: 4

Rep: Reputation: 3945Reputation: 3945Reputation: 3945Reputation: 3945Reputation: 3945Reputation: 3945Reputation: 3945Reputation: 3945Reputation: 3945Reputation: 3945Reputation: 3945
Dynamic loading and unloading of modules in applications occurs all the time. Search for, for instance, the term dyload.

Many modules, such as the ubiquitous glibc, are automatically loaded (that is to say, bound to the application ...) when any program that uses them is started, but programs can request and load or unload any other library at any time, particularly for situations such as the ones that you describe.
 
Old 02-16-2012, 06:28 PM   #4
Nominal Animal
Senior Member
 
Registered: Dec 2010
Location: Finland
Distribution: Xubuntu, CentOS, LFS
Posts: 1,723
Blog Entries: 3

Rep: Reputation: 948Reputation: 948Reputation: 948Reputation: 948Reputation: 948Reputation: 948Reputation: 948Reputation: 948
For C, see man 3 dlopen, man 3 dlsym and man 3 dlclose.

Designing good plugin API is more or less an art. You certainly want to implement only one filter per library. You probably should have a configuration or initialization function, and a matching release function, just in case the filter wants to set up some tables or something. Plus of course the filtering functions themselves.

If the filtering functions are fully independent, all the functions should have the same name in all filters. Otherwise you need to derive the function and symbol names somehow, e.g. based on the library name -- instead of just version, you'd look for prefix_version.

Whenever your application receives a SIGHUP signal, it should close all filters, and reload them. You can just scan a filter directory, for example. Each filter should obviously have some kind of an identifier. Then, you can build a linked list or a binary tree based on the identifiers, containing the function pointers. Essentially, you'll create a list or tree of "filter objects", each containing variable and function pointers you find out using dlsym().

For example, assume this is an example dynamic filter, which parses decimal integers:
Code:
/* Save as filter.c, then compile using
 *      gcc filter.c -o filter.so -shared -Wall
*/
#include <stdlib.h>
#include <errno.h>

const int    version      = 1;
const char   identifier[] = "foo";

int initialize(void)
{
    return 0;
}

void finalize(void)
{
    return;
}

int parse(const char *string, void *valueptr)
{
    int value, negative;

    if (!string)
        return EINVAL;

    while (*string > 0 && *string <= 32)
        string++;

    negative = 0;
    while (*string == '+' || *string == '-')
        if (*(string++) == '-')
            negative = !negative;

    if (*string >= '0' && *string <= '9')
        value = *(string++) - '0';
    else
        return ENOENT;

    while (*string >= '0' && *string <= '9') {
        const int old = value;
        value = 10*value + (int)(*(string++) - '0');
        if (value < old)
            return EDOM;
    }

    if (*string < 0 || *string > 32)
        return EDOM;

    if (negative)
        value = -value;

    if (valueptr)
        *((int *)valueptr) = value;

    return 0;
}
To load this filter dynamically, you could use
Code:
/* Save as loader.c, then compile using
 *      gcc loader.c -o loader -Wall -ldl
 * To test, run
 *      ./loader ./filter.so
 * (You need to specify the path to the library for dlopen(),
 *  or it will look for the library only in the standard locations.)
*/
#include <stdlib.h>
#include <dlfcn.h>
#include <errno.h>
#include <string.h>
#include <stdio.h>

const char *const test[] = {
    "0",
    " 1",
    " --43",
    "-+--+-+-2",
    NULL
};

/* Acceptable version numbers */
#define MIN_VERSION  1
#define MAX_VERSION  255

struct filter {
    struct filter  *next;     /* Singly-linked list */
    void           *library;  /* dlopen()/dlsym()/dlclose() handle */
    int             version;
    const char     *filename;
    const char     *identifier;
    int           (*initialize)(void);
    void          (*finalize)(void);
    int           (*parse)(const char *, void *);
};

/* Singly-linked list of known filters. */
struct filter *filters = NULL;

const char *load_filter(const char *const filename)
{
    struct filter *current;
    void          *library, *ptr;
    int            result;

    if (!filename || !*filename)
        return strerror(EINVAL);

    errno = 0;
    library = dlopen(filename, RTLD_LAZY | RTLD_LOCAL);
    if (!library)
        return dlerror();

    /* Locate the version symbol. */
    ptr = dlsym(library, "version");
    if (!ptr) {
        dlclose(library);
        return strerror(errno);
    }
    if (*((int *)ptr) < MIN_VERSION || *((int *)ptr) > MAX_VERSION) {
        dlclose(library);
        return strerror(ENOENT);
    }

    /* Allocate a filter structure. */
    current = malloc(sizeof (struct filter));
    if (!current) {
        dlclose(library);
        return strerror(ENOMEM);
    }

    current->library = library;
    current->version = *((int *)ptr);
    current->filename = filename; /* This should really be strdup(filename) */

    ptr = dlsym(library, "identifier");
    if (ptr)
        current->identifier = (const char *)ptr;
    else
        current->identifier = NULL;

    ptr = dlsym(library, "initialize");
    if (ptr)
        *(void **)&(current->initialize) = ptr;
    else
        current->initialize = NULL; /* Or a stub function */

    ptr = dlsym(library, "finalize");
    if (ptr)
        *(void **)&(current->finalize) = ptr;
    else
        current->finalize = NULL; /* Or a stub function */

    ptr = dlsym(library, "parse");
    if (ptr)
        *(void **)&(current->parse) = ptr;
    else
        current->parse = NULL; /* Or a stub function */

    /* Cancel if missing symbols. */
    if (!current->identifier ||
        !current->initialize ||
        !current->finalize ||
        !current->parse) {
        free(current);
        dlclose(library);
        return strerror(ENOENT);
    }

    /* Initialize filter. */
    result = current->initialize();
    if (result) {
        free(current);
        dlclose(library);
        return strerror(result);
    }

    /* Note: dlclose(current->library) is called
     *       when/if this filter is unloaded;
     *       current->finalize() must be called first.
    */

    /* Add to filter chain. */
    current->next = filters;
    filters       = current;

    return NULL;
}

int main(int argc, char *argv[])
{
    int                arg, result, value;
    const char        *failure;
    const char *const *proto;
    struct filter     *curr;

    for (arg = 1; arg < argc; arg++) {

        failure = load_filter(argv[arg]);
        if (failure)
            fprintf(stderr, "%s: Could not load filter: %s.\n", argv[arg], failure);
        else
            fprintf(stderr, "%s: Filter loaded successfully.\n", argv[arg]);
        fflush(stderr);
    }

    if (!filters) {
        fprintf(stderr, "Please supply the path(s) to dynamic filter(s).\n");
        return 1;
    }

    curr = filters;
    while (curr) {
        printf("Filter %s:\n", curr->filename);
        printf("\tVersion: %d\n", curr->version);
        if (curr->identifier)
            printf("\tIdentifier: '%s'\n", curr->identifier);
        printf("\tinitialize(): %p\n", curr->initialize);
        printf("\tfinalize(): %p\n", curr->finalize);
        printf("\tparse(): %p\n", curr->parse);

        for (proto = test; *proto; proto++) {
            result = curr->parse(*proto, &value);
            if (result)
                printf("\tparse(\"%s\", &value) == %d\n", *proto, result);
            else
                printf("\tparse(\"%s\", &value) == %d, value == %d\n", *proto, result, value);
        }

        curr = curr->next;
    }

    return 0;
}
Since you handle complex data formats, you should probably create a header file defining all the structures the filters can exchange data in -- basically, the filter API. You do not need to worry about endianness et cetera, because this is local to the process.

I'm pretty sure I could make better suggestions about what kind of structures to use, if you can describe the data and the operations in detail.

Last edited by Nominal Animal; 02-16-2012 at 06:32 PM.
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Loadable Kernel Modules jbrashear Debian 7 04-03-2005 05:11 PM
Loadable modules question dudds Linux - Newbie 5 07-20-2004 02:13 AM
loadable modules help -- unresolved symbols dklaming Linux - Newbie 1 04-30-2004 02:06 PM
of course I have loadable modules enabled! jmynkbht Linux - Software 0 04-26-2004 03:52 AM
loadable modules vs built-in modules nuzzy Linux - Hardware 1 07-21-2003 04:16 PM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 05:42 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration