The obvious answer is "it depends." It's a good thing I have all this space to explain. This is a somewhat complicated situation, but I'll try to keep it simple.

I've been tasked to create a framework application to analyze data in a database. The application needs to process the original data as text and populate the database with it, and it must be able to easily use updated data-analysis algorithms.

My plan as of now is to have a generic binary that loads the algorithms from shared libraries at run time; that way, any combination can be chosen without recompilation. Each type of algorithm will have an API that the specific libraries must define, both for the binary to use and for the other algorithms to use. This isolates the maintenance of each algorithm from the others and the binary, and vice-versa. I'm currently writing this in C and C++.

Upon preliminary investigation, it seems like dealing with the database might be a whole lot easier in lisp. I have tentative plans to wrap the database API so the algorithms aren't caught up in a mess of SELECT statements, so I'm now considering wrapping it with lisp. This would essentially require the executable to be in lisp, which would then load the algorithms using a C-to-lisp ABI. It would also have to inject function pointers in to the libraries. This can all be done with cffi; however, I don't know if the added complexity would be worth it. I was also thinking lisp would be a useful language to analyze the data with, which I could do by sourcing a lisp file instead of loading a shared library.

Does anyone regularly mix lisp and C? I'm talking about real lisp, not inline or translated code. Is it actually worth the trouble of getting them to interface? I'm pretty new to lisp, so I really don't know.

One of the main project considerations is simplicity (per the PI); however, this is the sort of project that will degrade into complete mayhem if not designed properly from the start. The more skills I add, the fewer people capable of maintaining it. Then again, this is a scientific application with anticipated intra-disciplinary use, so it might not be too much to ask to have skilled programmers maintain it.
Kevin Barry

Q: Will the system respond to commands like "Open the pod bay doors, HAL"?

PS:
It sounds like you should pretty easily be able to partition parts of the system in C/C++, other parts in Lisp. *Without* having to worry about interfacing C and Lisp at any lower level than that.

PPS:
There's a huge wealth of BI ("Business Intelligence") tools you might wish to consider. They're all pretty expensive (I'm not aware of any Open Source initiatives), but they might easily pay for themselves, depending on your project.

Thanks. This is an academic project to analyze a database of experimental findings. Right now it's just me and two other people, but I anticipate it growing into a multi-university project; therefore, nothing that gets paid more than I do. In fact, I already feel guilty for trying to introduce all of the dependencies required by the lisp libraries I want to use.

By "partition..." do you mean I should be able to do what I suggested, or I shouldn't have to do what I suggested (i.e. avoid cffi)? Thanks.
Kevin Barry

Hi, again -

By "partition", I basically meant you shouldn't have to as much as you suggested (if I understand you correctly). In other words: partition big parts of the problem into Lisp, specific subsections into C (or vice versa) and limit the interaction between the two into a relatively small, well-defined interface. The interface might well be implemented in cffi (if you want C to call into Lisp - which probably isn't necessary), but could just as easily be writing to/polling from a flat file, or mutually reading/writing rows from a database ... or some kind of "web service".

In the latter category, a "web service" most emphatically does *not* need to consist of a J2EE web server and a ton of SOAP messages. You might wish to consider a lightweight REST protocol, or share JSON data between two processes (one in LISP, the other in C/C++). Just a thought. Here are a couple of links:

http://www.twine.com/item/114w641qx-6pn/lisp-rest

http://code.google.com/p/rtm-lisp-api/

Those are interesting ideas. I want to keep superfluous data copying to a minimum, so I'm wondering if I can use one of those protocols as a control channel and have, for example, the C program send a request to the lisp process for a query, then have the lisp process fill a mmaped working area. I don't know how feasible that is in lisp, unless it just wrote it to the file and the C program stuck with the mapped area. This is actually a pattern I've been gradually building in to Resourcerver, but it isn't stable enough yet to integrate into a project I don't actually own.
Kevin Barry

I have very very little experience mixing C and Lisp, but ECL and CLISP are supposedly good at what it seems like you want to do (access C libraries from Lisp land). Lisp aside (as much as I hate to see a good Lisp project go away) if it becomes too difficult you might also try languages which are designed specifically to deal with C libraries like Lua or Tcl. Unfortunately even Lisp is not the answer to all problems.

Also, WRT how the system may "degrade into complete mayhem if not designed properly from the start" -- it's been my experience that Lisp languages are excellent for evolving systems. My own development style I think echoes what others have found to be fruitful: write tiny functions to fulfill little pieces in isolation, then write larger functions that use them to do the real work. This typically allows you to isolate data structures (the classic, use lists first then advanced data structures later) and other technical decisions in the smaller functions and move more high-level design into the larger ones. The net result is a system that is more flexible and easier to evolve over time as things change. If performance is an issue, replace those lists with vectors; if users are having trouble with complexity, introduce some macros or HOF to make it easier to program those larger functions. This approach also makes testing easier. As Paul Graham said, with Lisp you build both downwards and upwards at the same time, and eventually your code and the language meet in the middle.

Edit: You might also investigate Chicken Scheme which compiles to C code and reportedly also has a very good foreign interface. Additionally, it has tons of libraries (called eggs) to make life easier, among which are support for several databases. Other Scheme implementations like Gambit also compile to and interface with native code. It could take a long time to evaluate all the good options that are out there, so it's really more a question of whether it's worth putting time in up front for the amount of benefit you'll get later.

Part of my motivation is to learn lisp in an environment where I have a practical goal. It also seems like it would be very effective with what we're doing, which is basically trying to define spatial boundaries based on labeled points in a Euclidean space. We'll need to be very dynamic with the data representations to allow for disparate methods of point correlation.

The whole bit about "degrading to mayhem" is this. I can't count the number of "pet projects" not properly designed to pass on to others in reality that just eroded because 1) people didn't have the time to learn the system, 2) it was easier to mess with it (mess it up a little at a time) than to try to maintain it the way it was intended to be. Most of my effort is going into trying to make the system so it's much easier to maintain the right way than to just go in and do a bunch of one-timers, making it impossible for it to be easily maintained ever again. I'm attempting to create the most extreme isolation that's still practical and efficient so the algorithm writers can hack away all day without needing to see more than the API headers, and the future application maintainer can hack away at the application without having to interact with the algorithm developers (very much.)

You both bring up helpful advice. Thanks.
Kevin Barry

You're talking about "the industry" when you describe the situation in that last post.

I suggest that you "open the pod bay doors" in a different way: go ask the others who are working with you on this project; ask also for opinions from other Universities who might one day become stakeholders.

It is usually a very good idea to use high-level languages, especially when the chore is complex and an abstract representation of it is found to be useful. LISP, for all of its "irritating silly parentheses," really is a heck of a good tool. Maybe your present and future colleagues would have no difficulty using it. Certainly, pursue the option.