What are the possible approaches to Common-Lisp / Java Interoperability?

Question

So... in an attempt to use preexisting wheels, rather than reinvent my own at every turn, I've been trying to get a decent Common Lisp environment working with [a particular Java's library]. My ABCL adventures actually went reasonably well and I was able, eventually, to get ABCL talking nicely to [it]. Of course I wanted more than just that, I wanted interoperability between the [it] and my half-round wheel, chemicl, a cheminformatics package I started writing in Common Lisp. This is where the train began to fall of the tracks.

ABCL and cxml-stp

A while back, in an earlier, aborted attempt to get some of my chem/bioinformatics (https://github.com/slyrus/cl-bio) stuff working with ABCL I noticed that plexippus-xpath couldn't be loaded into ABCL. This was fixed, so I was encouraged that things might work with ABCL. However, cxml-stp seems to break ABCL.

Hopefully this is a fixable bug and some future version of ABCL will work with cxml-stp.

In the meantime...

Other CL and Java

So, I figured I'd try some other approaches to getting Java and a Common Lisp implementation to play nice. I know, you're thinking "why doesn't the dude just use clojure? After all, that's what clojure was designed for!" Well, that's a good question. I did use clojure for some earlier explorations with [this Java library] and, while the java integration generally works well, I have a bunch of existing Common Lisp code I'd like to use and, at the time at least, it seemed like all of the clojure wrappers where thin wrappers around ugly Java libraries. I've grown to know and love many Common Lisp libraries, many of which are nicely available in QuickLisp, and I'd like to be able to use those (things like cxml-stp, plexippus-xpath, opticl, etc...).

Answer 1

Clozure Common-Lisp (CCL), for five years now, has shipped with a fully ported distribution of JFLI (JFLI previously depended on the LispWorks FFI) as a standard component of the "examples" provided with the CCL source distribution. JFLI (by Rich Hickey, creator of Clojure) uses an in-process model and will likely be at least an order of magnitude more performant than anything you might put together from the model employed by Hickey's next attempt, a more widely compatible socket-based solution he named FOIL.

Have look at the following URL to browse the current JFLI source code as it currently exists in the Clozure development trunk:

https://github.com/Clozure/ccl/tree/master/examples/jfli

Rich Hickey introduced JFLI with the following summary of the approach he had taken (Substitute CCL's FFI where he references LW-FFI obviously):

My objective was to provide comprehensive, safe, dynamic and Lisp-y access to Java and Java libraries as if they were Lisp libraries, for use in Lisp programs, i.e. with an emphasis on working in Lisp rather than in Java. The approach I took was to embed a JVM instance in the Lisp process using JNI. I was able to do this using LispWorks' own FLI and no C (or Java! *) code, which is a tribute to the LW FLI. On top of the JNI layer (essentially a wrapper around the entire JNI API), I built this user-level API using Java Reflection.