Thursday, February 15, 2007

Free Lisp Executables in Windows

One of my latest projects has led me to think about making windows executables for a typical database application. The main requirement was using free software to make this project, so here's my starting point:

  • Clisp, 2.41 (2006-10-13) (full, with readline support)

  • ASDF
  • IDE: Emacs and SLIME (latest, from CVS at the date of this post)
The first issue I want to get behind is to make windows executables. If I can't make those, I might as well move to Python - but my programming language preference makes me stubborn about this one. I found out the magical command, ext:saveinitmem. But this alone wouldn't work, and I got to the conclusion SLIME was the problem - the binary image would try to output to slime, and result in an error. The fix is simple enough - just don't load swank, and make the binary without slime. I made a convenience build.bat file, with the following:

c:\dev\clisp\full\lisp.exe -B c:\dev\clisp\full -M c:\dev\clisp\full\lispinit.mem -ansi -norc -q -i build.lisp

(don't forget to change clisp's path to your configuration). One more thing I noted was the dll's clisp uses had to be in the same directory as the produced executable. (find them under "



Then, I put into build.lisp everything I want preloaded. ASDF, my project file loading, etc. and then, the magical function call (assuming an already loaded "test-app" package with a "main" function):

#+:clisp (ext:saveinitmem "test-app"

:init-function #'(lambda ()

(test-app::main))

:NORC t

:script t

:start-package :test-app

:executable t

:quiet t)

All fine so far. The executable is not small, but being about 5MB makes it smaller than the ~30MB I would get with the latest SBCL (1.01) for windows.



Next challenge, the database connection. This was a lot more troublesome than I initially thought. CLSQL was the elected choice, because I needed to access legacy data, initially from an Access .mdb file. I had to create an ODBC Data Source and use the clsql-odbc package to access it. I got this working in the clisp's repl, and then I made an image, run it and BOOM, all hell broke loose. Here is a thread where I got some valuable hints, specially from Ken Tilton, whose Celtk package I used. What happened is that clsql uses CFFI, with cffi-uffi-compat to interface to foreign functions defined in external dll's (in the windows case). Two issues ate more of my time than I would like:

  • Each defcfun (i.e., foreign function interface defined) must be evaluated after having used/loaded (only once) the foreign library. Ok, that makes sense. What puzzled me was that, unline Allegro CL, Clisp (and sbcl, while we're at it) requires us to repeat this use/load-foreign-library each time an image is (re)started. This makes it a bit messier, as I have to choose (1) to load clsql at the image execution-time or (2) fetch the needed libraries from the clsql project, and load them in my main (initial) function. I opted from the second approach, for image load-time's sake. The required lines are something like:

    (cffi-uffi-compat:load-foreign-library "clsql_uffi" :force-load t)

    (cffi-uffi-compat:load-foreign-library "odbc32" :force-load t).

  • CLSQL uses a variable to represent a null C pointer. this is ok, except that this pointer is created one time, so after clisp loads the created image, this pointer is invalid, yielding errors every time they are used. Solution? I've simply replaced all *null-ptr* with (make-null-pointer :byte) and *null-handle-ptr* with (make-null-pointer :void). This made the trick, although incurring in a small performancing penalty - I'm not that desperate, yet :).
After this issues, I was able to read and write to my access database from my (now 6.5MB) executable. On to the next issue, the GUI. Ahh, this made me think of lots of variables:

  • Language integration with the graphics (consider making calls for foreign functions vs. using simple abstractions in lisp);
  • Portability, I wouldn't like to build different GUI's for Mac and Windows, for instance;
  • Prettiness. Ok, It may be the best toolkit ever, but I may like to have, say, the xp look and feel. Or some other...
  • GUI design. There are some nice GUI designers out there, and there are some nice lisp abstrations to make the same result. What to choose?



Given this pre-requirements, my options for free were wxCL, graphic-forms, celtk, Ltk, and cells-gtk. (This list may not be comprehensive, feel free to comment, tell me about the amazing />

I tried wxCL first, as I'd already used wxPython, and felt at ease with wxGlade. So I started using it, created my first hello-world widget, and bang, had a whole lot of troubles, cffi related and others. I guess that they could be handled, but I would still be attached to the wxWidgets theme, not very handsome to me. For the same reason I ditched cells-gtk... I moved along, on to Graphic-forms. This is still in alpha stage, but is looking nice! Although, the interface is only for windows forms, and that makes it pretty useless in a mac/linux box. On I went, to the tcl/tk world. At first I thought I'd be attached to a specific theme/motif, but I've learned about Tile, and this made the trick for me. So, how do I choose between Ltk and celtk? Well, Ken Tilton's Cells project, basically, prodded me towards celtk :) It's just a programming language decision, cells can be used to bind behaviors to certain parts of the GUI in an oh-so-elegant way, I was hooked soon enough!



Ok, so what's the downside, you may ask? Celtk is still a work in progress, and is kind of not-ready for quick use/deployment. I also had cffi-related problems, simillar to those I had with clsql, and I worked around them with the following addition to may main function:

(cffi::use-foreign-library "/dev/Tcl/bin/tcl85.dll")

(cffi::use-foreign-library "/dev/Tcl/bin/tk85.dll")

(cffi::use-foreign-library "/dev/Tcl/lib/tile0.7.8/tile078.dll")



There's also lots of absolute paths to fix, in a decentralized way, the asdf is not up-to-date (Ken relies on ACL's *.lpr files), there's no readme/howto/start-here, and basically no homepage with this information. I think Ken is trying to get the project a home in http://common-lisp.net/, and meanwhile the comp.lang.lisp newsgroup will do to discuss issues we may have.



What I can say is that I've put his demo project, along with a sample MSAccess data access, in an ~8MB executable. I'm sure I can optimize this, but for me it's fair enough, and it loads fast enough for me!



Hope some of you may learn something from this. Feel free to shoot any questions, or suggestions you may have!



powered by performancing firefox

10 comments:

Flatlander said...

Nice article, I have had similar problems with clisp.
But does SBCL really require you to explicitly load the foreign library every time image is started? I'm fairly sure I have used both win & linux sbcl with just the load-library -> defcfun -> dump image routine.
It seems that at image loading time sbcl tries to find the dll's from the same locations they were loaded before dumping the image, and failing that, from the OS's standard locations(like .exe's folder etc. etc.)

Edgar Gonçalves said...

Well, that's probably true, and I may have mistakenly taken some error as the same I was getting in clisp (and since clisp got me to a nice working binary, I found my choice for now).

But anyway, I guess SBCL for windows is still in early development stages, and I often get errors like "GC invariant lost", that crashes the LDB :s

My point is, unless (1) someone tells me about an advantage for me to use SBCL*, (2) clisp fails me on some specific unforeseen task or, at least (3) until i see a similar post like this one, but on sbcl, I have no reasons to play again in sbcl.


* Multithreading in windows would be one great example!

John Wiegley said...

Hi Edgar, I know how to dump Swank into a core file and get it to load correctly on other systems (I do this for Ready Lisp, since I don't want users to want for Swank to be loaded at startup). Just e-mail me and I can describe what need to be done.

Edgar Gonçalves said...

Thanks for your help! In fact, I've been watching your progresses with Ready Lisp. I've recently switched my main development environment from Windows to Mac OS, and I've been using OpenMCL, but for some applications that need 32bit libraries I use Ready Lisp. And yes, loading a fully loaded lisp image is quite clever - I've used the same approach when coding academic scheme interpreters, 5 years ago. Quick question - why do you pick sbcl over openMCL on Mac OS? Is it "just" for its OS-wide availability? I've yet to found good reasons to opt for one over another, except that.

John Wiegley said...

I picked SBCL mainly because it's what I use; and I use it because the SLIME integration is the best out of all the implementations I currently use during development (Allegro, LispWorks, CMUCL and ECL). LispWorks may have a much better stepper, for example, but SBCL will jump me right to the offending sexp when I hit 'v' in a stack trace.

Edgar Gonçalves said...

Hmm, that's quite valid - I've been using the same shortcut with openmcl, but I've just checked there are a few differences. I'll have to check on those better, eventually! Thanks again for your comment, and have a merry lispy christmas!

John Wiegley said...

I just tried running the very latest Clozure CL (for 64-bit Intel Leopard), and ran into these issues right away: CL-FAD will not load, but this can be fixed by stubbing out the definition of %rmdir in openmcl.lisp in that project; and loading LOCAL-TIME results in the error "Error: Fault during read of memory address #x10". At least SERIES was able to load fine, which is where I usually have problems. But the LOCAL-TIME problem stops me for now -- at least until the new year when I have some time to help track down what's going on.

Edgar Gonçalves said...

I understand Clozure CL is still in its early stages; I've been toying around with the latest version of openmcl (for 64bit, also). cl-fad can asdf-install/load perfectly fine, but you're right about local-time, I get the same error. I have no need for such library for the moment; however, I suspect it can be easily ported to work under openMCL. However, I'm with you - if sbcl works fine the way it is, there's no need to swap until some further issue is raised.

Robert said...

I wasn't entirely sure why loading the swank stuff was an issue for you in building your executable.

When I build executables, I typically do not distribute the emacs-lisp-able version of the application, just the standalone. Since your executable clearly has a UI of its own, why is emacs integration an issue?

thanks!

Edgar Gonçalves said...

Robert, when I posted this I had no experience with lisp images, yet. So the first thing I tried to find was a magic all-in-one command that would spit out a binary executable. I quickly realized that in a single-treaded environment I wouldn't be able to get the application to "talk" to both the UI and the swank, so I dropped the swank loading and presto - it worked. The downside? I couldn't debug my binary application within Emacs. But that's hardly an issue, I know, as I can have SLIME firing up my application properly and the binary doesn't need to do any kind of debugging. Hope I've answered your questions!

Cheers, happy festivities!