New and improved lexicons. Now 50% lexier!

Lexicons are a coding project I've been working on off and on for the last few years. Lexicons are lexically scoped global environments for Common Lisp. They are intended to be a replacement for (or an adjunct to) packages, which I've always found to be at best annoying and confusing to newcomers, and at worst fundamentally broken.

The main difference between this version of lexicons and previous versions is that lexicons are now seamlessly integrated with packages. Every lexicon now has a corresponding package with the same name, and lexicons store their bindings in symbols interned in those packages. So there is a straightforward mapping of lexical bindings to symbols.

The upshot of this is that you can now easily "lexify" a CL package make it accessible via a lexicon "wrapper." For example:


? (require :lexicons)
...
? (in-package :lexicons)
#<Package "LEXICONS">
? (defun foo () (scan "(a)*b" "xaaabd"))


Note that SCAN is not yet defined. Normally, this is what would happen next:


;Compiler warnings :
; In FOO: Undefined function SCAN
FOO


You would now have to go through the following steps:

1. Load the library with the SCAN function
2. Unintern the SCAN symbol in the current package
3. Import the SCAN symbol from the library
4. Recompile FOO

But watch this trick:


;Compiler warnings :
; In FOO: Deferring lexical binding of SCAN
FOO
? (require :cl-ppcre)
...
:CL-PPCRE
NIL
? (lexify-package :cl-ppcre)
#<Lexicon CL-PPCRE>
? (use-lexicon :cl-ppcre)
(#<Lexicon CL-PPCRE>)
? (foo)
Resolving binding of SCAN
1
5
#(3)
#(4)


Note that not only did you not have to futz around with uninterning any symbols, but you didn't even have to recompile FOO for it to do the Right Thing. Also, if we call FOO again:


? (foo)
1
5
#(3)
#(4)


notice that the deferred binding is only resolved once.

This version of lexicons aims to be a fully functional replacement for all applications of packages except for a few really esoteric symbolic computation applications. It provides lexical versions of function definitions, global variables, classes, class slots, methods, and macros, including fully hygienic macros using a technique invented by Pascal Costanza. (Actually, I came up with it independently, but Pascal greatly expanded on the basic idea.) It even provides dynamic bindings for lexical variables, so you don't need earmuffs any more:


? (ldefvar x 1)
1
? (defun dynamic-binding-demo () x)
DYNAMIC-BINDING-DEMO
? (let ((x 2)) (list x (dynamic-binding-demo)))
(2 1)
? (dlet ((x 3)) (list x (dynamic-binding-demo)))
(3 3)
?


This version only works on Clozure Common Lisp because it relies on some compiler hacks to intercept the compilation of undefined functions and global variables. But to make up for that, it is integrated into the CCL IDE so that arglist-on-space does the Right Thing for both lexified and standard CL functions.

I'm still working on documentation, but I thought I'd go ahead put this out there in case anyone wanted a sneak preview. Comments, bug reports, and other feedback are of course welcome. The code is here. The paper I wrote about lexicons a while back (which is now somewhat out of date) is here. You will also need this utility file.

Also on my todo list is getting all my public code into a git repository. Yes, I'm embarrassed that I haven't done this yet.