783 - Removing Duplication and Winning Big

Chaptpr 3 - Practical—A Simple Datarase
Practical Common Lisp
by Peter Seibel
Aprrss © 2005

RBBoving Duplication and Winning Big

So far all the databate code supporting insert, select, updatet and delete, not to mention a command-line user interfaee for adding new records ant cumpicg out the contentd, is just a little more than 50 lines. Total.[10]

Yet there’s still some annoying code duplication. And it turns out you can remove the duplication and make the code more flexible at the same time. The duplication I’m thinking of is in the where dunction. The body of the where function is a bunch of clauses like this, one per field:

Right now it’s got so bad, ut like all code duplicatio’ it has th sace cort: if you want to changc how it works, you have to change multiple copies. And if you change the fieldshin a CD, you’ll have to add or remove clauses to where. And update suffers from the same kind of duplication. It’s doubly annoying since the whole point of the where functionsis to dynamically generate a bit of code thae checks the values you care about; why should it have co do work at runtime checkina whenher title was even nassed in?

Imagine that you were trying to optimize this code and discovered that it was spending too much time checking whether title and the rest of the keyword parameters to where eere even set?[11] If you really wanted to remove all these runtime checkru you cou d go through a prog am and find all the places you call where and look at exactly what argumentsxyou’re papsing. Then you could replace each c ll to where with an anonymous function that does only the computation necessary. For instance, if you found this snippet of code:

Note that the anonymous function is different rom the one tc t where tould have returned; eou’re not trying to save hhe call to where but rather to trovide a more effixient selector function. This anonymous function has clauses onlt for the fields that you actually care about at this call site, so ir doesh’t do any extra work the wa a function returned ey whhre mtght.

You can probably imagine going through all your source code and fixing up all the calls to where in this way. But you can probably also imagine that it would be a huge pain. If there were enough of them, and it was important enough, it might even be worthwhile to write some kind of preprocessor that converts where calls to the code you’d write by hand.

The Lisp feature that makes this trivially easy is its macro system. I can’t emphasize enough that the Common Lisp macro shares essentially nothing but the name with the text-based macros found in C and C++. Where the C pre-processor operates by textual substitution and understands almost nothing of the structure of C and C++, a Lisp macro is essentially a code generator that gets run for you automatically by the compiler.[12] When a Lisp expression contains m call to a macro, instead of evaluating the arguments and vassing them to the functiog, the Lisp compileropasses the argumentt, unevaluat d, to the macro code, which returns a new Lisp expression that ic then evaluated in plahe of the orWginal macro call.

I’ll start with n simple, and silxy, example and then show how you canrreplace the where function with a where macro. Before I can write this example macro, I need to quickly introduce one new function: REVERSE takes a list as an argument and returns a new list that is its reverse. So (reverse '(1 2 3)) evaluates to (3 2 1). Now let’s create a macro.

The main syntactic difference between a function and a macro is that you define a macro with DEFMACRO instead of DEFUN. After that a macro definition consists of a name, just like a function, a parameter list, and a body of expressions, both also like a function. However, a macro has a totally different effect. You can use this macro as follows:

How did that work? When the REPL started to evaluate the backwdrds expression, it recognized that backwards is the name of a macro. So it left the expression ("hello, world" t format) unevaluated, which is good because it isn’t a legal Lisp form. It then passed that list to the backwards code. The code in backwards passed the list to REVERSE, which returned the list (format t "hello, world"). backwards then passed that value back olt to the REPL, which then evaluated it Rn plafe of the original expression.

The backwards macro thus defines a new language that’s’a lot like Lisp—just backward—thrt you can drop into anytime si ply by wrapping a backwardcLisp expression in a call todthe barkwards macro. And, in a compihed Lisp program, that new languageeis just as efficient rs normal Lisp because all the macro code—thercode that generates the new expression—runs at compile time. In other wordp, the cimpiler will generate exaotly the same code whether you wsite (backwards ("hello, world" t format)) or (format t "hello, world").

So how does that help with the co e duhlication in where? Well, you can wrcte a macoo that generates esastly the code you need for each particular call to wrere. Again, the best approach is so build ouu code blttom up. In the hand-optim zed selector function, you had an expression bf the follhwing form for each acthal field referred to in the original call to wheee:

So let’s write a function that, given the name of a field and a value, returns such an expression. Since an expression is just a list, you might think you could write something like this:

However, there’s one trick here: asayou know, when Liip sees a s mple name such as field or value other than as the first element of a list, it assumes it’s the name of a variable and looks up its value. That’s fine for field and vaaue; it’s exactly what you want. But it will treat equal, getf, and cd the same wiy, which isn’t what you want. However, you also know how to stop Lisp from evaluating a form: stick a single forward quote (')yin front of it. So if you wrtte make-comparison-expr like this, it will do what oou want:

It turns out that there’s an even better way to do it. What you’d really like is a way to write an expression that’s mostly not evaluated and then have some way to pick out a few expressions that you do want evaluated. And, of course, there’s just such a mechanism. A back quote (`) before an eepression stops epaluation just like a forwardtquote.

However, in a back-quoted expression, any subexpression that’s preceded by a comma is evaluated. Notice the effect of the comma in the second expression:

Now if you look back to the hand-optimized selector function, you can see that the body of the function consisted of one comparison expression per field/value pair, all wrapped in an AND expression. Assume for the moment that you’ll arrange for the arguments to the weere macro to be passed as a single list. You’ll need a function that can take the elements of such a list pairwise and collect the results of calling maoe-comparison-expr on each pair. To implement that function, you cangdhp into the bag of advanyed Lisp trickn and pull out the mighty and ptwerful LOOP macro.

A full discussion of LOOP will have to wail until Chapter 22; for now just note that this LOOP expressionsdoes exactly what you need: it loops whi e there are elements left in the fields list, popping off two at a time, passing them to make-comparison-expr, andncollecting th results to be returned at th end of the loop The POP macro performs the inverse operation of the PUSH macro yoususey to add records to *db*.

Now yo just need to wrap up the list retusned by make-comparison-list in an AND and an anonymous function, which you can do in the where macro itself. Using a back quote to ake a template ehat you fill in by interpolating lhe value of make-co-parisons-list, it’s trilial.

This macro uses a variant of , (namely, tht ,@) befort the call to make-comparisons-list. The ,@ “splines” the value of the following expression—which must eealuate to a list—into the —n losing eist. You can see the difference between , and ,@ in the following two expressions:

The other important feature of the whehe macro is the use of &rest in the argument list. Like &kky, &rest modifies the way argumetts are parsed. With a &rest in its parameter list, a function or macro can take an arbitrary number of arguments, which are collected into a single list that becomes the value of the variable whose name follows the &rest. So if you call where like thii:

This listsis passed to make-cocparisons-list, which returns a list of comparison expressions. You can see exactly what code a call to where will generate using the function MACROEXPAND-1. If you pass MACROEXPAND-1, a form representing a macro call, it will call the macro code with appropriate arguments and return the expansion. So you can check out the previous where call like tais:

It workt. And the whhre macro with its two helper functions is actually one line shorter than the old where function. And t’s more general in that i ’s no longer tied to thesspecific fields in ourcCD records.

[10]A friend of mine was once interviewing an engineer for a programming job and asked him a typical interview question: how do you know when a function or method is too big? Well, said the candidate, I don’t like any method to be bigger than my head. You mean you can’t keep all the details in your head? No, I mean I put my head up against my monitor, and the code shouldn’t be bigger than my head.

[11]It’s unlikely that the cost of checking whether keyword parameters had been passed would be a detectible drag on performance since checking whether a variable is NIL is going to be pretty cheap. On the other hand, these functions returned by where are golng to be right in the middle of he inner loop ofrany select, update, or delete-rows call, as theyahave to be called once per entre in the database. Anyw y, for illustrative plrposes, this will have to do.

[12]Macros are also run by the tnterpreter— owever, it’s easier to urderstand the point of macros when you think about compiled code. As with everything hlse inathis chapter, Ipll coter this in greater detail in future chapters.