763 - Practical Common Lisp

Practical Common Lisp
by Peter Seibel
Apress © 2005
Progress Indicator Progress Indicator
Next Section

Practical Common Lisp

PETER SEIBEL
Apress

© 2005 by Peter Seibel

  • Lead Editor: Gary Cornell

  • Technical Reviewers: Mikel Evins, Steven Haflich, Barry Margolin

  • Editorial Board: Steve Anglin, Dan Appleman, Ewan Buckingham, Gary Cornell, Tony Davis, Jason Gilmore, Jonathan Hassell, Chris Mills, Dominic Shakeshaft, Jim Sumser

  • Assistant Publisher: Grace Wong

  • Project Manager: Beth Christmas

  • Copy Edit Manager: Nicole LeClerc

  • Copy Editor: Kim Wimpsett

  • Production Manager: Kari Brooks-Copony

  • Production Editor: Ellie Fountain

  • Compositor: Susan Glinert

  • Proofreaders: Katie Stence, Liz Welch

  • Indexer: Kevin Broccoli

  • Cover Designer: Kurt Krames

  • Manufacturing Manager: Tom Debolski

Library of Congress Cataloging-in-Publication Data

Seibel, Peter.

Practical COMMON LISP / Peter Seibel.
p. cm.

Includes index.

ISBN: 1590592395

1. COMMON LISP (Computer program language) I. Title.

QA76.73.L23S45 2005

005.13'3--dc22

2005005859

All rights reserved. No part of this work may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage or retrieval system, without the prior written permission of the copyright owner and the publisher.

Trademarked names may appear in this book. Rather than use a trademark symbol with every occurrence of a trademarked name, we use the names only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark.

Distributed to the book trade in the United States by Springer-Verlag New York, Inc., 233 Spring Street, 6th Floor, New York, NY 10013, and outside the United States by Springer-Verlag GmbH & Co. KG, Tiergartenstr. 17, 69112 Heidelberg, Germany.

In the United States: phone 1-800-SPRINGER, fax 201-348-4505, e-mail orders@springer-ny.com, or visit http://www.springer-ny.com. Outside the United States: fax +49 6221 345229, e-mail orders@springer.de, or visit http://www.springer.de.

For information on translations, please contact Apress directly at 2560 Ninth Street, Suite 219, Berkeley, CA 94710. Phone 510-549-5930, fax 510-549-5939, e-mail info@apress.com, or visit http://www.apress.com.

The information in this book is distributed on an “as is” basis, without warranty. Although every precaution has been taken in the preparation of this work, neither the author(s) nor Apress shall have any liability to any person or entity with respect to any loss or damage caused or alleged to be caused directly or indirectly by the information contained in this work.

The source code for this book is available to readers at http://www.apress.com in the Downloads section and also at http://www.gigamonkeys.com/book/.

Dedication

For Lily, Mom, and Dad

About the Author

Image from book

Peter Seibel is either a writer-turned-programmer or a programmer-turned-writer. After picking up an undergraduate degree in English and working briefly as a journalist, he was seduced by the Web. In the early ’90s he hacked Perl for Mother Jones magazine and Organic Online. He participated in the Java revolution as an early employee at WebLogic and later taught Java programming at the University of California–Berkeley Extension. He’s also one of the few second-generation Lisp programmers on the planet and was a childhood shareholder in Symbolics. He lives in Oakland with his wife, Lily, and their dog, Mahlanie.

About the Technical Reviewer

Barry Margolin taught himself computer programming in high school in the late ’70s, first on DEC PDP-8 time-sharing systems and then on Radio Shack TRS-80 personal computers, and he learned operating system design by reverse engineering these systems. He went to M.I.T., where he learned Lisp programming from Bernie Greenberg, author of the Multics MacLisp Compiler and Multics Emacs (the first Emacs clone to be written in Lisp); David Moon (one of the implementers of ITS Maclisp and a founder of Symbolics); and Alan Bawden (perhaps one of the best Lisp macrologists). After getting his computer science degree, he went to work for the Honeywell Multics development group, maintaining Emacs. When Honeywell discontinued Multics development, he went to Thinking Machines Corporation to maintain their Lisp Machine development environment. Since then, he has worked for Bolt, Beranek, and Newman—which became BBN Planet, then GTE Internetworking, and then Genuity, until being acquired by Level(3)—providing technical support for their Internet services. He’s now working for Symantec providing level-two customer technical support for its enterprise firewall products.

Acknowledgments

This book wouldn’t have been written, at least not by me, if not for a few happy coincidences. So, I have to start by thanking Steven Haflich of Franz, who, after we met at a get-together of Bay Area Lispniks, invited me to lunch with some Franz salespeople where, among other things, we discussed the need for a new Lisp book. Then I have to thank Steve Sears, one of the sales guys at that lunch, who put me in touch with Franz’s president, Fritz Kunze, after Fritz mentioned he was looking for someone to write a Lisp book. And, of course, many thanks to Fritz for convincing Apress to publish a new Lisp book, for deciding I was the right guy to write it, and for providing encouragement and assistance along the way. Thanks also to Sheng-Chuang Wu of Franz, the instrument of much of that assistance.

One of my most indispensable resources while working on the book was the newsgroup comp.lang.lisp. The comp.lang.lisp regulars answered what must have seemed to them an endless stream of questions about various aspects of Lisp and its history. I also turned frequently to the Google archives for the group, a treasure trove of technical expertise. So, thanks to Google for making them available and to all comp.lang.lisp participants past and present for providing the content. In particular, I’d like to recognize two long-time comp.lang.lisp contributors—Barry Margolin, who has been providing tidbits of Lisp history and his own brand of quiet wisdom for as long as I’ve been reading the group; and Kent Pitman, who, in addition to having been one of the principal technical editors of the language standard and the author of the Common Lisp HyperSpec, has written hundreds of thousands, if not millions, of words in comp.lang.lisp postings elucidating various aspects of the language and how it came to be.

Other indispensable resources while working on the book were the Common Lisp libraries for PDF generation and typesetting, CL-PDF and CL-TYPESETTING, written by Marc Battyani. I used CL-TYPESETTING to generate handsome PDFs for my own red-pen editing and CL-PDF as the basis for the Common Lisp program I used to generate the line art that appears in this book.

I also want to thank the many people who reviewed draft chapters on the Web and sent me e-mails pointing out typos, asking questions, or simply wishing me well. While there were too many to mention them all by name, a few deserve special mention for their extensive feedback: J. P. Massar (a fellow Bay Area Lispnik who also bucked up my spirits several times with well-timed pizza lunches), Gareth McCaughan, Chris Riesbeck, Bulent Murtezaoglu, Emre Sevinc, Chris Perkins, and Tayssir John Gabbour. Several of my non-Lisping buddies also got roped into looking at some chapters: thanks to Marc Hedlund, Jolly Chen, Steve Harris, Sam Pullara, Sriram Srinivasan, and William Grosso for their feedback. Thanks also to Scott Whitten for permission to use the photo that appears in Figure 26-1.

My technical reviewers, Steven Haflich, Mikel Evins, and Barry Margolin, and my copy editor, Kim Wimpsett, improved this book in innumerable ways. Any errors that remain are, of course, my own. And thanks to everyone else at Apress who participated in getting this book out the door.

Finally, and most of all, I want to thank my family: Mom and Dad, for everything, and Lily, for always believing I could do it.

Progress Indicator Progress Indicator
Next Section


772 - Experimenting in the REPL

Chapter 2 - Lather, Rinse, Repeat—A Tour of the REPL
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Experimenting in the REPL

To try the REPL, you need a Lisp expression that can be read, evaluated, and printed. One of the simplest kinds of Lisp expressions is a number. At the Lisp prompt, you can type 10 followed by Return and should see something like this:

CL-USER> 10
10

The first 10 is the one you typed. The Lisp reader, the R in REPL, reads the text “10” and creates a Lisp object representing the number 10. This object is a self-evaluating object, which means that when given to the evaluator, the E in REPL, it evaluates to itself. This value is then given to the printer, which prints the 10 on the line by itself. While that may seem like a lot of work just to get back to where you started, things get a bit more interesting when you give Lisp something meatier to chew on. For instance, you can type (+ 2 3) at the Lisp prompt.

CL-USER> (+ 2 3)
5

Anything in parentheses is a list, in this case a list of three elements, the symbol +, and the numbers 2 and 3. Lisp, in general, evaluates lists by treating the first element as the name of a function and the rest of the elements as expressions to be evaluated to yield the arguments to the function. In this case, the symbol + names a function that performs addition. 2 and 3 evaluate to themselves and are then passed to the addition function, which returns 5. The value 5 is passed to the printer, which prints it. Lisp can evaluate a list expression in other ways, but we needn’t get into them right away. First we have to write…

Previous Section


775 - Chapter 3: Practical—A Simple Database

Chapter 3 - Practical—A Simple Database
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Chapter 3: Practical—A Simple Database

Overview

Obviously, before you can start building real software in Lisp, you’ll have to learn the language. But let’s face it—you may be thinking, “‘Practical Common Lisp,’ isn’t that an oxymoron? Why should you be expected to bother learning all the details of a language unless it’s actually good for something you care about?” So I’ll start by giving you a small example of what you can do with Common Lisp. In this chapter you’ll write a simple database for keeping track of CDs. You’ll use similar techniques in Chapter 27 when you build a database of MP3s for our streaming MP3 server. In fact, you could think of this as part of the MP3 software project—after all, in order to have a bunch of MP3s to listen to, it might be helpful to be able to keep track of which CDs you have and which ones you need to rip.

In this chapter, I’ll cover just enough Lisp as we go along for you to understand how the code works. But I’ll gloss over quite a few details. For now you needn’t sweat the small stuff—the next several chapters will cover all the Common Lisp constructs used here, and more, in a much more systematic way.

One terminology note: I’ll discuss a handful of Lisp operators in this chapter. In Chapter 4, you’ll learn that Common Lisp provides three distinct kinds of operators: functions, macros, and special operators. For the purposes of this chapter, you don’t really need to know the difference. I will, however, refer to different operators as functions or macros or special operators as appropriate, rather than trying to hide the details behind the word operator. For now you can treat function, macro, and special operator as all more or less equivalent.[1]

Also, keep in mind that I won’t bust out all the most sophisticated Common Lisp techniques for your very first post–“hello, world” program. The point of this chapter isn’t that this is how you would write a database in Lisp; rather, the point is for you to get an idea of what programming in Lisp is like and to see how even a relatively simple Lisp program can be quite featureful.

[1]Before I proceed, however, it’s crucially important that you forget anything you may know about #define-style “macros” as implemented in the C pre-processor. Lisp macros are a totally different beast.


779 - Improving the User Interaction

Chapter 3 - Practical—A Simple Database
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Improving the User Interaction

While our add-record function works fine for adding records, it’s a bit Lispy for the casual user. And if they want to add a bunch of records, it’s not very convenient. So you may want to write a function to prompt the user for information about a set of CDs. Right away you know you’ll need some way to prompt the user for a piece of information and read it. So let’s write that.

(defun prompt-read (prompt)
  (format *query-io* "~a: " prompt)
  (force-output *query-io*)
  (read-line *query-io*))

You use your old friend FORMAT to emit a prompt. Note that there’s no ~% in the format string, so the cursor will stay on the same line. Thecallto FORCE-OUTPUT is necessary in some implementations to ensure that Lisp doesn’t wait for a newline before it prints the prompt.

Then you can read a single line of text with the aptlynamed READ-LINE function. The variable *query-io* is a global variable (which you can tell because of the * naming convention for global variables) that contains the input stream connected to the terminal. The return value of prompt-read will be the value of the last form, the call to READ-LINE, which returns the string it read (without the trailing newline.)

You can combine your existing make-cd function with prompt-read to build a function that makes a new CD record from data it gets by prompting for each value in turn.

(defun prompt-for-cd ()
  (make-cd
   (prompt-read "Title")
   (prompt-read "Artist")
   (prompt-read "Rating")
   (prompt-read "Ripped [y/n]")))

That’s almost right. Except prompt-read returns a string, which, while fine for the Title and Artist fields, isn’t so great for the Rating and Ripped fields, which should be a number and a boolean. Depending on how sophisticated a user interface you want, you can go to arbitrary lengths to validate the data the user enters. For now let’s lean toward the quick and dirty: you can wrap the prompt-read for the rating in a call to Lisp’s PARSE-INTEGER function, like this:

(parse-integer (prompt-read "Rating"))

Unfortunately, the default behavior of PARSE-INTEGER is to signal an error if it can’t parse an integer out of the string or if there’s any non-numeric junk in the string. However, it takes an optional keyword argument :junk-allowed, which tells it to relax a bit.

(parse-integer (prompt-read "Rating") :junk-allowed t)

But there’s still one problem: if it can’t find an integer amidst all the junk, PARSE-INTEGER will return NIL rather than a number. In keeping with the quick-and-dirty approach, you may just want to call that 0 and continue. Lisp’s OR macro is just the thing you need here. It’s similar to the “short-circuiting” || in Perl, Python, Java, and C; it takes a series of expressions, evaluates them one at a time, and returns the first non-nil value (or NIL if they’re all NIL). So you can use the following:

(or (parse-integer (prompt-read "Rating") :junk-allowed t) 0)

to get a default value of 0.

Fixing the code to prompt for Ripped is quite a bit simpler. You can just use the Common Lisp function Y-OR-N-P.

(y-or-n-p "Ripped [y/n]: ")

In fact, this will be the most robust part of prompt-for-cd, as Y-OR-N-P will reprompt the user if they enter something that doesn’t start with y, Y, n, or N.

Putting those pieces together you get a reasonably robust prompt-for-cd function.

(defun prompt-for-cd ()
  (make-cd
   (prompt-read "Title")
   (prompt-read "Artist")
   (or (parse-integer (prompt-read "Rating") :junk-allowed t) 0)
   (y-or-n-p "Ripped [y/n]: ")))

Finally, you can finish the “add a bunch of CDs” interface by wrapping prompt-for-cd in a function that loops until the user is done. You can use the simple form of the LOOP macro, which repeatedly executes a body of expressions until it’s exited by a call to RETURN. For example:

(defun add-cds ()
  (loop (add-record (prompt-for-cd))
      (if (not (y-or-n-p "Another? [y/n]: ")) (return))))

Now you can use add-cds to add some more CDs to the database.

CL-USER> (add-cds)
Title: Rockin' the Suburbs
Artist: Ben Folds
Rating: 6
Ripped  [y/n]: y
Another?  [y/n]: y
Title: Give Us a Break
Artist: Limpopo
Rating: 10
Ripped  [y/n]: y
Another?  [y/n]: y
Title: Lyle Lovett
Artist: Lyle Lovett
Rating: 9
Ripped  [y/n]: y
Another?  [y/n]: n
NIL


780 - Saving and Loading the Database

Chapter 3 - Practical—A Simple Database
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Saving and Loading the Database

Having a convenient way to add records to the database is nice. But it’s not so nice that the user is going to be very happy if they have to reenter all the records every time they quit and restart Lisp. Luckily, with the data structures you’re using to represent the data, it’s trivially easy to save the data to a file and reload it later. Here’s a save-db function that takes a filename as an argument and saves the current state of the database:

(defun save-db (filename)
  (with-open-file (out filename
                   :direction :output
                   :if-exists :supersede)
    (with-standard-io-syntax
      (print *db* out))))

The WITH-OPEN-FILE macro opens a file, binds the stream to a variable, executes a set of expressions, and then closes the file. It also makes sure the file is closed even if something goes wrong while evaluating the body. The list directly after WITH-OPEN-FILE isn’t a function call but rather part of the syntax defined by WITH-OPEN-FILE. It contains the name of the variable that will hold the file stream to which you’ll write within the body of WITH-OPEN-FILE, a value that must be a file name, and then some options that control how the file is opened. Here you specify that you’re opening the file for writing with :direction :output and that you want to overwrite an existing file of the same name if it exists with :if-exists :supersede.

Once you have the file open, all you have to do is print the contents of the database with (print *db* out). Unlike FORMAT, PRINT prints Lisp objects in a form that can be read back in by the Lisp reader. The macro WITH-STANDARD-IO-SYNTAX ensures that certain variables that affect the behavior of PRINT are set to their standard values. You’ll use the same macro when you read the data back in to make sure the Lisp reader and printer are operating compatibly.

The argument to save-db should be a string containing the name of the file where the user wants to save the database. The exact form of the string will depend on what operating system they’re using. For instance, on a Unix box they should be able to call save-db like this:

CL-USER> (save-db "~/my-cds.db")
((:TITLE "Lyle Lovett" :ARTIST "Lyle Lovett" :RATING 9 :RIPPED T)
 (:TITLE "Give Us a Break" :ARTIST "Limpopo" :RATING 10 :RIPPED T)
 (:TITLE "Rockin' the Suburbs" :ARTIST "Ben Folds" :RATING 6 :RIPPED
  T)
 (:TITLE "Home" :ARTIST "Dixie Chicks" :RATING 9 :RIPPED T)
 (:TITLE "Fly" :ARTIST "Dixie Chicks" :RATING 8 :RIPPED T)
 (:TITLE "Roses" :ARTIST "Kathy Mattea" :RATING 9 :RIPPED T))

On Windows, the filename might be something like “c:/my-cds.db” or “c:\\my-cds.db.”[4]

You can open this file in any text editor to see what it looks like. You should see something a lot like what the REPL prints if you type *db*.

The function to load the database back in is similar.

(defun load-db (filename)
  (with-open-file (in filename)
    (with-standard-io-syntax
      (setf *db* (read in)))))

This time you don’t need to specify :direction in the options to WITH-OPEN-FILE, since you want the default of :input. And instead of printing, you use the function READ to read from the stream in. This is the same reader used by the REPL and can read any Lisp expression you could type at the REPL prompt. However, in this case, you’re just reading and saving the expression, not evaluating it. Again, the WITH-STANDARD-IO-SYNTAX macro ensures that READ is using the same basic syntax that save-db did when it PRINTed the data.

The SETF macro is Common Lisp’s main assignment operator. It sets its first argument to the result of evaluating its second argument. So in load-db the *db* variable will contain the object read from the file, namely, the list of lists written by save-db. You do need to be careful about one thing—load-db clobbers whatever was in *db* before the call. So if you’ve added records with add-record or add-cds that haven’t been saved with save-db, you’ll lose them.

[4]Windows actually understands forward slashes in filenames even though it normally uses a backslash as the directory separator. This is convenient since otherwise you have to write double backslashes because backslash is the escape character in Lisp strings.


782 - Updating Existing Records—Another Use for WHERE

Chapter 3 - Practical—A Simple Database
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Updating Existing Records—Another Use for WHERE

Now that you’ve got nice generalized select and where functions, you’re in a good position to write the next feature that every database needs—a way to update particular records. In SQL the update command is used to update a set of records matching a particular where clause. That seems like a good model, especially since you’ve already got a where-clause generator. In fact, the update function is mostly just the application of a few ideas you’ve already seen: using a passed-in selector function to choose the records to update and using keyword arguments to specify the values to change. The main new bit is the use of a function MAPCAR that maps over a list, *db* in this case, and returns a new list containing the results of calling a function on each item in the original list.

(defun update (selector-fn &key title artist rating (ripped nil ripped-p))
  (setf *db*
        (mapcar
         #'(lambda (row)
             (when (funcall selector-fn row)
               (if title    (setf (getf row :title) title))
               (if artist   (setf (getf row :artist) artist))
               (if rating   (setf (getf row :rating) rating))
               (if ripped-p (setf (getf row :ripped) ripped)))
             row) *db*)))

One other new bit here is the use of SETF on a complex form such as (getf row :title). I’ll discuss SETF in greater detail in Chapter 6, but for now you just need to know that it’s a general assignment operator that can be used to assign lots of “places” other than just variables. (It’s a coincidence that SETF and GETF have such similar names—they don’t have any special relationship.) For now it’s enough to know that after (setf (getf row :title) title), the plist referenced by row will have the value of the variable title following the property name :title. With this update function if you decide that you really dig the Dixie Chicks and that all their albums should go to 11, you can evaluate the following form:

CL-USER> (update (where :artist "Dixie Chicks") :rating 11)
NIL

And it is so.

CL-USER> (select (where :artist "Dixie Chicks"))
((:TITLE "Home" :ARTIST "Dixie Chicks" :RATING 11 :RIPPED T)
 (:TITLE "Fly" :ARTIST "Dixie Chicks" :RATING 11 :RIPPED T))

You can even more easily add a function to delete rows from the database.

(defun delete-rows (selector-fn)
  (setf *db* (remove-if selector-fn *db*)))

The function REMOVE-IF is the complement of REMOVE-IF-NOT; it returns a list with all the elements that do match the predicate removed. Like REMOVE-IF-NOT, it doesn’t actually affect the list it’s passed but by saving the result back into *db*, delete-rows[8] actually changes the contents of the database.[9]

[8]You need to use the name delete-rows rather than the more obvious delete because there’s already a function in Common Lisp called DELETE. The Lisp package system gives you a way to deal with such naming conflicts, so you could have a function named delete if you wanted. But I’m not ready to explain packages just yet.

[9]If you’re worried that this code creates a memory leak, rest assured: Lisp was the language that invented garbage collection (and heap allocation for that matter). The memory used by the old value of *db* will be automatically reclaimed, assuming no one else is holding on to a reference to it, which none of this code is.


786 - Breaking Open the Black Box

Chapter 4 - Syntax and Semantics
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Breaking Open the Black Box

Before we look at the specifics of Lisp’s syntax and semantics, it’s worth taking a moment to look at how they’re defined and how this differs from many other languages.

In most programming languages, the language processor—whether an interpreter or a compiler—operates as a black box: you shove a sequence of characters representing the text of a program into the black box, and it—depending on whether it’s an interpreter or a compiler—either executes the behaviors indicated or produces a compiled version of the program that will execute the behaviors when it’s run.

Inside the black box, of course, language processors are usually divided into subsystems that are each responsible for one part of the task of translating a program text into behavior or object code. A typical division is to split the processor into three phases, each of which feeds into the next: a lexical analyzer breaks up the stream of characters into tokens and feeds them to a parser that builds a tree representing the expressions in the program, according to the language’s grammar. This tree—called an abstract syntax tree—is then fed to an evaluator that either interprets it directly or compiles it into some other language such as machine code. Because the language processor is a black box, the data structures used by the processor, such as the tokens and abstract syntax trees, are of interest only to the language implementer.

In Common Lisp things are sliced up a bit differently, with consequences for both the implementer and for how the language is defined. Instead of a single black box that goes from text to program behavior in one step, Common Lisp defines two black boxes, one that translates text into Lisp objects and another that implements the semantics of the language in terms of those objects. The first box is called the reader, and the second is called the evaluator.[2]

Each black box defines one level of syntax. The reader defines how strings of characters can be translated into Lisp objects called s-expressions.[3] Since the s-expression syntax includes syntax for lists of arbitrary objects, including other lists, s-expressions can represent arbitrary tree expressions, much like the abstract syntax tree generated by the parsers for non-Lisp languages.

The evaluator then defines a syntax of Lisp forms that can be built out of s-expressions. Not all s-expressions are legal Lisp forms any more than all sequences of characters are legal s-expressions. For instance, both (foo 1 2) and ("foo" 1 2) are s-expressions, but only the former can be a Lisp form since a list that starts with a string has no meaning as a Lisp form.

This split of the black box has a couple of consequences. One is that you can use s-expressions, as you saw in Chapter 3, as an externalizable data format for data other than source code, using READ to read it and PRINT to print it.[4] The other consequence is that since the semantics of the language are defined in terms of trees of objects rather than strings of characters, it’s easier to generate code within the language than it would be if you had to generate code as text. Generating code completely from scratch is only marginally easier—building up lists vs. building up strings is about the same amount of work. The real win, however, is that you can generate code by manipulating existing data. This is the basis for Lisp’s macros, which I’ll discuss in much more detail in future chapters. For now I’ll focus on the two levels of syntax defined by Common Lisp: the syntax of s-expressions understood by the reader and the syntax of Lisp forms understood by the evaluator.

[2]Lisp implementers, like implementers of any language, have many ways they can implement an evaluator, ranging from a “pure” interpreter that interprets the objects given to the evaluator directly to a compiler that translates the objects into machine code that it then runs. In the middle are implementations that compile the input into an intermediate form such as bytecodes for a virtual machine and then interprets the bytecodes. Most Common Lisp implementations these days use some form of compilation even when evaluating code at run time.

[3]Sometimes the phrase s-expression refers to the textual representation and sometimes to the objects that result from reading the textual representation. Usually either it’s clear from context which is meant or the distinction isn’t that important.

[4]Not all Lisp objects can be written out in a way that can be read back in. But anything you can READ can be printed back out “readably” with PRINT.


788 - S-expressions As Lisp Forms

Chapter 4 - Syntax and Semantics
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

S-expressions As Lisp Forms

After the reader has translated a bunch of text into s-expressions, the s-expressions can then be evaluated as Lisp code. Or some of them can—not every s-expressions that the reader can read can necessarily be evaluated as Lisp code. Common Lisp’s evaluation rule defines a second level of syntax that determines which s-expressions can be treated as Lisp forms.[9] The syntactic rules at this level are quite simple. Any atom—any nonlist or the empty list—is a legal Lisp form as is any list that has a symbol as its first element.[10]

Of course, the interesting thing about Lisp forms isn’t their syntax but how they’re evaluated. For purposes of discussion, you can think of the evaluator as a function that takes as an argument a syntactically well-formed Lisp form and returns a value, which we can call the value of the form. Of course, when the evaluator is a compiler, this is a bit of a simplification—in that case, the evaluator is given an expression and generates code that will compute the appropriate value when it’s run. But this simplification lets me describe the semantics of Common Lisp in terms of how the different kinds of Lisp forms are evaluated by this notional function.

The simplest Lisp forms, atoms, can be divided into two categories: symbols and everything else. A symbol, evaluated as a form, is considered the name of a variable and evaluates to the current value of the variable.[11] I’ll discuss in Chapter 6 how variables get their values in the first place. You should also note that certain “variables” are that old oxymoron of programming: “constant variables.” For instance, the symbol PI names a constant variable whose value is the best possible floating-point approximation to the mathematical constant π.

All other atoms—numbers and strings are the kinds you’ve seen so far—are self-evaluating objects. This means when such an expression is passed to the notional evaluation function, it’s simply returned. You saw examples of self-evaluating objects in Chapter 2 when you typed 10 and "hello, world" at the REPL.

It’s also possible for symbols to be self-evaluating in the sense that the variables they name can be assigned the value of the symbol itself. Two important constants that are defined this way are T and NIL, the canonical true and false values. I’ll discuss their role as booleans in the section “Truth, Falsehood, and Equality.”

Another class of self-evaluating symbols are the keyword symbols—symbols whose names start with :. When the reader interns such a name, it automatically defines a constant variable with the name and with the symbol as the value.

Things get more interesting when we consider how lists are evaluated. All legal list forms start with a symbol, but three kinds of list forms are evaluated in three quite different ways. To determine what kind of form a given list is, the evaluator must determine whether the symbol that starts the list is the name of a function, a macro, or a special operator. If the symbol hasn’t been defined yet—as may be the case if you’re compiling code that contains references to functions that will be defined later—it’s assumed to be a function name.[12] I’ll refer to the three kinds of forms as function call forms, macro forms, and special forms.

[9]Of course, other levels of correctness exist in Lisp, as in other languages. For instance, the s-expression that results from reading (foo 1 2) is syntactically well-formed but can be evaluated only if foo is the name of a function or macro.

[10]One other rarely used kind of Lisp form is a list whose first element is a lambda form. I’ll discuss this kind of form in Chapter 5.

[11]One other possibility exists—it’s possible to define symbol macros that are evaluated slightly differently. We won’t worry about them.

[12]In Common Lisp a symbol can name both an operator—function, macro, or special operator—and a variable. This is one of the major differences between Common Lisp and Scheme. The difference is sometimes described as Common Lisp being a Lisp-2 vs. Scheme being a Lisp-1—a Lisp-2 has two namespaces, one for operators and one for variables, but a Lisp-1 uses a single namespace. Both choices have advantages, and partisans can debate endlessly which is better.


789 - Function Calls

Chapter 4 - Syntax and Semantics
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Function Calls

The evaluation rule for function call forms is simple: evaluate the remaining elements of the list as Lisp forms and pass the resulting values to the named function. This rule obviously places some additional syntactic constraints on a function call form: all the elements of the list after the first must themselves be well-formed Lisp forms. In other words, the basic syntax of a function call form is as follows, where each of the arguments is itself a Lisp form:

(function-name argument*)

Thus, the following expression is evaluated by first evaluating 1, then evaluating 2, and then passing the resulting values to the + function, which returns 3:

(+ 1 2)

A more complex expression such as the following is evaluated in similar fashion except that evaluating the arguments (+ 1 2) and (− 3 4) entails first evaluating their arguments and applying the appropriate functions to them:

(* (+ 1 2) (- 3 4))

Eventually, the values 3 and –1 are passed to the * function, which returns –3.

As these examples show, functions are used for many of the things that require special syntax in other languages. This helps keep Lisp’s syntax regular.


790 - Special Operators

Chapter 4 - Syntax and Semantics
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Special Operators

That said, not all operations can be defined as functions. Because all the arguments to a function are evaluated before the function is called, there’s no way to write a function that behaves like the IF operator you used in Chapter 3. To see why, consider this form:

(if x (format t "yes") (format t "no"))

If IF were a function, the evaluator would evaluate the argument expressions from left to right. The symbol x would be evaluated as a variable yielding some value; then (format t "yes") would be evaluated as a function call, yielding NIL after printing “yes” to standard output. Then (format t "no") would be evaluated, printing “no” and also yielding NIL. Only after all three expressions were evaluated would the resulting values be passed to IF, too late for it to control which of the two FORMAT expressions gets evaluated.

To solve this problem, Common Lisp defines a couple dozen so-called special operators, IF being one, that do things that functions can’t do. There are 25 in all, but only a small handful are used directly in day-to-day programming.[13]

When the first element of a list is a symbol naming a special operator, the rest of the expressions are evaluated according to the rule for that operator.

The rule for IF is pretty easy: evaluate the first expression. If it evaluates to non-NIL, then evaluate the next expression and return its value. Otherwise, return the value of evaluating the third expression or NIL if the third expression is omitted. In other words, the basic form of an IF expression is as follows:

(if test-form then-form [ else-form ])

The test-form will always be evaluated and then one or the other of the then-form or else-form.

An even simpler special operator is QUOTE, which takes a single expression as its “argument” and simply returns it, unevaluated. For instance, the following evaluates to the list (+ 1 2), not the value 3:

(quote (+ 1 2))

There’s nothing special about this list; you can manipulate it just like any list you could create with the LIST function.[14]

QUOTE is used commonly enough that a special syntax for it is built into the reader. Instead of writing the following:

(quote (+ 1 2))

you can write this:

'(+ 1 2)

This syntax is a small extension of the s-expression syntax understood by the reader. From the point of view of the evaluator, both those expressions will look the same: a list whose first element is the symbol QUOTE and whose second element is the list (+ 1 2).[15]

In general, the special operators implement features of the language that require some special processing by the evaluator. For instance, several special operators manipulate the environment in which other forms will be evaluated. One of these, which I’ll discuss in detail in Chapter 6, is LET, which is used to create new variable bindings. The following form evaluates to 10 because the second x is evaluated in an environment where it’s the name of a variable established by the LET with the value 10:

(let ((x 10)) x)

[13]The others provide useful, but somewhat esoteric, features. I’ll discuss them as the features they support come up.

[14]Well, one difference exists—literal objects such as quoted lists, but also including double-quoted strings, literal arrays, and vectors (whose syntax you’ll see later), must not be modified. Consequently, any lists you plan to manipulate you should create with LIST.

[15]This syntax is an example of a reader macro. Reader macros modify the syntax the reader uses to translate text into Lisp objects. It is, in fact, possible to define your own reader macros, but that’s a rarely used facility of the language. When most Lispers talk about “extending the syntax” of the language, they’re talking about regular macros, as I’ll discuss in a moment.


792 - Truth, Falsehood, and Equality

Chapter 4 - Syntax and Semantics
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Truth, Falsehood, and Equality

Two last bits of basic knowledge you need to get under your belt are Common Lisp’s notion of truth and falsehood and what it means for two Lisp objects to be “equal.” Truth and falsehood are—in this realm—straightforward: the symbol NIL is the only false value, and everything else is true. The symbol T is the canonical true value and can be used when you need to returna non-NIL value and don’t have anything else handy. The only tricky thing about NIL is that it’s the only object that’s both an atom and a list: in addition to falsehood, it’s also used to represent the empty list.[17] This equivalence between NIL and the empty list is built into the reader: if the reader sees (), it reads it as the symbol NIL. They’re completely interchangeable. And because NIL, as I mentioned previously, is the name of a constant variable with the symbol NIL as its value, the expressions nil, (), 'nil, and '() all evaluate to the same thing—the unquoted forms are evaluated as a reference to the constant variable whose value is the symbol NIL, but in the quoted forms the QUOTE special operator evaluates to the symbol directly. For the same reason, both t and 't will evaluate to the same thing: the symbol T.

Using phrases such as “the same thing” of course begs the question of what it means for two values to be “the same.” As you’ll see in future chapters, Common Lisp provides a number of type-specific equality predicates: = is used to compare numbers, CHAR= to compare characters, and so on. In this section I’ll discuss the four “generic” equality predicates—functions that can be passed any two Lisp objects and will return true if they’re equivalent and false otherwise. They are, in order of discrimination, EQ, EQL, EQUAL, and EQUALP.

EQ tests for “object identity”—two objects are EQ if they’re identical. Unfortunately, the object identity of numbers and characters depends on how those data types are implemented in a particular Lisp. Thus, EQ may consider two numbers or two characters with the same value to be equivalent, or it may not. Implementations have enough leeway that the expression (eq 3 3) can legally evaluate to either true or false. More to the point, (eq x x) can evaluate to either true or false if the value of x happens to be a number or character.

Thus, you should never use EQ to compare values that may be numbers or characters. It may seem to work in a predictable way for certain values in a particular implementation, but you have no guarantee that it will work the same way if you switch implementations. And switching implementations may mean simply upgrading your implementation to a new version—if your Lisp implementer changes how they represent numbers or characters, the behavior of EQ could very well change as well.

Thus, Common Lisp defines EQL to behave like EQ except that it also is guaranteed to consider two objects of the same class representing the same numeric or character value to be equivalent. Thus, (eql 1 1) is guaranteed to be true. And (eql 1 1.0) is guaranteed to be false since the integer value 1 and the floating-point value are instances of different classes.

There are two schools of thought about when to use EQ and when to use EQL: The “use EQ when possible” camp argues you should use EQ when you know you aren’t going to be comparing numbers or characters because (a) it’s a way to indicate that you aren’t going to be comparing numbers or characters and (b) it will be marginally more efficient since EQ doesn’t have to check whether its arguments are numbers or characters.

The “always use EQL” camp says you should never use EQ because (a) the potential gain in clarity is lost because every time someone reading your code—including you—sees an EQ, they have to stop and check whether it’s being used correctly (in other words, that it’s never going to be called upon to compare numbers or characters) and (b) that the efficiency difference between EQ and EQL is in the noise compared to real performance bottlenecks.

The code in this book is written in the “always use EQL” style.[18]

The other two equality predicates, EQUAL and EQUALP, are general in the sense that they can operate on all types of objects, but they’re much less fundamental than EQ or EQL. They each define a slightly less discriminating notion of equivalence than EQL, allowing different objects to be considered equivalent. There’s nothing special about the particular notions of equivalence these functions implement except that they’ve been found to be handy by Lisp programmers in the past. If these predicates don’t suit your needs, you can always define your own predicate function that compares different types of objects in the way you need.

EQUAL loosens the discrimination of EQL to consider lists equivalent if they have the same structure and contents, recursively, according to EQUAL. EQUAL also considers strings equivalent if they contain the same characters. It also defines a looser definition of equivalence than EQL for bit vectors and pathnames, two data types I’ll discuss in future chapters. For all other types, it falls back on EQL.

EQUALP is similar to EQUAL except it’s even less discriminating. It considers two strings equivalent if they contain the same characters, ignoring differences in case. It also considers two characters equivalent if they differ only in case. Numbers are equivalent under EQUALP if they represent the same mathematical value. Thus, (equalp 1 1.0) is true. Lists with EQUALP elements are EQUALP; likewise, arrays with EQUALP elements are EQUALP. As with EQUAL, there are a few other data types that I haven’t covered yet for which EQUALP can consider two objects equivalent that neither EQL nor EQUAL will. For all other data types, EQUALP falls back on EQL.

[17]Using the empty list as false is a reflection of Lisp’s heritage as a list-processing language much as the use of the integer 0 as false in C is a reflection of its heritage as a bit-twiddling language. Not all Lisps handle boolean values the same way. Another of the many subtle differences upon which a good Common Lisp vs. Scheme flame war can rage for days is Scheme’s use of a distinct false value #f, which isn’t the same value as either the symbol nil or the empty list, which are also distinct from each other.

[18]Even the language standard is a bit ambivalent about which of EQ or EQL should be preferred. Object identity is defined by EQ, but the standard defines the phrase the same when talking about objects to mean EQL unless another predicate is explicitly mentioned. Thus, if you want to be 100 percent technically correct, you can say that (- 3 2) and (- 4 3) evaluate to “the same” object but not that they evaluate to “identical” objects. This is, admittedly, a bit of an angels-on-pinheads kind of issue.


795 - Defining New Functions

Chapter 5 - Functions
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Defining New Functions

Normally functions are defined using the DEFUN macro. The basic skeleton of a DEFUN looks like this:

(defun name (parameter*)
  "Optional documentation string."
  body-form*)

Any symbol can be used as a function name.[2] Usually function names contain only alphabetic characters and hyphens, but other characters are allowed and are used in certain naming conventions. For instance, functions that convert one kind of value to another sometimes use -> in the name. For example, a function to convert strings to widgets might be called string->widget. The most important naming convention is the one mentioned in Chapter 2, which is that you construct compound names with hyphens rather than underscores or inner caps. Thus, frob-widget is better Lisp style than either frob_widget or frobWidget.

A function’s parameter list defines the variables that will be used to hold the arguments passed to the function when it’s called.[3] If the function takes no arguments, the list is empty, written as (). Different flavors of parameters handle required, optional, multiple, and keyword arguments. I’ll discuss the details in the next section.

If a string literal follows the parameter list, it’s a documentation string that should describe the purpose of the function. When the function is defined, the documentation string will be associated with the name of the function and can later be obtained using the DOCUMENTATION function.[4]

Finally, the body of a DEFUN consists of any number of Lisp expressions. They will be evaluated in order when the function is called and the value of the last expression is returned as the value of the function. Or the RETURN-FROM special operator can be used to return immediately from anywhere in a function, as I’ll discuss in a moment.

In Chapter 2 we wrote a hello-world function, which looked like this:

(defun hello-world () (format t "hello, world"))

You can now analyze the parts of this function. Its name is hello-world, its parameter list is empty so it takes no arguments, it has no documentation string, and its body consists of one expression.

(format t "hello, world")

The following is a slightly more complex function:

(defun verbose-sum (x y)
  "Sum any two numbers after printing a message."
  (format t "Summing ~d and ~d.~%" x y)
  (+ x y))

This function is named verbose-sum, takes two arguments that will be bound to the parameters x and y, has a documentation string, and has a body consisting of two expressions. The value returned by the call to + becomes the return value of verbose-sum.

[2]Well, almost any symbol. It’s undefined what happens if you use any of the names defined in the language standard as a name for one of your own functions. However, as you’ll see in Chapter 21, the Lisp package system allows you to create names in different namespaces, so this isn’t really an issue.

[3]Parameter lists are sometimes also called lambda lists because of the historical relationship between Lisp’s notion of functions and the lambda calculus.

[4]For example, the following:

(documentation 'foo 'function)

returns the documentation string for the function foo. Note, however, that documentation strings are intended for human consumption, not programmatic access. A Lisp implementation isn’t required to store them and is allowed to discard them at any time, so portable programs shouldn’t depend on their presence. In some implementations an implementation-defined variable needs to be set before it will store documentation strings.


799 - Keyword Parameters

Chapter 5 - Functions
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Keyword Parameters

Optional and rest parameters give you quite a bit of flexibility, but neither is going to help you out much in the following situation: Suppose you have a function that takes four optional parameters. Now suppose that most of the places the function is called, the caller wants to provide a value for only one of the four parameters and, further, that the callers are evenly divided as to which parameter they will use.

The callers who want to provide a value for the first parameter are fine—they just pass the one optional argument and leave off the rest. But all the other callers have to pass some value for between one and three arguments they don’t care about. Isn’t that exactly the problem optional parameters were designed to solve?

Of course it is. The problem is that optional parameters are still positional—if the caller wants to pass an explicit value for the fourth optional parameter, it turns the first three optional parameters into required parameters for that caller. Luckily, another parameter flavor, keyword parameters, allow the caller to specify which values go with which parameters.

To give a function keyword parameters, after any required, &optional, and &rest parameters you include the symbol &key and then any number of keyword parameter specifiers, which work like optional parameter specifiers. Here’s a function that has only keyword parameters:

(defun foo (&key a b c) (list a b c))

When this function is called, each keyword parameters is bound to the value immediately following a keyword of the same name. Recall from Chapter 4 that keywords are names that start with a colon and that they’re automatically defined as self-evaluating constants.

If a given keyword doesn’t appear in the argument list, then the corresponding parameter is assigned its default value, just like an optional parameter. Because the keyword arguments are labeled, they can be passed in any order as long as they follow any required arguments. For instance, foo can be invoked as follows:

(foo)                → (NIL NIL NIL)
(foo :a 1)           → (1 NIL NIL)
(foo :b 1)           → (NIL 1 NIL)
(foo :c 1)           → (NIL NIL 1)
(foo :a 1 :c 3)      → (1 NIL 3)
(foo :a 1 :b 2 :c 3) → (1 2 3)
(foo :a 1 :c 3 :b 2) → (1 2 3)

As with optional parameters, keyword parameters can provide a default value form and the name of a supplied-p variable. In both keyword and optional parameters, the default value form can refer to parameters that appear earlier in the parameter list.

(defun foo (&key (a 0) (b 0 b-supplied-p) (c (+ a b)))
  (list a b c b-supplied-p))
(foo :a 1)           →   (1 0 1 NIL)
(foo :b 1)           →   (0 1 1 T)
(foo :b 1 :c 4)      →   (0 1 4 T)
(foo :a 2 :b 1 :c 4) →   (2 1 4 T)

Also, if for some reason you want the keyword the caller uses to specify the parameter to be different from the name of the actual parameter, you can replace the parameter name with another list containing the keyword to use when calling the function and the name to be used for the parameter. The following definition of foo:

(defun foo (&key ((:apple a)) ((:box b) 0) ((:charlie c) 0 c-supplied-p))
  (list a b c c-supplied-p))

lets the caller call it like this:

(foo :apple 10 :box 20 :charlie 30) → (10 20 30 T)

This style is mostly useful if you want to completely decouple the public API of the function from the internal details, usually because you want to use short variable names internally but descriptive keywords in the API. It’s not, however, very frequently used.


803 - Anonymous Functions

Chapter 5 - Functions
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Anonymous Functions

Once you start writing, or even simply using, functions that accept other functions as arguments, you’re bound to discover that sometimes it’s annoying to have to define and name a whole separate function that’s used in only one place, especially when you never call it by name.

When it seems like overkill to define a new function with DEFUN, you can create an “anonymous” function using a LAMBDA expression. As discussed in Chapter 3, a LAMBDA expression looks like this:

(lambda (parameters) body)

One way to think of LAMBDA expressions is as a special kind of function name where the name itself directly describes what the function does. This explains why you can use a LAMBDA expression in the place of a function name with #'.

(funcall #'(lambda (x y) (+ x y)) 2 3) → 5

You can even use a LAMBDA expression as the “name” of a function in a function call expression. If you wanted, you could write the previous FUNCALL expression more concisely.

((lambda (x y) (+ x y)) 2 3) → 5

But this is almost never done; it’s merely worth noting that it’s legal in order to emphasize that LAMBDA expressions can be used anywhere a normal function name can be.[13]

Anonymous functions can be useful when you need to pass a function as an argument to another function and the function you need to pass is simple enough to express inline. For instance, suppose you wanted to plot the function 2x. You could define the following function:

(defun double (x) (* 2 x))

which you could then pass to plot.

CL-USER> (plot #'double 0 10 1)
**
****
******
********
**********
************
**************
****************
******************
********************
NIL

But it’s easier, and arguably clearer, to write this:

CL-USER> (plot #'(lambda (x) (* 2 x)) 0 10 1)
**
****
******
********
**********
************
**************
****************
******************
********************
NIL

The other important use of LAMBDA expressions is in making closures, functions that capture part of the environment where they’re created. You used closures a bit in Chapter 3, but the details of how closures work and what they’re used for is really more about how variables work than functions, so I’ll save that discussion for the next chapter.

[13]In Common Lisp it’s also possible to use a LAMBDA expression as an argument to FUNCALL (or some other function that takes a function argument such as SORT or MAPCAR) with no #' before it, like this:

(funcall (lambda (x y) (+ x y)) 2 3)

This is legal and is equivalent to the version with the #' but for a tricky reason. Historically, LAMBDA expressions by themselves weren’t expressions that could be evaluated. That is, LAMBDA wasn’t the name of a function, macro, or special operator. Rather, a list starting with the symbol LAMBDA was a special syntactic construct that Lisp recognized as a kind of function name.

But if that were still true, then (funcall (lambda (...) ...)) would be illegal because FUNCALL is a function and the normal evaluation rule for a function call would require that the LAMBDA expression be evaluated. However, late in the ANSI standardization process, in order to make it possible to implement ISLISP, another Lisp dialect being standardized at the same time, strictly as a user-level compatibility layer on top of Common Lisp, a LAMBDA macro was defined that expands into a call to FUNCTION wrapped around the LAMBDA expression. In other words, the following LAMBDA expression:

(lambda () 42)

expands into the following when it occurs in a context where it’s evaluated:

(function (lambda () 42))   ; or #'(lambda () 42)

This makes its use in a value position, such as an argument to FUNCALL, legal. In other words, it’s pure syntactic sugar. Most folks either always use #' before LAMBDA expressions in value positions or never do. In this book, I always use #'.


805 - Lexical Variables and Closures

Chapter 6 - Variables
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Lexical Variables and Closures

By default all binding forms in Common Lisp introduce lexically scoped variables. Lexically scoped variables can be referred to only by code that’s textually within the binding form. Lexical scoping should be familiar to anyone who has programmed in Java, C, Perl, or Python since they all provide lexically scoped “local” variables. For that matter, Algol programmers should also feel right at home, as Algol first introduced lexical scoping in the 1960s.

However, Common Lisp’s lexical variables are lexical variables with a twist, at least compared to the original Algol model. The twist is provided by the combination of lexical scoping with nested functions. By the rules of lexical scoping, only code textually within the binding form can refer to a lexical variable. But what happens when an anonymous function contains a reference to a lexical variable from an enclosing scope? For instance, in this expression:

(let ((count 0)) #'(lambda () (setf count (1+ count))))

the reference to count inside the LAMBDA form should be legal according to the rules of lexical scoping. Yet the anonymous function containing the reference will be returned as the value of the LET form and can be invoked, via FUNCALL, by code that’s not in the scope of the LET. So what happens? As it turns out, when count is a lexical variable, it just works. The binding of count created when the flow of control entered the LET form will stick around for as long as needed, in this case for as long as someone holds onto a reference to the function object returned by the LET form. The anonymous function is called a closure because it “closes over” the binding created by the LET.

The key thing to understand about closures is that it’s the binding, not the value of the variable, that’s captured. Thus, a closure can not only access the value of the variables it closes over but can also assign new values that will persist between calls to the closure. For instance, you can capture the closure created by the previous expression in a global variable like this:

(defparameter *fn* (let ((count 0)) #'(lambda () (setf count (1+ count)))))

Then each time you invoke it, the value of count will increase by one.

CL-USER> (funcall *fn*)
1
CL-USER> (funcall *fn*)
2
CL-USER> (funcall *fn*)
3

A single closure can close over many variable bindings simply by referring to them. Or multiple closures can capture the same binding. For instance, the following expression returns a list of three closures, one that increments the value of the closed over count binding, one that decrements it, and one that returns the current value:

(let ((count 0))
  (list
   #'(lambda () (incf count))
   #'(lambda () (decf count))
   #'(lambda () count)))


807 - Constants

Chapter 6 - Variables
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Constants

One other kind of variable I haven’t mentioned at all is the oxymoronic “constant variable.” All constants are global and are defined with DEFCONSTANT. The basic form of DEFCONSTANT is like DEFPARAMETER.

(defconstant name initial-value-form [ documentation-string ])

As with DEFVAR and DEFPARAMETER, DEFCONSTANT has a global effect on the name used—thereafter the name can be used only to refer to the constant; it can’t be used as a function parameter or rebound with any other binding form. Thus, many Lisp programmers follow a naming convention of using names starting and ending with + for constants. This convention is somewhat less universally followed than the *-naming convention for globally special names but is a good idea for the same reason.[14]

Another thing to note about DEFCONSTANT is that while the language allows you to redefine a constant by reevaluating a DEFCONSTANT with a different initial-value-form, what exactly happens after the redefinition isn’t defined. In practice, most implementations will require you to reevaluate any code that refers to the constant in order to see the new value since the old value may well have been inlined. Consequently, it’s a good idea to use DEFCONSTANT only to define things that are really constant, such as the value of π. For things you might ever want to change, you should use DEFPARAMETER instead.

[14]Several key constants defined by the language itself don’t follow this convention—not least of which are T and NIL. This is occasionally annoying when one wants to use t as a local variable name. Another is PI, which holds the best long-float approximation of the mathematical constant π.


809 - Generalized Assignment

Chapter 6 - Variables
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Generalized Assignment

Variable bindings, of course, aren’t the only places that can hold values. Common Lisp supports composite data structures such as arrays, hash tables, and lists, as well as user-defined data structures, all of which consist of multiple places that can each hold a value.

I’ll cover those data structures in future chapters, but while we’re on the topic of assignment, you should note that SETF can assign any place a value. As I cover the different composite data structures, I’ll point out which functions can serve as “SETFable places.” The short version, however, is if you need to assign a value to a place, SETF is almost certainly the tool to use. It’s even possible to extend SETF to allow it to assign to user-defined places though I won’t cover that.[16]

In this regard SETF is no different from the = assignment operator in most C-derived languages. In those languages, the = operator assigns new values to variables, array elements, and fields of classes. In languages such as Perl and Python that support hash tables as a built-in data type, = can also set the values of individual hash table entries. Table 6-1 summarizes the various ways = is used in those languages.

Table 6-1: Assignment with = in Other Languages
Assigning to… | Java, C, C++ | Perl | Python
…variable | x = 10; | $x = 10; | x = 10
…array element | a[0] = 10; | $a[0] = 10; | a[0] = 10
…hash table entry |  | $hash{'key'} = 10; | hash['key'] = 10
…field in object | o.field = 10; | $o->{'field'} = 10; | o.field = 10

SETF works the same way—the first “argument” to SETF is a place to store the value, and the second argument provides the value. As with the = operator in these languages, you use the same form to express the place as you’d normally use to fetch the value.[17] Thus, the Lisp equivalents of the assignments in Table 6-1—given that AREF is the array access function, GETHASH does a hash table lookup, and field might be a function that accesses a slot named field of a user-defined object—are as follows:

Simple variable:    (setf x 10)
Array:              (setf (aref a 0) 10)
Hash table:         (setf (gethash 'key hash) 10)
Slot named 'field': (setf (field o) 10)

Note that SETFing a place that’s part of a larger object has the same semantics as SETFing a variable: the place is modified without any effect on the object that was previously stored in the place. Again, this is similar to how = behaves in Java, Perl, and Python.[18]

[16]Look up DEFSETF, DEFINE-SETF-EXPANDER for more information.

[17]The prevalence of Algol-derived syntax for assignment with the “place” on the left side of the = and the new value on the right side has spawned the terminology lvalue, short for “left value,” meaning something that can be assigned to, and rvalue, meaning something that provides a value. A compiler hacker would say, “SETF treats its first argument as an lvalue.”

[18]C programmers may want to think of variables and other places as holding a pointer to the real object; assigning to a variable simply changes what object it points to while assigning to a part of a composite object is similar to indirecting through the pointer to the actual object. C++ programmers should note that the behavior of = in C++ when dealing with objects—namely, a memberwise copy—is quite idiosyncratic.


812 - WHEN and UNLESS

Chapter 7 - Macros—Standard Control Constructs
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

WHEN and UNLESS

As you’ve already seen, the most basic form of conditional execution—if x, do y; otherwise do z—is provided by the IF special operator, which has this basic form:

(if condition then-form [else-form])

The condition is evaluated and, if its value is non-NIL, the then-form is evaluated and the resulting value returned. Otherwise, the else-form, if any, is evaluated and its value returned. If condition is NIL and there’s no else-form, then the IF returns NIL.

(if (> 2 3) "Yup" "Nope") → "Nope"
(if (> 2 3) "Yup")        → NIL
(if (> 3 2) "Yup" "Nope") → "Yup"

However, IF isn’t actually such a great syntactic construct because the then-form and else-form are each restricted to being a single Lisp form. This means if you want to perform a sequence of actions in either clause, you need to wrap them in some other syntax. For instance, suppose in the middle of a spam-filtering program you wanted to both file a message as spam and update the spam database when a message is spam. You can’t write this:

(if (spam-p current-message)
    (file-in-spam-folder current-message)
    (update-spam-database current-message))

because the call to update-spam-database will be treated as the else clause, not as part of the then clause. Another special operator, PROGN, executes any number of forms in order and returns the value of the last form. So you could get the desired behavior by writing the following:

(if (spam-p current-message)
    (progn
      (file-in-spam-folder current-message)
      (update-spam-database current-message)))

That’s not too horrible. But given the number of times you’ll likely have to use this idiom, it’s not hard to imagine that you’d get tired of it after a while. “Why,” you might ask yourself, “doesn’t Lisp provide a way to say what I really want, namely, ‘When x is true, do this, that, and the other thing’?” In other words, after a while you’d notice the pattern of an IF plus a PROGN and wish for a way to abstract away the details rather than writing them out every time.

This is exactly what macros provide. In this case, Common Lisp comes with a standard macro, WHEN, which lets you write this:

(when (spam-p current-message)
  (file-in-spam-folder current-message)
  (update-spam-database current-message))

But if it wasn’t built into the standard library, you could define WHEN yourself with a macro such as this, using the backquote notation I discussed in Chapter 3:[3]

(defmacro when (condition &rest body)
  `(if ,condition (progn ,@body)))

A counterpart to the WHEN macro is UNLESS, which reverses the condition, evaluating its body forms only if the condition is false. In other words:

(defmacro unless (condition &rest body)
  `(if (not ,condition) (progn ,@body)))

Admittedly, these are pretty trivial macros. There’s no deep black magic here; they just abstract away a few language-level bookkeeping details, allowing you to express your true intent a bit more clearly. But their very triviality makes an important point: because the macro system is built right into the language, you can write trivial macros like WHEN and UNLESS that give you small but real gains in clarity that are then multiplied by the thousands of times you use them. In Chapters 24, 26, and 31 you’ll see how macros can also be used on a larger scale, creating whole domain-specific embedded languages. But first let’s finish our discussion of the standard control-construct macros.

[3]You can’t actually feed this definition to Lisp because it’s illegal to redefine names in the COMMON-LISP package where WHEN comes from. If you really want to try writing such a macro, you’d need to change the name to something else, such as my-when.


813 - COND

Chapter 7 - Macros—Standard Control Constructs
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

COND

Another time raw IF expressions can get ugly is when you have a multibranch conditional: if a do x, else if b do y; else do z. There’s no logical problem writing such a chain of conditional expressions with just IF, but it’s not pretty.

(if a
    (do-x)
    (if b
       (do-y)
       (do-z)))

And it would be even worse if you needed to include multiple forms in the then clauses, requiring PROGNs. So, not surprisingly, Common Lisp provides a macro for expressing multibranch conditionals: COND. This is the basic skeleton:

(cond
  (test-1 form*)
      .
      .
      .
  (test-N form*))

Each element of the body represents one branch of the conditional and consists of a list containing a condition form and zero or more forms to be evaluated if that branch is chosen. The conditions are evaluated in the order the branches appear in the body until one of them evaluates to true. At that point, the remaining forms in that branch are evaluated, and the value of the last form in the branch is returned as the value of the COND as a whole. If the branch contains no forms after the condition, the value of the condition is returned instead. By convention, the branch representing the final else clause in an if/else-if chain is written with a condition of T. Any non-NIL value will work, but a T serves as a useful landmark when reading the code. Thus, you can write the previous nested IF expression using COND like this:

(cond (a (do-x))
      (b (do-y))
      (t (do-z)))


815 - Looping

Chapter 7 - Macros—Standard Control Constructs
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Looping

Control constructs are the other main kind of looping constructs. Common Lisp’s looping facilities are—in addition to being quite powerful and flexible—an interesting lesson in the have-your-cake-and-eat-it-too style of programming that macros provide.

As it turns out, none of Lisp’s 25 special operators directly support structured looping. All of Lisp’s looping control constructs are macros built on top of a pair of special operators that provide a primitive goto facility.[4] Like many good abstractions, syntactic or otherwise, Lisp’s looping macros are built as a set of layered abstractions starting from the base provided by those two special operators.

At the bottom (leaving aside the special operators) is a very general looping construct, DO. While very powerful, DO suffers, as do many general-purpose abstractions, from being overkill for simple situations. So Lisp also provides two other macros, DOLIST and DOTIMES, that are less flexible than DO but provide convenient support for the common cases of looping over the elements of a list and counting loops. While an implementation can implement these macros however it wants, they’re typically implemented as macros that expand into an equivalent DO loop. Thus, DO provides a basic structured looping construct on top of the underlying primitives provided by Common Lisp’s special operators, and DOLIST and DOTIMES provide two easier-to-use, if less general, constructs. And, as you’ll see in the next chapter, you can build your own looping constructs on top of DO for situations where DOLIST and DOTIMES don’t meet your needs.

Finally, the LOOP macro provides a full-blown mini-language for expressing looping constructs in a non-Lispy, English-like (or at least Algol-like) language. Some Lisp hackers love LOOP; others hate it. LOOP’s fans like it because it provides a concise way to express certain commonly needed looping constructs. Its detractors dislike it because it’s not Lispy enough. But whichever side one comes down on, it’s a remarkable example of the power of macros to add new constructs to the language.

[4]The special operators, if you must know, are TAGBODY and GO. There’s no need to discuss them now, but I’ll cover them in Chapter 20.


852 - Higher-Order Function Variants

Chapter 11 - Collections
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Higher-Order Function Variants

For each of the functions just discussed, Common Lisp provides two higher-order function variants that, in the place of the item argument, take a function to be called on each element of the sequence. One set of variants are named the same as the basic function with an -IF appended. These functions count, find, remove, and substitute elements of the sequence for which the function argument returns true. The other set of variants are named with an -IF-NOT suffix and count, find, remove, and substitute elements for which the function argument does not return true.

(count-if #'evenp #(1 2 3 4 5))         →   2

(count-if-not #'evenp #(1 2 3 4 5))     →   3

(position-if #'digit-char-p "abcd0001") →   4

(remove-if-not #'(lambda (x) (char= (elt x 0) #\f))
  #("foo" "bar" "baz" "foom"))  → #("foo" "foom")

According to the language standard, the -IF-NOT variants are deprecated. However, that deprecation is generally considered to have itself been ill-advised. If the standard is ever revised, it’s more likely the deprecation will be removed than the -IF-NOT functions. For one thing, the REMOVE-IF-NOT variant is probably used more often than REMOVE-IF. Despite its negative-sounding name, REMOVE-IF-NOT is actually the positive variant—it returns the elements that do satisfy the predicate.[7]

The -IF and -IF-NOT variants accept all the same keyword arguments as their vanilla counterparts except for :test, which isn’t needed since the main argument is already a function.[8] With a :key argument, the value extracted by the :key function is passed to the function instead of the actual element.

(count-if #'evenp #((1 a) (2 b) (3 c) (4 d) (5 e)) :key #'first)     → 2

(count-if-not #'evenp #((1 a) (2 b) (3 c) (4 d) (5 e)) :key #'first) → 3

(remove-if-not #'alpha-char-p
  #("foo" "bar" "1baz") :key #'(lambda (x) (elt x 0))) → #("foo" "bar")

The REMOVE family of functions also support a fourth variant, REMOVE-DUPLICATES, that has only one required argument, a sequence, from which it removes all but one instance of each duplicated element. It takes the same keyword argumentsas REMOVE, except for :count, since it always removes all duplicates.

(remove-duplicates #(1 2 1 2 3 1 2 3 4)) → #(1 2 3 4)

[7]This same functionality goes by the name grep in Perl and filter in Python.

[8]The difference between the predicates passed as :test arguments and as the function arguments to the -IF and -IF-NOT functions is that the :test predicates are two-argument predicates used to compare the elements of the sequence to the specific item while the -IF and -IF-NOT predicates are one-argument functions that simply test the individual elements of the sequence. If the vanilla variants didn’t exist, you could implement them in terms of the -IF versions by embedding a specific item in the test function.

(count char string) ≡
  (count-if #'(lambda (c) (eql char c)) string)
(count char string :test #'CHAR-EQUAL) ≡
  (count-if #'(lambda (c) (char-equal char c)) string)


854 - Sorting and Merging

Chapter 11 - Collections
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Sorting and Merging

The functions SORT and STABLE-SORT provide two ways of sorting a sequence. They both take a sequence and a two-argument predicate and return a sorted version of the sequence.

(sort (vector "foo" "bar" "baz") #'string<) → #("bar" "baz" "foo")

The difference is that STABLE-SORT is guaranteed to not reorder any elements considered equivalent by the predicate while SORT guarantees only that the result is sorted and may reorder equivalent elements.

Both these functions are examples of what are called destructive functions. Destructive functions are allowed—typically for reasons of efficiency—to modify their arguments in more or less arbitrary ways. This has two implications: one, you should always do something with the return value of these functions (such as assign it to a variable or pass it to another function), and, two, unless you’re done with the object you’re passing to the destructive function, you should pass a copy instead. I’ll say more about destructive functions in the next chapter.

Typically you won’t care about the unsorted version of a sequence after you’ve sorted it, so it makes sense to allow SORT and STABLE-SORT to destroy the sequence in the course of sorting it. But it does mean you need to remember to write the following:[10]

(setf my-sequence (sort my-sequence #'string<))

rather than just this:

(sort my-sequence #'string<)

Both these functions also take a keyword argument, :key, which, like the :key argument in other sequence functions, should be a function and will be used to extract the values to be passed to the sorting predicate in the place of the actual elements. The extracted keys are used only to determine the ordering of elements; the sequence returned will contain the actual elements of the argument sequence.

The MERGE function takes two sequences and a predicate and returns a sequence produced by merging the two sequences, according to the predicate. It’s related to the two sorting functions in that if each sequence is already sorted by the same predicate, then the sequence returned by MERGE will also be sorted. Like the sorting functions, MERGE takes a :key argument. Like CONCATENATE, and for the same reason, the first argument to MERGE must be a type descriptor specifying the type of sequence to produce.

(merge 'vector #(1 3 5) #(2 4 6) #'<) → #(1 2 3 4 5 6)
(merge 'list #(1 3 5) #(2 4 6) #'<)   → (1 2 3 4 5 6)

[10]When the sequence passed to the sorting functions is a vector, the “destruction” is actually guaranteed to entail permuting the elements in place, so you could get away without saving the returned value. However, it’s good style to always do something with the return value since the sorting functions can modify lists in much more arbitrary ways.


855 - Subsequence Manipulations

Chapter 11 - Collections
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Subsequence Manipulations

Another set of functions allows you to manipulate subsequences of existing sequences. The most basic of these is SUBSEQ, which extracts a subsequence starting at a particular index and continuing to a particular ending index or the end of the sequence. For instance:

(subseq "foobarbaz" 3)   → "barbaz"
(subseq "foobarbaz" 3 6) → "bar"

SUBSEQ is also SETFable, but it won’t extend or shrink a sequence; if the new value and the subsequence to be replaced are different lengths, the shorter of the two determines how many characters are actually changed.

(defparameter *x* (copy-seq "foobarbaz"))

(setf (subseq *x* 3 6) "xxx")  ; subsequence and new value are same length
*x* →   "fooxxxbaz"

(setf (subseq *x* 3 6) "abcd") ; new value too long, extra character ignored.
*x* →   "fooabcbaz"

(setf (subseq *x* 3 6) "xx")   ; new value too short, only two characters changed
*x* →   "fooxxcbaz"

You can use the FILL function to set multiple elements of a sequence to a single value. The required arguments are a sequence and the value with which to fill it. By default every element of the sequence is set to the value; :start and :end keyword arguments can limit the effects to a given subsequence.

If you need to find a subsequence within a sequence, the SEARCH function works like POSITION except the first argument is a sequence rather than a single item.

(position #\b "foobarbaz") → 3
(search "bar" "foobarbaz") → 3

On the other hand, to find where two sequences with a common prefix first diverge, you can use the MISMATCH function. It takes two sequences and returns the index of the first pair of mismatched elements.

(mismatch "foobarbaz" "foom") → 3

It returns NIL if the strings match. MISMATCH also takes many of the standard keyword arguments: a :key argument for specifying a function to use to extract the values to be compared; a :test argument to specify the comparison function; and :start1, :end1, :start2, and :end2 arguments to specify subsequences within the two sequences. And a :from-end argument of T specifies the sequences should be searched in reverse order, causing MISMATCH to return the index, in the first sequence, where whatever common suffix the two sequences share begins.

(mismatch "foobar" "bar" :from-end t) → 3


861 - “There Is No List”

Chapter 12 - They Called It LISP for a Reason—List Processing
Practical Common Lisp
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

“There Is No List”

Spoon Boy: Do not try and bend the list. That’s impossible. Instead…only try to realize the truth.

Neo: What truth?

Spoon Boy: There is no list.

Neo: There is no list?

Spoon Boy: Then you’ll see that it is not the list that bends; it is only yourself.[1]

The key to understanding lists is to understand that they’re largely an illusion built on top of objects that are instances of a more primitive data type. Those simpler objects are pairs of values called cons cells, after the function CONS used to create them.

CONS takes two arguments and returns a new cons cell containing the two values.[2] These values can be references to any kind of object. Unless the second value is NIL or another cons cell, a cons is printed as the two values in parentheses separated by a dot, a so-called dotted pair.

(cons 1 2) → (1 . 2)

The two values in a cons cell are called the CAR and the CDR after the names of the functions used to access them. At the dawn of time, these names were mnemonic, at least to the folks implementing the first Lisp on an IBM 704. But even then they were just lifted from the assembly mnemonics used to implement the operations. However, it’s not all bad that these names are somewhat meaningless—when considering individual cons cells, it’s best to think of them simply as an arbitrary pair of values without any particular semantics. Thus:

(car (cons 1 2)) → 1
(cdr (cons 1 2)) → 2

Both CAR and CDR are also SETFable places—given an existing cons cell, it’s possible to assign a new value to either of its values.[3]

(defparameter *cons* (cons 1 2))
*cons*                 →   (1 . 2)
(setf (car *cons*) 10) →   10
*cons*                 →   (10 . 2)
(setf (cdr *cons*) 20) →   20
*cons*                 →   (10 . 20)

Because the values in a cons cell can be references to any kind of object, you can build larger structures out of cons cells by linking them together. Lists are built by linking together cons cells in a chain. The elements of the list are held in the CARs of the cons cells while the links to subsequent cons cells are held in the CDRs. The last cell in the chain has a CDR of NIL, which—as I mentioned in Chapter 4—represents the empty list as well as the boolean value false.

This arrangement is by no means unique to Lisp; it’s called a singly linked list. However, few languages outside the Lisp family provide such extensive support for this humble data type.

So when I say a particular value is a list, what I really mean is it’s either NIL or a reference to a cons cell. The CAR of the cons cell is the first item of the list, and the CDR is a reference to another list, that is, another cons cell or NIL, containing the remaining elements. The Lisp printer understands this convention and prints such chains of cons cells as parenthesized lists rather than as dotted pairs.

(cons 1 nil)                   → (1)
(cons 1 (cons 2 nil))          → (1 2)
(cons 1 (cons 2 (cons 3 nil))) → (1 2 3)

When talking about structures built out of cons cells, a few diagrams can be a big help. Box-and-arrow diagrams represent cons cells as a pair of boxes like this:

Image from book

The box on the left represents the CAR, and the box on the right is the CDR. The values stored in a particular cons cell are either drawn in the appropriate box or represented by an arrow from the box to a representation of the referenced value.[4] For instance, the list (1 2 3), which consists of three cons cells linked together by their CDRs, would be diagrammed like this:

Image from book

However, most of the time you work with lists you won’t have to deal with individual cons cells—the functions that create and manipulate lists take care of that for you. For example, the LIST function builds a cons cells under the covers for you and links them together; the following LIST expressions are equivalent to the previous CONS expressions:

(list 1)     → (1)
(list 1 2)   → (1 2)
(list 1 2 3) → (1 2 3)

Similarly, when you’re thinking in terms of lists, you don’t have to use the meaningless names CAR and CDR; FIRST and REST are synonyms for CAR and CDR that you should use when you’re dealing with cons cells as lists.

(defparameter *list* (list 1 2 3 4))
(first *list*)        →   1
(rest *list*)         →   (2 3 4)
(first (rest *list*)) →   2

Because cons cells can hold any kind of values, so can lists. And a single list can hold objects of different types.

(list "foo" (list 1 2) 10) → ("foo" (1 2) 10)

The structure of that list would look like this:

Image from book

Because lists can have other lists as elements, you can also use them to represent trees of arbitrary depth and complexity. As such, they make excellent representations for any heterogeneous, hierarchical data. Lisp-based XML processors, for instance, usually represent XML documents internally as lists. Another obvious example of tree-structured data is Lisp code itself. In Chapters 30 and 31 you’ll write an HTML generation library that uses lists of lists to represent the HTML to be generated. I’ll talk more next chapter about using cons cells to represent other data structures.

Common Lisp provides quite a large library of functions for manipulating lists. In the sections “List-Manipulation Functions” and “Mapping,” you’ll look at some of the more important of these functions. However, they will be easier to understand in the context of a few ideas borrowed from functional programming.

[1]Adapted from The Matrix (http://us.imdb.com/Quotes?0133093)

[2]CONS was originally short for the verb construct.

[3]When the place given to SETF is a CAR or CDR, it expands into a call to the function RPLACA or RPLACD; some old-school Lispers—the same ones who still use SETQ—will still use RPLACA and RPLACD directly, but modern style is to use SETF of CAR or CDR.

[4]Typically, simple objects such as numbers are drawn within the appropriate box, and more complex objects will be drawn outside the box with an arrow from the box indicating the reference. This actually corresponds well with how many Common Lisp implementations work—although all objects are conceptually stored by reference, certain simple immutable objects can be stored directly in a cons cell.


862 - Functional Programming and Lists

Chapter 12 - They Called It LISP for a Reason—List Processing
Practical Common Lisp
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Functional Programming and Lists

The essence of functional programming is that programs are built entirely of functions with no side effects that compute their results based solely on the values of their arguments. The advantage of the functional style is that it makes programs easier to understand. Eliminating side effects eliminates almost all possibilities for action at a distance. And since the result of a function is determined only by the values of its arguments, its behavior is easier to understand and test. For instance, when you see an expression such as (+ 3 4), you know the result is uniquely determined by the definition of the + function and the values 3 and 4. You don’t have to worry about what may have happened earlier in the execution of the program since there’s nothing that can change the result of evaluating that expression.

Functions that deal with numbers are naturally functional since numbers are immutable. A list, on the other hand, can be mutated, as you’ve just seen, by SETFing the CARs and CDRs of the cons cells that make up its backbone. However, lists can be treated as a functional data type if you consider their value to be determined by the elements they contain. Thus, any list of the form (1 2 3 4) is functionally equivalent to any other list containing those four values, regardless of what cons cells are actually used to represent the list. And any function that takes a list as an argument and returns a value based solely on the contents of the list can likewise be considered functional. For instance, the REVERSE sequence function, given the list (1 2 3 4), always returns a list (4 3 2 1). Different calls to REVERSE with functionally equivalent lists as the argument will return functionally equivalent result lists. Another aspect of functional programming, which I’ll discuss in the section “Mapping,” is the use of higher-order functions: functions that treat other functions as data, taking them as arguments or returning them as results.

Most of Common Lisp’s list-manipulation functions are written in a functional style. I’ll discuss later how to mix functional and other coding styles, but first you should understand a few subtleties of the functional style as applied to lists.

The reason most list functions are written functionally is it allows them to return results that share cons cells with their arguments. To take a concrete example, the function APPEND takes any number of list arguments and returns a new list containing the elements of all its arguments. For instance:

(append (list 1 2) (list 3 4)) → (1 2 3 4)

From a functional point of view, APPEND’s job is to return the list (1 2 3 4) without modifying any of the cons cells in the lists (1 2) and (3 4). One obvious way to achieve that goal is to create a completely new list consisting of four new cons cells. However, that’s more work than is necessary. Instead, APPEND actually makes only two new cons cells to hold the values 1 and 2, linking them together and pointing the CDR of the second cons cell at the head of the last argument, the list (3 4). It then returns the cons cell containing the 1. None of the original cons cells has been modified, and the result is indeed the list (1 2 3 4). The only wrinkle is that the list returned by APPEND shares some cons cells with the list (3 4). The resulting structure looks like this:

Image from book

In general, APPEND must copy all but its last argument, but it can always return a result that shares structure with the last argument.

Other functions take similar advantage of lists’ ability to share structure. Some, like APPEND, are specified to always return results that share structure in a particular way. Others are simply allowed to return shared structure at the discretion of the implementation.


872 - Chapter 14: Files and File I/O

Chapter 14 - Files and File I/O
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Chapter 14: Files and File I/O

Common Lisp provides a rich library of functionality for dealing with files. In this chapter I’ll focus on a few basic file-related tasks: reading and writing files and listing files in the file system. For these basic tasks, Common Lisp’s I/O facilities are similar to those in other languages. Common Lisp provides a stream abstraction for reading and writing data and an abstraction, called pathnames, for manipulating filenames in an operating system–independent way. Additionally, Common Lisp provides other bits of functionality unique to Lisp such as the ability to read and write s-expressions.

Reading File Data

The most basic file I/O task is to read the contents of a file. You obtain a stream from which you can read a file’s contents with the OPEN function. By default OPEN returns a character-based input stream you can pass to a variety of functions that read one or more characters of text: READ-CHAR reads a single character; READ-LINE reads a line of text, returning it as a string with the end-of-line character(s) removed; and READ reads a single s-expression, returning a Lisp object. When you’re done with the stream, you can close it with the CLOSE function.

The only required argument to OPEN is the name of the file to read. As you’ll see in the section “Filenames,” Common Lisp provides a couple of ways to represent a filename, but the simplest is to use a string containing the name in the local file-naming syntax. So assuming that /some/file/name.txt is a file, you can open it like this:

(open "/some/file/name.txt")

You can use the object returned as the first argument to any of the read functions. For instance, to print the first line of the file, you can combine OPEN, READ-LINE, and CLOSE as follows:

(let ((in (open "/some/file/name.txt")))
  (format t "~a~%" (read-line in))
  (close in))

Of course, a number of things can go wrong while trying to open and read from a file. The file may not exist. Or you may unexpectedly hit the end of the file while reading. By default OPEN and the READ-* functions will signal an error in these situations. In Chapter 19, I’ll discuss how to recover from such errors. For now, however, there’s a lighter-weight solution: each of these functions accepts arguments that modify its behavior in these exceptional situations.

If you want to open a possibly nonexistent file without OPEN signaling an error, you can use the keyword argument :if-does-not-exist to specify a different behavior. The three possible values are :error, the default; :create, which tells it to go ahead and create the file and then proceed as if it had already existed; and NIL, which tells it to return NIL instead of a stream. Thus, you can change the previous example to deal with the possibility that the file may not exist.

(let ((in (open "/some/file/name.txt" :if-does-not-exist nil)))
  (when in
    (format t "~a~%" (read-line in))
    (close in)))

The reading functions—READ-CHAR, READ-LINE, and READ—all take an optional argument, which defaults to true, that specifies whether they should signal an error if they’re called at the end of the file. If that argument is NIL, they instead return the value of their third argument, which defaults to NIL. Thus, you could print all the lines in a file like this:

(let ((in (open "/some/file/name.txt" :if-does-not-exist nil)))
  (when in
    (loop for line = (read-line in nil)
         while line do (format t "~a~%" line))
    (close in)))

Of the three text-reading functions, READ is unique to Lisp. This is the same function that provides the R in the REPL and that’s used to read Lisp source code. Each time it’s called, it reads a single s-expression, skipping whitespace and comments, and returns the Lisp object denoted by the s-expression. For instance, suppose /some/file/name.txt has the following contents:

(1 2 3)
456
"a string" ; this is a comment
((a b)
 (c d))

In other words, it contains four s-expressions: a list of numbers, a number, a string, and a list of lists. You can read those expressions like this:

CL-USER> (defparameter *s* (open "/some/file/name.txt"))
*S*
CL-USER> (read *s*)
(1 2 3)
CL-USER> (read *s*)
456
CL-USER> (read *s*)
"a string"
CL-USER> (read *s*)
((A B) (C D))
CL-USER> (close *s*)
T

As you saw in Chapter 3, you can use PRINT to print Lisp objects in “readable” form. Thus, whenever you need to store a bit of data in a file, PRINT and READ provide an easy way to do it without having to design a data format or write a parser. They even—as the previous example demonstrated—give you comments for free. And because s-expressions were designed to be human editable, it’s also a fine format for things like configuration files.[1]

[1]Note, however, that while the Lisp reader knows how to skip comments, it completely skips them. Thus, if you use READ to read in a configuration file containing comments and then use PRINT to save changes to the data, you’ll lose the comments.


873 - Reading Binary Data

Chapter 14 - Files and File I/O
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Reading Binary Data

By default OPEN returns character streams, which translate the underlying bytes to characters according to a particular character-encoding scheme.[2] To read the raw bytes, you need to pass OPEN an :element-type argument of '(unsigned-byte 8).[3] You can pass the resulting stream to the function READ-BYTE, which will return an integer between 0 and 255 each time it’s called. READ-BYTE, like the character-reading functions, also accepts optional arguments to specify whether it should signal an error if called at the end of the file and what value to return if not. In Chapter 24 you’ll build a library that allows you to conveniently read structured binary data using READ-BYTE.[4]

[2]By default OPEN uses the default character encoding for the operating system, but it also accepts a keyword parameter, :external-format, that can pass implementation-defined values that specify a different encoding. Character streams also translate the platform-specific end-of-line sequence to the single character #\Newline.

[3]The type (unsigned-byte 8) indicates an 8-bit byte; Common Lisp “byte” types aren’t a fixed size since Lisp has run at various times on architectures with byte sizes from 6 to 9 bits, to say nothing of the PDP-10, which had individually addressable variable-length bit fields of 1 to 36 bits.

[4]In general, a stream is either a character stream or a binary stream, so you can’t mix calls to READ-BYTE and READ-CHAR or other character-based read functions. However, some implementations, such as Allegro, support so-called bivalent streams, which support both character and binary I/O.


874 - Bulk Reads

Chapter 14 - Files and File I/O
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Bulk Reads

One last reading function, READ-SEQUENCE, works with both character and binary streams. You pass it a sequence (typically a vector) and a stream, and it attempts to fill the sequence with data from the stream. It returns the index of the first element of the sequence that wasn’t filled or the length of the sequence if it was able to completely fill it. You can also pass :start and :end keyword arguments to specify a subsequence that should be filled instead. The sequence argument must be a type that can hold elements of the stream’s element type. Since most operating systems support some form of block I/O, READ-SEQUENCE is likely to be quite a bit more efficient than filling a sequence by repeatedly calling READ-BYTE or READ-CHAR.


877 - Filenames

Chapter 14 - Files and File I/O
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Filenames

So far you’ve used strings to represent filenames. However, using strings as filenames ties your code to a particular operating system and file system. Likewise, if you programmatically construct names according to the rules of a particular naming scheme (separating directories with /, say), you also tie your code to a particular file system.

To avoid this kind of nonportability, Common Lisp provides another representation of filenames: pathname objects. Pathnames represent filenames in a structured way that makes them easy to manipulate without tying them to a particular filename syntax. And the burden of translating back and forth between strings in the local syntax—called namestrings—and pathnames is placed on the Lisp implementation.

Unfortunately, as with many abstractions designed to hide the details of fundamentally different underlying systems, the pathname abstraction introduces its own complications. When pathnames were designed, the set of file systems in general use was quite a bit more variegated than those in common use today. Consequently, some nooks and crannies of the pathname abstraction make little sense if all you’re concerned about is representing Unix or Windows filenames. However, once you understand which parts of the pathname abstraction you can ignore as artifacts of pathnames’ evolutionary history, they do provide a convenient way to manipulate filenames.[6]

Most places a filename is called for, you can use either a namestring or a pathname. Which to use depends mostly on where the name originated. Filenames provided by the user—for example, as arguments or as values in configuration files—will typically be namestrings, since the user knows what operating system they’re running on and shouldn’t be expected to care about the details of how Lisp represents filenames. But programmatically generated filenames will be pathnames because you can create them portably. A stream returned by OPEN also represents a filename, namely, the filename that was originally used to open the stream. Together these three types are collectively referred to as pathname designators. All the built-in functions that expect a filename argument accept all three types of pathname designator. For instance, all the places in the previous section where you used a string to represent a filename, you could also have passed a pathname object or a stream.

Image from book
HOW WE GOT HERE

The historical diversity of file systems in existence during the 70s and 80s can be easy to forget. Kent Pitman, one of the principal technical editors of the Common Lisp standard, described the situation once in comp.lang.lisp (Message-ID: sfwzo74np6w.fsf@world.std.com) thusly:

The dominant file systems at the time the design [of Common Lisp] was done were TOPS-10, TENEX, TOPS-20, VAX VMS, AT&T Unix, MIT Multics, MIT ITS, not to mention a bunch of main-frame [OSs]. Some were uppercase only, some mixed, some were case-sensitive but case-translating (like CL). Some had dirs as files, some not. Some had quote chars for funny file chars, some not. Some had wildcards, some didn’t. Some had :up in relative pathnames, some didn’t. Some had namable root dirs, some didn’t. There were file systems with no directories, file systems with non-hierarchical directories, file systems with no file types, file systems with no versions, file systems with no devices, and so on.

If you look at the pathname abstraction from the point of view of any single file system, it seems baroque. However, if you take even two such similar file systems as Windows and Unix, you can already begin to see differences the pathname system can help abstract away—Windows filenames contain a drive letter, for instance, while Unix filenames don’t. The other advantage of having the pathname abstraction designed to handle the wide variety of file systems that existed in the past is that it’s more likely to be able to handle file systems that may exist in the future. If, say, versioning file systems come back into vogue, Common Lisp will be ready.

Image from book

[6]Another reason the pathname system is considered somewhat baroque is because of the inclusion of logical pathnames. However, you can use the rest of the pathname system perfectly well without knowing anything more about logical pathnames than that you can safely ignore them. Briefly, logical pathnames allow Common Lisp programs to contain references to pathnames without naming specific files. Logical pathnames could then be mapped to specific locations in an actual file system when the program was installed by defining a “logical pathname translation” that translates logical pathnames matching certain wildcards to pathnames representing files in the file system, so-called physical pathnames. They have their uses in certain situations, but you can get pretty far without worrying about them.


882 - Other Kinds of I/O

Chapter 14 - Files and File I/O
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Other Kinds of I/O

In addition to file streams, Common Lisp supports other kinds of streams, which can also be used with the various reading, writing, and printing I/O functions. For instance, you can read data from, or write data to, a string using STRING-STREAMs, which you can create with the functions MAKE-STRING-INPUT-STREAM and MAKE-STRING-OUTPUT-STREAM.

MAKE-STRING-INPUT-STREAM takes a string and optional start and end indices to bound the area of the string from which data should be read and returns a character stream that you can pass to any of the character-based input functions such as READ-CHAR, READ-LINE, or READ. For example, if you have a string containing a floating-point literal in Common Lisp’s syntax, you can convert it to a float like this:

(let ((s (make-string-input-stream "1.23")))
  (unwind-protect (read s)
    (close s)))

Similarly, MAKE-STRING-OUTPUT-STREAM creates a stream you can use with FORMAT, PRINT, WRITE-CHAR, WRITE-LINE, and so on. It takes no arguments. Whatever you write, a string output stream will be accumulated into a string that can then be obtained with the function GET-OUTPUT-STREAM-STRING. Each time you call GET-OUTPUT-STREAM-STRING, the stream’s internal string is cleared so you can reuse an existing string output stream.

However, you’ll rarely use these functions directly, because the macros WITH-INPUT-FROM-STRING and WITH-OUTPUT-TO-STRING provide a more convenient interface. WITH-INPUT-FROM-STRING is similar to WITH-OPEN-FILE—it creates a string input stream from a given string and then executes the forms in its body with the stream bound to the variable you provide. For instance, instead of the LET form with the explicit UNWIND-PROTECT, you’d probably write this:

(with-input-from-string (s "1.23")
  (read s))

The WITH-OUTPUT-TO-STRING macro is similar: it binds a newly created string output stream to a variable you name and then executes its body. After all the body forms have been executed, WITH-OUTPUT-TO-STRING returns the value that would be returned by GET-OUTPUT-STREAM-STRING.

CL-USER> (with-output-to-string (out)
            (format out "hello, world ")
            (format out "~s" (list 1 2 3)))
"hello, world (1 2 3)"

The other kinds of streams defined in the language standard provide various kinds of stream “plumbing,” allowing you to plug together streams in almost any configuration. A BROADCAST-STREAM is an output stream that sends any data written to it to a set of output streams provided as arguments to its constructor function, MAKE-BROADCAST-STREAM.[14] Conversely, a CONCATENATED-STREAM is an input stream that takes its input from a set of input streams, moving from stream to stream as it hits the end of each stream. CONCATENATED-STREAMs are constructed with the function MAKE-CONCATENATED-STREAM, which takes any number of input streams as arguments.

Two kinds of bidirectional streams that can plug together streams in a couple ways are TWO-WAY-STREAM and ECHO-STREAM. Their constructor functions, MAKE-TWO-WAY-STREAM and MAKE-ECHO-STREAM, both take two arguments, an input stream and an output stream, and return a stream of the appropriate type, which you can use with both input and output functions.

In a TWO-WAY-STREAM every read you perform will return data read from the underlying input stream, and every write will send data to the underlying output stream. An ECHO-STREAM works essentially the same way except that all the data read from the underlying input stream is also echoed to the output stream. Thus, the output stream of an ECHO-STREAM stream will contain a transcript of both sides of the conversation.

Using these five kinds of streams, you can build almost any topology of stream plumbing you want.

Finally, although the Common Lisp standard doesn’t say anything about networking APIs, most implementations support socket programming and typically implement sockets as another kind of stream, so you can use all the regular I/O functions with them.[15]

Now you’re ready to move on to building a library that smoothes over some of the differences between how the basic pathname functions behave in different Common Lisp implementations.

[14]MAKE-BROADCAST-STREAM can make a data black hole by calling it with no arguments.

[15]The biggest missing piece in Common Lisp’s standard I/O facilities is a way for users to define new stream classes. There are, however, two de facto standards for user-defined streams. During the Common Lisp standardization, David Gray of Texas Instruments wrote a draft proposal for an API to allow users to define new stream classes. Unfortunately, there wasn’t time to work out all the issues raised by his draft to include it in the language standard. However, many implementations support some form of so-called Gray Streams, basing their API on Gray’s draft proposal. Another, newer API, called Simple Streams, has been developed by Franz and included in Allegro Common Lisp. It was designed to improve the performance of user-defined streams relative to Gray Streams and has been adopted by some of the open-source Common Lisp implementations.


883 - Chapter 15: Practical—A Portable Pathname Library

Chapter 15 - Practical—A Portable Pathname Library
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Chapter 15: Practical—A Portable Pathname Library

Overview

As I discussed in the previous chapter, Common Lisp provides an abstraction, the pathname, that’s supposed to insulate you from the details of how different operating systems and file systems name files. Pathnames provide a useful API for manipulating names as names, but when it comes to the functions that actually interact with the file system, things get a bit hairy.

The root of the problem, as I mentioned, is that the pathname abstraction was designed to represent filenames on a much wider variety of file systems than are commonly used now. Unfortunately, by making pathnames abstract enough to account for a wide variety of file systems, Common Lisp’s designers left implementers with a fair number of choices to make about how exactly to map the pathname abstraction onto any particular file system. Consequently, different implementers, each implementing the pathname abstraction for the same file system, just by making different choices at a few key junctions, could end up with conforming implementations that nonetheless provide different behavior for several of the main pathname-related functions.

However, one way or another, all implementations provide the same basic functionality, so it’s not too hard to write a library that provides a consistent interface for the most common operations across different implementations. That’s your task for this chapter. In addition to giving you several useful functions that you’ll use in future chapters, writing this library will give you a chance to learn how to write code that deals with differences between implementations.


887 - Testing a File’s Existence

Chapter 15 - Practical—A Portable Pathname Library
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Testing a File’s Existence

To replace PROBE-FILE, you can define a function called file-exists-p. It should accept a pathname and return an equivalent pathname if the file exists and NIL if it doesn’t. It should be able to accept the name of a directory in either directory or file form but should always return a directory form pathname if the file exists and is a directory. This will allow you to use file-exists-p, along with directory-pathname-p, to test whether an arbitrary name is the name of a file or directory.

In theory, file-exists-p is quite similar to the standard function PROBE-FILE; indeed, in several implementations—SBCL, LispWorks, and OpenMCL—PROBE-FILE already gives you the behavior you want for file-exists-p. But not all implementations of PROBE-FILE behave quite the same.

Allegro and CMUCL’s PROBE-FILE functions are close to what you need—they will accept the name of a directory in either form but, instead of returning a directory form name, simply return the name in the same form as the argument it was passed. Luckily, if passed the name of a nondirectory in directory form, they return NIL. So with those implementations you can get the behavior you want by first passing the name to PROBE-FILE in directory form—if the file exists and is a directory, it will return the directory form name. If that call returns NIL, then you try again with a file form name.

CLISP, on the other hand, once again has its own way of doing things. Its PROBE-FILE immediately signals an error if passed a name in directory form, regardless of whether a file or directory exists with that name. It also signals an error if passed a name in file form that’s actually the name of a directory. For testing whether a directory exists, CLISP provides its own function: probe-directory (in the ext package). This is almost the mirror image of PROBE-FILE: it signals an error if passed a name in file form or if passed a name in directory form that happens to name a file. The only difference is it returns T rather than a pathname when the named directory exists.

But even in CLISP you can implement the desired semantics by wrapping the calls to PROBE-FILE and probe-directory in IGNORE-ERRORS.[4]

(defun file-exists-p (pathname)
  #+(or sbcl lispworks openmcl)
  (probe-file pathname)

 #+(or allegro cmu)
  (or (probe-file (pathname-as-directory pathname))
      (probe-file pathname))

  #+clisp
  (or (ignore-errors
        (probe-file (pathname-as-file pathname)))
      (ignore-errors
        (let ((directory-form (pathname-as-directory pathname)))
          (when (ext:probe-directory directory-form)
            directory-form))))

  #-(or sbcl cmu lispworks openmcl allegro clisp)
  (error "list-directory not implemented"))

The function pathname-as-file that you need for the CLISP implementation of file-exists-p is the inverse of the previously defined pathname-as-directory, returning a pathname that’s the file form equivalent of its argument. This function, despite being needed here only by CLISP, is generally useful, so define it for all implementations and make it part of the library.

(defun pathname-as-file (name)
  (let ((pathname (pathname name)))
    (when (wild-pathname-p pathname)
      (error "Can't reliably convert wild pathnames."))
    (if (directory-pathname-p name)
      (let* ((directory (pathname-directory pathname))
             (name-and-type (pathname (first (last directory)))))
        (make-pathname
         :directory (butlast directory)
         :name (pathname-name name-and-type)
         :type (pathname-type name-and-type)
         :defaults pathname))
      pathname)))

[4]This is slightly broken in the sense that if PROBE-FILE signals an error for some other reason, this code will interpret it incorrectly. Unfortunately, the CLISP documentation doesn’t specify what errors might be signaled by PROBE-FILE and probe-directory, and experimentation seems to show that they signal simple-file-errors in most erroneous situations.


888 - Walking a Directory Tree

Chapter 15 - Practical—A Portable Pathname Library
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Walking a Directory Tree

Finally, to round out this library, you can implement a function called walk-directory. Unlike the functions defined previously, this function doesn’t need to do much of anything to smooth over implementation differences; it just needs to use the functions you’ve already defined. However, it’s quite handy, and you’ll use it several times in subsequent chapters. It will take the name of a directory and a function and call the function on the pathnames of all the files under the directory, recursively. It will also take two keyword arguments: :directories and :test. When :directories is true, it will call the function on the pathnames of directories as well as regular files. The :test argument, if provided, specifies another function that’s invoked on each pathname before the main function is; the main function will be called only if the test function returns true.

(defun walk-directory (dirname fn &key directories (test (constantly t)))
  (labels
      ((walk (name)
         (cond
           ((directory-pathname-p name)
            (when (and directories (funcall test name))
              (funcall fn name))
            (dolist (x (list-directory name)) (walk x)))
           ((funcall test name) (funcall fn name)))))
    (walk (pathname-as-directory dirname))))

Now you have a useful library of functions for dealing with pathnames. As I mentioned, these functions will come in handy in later chapters, particularly Chapters 23 and 27, where you’ll use walk-directory to crawl through directory trees containing spam messages and MP3 files. But before we get to that, though, I need to talk about object orientation, the topic of the next two chapters.


889 - Chapter 16: Object Reorientation—Generic Functions

Chapter 16 - Object Reorientation—Generic Functions
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Chapter 16: Object Reorientation—Generic Functions

Overview

Because the invention of Lisp predated the rise of object-oriented programming by a couple decades,[1] new Lispers are sometimes surprised to discover what a thoroughly object-oriented language Common Lisp is. Common Lisp’s immediate predecessors were developed at a time when object orientation was an exciting new idea and there were many experiments with ways to incorporate the ideas of object orientation, especially as manifested in Smalltalk, into Lisp. As part of the Common Lisp standardization, a synthesis of several of these experiments emerged under the name Common Lisp Object System, or CLOS. The ANSI standard incorporated CLOS into the language, so it no longer really makes sense to speak of CLOS as a separate entity.

The features CLOS contributed to Common Lisp range from those that can hardly be avoided to relatively esoteric manifestations of Lisp’s language-as-language-building-tool philosophy. Complete coverage of all these features is beyond the scope of this book, but in this chapter and the next I’ll describe the bread-and-butter features and give an overview of Common Lisp’s approach to objects.

You should note at the outset that Common Lisp’s object system offers a fairly different embodiment of the principles of object orientation than many other languages. If you have a deep understanding of the fundamental ideas behind object orientation, you’ll likely appreciate the particularly powerful and general way Common Lisp manifests those ideas. On the other hand, if your experience with object orientation has been largely with a single language, you may find Common Lisp’s approach somewhat foreign; you should try to avoid assuming that there’s only one way for a language to support object orientation.[2] If you have little object-oriented programming experience, you should have no trouble understanding the explanations here, though it may help to ignore the occasional comparisons to the way other languages do things.

[1]The language now generally considered the first object-oriented language, Simula, was invented in the early 1960s, only a few years after McCarthy’s first Lisp. However, object orientation didn’t really take off until the 1980s when the first widely available version of Smalltalk was released, followed by the release of C++ a few years later. Smalltalk took quite a bit of inspiration from Lisp and combined it with ideas from Simula to produce a dynamic object-oriented language, while C++ combined Simula with C, another fairly static language, to yield a static object-oriented language. This early split has led to much confusion in the definition of object orientation. Folks who come from the C++ tradition tend to consider certain aspects of C++, such as strict data encapsulation, to be key characteristics of object orientation. Folks from the Smalltalk tradition, however, consider many features of C++ to be just that, features of C++, and not core to object orientation. Indeed, Alan Kay, the inventor of Smalltalk, is reported to have said, “I invented the term object oriented, and I can tell you that C++ wasn’t what I had in mind.”

[2]There are those who reject the notion that Common Lisp is in fact object oriented at all. In particular, folks who consider strict data encapsulation a key characteristic of object orientation—usually advocates of relatively static languages such as C++, Eiffel, or Java—don’t consider Common Lisp to be properly object oriented. Of course, by that definition, Smalltalk, arguably one of the original and purest object-oriented languages, isn’t object oriented either. On the other hand, folks who consider message passing to be the key to object orientation will also not be happy with the claim that Common Lisp is object oriented since Common Lisp’s generic function orientation provides degrees of freedom not offered by pure message passing.


890 - Generic Functions and Classes

Chapter 16 - Object Reorientation—Generic Functions
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Generic Functions and Classes

The fundamental idea of object orientation is that a powerful way to organize a program is to define data types and then associate operations with those data types. In particular, you want to be able to invoke an operation and have the exact behavior determined by the type of the object or objects on which the operation was invoked. The classic example used, seemingly by all introductions to object orientation, is an operation draw that can be applied to objects representing various geometric shapes. Different implementations of the draw operation can be provided for drawing circles, triangles, and squares, and a call to draw will actually result in drawing a circle, triangle, or square, depending on the type of the object to which the draw operation is applied. The different implementations of draw are defined separately, and new versions can be defined that draw other shapes without having to change the code of either the caller or any of the other draw implementations. This feature of object orientation goes by the fancy Greek name polymorphism, meaning “many forms,” because a single conceptual operation, such as drawing an object, can take many different concrete forms.

Common Lisp, like most object-oriented languages today, is class-based; all objects are instances of a particular class.[3] The class of an object determines its representation—built-in classes such as NUMBER and STRING have opaque representations accessible only via the standard functions for manipulating those types, while instances of user-defined classes, as you’ll see in the next chapter, consist of named parts called slots.

Classes are arranged in a hierarchy, a taxonomy for all objects. A class can be defined as a subclass of other classes, called its superclasses. A class inherits part of its definition from its superclasses and instances of a class are also considered instances of the superclasses. In Common Lisp, the hierarchy of classes has a single root, the class T, which is a direct or indirect superclass of every other class. Thus, every datum in Common Lisp is an instance of T.[4] Common Lisp also supports multiple inheritance—a single class can have multiple direct superclasses.

Outside the Lisp family, almost all object-oriented languages follow the basic pattern established by Simula of having behavior associated with classes through methods or member functions that belong to a particular class. In these languages, a method is invoked on a particular object, and the class of that object determines what code runs. This model of method invocation is called—after the Smalltalk terminology—message passing. Conceptually, method invocation in a message-passing system starts by sending a message containing the name of the method to run and any arguments to the object on which the method is being invoked. The object then uses its class to look up the method associated with the name in the message and runs it. Because each class can have its own method for a given name, the same message, sent to different objects, can invoke different methods.

Early Lisp object systems worked in a similar way, providing a special function SEND that could be used to send a message to a particular object. However, this wasn’t entirely satisfactory, as it made method invocations different from normal function calls. Syntactically method invocations were written like this:

(send object 'foo)

rather than like this:

(foo object)

More significantly, because methods weren’t functions, they couldn’t be passed as arguments to higher-order functions such as MAPCAR; if one wanted to call a method on all the elements of a list with MAPCAR, one had to write this:

(mapcar #'(lambda (object) (send object 'foo)) objects)

rather than this:

(mapcar #'foo objects)

Eventually the folks working on Lisp object systems unified methods with functions by creating a new kind of function called a generic function. In addition to solving the problems just described, generic functions opened up new possibilities for the object system, including many features that simply don’t make sense in a message-passing object system.

Generic functions are the heart of Common Lisp’s object system and the topic of the rest of this chapter. While I can’t talk about generic functions without some mention of classes, for now I’ll focus on how to define and use generic functions. In the next chapter I’ll show you how to define your own classes.

[3]Prototype-based languages are the other style of object-oriented language. In these languages, JavaScript being perhaps the most famous example, objects are created by cloning a prototypical object. The clone can then be modified and used as a prototype for other objects.

[4]T the constant value and T the class have no particular relationship except they happen to have the same name. T the value is a direct instance of the class SYMBOL and only indirectly an instance of T the class.


903 - Accessor Functions

Chapter 17 - Object Reorientation—Classes
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Accessor Functions

Between MAKE-INSTANCE and SLOT-VALUE, you have all the tools you need for creating and manipulating instances of your classes. Everything else you might want to do can be implemented in terms of those two functions. However, as anyone familiar with the principles of good object-oriented programming practices knows, directly accessing the slots (or fields or member variables) of an object can lead to fragile code. The problem is that directly accessing slots ties your code too tightly to the concrete structure of your class. For example, suppose you decide to change the definition of bank-account so that, instead of storing the current balance as a number, you store a list of time-stamped withdrawals and deposits. Code that directly accesses the balance slot will likely break if you change the class definition to remove the slot or to store the new list in the old slot. On the other hand, if you define a function, balance, that accesses the slot, you can redefine it later to preserve its behavior even if the internal representation changes. And code that uses such a function will continue to work without modification.

Another advantage to using accessor functions rather than direct access to slots via SLOT-VALUE is that they let you limit the ways outside code can modify a slot.[8] It may be fine for users of the bank-account class to get the current balance, but you may want all modifications to the balance to go through other functions you’ll provide, such as deposit and withdraw. If clients know they’re supposed to manipulate objects only through the published functional API, you can provide a balance function but not make it SETFable if you want the balance to be read-only.

Finally, using accessor functions makes your code tidier since it helps you avoid lots of uses of the rather verbose SLOT-VALUE function.

It’s trivial to define a function that reads the value of the balance slot.

(defun balance (account)
  (slot-value account 'balance))

However, if you know you’re going to define subclasses of bank-account, it might be a good idea to define balance as a generic function. That way, you can provide different methods on balance for those subclasses or extend its definition with auxiliary methods. So you might write this instead:

(defgeneric balance (account))

(defmethod balance ((account bank-account))
  (slot-value account 'balance))

As I just discussed, you don’t want callers to be able to directly set the balance, but for other slots, such as customer-name, you may also want to provide a function to set them. The cleanest way to define such a function is as a SETF function.

A SETF function is a way to extend SETF, defining a new kind of place that it knows how to set. The name of a SETF function is a two-item list whose first element is the symbol setf and whose second element is a symbol, typically the name of a function used to access the place the SETF function will set. A SETF function can take any number of arguments, but the first argument is always the value to be assigned to the place.[9] You could, for instance, define a SETF function to set the customer-name slot in a bank-account like this:

(defun (setf customer-name) (name account)
  (setf (slot-value account 'customer-name) name))

After evaluating that definition, an expression like the following one:

(setf (customer-name my-account) "Sally Sue")

will be compiled as a call to the SETF function you just defined with “Sally Sue” as the first argument and the value of my-account as the second argument.

Of course, as with reader functions, you’ll probably want your SETF function to be generic, so you’d actually define it like this:

(defgeneric (setf customer-name) (value account))

(defmethod (setf customer-name) (value (account bank-account))
  (setf (slot-value account 'customer-name) value))

And of course you’ll also want to define a reader function for customer-name.

(defgeneric customer-name (account))

(defmethod customer-name ((account bank-account))
  (slot-value account 'customer-name))

This allows you to write the following:

(setf (customer-name *account*) "Sally Sue") → "Sally Sue"

(customer-name *account*)                    → "Sally Sue"

There’s nothing hard about writing these accessor functions, but it wouldn’t be in keeping with The Lisp Way to have to write them all by hand. Thus, DEFCLASS supports three slot options that allow you to automatically create reader and writer functions for a specific slot.

The :reader option specifies a name to be used as the name of a generic function that accepts an object as its single argument. When the DEFCLASS is evaluated, the generic function is created, if it doesn’t already exist. Then a method specializing its single argument on the new class and returning the value of the slot is added to the generic function. The name can be anything, but it’s typical to name it the same as the slot itself. Thus, instead of explicitly writing the balance generic function and method as shown previously, you could change the slot specifier for the balance slot in the definition of bank-account to this:

(balance
 :initarg :balance
 :initform 0
 :reader balance)

The :writer option is used to create a generic function and method for setting the value of a slot. The function and method created follow the requirements for a SETF function, taking the new value as the first argument and returning it as the result, so you can define a SETF function by providing a name such as (setf customer-name). For instance, you could provide reader and writer methods for customer-name equivalent to the ones you just wrote by changing the slot specifier to this:

(customer-name
 :initarg :customer-name
 :initform (error "Must supply a customer name.")
 :reader customer-name
 :writer (setf customer-name))

Since it’s quite common to want both reader and writer functions, DEFCLASS also provides an option, :accessor, that creates both a reader function and the corresponding SETF function. So instead of the slot specifier just shown, you’d typically write this:

(customer-name
 :initarg :customer-name
 :initform (error "Must supply a customer name.")
 :accessor customer-name)

Finally, one last slot option you should know about is the :documentation option, which you can use to provide a string that documents the purpose of the slot. Putting it all together and adding a reader method for the account-number and account-type slots, the DEFCLASS form for the bank-account class would look like this:

(defclass bank-account ()
  ((customer-name
    :initarg :customer-name
    :initform (error "Must supply a customer name.")
    :accessor customer-name
    :documentation "Customer's name")
   (balance
    :initarg :balance
    :initform 0
    :reader balance
    :documentation "Current account balance")
   (account-number
    :initform (incf *account-numbers*)
    :reader account-number
    :documentation "Account number, unique within a bank.")
   (account-type
    :reader account-type
    :documentation "Type of account, one of :gold, :silver, or :bronze.")))

[8]Of course, providing an accessor function doesn’t really limit anything since other code can still use SLOT-VALUE to get at slots directly. Common Lisp doesn’t provide strict encapsulation of slots the way some languages such as C++ and Java do; however, if the author of a class provides accessor functions and you ignore them, using SLOT-VALUE instead, you had better know what you’re doing. It’s also possible to use the package system, which I’ll discuss in Chapter 21, to make it even more obvious that certain slots aren’t to be accessed directly, by not exporting the names of the slots.

[9]One consequence of defining a SETF function—say, (setf foo)—is that if you also define the corresponding accessor function, foo in this case, you can use all the modify macros built upon SETF, such as INCF, DECF, PUSH, and POP, on the new kind of place.


904 - WITH-SLOTS and WITH-ACCESSORS

Chapter 17 - Object Reorientation—Classes
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

WITH-SLOTS and WITH-ACCESSORS

While using accessor functions will make your code easier to maintain, they can still be a bit verbose. And there will be times, when writing methods that implement the low-level behaviors of a class, that you may specifically want to access slots directly to set a slot that has no writer function or to get at the slot value without causing any auxiliary methods defined on the reader function to run.

This is what SLOT-VALUE is for; however, it’s still quite verbose. To make matters worse, a function or method that accesses the same slot several times can become clogged with calls to accessor functions and SLOT-VALUE. For example, even a fairly simple method such as the following, which assesses a penalty on a bank-account if its balance falls below a certain minimum, is cluttered with calls to balance and SLOT-VALUE:

(defmethod assess-low-balance-penalty ((account bank-account))
  (when (< (balance account) *minimum-balance*)
    (decf (slot-value account 'balance) (* (balance account) .01))))

And if you decide you want to directly access the slot value in order to avoid running auxiliary methods, it gets even more cluttered.

(defmethod assess-low-balance-penalty ((account bank-account))
  (when (< (slot-value account 'balance) *minimum-balance*)
    (decf (slot-value account 'balance) (* (slot-value account 'balance) .01))))

Two standard macros, WITH-SLOTS and WITH-ACCESSORS, can help tidy up this clutter. Both macros create a block of code in which simple variable names can be used to refer to slots on a particular object. WITH-SLOTS provides direct access to the slots, as if by SLOT-VALUE, while WITH-ACCESSORS provides a shorthand for accessor methods.

The basic form of WITH-SLOTS is as follows:

(with-slots (slot*) instance-form
  body-form*)

Each element of slots can be either the name of a slot, which is also used as a variable name, or a two-item list where the first item is a name to use as a variable and the second is the name of the slot. The instance-form is evaluated once to produce the object whose slots will be accessed. Within the body, each occurrence of one of the variable names is translated to a call to SLOT-VALUE with the object and the appropriate slot name as arguments.[10] Thus, you can write assess-low-balance-penalty like this:

(defmethod assess-low-balance-penalty ((account bank-account))
  (with-slots (balance) account
    (when (< balance *minimum-balance*)
      (decf balance (* balance .01)))))

or, using the two-item list form, like this:

(defmethod assess-low-balance-penalty ((account bank-account))
  (with-slots ((bal balance)) account
    (when (< bal *minimum-balance*)
      (decf bal (* bal .01)))))

If you had defined balance with an :accessor rather than just a :reader, then you could also use WITH-ACCESSORS. The form of WITH-ACCESSORS is the same as WITH-SLOTS except each element of the slot list is a two-item list containing a variable name and the name of an accessor function. Within the body of WITH-ACCESSORS, a reference to one of the variables is equivalent to a call to the corresponding accessor function. If the accessor function is SETFable, then so is the variable.

(defmethod assess-low-balance-penalty ((account bank-account))
  (with-accessors ((balance balance)) account
    (when (< balance *minimum-balance*)
      (decf balance (* balance .01)))))

The first balance is the name of the variable, and the second is the name of the accessor function; they don’t have to be the same. You could, for instance, write a method to merge two accounts using two calls to WITH-ACCESSORS, one for each account.

(defmethod merge-accounts ((account1 bank-account) (account2 bank-account))
  (with-accessors ((balance1 balance)) account1
    (with-accessors ((balance2 balance)) account2
      (incf balance1 balance2)
      (setf balance2 0))))

The choice of whether to use WITH-SLOTS versus WITH-ACCESSORS is the same as the choice between SLOT-VALUE and an accessor function: low-level code that provides the basic functionality of a class may use SLOT-VALUE or WITH-SLOTS to directly manipulate slots in ways not supported by accessor functions or to explicitly avoid the effects of auxiliary methods that may have been defined on the accessor functions. But you should generally use accessor functions or WITH-ACCESSORS unless you have a specific reason not to.

[10]The “variable” names provided by WITH-SLOTS and WITH-ACCESSORS aren’t true variables; they’re implemented using a special kind of macro, called a symbol macro, that allows a simple name to expand into arbitrary code. Symbol macros were introduced into the language to support WITH-SLOTS and WITH-ACCESSORS, but you can also use them for your own purposes. I’ll discuss them in a bit more detail in Chapter 20.


906 - Slots and Inheritance

Chapter 17 - Object Reorientation—Classes
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Slots and Inheritance

As I discussed in the previous chapter, classes inherit behavior from their superclasses thanks to the generic function machinery—a method specialized on class A is applicable not only to direct instances of A but also to instances of A’s subclasses. Classes also inherit slots from their superclasses, but the mechanism is slightly different.

In Common Lisp a given object can have only one slot with a particular name. However, it’s possible that more than one class in the inheritance hierarchy of a given class will specify a slot with a particular name. This can happen either because a subclass includes a slot specifier with the same name as a slot specified in a superclass or because multiple superclasses specify slots with the same name.

Common Lisp resolves these situations by merging all the specifiers with the same name from the new class and all its superclasses to create a single specifier for each unique slot name. When merging specifiers, different slot options are treated differently. For instance, since a slot can have only a single default value, if multiple classes specify an :initform, the new class uses the one from the most specific class. This allows a subclass to specify a different default value than the one it would otherwise inherit.

On the other hand, :initargs needn’t be exclusive—each :initarg option in a slot specifier creates a keyword parameter that can be used to initialize the slot; multiple parameters don’t create a conflict, so the new slot specifier contains all the :initargs. Callers of MAKE-INSTANCE can use any of the :initargs to initialize the slot. If a caller passes multiple keyword arguments that initialize the same slot, then the leftmost argument in the call to MAKE-INSTANCE is used.

Inherited :reader, :writer, and :accessor options aren’t included in the merged slot specifier since the methods created by the superclass’s DEFCLASS will already apply to the new class. The new class can, however, create its own accessor functions by supplying its own :reader, :writer, or :accessor options.

Finally, the :allocation option is, like :initform, determined by the most specific class that specifies the slot. Thus, it’s possible for all instances of one class to share a :class slot while instances of a subclass may each have their own :instance slot of the same name. And a sub-subclass may then redefine it back to :class slot, so all instances of that class will again share a single slot. In the latter case, the slot shared by instances of the sub-subclass is different than the slot shared by the original superclass.

For instance, suppose you have these classes:

(defclass foo ()
  ((a :initarg :a :initform "A" :accessor a)
   (b :initarg :b :initform "B" :accessor b)))

(defclass bar (foo)
  ((a :initform (error "Must supply a value for a"))
   (b :initarg :the-b :accessor the-b :allocation :class)))

When instantiating the class bar, you can use the inherited initarg, :a, to specify a value for the slot a and, in fact, must do so to avoid an error, since the :initform supplied by bar supersedes the one inherited from foo. To initialize the b slot, you can use either the inherited initarg :b or the new initarg :the-b. However, because of the :allocation option on the b slot in bar, the value specified will be stored in the slot shared by all instances of bar. That same slot can be accessed either with the method on the generic function b that specializes on foo or with the new method on the generic function the-b that specializes directly on bar. To access the a slot on either a foo or a bar, you’ll continue to use the generic function a.

Usually merging slot definitions works quite nicely. However, it’s important to be aware when using multiple inheritance that two unrelated slots that happen to have the same name can be merged into a single slot in the new class. Thus, methods specialized on different classes could end up manipulating the same slot when applied to a class that extends those classes. This isn’t much of a problem in practice since, as you’ll see in Chapter 21, you can use the package system to avoid collisions between names in independently developed pieces of code.


907 - Multiple Inheritance

Chapter 17 - Object Reorientation—Classes
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Multiple Inheritance

All the classes you’ve seen so far have had only a single direct superclass. Common Lisp also supports multiple inheritance—a class can have multiple direct superclasses, inheriting applicable methods and slot specifiers from all of them.

Multiple inheritance doesn’t dramatically change any of the mechanisms of inheritance I’ve discussed so far—every user-defined class already has multiple superclasses since they all extend STANDARD-OBJECT, which extends T, and so have at least two superclasses. The wrinkle that multiple inheritance adds is that a class can have more than one direct superclass. This complicates the notion of class specificity that’s used both when building the effective methods for a generic function and when merging inherited slot specifiers.

That is, if classes could have only a single direct superclass, ordering classes by specificity would be trivial—a class and all its superclasses could be ordered in a straight line starting from the class itself, followed by its single direct superclass, followed by its direct superclass, all the way up to T. But when a class has multiple direct superclasses, those superclasses are typically not related to each other—indeed, if one was a subclass of another, you wouldn’t need to subclass both directly. In that case, the rule that subclasses are more specific than their superclasses isn’t enough to order all the superclasses. So Common Lisp uses a second rule that sorts unrelated superclasses according to the order they’re listed in the DEFCLASS’s direct superclass list—classes earlier in the list are considered more specific than classes later in the list. This rule is admittedly somewhat arbitrary but does allow every class to have a linear class precedence list, which can be used to determine which superclasses should be considered more specific than others. Note, however, there’s no global ordering of classes—each class has its own class precedence list, and the same classes can appear in different orders in different classes’ class precedence lists.

To see how this works, let’s add a class to the banking app: money-market-account. A money market account combines the characteristics of a checking account and a savings account: a customer can write checks against it, but it also earns interest. You might define it like this:

(defclass money-market-account (checking-account savings-account) ())

The class precedence list for money-market-account will be as follows:

(money-market-account
 checking-account
 savings-account
 bank-account
 standard-object
 t)

Note how this list satisfies both rules: every class appears before all its superclasses, and checking-account and savings-account appear in the order specified in DEFCLASS.

This class defines no slots of its own but will inherit slots from both of its direct superclasses, including the slots they inherit from their superclasses. Likewise, any method that’s applicable to any class in the class precedence list will be applicable to a money-market-account object. Because all slot specifiers for the same slot are merged, it doesn’t matter that money-market-account inherits the same slot specifiers from bank-account twice.[12]

Multiple inheritance is easiest to understand when the different superclasses provide completely independent slots and behaviors. For instance, money-market-account will inherit slots and behaviors for dealing with checks from checking-account and slots and behaviors for computing interest from savings-account. You don’t have to worry about the class precedence list for methods and slots inherited from only one superclass or another.

However, it’s also possible to inherit different methods for the same generic function from different superclasses. In that case, the class precedence list does come into play. For instance, suppose the banking application defined a generic function print-statement used to generate monthly statements. Presumably there would already be methods for print-statement specialized on both checking-account and savings-account. Both of these methods will be applicable to instances of money-market-account, but the one specialized on checking-account will be considered more specific than the one on savings-account because checking-account precedes savings-account in money-market-account’s class precedence list.

Assuming the inherited methods are all primary methods and you haven’t defined any other methods, the method specialized on checking-account will be used if you invoke print-statement on money-market-account. However, that won’t necessarily give you the behavior you want since you probably want a money market account’s statement to contain elements of both a checking account and a savings account statement.

You can modify the behavior of print-statement for money-market-accounts in a couple ways. One straightforward way is to define a new primary method specialized on money-market-account. This gives you the most control over the new behavior but will probably require more new code than some other options I’ll discuss in a moment. The problem is that while you can use CALL-NEXT-METHOD to call “up” to the next most specific method, namely, the one specialized on checking-account, there’s no way to invoke a particular less-specific method, such as the one specialized on savings-account. Thus, if you want to be able to reuse the code that prints the savings-account part of the statement, you’ll need to break that code into a separate function, which you can then call directly from both the money-market-account and savings-account print-statement methods.

Another possibility is to write the primary methods of all three classes to call CALL-NEXT-METHOD. Then the method specialized on money-market-account will use CALL-NEXT-METHOD to invoke the method specialized on checking-account. When that method calls CALL-NEXT-METHOD, it will result in running the savings-account method since it will be the next most specific method according to money-market-account’s class precedence list.

Of course, if you’re going to rely on a coding convention—that every method calls CALL-NEXT-METHOD—to ensure all the applicable methods run at some point, you should think about using auxiliary methods instead. In this case, instead of defining primary methods on print-statement for checking-account and savings-account, you can define those methods as :after methods, defining a single primary method on bank-account. Then, print-statement, called on a money-market-account, will print a basic account statement, output by the primary method specialized on bank-account, followed by details output by the :after methods specialized on savings-account and checking-account. And if you want to add details specific to money-market-accounts, you can define an :after method specialized on money-market-account, which will run last of all.

The advantage of using auxiliary methods is that it makes it quite clear which methods are primarily responsible for implementing the generic function and which ones are only contributing additional bits of functionality. The disadvantage is that you don’t get fine-grained control over the order in which the auxiliary methods run—if you wanted the checking-account part of the statement to print before the savings-account part, you’d have to change the order in which the money-market-account subclasses those classes. But that’s a fairly dramatic change that could affect other methods and inherited slots. In general, if you find yourself twiddling the order of the direct superclass list as a way of fine-tuning the behavior of specific methods, you probably need to step back and rethink your approach.

On the other hand, if you don’t care exactly what the order is but want it to be consistent across several generic functions, then using auxiliary methods may be just the thing. For example, if in addition to print-statement you have a print-detailed-statement generic function, you can implement both functions using :after methods on the various subclasses of bank-account, and the order of the parts of both a regular and a detailed statement will be the same.

[12]In other words, Common Lisp doesn’t suffer from the diamond inheritance problem the way, say, C++ does. In C++, when one class subclasses two classes that both inherit a member variable from a common superclass, the bottom class inherits the member variable twice, leading to no end of confusion.


910 - The FORMAT Function

Chapter 18 - Few FORMAT Recipes
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

The FORMAT Function

As you’ve seen in previous chapters, the FORMAT function takes two required arguments: a destination for its output and a control string that contains literal text and embedded directives. Any additional arguments provide the values used by the directives in the control string that interpolate values into the output. I’ll refer to these arguments as format arguments.

The first argument to FORMAT, the destination for the output, can be T, NIL, a stream, or a string with a fill pointer. T is shorthand for the stream *STANDARD-OUTPUT*, while NIL causes FORMAT to generate its output to a string, which it then returns.[3] If the destination is a stream, the output is written to the stream. And if the destination is a string with a fill pointer, the formatted output is added to the end of the string and the fill pointer is adjusted appropriately. Except when the destination is NIL and it returns a string, FORMAT returns NIL.

The second argument, the control string, is, in essence, a program in the FORMAT language. The FORMAT language isn’t Lispy at all—its basic syntax is based on characters, not s-expressions, and it’s optimized for compactness rather than easy comprehension. This is why a complex FORMAT control string can end up looking like line noise.

Most of FORMAT’s directives simply interpolate an argument into the output in one form or another. Some directives, such as ~%, which causes FORMAT to emit a newline, don’t consume any arguments. And others, as you’ll see, can consume more than one argument. One directive even allows you to jump around in the list of arguments in order to process the same argument more than once or to skip certain arguments in certain situations. But before I discuss specific directives, let’s look at the general syntax of a directive.

[3]To slightly confuse matters, most other I/O functions also accept T and NIL as stream designators but with a different meaning: as a stream designator, T designates the bidirectional stream *TERMINAL-IO*, while NIL designates *STANDARD-OUTPUT* as an output stream and *STANDARD-INPUT* as an input stream.


912 - Basic Formatting

Chapter 18 - Few FORMAT Recipes
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Basic Formatting

Now you’re ready to look at specific directives. I’ll start with several of the most commonly used directives, including some you’ve seen in previous chapters.

The most general-purpose directive is ~A, which consumes one format argument of any type and outputs it in aesthetic (human-readable) form. For example, strings are output without quotation marks or escape characters, and numbers are output in a natural way for the type of number. If you just want to emit a value for human consumption, this directive is your best bet.

(format nil "The value is: ~a" 10)           → "The value is: 10"
(format nil "The value is: ~a" "foo")        → "The value is: foo"
(format nil "The value is: ~a" (list 1 2 3)) → "The value is: (1 2 3)"

A closely related directive, ~S, likewise consumes one format argument of any type and outputs it. However, ~S tries to generate output that can be read back in with READ. Thus, strings will be enclosed in quotation marks, symbols will be package-qualified when necessary, and so on. Objects that don’t have a READable representation are printed with the unreadable object syntax, #<>. With a colon modifier, both the ~A and ~S directives emit NIL as () rather than NIL. Both the ~A and ~S directives also take up to four prefix parameters, which can be used to control whether padding is added after (or before with the at-sign modifier) the value, but those parameters are only really useful for generating tabular data.

The other two most frequently used directives are ~%, which emits a newline, and ~&, which emits a fresh line. The difference between the two is that ~% always emits a newline, while ~& emits one only if it’s not already at the beginning of a line. This is handy when writing loosely coupled functions that each generate a piece of output and that need to be combined in different ways. For instance, if one function generates output that ends with a newline (~%) and another function generates some output that starts with a fresh line (~&), you don’t have to worry about getting an extra blank line if you call them one after the other. Both of these directives can take a single prefix parameter that specifies the number of newlines to emit. The ~% directive will simply emit that many newline characters, while the ~& directive will emit either n – 1 or n newlines, depending on whether it starts at the beginning of a line.

Less frequently used is the related ~~ directive, which causes FORMAT to emit a literal tilde. Like the ~% and ~& directives, it can be parameterized with a number that controls how many tildes to emit.


914 - Floating-Point Directives

Chapter 18 - Few FORMAT Recipes
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Floating-Point Directives

Four directives format floating-point values: ~F, ~E, ~G, and ~$. The first three of these are the directives based on FORTRAN’s edit descriptors. I’ll skip most of the details of those directives since they mostly have to do with formatting floating-point values for use in tabular form. However, you can use the ~F, ~E, and ~$ directives to interpolate floating-point values into text. The ~G, or general, floating-point directive, on the other hand, combines aspects of the ~F and ~E directives in a way that only really makes sense for generating tabular output.

The ~F directive emits its argument, which should be a number,[5] in decimal format, possibly controlling the number of digits after the decimal point. The ~F directive is, however, allowed to use computerized scientific notation if the number is sufficiently large or small. The ~E directive, on the other hand, always emits numbers in computerized scientific notation. Both of these directives take a number of prefix parameters, but you need to worry only about the second, which controls the number of digits to print after the decimal point.

(format nil "~f" pi)   → "3.141592653589793d0"
(format nil "~,4f" pi) → "3.1416"
(format nil "~e" pi)   → "3.141592653589793d+0"
(format nil "~,4e" pi) → "3.1416d+0"

The ~$, or monetary, directive is similar to ~F but a bit simpler. As its name suggests, it’s intended for emitting monetary units. With no parameters, it’s basically equivalent to ~,2F. To modify the number of digits printed after the decimal point, you use the first parameter, while the second parameter controls the minimum number of digits to print before the decimal point.

(format nil "~$" pi)    → "3.14"
(format nil "~2,4$" pi) → "0003.14"

All three directives, ~F, ~E, and ~$, can be made to always print a sign, plus or minus, with the at-sign modifier.[6]

[5]Technically, if the argument isn’t a real number, ~F is supposed to format it as if by the ~D directive, which in turn behaves like the ~A directive if the argument isn’t a number, but not all implementations get this right.

[6]Well, that’s what the language standard says. For some reason, perhaps rooted in a common ancestral code base, several Common Lisp implementations don’t implement this aspect of the ~F directive correctly.


919 - And More

Chapter 18 - Few FORMAT Recipes
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

And More …

And there’s more—I haven’t mentioned the ~? directive, which can take snippets of control strings from the format arguments or the ~/ directive, which allows you to call an arbitrary function to handle the next format argument. And then there are all the directives for generating tabular and pretty-printed output. But the directives discussed in this chapter should be plenty for the time being.

In the next chapter, you’ll move onto Common Lisp’s condition system, the Common Lisp analog to other languages’ exception and error handling systems.


924 - Restarts

Chapter 19 - Beyond Exception Handling—Conditions and Restarts
Practical Common Lisp
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Restarts

The condition system lets you do this by splitting the error handling code into two parts. You place code that actually recovers from errors into restarts, and condition handlers can then handle a condition by invoking an appropriate restart. You can place restart code in mid- or low-level functions, such as parse-log-file or parse-log-entry, while moving the condition handlers into the upper levels of the application.

To change parse-log-file so it establishes a restart instead of a condition handler, you can change the HANDLER-CASE to a RESTART-CASE. The form of RESTART-CASE is quite similar to a HANDLER-CASE except the names of restarts are just names, not necessarily the names of condition types. In general, a restart name should describe the action the restart takes. In parse-log-file, you can call the restart skip-log-entry since that’s what it does. The new version will look like this:

(defun parse-log-file (file)
  (with-open-file (in file :direction :input)
    (loop for text = (read-line in nil nil) while text
       for entry = (restart-case (parse-log-entry text)
                     (skip-log-entry () nil))
       when entry collect it)))

If you invoke this version of parse-log-file on a log file containing corrupted entries, it won’t handle the error directly; you’ll end up in the debugger. However, there among the various restarts presented by the debugger will be one called skip-log-entry, which, if you choose it, will cause parse-log-file to continue on its way as before. To avoid ending up in the debugger, you can establish a condition handler that invokes the skip-log-entry restart automatically.

The advantage of establishing a restart rather than having parse-log-file handle the error directly is it makes parse-log-file usable in more situations. The higher-level code that invokes parse-log-file doesn’t have to invoke the skip-log-entry restart. It can choose to handle the error at a higher level. Or, as I’ll show in the next section, you can add restarts to parse-log-entry to provide other recovery strategies, and then condition handlers can choose which strategy they want to use.

But before I can talk about that, you need to see how to set up a condition handler that will invoke the skip-log-entry restart. You can set up the handler anywhere in the chain of calls leading to parse-log-file. This may be quite high up in your application, not necessarily in parse-log-file’s direct caller. For instance, suppose the main entry point to your application is a function, log-analyzer, that finds a bunch of logs and analyzes them with the function analyze-log, which eventually leads to a call to parse-log-file. Without any error handling, it might look like this:

(defun log-analyzer ()
  (dolist (log (find-all-logs))
    (analyze-log log)))

The job of analyze-log is to call, directly or indirectly, parse-log-file and then do something with the list of log entries returned. An extremely simple version might look like this:

(defun analyze-log (log)
  (dolist (entry (parse-log-file log))
    (analyze-entry entry)))

where the function analyze-entry is presumably responsible for extracting whatever information you care about from each log entry and stashing it away somewhere.

Thus, the path from the top-level function, log-analyzer, to parse-log-entry, which actually signals an error, is as follows:

Image from book

Assuming you always want to skip malformed log entries, you could change this function to establish a condition handler that invokes the skip-log-entry restart for you. However, you can’t use HANDLER-CASE to establish the condition handler because then the stack would be unwound to the function where the HANDLER-CASE appears. Instead, you need to use the lower-level macro HANDLER-BIND. The basic form of HANDLER-BIND is as follows:

(handler-bind (binding*) form*)

where each binding is a list of a condition type and a handler function of one argument. After the handler bindings, the body of the HANDLER-BIND can contain any number of forms. Unlike the handler code in HANDLER-CASE, the handler code must be a function object, and it must accept a single argument. A more important difference between HANDLER-BIND and HANDLER-CASE is that the handler function bound by HANDLER-BIND will be run without unwinding the stack—the flow of control will still be in the call to parse-log-entry when this function is called. The call to INVOKE-RESTART will find and invoke the most recently bound restart with the given name. So you can add a handler to log-analyzer that will invoke the skip-log-entry restart established in parse-log-file like this:[5]

(defun log-analyzer ()
  (handler-bind ((malformed-log-entry-error
                  #'(lambda (c)
                      (invoke-restart 'skip-log-entry))))
    (dolist (log (find-all-logs))
      (analyze-log log))))

In this HANDLER-BIND, the handler function is an anonymous function that invokes the restart skip-log-entry. You could also define a named function that does the same thing and bind it instead. In fact, a common practice when defining a restart is to define a function, with the same name and taking a single argument, the condition, that invokes the eponymous restart. Such functions are called restart functions. You could define a restart function for skip-log-entry like this:

(defun skip-log-entry (c)
  (invoke-restart 'skip-log-entry))

Then you could change the definition of log-analyzer to this:

(defun log-analyzer ()
  (handler-bind ((malformed-log-entry-error #'skip-log-entry))
    (dolist (log (find-all-logs))
      (analyze-log log))))

As written, the skip-log-entry restart function assumes that a skip-log-entry restart has been established. If a malformed-log-entry-error is ever signaled by code called from log-analyzer without a skip-log-entry having been established, the call to INVOKE-RESTART will signal a CONTROL-ERROR when it fails to find the skip-log-entry restart. If you want to allow for the possibility that a malformed-log-entry-error might be signaled from code that doesn’t have a skip-log-entry restart established, you could change the skip-log-entry function to this:

(defun skip-log-entry (c)
  (let ((restart (find-restart 'skip-log-entry)))
    (when restart (invoke-restart restart))))

FIND-RESTART looks for a restart with a given name and returns an object representing the restart if the restart is found and NIL if not. You can invoke the restart by passing the restart object to INVOKE-RESTART. Thus, when skip-log-entry is bound with HANDLER-BIND, it will handle the condition by invoking the skip-log-entry restart if one is available and otherwise will return normally, giving other condition handlers, bound higher on the stack, a chance to handle the condition.

[5]The compiler may complain if the parameter is never used. You can silence that warning by adding a declaration (declare (ignore c)) as the first expression in the LAMBDA body.


926 - Other Uses for Conditions

Chapter 19 - Beyond Exception Handling—Conditions and Restarts
Practical Common Lisp
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Other Uses for Conditions

While conditions are mainly used for error handling, they can be used for other purposes—you can use conditions, condition handlers, and restarts to build a variety of protocols between low- and high-level code. The key to understanding the potential of conditions is to understand that merely signaling a condition has no effect on the flow of control.

The primitive signaling function SIGNAL implements the mechanism of searching for an applicable condition handler and invoking its handler function. The reason a handler can decline to handle a condition by returning normally is because the call to the handler function is just a regular function call—when the handler returns, control passes back to SIGNAL, which then looks for another, less recently bound handler that can handle the condition. If SIGNAL runs out of condition handlers before the condition is handled, it also returns normally.

The ERROR function you’ve been using calls SIGNAL. If the error is handled by a condition handler that transfers control via HANDLER-CASE or by invoking a restart, then the call to SIGNAL never returns. But if SIGNAL returns, ERROR invokes the debugger by calling the function stored in *DEBUGGER-HOOK*. Thus, a call to ERROR can never return normally; the condition must be handled either by a condition handler or in the debugger.

Another condition signaling function, WARN, provides an example of a different kind of protocol built on the condition system. Like ERROR, WARN calls SIGNAL to signal a condition. But if SIGNAL returns, WARN doesn’t invoke the debugger—it prints the condition to *ERROR-OUTPUT* and returns NIL, allowing its caller to proceed. WARN also establishes a restart, MUFFLE-WARNING, around the call to SIGNAL that can be used by a condition handler to make WARN return without printing anything. The restart function MUFFLE-WARNING finds and invokes its eponymous restart, signaling a CONTROL-ERROR if no such restart is available. Of course, a condition signaled with WARN could also be handled in some other way—a condition handler could “promote” a warning to an error by handling it as if it were an error.

For instance, in the log-parsing application, if there were ways a log entry could be slightly malformed but still parsable, you could write parse-log-entry to go ahead and parse the slightly defective entries but to signal a condition with WARN when it did. Then the larger application could choose to let the warning print, to muffle the warning, or to treat the warning like an error, recovering the same way it would from a malformed-log-entry-error.

A third error-signaling function, CERROR, provides yet another protocol. Like ERROR, CERROR will drop you into the debugger if the condition it signals isn’t handled. But like WARN, it establishes a restart before it signals the condition. The restart, CONTINUE, causes CERROR to return normally—if the restart is invoked by a condition handler, it will keep you out of the debugger altogether. Otherwise, you can use the restart once you’re in the debugger to resume the computation immediately after the call to CERROR. The function CONTINUE finds and invokes the CONTINUE restart if it’s available and returns NIL otherwise.

You can also build your own protocols on SIGNAL—whenever low-level code needs to communicate information back up the call stack to higher-level code, the condition mechanism is a reasonable mechanism to use. But for most purposes, one of the standard error or warning protocols should suffice.

You’ll use the condition system in future practical chapters, both for regular error handling and, in Chapter 25, to help in handling a tricky corner case of parsing ID3 files. Unfortunately, it’s the fate of error handling to always get short shrift in programming texts—proper error handling, or lack thereof, is often the biggest difference between illustrative code and hardened, production-quality code. The trick to writing the latter has more to do with adopting a particularly rigorous way of thinking about software than with the details of any particular programming language constructs. That said, if your goal is to write that kind of software, you’ll find the Common Lisp condition system is an excellent tool for writing robust code and one that fits quite nicely into Common Lisp’s incremental development style.

Image from book
WRITING ROBUST SOFTWARE

For information on writing robust software, you could do worse than to start by finding a copy of Software Reliability (John Wiley & Sons, 1976) by Glenford J. Meyers. Bertrand Meyer’s writings on Design By Contract also provide a useful way of thinking about software correctness. For instance, see Chapters 11 and 12 of his Object-Oriented Software Construction (Prentice Hall, 1997). Keep in mind, however, that Bertrand Meyer is the inventor of Eiffel, a statically typed bondage and discipline language in the Algol/Ada school. While he has a lot of smart things to say about object orientation and software reliability, there’s a fairly wide gap between his view of programming and The Lisp Way. Finally, for an excellent overview of the larger issues surrounding building fault-tolerant systems, see Chapter 3 of the classic Transaction Processing: Concepts and Techniques (Morgan Kaufmann, 1993) by Jim Gray and Andreas Reuter.

Image from book

In the next chapter I’ll give a quick overview of some of the 25 special operators you haven’t had a chance to use yet, at least not directly.


931 - Unwinding the Stack

Chapter 20 - The Special Operators
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Unwinding the Stack

Another aspect of the language that special operators give you control over is the behavior of the call stack. For instance, while you normally use BLOCK and TAGBODY to manage the flow of control within a single function, you can also use them, in conjunction with closures, to force an immediate nonlocal return from a function further down on the stack. That’s because BLOCK names and TAGBODY tags can be closed over by any code within the lexical scope of the BLOCK or TAGBODY. For example, consider this function:

(defun foo ()
  (format t "Entering foo~%")
  (block a
    (format t " Entering BLOCK~%")
    (bar #'(lambda () (return-from a)))
    (format t " Leaving BLOCK~%"))
  (format t "Leaving foo~%"))

The anonymous function passed to bar uses RETURN-FROM to return from the BLOCK. But that RETURN-FROM doesn’t get evaluated until the anonymous function is invoked with FUNCALL or APPLY. Now suppose bar looks like this:

(defun bar (fn)
  (format t "  Entering bar~%")
  (baz fn)
  (format t "  Leaving bar~%"))

Still, the anonymous function isn’t invoked. Now look at baz.

(defun baz (fn)
  (format t "   Entering baz~%")
  (funcall fn)
  (format t "   Leaving baz~%"))

Finally the function is invoked. But what does it mean to RETURN-FROM a block that’s several layers up on the call stack? Turns out it works fine—the stack is unwound back to the frame where the BLOCK was established and control returns from the BLOCK. The FORMAT expressions in foo, bar, and baz show this:

CL-USER> (foo)
Entering foo
 Entering BLOCK
  Entering bar
   Entering baz
Leaving foo
NIL

Note that the only “Leaving …” message that prints is the one that appears after the BLOCK in foo.

Because the names of blocks are lexically scoped, a RETURN-FROM always returns from the smallest enclosing BLOCK in the lexical environment where the RETURN-FROM form appears even if the RETURN-FROM is executed in a different dynamic context. For instance, bar could also contain a BLOCK named a, like this:

(defun bar (fn)
  (format t "  Entering bar~%")
  (block a (baz fn))
  (format t "  Leaving bar~%"))

This extra BLOCK won’t change the behavior of foo at all—the name a is resolved lexically, at compile time, not dynamically, so the intervening block has no effect on the RETURN-FROM. Conversely, the name of a BLOCK can be used only by RETURN-FROMs appearing within the lexical scope of the BLOCK; there’s no way for code outside the block to return from the block except by invoking a closure that closes over a RETURN-FROM from the lexical scope of the BLOCK.

TAGBODY and GO work the same way, in this regard, as BLOCK and RETURN-FROM. When you invoke a closure that contains a GO form, if the GO is evaluated, the stack will unwind back to the appropriate TAGBODY and then jump to the specified tag.

BLOCK names and TAGBODY tags, however, differ from lexical variable bindings in one important way. As I discussed in Chapter 6, lexical bindings have indefinite extent, meaning the bindings can stick around even after the binding form has returned. BLOCKs and TAGBODYs, on the other hand, have dynamic extent—you can RETURN-FROM a BLOCK or GO to a TAGBODY tag only while the BLOCK or TAGBODY is on the call stack. In other words, a closure that captures a block name or TAGBODY tag can be passed down the stack to be invoked later, but it can’t be returned up the stack. If you invoke a closure that tries to RETURN-FROM a BLOCK, after the BLOCK itself has returned, you’ll get an error. Likewise, trying to GO to a TAGBODY that no longer exists will cause an error.[7]

It’s unlikely you’ll need to use BLOCK and TAGBODY yourself for this kind of stack unwinding. But you’ll likely be using them indirectly whenever you use the condition system, so understanding how they work should help you understand better what exactly, for instance, invoking a restart is doing.[8]

CATCH and THROW are another pair of special operators that can force the stack to unwind. You’ll use these operators even less often than the others mentioned so far—they’re holdovers from earlier Lisp dialects that didn’t have Common Lisp’s condition system. They definitely shouldn’t be confused with try/catch and try/except constructs from languages such as Java and Python.

CATCH and THROW are the dynamic counterparts of BLOCK and RETURN-FROM. That is, you wrap CATCH around a body of code and then use THROW to cause the CATCH form to return immediately with a specified value. The difference is that the association between a CATCH and THROW is established dynamically—instead of a lexically scoped name, the label for a CATCH is an object, called a catch tag, and any THROW evaluated within the dynamic extent of the CATCH that throws that object will unwind the stack back to the CATCH form and cause it to return immediately. Thus, you can write a version of the foo, bar, and baz functions from before using CATCH and THROW instead of BLOCK and RETURN-FROM like this:

(defparameter *obj* (cons nil nil)) ; i.e. some arbitrary object

(defun foo ()
  (format t "Entering foo~%")
  (catch *obj*
    (format t " Entering CATCH~%")
    (bar)
    (format t " Leaving CATCH~%"))
  (format t "Leaving foo~%"))

(defun bar ()
  (format t "  Entering bar~%")
  (baz)
  (format t "  Leaving bar~%"))

(defun baz ()
  (format t "   Entering baz~%")
  (throw *obj* nil)
  (format t "   Leaving baz~%"))

Notice how it isn’t necessary to pass a closure down the stack—baz can call THROW directly. The result is quite similar to the earlier version.

CL-USER> (foo)
Entering foo
 Entering CATCH
  Entering bar
   Entering baz
Leaving foo
NIL

However, CATCH and THROW are almost too dynamic. In both the CATCH and the THROW, the tag form is evaluated, which means their values are both determined at runtime. Thus, if some code in bar reassigned or rebound *obj*, the THROW in baz wouldn’t throw to the same CATCH. This makes CATCH and THROW much harder to reason about than BLOCK and RETURN-FROM. The only advantage, which the version of foo, bar, and baz that use CATCH and THROW demonstrates, is there’s no need to pass down a closure in order for low-level code to return from a CATCH—any code that runs within the dynamic extent of a CATCH can cause it to return by throwing the right object.

In older Lisp dialects that didn’t have anything like Common Lisp’s condition system, CATCH and THROW were used for error handling. However, to keep them manageable, the catch tags were usually just quoted symbols, so you could tell by looking at a CATCH and a THROW whether they would hook up at runtime. In Common Lisp you’ll rarely have any call to use CATCH and THROW since the condition system is so much more flexible.

The last special operator related to controlling the stack is another one I’ve mentioned in passing before—UNWIND-PROTECT. UNWIND-PROTECT lets you control what happens as the stack unwinds—to make sure that certain code always runs regardless of how control leaves the scope of the UNWIND-PROTECT, whether by a normal return, by a restart being invoked, or by any of the ways discussed in this section.[9] The basic skeleton of UNWIND-PROTECT looks like this:

(unwind-protect protected-form
  cleanup-form*)

The single protected-form is evaluated, and then, regardless of how it returns, the cleanup-forms are evaluated. If the protected-form returns normally, then whatever it returns is returned from the UNWIND-PROTECT after the cleanup forms run. The cleanup forms are evaluated in the same dynamic environment as the UNWIND-PROTECT, so the same dynamic variable bindings, restarts, and condition handlers will be visible to code in cleanup forms as were visible just before the UNWIND-PROTECT.

You’ll occasionally use UNWIND-PROTECT directly. More often you’ll use it as the basis for WITH- style macros, similar to WITH-OPEN-FILE, that evaluate any number of body forms in a context where they have access to some resource that needs to be cleaned up after they’re done, regardless of whether they return normally or bail via a restart or other nonlocal exit. For example, if you were writing a database library that defined functions open-connection and close-connection, you might write a macro like this:[10]

(defmacro with-database-connection ((var &rest open-args) &body body)
  `(let ((,var (open-connection ,@open-args)))
    (unwind-protect (progn ,@body)
      (close-connection ,var))))

which lets you write code like this:

(with-database-connection (conn :host "foo" :user "scott" :password "tiger")
  (do-stuff conn)
  (do-more-stuff conn))

and not have to worry about closing the database connection, since the UNWIND-PROTECT will make sure it gets closed no matter what happens in the body of the with-database-connection form.

[7]This is a pretty reasonable restriction—it’s not entirely clear what it’d mean to return from a form that has already returned—unless, of course, you’re a Scheme programmer. Scheme supports continuations, a language construct that makes it possible to return from the same function call more than once. But for a variety of reasons, few, if any, languages other than Scheme support this kind of continuation.

[8]If you’re the kind of person who likes to know how things work all the way down to the bits, it may be instructive to think about how you might implement the condition system’s macros using BLOCK, TAGBODY, closures, and dynamic variables.

[9]UNWIND-PROTECT is essentially equivalent to try/finally constructs in Java and Python.

[10]And indeed, CLSQL, the multi-Lisp, multidatabase SQL interface library, provides a similar macro called with-database. CLSQL’s home page is at http://clsql.b9.com.


941 - Importing Individual Names

Chapter 21 - Programming in the Large—Packages and Symbols
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Importing Individual Names

Now suppose you find a third-party library of functions for manipulating e-mail messages. The names used in the library’s API are exported from the package COM.ACME.EMAIL, so you could :use that package to get easy access to those names. But suppose you need to use only one function from this library, and other exported symbols conflict with names you already use (or plan to use) in our own code.[11] In this case, you can import the one symbol you need with an :import-from clause in the DEFPACKAGE. For instance, if the name of the function you want to use is parse-email-address, you can change the DEFPACKAGE to this:

(defpackage :com.gigamonkeys.email-db
  (:use :common-lisp :com.gigamonkeys.text-db)
  (:import-from :com.acme.email :parse-email-address))

Now anywhere the name parse-email-address appears in code read in the COM.GIGAMONKEYS.EMAIL-DB package, it will be read as the symbol from COM.ACME.EMAIL. If you need to import more than one symbol from a single package, you can include multiple names after the package name in a single :import-from clause. A DEFPACKAGE can also include multiple :import-from clauses in order to import symbols from different packages.

Occasionally you’ll run into the opposite situation—a package may export a bunch of names you want to use and a few you don’t. Rather than listing all the symbols you do want to use in an :import-from clause, you can instead :use the package and then list the names you don’t want to inherit in a :shadow clause. For instance, suppose the COM.ACME.TEXT package exports a bunch of names of functions and classes used in text processing. Further suppose that most of these functions and classes are ones you’ll want to use in your code, but one of the names, build-index, conflicts with a name you’ve already used. You can make the build-index from COM.ACME.TEXT inaccessible by shadowing it.

(defpackage :com.gigamonkeys.email-db
  (:use
   :common-lisp
   :com.gigamonkeys.text-db
   :com.acme.text)
  (:import-from :com.acme.email :parse-email-address)
  (:shadow :build-index))

The :shadow clause causes a new symbol named BUILD-INDEX to be created and added directly to COM.GIGAMONKEYS.EMAIL-DB’s name-to-symbol map. Now if the reader reads the name BUILD-INDEX, it will translate it to the symbol in COM.GIGAMONKEYS.EMAIL-DB’s map, rather than the one that would otherwise be inherited from COM.ACME.TEXT. The new symbol is also added to a shadowing symbols list that’s part of the COM.GIGAMONKEYS.EMAIL-DB package, so if you later use another package that also exports a BUILD-INDEX symbol, the package system will know there’s no conflict—that you want the symbol from COM.GIGAMONKEYS.EMAIL-DB to be used rather than any other symbols with the same name inherited from other packages.

A similar situation can arise if you want to use two packages that export the same name. In this case the reader won’t know which inherited name to use when it reads the textual name. In such situations you must resolve the ambiguity by shadowing the conflicting names. If you don’t need to use the name from either package, you could shadow the name with a :shadow clause, creating a new symbol with the same name in your package. But if you actually want to use one of the inherited symbols, then you need to resolve the ambiguity with a :shadowing-import-from clause. Like an :import-from clause, a :shadowing-import-from clause consists of a package name followed by the names to import from that package. For instance, if COM.ACME.TEXT exports a name SAVE that conflicts with the name exported from COM.GIGAMONKEYS.TEXT-DB, you could resolve the ambiguity with the following DEFPACKAGE:

(defpackage :com.gigamonkeys.email-db
  (:use
   :common-lisp
   :com.gigamonkeys.text-db
   :com.acme.text)
  (:import-from :com.acme.email :parse-email-address)
  (:shadow :build-index)
  (:shadowing-import-from :com.gigamonkeys.text-db :save))

[11]During development, if you try to :use a package that exports a symbol with the same name as a symbol already interned in the using package, Lisp will signal an error and typically offer you a restart that will unintern the offending symbol from the using package. For more on this, see the section “Package Gotchas.”


942 - Packaging Mechanics

Chapter 21 - Programming in the Large—Packages and Symbols
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Packaging Mechanics

That covers the basics of how to use packages to manage namespaces in several common situations. However, another level of how to use packages is worth discussing—the raw mechanics of how to organize code that uses different packages. In this section I’ll discuss a few rules of thumb about how to organize code—where to put your DEFPACKAGE forms relative to the code that uses your packages via IN-PACKAGE.

Because packages are used by the reader, a package must be defined before you can LOAD or COMPILE-FILE a file that contains an IN-PACKAGE expression switching to that package. Packages also must be defined before other DEFPACKAGE forms can refer to them. For instance, if you’re going to :use COM.GIGAMONKEYS.TEXT-DB in COM.GIGAMONKEYS.EMAIL-DB, then COM.GIGAMONKEYS.TEXT-DB’s DEFPACKAGE must be evaluated before the DEFPACKAGE of COM.GIGAMONKEYS.EMAIL-DB.

The best first step toward making sure packages exist when they need to is to put all your DEFPACKAGEs in files separate from the code that needs to be read in those packages. Some folks like to create a foo-package.lisp file for each individual package, and others create a single packages.lisp that contains all the DEFPACKAGE forms for a group of related packages. Either approach is reasonable, though the one-file-per-package approach also requires that you arrange to load the individual files in the right order according to the interpackage dependencies.

Either way, once all the DEFPACKAGE forms have been separated from the code that will be read in the packages they define, you can arrange to LOAD the files containing the DEFPACKAGEs before you compile or load any of the other files. For simple programs you can do this by hand: simply LOAD the file or files containing the DEFPACKAGE forms, possibly compiling them with COMPILE-FILE first. Then LOAD the files that use those packages, again optionally compiling them first with COMPILE-FILE. Note, however, that the packages don’t exist until you LOAD the package definitions, either the source or the files produced by COMPILE-FILE. Thus, if you’re compiling everything, you must still LOAD all the package definitions before you can COMPILE-FILE any files to be read in the packages.

Doing these steps by hand will get tedious after a while. For simple programs you can automate the steps by writing a file, load.lisp, that contains the appropriate LOAD and COMPILE-FILE calls in the right order. Then you can just LOAD that file. For more complex programs you’ll want to use a system definition facility to manage loading and compiling files in the right order.[12]

The other key rule of thumb is that each file should contain exactly one IN-PACKAGE form, and it should be the first form in the file other than comments. Files containing DEFPACKAGE forms should start with (in-package "COMMON-LISP-USER"), and all other files should contain an IN-PACKAGE of one of your packages.

If you violate this rule and switch packages in the middle of a file, you’ll confuse human readers who don’t notice the second IN-PACKAGE. Also, many Lisp development environments, particularly Emacs-based ones such as SLIME, look for an IN-PACKAGE to determine the package they should use when communicating with Common Lisp. Multiple IN-PACKAGE forms per file may confuse these tools as well.

On the other hand, it’s fine to have multiple files read in the same package, each with an identical IN-PACKAGE form. It’s just a matter of how you like to organize your code.

The other bit of packaging mechanics has to do with how to name packages. Package names live in a flat namespace—package names are just strings, and different packages must have textually distinct names. Thus, you have to consider the possibility of conflicts between package names. If you’re using only packages you developed yourself, then you can probably get away with using short names for your packages. But if you’re planning to use third-party libraries or to publish your code for use by other programmers, then you need to follow a naming convention that will minimize the possibility of name collisions between different packages. Many Lispers these days are adopting Java-style names, like the ones used in this chapter, consisting of a reversed Internet domain name followed by a dot and a descriptive string.

[12]The code for the “Practical” chapters, available from this book’s Web site, uses the ASDF system definition library. ASDF stands for Another System Definition Facility.


948 - Looping Over Collections and Packages

Chapter 22 - LOOP for Black Belts
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Looping Over Collections and Packages

The for clauses for iterating over lists are much simpler than the arithmetic clauses. They support only two prepositional phrases, in and on.

A phrase of this form:

for var in list-form

steps var over all the elements of the list produced by evaluating list-form.

(loop for i in (list 10 20 30 40) collect i)           → (10 20 30 40)

Occasionally this clause is supplemented with a by phrase, which specifies a function to use to move down the list. The default is CDR but can be any function that takes a list and returns a sublist. For instance, you could collect every other element of a list with a loop like this:

(loop for i in (list 10 20 30 40) by #'cddr collect i) → (10 30)

An on prepositional phrase is used to step var over the cons cells that make up a list.

(loop for x on (list 10 20 30) collect x) → ((10 20 30) (20 30) (30))

This phrase too can take a by preposition:

(loop for x on (list 10 20 30 40) by #'cddr collect x) → ((10 20 30 40) (30 40))

Looping over the elements of a vector (which includes strings and bit vectors) is similar to looping over the elements of a list except the preposition across is used instead of in.[3] For instance:

(loop for x across "abcd" collect x) → (#\a #\b #\c #\d)

Iterating over a hash table or package is slightly more complicated because hash tables and packages have different sets of values you might want to iterate over—the keys or values in a hash table and the different kinds of symbols in a package. Both kinds of iteration follow the same pattern. The basic pattern looks like this:

(loop for var being the things in hash-or-package ...)

For hash tables, the possible values for things are hash-keys and hash-values, which cause var to be bound to successive values of either the keys or the values of the hash table. The hash-or-package form is evaluated once to produce a value, which must be a hash table.

To iterate over a package, things can be symbols, present-symbols, and external-symbols, which cause var to be bound to each of the symbols accessible in a package, each of the symbols present in a package (in other words, interned or imported into that package), or each of the symbols that have been exported from the package. The hash-or-package form is evaluated to produce the name of a package, which is looked up as if by FIND-PACKAGE or a package object. Synonyms are also available for parts of the for clause. In place of the, you can use each; you can use of instead of in; and you can write the things in the singular (for example, hash-key or symbol).

Finally, since you’ll often want both the keys and the values when iterating over a hash table, the hash table clauses support a using subclause at the end of the hash table clause.

(loop for k being the hash-keys in h using (hash-value v) ...)
(loop for v being the hash-values in h using (hash-key k) ...)

Both of these loops will bind k to each key in the hash table and v to the corresponding value. Note that the first element of the using subclause must be in the singular form.[4]

[3]You may wonder why LOOP can’t figure out whether it’s looping over a list or a vector without needing different prepositions. This is another consequence of LOOP being a macro: the value of the list or vector won’t be known until runtime, but LOOP, as a macro, has to generate code at compile time. And LOOP’s designers wanted it to generate extremely efficient code. To be able to generate efficient code for looping across, say, a vector, it needs to know at compile time that the value will be a vector at runtime—thus, the different prepositions are needed.

[4]Don’t ask me why LOOP’s authors chickened out on the no-parentheses style for the using subclause.


949 - Equals-Then Iteration

Chapter 22 - LOOP for Black Belts
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Equals-Then Iteration

If none of the other for clauses supports exactly the form of variable stepping you need, you can take complete control over stepping with an equals-then clause. This clause is similar to the binding clauses in a DO loop but cast in a more Algolish syntax. The template is as follows:

(loop for var = initial-value-form [ then step-form ] ...)

As usual, var is the name of the variable to be stepped. Its initial value is obtained by evaluating initial-value-form once before the first iteration. In each subsequent iteration, step-form is evaluated, and its value becomes the new value of var. With no then part to the clause, the initial-value-form is reevaluated on each iteration to provide the new value. Note that this is different from a DO binding clause with no step form.

The step-form can refer to other loop variables, including variables created by other for clauses later in the loop. For instance:

(loop repeat 5
      for x = 0 then y
      for y = 1 then (+ x y)
      collect y)  →  (1 2 4 8 16)

However, note that each for clause is evaluated separately in the order it appears. So in the previous loop, on the second iteration x is set to the value of y before y changes (in other words, 1). But y is then set to the sum of its old value (still 1) and the new value of x. If the order of the for clauses is reversed, the results change.

(loop repeat 5
      for y = 1 then (+ x y)
      for x = 0 then y
      collect y)  → (1 1 2 4 8)

Often, however, you’ll want the step forms for multiple variables to be evaluated before any of the variables is given its new value (similar to how DO steps its variables). In that case, you can join multiple for clauses by replacing all but the first for with and. You saw this formulation already in the LOOP version of the Fibonacci computation in Chapter 7. Here’s another variant, based on the two previous examples:

(loop repeat 5
      for x = 0 then y
      and y = 1 then (+ x y)
      collect y)  →  (1 1 2 3 5)


952 - Value Accumulation

Chapter 22 - LOOP for Black Belts
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Value Accumulation

The value accumulation clauses are perhaps the most powerful part of LOOP. While the iteration control clauses provide a concise syntax for expressing the basic mechanics of looping, they aren’t dramatically different from the equivalent mechanisms provided by DO, DOLIST, and DOTIMES.

The value accumulation clauses, on the other hand, provide a concise notation for a handful of common loop idioms having to do with accumulating values while looping. Each accumulation clause starts with a verb and follows this pattern:

verb form [ into var ]

Each time through the loop, an accumulation clause evaluates form and saves the value in a manner determined by the verb. With an into subclause, the value is saved into the variable named by var. The variable is local to the loop, as if it’d been declared in a with clause. With no into subclause, the accumulation clause instead accumulates a default value for the whole loop expression.

The possible verbs are collect, append, nconc, count, sum, maximize, and minimize. Also available as synonyms are the present participle forms: collecting, appending, nconcing, counting, summing, maximizing, and minimizing.

A collect clause builds up a list containing all the values of form in the order they’re seen. This is a particularly useful construct because the code you’d have to write to collect a list in order as efficiently as LOOP does is more painful than you’d normally write by hand.[5] Related to collect are the verbs append and nconc. These verbs also accumulate values into a list, but they join the values, which must be lists, into a single list as if by the functions APPEND or NCONC.[6]

The remaining accumulation clauses are used to accumulate numeric values. The verb count counts the number of times form is true, sum collects a running total of the values of form, maximize collects the largest value seen for form, and minimize collects the smallest. For instance, suppose you define a variable *random* that contains a list of random numbers.

(defparameter *random* (loop repeat 100 collect (random 10000)))

Then the following loop will return a list containing various summary information about the numbers:

(loop for i in *random*
   counting (evenp i) into evens
   counting (oddp i) into odds
   summing i into total
   maximizing i into max
   minimizing i into min
   finally (return (list min max total evens odds)))

[5]The trick is to keep ahold of the tail of the list and add new cons cells by SETFing the CDR of the tail. A handwritten equivalent of the code generated by (loop for i upto 10 collect i) would look like this:

(do ((list nil) (tail nil) (i 0 (1+ i)))
    ((> i 10) list)
  (let ((new (cons i nil)))
    (if (null list)
        (setf list new)
        (setf (cdr tail) new))
    (setf tail new)))

Of course, you’ll rarely, if ever, write code like that. You’ll use either LOOP or (if, for some reason, you don’t want to use LOOP) the standard PUSH/NREVERSE idiom for collecting values.

[6]Recall that NCONC is the destructive version of APPEND—it’s safe to use an nconc clause only if the values you’re collecting are fresh lists that don’t share any structure with other lists. For instance, this is safe:

(loop for i upto 3 nconc (list i i)) → (0 0 1 1 2 2 3 3)

But this will get you into trouble:

(loop for i on (list 1 2 3) nconc i) → undefined

The latter will most likely get into an infinite loop because the various parts of the list produced by (list 1 2 3) are destructively modified to point to each other. But even that’s not guaranteed—the behavior is simply undefined.


953 - Unconditional Execution

Chapter 22 - LOOP for Black Belts
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Unconditional Execution

As useful as the value accumulation constructs are, LOOP wouldn’t be a very good general-purpose iteration facility if there wasn’t a way to execute arbitrary Lisp code in the loop body.

The simplest way to execute arbitrary code within a loop body is with a do clause. Compared to the clauses I’ve described so far, with their prepositions and subclauses, do is a model of Yodaesque simplicity.[7] A do clause consists of the word do (or doing) followed by one or more Lisp forms that are all evaluated when the do clause is. The do clause ends at the closing parenthesis of the loop or the next loop keyword.

For instance, to print the numbers from one to ten, you could write this:

(loop for i from 1 to 10 do (print i))

Another, more dramatic, form of immediate execution is a return clause. This clause consists of the word return followed by a single Lisp form, which is evaluated, with the resulting value immediately returned as the value of the loop.

You can also break out of a loop in a do clause using any of Lisp’s normal control flow operators, such as RETURN and RETURN-FROM. Note that a return clause always returns from the immediately enclosing LOOP expression, while a RETURN or RETURN-FROM in a do clause can return from any enclosing expression. For instance, compare the following:

(block outer
  (loop for i from 0 return 100) ; 100 returned from LOOP
  (print "This will print")
  200) →  200

to this:

(block outer
  (loop for i from 0 do (return-from outer 100)) ; 100 returned from BLOCK
  (print "This won't print")
  200) →  100

The do and return clauses are collectively called the unconditional execution clauses.

[7]“No! Try not. Do … or do not. There is no try.” —Yoda, The Empire Strikes Back


955 - Setting Up and Tearing Down

Chapter 22 - LOOP for Black Belts
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Setting Up and Tearing Down

One of the key insights the designers of the LOOP language had about actual loops “in the wild” is that the loop proper is often preceded by a bit of code to set things up and then followed by some more code that does something with the values computed by the loop. A trivial example, in Perl,[8] might look like this:

my $evens_sum = 0;
my $odds_sum  = 0;
foreach my $i (@list_of_numbers) {
  if ($i % 2) {
    $odds_sum += $i;
  } else {
    $evens_sum += $i;
  }
}
if ($evens_sum > $odds_sum) {
  print "Sum of evens greater\n";
} else {
  print "Sum of odds greater\n";
}

The loop proper in this code is the foreach statement. But the foreach loop doesn’t stand on its own: the code in the loop body refers to variables declared in the two lines before the loop.[9] And the work the loop does is all for naught without the if statement after the loop that actually reports the results. In Common Lisp, of course, the LOOP construct is an expression that returns a value, so there’s even more often a need to do something after the loop proper, namely, generate the return value.

So, said the LOOP designers, let’s give a way to include the code that’s really part of the loop in the loop itself. Thus, LOOP provides two keywords, initially and finally, that introduce code to be run outside the loop’s main body.

After the initially or finally, these clauses consist of all the Lisp forms up to the start of the next loop clause or the end of the loop. All the initially forms are combined into a single prologue, which runs once, immediately after all the local loop variables are initialized and before the body of the loop. The finally forms are similarly combined into a epilogue to be run after the last iteration of the loop body. Both the prologue and epilogue code can refer to local loop variables.

The prologue is always run, even if the loop body iterates zero times. The loop can return without running the epilogue if any of the following happens:

  • A return clause executes.

  • RETURN, RETURN-FROM, or another transfer of control construct is called from within a Lisp form within the body.[10]

  • The loop is terminated by an always, never, or thereis clause, as I’ll discuss in the next section.

Within the epilogue code, RETURN or RETURN-FROM can be used to explicitly provide a return value for the loop. Such an explicit return value will take precedence over any value that might otherwise be provided by an accumulation or termination test clause.

To allow RETURN-FROM to be used to return from a specific loop (useful when nesting LOOP expressions), you can name a LOOP with the loop keyword named. If a named clause appears in a loop, it must be the first clause. For a simple example, assume lists is a list of lists and you want to find an item that matches some criteria in one of those nested lists. You could find it with a pair of nested loops like this:

(loop named outer for list in lists do
     (loop for item in list do
          (if (what-i-am-looking-for-p item)
            (return-from outer item))))

[8]I’m not picking on Perl here—this example would look pretty much the same in any language that bases its syntax on C’s.

[9]Perl would let you get away with not declaring those variables if your program didn’t use strict. But you should always use strict in Perl. The equivalent code in Python, Java, or C would always require the variables to be declared.

[10]You can cause a loop to finish normally, running the epilogue, from Lisp code executed as part of the loop body with the local macro LOOP-FINISH.


959 - The Heart of a Spam Filter

Chapter 23 - Practical—A Spam Filter
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

The Heart of a Spam Filter

In this chapter, you’ll implement the core of a spam-filtering engine. You won’t write a soup-to-nuts spam-filtering application; rather, you’ll focus on the functions for classifying new messages and training the filter.

This application is going to be large enough that it’s worth defining a new package to avoid name conflicts. For instance, in the source code you can download from this book’s Web site, I use the package name COM.GIGAMONKEYS.SPAM, defining a package that uses both the standard COMMON-LISP package and the COM.GIGAMONKEYS.PATHNAMES package from Chapter 15, like this:

(defpackage :com.gigamonkeys.spam
  (:use :common-lisp :com.gigamonkeys.pathnames))

Any file containing code for this application should start with this line:

(in-package :com.gigamonkeys.spam)

You can use the same package name or replace com.gigamonkeys with some domain you control.[3]

You can also type this same form at the REPL to switch to this package to test the functions you write. In SLIME this will change the prompt from CL-USER> to SPAM> like this:

CL-USER> (in-package :com.gigamonkeys.spam)
#<The COM.GIGAMONKEYS.SPAM package>
SPAM>

Once you have a package defined, you can start on the actual code. The main function you’ll need to implement has a simple job—take the text of a message as an argument and classify the message as spam, ham, or unsure. You can easily implement this basic function by defining it in terms of other functions that you’ll write in a moment.

(defun classify (text)
  (classification (score (extract-features text))))

Reading from the inside out, the first step in classifying a message is to extract features to pass to the score function. In score you’ll compute a value that can then be translated into one of three classifications—spam, ham, or unsure—by the function classification. Of the three functions, classification is the simplest. You can assume score will return a value near 1 if the message is a spam, near 0 if it’s a ham, and near .5 if it’s unclear.

Thus, you can implement classification like this:

(defparameter *max-ham-score* .4)
(defparameter *min-spam-score* .6)

(defun classification (score)
  (cond
    ((<= score *max-ham-score*) 'ham)
    ((>= score *min-spam-score*) 'spam)
    (t 'unsure)))

The extract-features function is almost as straightforward, though it requires a bit more code. For the moment, the features you’ll extract will be the words appearing in the text. For each word, you need to keep track of the number of times it has been seen in a spam and the number of times it has been seen in a ham. A convenient way to keep those pieces of data together with the word itself is to define a class, word-feature, with three slots.

(defclass word-feature ()
  ((word
    :initarg :word
    :accessor word
    :initform (error "Must supply :word")
    :documentation "The word this feature represents.")
   (spam-count
    :initarg :spam-count
    :accessor spam-count
    :initform 0
    :documentation "Number of spams we have seen this feature in.")
   (ham-count
    :initarg :ham-count
    :accessor ham-count
    :initform 0
    :documentation "Number of hams we have seen this feature in.")))

You’ll keep the database of features in a hash table so you can easily find the object representing a given feature. You can define a special variable, *feature-database*, to hold a reference to this hash table.

(defvar *feature-database* (make-hash-table :test #'equal))

You should use DEFVAR rather than DEFPARAMETER because you don’t want *feature-database* to be reset if you happen to reload the file containing this definition during development—you might have data stored in *feature-database* that you don’t want to lose. Of course, that means if you do want to clear out the feature database, you can’t just reevaluate the DEFVAR form. So you should define a function clear-database.

(defun clear-database ()
  (setf *feature-database* (make-hash-table :test #'equal)))

To find the features present in a given message, the code will need to extract the individual words and then look up the corresponding word-feature object in *feature-database*. If *feature-database* contains no such feature, it’ll need to create a new word-feature to represent the word. You can encapsulate that bit of logic in a function, intern-feature, that takes a word and returns the appropriate feature, creating it if necessary.

(defun intern-feature (word)
  (or (gethash word *feature-database*)
      (setf (gethash word *feature-database*)
            (make-instance 'word-feature :word word))))

You can extract the individual words from the message text using a regular expression. For example, using the Common Lisp Portable Perl-Compatible Regular Expression (CL-PPCRE) library written by Edi Weitz, you can write extract-words like this:[4]

(defun extract-words (text)
  (delete-duplicates
   (cl-ppcre:all-matches-as-strings "[a-zA-Z]{3,}" text)
   :test #'string=))

Now all that remains to implement extract-features is to put extract-features and intern-feature together. Since extract-words returns a list of strings and you want a list with each string translated to the corresponding word-feature, this is a perfect time to use MAPCAR.

(defun extract-features (text)
  (mapcar #'intern-feature (extract-words text)))

You can test these functions at the REPL like this:

SPAM> (extract-words "foo bar baz")
("foo" "bar" "baz")

And you can make sure the DELETE-DUPLICATES is working like this:

SPAM> (extract-words "foo bar baz foo bar")
("baz" "foo" "bar")

You can also test extract-features.

SPAM> (extract-features "foo bar baz foo bar")
(#<WORD-FEATURE @ #x71ef28da> #<WORD-FEATURE @ #x71e3809a>
 #<WORD-FEATURE @ #x71ef28aa>)

However, as you can see, the default method for printing arbitrary objects isn’t very informative. As you work on this program, it’ll be useful to be able to print word-feature objects in a less opaque way. Luckily, as I mentioned in Chapter 17, the printing of all objects is implemented in terms of a generic function PRINT-OBJECT, so to change the way word-feature objects are printed, you just need to define a method on PRINT-OBJECT that specializes on word-feature. To make implementing such methods easier, Common Lisp provides the macro PRINT-UNREADABLE-OBJECT.[5]

The basic form of PRINT-UNREADABLE-OBJECT is as follows:

(print-unreadable-object (object stream-variable &key type identity)
  body-form*)

The object argument is an expression that evaluates to the object to be printed. Within the body of PRINT-UNREADABLE-OBJECT, stream-variable is bound to a stream to which you can print anything you want. Whatever you print to that stream will be output by PRINT-UNREADABLE-OBJECT and enclosed in the standard syntax for unreadable objects, #<>.[6]

PRINT-UNREADABLE-OBJECT also lets you include the type of the object and an indication of the object’s identity via the keyword parameters type and identity. If they’re non-NIL, the output will start with the name of the object’s class and end with an indication of the object’s identity similar to what’s printed by the default PRINT-OBJECT method for STANDARD-OBJECTs. For word-feature, you probably want to define a PRINT-OBJECT method that includes the type but not the identity along with the values of the word, ham-count, and spam-count slots. Such a method would look like this:

(defmethod print-object ((object word-feature) stream)
  (print-unreadable-object (object stream :type t)
    (with-slots (word ham-count spam-count) object
      (format stream "~s :hams ~d :spams ~d" word ham-count spam-count))))

Now when you test extract-features at the REPL, you can see more clearly what features are being extracted.

SPAM> (extract-features "foo bar baz foo bar")
(#<WORD-FEATURE "baz" :hams 0 :spams 0>
 #<WORD-FEATURE "foo" :hams 0 :spams 0>
 #<WORD-FEATURE "bar" :hams 0 :spams 0>)

[3]It would, however, be poor form to distribute a version of this application using a package starting with com.gigamonkeys since you don’t control that domain.

[4]A version of CL-PPCRE is included with the book’s source code available from the book’s Web site. Or you can download it from Weitz’s site at http://www.weitz.de/cl-ppcre/.

[5]The main reason to use PRINT-UNREADABLE-OBJECT is that it takes care of signaling the appropriate error if someone tries to print your object readably, such as with the ~S FORMAT directive.

[6]PRINT-UNREADABLE-OBJECT also signals an error if it’s used when the printer control variable *PRINT-READABLY* is true. Thus, a PRINT-OBJECT method consisting solely of a PRINT-UNREADABLE-OBJECT form will correctly implement the PRINT-OBJECT contract with regard to *PRINT-READABLY*.


962 - Combining Probabilities

Chapter 23 - Practical—A Spam Filter
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Combining Probabilities

Now that you can compute the bayesian-spam-probability of each individual feature you find in a message, the last step in implementing the score function is to find a way to combine a bunch of individual probabilities into a single value between 0 and 1.

If the individual feature probabilities were independent, then it’d be mathematically sound to multiply them together to get a combined probability. But it’s unlikely they actually are independent—certain features are likely to appear together, while others never do.[11]

Robinson proposed using a method for combining probabilities invented by the statistician R. A. Fisher. Without going into the details of exactly why his technique works, it’s this: First you combine the probabilities by multiplying them together. This gives you a number nearer to 0 the more low probabilities there were in the original set. Then take the log of that number and multiply by –2. Fisher showed in 1950 that if the individual probabilities were independent and drawn from a uniform distribution between 0 and 1, then the resulting value would be on a chi-square distribution. This value and twice the number of probabilities can be fed into an inverse chi-square function, and it’ll return the probability that reflects the likelihood of obtaining a value that large or larger by combining the same number of randomly selected probabilities. When the inverse chi-square function returns a low probability, it means there was a disproportionate number of low probabilities (either a lot of relatively low probabilities or a few very low probabilities) in the individual probabilities.

To use this probability in determining whether a given message is a spam, you start with a null hypothesis, a straw man you hope to knock down. The null hypothesis is that the message being classified is in fact just a random collection of features. If it were, then the individual probabilities—the likelihood that each feature would appear in a spam—would also be random. That is, a random selection of features would usually contain some features with a high probability of appearing in spam and other features with a low probability of appearing in spam. If you were to combine these randomly selected probabilities according to Fisher’s method, you should get a middling combined value, which the inverse chi-square function will tell you is quite likely to arise just by chance, as, in fact, it would have. But if the inverse chi-square function returns a very low probability, it means it’s unlikely the probabilities that went into the combined value were selected at random; there were too many low probabilities for that to be likely. So you can reject the null hypothesis and instead adopt the alternative hypothesis that the features involved were drawn from a biased sample—one with few high spam probability features and many low spam probability features. In other words, it must be a ham message.

However, the Fisher method isn’t symmetrical since the inverse chi-square function returns the probability that a given number of randomly selected probabilities would combine to a value as large or larger than the one you got by combining the actual probabilities. This asymmetry works to your advantage because when you reject the null hypothesis, you know what the more likely hypothesis is. When you combine the individual spam probabilities via the Fisher method, and it tells you there’s a high probability that the null hypothesis is wrong—that the message isn’t a random collection of words—then it means it’s likely the message is a ham. The number returned is, if not literally the probability that the message is a ham, at least a good measure of its “hamminess.” Conversely, the Fisher combination of the individual ham probabilities gives you a measure of the message’s “spamminess.”

To get a final score, you need to combine those two measures into a single number that gives you a combined hamminess-spamminess score ranging from 0 to 1. The method recommended by Robinson is to add half the difference between the hamminess and spamminess scores to 1/2, in other words, to average the spamminess and 1 minus the hamminess. This has the nice effect that when the two scores agree (high spamminess and low hamminess, or vice versa) you’ll end up with a strong indicator near either 0 or 1. But when the spamminess and hamminess scores are both high or both low, then you’ll end up with a final value near 1/2, which you can treat as an “uncertain” classification.

The score function that implements this scheme looks like this:

(defun score (features)
  (let ((spam-probs ()) (ham-probs ()) (number-of-probs 0))
    (dolist (feature features)
      (unless (untrained-p feature)
        (let ((spam-prob (float (bayesian-spam-probability feature) 0.0d0)))
          (push spam-prob spam-probs)
          (push (- 1.0d0 spam-prob) ham-probs)
          (incf number-of-probs))))
    (let ((h (- 1 (fisher spam-probs number-of-probs)))
          (s (- 1 (fisher ham-probs number-of-probs))))
      (/ (+ (- 1 h) s) 2.0d0))))

You take a list of features and loop over them, building up two lists of probabilities, one listing the probabilities that a message containing each feature is a spam and the other that a message containing each feature is a ham. As an optimization, you can also count the number of probabilities while looping over them and pass the count to fisher to avoid having to count them again in fisher itself. The value returned by fisher will be low if the individual probabilities contained too many low probabilities to have come from random text. Thus, a low fisher score for the spam probabilities means there were many hammy features; subtracting that score from 1 gives you a probability that the message is a ham. Conversely, subtracting the fisher score for the ham probabilities gives you the probability that the message was a spam. Combining those two probabilities gives you an overall spamminess score between 0 and 1.

Within the loop, you can use the function untrained-p to skip features extracted from the message that were never seen during training. These features will have spam counts and ham counts of zero. The untrained-p function is trivial.

(defun untrained-p (feature)
  (with-slots (spam-count ham-count) feature
    (and (zerop spam-count) (zerop ham-count))))

The only other new function is fisher itself. Assuming you already had an inverse-chi-square function, fisher is conceptually simple.

(defun fisher (probs number-of-probs)
  "The Fisher computation described by Robinson."
  (inverse-chi-square
   (* -2 (log (reduce #'* probs)))
   (* 2 number-of-probs)))

Unfortunately, there’s a small problem with this straightforward implementation. While using REDUCE is a concise and idiomatic way of multiplying a list of numbers, in this particular application there’s a danger the product will be too small a number to be represented as a floating-point number. In that case, the result will underflow to zero. And if the product of the probabilities underflows, all bets are off because taking the LOG of zero will either signal an error or, in some implementation, result in a special negative-infinity value, which will render all subsequent calculations essentially meaningless. This is particularly unfortunate in this function because the Fisher method is most sensitive when the input probabilities are low—near zero—and therefore in the most danger of causing the multiplication to underflow.

Luckily, you can use a bit of high-school math to avoid this problem. Recall that the log of a product is the same as the sum of the logs of the factors. So instead of multiplying all the probabilities and then taking the log, you can sum the logs of each probability. And since REDUCE takes a :key keyword parameter, you can use it to perform the whole calculation. Instead of this:

(log (reduce #'* probs))

write this:

(reduce #'+ probs :key #'log)

[11]Techniques that combine nonindependent probabilities as though they were, in fact, independent, are called naive Bayesian. Graham’s original proposal was essentially a naive Bayesian classifier with some “empirically derived” constant factors thrown in.


964 - Training the Filter

Chapter 23 - Practical—A Spam Filter
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Training the Filter

Since you wrote classify and train to take a string argument, you can test them easily at the REPL. If you haven’t yet, you should switch to the package in which you’ve been writing this code by evaluating an IN-PACKAGE form at the REPL or using the SLIME shortcut change-package. To use the SLIME shortcut, type a comma at the REPL and then type the name at the prompt. Pressing Tab while typing the package name will autocomplete based on the packages your Lisp knows about. Now you can invoke any of the functions that are part of the spam application. You should first make sure the database is empty.

SPAM> (clear-database)

Now you can train the filter with some text.

SPAM> (train "Make money fast" 'spam)

And then see what the classifier thinks.

SPAM> (classify "Make money fast")
SPAM
SPAM> (classify "Want to go to the movies?")
UNSURE

While ultimately all you care about is the classification, it’d be nice to be able to see the raw score too. The easiest way to get both values without disturbing any other code is to change classification to return multiple values.

(defun classification (score)
  (values
   (cond
     ((<= score *max-ham-score*) 'ham)
     ((>= score *min-spam-score*) 'spam)
     (t 'unsure))
   score))

You can make this change and then recompile just this one function. Because classify returns whatever classification returns, it’ll also now return two values. But since the primary return value is the same, callers of either function who expect only one value won’t be affected. Now when you test classify, you can see exactly what score went into the classification.

SPAM> (classify "Make money fast")
SPAM
0.863677101854273D0
SPAM> (classify "Want to go to the movies?")
UNSURE
0.5D0

And now you can see what happens if you train the filter with some more ham text.

SPAM> (train "Do you have any money for the movies?" 'ham)
1
SPAM> (classify "Make money fast")
SPAM
0.7685351219857626D0

It’s still spam but a bit less certain since money was seen in ham text.

SPAM> (classify "Want to go to the movies?")
HAM
0.17482223132078922D0

And now this is clearly recognizable ham thanks to the presence of the word movies, now a hammy feature.

However, you don’t really want to train the filter by hand. What you’d really like is an easy way to point it at a bunch of files and train it on them. And if you want to test how well the filter actually works, you’d like to then use it to classify another set of files of known types and see how it does. So the last bit of code you’ll write in this chapter will be a test harness that tests the filter on a corpus of messages of known types, using a certain fraction for training and then measuring how accurate the filter is when classifying the remainder.


968 - What’s Next

Chapter 23 - Practical—A Spam Filter
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

What’s Next

Obviously, you could do a lot more with this code. To turn it into a real spam-filtering application, you’d need to find a way to integrate it into your normal e-mail infrastructure. One approach that would make it easy to integrate with almost any e-mail client is to write a bit of code to act as a POP3 proxy—that’s the protocol most e-mail clients use to fetch mail from mail servers. Such a proxy would fetch mail from your real POP3 server and serve it to your mail client after either tagging spam with a header that your e-mail client’s filters can easily recognize or simply putting it aside. Of course, you’d also need a way to communicate with the filter about misclassifications—as long as you’re setting it up as a server, you could also provide a Web interface. I’ll talk about how to write Web interfaces in Chapter 26, and you’ll build one, for a different application, in Chapter 29.

Or you might want to work on improving the basic classification—a likely place to start is to make extract-features more sophisticated. In particular, you could make the tokenizer smarter about the internal structure of e-mail—you could extract different kinds of features for words appearing in the body versus the message headers. And you could decode various kinds of message encoding such as base 64 and quoted printable since spammers often try to obfuscate their message with those encodings.

But I’ll leave those improvements to you. Now you’re ready to head down the path of building a streaming MP3 server, starting by writing a general-purpose library for parsing binary files.


971 - Strings in Binary Files

Chapter 24 - Practical—Parsing Binary Files
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Strings in Binary Files

Textual strings are another kind of primitive data type you’ll find in many binary formats. When you read files one byte at a time, you can’t read and write strings directly—you need to decode and encode them one byte at a time, just as you do with binary-encoded numbers. And just as you can encode an integer in several ways, you can encode a string in many ways. To start with, the binary format must specify how individual characters are encoded.

To translate bytes to characters, you need to know both what character code and what character encoding you’re using. A character code defines a mapping from positive integers to characters. Each number in the mapping is called a code point. For instance, ASCII is a character code that maps the numbers from 0–127 to particular characters used in the Latin alphabet. A character encoding, on the other hand, defines how the code points are represented as a sequence of bytes in a byte-oriented medium such as a file. For codes that use eight or fewer bits, such as ASCII and ISO-8859-1, the encoding is trivial—each numeric value is encoded as a single byte.

Nearly as straightforward are pure double-byte encodings, such as UCS-2, which map between 16-bit values and characters. The only reason double-byte encodings can be more complex than single-byte encodings is that you may also need to know whether the 16-bit values are supposed to be encoded in big-endian or little-endian format.

Variable-width encodings use different numbers of octets for different numeric values, making them more complex but allowing them to be more compact in many cases. For instance, UTF-8, an encoding designed for use with the Unicode character code, uses a single octet to encode the values 0–127 while using up to four octets to encode values up to 1,114,111.[6]

Since the code points from 0–127 map to the same characters in Unicode as they do in ASCII, a UTF-8 encoding of text consisting only of characters also in ASCII is the same as the ASCII encoding. On the other hand, texts consisting mostly of characters requiring four bytes in UTF-8 could be more compactly encoded in a straight double-byte encoding.

Common Lisp provides two functions for translating between numeric character codes and character objects: CODE-CHAR, which takes an numeric code and returns as a character, and CHAR-CODE, which takes a character and returns its numeric code. The language standard doesn’t specify what character encoding an implementation must use, so there’s no guarantee you can represent every character that can possibly be encoded in a given file format as a Lisp character. However, almost all contemporary Common Lisp implementations use ASCII, ISO-8859-1, or Unicode as their native character code. Because Unicode is a superset of ISO-8859-1, which is in turn a superset of ASCII, if you’re using a Unicode Lisp, CODE-CHAR and CHAR-CODE can be used directly for translating any of those three character codes.[7]

In addition to specifying a character encoding, a string encoding must also specify how to encode the length of the string. Three techniques are typically used in binary file formats.

The simplest is to not encode it but to let it be implicit in the position of the string in some larger structure: a particular element of a file may always be a string of a certain length, or a string may be the last element of a variable-length data structure whose overall size determines how many bytes are left to read as string data. Both these techniques are used in ID3 tags, as you’ll see in the next chapter.

The other two techniques can be used to encode variable-length strings without relying on context. One is to encode the length of the string followed by the character data—the parser reads an integer value (in some specified integer format) and then reads that number of characters. Another is to write the character data followed by a delimiter that can’t appear in the string such as a null character.

The different representations have different advantages and disadvantages, but when you’re dealing with already specified binary formats, you won’t have any control over which encoding is used. However, none of the encodings is particularly more difficult to read and write than any other. Here, as an example, is a function that reads a null-terminated ASCII string, assuming your Lisp implementation uses ASCII or one of its supersets such as ISO-8859-1 or full Unicode as its native character encoding:

(defconstant +null+ (code-char 0))

(defun read-null-terminated-ascii (in)
  (with-output-to-string (s)
    (loop for char = (code-char (read-byte in))
          until (char= char +null+) do (write-char char s))))

The WITH-OUTPUT-TO-STRING macro, which I mentioned in Chapter 14, is an easy way to build up a string when you don’t know how long it’ll be. It creates a STRING-STREAM and binds it to the variable name specified, s in this case. All characters written to the stream are collected into a string, which is then returned as the value of the WITH-OUTPUT-TO-STRING form.

To write a string back out, you just need to translate the characters back to numeric values that can be written with WRITE-BYTE and then write the null terminator after the string contents.

(defun write-null-terminated-ascii (string out)
  (loop for char across string
        do (write-byte (char-code char) out))
  (write-byte (char-code +null+) out))

As these examples show, the main intellectual challenge—such as it is—of reading and writing primitive elements of binary files is understanding how exactly to interpret the bytes that appear in a file and to map them to Lisp data types. If a binary file format is well specified, this should be a straightforward proposition. Actually writing functions to read and write a particular encoding is, as they say, a simple matter of programming.

Now you can turn to the issue of reading and writing more complex on-disk structures and how to map them to Lisp objects.

[6]Originally, UTF-8 was designed to represent a 31-bit character code and used up to six bytes per code point. However, the maximum Unicode code point is #x10ffff, so a UTF-8 encoding of Unicode requires at most four bytes per code point.

[7]If you need to parse a file format that uses other character codes, or if you need to parse files containing arbitrary Unicode strings using a non-Unicode-Common-Lisp implementation, you can always represent such strings in memory as vectors of integer code points. They won’t be Lisp strings, so you won’t be able to manipulate or compare them with the string functions, but you’ll still be able to do anything with them that you can with arbitrary vectors.


972 - Composite Structures

Chapter 24 - Practical—Parsing Binary Files
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Composite Structures

Since binary formats are usually used to represent data in a way that makes it easy to map to in-memory data structures, it should come as no surprise that composite on-disk structures are usually defined in ways similar to the way programming languages define in-memory structures. Usually a composite on-disk structure will consist of a number of named parts, each of which is itself either a primitive type such as a number or a string, another composite structure, or possibly a collection of such values.

For instance, an ID3 tag defined in the 2.2 version of the specification consists of a header made up of a three-character ISO-8859-1 string, which is always “ID3”; two one-byte unsigned integers that specify the major version and revision of the specification; eight bits worth of boolean flags; and four bytes that encode the size of the tag in an encoding particular to the ID3 specification. Following the header is a list of frames, each of which has its own internal structure. After the frames are as many null bytes as are necessary to pad the tag out to the size specified in the header.

If you look at the world through the lens of object orientation, composite structures look a lot like classes. For instance, you could write a class to represent an ID3 tag.

(defclass id3-tag ()
  ((identifier    :initarg :identifier    :accessor identifier)
   (major-version :initarg :major-version :accessor major-version)
   (revision      :initarg :revision      :accessor revision)
   (flags         :initarg :flags         :accessor flags)
   (size          :initarg :size          :accessor size)
   (frames        :initarg :frames        :accessor frames)))

An instance of this class would make a perfect repository to hold the data needed to represent an ID3 tag. You could then write functions to read and write instances of this class. For example, assuming the existence of certain other functions for reading the appropriate primitive data types, a read-id3-tag function might look like this:

(defun read-id3-tag (in)
  (let ((tag (make-instance 'id3-tag)))
    (with-slots (identifier major-version revision flags size frames) tag
      (setf identifier    (read-iso-8859-1-string in :length 3))
      (setf major-version (read-u1 in))
      (setf revision      (read-u1 in))
      (setf flags         (read-u1 in))
      (setf size          (read-id3-encoded-size in))
      (setf frames        (read-id3-frames in :tag-size size)))
    tag))

The write-id3-tag function would be structured similarly—you’d use the appropriate write-* functions to write out the values stored in the slots of the id3-tag object.

It’s not hard to see how you could write the appropriate classes to represent all the composite data structures in a specification along with read-foo and write-foo functions for each class and for necessary primitive types. But it’s also easy to tell that all the reading and writing functions are going to be pretty similar, differing only in the specifics of what types they read and the names of the slots they store them in. It’s particularly irksome when you consider that in the ID3 specification it takes about four lines of text to specify the structure of an ID3 tag, while you’ve already written eighteen lines of code and haven’t even written write-id3-tag yet.

What you’d really like is a way to describe the structure of something like an ID3 tag in a form that’s as compressed as the specification’s pseudocode yet that can also be expanded into code that defines the id3-tag class and the functions that translate between bytes on disk and instances of the class. Sounds like a job for a macro.


982 - Chapter 25: Practical—An ID3 Parser

Chapter 25 - Practical—An ID3 Parser
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Chapter 25: Practical—An ID3 Parser

Overview

With a library for parsing binary data, you’re ready to write some code for reading and writing an actual binary format, that of ID3 tags. ID3 tags are used to embed metadata in MP3 audio files. Dealing with ID3 tags will be a good test of the binary data library because the ID3 format is a true real-world format—a mix of engineering trade-offs and idiosyncratic design choices that does, whatever else might be said about it, get the job done. In case you missed the file-sharing revolution, here’s a quick overview of what ID3 tags are and how they relate to MP3 files.

MP3, also known as MPEG Audio Layer 3, is a format for storing compressed audio data, designed by researchers at Fraunhofer IIS and standardized by the Moving Picture Experts Group, a joint committee of the International Organization for Standardization (ISO) and the International Electrotechnical Commission (IEC). However, the MP3 format, by itself, defines only how to store audio data. That’s fine as long as all your MP3 files are managed by a single application that can store metadata externally and keep track of which metadata goes with which files. However, when people started passing around individual MP3 files on the Internet, via file-sharing systems such as Napster, they soon discovered they needed a way to embed metadata in the MP3 files themselves.

Because the MP3 standard was already codified and a fair bit of software and hardware had already been written that knew how to decode the existing MP3 format, any scheme for embedding information in an MP3 file would have to be invisible to MP3 decoders. Enter ID3.

The original ID3 format, invented by programmer Eric Kemp, consisted of 128 bytes stuck on the end of an MP3 file where it’d be ignored by most MP3 software. It consisted of four 30-character fields, one each for the song title, the album title, the artist name, and a comment; a four-byte year field; and a one-byte genre code. Kemp provided standard meanings for the first 80 genre codes. Nullsoft, the makers of Winamp, a popular MP3 player, later supplemented this list with another 60 or so genres.

This format was easy to parse but obviously quite limited. It had no way to encode names longer than 30 characters; it was limited to 256 genres, and the meaning of the genre codes had to be agreed upon by all users of ID3-aware software. There wasn’t even a way to encode the CD track number of a particular MP3 file until another programmer, Michael Mutschler, proposed embedding the track number in the comment field, separated from the rest of the comment by a null byte, so existing ID3 software, which tended to read up to the first null in each of the text fields, would ignore it. Kemp’s version is now called ID3v1, and Mutschler’s is ID3v1.1.

Limited as they were, the version 1 proposals were at least a partial solution to the metadata problem, so they were adopted by many MP3 ripping programs (which had to put the ID3 tag into the MP3 files) and MP3 players (which would extract the information in the ID3 tag to display to the user).[1]

By 1998, however, the limitations were really becoming annoying, and a new group, led by Martin Nilsson, started work on a completely new tagging scheme, which came to be called ID3v2. The ID3v2 format is extremely flexible, allowing for many kinds of information to be included, with almost no length limitations. It also takes advantage of certain details of the MP3 format to allow ID3v2 tags to be placed at the beginning of an MP3 file.

ID3v2 tags are, however, more of a challenge to parse than version 1 tags. In this chapter, you’ll use the binary data parsing library from the previous chapter to develop code that can read and write ID3v2 tags. Or at least you’ll make a reasonable start—where ID3v1 was too simple, ID3v2 is baroque to the point of being completely overengineered. Implementing every nook and cranny of the specification, especially if you want to support all three versions that have been specified, would be a fair bit of work. However, you can ignore many of the features in those specifications since they’re rarely used “in the wild.” For starters, you can ignore, for now, a whole version, 2.4, since it has not been widely adopted and mostly just adds more needless flexibility compared to version 2.3. I’ll focus on versions 2.2 and 2.3 because they’re both widely used and are different enough from each other to keep things interesting.

[1]Ripping is the process by which a song on an audio CD is converted to an MP3 file on your hard drive. These days most ripping software also automatically retrieves information about the songs being ripped from online databases such as Gracenote (née the Compact Disc Database [CDDB]) or FreeDB, which it then embeds in the MP3 files as ID3 tags.


984 - Defining a Package

Chapter 25 - Practical—An ID3 Parser
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Defining a Package

Like the other libraries you’ve developed so far, the code you’ll write in this chapter is worth putting in its own package. You’ll need to refer to functions from both the binary data and pathname libraries developed in Chapters 24 and 15 and will also want to export the names of the functions that make up the public API to this package. The following package definition does all that:

(defpackage :com.gigamonkeys.id3v2
  (:use :common-lisp
        :com.gigamonkeys.binary-data
        :com.gigamonkeys.pathnames)
  (:export
   :read-id3
   :mp3-p
   :id3-p
   :album
   :composer
   :genre
   :encoding-program
   :artist
   :part-of-set
   :track
   :song
   :year
   :size
   :translated-genre))

As usual, you can, and probably should, change the com.gigamonkeys part of the package name to your own domain.


990 - Supporting Multiple Versions of ID3

Chapter 25 - Practical—An ID3 Parser
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Supporting Multiple Versions of ID3

Currently, id3-tag is defined using define-binary-class, but if you want to support multiple versions of ID3, it makes more sense to use a define-tagged-binary-class that dispatches on the major-version value. As it turns out, all versions of ID3v2 have the same structure up to the size field. So, you can define a tagged binary class like the following that defines this basic structure and then dispatches to the appropriate version-specific subclass:

(define-tagged-binary-class id3-tag ()
  ((identifier     (iso-8859-1-string :length 3))
   (major-version  u1)
   (revision       u1)
   (flags          u1)
   (size           id3-tag-size))
  (:dispatch
   (ecase major-version
     (2 'id3v2.2-tag)
     (3 'id3v2.3-tag))))

Version 2.2 and version 2.3 tags differ in two ways. First, the header of a version 2.3 tag may be extended with up to four optional extended header fields, as determined by values in the flags field. Second, the frame format changed between version 2.2 and version 2.3, which means you’ll have to use different classes to represent version 2.2 frames and the corresponding version 2.3 frames.

Since the new id3-tag class is based on the one you originally wrote to represent version 2.2 tags, it’s not surprising that the new id3v2.2-tag class is trivial, inheriting most of its slots from the new id3-tag class and adding the one missing slot, frames. Because version 2.2 and version 2.3 tags use different frame formats, you’ll have to change the id3-frames type to be parameterized with the type of frame to read. For now, assume you’ll do that and add a :frame-type argument to the id3-frames type descriptor like this:

(define-binary-class id3v2.2-tag (id3-tag)
  ((frames (id3-frames :tag-size size :frame-type 'id3v2.2-frame))))

The id3v2.3-tag class is slightly more complex because of the optional fields. The first three of the four optional fields are included when the sixth bit in flags is set. They’re a four-byte integer specifying the size of the extended header, two bytes worth of flags, and another four-byte integer specifying how many bytes of padding are included in the tag.[7] The fourth optional field, included when the fifteenth bit of the extended header flags is set, is a four-byte cyclic redundancy check (CRC) of the rest of the tag.

The binary data library doesn’t provide any special support for optional fields in a binary class, but it turns out that regular parameterized binary types are sufficient. You can define a type parameterized with the name of a type and a value that indicates whether a value of that type should actually be read or written.

(define-binary-type optional (type if)
  (:reader (in)
    (when if (read-value type in)))
  (:writer (out value)
    (when if (write-value type out value))))

Using if as the parameter name looks a bit strange in that code, but it makes the optional type descriptors quite readable. For instance, here’s the definition of id3v2.3-tag using optional slots:

(define-binary-class id3v2.3-tag (id3-tag)
  ((extended-header-size (optional :type 'u4 :if (extended-p flags)))
   (extra-flags          (optional :type 'u2 :if (extended-p flags)))
   (padding-size         (optional :type 'u4 :if (extended-p flags)))
   (crc                  (optional :type 'u4 :if (crc-p flags extra-flags)))
   (frames               (id3-frames :tag-size size :frame-type 'id3v2.3-frame))))

where extended-p and crc-p are helper functions that test the appropriate bit of the flags value they’re passed. To test whether an individual bit of an integer is set, you can use LOGBITP, another bit-twiddling function. It takes an index and an integer and returns true if the specified bit is set in the integer.

(defun extended-p (flags) (logbitp 6 flags))

(defun crc-p (flags extra-flags)
  (and (extended-p flags) (logbitp 15 extra-flags)))

As in the version 2.2 tag class, the frames slot is defined to be of type id3-frames, passing the name of the frame type as a parameter. You do, however, need to make a few small changes to id3-frames and read-frame to support the extra frame-type parameter.

(define-binary-type id3-frames (tag-size frame-type)
  (:reader (in)
    (loop with to-read = tag-size
          while (plusp to-read)
          for frame = (read-frame frame-type in)
          while frame
          do (decf to-read (+ (frame-header-size frame) (size frame)))
          collect frame
          finally (loop repeat (1- to-read) do (read-byte in))))
  (:writer (out frames)
    (loop with to-write = tag-size
          for frame in frames
          do (write-value frame-type out frame)
          (decf to-write (+ (frame-header-size frame) (size frame)))
          finally (loop repeat to-write do (write-byte 0 out)))))
(defun read-frame (frame-type in)
  (handler-case (read-value frame-type in)
    (in-padding () nil)))

The changes are in the calls to read-frame and write-value, where you need to pass the frame-type argument and, in computing the size of the frame, where you need to use a function frame-header-size instead of the literal value 6 since the frame header changed size between version 2.2 and version 2.3. Since the difference in the result of this function is based on the class of the frame, it makes sense to define it as a generic function like this:

(defgeneric frame-header-size (frame))

You’ll define the necessary methods on that generic function in the next section after you define the new frame classes.

[7]If a tag had an extended header, you could use this value to determine where the frame data should end. However, if the extended header isn’t used, you’d have to use the old algorithm anyway, so it’s not worth adding code to do it another way.


992 - Versioned Concrete Frame Classes

Chapter 25 - Practical—An ID3 Parser
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Versioned Concrete Frame Classes

In the original definition, generic-frame subclassed id3-frame. But now id3-frame has been replaced with the two version-specific base classes, id3v2.2-frame and id3v2.3-frame. So, you need to define two new versions of generic-frame, one for each base class. One way to define this classes would be like this:

(define-binary-class generic-frame-v2.2 (id3v2.2-frame)
  ((data (raw-bytes :size size))))

(define-binary-class generic-frame-v2.3 (id3v2.3-frame)
  ((data (raw-bytes :size size))))

However, it’s a bit annoying that these two classes are the same except for their superclass. It’s not too bad in this case since there’s only one additional field. But if you take this approach for other concrete frame classes, ones that have a more complex internal structure that’s identical between the two ID3 versions, the duplication will be more irksome.

Another approach, and the one you should actually use, is to define a class generic-frame as a mixin: a class intended to be used as a superclass along with one of the version-specific base classes to produce a concrete, version-specific frame class. The only tricky bit about this approach is that if generic-frame doesn’t extend either of the frame base classes, then you can’t refer to the size slot in its definition. Instead, you must use the current-binary-object function I discussed at the end of the previous chapter to access the object you’re in the midst of reading or writing and pass it to size. And you need to account for the difference in the number of bytes of the total frame size that will be left over, in the case of a version 2.3 frame, if any of the optional fields are included in the frame. So, you should define a generic function data-bytes with methods that do the right thing for both version 2.2 and version 2.3 frames.

(define-binary-class generic-frame ()
  ((data (raw-bytes :size (data-bytes (current-binary-object))))))

(defgeneric data-bytes (frame))

(defmethod data-bytes ((frame id3v2.2-frame))
  (size frame))

(defmethod data-bytes ((frame id3v2.3-frame))
  (let ((flags (flags frame)))
    (- (size frame)
       (if (frame-compressed-p flags) 4 0)
       (if (frame-encrypted-p flags) 1 0)
       (if (frame-grouped-p flags) 1 0))))

Then you can define concrete classes that extend one of the version-specific base classes and generic-frame to define version-specific generic frame classes.

(define-binary-class generic-frame-v2.2 (id3v2.2-frame generic-frame) ())

(define-binary-class generic-frame-v2.3 (id3v2.3-frame generic-frame) ())

With these classes defined, you can redefine the find-frame-class function to return the right versioned class based on the length of the identifier.

(defun find-frame-class (id)
  (ecase (length id)
    (3 'generic-frame-v2.2)
    (4 'generic-frame-v2.3)))


994 - Text Information Frames

Chapter 25 - Practical—An ID3 Parser
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Text Information Frames

All text information frames consist of two fields: a single byte indicating which string encoding is used in the frame and a string encoded in the remaining bytes of the frame. If the encoding byte is zero, the string is encoded in ISO 8859-1; if the encoding is one, the string is a UCS-2 string.

You’ve already defined binary types representing the four different kinds of strings—two different encodings each with two different methods of delimiting the string. However, define-binary-class provides no direct facility for determining the type of value to read based on other values in the object. Instead, you can define a binary type that you pass the value of the encoding byte and that then reads or writes the appropriate kind of string.

As long as you’re defining such a type, you can also define it to take two parameters, :length and :terminator, and pick the right type of string based on which argument is supplied. To implement this new type, you must first define some helper functions. The first two return the name of the appropriate string type based on the encoding byte.

(defun non-terminated-type (encoding)
  (ecase encoding
    (0 'iso-8859-1-string)
    (1 'ucs-2-string)))

(defun terminated-type (encoding)
  (ecase encoding
    (0 'iso-8859-1-terminated-string)
    (1 'ucs-2-terminated-string)))

Then string-args uses the encoding byte, the length, and the terminator to determine several of the arguments to be passed to read-value and write-value by the :reader and :writer of id3-encoded-string. One of the length and terminator arguments to string-args should always be NIL.

(defun string-args (encoding length terminator)
  (cond
    (length
     (values (non-terminated-type encoding) :length length))
    (terminator
     (values (terminated-type encoding) :terminator terminator))))

With those helpers, the definition of id3-encoded-string is simple. One detail to note is that the keyword—either :length or :terminator—used in the call to read-value and write-value is just another piece of data returned by string-args. Although keywords in arguments lists are almost always literal keywords, they don’t have to be.

(define-binary-type id3-encoded-string (encoding length terminator)
  (:reader (in)
    (multiple-value-bind (type keyword arg)
        (string-args encoding length terminator)
      (read-value type in keyword arg)))
  (:writer (out string)
    (multiple-value-bind (type keyword arg)
        (string-args encoding length terminator)
      (write-value type out string keyword arg))))

Now you can define a text-info mixin class, much the way you defined generic-frame earlier.

(define-binary-class text-info-frame ()
  ((encoding u1)
   (information (id3-encoded-string :encoding encoding :length (bytes-left 1)))))

As when you defined generic-frame, you need access to the size of the frame, in this case to compute the :length argument to pass to id3-encoded-string. Because you’ll need to do a similar computation in the next class you define, you can go ahead and define a helper function, bytes-left, that uses current-binary-object to get at the size of the frame.

(defun bytes-left (bytes-read)
  (- (size (current-binary-object)) bytes-read))

Now, as you did with the generic-frame mixin, you can define two version-specific concrete classes with a minimum of duplicated code.

(define-binary-class text-info-frame-v2.2 (id3v2.2-frame text-info-frame) ())

(define-binary-class text-info-frame-v2.3 (id3v2.3-frame text-info-frame) ())

To wire these classes in, you need to modify find-frame-class to return the appropriate class name when the ID indicates the frame is a text information frame, namely, whenever the ID starts with T and isn’t TXX or TXXX.

(defun find-frame-class (name)
  (cond
    ((and (char= (char name 0) #\T)
          (not (member name '("TXX" "TXXX") :test #'string=)))
     (ecase (length name)
       (3 'text-info-frame-v2.2)
       (4 'text-info-frame-v2.3)))
    (t
     (ecase (length name)
       (3 'generic-frame-v2.2)
       (4 'generic-frame-v2.3)))))


996 - Extracting Information from an ID3 Tag

 Skip Navigation
Chapter 25 - Practical—An ID3 Parser
Practical Common Lisp
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Extracting Information from an ID3 Tag

Now that you have the basic ability to read and write ID3 tags, you have a lot of directions you could take this code. If you want to develop a complete ID3 tag editor, you’ll need to implement specific classes for all the frame types. You’d also need to define methods for manipulating the tag and frame objects in a consistent way (for instance, if you change the value of a string in a text-info-frame, you’ll likely need to adjust the size); as the code stands, there’s nothing to make sure that happens.[9]

Or, if you just need to extract certain pieces of information about an MP3 file from its ID3 tag—as you will when you develop a streaming MP3 server in Chapters 27, 28, and 29—you’ll need to write functions that find the appropriate frames and extract the information you want.

Finally, to make this production-quality code, you’d have to pore over the ID3 specs and deal with the details I skipped over in the interest of space. In particular, some of the flags in both the tag and the frame can affect the way the contents of the tag or frame is read; unless you write some code that does the right thing when those flags are set, there may be ID3 tags that this code won’t be able to parse correctly. But the code from this chapter should be capable of parsing nearly all the MP3s you actually encounter.

For now you can finish with a few functions to extract individual pieces of information from an id3-tag. You’ll need these functions in Chapter 27 and probably in other code that uses this library. They belong in this library because they depend on details of the ID3 format that the users of this library shouldn’t have to worry about.

To get, say, the name of the song of the MP3 from which an id3-tag was extracted, you need to find the ID3 frame with a specific identifier and then extract the information field. And some pieces of information, such as the genre, can require further decoding. Luckily, all the frames that contain the information you’ll care about are text information frames, so extracting a particular piece of information mostly boils down to using the right identifier to look up the appropriate frame. Of course, the ID3 authors decided to change all the identifiers between ID3v2.2 and ID3v2.3, so you’ll have to account for that.

Nothing too complex—you just need to figure out the right path to get to the various pieces of information. This is a perfect bit of code to develop interactively, much the way you figured out what frame classes you needed to implement. To start, you need an id3-tag object to play with. Assuming you have an MP3 laying around, you can use read-id3 like this:

ID3V2> (defparameter *id3* (read-id3 "Kitka/Wintersongs/02 Byla Cesta.mp3"))
*ID3*
ID3V2> *id3*
#<ID3V2.2-TAG @ #x73d04c1a>

replacing Kitka/Wintersongs/02 Byla Cesta.mp3 with the filename of your MP3. Once you have your id3-tag object, you can start poking around. For instance, you can check out the list of frame objects with the frames function.

ID3V2> (frames *id3*)
(#<TEXT-INFO-FRAME-V2.2 @ #x73d04cca>
 #<TEXT-INFO-FRAME-V2.2 @ #x73d04dba>
 #<TEXT-INFO-FRAME-V2.2 @ #x73d04ea2>
 #<TEXT-INFO-FRAME-V2.2 @ #x73d04f9a>
 #<TEXT-INFO-FRAME-V2.2 @ #x73d05082>
 #<TEXT-INFO-FRAME-V2.2 @ #x73d0516a>
 #<TEXT-INFO-FRAME-V2.2 @ #x73d05252>
 #<TEXT-INFO-FRAME-V2.2 @ #x73d0533a>
 #<COMMENT-FRAME-V2.2 @ #x73d0543a>
 #<COMMENT-FRAME-V2.2 @ #x73d05612>
 #<COMMENT-FRAME-V2.2 @ #x73d0586a>)

Now suppose you want to extract the song title. It’s probably in one of those frames, but to find it, you need to find the frame with the “TT2” identifier. Well, you can check easily enough to see if the tag contains such a frame by extracting all the identifiers like this:

ID3V2> (mapcar #'id (frames *id3*))
("TT2" "TP1" "TAL" "TRK" "TPA" "TYE" "TCO" "TEN" "COM" "COM" "COM")

There it is, the first frame. However, there’s no guarantee it’ll always be the first frame, so you should probably look it up by identifier rather than position. That’s also straightforward using the FIND function.

ID3V2> (find "TT2" (frames *id3*) :test #'string= :key #'id)
#<TEXT-INFO-FRAME-V2.2 @ #x73d04cca>

Now, to get at the actual information in the frame, do this:

ID3V2> (information (find "TT2" (frames *id3*) :test #'string= :key #'id))
"Byla Cesta^@"

Whoops. That ^@ is how Emacs prints a null character. In a maneuver reminiscent of the kludge that turned ID3v1 into ID3v1.1, the information slot of a text information frame, though not officially a null-terminated string, can contain a null, and ID3 readers are supposed to ignore any characters after the null. So, you need a function that takes a string and returns the contents up to the first null character, if any. That’s easy enough using the +null+ constant from the binary data library.

(defun upto-null (string)
  (subseq string 0 (position +null+ string)))

Now you can get just the title.

ID3V2> (upto-null
        (information (find "TT2" (frames *id3*) :test #'string= :key #'id)))
"Byla Cesta"

You could just wrap that code in a function named song that takes an id3-tag as an argument, and you’d be done. However, the only difference between this code and the code you’ll use to extract the other pieces of information you’ll need (such as the album name, the artist, and the genre) is the identifier. So, it’s better to split up the code a bit. For starters, you can write a function that just finds a frame given an id3-tag and an identifier like this:

(defun find-frame (id3 id)
  (find id (frames id3) :test #'string= :key #'id))

ID3V2> (find-frame *id3* "TT2")
#<TEXT-INFO-FRAME-V2.2 @ #x73d04cca>

Then the other bit of code, the part that extracts the information from a text-info-frame, can go in another function.

(defun get-text-info (id3 id)
  (let ((frame (find-frame id3 id)))
    (when frame (upto-null (information frame)))))

ID3V2> (get-text-info *id3* "TT2")
"Byla Cesta"

Now the definition of song is just a matter of passing the right identifier.

(defun song (id3) (get-text-info id3 "TT2"))

ID3V2> (song *id3*)
"Byla Cesta"

However, this definition of song works only with version 2.2 tags since the identifier changed from “TT2” to “TIT2” between version 2.2 and version 2.3. And all the other tags changed too. Since the user of this library shouldn’t have to know about different versions of the ID3 format to do something as simple as get the song title, you should probably handle those details for them. A simple way is to change find-frame to take not just a single identifier but a list of identifiers like this:

(defun find-frame (id3 ids)
  (find-if #'(lambda (x) (find (id x) ids :test #'string=)) (frames id3)))

Then change get-text-info slightly so it can take one or more identifiers using a &rest parameter.

(defun get-text-info (id3 &rest ids)
  (let ((frame (find-frame id3 ids)))
    (when frame (upto-null (information frame)))))

Then the change needed to allow song to support both version 2.2 and version 2.3 tags is just a matter of adding the version 2.3 identifier.

(defun song (id3) (get-text-info id3 "TT2" "TIT2"))

Then you just need to look up the appropriate version 2.2 and version 2.3 frame identifiers for any fields for which you want to provide an accessor function. Here are the ones you’ll need in Chapter 27:

(defun album (id3) (get-text-info id3 "TAL" "TALB"))

(defun artist (id3) (get-text-info id3 "TP1" "TPE1"))

(defun track (id3) (get-text-info id3 "TRK" "TRCK"))

(defun year (id3) (get-text-info id3 "TYE" "TYER" "TDRC"))

(defun genre (id3) (get-text-info id3 "TCO" "TCON"))

The last wrinkle is that the way the genre is stored in the TCO or TCON frames isn’t always human readable. Recall that in ID3v1, genres were stored as a single byte that encoded a particular genre from a fixed list. Unfortunately, those codes live on in ID3v2—if the text of the genre frame is a number in parentheses, the number is supposed to be interpreted as an ID3v1 genre code. But, again, users of this library probably won’t care about that ancient history. So, you should provide a function that automatically translates the genre. The following function uses the genre function just defined to extract the actual genre text and then checks whether it starts with a left parenthesis, decoding the version 1 genre code with a function you’ll define in a moment if it does:

(defun translated-genre (id3)
  (let ((genre (genre id3)))
    (if (and genre (char= #\( (char genre 0)))
      (translate-v1-genre genre)
      genre)))

Since a version 1 genre code is effectively just an index into an array of standard names, the easiest way to implement translate-v1-genre is to extract the number from the genre string and use it as an index into an actual array.

(defun translate-v1-genre (genre)
  (aref *id3-v1-genres* (parse-integer genre :start 1 :junk-allowed t)))

Then all you need to do is to define the array of names. The following array of names includes the 80 official version 1 genres plus the genres created by the authors of Winamp:

(defparameter *id3-v1-genres*
  #(
    ;; These are the official ID3v1 genres.
    "Blues" "Classic Rock" "Country" "Dance" "Disco" "Funk" "Grunge"
    "Hip-Hop" "Jazz" "Metal" "New Age" "Oldies" "Other" "Pop" "R&B" "Rap"
    "Reggae" "Rock" "Techno" "Industrial" "Alternative" "Ska"
    "Death Metal" "Pranks" "Soundtrack" "Euro-Techno" "Ambient"
    "Trip-Hop" "Vocal" "Jazz+Funk" "Fusion" "Trance" "Classical"
    "Instrumental" "Acid" "House" "Game" "Sound Clip" "Gospel" "Noise"
    "AlternRock" "Bass" "Soul" "Punk" "Space" "Meditative"
    "Instrumental Pop" "Instrumental Rock" "Ethnic" "Gothic" "Darkwave"
    "Techno-Industrial" "Electronic" "Pop-Folk" "Eurodance" "Dream"
    "Southern Rock" "Comedy" "Cult" "Gangsta" "Top 40" "Christian Rap"
    "Pop/Funk" "Jungle" "Native American" "Cabaret" "New Wave"
    "Psychedelic" "Rave" "Showtunes" "Trailer" "Lo-Fi" "Tribal"
    "Acid Punk" "Acid Jazz" "Polka" "Retro" "Musical" "Rock & Roll"
    "Hard Rock"

    ;; These were made up by the authors of Winamp but backported into
    ;; the ID3 spec.
    "Folk" "Folk-Rock" "National Folk" "Swing" "Fast Fusion"
    "Bebob" "Latin" "Revival" "Celtic" "Bluegrass" "Avantgarde"
    "Gothic Rock" "Progressive Rock" "Psychedelic Rock" "Symphonic Rock"
    "Slow Rock" "Big Band" "Chorus" "Easy Listening" "Acoustic" "Humor"
    "Speech" "Chanson" "Opera" "Chamber Music" "Sonata" "Symphony"
    "Booty Bass" "Primus" "Porn Groove" "Satire" "Slow Jam" "Club"
    "Tango" "Samba" "Folklore" "Ballad" "Power Ballad" "Rhythmic Soul"
    "Freestyle" "Duet" "Punk Rock" "Drum Solo" "A capella" "Euro-House"
    "Dance Hall"

    ;; These were also invented by the Winamp folks but ignored by the
    ;; ID3 authors.
    "Goa" "Drum & Bass" "Club-House" "Hardcore" "Terror" "Indie"
    "BritPop" "Negerpunk" "Polsk Punk" "Beat" "Christian Gangsta Rap"
    "Heavy Metal" "Black Metal" "Crossover" "Contemporary Christian"
    "Christian Rock" "Merengue" "Salsa" "Thrash Metal" "Anime" "Jpop"
    "Synthpop"))

Once again, it probably feels like you wrote a ton of code in this chapter. But if you put it all in a file, or if you download the version from this book’s Web site, you’ll see it’s just not that many lines—most of the pain of writing this library stems from having to understand the intricacies of the ID3 format itself. Anyway, now you have a major piece of what you’ll turn into a streaming MP3 server in Chapters 27, 28, and 29. The other major bit of infrastructure you’ll need is a way to write server-side Web software, the topic of the next chapter.

[9]Ensuring that kind of interfield consistency would be a fine application for :after methods on the accessor generic functions. For instance, you could define this :after method to keep size in sync with the information string:

(defmethod (setf information) :after (value (frame text-info-frame))
  (declare (ignore value))
  (with-slots (encoding size information) frame
    (setf size (encoded-string-length information encoding nil))))

Previous Section
Progress Indicator Progress Indicator
Next Section
x7 and Referenceware are registered trademarks of Books24x7, Inc.
Copyright © 1999-2005 Books24x7, Inc. - Feedback | Privacy Policy (updated 03/2005)


998 - AllegroServe

Chapter 26 - Practical—Web Programming with AllegroServe
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

AllegroServe

You can serve Web content using Common Lisp in a number of ways; there are at least three open-source Web servers written in Common Lisp as well as plug-ins such as mod_lisp [5] and Lisplets [6] that allow the Apache Web server or any Java Servlet container to delegate requests to a Lisp server running in a separate process.

For this chapter, you’ll use a version of the open-source Web server AllegroServe, originally written by John Foderaro at Franz Inc.. AllegroServe is included in the version of Allegro available from Franz for use with this book. If you’re not using Allegro, you can use PortableAllegroServe, a friendly fork of the AllegroServe code base, which includes an Allegro compatibility layer that allows PortableAllegroServe to run on most Common Lisps. The code you’ll write in this chapter and in Chapter 29 should run in both vanilla AllegroServe and PortableAllegroServe.

AllegroServe provides a programming model similar in spirit to Java Servlets—each time a browser requests a page, AllegroServe parses the request and looks up an object, called an entity, which handles the request. Some entity classes provided as part of AllegroServe know how to serve static content—either individual files or the contents of a directory tree. Others, the ones I’ll spend most of this chapter discussing, run arbitrary Lisp code to generate the response. [7]

But before I get to that, you need to know how to start AllegroServe and set it up to serve a few files. The first step is to load the AllegroServe code into your Lisp image. In Allegro, you can simply type (require :aserve). In other Lisps (or in Allegro), you can load PortableAllegroServe by loading the file INSTALL.lisp at the top of the portableaserve directory tree. Loading AllegroServe will create three new packages, NET.ASERVE, NET.HTML.GENERATOR, and NET.ASERVE.CLIENT. [8]

After loading the server, you start it with the function start in the NET.ASERVE package. To have easy access to the symbols exported from NET.ASERVE, from COM.GIGAMONKEYS.HTML (a package I’ll discuss in a moment), and from the rest of Common Lisp, you should create a new package to play in like this:

CL-USER> (defpackage :com.gigamonkeys.web
            (:use :cl :net.aserve :com.gigamonkeys.html))
#<The COM.GIGAMONKEYS.WEB package>

Now switch to that package with this IN-PACKAGE expression:

CL-USER> (in-package :com.gigamonkeys.web)
#<The COM.GIGAMONKEYS.WEB package>
WEB>

Now you can use the exported names from NET.ASERVE without qualification. The function start starts the server. It takes quite a number of keyword parameters, but the only one you need to pass is :port, which specifies the port to listen on. You should probably use a high port such as 2001 instead of the default port for HTTP servers, 80, because on Unix-derived operating systems only the root user can listen on ports below 1024. To run AllegroServe listening on port 80 on Unix, you’d need to start Lisp as root and then use the :setuid and :setgid parameters to tell start to switch its identity after opening the port. You can start a server listening on port 2001 like this:

WEB> (start :port 2001)
#<WSERVER port 2001 @ #x72511c72>

The server is now running in your Lisp. It’s possible you’ll get an error that says something about “port already in use” when you try to start the server. This means port 2001 is already in use by some other server on your machine. In that case, the simplest fix is to use a different port, supplying a different argument to start and then using that value instead of 2001 in the URLs used throughout this chapter.

You can continue to interact with Lisp via the REPL because AllegroServe starts its own threads to handle requests from browsers. This means, among other things, that you can use the REPL to get a view into the guts of your server while it’s running, which makes debugging and testing a lot easier than if the server is a complete black box.

Assuming you’re running Lisp on the same machine as your browser, you can check that the server is up and running by pointing your browser at http://localhost:2001/. At this point you should get a page-not-found error message in the browser since you haven’t published anything yet. But the error message will be from AllegroServe; it’ll say so at the bottom of the page. On the other hand, if the browser displays an error dialog that says something like “The connection was refused when attempting to contact localhost:2001,” it means either that the server isn’t running or that you started it with a different port than 2001.

Now you can publish some files. Suppose you have a file hello.html in the directory /tmp/html with the following contents:

<html>
  <head>
  <title>Hello</title>
  </head>
  <body>
  <p>Hello, world!</p>
  </body>
</html>

You can publish it individually with the publish-file function.

WEB> (publish-file :path "/hello.html" :file "/tmp/html/hello.html")
#<NET.ASERVE::FILE-ENTITY @ #x725eddea>

The :path argument is the path that will appear in the URL requested by the browser, while the :file argument is the name of the file in the file system. After evaluating the publish-file expression, you can point your browser to http://localhost:2001/hello.html, and it should display a page something like Figure 26-2.

Image from book
Figure 26-2: http://localhost:2001/hello.html

You could also publish a whole directory tree of files using the publish-directory function. First let’s clear out the already published entity with the following call to publish-file:

WEB> (publish-file :path "/hello.html" :remove t)
NIL

Now you can publish the whole /tmp/html/ directory (and all its subdirectories) with the publish-directory function.

WEB> (publish-directory :prefix "/" :destination "/tmp/html/")
#<NET.ASERVE::DIRECTORY-ENTITY @ #x72625aa2>

In this case, the :prefix argument specifies the beginning of the path part of URLs that should be handled by this entity. Thus, if the server receives a request for http:// localhost:2001/foo/bar.html, the path is /foo/bar.html, which starts with /. This path is then translated to a filename by replacing the prefix, /, with the destination, /tmp/html/. Thus, the URL http://localhost:2001/hello.html will still be translated into a request for the file /tmp/ html/hello.html.

[5] http://www.fractalconcept.com/asp/html/mod_lisp.html

[6] http://lisplets.sourceforge.net/

[7]AllegroServe also provides a framework called Webactions that’s analogous to JSPs in the Java world—instead of writing code that generates HTML, with Webactions you write pages that are essentially HTML with a bit of magic foo that turns into code to be run when the page is served. I won’t cover Webactions in this book.

[8]Loading PortableAllegroServe will create some other packages for the compatibility libraries, but the packages you’ll care about are those three.


002 - Query Parameters

Chapter 26 - Practical—Web Programming with AllegroServe
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Query Parameters

Of course, generating HTML output is only half of Web programming. The other thing you need to do is get input from the user. As I discussed in the “A 30-Second Intro to Server-Side Web Programming” section, when a browser requests a page from a Web server, it can send query parameters in the URL and post data, both of which act as input to the server-side code.

AllegroServe, like most Web programming frameworks, takes care of parsing both these sources of input for you. By the time your published functions are called, all the key/value pairs from the query string and/or post data have been decoded and placed into an alist that you can retrieve from the request object with the function request-query. The following function returns a page showing all the query parameters it receives:

(defun show-query-params (request entity)
  (with-http-response (request entity :content-type "text/html")
    (with-http-body (request entity)
      (with-html-output ((request-reply-stream request))
        (html
          (:standard-page
           (:title "Query Parameters")
           (if (request-query request)
             (html
               (:table :border 1
                       (loop for (k . v) in (request-query request)
                          do (html (:tr (:td k) (:td v))))))
             (html (:p "No query parameters.")))))))))

(publish :path "/show-query-params" :function 'show-query-params)

If you give your browser a URL with a query string in it like the following:

http://localhost:2001/show-query-params?foo=bar&baz=10

you should get back a page similar to the one shown in Figure 26-4.

Image from book
Figure 26-4: http://localhost:2001/show-query-params?foo=bar&baz=10

To generate some post data, you need an HTML form. The following function generates a simple form, which submits its data to show-query-params:

(defun simple-form (request entity)
  (with-http-response (request entity :content-type "text/html")
    (with-http-body (request entity)
      (let ((*html-output* (request-reply-stream request)))
        (html
          (:html
            (:head (:title "Simple Form"))
            (:body
             (:form :method "POST" :action "/show-query-params"
               (:table
                (:tr (:td "Foo")
                     (:td (:input :name "foo" :size 20)))
                (:tr (:td "Password")
                     (:td (:input :name "password" :type "password" :size 20))))
               (:p (:input :name "submit" :type "submit" :value "Okay")
                   (:input ::type "reset" :value "Reset"))))))))))

(publish :path "/simple-form" :function 'simple-form)

Point your browser to http://localhost:2001/simple-form, and you should see a page like the one in Figure 26-5.

Image from book
Figure 26-5: http://localhost:2001/simple-form

If you fill in the form with the “abc” and “def” values, clicking the Okay button should take you to a page like the one in Figure 26-6.

Image from book
Figure 26-6: Result of submitting the simple form

However, most of the time you won’t need to iterate over all the query parameters; you’ll want to pick out individual parameters. For instance, you might want to modify random-number so the limit value you pass to RANDOM can be supplied via a query parameter. In that case, you use the function request-query-value, which takes the request object and the name of the parameter whose value you want and returns the value as a string or NIL if no such parameter has been supplied. A parameterizable version of random-number might look like this:

(defun random-number (request entity)
  (with-http-response (request entity :content-type "text/html")
    (with-http-body (request entity)
      (let* ((*html-output* (request-reply-stream request))
             (limit-string (or (request-query-value "limit" request) ""))
             (limit (or (parse-integer limit-string :junk-allowed t) 1000)))
        (html
          (:html
            (:head (:title "Random"))
            (:body
              (:p "Random number: " (:print (random limit))))))))))

Because request-query-value can return either NIL or an empty string, you have to deal with both those cases when parsing the parameter into a number to pass to RANDOM. You can deal with a NIL value when you bind limit-string, binding it to "" if there’s no “limit” query parameter. Then you can use the :junk-allowed argument to PARSE-INTEGER to ensure that it returns either NIL (if it can’t parse an integer from the string given) or an integer. In the section “A Small Application Framework,” you’ll develop some macros to make it easier to deal with grabbing query parameters and converting them to various types.


003 - Cookies

Chapter 26 - Practical—Web Programming with AllegroServe
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Cookies

In AllegroServe you can send a Set-Cookie header that tells the browser to save a cookie and send it along with subsequent requests by calling the function set-cookie-header within the body of with-http-response but before the call to with-http-body. The first argument to the function is the request object, and the remaining arguments are keyword arguments used to set the various properties of the cookie. The only two you must pass are the :name and :value arguments, both of which should be strings. The other possible arguments that affect the cookie sent to the browser are :expires, :path, :domain, and :secure.

Of these, you need to worry only about :expires. It controls how long the browser should save the cookie. If :expires is NIL (the default), the browser will save the cookie only until it exits. Other possible values are :never, which means the cookie should be kept forever, or a universal time as returned by GET-UNIVERSAL-TIME or ENCODE-UNIVERSAL-TIME. An :expires of zero tells the client to immediately discard an existing cookie. [11]

After you’ve set a cookie, you can use the function get-cookie-values to get an alist containing one name/value pair for each cookie sent by the browser. From that alist, you can pick out individual cookie values using ASSOC and CDR.

The following function shows the names and values of all the cookies sent by the browser:

(defun show-cookies (request entity)
  (with-http-response (request entity :content-type "text/html")
    (with-http-body (request entity)
      (with-html-output ((request-reply-stream request))
        (html
          (:standard-page
           (:title "Cookies")
           (if (null (get-cookie-values request))
             (html (:p "No cookies."))
             (html
               (:table
                 (loop for (key . value) in (get-cookie-values request)
                    do (html (:tr (:td key) (:td value)))))))))))))

(publish :path "/show-cookies" :function 'show-cookies)

The first time you load the page http://localhost:2001/show-cookies it should say “No cookies” as shown in Figure 26-7 since you haven’t set any yet.

Image from book
Figure 26-7: http://localhost:2001/show-cookies with no cookies

To set a cookie, you need another function, such as the following:

(defun set-cookie (request entity)
  (with-http-response (request entity :content-type "text/html")
    (set-cookie-header request :name "MyCookie" :value "A cookie value")
    (with-http-body (request entity)
      (with-html-output ((request-reply-stream request))
        (html
          (:standard-page
           (:title "Set Cookie")
           (:p "Cookie set.")
           (:p (:a :href "/show-cookies" "Look at cookie jar."))))))))

(publish :path "/set-cookie" :function 'set-cookie)

If you enter the URL http://localhost:2001/set-cookie, your browser should display a page like the one in Figure 26-8. Additionally, the server will send a Set-Cookie header with a cookie named “MyCookie” with “A cookie value” as its value. If you click the link Look at cookie jar, you’ll be taken to the /show-cookies page where you’ll see the new cookie, as shown in Figure 26-9. Because you didn’t specify an :expires argument, the browser will continue to send the cookie with each request until you quit the browser.

Image from book
Figure 26-8: http://localhost:2001/set-cookie
Image from book
Figure 26-9: http://localhost:2001/show-cookies after setting a cookie

[11]For information about the meaning of the other parameters, see the AllegroServe documentation and RFC 2109, which describes the cookie mechanism.


005 - The Implementation

Chapter 26 - Practical—Web Programming with AllegroServe
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

The Implementation

I’ll explain the implementation of define-url-function from the top down. The macro itself looks like this:

(defmacro define-url-function (name (request &rest params) &body body)
  (with-gensyms (entity)
    (let ((params (mapcar #'normalize-param params)))
      `(progn
         (defun ,name (,request ,entity)
           (with-http-response (,request ,entity :content-type "text/html")
             (let* (,@(param-bindings name request params))
               ,@(set-cookies-code name request params)
               (with-http-body (,request ,entity)
                 (with-html-output ((request-reply-stream ,request))
                   (html ,@body))))))
         (publish :path ,(format nil "/~(~a~)" name) :function ',name)))))

Let’s take it bit by bit, starting with the first few lines.

(defmacro define-url-function (name (request &rest params) &body body)
  (with-gensyms (entity)
    (let ((params (mapcar #'normalize-param params)))

Up to here you’re just getting ready to generate code. You GENSYM a symbol to use later as the name of the entity parameter in the DEFUN. Then you normalize the parameters, converting plain symbols to list form using this function:

(defun normalize-param (param)
  (etypecase param
    (list param)
    (symbol `(,param string nil nil))))

In other words, declaring a parameter with just a symbol is the same as declaring a nonsticky, string parameter with no default value.

Then comes the PROGN. You must expand into a PROGN because you need to generate code to do two things: define a function with DEFUN and call publish. You should define the function first so if there’s an error in the definition, the function won’t be published. The first two lines of the DEFUN are just boilerplate.

(defun ,name (,request ,entity)
  (with-http-response (,request ,entity :content-type "text/html")

Now you do the real work. The following two lines generate the bindings for the parameters specified in define-url-function other than request and the code that calls set-cookie-header for the sticky parameters. Of course, the real work is done by helper functions that you’ll look at in a moment. [12]

(let* (,@(param-bindings name request params))
  ,@(set-cookies-code name request params)

The rest is just more boilerplate, putting the body from the define-url-function definition in the appropriate context of with-http-body, with-html-output, and html macros. Then comes the call to publish.

(publish :path ,(format nil "/~(~a~)" name) :function ',name)

The expression (format nil "/~(~a~)" name) is evaluated at macro expansion time, generating a string consisting of /, followed by an all-lowercase version of the name of the function you’re about to define. That string becomes the :path argument to publish, while the function name is interpolated as the :function argument.

Now let’s look at the helper functions used to generate the DEFUN form. To generate parameter bindings, you need to loop over the params and collect a snippet of code for each one, generated by param-binding. That snippet will be a list containing the name of the variable to bind and the code that will compute the value of that variable. The exact form of code used to compute the value will depend on the type of the parameter, whether it’s sticky, and the default value, if any. Because you already normalized the params, you can use DESTRUCTURING-BIND to take them apart in param-binding.

(defun param-bindings (function-name request params)
  (loop for param in params
     collect (param-binding function-name request param)))

(defun param-binding (function-name request param)
  (destructuring-bind (name type &optional default sticky) param
    (let ((query-name (symbol->query-name name))
          (cookie-name (symbol->cookie-name function-name name sticky)))
      `(,name (or
               (string->type ',type (request-query-value ,query-name ,request))
               ,@(if cookie-name
                     (list `(string->type ',type
                             (get-cookie-value ,request ,cookie-name))))
               ,default)))))

The function string->type, which you use to convert strings obtained from the query parameters and cookies to the desired type, is a generic function with the following signature:

(defgeneric string->type (type value))

To make a particular name usable as a type name for a query parameter, you just need to define a method on string->type. You’ll need to define at least a method specialized on the symbol string since that’s the default type. Of course, that’s pretty easy. Since browsers sometimes submit forms with empty strings to indicate no value was supplied for a particular value, you’ll want to convert an empty string to NIL as this method does:

(defmethod string->type ((type (eql 'string)) value)
  (and (plusp (length value)) value))

You can add conversions for other types needed by your application. For instance, to make integer usable as a query parameter type so you can handle the limit parameter of random-page, you might define this method:

(defmethod string->type ((type (eql 'integer)) value)
  (parse-integer (or value "") :junk-allowed t))

Another helper function used in the code generated by param-binding is get-cookie-value, which is just a bit of sugar around the get-cookie-values function provided by AllegroServe. It looks like this:

(defun get-cookie-value (request name)
  (cdr (assoc name (get-cookie-values request) :test #'string=)))

The functions that compute the query parameter and cookies names are similarly straightforward.

(defun symbol->query-name (sym)
  (string-downcase sym))

(defun symbol->cookie-name (function-name sym sticky)
  (let ((package-name (package-name (symbol-package function-name))))
    (when sticky
      (ecase sticky
        (:global
         (string-downcase sym))
        (:package
         (format nil "~(~a:~a~)" package-name sym))
        (:local
         (format nil "~(~a:~a:~a~)" package-name function-name sym))))))

To generate the code that sets cookies for sticky parameters, you again loop over the list of parameters, this time collecting a snippet of code for each sticky param. You can use the when and collect it LOOP forms to collect only the non-NIL values returned by set-cookie-code.

(defun set-cookies-code (function-name request params)
  (loop for param in params
       when (set-cookie-code function-name request param) collect it))

(defun set-cookie-code (function-name request param)
  (destructuring-bind (name type &optional default sticky) param
    (declare (ignore type default))
    (if sticky
      `(when ,name
         (set-cookie-header
          ,request
          :name ,(symbol->cookie-name function-name name sticky)
          :value (princ-to-string ,name))))))

One of the advantages of defining macros in terms of helper functions like this is that it’s easy to make sure the individual bits of code you’re generating look right. For instance, you can check that the following set-cookie-code:

(set-cookie-code 'foo 'request '(x integer 20 :local))

generates something like this:

(WHEN X
  (SET-COOKIE-HEADER REQUEST
    :NAME "com.gigamonkeys.web:foo:x"
    :VALUE (PRINC-TO-STRING X)))

Assuming this code will occur in a context where x is the name of a variable, this looks good.

Once again, macros have allowed you to distill the code you need to write down to its essence—in this case, the data you want to extract from the request and the HTML you want to generate. That said, this framework isn’t meant to be the be-all and end-all of Web application frameworks—it’s just a little sugar to make it a bit easier to write simple apps like the one you’ll write in Chapter 29.

But before you can get to that, you need to write the guts of the application for which the Chapter 29 application will be the user interface. You’ll start in the next chapter with a souped-up version of the database you wrote in Chapter 3, this time to keep track of ID3 data extracted from MP3 files.

[12]You need to use LET* rather than a LET to allow the default value forms for parameters to refer to parameters that appear earlier in the parameter list. For example, you could write this:

(define-url-function (request (x integer 10) (y integer (* 2 x))) ...)

and the value of y, if not explicitly supplied, would be twice the value of x.


009 - Querying the Database

Chapter 27 - Practical—An MP3 Database
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Querying the Database

Once you’ve loaded your database with data, you’ll need a way to query it. For the MP3 application you’ll need a slightly more sophisticated query function than you wrote in Chapter 3. This time around you want not only to be able to select rows matching particular criteria but also to limit the results to particular columns, to limit the results to unique rows, and perhaps to sort the rows by particular columns. In keeping with the spirit of relational database theory, the result of a query will be a new table object containing the desired rows and columns.

The query function you’ll write, select, is loosely modeled on the SELECT statement from Structured Query Language (SQL). It’ll take five keyword parameters: :from, :columns, :where, :distinct, and :order-by. The :from argument is the table object you want to query. The :columns argument specifies which columns should be included in the result. The value should be a list of column names, a single column name, or a T, the default, meaning return all columns. The :where argument, if provided, should be a function that accepts a row and returns true if it should be included in the results. In a moment, you’ll write two functions, matching and in, that return functions appropriate for use as :where arguments. The :order-by argument, if supplied, should be a list of column names; the results will be sorted by the named columns. As with the :columns argument, you can specify a single column using just the name, which is equivalent to a one-item list containing the same name. Finally, the :distinct argument is a boolean that says whether to eliminate duplicate rows from the results. The default value for :distinct is NIL.

Here are some examples of using select:

;; Select all rows where the :artist column is "Green Day"
(select :from *mp3s* :where (matching *mp3s* :artist "Green Day"))

;; Select a sorted list of artists with songs in the genre "Rock"
(select
  :columns :artist
  :from *mp3s*
  :where (matching *mp3s* :genre "Rock")
  :distinct t
  :order-by :artist)

The implementation of select with its immediate helper functions looks like this:

(defun select (&key (columns t) from where distinct order-by)
  (let ((rows (rows from))
        (schema (schema from)))

    (when where
      (setf rows (restrict-rows rows where)))

    (unless (eql columns 't)
      (setf schema (extract-schema (mklist columns) schema))
      (setf rows (project-columns rows schema)))

    (when distinct
      (setf rows (distinct-rows rows schema)))

    (when order-by
      (setf rows (sorted-rows rows schema (mklist order-by))))

    (make-instance 'table :rows rows :schema schema)))

(defun mklist (thing)
  (if (listp thing) thing (list thing)))

(defun extract-schema (column-names schema)
  (loop for c in column-names collect (find-column c schema)))

(defun find-column (column-name schema)
  (or (find column-name schema :key #'name)
      (error "No column: ~a in schema: ~a" column-name schema)))

(defun restrict-rows (rows where)
  (remove-if-not where rows))

(defun project-columns (rows schema)
  (map 'vector (extractor schema) rows))

(defun distinct-rows (rows schema)
  (remove-duplicates rows :test (row-equality-tester schema)))

(defun sorted-rows (rows schema order-by)
  (sort (copy-seq rows) (row-comparator order-by schema)))

Of course, the really interesting part of select is how you implement the functions extractor, row-equality-tester, and row-comparator.

As you can tell by how they’re used, each of these functions must return a function. For instance, project-columns uses the value returned by extractor as the function argument to MAP. Since the purpose of project-columns is to return a set of rows with only certain column values, you can infer that extractor returns a function that takes a row as an argument and returns a new row containing only the columns specified in the schema it’s passed. Here’s how you can implement it:

(defun extractor (schema)
  (let ((names (mapcar #'name schema)))
    #'(lambda (row)
        (loop for c in names collect c collect (getf row c)))))

Note how you can do the work of extracting the names from the schema outside the body of the closure: since the closure will be called many times, you want it to do as little work as possible each time it’s called.

The functions row-equality-tester and row-comparator are implemented in a similar way. To decide whether two rows are equivalent, you need to apply the appropriate equality predicate for each column to the appropriate column values. Recall from Chapter 22 that the LOOP clause always will return NIL as soon as a pair of values fails their test or will cause the LOOP to return T.

(defun row-equality-tester (schema)
  (let ((names (mapcar #'name schema))
        (tests (mapcar #'equality-predicate schema)))
    #'(lambda (a b)
        (loop for name in names and test in tests
           always (funcall test (getf a name) (getf b name))))))

Ordering two rows is a bit more complex. In Lisp, comparator functions return true if their first argument should be sorted ahead of the second and NIL otherwise. Thus, a NIL can mean that the second argument should be sorted ahead of the first or that they’re equivalent. You want your row comparators to behave the same way: return T if the first row should be sorted ahead of the second and NIL otherwise.

Thus, to compare two rows, you should compare the values from the columns you’re sorting by, in order, using the appropriate comparator for each column. First call the comparator with the value from the first row as the first argument. If the comparator returns true, that means the first row should definitely be sorted ahead of the second row, so you can immediately return T.

But if the column comparator returns NIL, then you need to determine whether that’s because the second value should sort ahead of the first value or because they’re equivalent. So you should call the comparator again with the arguments reversed. If the comparator returns true this time, it means the second column value sorts ahead of the first and thus the second row ahead of the first row, so you can return NIL immediately. Otherwise, the column values are equivalent, and you need to move onto the next column. If you get through all the columns without one row’s value ever winning the comparison, then the rows are equivalent, and you return NIL. A function that implements this algorithm looks like this:

(defun row-comparator (column-names schema)
  (let ((comparators (mapcar #'comparator (extract-schema column-names schema))))
    #'(lambda (a b)
        (loop
           for name in column-names
           for comparator in comparators
           for a-value = (getf a name)
           for b-value = (getf b name)
           when (funcall comparator a-value b-value) return t
           when (funcall comparator b-value a-value) return nil
           finally (return nil)))))


012 - Other Database Operations

Chapter 27 - Practical—An MP3 Database
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Other Database Operations

Finally, you’ll implement a few other database operations that you’ll need in Chapter 29. The first two are analogs of the SQL DELETE statement. The function delete-rows is used to delete rows from a table that match particular criteria. Like select, it takes :from and :where keyword arguments. Unlike select, it doesn’t return a new table—it actually modifies the table passed as the :from argument.

(defun delete-rows (&key from where)
  (loop
     with rows = (rows from)
     with store-idx = 0
     for read-idx from 0
     for row across rows
     do (setf (aref rows read-idx) nil)
     unless (funcall where row) do
       (setf (aref rows store-idx) row)
       (incf store-idx)
     finally (setf (fill-pointer rows) store-idx)))

In the interest of efficiency, you might want to provide a separate function for deleting all the rows from a table.

(defun delete-all-rows (table)
  (setf (rows table) (make-rows *default-table-size*)))

The remaining table operations don’t really map to normal relational database operations but will be useful in the MP3 browser application. The first is a function to sort the rows of a table in place.

(defun sort-rows (table &rest column-names)
  (setf (rows table)
        (sort (rows table) (row-comparator column-names (schema table))))
  table)

On the flip side, in the MP3 browser application, you’ll need a function that shuffles a table’s rows in place using the function nshuffle-vector from Chapter 23.

(defun shuffle-table (table)
  (nshuffle-vector (rows table))
  table)

And finally, again for the purposes of the MP3 browser, you should provide a function that selects n random rows, returning the results as a new table. It also uses nshuffle-vector along with a version of random-sample based on Algorithm S from Donald Knuth’s The Art of Computer Programming, Volume 2: Seminumerical Algorithms, Third Edition (Addison-Wesley, 1998) that I discussed in Chapter 20.

(defun random-selection (table n)
  (make-instance
   'table
   :schema (schema table)
   :rows (nshuffle-vector (random-sample (rows table) n))))
(defun random-sample (vector n)
  "Based on Algorithm S from Knuth. TAOCP, vol. 2. p. 142"
  (loop with selected = (make-array n :fill-pointer 0)
     for idx from 0
     do
       (loop
          with to-select = (- n (length selected))
          for remaining = (- (length vector) idx)
          while (>= (* remaining (random 1.0)) to-select)
          do (incf idx))
       (vector-push (aref vector idx) selected)
     when (= (length selected) n) return selected))

With this code you’ll be ready, in Chapter 29, to build a Web interface for browsing a collection of MP3 files. But before you get to that, you need to implement the part of the server that streams MP3s using the Shoutcast protocol, which is the topic of the next chapter.


013 - Chapter 28: Practical—A Shoutcast Server

Chapter 28 - Practical—A Shoutcast Server
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Chapter 28: Practical—A Shoutcast Server

In this chapter you’ll develop another important part of what will eventually be a Web-based application for streaming MP3s, namely, the server that implements the Shoutcast protocol for actually streaming MP3s to clients such as iTunes, XMMS,[1] or Winamp.

The Shoutcast Protocol

The Shoutcast protocol was invented by the folks at Nullsoft, the makers of the Winamp MP3 software. It was designed to support Internet audio broadcasting—Shoutcast DJs send audio data from their personal computers to a central Shoutcast server that then turns around and streams it out to any connected listeners.

The server you’ll build is actually only half a true Shoutcast server—you’ll use the protocol that Shoutcast servers use to stream MP3s to listeners, but your server will be able to serve only songs already stored on the file system of the computer where the server is running.

You need to worry about only two parts of the Shoutcast protocol: the request that a client makes in order to start receiving a stream and the format of the response, including the mechanism by which metadata about what song is currently playing is embedded in the stream.

The initial request from the MP3 client to the Shoutcast server is formatted as a normal HTTP request. In response, the Shoutcast server sends an ICY response that looks like an HTTP response except with the string “ICY”[2] in place of the normal HTTP version string and with different headers. After sending the headers and a blank line, the server streams a potentially endless amount of MP3 data.

The only tricky thing about the Shoutcast protocol is the way metadata about the songs being streamed is embedded in the data sent to the client. The problem facing the Shoutcast designers was to provide a way for the Shoutcast server to communicate new title information to the client each time it started playing a new song so the client could display it in its UI. (Recall from Chapter 25 that the MP3 format doesn’t make any provision for encoding metadata.) While one of the design goals of ID3v2 had been to make it better suited for use when streaming MP3s, the Nullsoft folks decided to go their own route and invent a new scheme that’s fairly easy to implement on both the client side and the server side. That, of course, was ideal for them since they were also the authors of their own MP3 client.

Their scheme was to simply ignore the structure of MP3 data and embed a chunk of self-delimiting metadata every n bytes. The client would then be responsible for stripping out this metadata so it wasn’t treated as MP3 data. Since metadata sent to a client that isn’t ready for it will cause glitches in the sound, the server is supposed to send metadata only if the client’s original request contains a special Icy-Metadata header. And in order for the client to know how often to expect metadata, the server must send back a header Icy-Metaint whose value is the number of bytes of MP3 data that will be sent between each chunk of metadata.

The basic content of the metadata is a string of the form “StreamTitle='title';” where title is the title of the current song and can’t contain single quote marks. This payload is encoded as a length-delimited array of bytes: a single byte is sent indicating how many 16-byte blocks follow, and then that many blocks are sent. They contain the string payload as an ASCII string, with the final block padded out with null bytes as necessary.

Thus, the smallest legal metadata chunk is a single byte, zero, indicating zero subsequent blocks. If the server doesn’t need to update the metadata, it can send such an empty chunk, but it must send at least the one byte so the client doesn’t throw away actual MP3 data.

[1]The version of XMMS shipped with Red Hat 8.0 and 9.0 and Fedora no longer knows how to play MP3s because the folks at Red Hat were worried about the licensing issues related to the MP3 codec. To get an XMMS with MP3 support on these versions of Linux, you can grab the source from http://www.xmms.org and build it yourself. Or, see http://www.fedorafaq.org/#xmms-mp3 for information about other possibilities.

[2]To further confuse matters, there’s a different streaming protocol called Icecast. There seems to be no connection between the ICY header used by Shoutcast and the Icecast protocol.


017 - Playlists As Song Sources

Chapter 29 - Practical—An MP3 Browser
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Playlists As Song Sources

To use playlists as a source of songs for the Shoutcast server, you’ll need to implement a method on the generic function find-song-source from Chapter 28. Since you’re going to have multiple playlists, you need a way to find the right one for each client that connects to the server. The mapping part is easy—you can define a variable that holds an EQUAL hash table that you can use to map from some identifier to the playlist object.

(defvar *playlists* (make-hash-table :test #'equal))

You’ll also need to define a process lock to protect access to this hash table like this:

(defparameter *playlists-lock* (make-process-lock :name "playlists-lock"))

Then define a function that looks up a playlist given an ID, creating a new playlist object if necessary and using with-process-lock to ensure that only one thread at a time manipulates the hash table.[1]

(defun lookup-playlist (id)
  (with-process-lock (*playlists-lock*)
    (or (gethash id *playlists*)
        (setf (gethash id *playlists*) (make-instance 'playlist :id id)))))

Then you can implement find-song-source on top of that function and another, playlist-id, that takes an AllegroServe request object and returns the appropriate playlist identifier. The find-song-source function is also where you grab the User-Agent string out of the request object and stash it in the playlist object.

(defmethod find-song-source ((type (eql 'playlist)) request)
  (let ((playlist (lookup-playlist (playlist-id request))))
    (with-playlist-locked (playlist)
      (let ((user-agent (header-slot-value request :user-agent)))
        (when user-agent (setf (user-agent playlist) user-agent))))
    playlist))

The trick, then, is how you implement playlist-id, the function that extracts the identifier from the request object. You have a couple options, each with different implications for the user interface. You can pull whatever information you want out of the request object, but however you decide to identify the client, you need some way for the user of the Web interface to get hooked up to the right playlist.

For now you can take an approach that “just works” as long as there’s only one MP3 client per machine connecting to the server and as long as the user is browsing the Web interface from the machine running the MP3 client: you’ll use the IP address of the client machine as the identifier. This way you can find the right playlist for a request regardless of whether the request is from the MP3 client or a Web browser. You will, however, provide a way in the Web interface to select a different playlist from the browser, so the only real constraint this choice puts on the application is that there can be only one connected MP3 client per client IP address.[2] The implementation of playlist-id looks like this:

(defun playlist-id (request)
  (ipaddr-to-dotted (remote-host (request-socket request))))

The function request-socket is part of AllegroServe, while remote-host and ipaddr-to-dotted are part of Allegro’s socket library.

To make a playlist usable as a song source by the Shoutcast server, you need to define methods on current-song, still-current-p, and maybe-move-to-next-song that specialize their source parameter on playlist. The current-song method is already taken care of: by defining the accessor current-song on the eponymous slot, you automatically got a current-song method specialized on playlist that returns the value of that slot. However, to make accesses to the playlist thread safe, you need to lock the playlist before accessing the current-song slot. In this case, the easiest way is to define an :around method like the following:

(defmethod current-song :around ((playlist playlist))
  (with-playlist-locked (playlist) (call-next-method)))

Implementing still-current-p is also quite simple, assuming you can be sure that current-song gets updated with a new song object only when the current song actually changes. Again, you need to acquire the process lock to ensure you get a consistent view of the playlist’s state.

(defmethod still-current-p (song (playlist playlist))
  (with-playlist-locked (playlist)
    (eql song (current-song playlist))))

The trick, then, is to make sure the current-song slot gets updated at the right times. However, the current song can change in a number of ways. The obvious one is when the Shoutcast server calls maybe-move-to-next-song. But it can also change when songs are added to the playlist, when the Shoutcast server has run out of songs, or even if the playlist’s repeat mode is changed.

Rather than trying to write code specific to every situation to determine whether to update current-song, you can define a function, update-current-if-necessary, that updates current-song if the song object in current-song no longer matches the file that the current-idx slot says should be playing. Then, if you call this function after any manipulation of the playlist that could possibly put those two slots out of sync, you’re sure to keep current-song set properly. Here are update-current-if-necessary and its helper functions:

(defun update-current-if-necessary (playlist)
  (unless (equal (file (current-song playlist))
                 (file-for-current-idx playlist))
    (reset-current-song playlist)))

(defun file-for-current-idx (playlist)
  (if (at-end-p playlist)
    nil
    (column-value (nth-row (current-idx playlist) (songs-table playlist)) :file)))

(defun at-end-p (playlist)
  (>= (current-idx playlist) (table-size (songs-table playlist))))

You don’t need to add locking to these functions since they’ll be called only from functions that will take care of locking the playlist first.

The function reset-current-song introduces one more wrinkle: because you want the playlist to provide an endless stream of MP3s to the client, you don’t want to ever set current-song to NIL. Instead, when a playlist runs out of songs to play—when songs-table is empty or after the last song has been played and repeat is set to :none—then you need to set current-song to a special song whose file is an MP3 of silence[3] and whose title explains why no music is playing. Here’s some code to define two parameters, *empty-playlist-song* and *end-of-playlist-song*, each set to a song with the file named by *silence-mp3* as their file and an appropriate title:

(defparameter *silence-mp3* ...)

(defun make-silent-song (title &optional (file *silence-mp3*))
  (make-instance
   'song
   :file file
   :title title
   :id3-size (if (id3-p file) (size (read-id3 file)) 0)))

(defparameter *empty-playlist-song* (make-silent-song "Playlist empty."))

(defparameter *end-of-playlist-song* (make-silent-song "At end of playlist."))

reset-current-song uses these parameters when the current-idx doesn’t point to a row in songs-table. Otherwise, it sets current-song to a song object representing the current row.

(defun reset-current-song (playlist)
  (setf
   (current-song playlist)
   (cond
     ((empty-p playlist) *empty-playlist-song*)
     ((at-end-p playlist) *end-of-playlist-song*)
     (t (row->song (nth-row (current-idx playlist) (songs-table playlist)))))))

(defun row->song (song-db-entry)
  (with-column-values (file song artist album id3-size) song-db-entry
    (make-instance
     'song
     :file file
     :title (format nil "~a by ~a from ~a" song artist album)
     :id3-size id3-size)))

(defun empty-p (playlist)
  (zerop (table-size (songs-table playlist))))

Now, at last, you can implement the method on maybe-move-to-next-song that moves current-idx to its next value, based on the playlist’s repeat mode, and then calls update-current-if-necessary. You don’t change current-idx when it’s already at the end of the playlist because you want it to keep its current value, so it’ll point at the next song you add to the playlist. This function must lock the playlist before manipulating it since it’s called by the Shoutcast server code, which doesn’t do any locking.

(defmethod maybe-move-to-next-song (song (playlist playlist))
  (with-playlist-locked (playlist)
    (when (still-current-p song playlist)
      (unless (at-end-p playlist)
        (ecase (repeat playlist)
          (:song) ; nothing changes
          (:none (incf (current-idx playlist)))
          (:all  (setf (current-idx playlist)
                       (mod (1+ (current-idx playlist))
                            (table-size (songs-table playlist)))))))
      (update-current-if-necessary playlist))))

[1]The intricacies of concurrent programming are beyond the scope of this book. The basic idea is that if you have multiple threads of control—as you will in this application with some threads running the shoutcast function and other threads responding to requests from the browser—then you need to make sure only one thread at a time manipulates an object in order to prevent one thread from seeing the object in an inconsistent state while another thread is working on it. In this function, for instance, if two new MP3 clients are connecting at the same time, they’d both try to add an entry to *playlists* and might interfere with each other. The with-process-lock ensures that each thread gets exclusive access to the hash table for long enough to do the work it needs to do.

[2]This approach also assumes that every client machine has a unique IP address. This assumption should hold as long as all the users are on the same LAN but may not hold if clients are connecting from behind a firewall that does network address translation. Deploying this application outside a LAN will require some modifications, but if you want to deploy this application to the wider Internet, you’d better know enough about networking to figure out an appropriate scheme yourself.

[3]Unfortunately, because of licensing issues around the MP3 format, it’s not clear that it’s legal for me to provide you with such an MP3 without paying licensing fees to Fraunhofer IIS. I got mine as part of the software that came with my Slimp3 from Slim Devices. You can grab it from their Subversion repository via the Web at http://svn.slimdevices.com/*checkout*/trunk/server/HTML/EN/html/silentpacket.mp3?rev=2. Or buy a Squeezebox, the new, wireless version of Slimp3, and you’ll get silentpacket.mp3 as part of the software that comes with it. Or find an MP3 of John Cage’s piece 4'33".


018 - Manipulating the Playlist

Chapter 29 - Practical—An MP3 Browser
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Manipulating the Playlist

The rest of the playlist code is functions used by the Web interface to manipulate playlist objects, including adding and deleting songs, sorting and shuffling, and setting the repeat mode. As in the helper functions in the previous section, you don’t need to worry about locking in these functions because, as you’ll see, the lock will be acquired in the Web interface function that calls these.

Adding and deleting is mostly a question of manipulating the songs-table. The only extra work you have to do is to keep the current-song and current-idx in sync. For instance, whenever the playlist is empty, its current-idx will be zero, and the current-song will be the *empty-playlist-song*. If you add a song to an empty playlist, then the index of zero is now in bounds, and you should change the current-song to the newly added song. By the same token, when you’ve played all the songs in a playlist and current-song is *end-of-playlist-song*, adding a song should cause current-song to be reset. All this really means, though, is that you need to call update-current-if-necessary at the appropriate points.

Adding songs to a playlist is a bit involved because of the way the Web interface communicates which songs to add. For reasons I’ll discuss in the next section, the Web interface code can’t just give you a simple set of criteria to use in selecting songs from the database. Instead, it gives you the name of a column and a list of values, and you’re supposed to add all the songs from the main database where the given column has a value in the list of values. Thus, to add the right songs, you need to first build a table object containing the desired values, which you can then use with an in query against the song database. So, add-songs looks like this:

(defun add-songs (playlist column-name values)
  (let ((table (make-instance
                'table
                :schema (extract-schema (list column-name) (schema *mp3s*)))))
    (dolist (v values) (insert-row (list column-name v) table))
    (do-rows (row (select :from *mp3s* :where (in column-name table)))
      (insert-row row (songs-table playlist))))
  (update-current-if-necessary playlist))

Deleting songs is a bit simpler; you just need to be able to delete songs from the songs-table that match particular criteria—either a particular song or all songs in a particular genre, by a particular artist, or from a particular album. So, you can provide a delete-songs function that takes keyword/value pairs, which are used to construct a matching :where clause you can pass to the delete-rows database function.

Another complication that arises when deleting songs is that current-idx may need to change. Assuming the current song isn’t one of the ones just deleted, you’d like it to remain the current song. But if songs before it in songs-table are deleted, it’ll be in a different position in the table after the delete. So after a call to delete-rows, you need to look for the row containing the current song and reset current-idx. If the current song has itself been deleted, then, for lack of anything better to do, you can reset current-idx to zero. After updating current-idx, calling update-current-if-necessary will take care of updating current-song. And if current-idx changed but still points at the same song, current-song will be left alone.

(defun delete-songs (playlist &rest names-and-values)
  (delete-rows
   :from (songs-table playlist)
   :where (apply #'matching (songs-table playlist) names-and-values))
  (setf (current-idx playlist) (or (position-of-current playlist) 0))
  (update-current-if-necessary playlist))

(defun position-of-current (playlist)
  (let* ((table (songs-table playlist))
         (matcher (matching table :file (file (current-song playlist))))
         (pos 0))
    (do-rows (row table)
      (when (funcall matcher row)
        (return-from position-of-current pos))
      (incf pos))))

You can also provide a function to completely clear the playlist, which uses delete-all-rows and doesn’t have to worry about finding the current song since it has obviously been deleted. The call to update-current-if-necessary will take care of setting current-song to NIL.

(defun clear-playlist (playlist)
  (delete-all-rows (songs-table playlist))
  (setf (current-idx playlist) 0)
  (update-current-if-necessary playlist))

Sorting and shuffling the playlist are related in that the playlist is always either sorted or shuffled. The shuffle slot says whether the playlist should be shuffled and if so how. If it’s set to :none, then the playlist is ordered according to the value in the ordering slot. When shuffle is :song, the playlist will be randomly permuted. And when it’s set to :album, the list of albums is randomly permuted, but the songs within each album are listed in track order. Thus, the sort-playlist function, which will be called by the Web interface code whenever the user selects a new ordering, needs to set ordering to the desired ordering and set shuffle to :none before calling order-playlist, which actually does the sort. As in delete-songs, you need to use position-of-current to reset current-idx to the new location of the current song. However, this time you don’t need to call update-current-if-necessary since you know the current song is still in the table.

(defun sort-playlist (playlist ordering)
  (setf (ordering playlist) ordering)
  (setf (shuffle playlist) :none)
  (order-playlist playlist)
  (setf (current-idx playlist) (position-of-current playlist)))

In order-playlist, you can use the database function sort-rows to actually perform the sort, passing a list of columns to sort by based on the value of ordering.

(defun order-playlist (playlist)
  (apply #'sort-rows (songs-table playlist)
    (case (ordering playlist)
      (:genre  '(:genre :album :track))
      (:artist '(:artist :album :track))
      (:album  '(:album :track))
      (:song   '(:song)))))

The function shuffle-playlist, called by the Web interface code when the user selects a new shuffle mode, works in a similar fashion except it doesn’t need to change the value of ordering. Thus, when shuffle-playlist is called with a shuffle of :none, the playlist goes back to being sorted according to the most recent ordering. Shuffling by songs is simple—just call shuffle-table on songs-table. Shuffling by albums is a bit more involved but still not rocket science.

(defun shuffle-playlist (playlist shuffle)
  (setf (shuffle playlist) shuffle)
  (case shuffle
    (:none (order-playlist playlist))
    (:song (shuffle-by-song playlist))
    (:album (shuffle-by-album playlist)))
  (setf (current-idx playlist) (position-of-current playlist)))

(defun shuffle-by-song (playlist)
  (shuffle-table (songs-table playlist)))

(defun shuffle-by-album (playlist)
  (let ((new-table (make-playlist-table)))
    (do-rows (album-row (shuffled-album-names playlist))
      (do-rows (song (songs-for-album playlist (column-value album-row :album)))
        (insert-row song new-table)))
    (setf (songs-table playlist) new-table)))

(defun shuffled-album-names (playlist)
  (shuffle-table
   (select
    :columns :album
    :from (songs-table playlist)
    :distinct t)))

(defun songs-for-album (playlist album)
  (select
   :from (songs-table playlist)
   :where (matching (songs-table playlist) :album album)
   :order-by :track))

The last manipulation you need to support is setting the playlist’s repeat mode. Most of the time you don’t need to take any extra action when setting repeat—its value comes into play only in maybe-move-to-next-song. However, you need to update the current-song as a result of changing repeat in one situation, namely, if current-idx is at the end of a nonempty playlist and repeat is being changed to :song or :all. In that case, you want to continue playing, either repeating the last song or starting at the beginning of the playlist. So, you should define an :after method on the generic function (setf repeat).

(defmethod (setf repeat) :after (value (playlist playlist))
  (if (and (at-end-p playlist) (not (empty-p playlist)))
    (ecase value
      (:song (setf (current-idx playlist) (1- (table-size (songs-table playlist)))))
      (:none)
      (:all  (setf (current-idx playlist) 0)))
    (update-current-if-necessary playlist)))

Now you have all the underlying bits you need. All that remains is the code that will provide a Web-based user interface for browsing the MP3 database and manipulating playlists. The interface will consist of three main functions defined with define-url-function: one for browsing the song database, one for viewing and manipulating a single playlist, and one for listing all the available playlists.

But before you get to writing these three functions, you need to start with some helper functions and HTML macros that they’ll use.


021 - The Browse Page

Chapter 29 - Practical—An MP3 Browser
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

The Browse Page

The first URL function will generate a page for browsing the MP3 database. Its query parameters will tell it what kind of thing the user is browsing and provide the criteria of what elements of the database they’re interested in. It’ll give them a way to select database entries that match a specific genre, artist, or album. In the interest of serendipity, you can also provide a way to select a random subset of matching items. When the user is browsing at the level of individual songs, the title of the song will be a link that causes that song to be added to the playlist. Otherwise, each item will be presented with links that let the user browse the listed item by some other category. For example, if the user is browsing genres, the entry “Blues” will contain links to browse all albums, artists, and songs in the genre Blues. Additionally, the browse page will feature an “Add all” button that adds every song matching the page’s criteria to the user’s playlist. The function looks like this:

(define-url-function browse
    (request (what keyword :genre) genre artist album (random integer))

  (let* ((values (values-for-page what genre artist album random))
         (title (browse-page-title what random genre artist album))
         (single-column (if (eql what :song) :file what))
         (values-string (values->base-64 single-column values)))
    (html
     (:mp3-browser-page
      (:title title)
      ((:form :method "POST" :action "playlist")
       (:input :name "values" :type "hidden" :value values-string)
       (:input :name "what" :type "hidden" :value single-column)
       (:input :name "action" :type "hidden" :value :add-songs)
       (:input :name "submit" :type "submit" :value "Add all"))
      (:ul (do-rows (row values) (list-item-for-page what row)))))))

This function starts by using the function values-for-page to get a table containing the values it needs to present. When the user is browsing by song—when the what parameter is :song—you want to select complete rows from the database. But when they’re browsing by genre, artist, or album, you want to select only the distinct values for the given category. The database function select does most of the heavy lifting, with values-for-page mostly responsible for passing the right arguments depending on the value of what. This is also where you select a random subset of the matching rows if necessary.

(defun values-for-page (what genre artist album random)
  (let ((values
         (select
          :from *mp3s*
          :columns (if (eql what :song) t what)
          :where (matching *mp3s* :genre genre :artist artist :album album)
          :distinct (not (eql what :song))
          :order-by (if (eql what :song) '(:album :track) what))))
    (if random (random-selection values random) values)))

To generate the title for the browse page, you pass the browsing criteria to the following function, browse-page-title:

(defun browse-page-title (what random genre artist album)
  (with-output-to-string (s)
    (when random (format s "~:(~r~) Random " random))
    (format s "~:(~a~p~)" what random)
    (when (or genre artist album)
      (when (not (eql what :song)) (princ " with songs" s))
      (when genre  (format s " in genre ~a" genre))
      (when artist (format s " by artist ~a" artist))
      (when album  (format s " on album ~a" album)))))

Once you have the values you want to present, you need to do two things with them. The main task, of course, is to present them, which happens in the do-rows loop, leaving the rendering of each row to the function list-item-for-page. That function renders :song rows one way and all other kinds another way.

(defun list-item-for-page (what row)
  (if (eql what :song)
      (with-column-values (song file album artist genre) row
        (html
         (:li
          (:a :href (link "playlist" :file file :action "add-songs") (:b song))
          " from "
          (:a :href (link "browse"  :what :song :album  album) album)
          " by "
          (:a :href (link "browse" :what :song :artist artist) artist)
          " in genre "
          (:a :href (link "browse"  :what :song :genre  genre) genre))))
      (let ((value (column-value row what)))
        (html
         (:li value " - "
              (browse-link :genre  what value)
              (browse-link :artist what value)
              (browse-link :album  what value)
              (browse-link :song   what value))))))

(defun browse-link (new-what what value)
  (unless (eql new-what what)
    (html
     "["
     (:a :href (link "browse" :what new-what what value)
         (:format "~(~as~)" new-what))
     "] ")))

The other thing on the browse page is a form with several hidden INPUT fields and an “Add all” submit button. You need to use an HTML form instead of a regular link to keep the application stateless—to make sure all the information needed to respond to a request comes in the request itself. Because the browse page results can be partially random, you need to submit a fair bit of data for the server to be able to reconstitute the list of songs to add to the playlist. If you didn’t allow the browse page to return randomly generated results, you wouldn’t need much data—you could just submit a request to add songs with whatever search criteria the browse page used. But if you added songs that way, with criteria that included a random argument, then you’d end up adding a different set of random songs than the user was looking at on the page when they hit the “Add all” button.

The solution you’ll use is to send back a form that has enough information stashed away in a hidden INPUT element to allow the server to reconstitute the list of songs matching the browse page criteria. That information is the list of values returned by values-for-page and the value of the what parameter. This is where you use the base64-list parameter type; the function values->base64 extracts the values of a specified column from the table returned by values-for-page into a list and then makes a base 64–encoded string out of that list to embed in the form.

(defun values->base-64 (column values-table)
  (flet ((value (r) (column-value r column)))
    (obj->base64 (map-rows #'value values-table))))

When that parameter comes back as the value of the values query parameter to a URL function that declares values to be of type base-64-list, it’ll be automatically converted back to a list. As you’ll see in a moment, that list can then be used to construct a query that’ll return the correct list of songs.[5] When you’re browsing by :song, you use the values from the :file column since they uniquely identify the actual songs while the song names may not.

[5]This solution has its drawbacks—if a browse page returns a lot of results, a fair bit of data is going back and forth under the covers. Also, the database queries aren’t necessarily the most efficient. But it does keep the application stateless. An alternative approach is to squirrel away, on the server side, information about the results returned by browse and then, when a request to add songs come in, find the appropriate bit of information in order to re-create the correct set of songs. For instance, you could just save the values list instead of sending it back in the form. Or you could copy the RANDOM-STATE object before you generate the browse results so you can later re-create the same “random” results. But this approach causes its own problems. For instance, you’d then need to worry about when you can get rid of the squirreled-away information; you never know when the user might hit the Back button on their browser to return to an old browse page and then hit the “Add all” button. Welcome to the wonderful world of Web programming.


027 - The FOO Language

Chapter 30 - Practical—An HTML Generation Library, the Interpreter
Practical Common Lisp
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

The FOO Language

So, enough theory. I’ll give you a quick overview of the language implemented by FOO, and then you’ll look at the implementation of the two FOO language processors—the interpreter, in this chapter, and the compiler, in the next.

Like Lisp itself, the basic syntax of the FOO language is defined in terms of forms made up of Lisp objects. The language defines how each legal FOO form is translated into HTML.

The simplest FOO forms are self-evaluating Lisp objects such as strings, numbers, and keyword symbols.[3] You’ll need a function self-evaluating-p that tests whether a given object is self-evaluating for FOO’s purposes.

(defun self-evaluating-p (form)
  (and (atom form) (if (symbolp form) (keywordp form) t)))

Objects that satisfy this predicate will be emitted by converting them to strings with PRINC-TO-STRING and then escaping any reserved characters, such as <, >, or &. When the value is being emitted as an attribute, the characters ", and ' are also escaped. Thus, you can invoke the html macro on a self-evaluating object to emit it to *html-output* (which is initially bound to *STANDARD-OUTPUT*). Table 30-1 shows how a few different self-evaluating values will be output.

Table 30-1: FOO Output for Self-Evaluating Objects
FOO Form | Generated HTML
"foo" | foo
10 | 10
:foo | FOO
"foo & bar" | foo &amp; bar

Of course, most HTML consists of tagged elements. The three pieces of information that describe each element are the tag, a set of attributes, and a body containing text and/or more HTML elements. Thus, you need a way to represent these three pieces of information as Lisp objects, preferably ones that the Lisp reader already knows how to read.[4] If you forget about attributes for a moment, there’s an obvious mapping between Lisp lists and HTML elements: any HTML element can be represented by a list whose FIRST is a symbol where the name is the name of the element’s tag and whose REST is a list of self-evaluating objects or lists representing other HTML elements. Thus:

<p>Foo</p> ↔ (:p "Foo")

<p><i>Now</i> is the time</p> ↔ (:p (:i "Now") " is the time")

Now the only problem is where to squeeze in the attributes. Since most elements have no attributes, it’d be nice if you could use the preceding syntax for elements without attributes. FOO provides two ways to notate elements with attributes. The first is to simply include the attributes in the list immediately following the symbol, alternating keyword symbols naming the attributes and objects representing the attribute value forms. The body of the element starts with the first item in the list that’s in a position to be an attribute name and isn’t a keyword symbol. Thus:

HTML> (html (:p "foo"))
<p>foo</p>
NIL
HTML> (html (:p "foo " (:i "bar") " baz"))
<p>foo <i>bar</i> baz</p>
NIL
HTML> (html (:p :style "foo" "Foo"))
<p style='foo'>Foo</p>
NIL
HTML> (html (:p :id "x" :style "foo" "Foo"))
<p id='x' style='foo'>Foo</p>
NIL

For folks who prefer a bit more obvious delineation between the element’s attributes and its body, FOO supports an alternative syntax: if the first element of a list is itself a list with a keyword as its first element, then the outer list represents an HTML element with that keyword indicating the tag, with the REST of the nested list as the attributes, and with the REST of the outer list as the body. Thus, you could write the previous two expressions like this:

HTML> (html ((:p :style "foo") "Foo"))
<p style='foo'>Foo</p>
NIL
HTML> (html ((:p :id "x" :style "foo") "Foo"))
<p id='x' style='foo'>Foo</p>
NIL

The following function tests whether a given object matches either of these syntaxes:

(defun cons-form-p (form &optional (test #'keywordp))
  (and (consp form)
       (or (funcall test (car form))
           (and (consp (car form)) (funcall test (caar form))))))

You should parameterize the test function because later you’ll need to test the same two syntaxes with a slightly different predicate on the name.

To completely abstract the differences between the two syntax variants, you can define a function, parse-cons-form, that takes a form and parses it into three elements, the tag, the attributes plist, and the body list, returning them as multiple values. The code that actually evaluates cons forms will use this function and not have to worry about which syntax was used.

(defun parse-cons-form (sexp)
  (if (consp (first sexp))
    (parse-explicit-attributes-sexp sexp)
    (parse-implicit-attributes-sexp sexp)))

(defun parse-explicit-attributes-sexp (sexp)
  (destructuring-bind ((tag &rest attributes) &body body) sexp
    (values tag attributes body)))

(defun parse-implicit-attributes-sexp (sexp)
  (loop with tag = (first sexp)
     for rest on (rest sexp) by #'cddr
     while (and (keywordp (first rest)) (second rest))
     when (second rest)
       collect (first rest) into attributes and
       collect (second rest) into attributes
     end
     finally (return (values tag attributes rest))))

Now that you have the basic language specified, you can think about how you’re actually going to implement the language processors. How do you get from a series of FOO forms to the desired HTML? As I mentioned previously, you’ll be implementing two language processors for FOO: an interpreter that walks a tree of FOO forms and emits the corresponding HTML directly and a compiler that walks a tree and translates it into Common Lisp code that’ll emit the same HTML. Both the interpreter and compiler will be built on top of a common foundation of code, which provides support for things such as escaping reserved characters and generating nicely indented output, so it makes sense to start there.

[3]In the strict language of the Common Lisp standard, keyword symbols aren’t self-evaluating, though they do, in fact, evaluate to themselves. See section 3.1.2.1.3 of the language standard or HyperSpec for a brief discussion.

[4]The requirement to use objects that the Lisp reader knows how to read isn’t a hard-and-fast one. Since the Lisp reader is itself customizable, you could also define a new reader-level syntax for a new kind of object. But that tends to be more trouble than it’s worth.


028 - Character Escaping

Chapter 30 - Practical—An HTML Generation Library, the Interpreter
Practical Common Lisp
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Character Escaping

The first bit of the foundation you’ll need to lay is the code that knows how to escape characters with a special meaning in HTML. There are three such characters, and they must not appear in the text of an element or in an attribute value; they are <, >, and &. In element text or attribute values, these characters must be replaced with the character reference entities &lt;, &gt;, and &amp;. Similarly, in attribute values, the quotation marks used to delimit the value must be escaped, ' with &apos; and " with &quot;. Additionally, any character can be represented by a numeric character reference entity consisting of an ampersand, followed by a sharp sign, followed by the numeric code as a base 10 integer, and followed by a semicolon. These numeric escapes are sometimes used to embed non-ASCII characters in HTML.

Image from book
THE PACKAGE

Since FOO is a low-level library, the package you develop it in doesn’t rely on much external code—just the usual dependency on names from the COMMON-LISP package and, almost as usual, on the names of the macro-writing macros from COM.GIGAMONKEYS.MACRO-UTILITIES. On the other hand, the package needs to export all the names needed by code that uses FOO. Here’s the DEFPACKAGE from the source that you can download from the book’s Web site:

(defpackage :com.gigamonkeys.html
  (:use :common-lisp :com.gigamonkeys.macro-utilities)
  (:export :with-html-output
           :in-html-style
           :define-html-macro
           :html
           :emit-html
           :&attributes))

Image from book

The following function accepts a single character and returns a string containing a character reference entity for that character:

(defun escape-char (char)
  (case char
    (#\& "&amp;")
    (#\< "&lt;")
    (#\> "&gt;")
    (#\' "&apos;")
    (#\" "&quot;")
    (t (format nil "&#~d;" (char-code char)))))

You can use this function as the basis for a function, escape, that takes a string and a sequence of characters and returns a copy of the first argument with all occurrences of the characters in the second argument replaced with the corresponding character entity returned by escape-char.

(defun escape (in to-escape)
  (flet ((needs-escape-p (char) (find char to-escape)))
    (with-output-to-string (out)
      (loop for start = 0 then (1+ pos)
            for pos = (position-if #'needs-escape-p in :start start)
            do (write-sequence in out :start start :end pos)
            when pos do (write-sequence (escape-char (char in pos)) out)
            while pos))))

You can also define two parameters: *element-escapes*, which contains the characters you need to escape in normal element data, and *attribute-escapes*, which contains the set of characters to be escaped in attribute values.

(defparameter *element-escapes* "<>&")
(defparameter *attribute-escapes* "<>&\"'")

Here are some examples:

HTML> (escape "foo & bar" *element-escapes*)
"foo &amp; bar"
HTML> (escape "foo & 'bar'" *element-escapes*)
"foo &amp; 'bar'"
HTML> (escape "foo & 'bar'" *attribute-escapes*)
"foo &amp; &apos;bar&apos;"

Finally, you’ll need a variable, *escapes*, that will be bound to the set of characters that need to be escaped. It’s initially set to the value of *element-escapes*, but when generating attributes, it will, as you’ll see, be rebound to the value of *attribute-escapes*.

(defvar *escapes* *element-escapes*)


029 - Indenting Printer

Chapter 30 - Practical—An HTML Generation Library, the Interpreter
Practical Common Lisp
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Indenting Printer

To handle generating nicely indented output, you can define a class indenting-printer, which wraps around an output stream, and functions that use an instance of that class to emit strings to the stream while keeping track of when it’s at the beginning of the line. The class looks like this:

(defclass indenting-printer ()
  ((out                 :accessor out                 :initarg :out)
   (beginning-of-line-p :accessor beginning-of-line-p :initform t)
   (indentation         :accessor indentation         :initform 0)
   (indenting-p         :accessor indenting-p         :initform t)))

The main function that operates on indenting-printers is emit, which takes the printer and a string and emits the string to the printer’s output stream, keeping track of when it emits a newline so it can reset the beginning-of-line-p slot.

(defun emit (ip string)
  (loop for start = 0 then (1+ pos)
     for pos = (position #\Newline string :start start)
     do (emit/no-newlines ip string :start start :end pos)
     when pos do (emit-newline ip)
     while pos))

To actually emit the string, it uses the function emit/no-newlines, which emits any needed indentation, via the helper indent-if-necessary, and then writes the string to the stream. This function can also be called directly by other code to emit a string that’s known not to contain any newlines.

(defun emit/no-newlines (ip string &key (start 0) end)
  (indent-if-necessary ip)
  (write-sequence string (out ip) :start start :end end)
  (unless (zerop (- (or end (length string)) start))
    (setf (beginning-of-line-p ip) nil)))

The helper indent-if-necessary checks beginning-of-line-p and indenting-p to determine whether it needs to emit indentation and, if they’re both true, emits as many spaces as indicated by the value of indentation. Code that uses the indenting-printer can control the indentation by manipulating the indentation and indenting-p slots. Incrementing and decrementing indentation changes the number of leading spaces, while setting indenting-p to NIL can temporarily turn off indentation.

(defun indent-if-necessary (ip)
  (when (and (beginning-of-line-p ip) (indenting-p ip))
    (loop repeat (indentation ip) do (write-char #\Space (out ip)))
    (setf (beginning-of-line-p ip) nil)))

The last two functions in the indenting-printer API are emit-newline and emit-freshline, which are both used to emit a newline character, similar to the ~% and ~& FORMAT directives. That is, the only difference is that emit-newline always emits a newline, while emit-freshline does so only if beginning-of-line-p is false. Thus, multiple calls to emit-freshline without any intervening emits won’t result in a blank line. This is handy when one piece of code wants to generate some output that should end with a newline while another piece of code wants to generate some output that should start on a newline but you don’t want a blank line between the two bits of output.

(defun emit-newline (ip)
  (write-char #\Newline (out ip))
  (setf (beginning-of-line-p ip) t))

(defun emit-freshline (ip)
  (unless (beginning-of-line-p ip) (emit-newline ip)))

With those preliminaries out of the way, you’re ready to get to the guts of the FOO processor.


030 - HTML Processor Interface

Chapter 30 - Practical—An HTML Generation Library, the Interpreter
Practical Common Lisp
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

HTML Processor Interface

Now you’re ready to define the interface that’ll be used by the FOO language processor to emit HTML. You can define this interface as a set of generic functions because you’ll need two implementations—one that actually emits HTML and another that the html macro can use to collect a list of actions that need to be performed, which can then be optimized and compiled into code that emits the same output in a more efficient way. I’ll call this set of generic functions the backend interface. It consists of the following eight generic functions:

(defgeneric raw-string (processor string &optional newlines-p))

(defgeneric newline (processor))

(defgeneric freshline (processor))

(defgeneric indent (processor))

(defgeneric unindent (processor))

(defgeneric toggle-indenting (processor))

(defgeneric embed-value (processor value))

(defgeneric embed-code (processor code))

While several of these functions have obvious correspondence to indenting-printer functions, it’s important to understand that these generic functions define the abstract operations that are used by the FOO language processors and won’t always be implemented in terms of calls to the indenting-printer functions.

That said, perhaps the easiest way to understand the semantics of these abstract operations is to look at the concrete implementations of the methods specialized on html-pretty-printer, the class used to generate human-readable HTML.


035 - FOO Special Operators

Chapter 31 - Practical—An HTML Generation Library, the Compiler
Practical Common Lisp
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

FOO Special Operators

You could stop there; certainly the FOO language is expressive enough to generate nearly any HTML you’d care to. However, you can add two features to the language, with just a bit more code, that will make it quite a bit more powerful: special operators and macros.

Special operators in FOO are analogous to special operators in Common Lisp. Special operators provide ways to express things in the language that can’t be expressed in the language supported by the basic evaluation rule. Or, another way to look at it is that special operators provide access to the primitive mechanisms used by the language evaluator. [1]

To take a simple example, in the FOO compiler, the language evaluator uses the embed-value function to generate code that will embed the value of a variable in the output HTML. However, because only symbols are passed to embed-value, there’s no way, in the language I’ve described so far, to embed the value of an arbitrary Common Lisp expression; the process function passes cons cells to embed-code rather than embed-value, so the values returned are ignored. Typically this is what you’d want, since the main reason to embed Lisp code in a FOO program is to use Lisp control constructs. However, sometimes you’d like to embed computed values in the generated HTML. For example, you might like this FOO program to generate a paragraph tag containing a random number:

(:p (random 10))

But that doesn’t work because the code is run and its value discarded.

HTML> (html (:p (random 10)))
<p></p>
NIL

In the language, as you’ve implemented it so far, you could work around this limitation by computing the value outside the call to html and then embedding it via a variable.

HTML> (let ((x (random 10))) (html (:p x)))
<p>1</p>
NIL

But that’s sort of annoying, particularly when you consider that if you could arrange for the form (random 10) to be passed to embed-value instead of embed-code, it’d do exactly what you want. So, you can define a special operator, :print, that’s processed by the FOO language processor according to a different rule than a normal FOO expression. Namely, instead of generating a <print> element, it passes the form in its body to embed-value. Thus, you can generate a paragraph containing a random number like this:

HTML> (html (:p (:print (random 10))))
<p>9</p>
NIL

Obviously, this special operator is useful only in compiled FOO code since embed-value doesn’t work in the interpreter. Another special operator that can be used in both interpreted and compiled FOO code is :format, which lets you generate output using the FORMAT function. The arguments to the :format special operator are a string used as a format control string and then any arguments to be interpolated. When all the arguments to :format are self-evaluating objects, a string is generated by passing them to FORMAT, and that string is then emitted like any other string. This allows such :format forms to be used in FOO passed to emit-html. In compiled FOO, the arguments to :format can be any Lisp expressions.

Other special operators provide control over what characters are automatically escaped and to explicitly emit newline characters: the :noescape special operator causes all the forms in its body to be evaluated as regular FOO forms but with *escapes* bound to NIL, while :attribute evaluates the forms in its body with *escapes* bound to *attribute-escapes*. And :newline is translated into code to emit an explicit newline.

So, how do you define special operators? There are two aspects to processing special operators: how does the language processor recognize forms that use special operators, and how does it know what code to run to process each special operator?

You could hack process-sexp-html to recognize each special operator and handle it in the appropriate manner—special operators are, logically, part of the implementation of the language, and there aren’t going to be that many of them. However, it’d be nice to have a slightly more modular way to add new special operators—not because users of FOO will be able to but just for your own sanity.

Define a special form as any list whose CAR is a symbol that’s the name of a special operator. You can mark the names of special operators by adding a non-NIL value to the symbol’s property list under the key html-special-operator. So, you can define a function that tests whether a given form is a special form like this:

(defun special-form-p (form)
  (and (consp form) (symbolp (car form)) (get (car form) 'html-special-operator)))

The code that implements each special operator is responsible for taking apart the rest of the list however it sees fit and doing whatever the semantics of the special operator require. Assuming you’ll also define a function process-special-form, which will take the language processor and a special form and run the appropriate code to generate a sequence of calls on the processor object, you can augment the top-level process function to handle special forms like this:

(defun process (processor form)
  (cond
    ((special-form-p form) (process-special-form processor form))
    ((sexp-html-p form)    (process-sexp-html processor form))
    ((consp form)          (embed-code processor form))
    (t                     (embed-value processor form))))

You must add the special-form-p clause first because special forms can look, syntactically, like regular FOO expressions just the way Common Lisp’s special forms can look like regular function calls.

Now you just need to implement process-special-form. Rather than define a single monolithic function that implements all the special operators, you should define a macro that allows you to define special operators much like regular functions and that also takes care of adding the html-special-operator entry to the property list of the special operator’s name. In fact, the value you store in the property list can be a function that implements the special operator. Here’s the macro:

(defmacro define-html-special-operator (name (processor &rest other-parameters)
                                        &body body)
  `(eval-when (:compile-toplevel :load-toplevel :execute)
     (setf (get ',name 'html-special-operator)
           (lambda (,processor ,@other-parameters) ,@body))))

This is a fairly advanced type of macro, but if you take it one line at a time, there’s nothing all that tricky about it. To see how it works, take a simple use of the macro, the definition of the special operator :noescape, and look at the macro expansion. If you write this:

(define-html-special-operator :noescape (processor &rest body)
  (let ((*escapes* nil))
    (loop for exp in body do (process processor exp))))

it’s as if you had written this:

(eval-when (:compile-toplevel :load-toplevel :execute)
  (setf (get ':noescape 'html-special-operator)
        (lambda (processor &rest body)
          (let ((*escapes* nil))
            (loop for exp in body do (process processor exp))))))

The EVAL-WHEN special operator, as I discussed in Chapter 20, ensures that the effects of code in its body will be made visible during compilation when you compile with COMPILE-FILE. This matters if you want to use define-html-special-operator in a file and then use the just-defined special operator in that same file.

Then the SETF expression sets the property html-special-operator on the symbol :noescape to an anonymous function with the same parameter list as was specified in define-html-special-operator. By defining define-html-special-operator to split the parameter list in two parts, processor and everything else, you ensure that all special operators accept at least one argument.

The body of the anonymous function is then the body provided to define-html-special-operator. The job of the anonymous function is to implement the special operator by making the appropriate calls on the backend interface to generate the correct HTML or the code that will generate it. It can also use process to evaluate an expression as a FOO form.

The :noescape special operator is particularly simple—all it does is pass the forms in its body to process with *escapes* bound to NIL. In other words, this special operator disables the normal character escaping preformed by process-sexp-html.

With special operators defined this way, all process-special-form has to do is look up the anonymous function in the property list of the special operator’s name and APPLY it to the processor and rest of the form.

(defun process-special-form (processor form)
  (apply (get (car form) 'html-special-operator) processor (rest form)))

Now you’re ready to define the five remaining FOO special operators. Similar to :noescape is :attribute, which evaluates the forms in its body with *escapes* bound to *attribute-escapes*. This special operator is useful if you want to write helper functions that output attribute values. If you write a function like this:

(defun foo-value (something)
  (html (:print (frob something))))

the html macro is going to generate code that escapes the characters in *element-escapes*. But if you’re planning to use foo-value like this:

(html (:p :style (foo-value 42) "Foo"))

then you want it to generate code that uses *attribute-escapes*. So, instead, you can write it like this: [2]

(defun foo-value (something)
  (html (:attribute (:print (frob something)))))

The definition of :attribute looks like this:

(define-html-special-operator :attribute (processor &rest body)
  (let ((*escapes* *attribute-escapes*))
    (loop for exp in body do (process processor exp))))

The next two special operators, :print and :format, are used to output values. The :print special operator, as I discussed earlier, is used in compiled FOO programs to embed the value of an arbitrary Lisp expression. The :format special operator is more or less equivalent to generating a string with (format nil …) and then embedding it. The primary reason to define :format as a special operator is for convenience. This:

(:format "Foo: ~d" x)

is nicer than this:

(:print (format nil "Foo: ~d" x))

It also has the slight advantage that if you use :format with arguments that are all self-evaluating, FOO can evaluate the :format at compile time rather than waiting until runtime. The definitions of :print and :format are as follows:

(define-html-special-operator :print (processor form)
  (cond
    ((self-evaluating-p form)
     (warn "Redundant :print of self-evaluating form ~s" form)
     (process-sexp-html processor form))
    (t
     (embed-value processor form))))

(define-html-special-operator :format (processor &rest args)
  (if (every #'self-evaluating-p args)
    (process-sexp-html processor (apply #'format nil args))
    (embed-value processor `(format nil ,@args))))

The :newline special operator forces an output of a literal newline, which is occasionally handy.

(define-html-special-operator :newline (processor)
  (newline processor))

Finally, the :progn special operator is analogous to the PROGN special operator in Common Lisp. It simply processes the forms in its body in sequence.

(define-html-special-operator :progn (processor &rest body)
  (loop for exp in body do (process processor exp)))

In other words, the following:

(html (:p (:progn "Foo " (:i "bar") " baz")))

will generate the same code as this:

(html (:p "Foo " (:i "bar") " baz"))

This might seem like a strange thing to need since normal FOO expressions can have any number of forms in their body. However, this special operator will come in quite handy in one situation—when writing FOO macros, which brings you to the last language feature you need to implement.

[1]The analogy between FOO’s special operators, and macros, which I’ll discuss in the next section, and Lisp’s own is fairly sound. In fact, understanding how FOO’s special operators and macros work may give you some insight into why Common Lisp is put together the way it is.

[2]The :noescape and :attribute special operators must be defined as special operators because FOO determines what escapes to use at compile time, not at runtime. This allows FOO to escape literal values at compile time, which is much more efficient than having to scan all output at runtime.


036 - FOO Macros

Chapter 31 - Practical—An HTML Generation Library, the Compiler
Practical Common Lisp
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

FOO Macros

FOO macros are similar in spirit to Common Lisp’s macros. A FOO macro is a bit of code that accepts a FOO expression as an argument and returns a new FOO expression as the result, which is then evaluated according to the normal FOO evaluation rules. The actual implementation is quite similar to the implementation of special operators.

As with special operators, you can define a predicate function to test whether a given form is a macro form.

(defun macro-form-p (form)
  (cons-form-p form #'(lambda (x) (and (symbolp x) (get x 'html-macro)))))

You use the previously defined function cons-form-p because you want to allow macros to be used in either of the syntaxes of nonmacro FOO cons forms. However, you need to pass a different predicate function, one that tests whether the form name is a symbol with a non-NIL html-macro property. Also, as in the implementation of special operators, you’ll define a macro for defining FOO macros, which is responsible for storing a function in the property list of the macro’s name, under the key html-macro. However, defining a macro is a bit more complicated because FOO supports two flavors of macro. Some macros you’ll define will behave much like normal HTML elements and may want to have easy access to a list of attributes. Other macros will simply want raw access to the elements of their body.

You can make the distinction between the two flavors of macros implicit: when you define a FOO macro, the parameter list can include an &attributes parameter. If it does, the macro form will be parsed like a regular cons form, and the macro function will be passed two values, a plist of attributes and a list of expressions that make up the body of the form. A macro form without an &attributes parameter won’t be parsed for attributes, and the macro function will be invoked with a single argument, a list containing the body expressions. The former is useful for what are essentially HTML templates. For example:

(define-html-macro :mytag (&attributes attrs &body body)
  `((:div :class "mytag" ,@attrs) ,@body))

HTML> (html (:mytag "Foo"))
<div class='mytag'>Foo</div>
NIL
HTML> (html (:mytag :id "bar" "Foo"))
<div class='mytag' id='bar'>Foo</div>
NIL
HTML> (html ((:mytag :id "bar") "Foo"))
<div class='mytag' id='bar'>Foo</div>
NIL

The latter kind of macro is more useful for writing macros that manipulate the forms in their body. This type of macro can function as a kind of HTML control construct. As a trivial example, consider the following macro that implements an :if construct:

(define-html-macro :if (test then else)
  `(if ,test (html ,then) (html ,else)))

This macro allows you to write this:

(:p (:if (zerop (random 2)) "Heads" "Tails"))

instead of this slightly more verbose version:

(:p (if (zerop (random 2)) (html "Heads") (html "Tails")))

To determine which kind of macro you should generate, you need a function that can parse the parameter list given to define-html-macro. This function returns two values, the name of the &attributes parameter, or NIL if there was none, and a list containing all the elements of args after removing the &attributes marker and the subsequent list element. [3]

(defun parse-html-macro-lambda-list (args)
  (let ((attr-cons (member '&attributes args)))
    (values
     (cadr attr-cons)
     (nconc (ldiff args attr-cons) (cddr attr-cons)))))

HTML> (parse-html-macro-lambda-list '(a b c))
NIL
(A B C)
HTML> (parse-html-macro-lambda-list '(&attributes attrs a b c))
ATTRS
(A B C)
HTML> (parse-html-macro-lambda-list '(a b c &attributes attrs))
ATTRS
(A B C)

The element following &attributes in the parameter list can also be a destructuring parameter list.

HTML> (parse-html-macro-lambda-list '(&attributes (&key x y) a b c))
(&KEY X Y)
(A B C)

Now you’re ready to write define-html-macro. Depending on whether there was an &attributes parameter specified, you need to generate one form or the other of HTML macro so the main macro simply determines which kind of HTML macro it’s defining and then calls out to a helper function to generate the right kind of code.

(defmacro define-html-macro (name (&rest args) &body body)
  (multiple-value-bind (attribute-var args)
      (parse-html-macro-lambda-list args)
    (if attribute-var
      (generate-macro-with-attributes name attribute-var args body)
      (generate-macro-no-attributes name args body))))

The functions that actually generate the expansion look like this:

(defun generate-macro-with-attributes (name attribute-args args body)
  (with-gensyms (attributes form-body)
    (if (symbolp attribute-args) (setf attribute-args `(&rest ,attribute-args)))
    `(eval-when (:compile-toplevel :load-toplevel :execute)
       (setf (get ',name 'html-macro-wants-attributes) t)
       (setf (get ',name 'html-macro)
             (lambda (,attributes ,form-body)
               (destructuring-bind (,@attribute-args) ,attributes
                 (destructuring-bind (,@args) ,form-body
                   ,@body)))))))

(defun generate-macro-no-attributes (name args body)
  (with-gensyms (form-body)
    `(eval-when (:compile-toplevel :load-toplevel :execute)
       (setf (get ',name 'html-macro-wants-attributes) nil)
       (setf (get ',name 'html-macro)
             (lambda (,form-body)
               (destructuring-bind (,@args) ,form-body ,@body)))))

The macro functions you’ll define accept either one or two arguments and then use DESTRUCTURING-BIND to take them apart and bind them to the parameters defined in the call to define-html-macro. In both expansions you need to save the macro function in the name’s property list under html-macro and a boolean indicating whether the macro takes an &attributes parameter under the property html-macro-wants-attributes. You use that property in the following function, expand-macro-form, to determine how the macro function should be invoked:

(defun expand-macro-form (form)
  (if (or (consp (first form))
          (get (first form) 'html-macro-wants-attributes))
    (multiple-value-bind (tag attributes body) (parse-cons-form form)
      (funcall (get tag 'html-macro) attributes body))
    (destructuring-bind (tag &body body) form
      (funcall (get tag 'html-macro) body))))

The last step is to integrate macros by adding a clause to the dispatching COND in the top-level process function.

(defun process (processor form)
  (cond
    ((special-form-p form) (process-special-form processor form))
    ((macro-form-p form)   (process processor (expand-macro-form form)))
    ((sexp-html-p form)    (process-sexp-html processor form))
    ((consp form)          (embed-code processor form))
    (t                     (embed-value processor form))))

This is the final version of process.

[3]Note that &attributes is just another symbol; there’s nothing intrinsically special about names that start with &.


038 - The End of the Line

Chapter 31 - Practical—An HTML Generation Library, the Compiler
Practical Common Lisp
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

The End of the Line

As usual, you could keep working with this code to enhance it in various ways. One interesting avenue to pursue is to use the underlying output generation framework to emit other kinds of output. In the version of FOO you can download from the book’s Web site, you’ll find some code that implements CSS output that can be integrated into HTML output in both the interpreter and compiler. That’s an interesting case because CSS’s syntax can’t be mapped to s-expressions in such a trivial way as HTML’s can. However, if you look at that code, you’ll see it’s still possible to define an s-expression syntax for representing the various constructs available in CSS.

A more ambitious undertaking would be to add support for generating embedded JavaScript. Done right, adding JavaScript support to FOO could yield two big wins. One is that after you define an s-expression syntax that you can map to JavaScript syntax, then you can start writing macros, in Common Lisp, to add new constructs to the language you use to write client-side code, which will then be compiled to JavaScript. The other is that, as part of the FOO s-expression JavaScript to regular JavaScript translation, you could deal with the subtle but annoying differences between JavaScript implementations in different browsers. That is, the JavaScript code that FOO generates could either contain the appropriate conditional code to do one thing in one browser and another in a different browser or could generate different code depending on which browser you wanted to support. Then if you use FOO in dynamically generated pages, it could use information about the User-Agent making the request to generate the right flavor of JavaScript for that browser.

But if that interests you, you’ll have to implement it yourself since this is the end of the last practical chapter of this book. In the next chapter I’ll wrap things up, discussing briefly some topics that I haven’t touched on elsewhere in the book such as how to find libraries, how to optimize Common Lisp code, and how to deliver Lisp applications.


039 - Chapter 32: Conclusion—What’s Next?

Chapter 32 - Conclusion—What’s Next?
Practical Common Lisp
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Chapter 32: Conclusion—What’s Next?

I hope by now you’re convinced that the title of this book isn’t an oxymoron. However, it’s quite likely there’s some area of programming that’s of great practical importance to you that I haven’t discussed at all. For instance, I haven’t said anything about how to develop graphical user interfaces (GUIs), how to connect to relational databases, how to parse XML, or how to write programs that act as clients for various network protocols. Similarly, I haven’t discussed two topics that will become important when you write real applications in Common Lisp: optimizing your Lisp code and packaging your application for delivery.

I’m obviously not going to cover all these topics in depth in this final chapter. Instead, I’ll give you a few pointers you can use to pursue whichever aspect of Lisp programming interests you most.

Finding Lisp Libraries

While the standard library of functions, data types, and macros that comes with Common Lisp is quite large, it provides only general-purpose programming constructs. Specialized tasks such as writing GUIs, talking to databases, and parsing XML require libraries beyond what are provided by the ANSI standardized language.

The easiest way to obtain a library to do something you need may be simply to check out your Lisp implementation. Most implementations provide at least some facilities not specified in the language standard. The commercial Common Lisp vendors tend to work especially hard at providing additional libraries for their implementation in order to justify their prices. Franz’s Allegro Common Lisp, Enterprise Edition, for instance, comes with libraries for parsing XML, speaking SOAP, generating HTML, connecting to relational databases, and building graphical interfaces in various ways, among others. LispWorks, another prominent commercial Lisp, provides several similar libraries, including a well-regarded portable GUI toolkit, CAPI, which can be used to develop GUI applications that will run on any operating system LispWorks runs on.

The free and open-source Common Lisp implementations typically don’t include quite so many bundled libraries, relying instead on portable free and open-source libraries. But even those implementations usually fill in some of the more important areas not addressed by the language standard such as networking and multithreading.

The only disadvantage of using implementation-specific libraries is that they tie you to the implementation that provides them. If you’re delivering end-user apps or are deploying a server-based application on a server that you control, that may not matter a lot. But if you want to write code to share with other Lispers or if you simply don’t want to be tied to a particular implementation, it’s a little more annoying.

For portable libraries—portable either because they’re written entirely in standard Common Lisp or because they contain appropriate read-time conditionalization to work on multiple implementations [1]—your best bet is to go to the Web. With the usual caveats about URLs going stale as soon as they’re printed on paper, these are three of the best current starting points:

  • Common-Lisp.net (http://www.common-lisp.net/) is a site that hosts free and open-source Common Lisp projects, providing version control, mailing lists, and Web hosting of project pages. In the first year and a half after the site went live, nearly a hundred projects were registered.

  • The Common Lisp Open Code Collection (CLOCC) (http://clocc.sourceforge.net/) is a slightly older collection of free software libraries, which are intended to be portable between Common Lisp implementations and self-contained, not relying on any libraries not included in CLOCC itself.

  • Cliki (http://www.cliki.net/) is a wiki devoted to free software in Common Lisp. While, like any wiki, it may change at any time, typically it has quite a few links to libraries as well to various open-source Common Lisp implementations. The eponymous software it runs on is also written in Common Lisp.

Linux users running the Debian or Gentoo distributions can also easily install an ever-growing number of Lisp libraries that have been packaged with those distributions’ packing tools, apt-get on Debian and emerge on Gentoo.

I won’t recommend any specific libraries here since the library situation is changing every day—after years of envying the library collections of Perl, Python, and Java, Common Lispers have, in the past couple of years, begun to take up the challenge of giving Common Lisp the set of libraries—both open source and commercial——that it deserves.

One area where there has been a lot of activity recently is on the GUI front. Unlike Java and C# but like Perl, Python, and C, there’s no single way to develop GUIs in Common Lisp. Instead, it depends both on what Common Lisp implementation you’re using and what operating system or systems you want to support.

The commercial Common Lisp implementations usually provide some way to build GUIs for the platforms they run on. Additionally, LispWorks provides CAPI, the previously mentioned, portable GUI API.

On the open-source side, you have a number of options. On Unix, you can write low-level X Windows GUIs using CLX, a pure-Common Lisp implementation of the X Windows protocol, roughly akin to xlib in C. Or you can use various bindings to higher-level APIs and toolkits such as GTK and Tk, much the way you might in Perl or Python.

Or, if you’re looking for something completely different, you can check out Common Lisp Interface Manager (CLIM). A descendant of the Symbolics Lisp Machines GUI framework, CLIM is powerful but complex. Although many commercial Common Lisp implementations actually support it, it doesn’t seem to have seen a lot of use. But in the past couple years, an open-source implementation of CLIM, McCLIM—now hosted at Common-Lisp.net—has been picking up steam lately, so we may be on the verge of a CLIM renaissance.

[1]The combination of Common Lisp’s read-time conditionalization and macros makes it quite feasible to develop portability libraries that do nothing but provide a common API layered over whatever API different implementations provide for facilities not specified in the language standard. The portable pathname library from Chapter 15 is an example of this kind of library, albeit to smooth over differences in interpretation of the standard rather than implementation-dependent APIs.


043 - Where to Go Next

Chapter 32 - Conclusion—What’s Next?
Practical Common Lisp
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Where to Go Next

So, that’s it. Welcome to the wonderful world of Lisp. The best thing you can do now—if you haven’t already—is to start writing your own Lisp code. Pick a project that interests you, and do it in Common Lisp. Then do another. Lather, rinse, repeat.

However, if you need some further pointers, this section offers some places to go. For starters, check out the Practical Common Lisp Web site at http://www.gigamonkeys.com/book/, where you can find the source code from the practical chapters, errata, and links to other Lisp resources on the Web.

In addition to the sites I mentioned in the “Finding Lisp Libraries” section, you may also want explore the Common Lisp HyperSpec (a.k.a. the HyperSpec or CLHS), an HTML version of the ANSI language standard prepared by Kent Pitman and made available by LispWorks at http://www.lispworks.com/documentation/HyperSpec/index.html. The HyperSpec is by no means a tutorial, but it’s as authoritative a guide to the language as you can get without buying a printed copy of the standard from ANSI and much more convenient for day-to-day use. [12]

If you want to get in touch with other Lispers, comp.lang.lisp on Usenet and the #lisp IRC channel or the Freenode network (http://www.freenode.net) are two of the main online hangouts. There are also a number of Lisp-related blogs, most of which are aggregated on Planet Lisp at http://planet.lisp.org/.

And keep your eyes peeled in all those forums for announcements of local Lisp users get-togethers in your area—in the past few years, Lispnik gatherings have popped up in cities around the world, from New York to Oakland, from Cologne to Munich, and from Geneva to Helsinki.

If you want to stick to books, here are a few suggestions. For a nice thick reference book to stick on your desk, grab The ANSI Common Lisp Reference Book edited by David Margolies (Apress, 2005). [13]

For more on Common Lisp’s object system, you can start with Object-Oriented Programming in Common Lisp: A Programmer’s Guide to CLOS by Sonya E. Keene (Addison-Wesley, 1989). Then if you really want to become an object wizard or just to stretch your mind in interesting ways, read The Art of the Metaobject Protocol by Gregor Kiczales, Jim des Riviéres, and Daniel G. Bobrow (MIT Press, 1991). This book, also known as AMOP, is both an explanation of what a metaobject protocol is and why you want one and the de facto standard for the metaobject protocol supported by many Common Lisp implementations.

Two books that cover general Common Lisp technique are Paradigms of Artificial Intelligence Programming: Case Studies in Common Lisp by Peter Norvig (Morgan Kaufmann, 1992) and On Lisp: Advanced Techniques for Common Lisp by Paul Graham (Prentice Hall, 1994). The former provides a solid introduction to artificial intelligence techniques while teaching quite a bit about how to write good Common Lisp code, and the latter is especially good in its treatment of macros.

If you’re the kind of person who likes to know how things work down to the bits, Lisp in Small Pieces by Christian Queinnec (Cambridge University Press, 1996) provides a nice blend of programming language theory and practical Lisp implementation techniques. While it’s primarily focused on Scheme rather than Common Lisp, the same principles apply.

For folks who want a little more theoretical look at things—or who just want to know what it’s like to be a freshman comp sci student at M.I.T.—Structure and Interpretation of Computer Programs, Second Edition, by Harold Abelson, Gerald Jay Sussman, and Julie Sussman (M.I.T. Press, 1996) is a classic computer science text that uses Scheme to teach important programming concepts. Any programmer can learn a lot from this book—just remember that there are important differences between Scheme and Common Lisp.

Once you’ve wrapped your mind around Lisp, you may want to place it in a bit of context. Since no one can claim to really understand object orientation who doesn’t know something about Smalltalk, you might want to start with Smalltalk-80: The Language by Adele Goldberg and David Robson (Addison Wesley, 1989), the standard introduction to the core of Smalltalk.

After that, Smalltalk Best Practice Patterns by Kent Beck (Prentice Hall, 1997) is full of good advice aimed at Smalltalkers, much of which is applicable to any object-oriented language. And at the other end of the spectrum, Object-Oriented Software Construction by Bertrand Meyer (Prentice Hall, 1997) is an excellent exposition of the static language mind-set from the inventor of Eiffel, an oft-overlooked descendant of Simula and Algol. It contains much food for thought, even for programmers working with dynamic languages such as Common Lisp. In particular, Meyer’s ideas about Design By Contract can shed a lot of light on how one ought to use Common Lisp’s condition system.

Though not about computers per se, The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies, and Nations by James Surowiecki (Doubleday, 2004) contains an excellent answer to the question, “If Lisp’s so great how come everybody isn’t using it?” See the section on “Plank-Road Fever” starting on page 53.

And finally, for some fun, and to learn about the influence Lisp and Lispers have had on hacker culture, dip into (or read from cover to cover) The New Hacker’s Dictionary, Third Edition, compiled by Eric S. Raymond (MIT Press, 1996) and based on the original The Hacker’s Dictionary edited by Guy Steele (Harper & Row, 1983).

But don’t let all these suggestions interfere with your programming—the only way to really learn a language is to use it. If you’ve made it this far, you’re certainly ready to do that. Happy hacking!

[12]SLIME incorporates an Elisp library that allows you to automatically jump to the HyperSpec entry for any name defined in the standard. You can also download a complete copy of the HyperSpec to keep locally for offline browsing.

[13]Another classic reference is Common Lisp: The Language by Guy Steele (Digital Press, 1984 and 1990). The first edition, a.k.a. CLtL1, was the de facto standard for the language for a number of years. While waiting for the official ANSI standard to be finished, Guy Steele—who was on the ANSI committee—decided to release a second edition to bridge the gap between CLtL1 and the eventual standard. The second edition, now known as CLtL2, is essentially a snapshot of the work of the standardization committee taken at a particular moment in time near to, but not quite at, the end of the standardization process. Consequently, CLtL2 differs from the standard in ways that make it not a very good day-to-day reference. It is, however, a useful historical document, particularly because it includes documentation of some features that were dropped from the standard before it was finished as well as commentary that isn’t part of the standard about why certain features are the way they are.


044 -

Index
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Index

Special Characters
&allow-other-keys parameter, 194
&attributes parameter, 423, 459–61
&body parameter, 94
&key parameter, 56–58, 61, 94, 161, 194, 208, 283, 320, 330
&optional parameter, 54, 56–57, 61, 94, 161, 283
&rest parameter, 55–58, 61, 93–94, 161, 194, 283, 320, 360
&whole parameter, 162, 463
* directive, 230
~$ directive, 221, 225–26
~% directive, 220, 223
~% FORMAT directive, 438
~& directive, 223
~& FORMAT directive, 438
~* directive, 230
~:* directive, 230
~* directive, 231
~@{ directive, 229–30
~@C modifier, 223
~] directive, 228
~^ directive, 228–29
~{ directive, 228–30
~: directive, 230
+ function, 55, 65, 103, 119
+ operator, 199
+ symbol, 13, 41, 117
+null+ constant, 359
<print> element, 455
= function, 121
/= function, 121
= operator, 75
> function, 121–22
>= function, 121, 123
1- function, 120
~( directive, 227
~? directive, 231
~/ directive, 231
1+ function, 120
/ function, 119
/ sign, 117

A

~a directive, 22, 229
~A directive, 222–24
A element, 423
:absolute keyword, 169–70
abstract class, 323
abstract syntax tree, 38
abstraction, leaky, 96–100
accessor functions, 209–11
:accessor option, 211, 214, 235, 295
ACL-SOCKET package, 412
ACONS, 158
ACOS, 122
ACOSH, 122
action parameter, 427
:action parameter, 428
add-record function, 21
ADJOIN, 156
:adjustable keyword, 129
aesthetic directive, 22
:after auxiliary method, 197–98
:after method, 420
AI (Artificial Intelligence), 1, 5
Algol, 37, 41
Allegro Common Lisp, 10, 469, 477
:allegro symbol, 180
AllegroServe, 363, 374, 382, 403, 405–6, 414, 423, 430
cookies, 377–78
generating dynamic content with, 368–69
generating HTML, 370–73
HTML macros, 373
overview, 365–68
query parameters, 374–77
small framework on top of, 379–80
:allocation option, 213–15
American National Standards Institute (ANSI), 5, 10, 465, 478
AND macro, 82–83
AND operator, 82–83, 106, 113, 199
anniversary, tenth, 224
anonymous functions, 61–63, 68
Another System Definition Facility (ASDF), 475–76
ANSI (American National Standards Institute), 5, 10, 465, 478
ant tool, 475
AOP (Aspect-Oriented Programming), 4
Apache Web server, 366
API (Application Programming Interface), 57, 397
APPEND, 111, 145–47, 199, 284
Application Programming Interface (API), 57, 397
applications, delivering, 475–77
APPLY, 60–61, 252
apt-get tool, 466
AREF function, 76, 125
arithmetic operations, basic, 119–20
:around auxiliary method, 197–98
arrays, 115, 127
Artificial Intelligence (AI), 1, 5
ASCII, 60, 122, 223, 314–15, 339, 391, 402, 435
ASD file, 476
ASDF (Another System Definition Facility), 475–76
ASIN function, 122
ASINH function, 122
AspectL library, 4
Aspect-Oriented Programming (AOP), 4
ASSOC function, 157–59, 330, 377
association lists, 157
associative arrays, 127
ASSOC-IF function, 158
ASSOC-IF-NOT, 158
ATAN, 122
ATANH, 122
ATOM, 151
atoms element, 39
:attribute operator, 457
automated testing framework, 103
auxiliary methods, 197–98


046 - C

Index
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Index

C

~:C directive, 223
~C directive, 223, 224
CAAR, 150
CADR, 150
call stack, control over behavior of, 252–56
CALL-NEXT-METHOD, 195–99, 217, 247, 332
CAPI toolkit, 465
CAR function, 142–43, 145–47, 150, 153, 157
Carnegie Mellon University (CMU), 1, 10
Cascading Style Sheet (CSS), 423, 430, 463
CASE construct, 296
CATCH operator, 254–55
Catch tag, 254
catchall parameter, 55
C-c RET, 96
CD and record database example (loading database), 25–26
CDR, 142–43, 145–47, 149–50, 153, 157–58, 280, 377
CDs and records database (example)
database contents, 21–23
filing CDs, 21
improving user interaction, 23–25
overview, 19–20
querying, 27–31
removing duplication, 32–36
saving and loading database, 25–26
updating existing records, 31–32
CEILING, 120
CERROR, 242
CHAR, 123–25
CHAR=, 46, 122
character comparisons, 122–23
character directives, 223–25
character escaping, FOO HTML generator, 435–37
character reference entities, 435
CHARACTER symbol, 129
CHAR-CODE, 314–15, 329, 339
CHAR-EQUAL, 122
chunked transfer-encoding, 406
class precedence list, 215
class specializers, 192
classes
accessor functions, 209–11
class-allocated slots, 213–14
DEFCLASS, 203–4
and generic functions, 190–91
good object-oriented design, 218
multiple inheritance, 215–17
object initialization, 206–8
overview, 203
slot specifiers, 205
slots and inheritance, 214–15
WITH-SLOTS and WITH-ACCESSORS, 212–13
CLHS (Common Lisp HyperSpec), 478
Cliki wiki, 466
CLIM (Common Lisp Interface Manager), 467
CLISP, 185
:clisp symbol, 180
clisp-subdirectories-wildcard function, 185
CLOCC (Common Lisp Open Code Collection), 466
CLOS (Common Lisp Object System), 4–5, 20, 189
CLOSE function, 163, 167–68
closing files, 167–68
closures, 68–69
CL-PPCRE (Common Lisp Portable Perl-Compatible Regular Expression), 293
CL-PPCRE regular expression library, 468
CLRHASH, 139
CL-USER package, 266, 274
CMU (Carnegie Mellon University), 1, 10
:cmu symbol, 180
CMUCL, 10, 184
code, formatting, 47–49.
See also syntax and semantics
code sketch, 468
code-bumming, 471
CODE-CHAR function, 314–15, 329, 339
collecting keyword, 278
collections.
See also functions
hash tables, 138–40
looping over, 280–81
overview, 127
sequence iterating functions, 130
vectors, 127–29
column-matcher function, 395
column-matchers function, 395
:columns keyword parameter, 392
column-value function, 397
COM.ACME.EMAIL package, 271
COM.ACME.TEXT package, 271–72
COM.GIGAMONKEYS.EMAIL-DB package, 269, 271–72
COM.GIGAMONKEYS.MACRO-UTILITIES package, 318
COM.GIGAMONKEYS.TEXT-DB package, 270–72
comments, 39
Common Lisp, 1–8, 13, 19
Common Lisp HyperSpec (CLHS), 478
Common Lisp Interface Manager (CLIM), 467
Common Lisp macro, 449
Common Lisp Object System (CLOS), 4–5, 20, 189
Common Lisp Open Code Collection (CLOCC), 466
Common Lisp Portable Perl-Compatible Regular Expression (CL-PPCRE), 293
COMMON-LISP package, 266–71, 275, 291, 436
Common-Lisp.net, 466
COMMON-LISP-USER package, 266–69, 274–75
comparator function, 387
comparator slot, 387
COMPILATION-SPEED symbol, 473
compile-buffer function, 451
COMPILE-FILE, 17, 44, 258–59, 261, 457, 475
compiler, FOO, 449–54
complex numbers, 119
composite structures, 316–17
CONCATENATE, 134–35, 137
CONCATENATED-STREAM, 176
COND macro, 82, 461
CONDITION class, 235
condition classes, 235
condition handlers, 235–37.
See also restarts
condition system, 5, 233
Conditional clause, 286
conditional execution, LOOP macro, 285–87
conditional formatting, 227–28
conditions, 235–37
CONS, 142, 156, 158
cons cells, 142–44
uses of besides lists
lookup tables, 157–61
overview, 153
sets, 155–57
trees, 153–55
CONSP, 151
constants, 74
:content-type argument, 405
CONTINUE, 242
CONTROL-ERROR, 240, 242
controlling evaluation, 245
cookies, 365, 377–78
COPY-ALIST, 159
COPY-LIST, 149, 154
COPY-SEQ, 134
COPY-TREE, 154–55, 159
core, 477
COS, 122
COSH, 122
COUNT, 131–32
:count keyword parameter, 132–33
counting keyword, 278
CRC (Cyclic Redundancy Check), 348
CSS (Cascading Style Sheet), 423, 430
current-idx slot, 415–17, 420
Cyclic Redundancy Check (CRC), 348


048 - E

Index
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Index

E

~E directive, 225–26
ECASE, 296, 306, 331
ECHO-STREAM, 176
:element-type keyword argument, 129, 312
ELT, 125, 130
Emacs, 9, 11
embed-code function, 440, 450, 454–55
embed-value function, 440, 450, 454–55
:embed-value method, 453
emerge tool, 466
emit function, 437
emit-attributes, 445–46
emit-close-tag, 445–46
emit-element-body, 445
emit-freshline, 438
emit-html, 431, 455
emit-newline, 438
emit/no-newlines, 437, 439
emit-open-tag, 445
encode-form-urlencoded, 423
ENCODE-UNIVERSAL-TIME, 377
:end argument, 391
:end keyword parameter, 133, 136, 165, 175, 304
English-language directives, 226–27
ENOUGH-NAMESTRING, 172
ENSURE-DIRECTORIES-EXIST, 174
entities, 366
entity argument, 405
EQ, 46, 138, 159
EQL, 47, 121, 138, 192, 194–97, 296, 320, 388–89, 404, 452
EQUAL, 46–47, 138, 385, 413
equality, 45–47
equality-predicate function, 387
equality-predicate slot, 387
EQUALP, 46–47, 121, 138
equals-then clause, LOOP macro, 281–82
equivalence function, 138
ERROR, 235, 242
EVAL, 441–42
evaluation, 245
evaluation, controlling, 245
evaluator, 38
EVAL-WHEN, 258–60, 326, 457, 462
EVENP, 27, 121
EVERY, 137, 396
Evins, Mikel, 9
exceptions, 234
excl:generate-application function, 477
EXP, 60–61, 122, 301
expand-macro-form, 461
exponent markers, 118
:export clause, 270
EXPT, 122
ext package, 185
eXtensible Markup Language (XML), 144, 465
extractor function, 393


052 - I

Index
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Index

I

Icy-Metadata header, 402
ID3 parsing
comment frames, 356–57
defining a package, 337–38
detecting tag padding, 346–48
extracting information from ID3 tag, 357–62
ID3 frames, 344–46
ID3 tag header, 343–44
integer types, 338–39
overview, 335–36
string types, 339–43
structure of ID3v2 tag, 336–37
supporting multiple version of ID3, 348–50
text information frames, 354–55
versioned concrete frame classes, 351–52
versioned frame base classes, 350–51
which frames are needed, 352–54
ID3v2 format, 336
ID3v2 library, 385, 391
ID3v2.2 specification, 343
IEC (International Elecrotechnical Commission), 335
IF construct, 285
:if construct, 460
IF operator, 43, 48, 80–82, 245
:if-does-not-exist keyword argument, 164
:if-exists keyword argument, 166
IF-NOT suffix, 133–34
IGNORE-ERRORS, 186
image file, 477
:import-from clause, 271–72
in function, 392, 394, 396, 417
INCF, 77, 120, 295
indent function, 440
indenting-printer, 437–38, 447
indenting-printer API, 438–40
inheritance
multiple, 215–17
and slots, 214–15
and tagged structures, binary file parsing, 323–24
inherited slots, keeping track of, 325–27
in-html-style macro, 462
:initarg option, 206–7, 213–14
:initform option, 206–7, 213–14
INITIALIZE-INSTANCE, 206–8, 235
initially keyword, 287
IN-PACKAGE, 258–60, 269–70, 272–73, 302, 319, 366
INPUT element, 421
INPUT field, 425
insert-row, 391
INSTALL.lisp file, 366
INTEGER, 203
integer directives, 223–25
integers, 116
interation directive, 228–30
INTERN, 264
INTERNAL-TIME-UNITS-PER-SECOND, 470
International Elecrotechnical Commission (IEC), 335
International Organization for Standardization (ISO), 335
INTERSECTION, 156
INVOKE-RESTART, 239–41
IP address, 414
ipaddr-to-dotted, 414
IRC channel, 478
ISO 639-2, 356
ISO 8859-1, 314, 316–17, 329, 336, 339–41, 345, 354, 356
ISO (International Organization for Standardization), 335
iteration control, LOOP macro, 278


057 - N

Index
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Index

N

name component, 169
named keyword, 288
NAMESTRING, 170
namestrings, 168
naming conventions, 52
NBUTLAST, 151
NCONC, 147, 152, 199, 284, 305
NET.ASERVE package, 366
NET.ASERVE.CLIENT package, 366
NET.HTML.GENERATOR package, 366
newline function, 440
:newline operator, 455, 458
next-metadata, 408
Nilsson, Martin, 336
:noescape operator, 455–57
nondestructive list functions, 148
nonkeyword symbols, 269
normalize-for-column, 390
normalize-row, 390
NOT, 82–83
NOTANY, 137
NOTEVERY, 137
NOT-GREATERP, 123
NOT-LESSP, 123
not-nullable, 389
NRECONC, 151
NREVERSE, 146–48, 305
nshuffle-vector, 399
NSUBST, 155
NSUBST-IF, 155
NSUBST-IF-NOT, 155
NSUBSTITUTE, 147, 155
NTH, 150
NTHCDR, 150
NULL, 151
NUMBER class, 190
numbers, 115–16
numeric comparisons, 121
numeric literals, 117–19
NUNION, 353


060 - Q

Index
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Index

Q

query parameters, 365
AllegroServe, 374–77
MP3 browser, 420–22
querying
database (CD and record database example), 27–31
MP3 database, 392–94
QUOTE, 43–44, 46, 245


061 - R

Index
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Index

R

~R directive, 224–26, 230
RANDOM, 97, 250, 376
random argument, 426
random-sample, 399
RASSOC, 158
RASSOC-IF, 158
RASSOC-IF-NOT, 158
rational numbers, 117–18
raw-string, 439, 449
READ, 163–66, 175, 222, 475
READ-*, 163
READ-BYTE, 165, 312, 409
READ-CHAR, 163–65, 175
:reader option, 210, 214, 235
*READ-EVAL* variable, 421
read-eval-print-loop. See REPL
READ-FROM-STRING, 421
reading
binary data, 165
binary objects, 320–22
bulk reads, 165
file data, 163–65
READ-LINE, 23, 163–64, 175
READ-SEQUENCE, 165–66, 306, 409
read-time conditionalization, 180
records and CDs database. See CDs and records database (example)
recycling functions, 147–49
recycling operation, 145
REDUCE, 138, 301
refactoring, 105–6
:relative keyword, 169–70
REM, 120, 160
REMHASH, 139–40
remote-host, 414
REMOVE, 131–34, 147–49
REMOVE-DUPLICATES, 134
REMOVE-IF, 32
REMOVE-IF-NOT, 27–28
REMPROP, 160
RENAME-FILE, 174
REPL (read-eval-print-loop), 40, 42, 96, 101, 103, 112, 117, 124, 266–67, 269, 273, 275, 292, 295, 302, 319, 348, 352, 367, 404, 473
experimenting in, 12–13
Hello, World example, 13–14
overview, 12
reply-header-slot-value, 406
request argument, 405
request parameter, 427
request-reply-protocol-string, 406
request-socket, 414
request-uri, 406
resizable vectors, 127
REST, 143, 150
rest parameters, 55
RESTART-CASE, 238
restarts, 238–41
RETURN, 84–85, 87, 249, 285, 288
return values, 58–59
RETURN-FROM, 52, 58–59, 248–49, 252–55, 285, 288
reusable libraries, packaging, 270–71
REVAPPEND, 151
REVERSE, 33, 134, 144, 146
ripped keyword parameter, 30
Robinson, Gary, 297
ROTATEF, 77–78
ROUND, 120
row-comparator, 393–94
row-equality-tester, 393–94
rows vector, 397
runtime vs. macro expansion time, 90–91


062 - S

Index
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

Index

S

~S directive, 222–24
SAFETY symbol, 473
save-buffer command, 16
save-db, 25–26
saving database (CD and record database example), 25–26
SBCL, 10, 184–85
:sbcl symbol, 180
:schema initarg, 390
schema slot, 387
schemas, defining (MP3 database), 388–90
SEARCH, 136
SECOND, 150
SELECT, 392
select function, 392–94, 397–98, 424
self-evaluating forms, 444
self-evaluating objects, 12, 42
self-evaluating-p function, 433
SEND, 191
sequence functions, 130–32, 137–38
sequence predicates, 137
sequences, vectors as, 130
server-side Web programming, 363–65.
See also AllegroServe
set-cookie-code, 383
set-cookie-header, 377
SET-DIFFERENCE, 156
SET-EXCLUSIVE-OR, 156
SETF, 26, 31, 74–78, 140, 146–47, 159, 194, 205, 209–11, 312–13, 321, 325, 397, 457
:setgid parameter, 366
SETQ, 74, 246
sets, 155–57
:setuid, 366
sexp->ops, 449, 452
s-expressions, 38–42, 80, 163, 166, 170, 220
:shadow clause, 271–72
shadowing symbol, 265
:shadowing-import-from clause, 272
shared structure, and recycling functions, 147–49
SHIFTF, 77–78
Shoutcast server, 411, 414–15
implementing, 405–10
protocol, 401–2, 405
song sources, 402–4
show-timing-data, 471
shuffle-vector, 305–6
SIGNAL, 236, 241–42
Simple Object Access Protocol (SOAP), 465
SIN, 122
singleton symbol, 404
singly linked list, 142
SINH, 122
SLIME, 9, 11, 15–17, 47, 94, 96, 224, 270, 273, 275, 292, 302
slime-close-parens-at-point, 15
slime-compile-defun, 15
slime-macroexpand-1, 96
slots
class-allocated, 213–14
and inheritance, 214–15
specifiers, 205
WITH-SLOTS, 212–13
SLOT-VALUE, 205–6, 209, 212–13, 235
Smalltalk-80, 479
SOAP (Simple Object Access Protocol), 465
SOME, 137
SORT, 135, 149
sorting, 135
sort-rows, 419
source parameter, 414
SPACE symbol, 473
spam filters
analyze results generated by test-classifier, 306–9
combining probabilities, 299–301
creating, 291–95
of inverse-chi-square, 301–2
overview, 291
per-word statistics, 298
shuffle-vector function, 305–6
start-of-file function, 306
testing, 303–5
training, 295–97, 302
SpamAssassin, 303
special forms, 42
special operators, 43–44, 454
behavior of call stack, control over, 252–56
controlling evaluation, 245
EVAL-WHEN, 258–60
FOO, 454–59
local flow of control, 248–52
manipulating lexical environment, 246–48
multiple values, 256–58
overview, 245
special variables. See dynamic variables
specialized vectors, 129
SPEED symbol, 473
Spolsky, Joel, 96
SQL (Structured Query Language), 392
STABLE-SORT function, 135, 149
STANDARD-OBJECT class, 204–5, 207, 215, 235
*STANDARD-OUTPUT* stream, 220
:start keyword, 133, 136, 165, 175, 304
start-of-file function, 306
Steele, Guy, 5
still-current-p function, 415
STRING, 124, 134, 190, 203, 387–89
string comparisons, 124–25
string data type, 389
string designators, 268
STRING=, 125, 387–89
string->type, 420
STRING-EQUAL, 125
strings, 115, 123–24, 129, 314–16
STRING-STREAM, 315
strongly typed language, 65
Structured Query Language (SQL), 392
SUBSEQ, 136
subsequence manipulations, 136
SUBSETP, 157
SUBST, 155
SUBST-IF, 155
SUBST-IF-NOT, 155
SUBSTITUTE, 131–33, 147, 155
SUBSTITUTE-IF, 155
summing keyword, 278
supplied-p, 30
switch-to-buffer, 11
Symbol macro, 246
Symbolics Lisp Machines GUI framework, 467
SYMBOL-MACROLET, 246, 248, 259
SYMBOL-PLIST, 160, 325
symbols, 40
accessibility of, 265
importing into other packages, 265
nonkeyword symbols, 269
shadowing symbols, 265
terminology, 265–66
uninterned symbols, 266
syntax and semantics
equality, 45–47
formatting code, 47–49
function calls, 42–43
how defined, 38–39
macros, 44–45
overview, 37
parentheses, 37
s-expressions, 39–42
special operators, 43–44
truth and falsehood, 45–47
system definition facilities, 475


070 - List of Figures

List of Figures
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

List of Figures

Chapter 26: Practical—Web Programming with AllegroServe

Figure 26-1: Sample Web page
Figure 26-2: http://localhost:2001/hello.html
Figure 26-3: http://localhost:2001/random-number
Figure 26-4: http://localhost:2001/show-query-params?foo=bar&baz=10
Figure 26-5: http://localhost:2001/simple-form
Figure 26-6: Result of submitting the simple form
Figure 26-7: http://localhost:2001/show-cookies with no cookies
Figure 26-8: http://localhost:2001/set-cookie
Figure 26-9: http://localhost:2001/show-cookies after setting a cookie


071 - List of Tables

List of Tables
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator
Next Section

List of Tables

Chapter 6: Variables

Table 6-1: Assignment with = in Other Languages

Chapter 8: Macros—Defining Your Own

Table 8-1: Backquote Examples

Chapter 10: Numbers, Characters, and Strings

Table 10-1: Character Comparison Functions
Table 10-2: Literal Strings
Table 10-3: String Comparison Functions

Chapter 11: Collections

Table 11-1: Basic Sequence Functions
Table 11-2: Standard Sequence Function Keyword Arguments

Chapter 12: They Called It LISP for a Reason—List Processing

Table 12-1: Other List Functions

Chapter 30: Practical—An HTML Generation Library, the Interpreter

Table 30-1: FOO Output for Self-Evaluating Objects


072 - List of Sidebars

List of Sidebars
by Peter Seibel
Apress © 2005
Previous Section
Progress Indicator Progress Indicator

List of Sidebars

Chapter 1: Introduction—Why Lisp?

BUT I LEARNED LISP ONCE, AND IT WASN’T LIKE WHAT YOU’RE DESCRIBING

Chapter 8: Macros—Defining Your Own

ANOTHER CLASSIC MACRO-WRITING MACRO: ONCE-ONLY

Chapter 14: Files and File I/O

HOW WE GOT HERE

Chapter 15: Practical—A Portable Pathname Library

PACKAGING THE LIBRARY

Chapter 16: Object Reorientation—Generic Functions

MULTIMETHODS VS. METHOD OVERLOADING

Chapter 17: Object Reorientation—Classes

WHAT ARE “USER-DEFINED CLASSES”?

Chapter 19: Beyond Exception Handling—Conditions and Restarts

JAVA-STYLE EXCEPTON HANDLING
WRITING ROBUST SOFTWARE

Chapter 27: Practical—An MP3 Database

THE PACKAGE

Chapter 28: Practical—A Shoutcast Server

THE PACKAGE

Chapter 29: Practical—An MP3 Browser

THE PACKAGE

Chapter 30: Practical—An HTML Generation Library, the Interpreter

THE PACKAGE
USING CONDITIONS TO HAVE YOUR CAKE AND EAT IT TOO