Lisp for Web-Based Applications
After a link to Beating the Averages was posted on slashdot, some readers wanted to hear in more detail about the specific technical advantages we got from using Lisp in Viaweb. For those who are interested, here are some excerpts from a talk I gave in April 2001 at BBN Labs in Cambridge, MA.
Lisp in Web-Based Applications
Paul Graham
(This is an excerpt of a talk given at BBN Labs in Cambridge, MA, in April 2001.)
Any Language You Want
One of the reasons to use Lisp in writing Web-based applications is that you can use Lisp. When you're writing software that is only going to run on your own servers, you can use whatever language you want.
For a long time programmers didn't have a lot of choice about what language to use for writing application programs. Until recently, writing application programs meant writing software to run on desktop computers.
In desktop software there was a strong bias toward writing the application in the same language as the operating system. Ten years ago, for all practical purposes, applications were written in C.
With Web-based applications, that changes. You control the servers, and you can write your software in any language you want. You can take it for granted now that you have the source code of both your operating system and your compilers. If there does turn out to be any kind of problem between the language and the OS, you can fix it yourself.
This new freedom is a double-edged sword, however. Having more choices means that you now have to think about which choice to make. It was easier in the old days. If you were in charge of a software project, and some troublesome person suggested writing the software in a different language from whatever you usually used, you could just tell them that it would be impractical, and that would be the end of it.
Now, with server-based applications, everything is changed.
You're now subject to market forces in what language you choose. If you try to pretend that nothing has changed, and just use C and C++, like most of our competitors did, you are setting yourself up for a fall. A little startup using a more powerful language will eat your lunch.
Incremental Development
There is a certain style of software development associated with Lisp. One of its traditions is incremental development: you start by writing, as quickly as possible, a program that does almost nothing. Then you gradually add features to it, but at every step you have working code.
I think this way you get better software, written faster. Everything about Lisp is tuned to this style of programming, because Lisp programmers have worked this way for at least thirty years.
The Viaweb editor must be one of the most extreme cases of incremental development. It began with a 120-line program for generating Web sites that I had used in an example in a book that I finished just before we started Viaweb.
The Viaweb editor, which eventually grew to be about 25,000 lines of code, grew incrementally from this program. I never once sat down and rewrote the whole thing. I don't think I was ever more than a day or two without running code. The whole development process was one long series of gradual changes.
This style of development fits well with the rolling releases that are possible with Web-based software. It's also a faster way to get software written generally.
Interactive Toplevel
Lisp's interactive toplevel is a great help in developing software rapidly. But the biggest advantage for us was probably in finding bugs. As I mentioned before, with Web-based applications you have the users' data on your servers and can usually reproduce bugs.
When one of the customer support people came to me with a report of a bug in the editor, I would load the code into the Lisp interpreter and log into the user's account.
If I was able to reproduce the bug I'd get an actual break loop, telling me exactly what was going wrong. Often I could fix the code and release a fix right away. And when I say right away, I mean while the user was still on the phone.
Such fast turnaround on bug fixes put us into an impossibly tempting position. If we could catch and fix a bug while the user was still on the phone, it was very tempting for us to give the user the impression that they were imagining it. And so we sometimes (to their delight) had the customer support people tell the user to just try logging in again and see if they still had the problem. And of course when the user logged back in they'd get the newly released version of the software with the bug fixed, and everything would work fine. I realize this was a bit sneaky of us, but it was also a lot of fun.
Macros for Html
Lisp macros were another big win for us. We used them very extensively in the Viaweb editor. It could accurately be described as one big macro.
And that gives you an idea of how much we depended on Lisp, because no other language has macros in the sense that Lisp does.
One way we used macros was to generate Html. There is a very natural fit between macros and Html, because Html is a prefix notation like Lisp, and Html is recursive like Lisp. So we had macro calls within macro calls, generating the most complicated Html, and it was all still very manageable.
Embedded Languages
Another big use for macros was the embedded language we had for describing pages, called Rtml. (We made up various explanations for what Rtml was supposed to stand for, but actually I named it after Robert Morris, the other founder of Viaweb, whose username is Rtm.)
Every page made by our software was generated by a program written in Rtml. We called these programs templates to make them less frightening, but they were real programs. In fact, they were Lisp programs.
Rtml was a combination of macros and the built-in Lisp operators.
Users could write their own Rtml templates to describe what they wanted their pages to look like. We had a structure editor for manipulating these templates, a lot like the structure editor they had in Interlisp. Instead of typing free-form text, you cut and pasted bits of code together. This meant that it was impossible to get syntax errors. It also meant that we didn't have to display the parentheses in the underlying s-expressions: we could show structure by indentation. By this means we made the language look a lot less threatening.
We also designed Rtml so that there could be no errors at runtime: every Rtml program yielded some kind of Web page, and you could debug it by hacking it until it produced the page you meant it to.
Initially we expected our users to be Web consultants, and we expected them to use Rtml a lot.
We provided some default templates for section pages and item pages and so on, and the idea was that the users could take them and modify them to make whatever pages they wanted.
In fact it turned out that Web consultants didn't like Viaweb. Consultants, as a general rule, like to use products that are too hard for their clients to use, because it guarantees them ongoing employment. Consultants would come to our Web site, which said all over it that our software was so easy to use that it would let anyone make an online store in five minutes, and they'd say, there's no way we're using that. So we didn't get a lot of interest from Web consultants. Instead the users all tended to be end-users, the actual merchants themselves. They loved the idea of being in control of their own Web sites. And this kind of user did not want to do any kind of programming. They just used the default templates.
So Rtml didn't end up being the main interface to the program. It ended up playing two roles.
First of all, it was an escape valve for the really sophisticated users, who wanted something our built-in templates couldn't provide. Somewhere in the course of doing Viaweb, someone gave me a very useful piece of advice: users always want an upgrade path, even though as a rule they'll never take it. Rtml was our upgrade path. If you wanted to, you could get absolute control over everything on your pages.
Only one out of every couple hundred users actually wrote their own templates. And this led to the second advantage of Rtml. By looking at the way these users modified our built-in templates, we knew what we needed to add to them. Eventually we made it our goal that no one should ever have to use Rtml. Our built-in templates should do everything people wanted. In this new approach, Rtml served us as a warning sign that something was missing in our software.
The third and biggest win from using Rtml was the advantage we ourselves got from it.
Even if we had been the only people who used Rtml, it would have been very much worth while writing the software that way. Having that extra layer of abstraction in our software gave us a big advantage over competitors. It made the design of our software much cleaner, for one thing. Instead of just having bits of actual C or Perl code that generated our Web pages, like our competitors, we had a very high-level language for generating Web pages, and our page styles specified in that. It made the code much cleaner and easier to modify. I've already mentioned that Web-based applications get released as a series of many small modifications. When you do that you want to be able to know how serious any given modification is. By dividing your code into layers, you get a better handle on this. Modifying stuff in lower layers (Rtml itself) was a serious matter to be done rarely, and after much thought.
Whereas modifying the top layers (template code) was something you could do quickly without worrying too much about the consequences.
Rtml was a very Lispy proposition. It was mostly Lisp macros, to start with. The online editor was, behind the scenes, manipulating s-expressions. And when people ran templates, they got compiled into Lisp functions by calling compile at runtime.
Rtml even depended heavily on keyword parameters, which up to that time I had always considered one of the more dubious features of Common Lisp. Because of the way Web-based software gets released, you have to design the software so that it's easy to change. And Rtml itself had to be easy to change, just like any other part of the software. Most of the operators in Rtml were designed to take keyword parameters, and what a help that turned out to be. If I wanted to add another dimension to the behavior of one of the operators, I could just add a new keyword parameter, and everyone's existing templates would continue to work.
A few of the Rtml operators didn't take keyword parameters, because I didn't think I'd ever need to change them, and almost every one I ended up kicking myself about later. If I could go back and start over from scratch, one of the things I'd change would be that I'd make every Rtml operator take keyword parameters.
We had a couple embedded languages within the editor, in fact. Another one, which we didn't expose directly to the users, was for describing images. Viaweb included an image generator, written in C, that could take a description of an image, create that image, and return its url. We used s-expressions to describe these images as well.
Closures Simulate Subroutines
One of the problems with using Web pages as a UI is the inherent statelessness of Web sessions. We got around this by using lexical closures to simulate subroutine-like behavior.
If you understand about continuations, one way to explain what we did would be to say that we wrote our software in continuation-passing style.
When most web-based software generates a link on a page, it tends to be thinking, if the user clicks on this link, I want to call this cgi script with these arguments. When our software generated a link, it could think, if the user clicks on this link, I want to run this piece of code. And the piece of code could an arbitrary piece of code, possibly (in fact, usually) containing free variables whose value came from the surrounding context.
The way we did this was to write a macro that took an initial argument expected to be a closure, followed by a body of code. The code would then get stored in a global hash table under a unique id, and whatever output was generated by the code in the body would appear within a link whose url contained that hash key.
If that link was the next one clicked on, our software would find and call the corresponding bit of code, and the chain would continue. Effectively we were writing cgi scripts on the fly, except that they were closures that could refer to the surrounding context.
So far this sounds very theoretical, so let me give you an example of where this technique made an obvious difference. One of the things you often want to do in Web-based applications is edit an object with various types of properties. Many of the properties of an object can be represented as form fields or menus. If you're editing an object representing a person, for example, you might get a field, for their name, a menu choice for their title, and so on.
Now what happens when some object has a property that is a color? If you use ordinary cgi scripts, where everything has to happen on one form, with an Update button at the bottom, you are going to have a hard time.
You could use a text field and make the user type an rgb number into it, but end-users don't like that. Or you could have a menu of possible colors, but then you have to limit the possible colors, or otherwise even to offer just the standard Web colormap, you'd need 256 menu items with barely distinguishable names.
What we were able to do, in Viaweb, was display a color as a swatch representing the current value, followed by a button that said "Change." If the user clicked on the Change button they'd go to a page with an imagemap of colors to choose among. And after they chose a color, they'd be back on the page where they were editing the object's properties, with that color changed. This is what I mean about simulating subroutine-like behavior. The software could behave as if it were returning from having chosen a color. It wasn't, of course; it was making a new cgi call that looked like going back up a stack.
But by using closures, we could make it look to the user, and to ourselves, as if we were just doing a subroutine call. We could write the code to say, if the user clicks on this link, go to the color selection page, and then come back here. This was just one of the places were we took advantage of this possibility. It made our software visibly more sophisticated than that of our competitors.