On the use of parser/lexer generators

Today, I came across a blog post on Advogato about the use of parser and lexer generators. The author makes a case about writing hand-crafted lexers (in his case, a lexer for XML). As the maintainer of a LALR(1) parser generator for Scheme, I could not resist and wanted to share my thoughts on this subject.

First and foremost, let me say I have mixed feelings about the article. Although it addresses important issues, it concludes with a very simple lexing problem, not a challenging one. I think this alone greatly diminishes the strength of his argument.

Now let me give my own list of pros and cons of parser/lexer generators.

Pros

I am a big fan of DSLs. And parser/lexer generators obviously fall in this category of tools. They operate on a high level, declarative description of the language we wish to parse. This results in more maintainable software, with less bugs (most popular generators have been used extensively, so you can already be confident on the correctness of generated code).

At Nu Echo, I wrote two parsers for a language called ABNF (it's a language for defining grammars used in speech recognition applications), one using an LALR(1) parser generator and the other one being an hand-crafted recursive-descent parser. The latter is certainly faster and give better error diagnostics (I'll get back to this in a moment), but from a maintenance point of view, the former wins.

Another important aspect of parser generators is that they often support different parsing technologies/drivers. For instance, bison can generate LALR(1) or GLR parser (lalr-scm can, too). And ANTLR supports LL(1) but also LL(k) grammars. Without having to rewrite the source grammar. You can't do that with an hand-crafted parser. (Of course, you must stay in the same "family" of parsing technologies. A grammars written for an LR parser generator may not be appropriate for an LL parser generator, and vice versa.)

A corollary to this is that you can concentrate on offering the best interface to your users: the language itself. You don't have to adapt the language to a parsing technique. This is very important from a design point of view. Usability is (or should be?) one of the driving goals when designing a new language.

Cons

Unfortunately, they are more pragmatic issues that can preclude the use of parser/lexer generators. Here are a few:

• My experience with parser generators showed me that they are nice for command-line tools, but they do not produce good parsers for use in interactive environments (like an editor). (I must confess that I have almost exclusively used LR parser generators a la yacc/bison, so this argument may be a bit stretched. And I know that there have been quite a lot of work on interactive programming environments and incremental parsing.) Error recovery mechanisms are often limited and tend to produce cryptic error messages.
  


• Hand-crafted parsers are often much easier to debug. Ever tried to understand what a shift/reduce conflict is? And tried to figure out how to rewrite your grammar to resolve it? You have to understand the parsing technology. Talk about an abstraction leak!


• Generated parsers are derived resources in a development project (i.e. resources obtained from other resources). This means they are usually not kept in the CVS/git/SVN/... repository. This can complicate the build process. This may not be a big deal, but in some projects this can be trickier.
  
  In some languages other than C/C++/Java, like Scheme, this may not be a problem since the parser generator can be called at macro-expansion time.
  

Et voilà! I'm surely missing a lot of other equally important aspects, but it's getting late and I am tired. And you? What are you reasons for using or not using a parser generator/lexer generator?