Tuesday, April 26, 2011

Is it Lisp or is it me?

John D. Cook writes:


I’m skeptical when I hear someone say that he was able to program circles around his colleagues and it’s all because he writes Lisp. Assuming such a person accurately assesses his productivity relative to his peers, it’s hard to attribute such a vast difference to Lisp (or any other programming language).

...

There are genius programmers who write Lisp, and Lisp may suit them well. But these same folks would also be able to accomplish amazing things in other languages.


As someone who has specifically gone on the record saying pretty much exactly that (more than once) I feel the need to refute this claim.

Since the argument is based on the premise that I was in fact more productive than my peers, I have to toot my own horn a bit more than I'm generally comfortable with. I spent twelve years of my early career at the Jet Propulsion Lab during which time* I was promoted to the rank of Principal, the highest rung on the technical career ladder. I think it's safe to say that I did not achieve this through my sparkling personality, my willingness to brown-nose, or my finely honed political skills. In fact, looking back on it, I think at one time or another I managed to alienate about 90% of the people I came in contact with. So the only reasonable explanation of the fact that I was promoted instead of fired was that I produced results.

That still leaves open the possibility that I'm just freakin' brilliant, I could have (and would have) produced the same results in any old programming language. But there is actual data to refute that theory.

First, when I was at Google I got to see firsthand what real coding genius looks like, and it ain't me. There were dozens of people at Google who could code so fast and so effectively that it literally left me slack-jawed. My colleagues regularly did things that I would not have even thought possible had I not seen them with my own eyes.

Second, when I tried to learn to do what they were doing, I failed miserably. I tried to wrap my brain around Java and C++ and I just couldn't. I found myself so frustrated by the fact that I had to manually worry about a zillion little details that I could just ignore if I were using Lisp that I got a mental wedgie and I just couldn't get past it. I'd get bogged down in protection faults, STL errors that scrolled off the top of the screen, the fetid cesspool that is Perl, and get so frustrated that I just couldn't make any progress.

Third, I'm not the only one this has happened to. When I was working on the remote agent experiment an attempt was made to port part of that code (the planner) from Lisp to C++. After a year that effort had to be abandoned because the planner team just couldn't make it work. So it's not just me.

Fourth, it's just seems obvious from first principles that it you're using a language where you don't have to worry about memory management, you're going to get things done quicker and more reliably than a language where you do.

It is because of all this that I attribute my own success more to Lisp than to my personal coding prowess.

This is not to say that Lisp is a panacea. I do believe in the Lisp curse that the power of the language is in some respects self-undermining because it empowers the individual and so tends to attract people who don't work well in teams. A team of competent programmers willing to put up with bullshit will out-compete a lone wolf striving for elegance no matter how brilliant he is. It's sad, but that's the way it is.

---
[*] To be strictly accurate, after 12 years I was a Senior, one rung below Principal. I then left for a year to work at Google, and when I returned I was re-hired as a Principal.

16 comments:

Ricardo Herrmann said...

First of all, congrats for the blog, here's one more follower ;-) On the topic, the RA planner must be an incredible piece of work (I've read about it, mostly from the book from Nau, Ghallab and Traverso). Could you elaborate on what were the basic pain points when trying to do the port? I wrote a Graphplan+SAT (blackbox-like) planner in C++ once and then later a hierarchical non-deterministic planner in Haskell, and I can attest FP languages make it much easier, but C++ with gereric algorithms should not be that bad for this purpose (putting low-level performance aside).

Anonymous said...

Good article. But please tell us some stories about these Google geniuses and their impossible accomplishments.

Ron said...

> First of all, congrats for the blog, here's one more follower ;-)

Thanks!

> Could you elaborate on what were the basic pain points when trying to do the port?

Not really. It was a long time ago and I was not directly involved.

> please tell us some stories about these Google geniuses and their impossible accomplishments.

http://rondam.blogspot.com/2009/12/xooglers-rises-from-ashes.html

Ruben Berenguel said...

Hi Ron, thanks for posting this. Although I'm a regular reader of The Endeavour, I was expecting some answer to the post (I couldn't do it: my "biggest" pieces of Lisp are just to draw Lavaurs chords, which could be done in C or whatever and a toy ray-tracer based on pg's in ACL).

Thanks.

Ruben

Anonymous said...

"the fetid cesspool that is Perl"! LOL! Love it!

ruben said...

You are freaking brilliang, but that aside...

perhaps it's not a matter of YOU or LISP, but of YOU *AND* LISP. The merit may not be strictly yours, or of LISP, but you make a good team.

You know, like when we say of film actors "the relationship works, there's good chemistry".

But that's not the only possible team. And others may be LISP challenged but still excel at other languages. Even *yuck* Perl ;)

Anonymous said...

You come to appreciate Lisp when you face problems with creation of solution space and it's search. Until then C/C++ will do great ;)

Ken said...

It seems to me that basically what you said up there in your second point is that you tried to learn other languages, it was hard because it was different, and you gave up. I would like to think I read that wrong, so please correct me.

As for the whole memory management and detail oriented nature of C/C++ and even Java, it really has its advantages. There are things I can do with C on a bit by bit basis that I would be hard-pressed to find an alternative for in LISP.

Every language has its high and low points. I remain skeptical of anyone who says " is by far the best language out there."

Ron said...

@Ken: Not quite.

First, I didn't really give up. I kept beating on C++ until after many years I was able to produce this:

https://github.com/rongarret/Ciel

Second, it wasn't hard because it was different, it was hard because it was different in a very particular way that was not (and is not) a good impedance match to my mindset. For example, C++ actually *requires* you to violate the DRY (do not repeat yourself) principle because it uses C's brain-damaged #include construct. The amount of effort that gets wasted as a result of this just makes my blood boil. Life is too short.

Yes, there are times when you want to get close to the bare metal. But it turns out that you can much closer in Lisp than you might suspect. See for example:

http://vorlon.case.edu/~beer/Software/FPC-PPC/

Anonymous said...

@Ken:

> There are things I can do with C on a bit by bit basis that I would be hard-pressed to find an alternative for in LISP.

You can twiddle bits in the Lisps as well.

And only people who have never written a non-trivial application call it "LISP", unless you're talking about that thing from the 50's.

A_flj_ said...

Idunno ... would you code a UI widget toolkit in Lisp, rather than in C++?

I do agree that using Lisp, or Lisp-like langauges, increases your productivity significantly. But this increase in productivity comes at a cost. I don't see the guys in the automotive industry deploying Lisp interpreters to their highly constrained embedded systems, for instance. Nor do I see Oracle rewriting their database engine in Lisp. Or a new database written in Lisp overtaking Oracle.

There are also things that don't need to be very smart, but just large. IMO, C or C++ provide ways to manually manage bits and pieces that a language with managed memory and a garbage collector don't.

My point: C or C++ may come in handier when you actually do want to get bothered with all the tiny bits which Lisp would manage for you.

Java, on the other hand, provides something you don't get from neither Lisp nor C/C++: compiled code which can be deployed unchanged to a variety of platforms.

Could it be that while at Google you simply didn't grok why a particular language was used for a particular piece of code?

I also don't get what you mean with DRY being broken when using #include. Could you expand on this?

Ron said...

> would you code a UI widget toolkit in Lisp, rather than in C++?

Absolutely. Macintosh Common Lisp had one of the best UI builders ever. In fact, it still hasn't been surpassed in some respects 25 years later. Today I do quite a bit of UI hacking in CCL, which is directly descended from MCL. Lisp totally rocks for UI work.

> I don't see the guys in the automotive industry deploying Lisp interpreters to their highly constrained embedded systems, for instance.

When I was at NASA we used Lisp to program embedded processors with a few kilobytes of RAM. We didn't run Lisp directly. Instead we wrote compilers in Lisp for little Lisp-like DSLs.

> I also don't get what you mean with DRY being broken when using #include. Could you expand on this?

When writing anything non-trivial in C/C++ you have to write the function prototypes twice, once in the .c file and once in the .h file. In C++ in particular this gets to be quite a chore for large classes.

gary knott said...

you might like to go see
www.civilized.com/files/lispbook.pdf
(for a section on what Lisp is good for) and you might want to
see www.civilized.com/programming.html
as well.

A_flj_ said...

I know too little about MCL's UI builder, writing DSL compilers in Lisp or the like, so I guess it's possible. I still don't see anybody writing the next killer DB engine in Lisp.

I don't think the .h files are that much of a hassle/DRY breakage. In case of a smart IDE, it's a code generator that manages this for you, and you could always include the .c/.cpp file directly.

Still, I think Lisp is unfit for projects with a long duration and high programmer turnover, and yielding products which have to live through very many versions over a long time.
Lisp may be the best choice for a very smart programmer working mostly alone on code which has to be deployed only once, or at least doesn't have to undergo significant refactoring over time.

Maybe you'll like this: http://www.jakevoytko.com/blog/2008/10/06/lisp-is-changing-my-c/. However, what this tells me is something I've noticed since I started programming. Usually high school or university courses in C++ or Java teach you the language, not how to program in it. Which doesn't necessarily mean that one language or the other is always better.

John Rohde Jensen said...

I'm a bit disturbed that You have given up on Lisp at the same time as I've given up the other languages in favor of Lisp.

I see Lisp as a power tool that enables me to be more productive.My problem is the really big programs where brute force and manpower doesn't cut it anymore. This is where I look towards Lisp for more leverage.

Best Regards, Your work is an inspiration

John

Ron said...

Thanks for the kind words. And I haven't completely given up on Lisp. I still use it for personal projects, and I'm still hoping to be able to use it for real work again some day.