Failure must always be an option

An essay I wrote six years ago about some of my experiences working for NASA has gotten some recent attention on Reddit and Hacker News. I thought I'd write a little update, particularly since some of the things I learned from that experience are, I think, relevant to the government bailout of the mortgage industry.

JPL uses an organizational structure called matrix management. There are two orthogonal management structures, one organized according to expertise (the "line management") and another, almost completely independent one organized according to task (the "program office"). The program office's job was to win contracts, and the line management's job was to provide the people to work on the resulting projects. Through most of JPL's history, its contracts were large NASA missions with budgets in the billions of dollars, and so this management structure made a certain amount of sense. A vast amount of paperwork had to be generated to win even a single contract.

For most of my career I worked on contracts that had already been awarded, so the inner workings of the program office were completely opaque to me (and still are to this day). By the time I got involved in a project the contract had already been awarded according to a proposal that had been generated by some mysterious process that I never fully understood, and which I never actually saw. All I knew was that my line management gave me a set of account numbers, and I wrote those account numbers down on my time card, and I got paid every other week. And this was true of most of the rank-and-file engineers that I worked with.

The result of this opacity was that the incentives and reward structure for individual employees was often in direct conflict with the goals of the Lab and NASA. For example, one of the factors that went into my performance review every year was how many papers I had published. In fact, this was one of the major considerations because it was one of the few things that management could get a quantitative handle on. So naturally I put a lot of effort into getting published. The problem is that the things you have to do in order to get published are often very different from the things you have to do in order to actually be productive on a NASA project. Getting published requires getting approval from your academic peers, who work for different institutions, often competing for the same contracts that your institution is trying to win. The result is a lot of politics and mutual back-scratching (and back-stabbing), because those are often more effective strategies to get papers published than actually doing worthwhile research.

I played this game, but I never liked it, and I was never particularly good at it, mostly because I was never able to suppress this annoying tendency I have of calling bullshit by its proper name, which did not make me the most popular person on the conference circuit. I was able to survive for two reasons. First, some of the work I did was actually good, at least as measured by the number of times it was cited by other researchers. (For a number of years I was, as far as I could tell, the most cited CS researcher in all of NASA. I started making that claim on my resume, and no one ever challenged me on it. It's possible I still hold that title despite the fact that I haven't published anything in years.) And second, my line management also had a perverse incentive: while my performance was being judged on how many papers I published, their performance was judged on how well they kept me supplied with account numbers to charge my time to.

The resulting situation at times approached something straight out of Kafka. I was being paid with money raised by people in the program office that I never met, whose identities I didn't even know, distributed through a process whose quality metric was not getting any productive work done but merely keeping everyone on the rolls employed, and given performance incentives that actively discouraged me from doing anything that would actually be useful to those people, but instead rewarded me for pleasing people at competing institutions. That combined with my distaste for politics and the leverage I got from using Lisp led to some truly bizarre situations. Towards the end of my JPL career, when budgets started getting tight and line management started circling the wagons, I had to start writing my own proposals. The review cycle for these proposals was ridiculously long -- months to years -- and in the meantime my line management still had to keep me employed somehow. On a number of occasions I had proposals rejected on the grounds that what I was proposing was impossible, when in fact I had already done the work in the intervening time.

At the peak of my JPL career I attained the rank of Principal, which is the highest rung on the technical career ladder whose existence is publicly known. (It turns out there are "secret" promotions you can get after that.) It's essentially the equivalent of getting tenure at a university, only with no teaching responsibilities. The decision to promote someone to Principal is made by a committee. I never found out who was on that committee, or what criteria they used to make the decision. But whoever they were, they had the power to render me more or less un-fireable, despite the fact that by then I was not really contributing anything to the Lab's mission.

All this was paid for with, including overhead, several million of your taxpayer dollars over a period of fifteen years. And my experience was not an isolated incident. This is not to say that there are not good, productive, hard-working people at NASA and JPL. There are. But there is also an awful lot of dead wood too, and no effective mechanism for getting rid of it.

What does all this have to do with the mortgage crisis? I believe that the mortgage crisis is essentially my experience at JPL writ large. Both situations were produced by a set of rules that produced perverse incentives that rewarded people for acting contrary to the greater good. In my case, I was rewarded for publishing useless papers. In the case of the mortgage industry, banks and brokers were rewarded for making bad loans. And in both cases, if push came to shove, the government with its deep pockets was there to foot the bill.

The big difference, of course, being that the bill for the mortgage debacle is five or six orders of magnitude bigger than the bill for keeping me on at JPL.

The fundamental problem in both cases was a lack of accountability combined with government-backed job security. Those two factors make a truly toxic combination. I was lucky as hell that Google was successful enough that it provided me with the capital I needed to escape the gilded cage I had locked myself into at JPL. But for that good fortune I'd very likely still be there doing God only knows what.

One of the reasons that Google succeeded was that everyone there knew that failure was a possibility. This seems strange now that Google is the 800-pound gorilla, but back in 2000 we were just another obscure little startup in the midst of the market crash brought on by the bursting of the dotcom bubble. We were well capitalized and were seeing steady growth, but success was far from a foregone conclusion, and that motivated everyone to make sure that they were all pulling in the same direction. And every now and then, someone got fired.

I think that the possibility of failure is essential to long-term success. If you have no possibility of failure, if there's always the rich uncle or Uncle Sam to bail you out, you have no incentive to sacrifice or compromise or take risks or stay up late or even to see to it that your actions are serving the greater good. There is a lot of talk nowadays of the "moral hazard" resulting from government bailouts. This is a misnomer. It's not a moral hazard, it's an economic one.

While I tend to think that the current government bailout really is necessary in order to avoid a short-term catastrophe, I really worry about its long-term effects. If we're not very, very careful we could easily render the entire country as dysfunctional as JPL. At that point there will be no rich uncle to bail us out. When that bubble bursts it will make last week, and possibly even 1929, look like a cakewalk.