I am (not) a Failure: Lessons Learned From Six (and a half) Failed Startup Attempts

A while back I wrote:

I've had many, many failures in my life.  (Hm, maybe I should write a blog post about that.)

This is that post.  I'm writing it not as a lament, but rather because I've ended up in a good place in life despite my extraordinary track record of failing at just about everything I've ever tried.  If my younger self had heard these stories he might have had a less stressful life knowing that even a long string of failures in specific endeavors doesn't necessarily translate to failure in life.  Maybe some version of my younger self is still out there and will get some similar value out of it.

Intro

Growing up, I had two major life ambitions: to become a tenured university professor, and to found a successful startup company.  My role models were people like Rodney Brooks (who was on my Ph.D. thesis committee) who made a big splash in the AI/Robotics world in the early 1990s with the Subsumption Architecture, and was a co-founder of Lucid and iRobot.

I didn't accomplish either of those goals.  I never even got an interview for an academic job, let alone tenure.  I made six (and a half) attempts to found a startup.  All of them failed.

Failure #0: Professor Ron

In elementary school, my ambition was to be a doctor, specifically a brain surgeon.  That dream died in high school biology class when we had to dissect fetal pigs and I discovered that I was a lot more squeamish than I had thought.  So I traded my surgical scrubs for a computer keyboard and set my sights on AI instead.  Fast-forward to 1991, when a freshly minted PH.D. started sending out curriculum vitae (that's academic-speak for "resumes") to CS departments.  Unfortunately, my timing could not have been worse.  It was the height of AI Winter and the first Bush recession, and the few universities that were hiring were getting hundreds of applications for single openings.  My publication record was not great.  I hadn't published a single journal article.  My main conference publication (PDF is here) was well received, but (apparently) insufficiently academic for academia.  I didn't even get an interview.

So I just stayed where I was, which was JPL.  I've told that story elsewhere so I won't repeat it here.  Suffice to say, my JPL career had ups and downs, but eventually entered a long and steady decline from which it never recovered.  But the reason that my career declined had nothing to do with JPL or any external circumstance, it happened because I gradually came to realize that scientific research was not what I had imagined it to be.  In the real world, research is not a Platonic quest for objective truth. It is, first and foremost, a human endeavor, deeply intertwined with human ambitions and foibles, including my own.  The reason my career lasted as long as it did was that I learned, at least to a certain extent, how to work the system.  I was nominally successful, eventually becoming the most cited AI researcher in all of NASA according to a popular metric at the time (Citeseer).  But as time went by I became less and less proud of the work I was doing, and the entire field seemed mired in a stasis that I felt powerless to dislodge.  So I quit.

Lesson learned: academic research is no panacea.  (I think things are actually worse now than they were in my day.)

Failure #1: FlowNet

I spent the summer of 1990 as a visiting graduate student in Rodney Brooks's lab at MIT.  While there I met a brilliant hardware engineer named Mike Ciholas, who went on to found a company that today employs dozens of people and bears his name.  He pitched an idea to me for a new design for a computer network that he called FlowNet (full text is here).  We built working prototypes.  If they had gone into production they would have had more than an order of magnitude price-performance advantage over the competition at the time.  We had numerous meetings with VC's, but none of them ever gave us a term sheet, probably because they knew what we did not: Fast Ethernet was coming, and although it would be expensive at first, it would quickly get commodified and beat out any potential competition no matter how technically superior it might be.  We built our prototypes in 1993.  Fast Ethernet came out in 1995.  We folded up the tent in 1997.

Lesson learned: direct frontal assaults on infrastructure and standards are very unlikely to succeed.  More worthy players than us have learned this lesson the hard way over the years.

Interlude: Google

In 2000 I went to work for an obscure little Silicon Valley startup company.  I've told that story elsewhere so I won't tell it again, but I wanted to mention it for two reasons.  First, it sets the stage for what was to come, and second, while it was unquestionably a success, it was not my success.  In fact, on a personal level I count Google as a failure as well.  My tenure there was marked by a long series of screwups, and I had a serious case of impostor syndrome.    I lasted a year, but after that I went back to JPL to bide my time until the IPO.

Failure #2: IndieBuyer

The Google IPO allowed me to quit JPL and pursue my entrepreneurial ambitions more seriously.  With the benefit of hindsight I should have moved to Northern California and hung out at Y Combinator, but I had lived in Los Angeles for 17 years at that point, which was longer than any place else I had ever lived before by a wide margin.  By my standards at the time, I had deep roots there.  I have also always had a love for magic and movies, and Los Angeles was the epicenter of both of those industries.  So I decided to try to do something in the entertainment space.

The result of that was a company called IndieBuyer, which was going to be Netflix for independent films.  The idea was that technology was making filmmaking more and more accessible, leading to a long tail of undervalued product that couldn't find an audience because of the bottleneck presented by a distribution system designed for more capital-intensive projects.  The business model was that we would sell DVDs (which were still very much a thing back then) but with a money-back guarantee: if you bought a DVD that we recommended and you didn't like it, you could return it for a full refund.  The returns would provide the data for a recommendation engine that would, we hoped, over time become a better and better predictor of who would like what kinds of movies.  The long-term business model was to intelligently match niche products with niche audiences at scale.

It was a plausible idea, and it might have even worked if we had been better capitalized.  But I was funding it entirely from my own pocket, and after a couple of years with very little growth I lost my nerve and decided to pull the plug.  We did try to find some outside investors, but because we had already launched and weren't showing much traction, we didn't attract any interest.

Lesson learned: it's really hard to make money in the movie business.

Failure #2.5: Evryx

This is the half-failure alluded to in the title.  I count it as half a failure because, although I was actively involved, I was not a founder.  Evryx (which eventually rebranded itself as SnapNow) was founded by another engineer at JPL.  The company was an early pioneer in reverse image search.  It used an algorithm called SIFT to match an image taken with a cell phone to a library of stored images and provide information about whatever was in the image.  I was never entirely clear on what the business model was intended to be, but the technology seemed cool.  I happened to mention it in passing to a friend of mine whom I had met at a political fundraiser.  He looked into it and got sufficiently excited about it to become their CEO.  The founder became CTO.

About a year later the company had a crisis: the founder/CTO was behaving erratically, and my friend, the CEO, decided he had to fire him.  He tapped me to come aboard as acting CTO until a permanent replacement could be recruited.  The company limped along for another year or so but ultimately folded in 2007, in no small measure because of the fallout from the financial crisis that began that year.  The CEO had actually managed to secure another round of funding, but it was voted down at a shareholder meeting, with the (fired) CTO casting the deciding vote against.  The company folded a few weeks later.  The bankrupt entity was eventually acquired at a fire-sale price by a private investor, and I have no idea what happened to it after that.  It was sitting on some potentially valuable IP, so maybe they were able to do something with it.

Lesson learned: even smart people can do incredibly stupid things.  The CTO said he voted against the financing because he didn't like the terms and didn't want to be diluted.  The result was that his intact share of the company ended up being worthless.

(I actually saw a similar thing play out at another company in which I was a passive investor.  The technical co-founders basically torpedoed the company, but in their case they went on to start a new company with the exact same IP and business model.  It was blatantly illegal, but to that point the company was funded entirely by small angel investors, and no one was willing to put up the resources to try to stop them.  So they got away with it.)

Failure #3: iCab

In 2009 my wife and I took a cruise around the Pacific rim.  At one point we ended up in Saigon and needed to get back to the cruise ship, but couldn’t find a taxi.  I thought to myself: this place is chock-full of taxi drivers, all of whom have cell phones, and many of whom have smart phones.  Someone should write an app that would let me broadcast the fact that I'm standing in this location looking for a ride.  It was a brilliant idea.  Unfortunately, Garrett Camp and Travis Kalanick had the exact same idea at the exact same time.  I founded a company called iCab, and they founded a company called Ubercab, which eventually changed its name to Uber.

iCab was stillborn.  It was never even established as an actual company.  I assembled a team, mainly people I knew from Evryx, and wrote a prototype app which actually worked.  But we couldn't figure out a way to procure drivers.  We were in LA and we tried pitching the idea to cab companies, but they were not interested.  The idea of using black car limos never even occurred to me.  With no supply chain, we folded up the tent before we had even pitched it.

Lesson learned: a brilliant idea is not enough.  You have to execute.  And sometimes that requires another brilliant idea.

Failure #4: Virgin Charter (nee Smart Charter)

In 2010 my wife and I decided to get an apartment in Santa Monica so we could spend more time by the beach.  I started hanging out in the offices of a venture capital firm with an office on the Third Street Promenade.  One day a fellow named Scott Duffy walked into the office and asked if he could hire us to do due diligence on an acquisition he was considering.  I don't remember the name of the company, but it collected business intelligence on the private aviation industry.

In the course of conducting this due diligence we learned that the private jet market is incredibly fragmented and inefficient, with dozens of operators most of whom had only a small handful of planes.  At the time there were a few thousand private jets operating as charters, but there were only three operators with fleets of more than 100 planes.  The largest of these was NetJets with (if memory serves) about 250, only about 10% of the total.

To match customers with aircraft, a secondary industry of brokers had cropped up.  If you wanted to charter a jet you would call up a broker, give them your mission parameters, and then they would start making phone calls and sending emails and faxes (yes, fax was still a thing back then) to try to find a matching aircraft.  The turnaround time to get a quote was measured in hours, and actually securing a reservation could take days.

But even worse than the time involved was the fact that when an aircraft took you somewhere, it eventually had to return to its home base.  Most of the time this happened in one of two ways: you either paid to have the aircraft (and its crew) wait for you and take you back, or you had to pay for two round trips.  About 40% of legs flown by private jets were flown empty.  This led to a secondary market of brokers trying to sell these empty legs, which had already been paid for.  If you were lucky and happened to find one that was flying a route that you wanted to travel, you could pick one up for a song.  The empty flight was already paid for, so any revenue at all was pure profit.

The problem is that most of the customer base for private jets want to go where and when they want to go, not where and when a plane happens to be available, so it was rare to actually sell an empty leg.  But if you've ever wanted to fly on a private jet and you are flexible with regards to your timing and route, you could (and still can) find some very good deals.

One man's inefficiency is always another man's business opportunity.  I realized that the problem was the lack of a centralized clearing house for demand.  If we could somehow collect all the data on who wanted to go where and when, we could route the fleet much more efficiently.  We crunched the numbers and it turned out to be a billion-dollar business -- if we could somehow pull it off.  The problem was that the customers and the suppliers all depended on the brokers.  In order to succeed we wold need to wrest both away.  How to do it?

I came up with the following diabolical plan: build a web site that would let people enter their trip details, and then give access to that data to the brokers as a lead generation service for a very modest fee.  The fee was actually a red herring to disguise the fact that we were secretly plotting to put the brokers out of business.  It would be an offer they couldn't refuse: more customers for a small amount of money.  But once we reached a critical mass of users, we would start phase two: offer that lead data directly to charter operators so they could bypass the brokers.  Again, it would be an offer they could not refuse.  We would offer these leads to the operators for free, and it would be a much faster and more efficient way of getting bookings because the whole process would be automated.  As a bonus, they could pocket what would have been the broker's commission.

Finally, once we had a critical mass of operators on board, we would start phase 3, a centralized scheduling service that would dispatch the entire fleet in order to minimize the number of empty legs being flown.  Again, it would be an offer they couldn't refuse.  Anyone who didn't subscribe would have to charge significantly more for the same product in order to pay for the additional empty legs, and market forces would eventually drive all the customers to us.

To this day I'm pretty sure that plan would have worked if we had actually executed it.  It was a brilliant plan if I do say so myself.  In fact, it was so brilliant that it convinced Richard Branson to acquire the company before we launched for $10M.  We started the company as Smart Charter, but we launched as Virgin Charter.

I vehemently opposed the acquisition, but I was overridden by my two co-founders.  When I lost that fight, I resigned from the company, but I kept my shares on the off chance that my intuitions were wrong and Virgin would actually make it work.  Unfortunately, my fears were well-founded.  As soon as they took over, they changed the business model to be directly consumer-facing, because that is Virgin's strength.  Instead of a stealthy three-phase plan, they went for a direct assault on an established industry with very little understanding of the actual dynamics of that industry.  Charter jets are a very rarified market.  There's a lot of dollars involved, but not a whole lot of people, and those people don't like to be treated like ordinary consumers.

At our peak we had dozens of sales people.  They all sat around the office with almost nothing to do.  This was made all the more ironic by the fact that the original business model required fewer than ten people and probably could have been run on a single server.  Even at 100% market share, the volumes in private jets are just not that high.

Lesson learned: Be careful who you take on as a partner.  Success in one industry does not necessarily translate into success in another.  This is another lesson that others have learned the hard way.

Failure #5: Founder's Forge

After being stung so hard by the private jet fiasco, I decided to design a business that I could run entirely without partners.   This was around the time that Bitcoin first appeared, though it would still be many, many years before it broke out of hacker circles and into public awareness.  I had always been interested in cryptography, and there were parts of Bitcoin's technology that I found very appealing, particularly its use of public-key cryptography and digital signatures.  I foresaw increasing problems with credit card fraud because of the fact that the protocol was fundamentally broken: the credentials that you use to authorize a transaction are not bound to that transaction.  They can be re-used.  This is still the basis of most credit card fraud today, and it's the reason your card has a chip in it.  That chip has a secret key in it, and it computes digital signatures.  That makes it impossible to re-use the authorization for one transaction on a different transaction, and makes the credentials in general harder (but still not impossible) to steal.

The problem is that you can't use that chip for e-commerce.  It's not because this is inherently impossible, it's just that the infrastructure to do it doesn't exist.  There is no reason you could not have a card reader peripheral plugged into a USB port that would execute the same protocol that point-of-sale terminals use and provide the same protection.  In fact, there is no reason why you even need any hardware at all.  The entire protocol could be executed entirely in software on your computer or smart phone.  It would cost next to nothing and it would eliminate most credit card fraud.  So why hasn't it been done?

It is because the process is controlled by the banks, and the banks don't care.  Why don't the banks care?  Because they treat the cost of fraud as just another cost of doing business, and they pass it along to you, the consumer.  And they do it in a diabolical, stealthy way that you don't notice.  But that's another story.  Back in the day, I was not yet aware of all this.  I thought that fixing credit card fraud would be, if not welcomed, at least not actively opposed.  I was wrong.

I knew that the banks controlled the credit card system, and I knew they comprised a massive, ossified industry that was going to be next to impossible to penetrate.  So I came up with a plan that would hide the fact that I was trying to solve credit card fraud by presenting it as a completely different product addressing a completely different problem and market.

The idea was this: by this time I had a fair amount of experience starting and running companies, and one of the "pain points" was financial record-keeping.  Making sure the books were up to date was a sufficiently serious chore that most companies employed an entire team of people to do it.  (There's a reason that "the finance department" is a thing.)

The fundamental reason that record-keeping is a chore is that the records are decoupled from the transactions they record.  This is for historical reasons.  You can easily hand someone a dollar, or write them a check, or swipe your credit card without writing down that you did these things.  In the vast majority of cases you have to manually connect the record of a transaction with the actual transaction itself.

But in an age where most transactions are electronic, this doesn't have to be the case.  There is no reason why you can't build an interface in which the authorization and the record-keeping are one and the same operation, and if you do that, they cannot fall out of sync.  So the idea was to provide a service that coupled transactions and record-keeping to guarantee that they always remained in sync.  And the interface to the authorization system would quietly use digital signatures as the authorization mechanism.  The users of the system would mostly be unaware that there were digital signatures happening under the hood, just as the users of chipped credit cards are mostly unaware of this.

The company was called "Founders Forge" because the initial target market was going to be startup companies.  The name was a play on "SourceForge", the then-popular open-source hosting site.  But all of that was a red herring because the real plan was to replace credit cards with digital signatures for e-commerce.

Of course, to make all this work I needed a way to move money around, and for this I needed a bank.  I always knew that was going to be the heavy-lift so I focused on that early, long before I had even a prototype product ready to launch.  As things happened, I never even built a prototype because the banks turned out to be an impenetrable brick wall, but that impenetrability manifested itself in a very unexpected way.

I approached banks with the following pitch: I am building an accounting system for startup companies, and I want to link it with their actual financial transactions.  So I need access to an API to allow me to authorize transactions on my client's behalf.  The clients will authorize me as their agent so it will all be on the up-and-up.  The benefit for the bank is that I would funnel them a steady stream of new business in the form of the most promising Silicon Valley startup companies.

Somewhat to my surprise, this pitch was very well received.  I got positive responses and follow-up meetings from half a dozen different banks, and I thought I was on my way.

But then events took a very unexpected turn.  One by one, every bank that had initially responded positively changed their minds.  Worse, not a single one of them would tell me why.  They said, "Sorry, turns out we can't do it" and then stopped returning my calls.  In one case I actually got as high as the bank's CTO, who seemed very excited about the idea, but then he too suddenly backed off.  At that point I decided to quit, and went into a deep existential crisis for a while.  I had been working on Founders Forge for about six years.

I never got a straight answer from any of the banks about why they changed their minds.

Lesson learned: never underestimate the capacity of the banking industry to stifle innovation.  (Also, it really does appear to be run by a shadowy cabal of people you will likely never meet.)

Failure #6: Spark Innovations

I don't actually remember exactly how my last company got started.  I did this one with a partner who handled the business side of things, but was also a really good designer and front-end developer, so we made a kick-ass team.  The product was motivated by the observation that people used Microsoft Excel for purposes that it was never really intended to serve.  In particular, people tended to use it as a replacement for a SQL database.  The idea was to build a product with a spreadsheet-like UI, but with a SQL database on the back end.  We had three launch customers lined up, all of whom seemed to be very excited about the idea.  While we were building the MVP we had a previously scheduled meeting with a VC acquaintance.  We told him what we were working on, not really intending it to be a pitch, but just FYI.  In one of those legendary Silicon-Valley moments he got so excited about the idea that he asked us to let him invest just to get a foot in the door.  We didn't really want to take any money until after we launched, but he insisted, so we capitulated.

To make a long story short, we launched, and all three of our launch customers suddenly lost interest and didn't tell us why.  It was a similar situation that I had encountered previously with the banks.  No feedback, just suddenly not interested and not returning our calls.  At that point my partner had to quit because he needed a paycheck.  In a final desperate hail-mary I tried to pivot the company to something I could do on my own, but that didn't work either (obviously).  I'll spare you the details on that one.

And that's it.  My tale of six and a half failed startups plus one failed academic career, all of which somehow added up to a good life at the end of the day.  Of course, hitting the jackpot with Google didn't hurt, but I really don't think that was the whole story.  I learned a lot.  I grew a lot.  And  I think my current happiness stems mainly from the fact that I like the person I've become, someone who can fail again and again and again and again and still find a way, for the most part, to be happy.

A Reply to Don Geddis

Don Geddis left a comment on my last post.  My reply grew far longer than would reasonably fit into a comment reply so I decided to post it as an article.  Don wrote:

I wonder if you've considered that perhaps you have more in common with the people who frustrate you, than your current self-image suggests.

My reply:

I've not just considered it, I will happily concede that I am not as rational as I fancy myself to be.  In my case, I would prefer to put a slightly different spin on it and say that I am not as rational as I would like to be, or as I try to be, but whatever.  I'm a fallible human, just like everyone else.  (Duh!)  I make mistakes, and one of the mistakes I've made on more than one occasion is to be overly confident in my own abilities.  The best I can say about myself (or at least could until recently) is that I try.  I try to recognize when I make mistakes.  I try to learn from them.  I try to improve.

The failure I'm referring to in this post is not that I've failed to be sufficiently rational, or even that I've failed to persuade others to become more rational.  I only mentioned that to put my gloomy assessment of the situation into context.  My failure is that I reached the limits of my willingness to try.  I've had arguments with Protestants and Catholics and Muslims and Witnesses and Mormons and YECs and lunar landing denialists and even Republicans.  (I've never had the pleasure of interacting directly with a flat-earther, but if there are any among my audience who would be willing to engage I'd welcome the opportunity.)  The vast majority of these discussions have been civil and constructive.  I've learned a lot, and found a surprising amount of common ground.  I can steel-man just about any position (even yours, I'll bet).  In a not-insignificant number of cases I've ended up becoming friends with my interlocutor (at least for a while).  I even managed to make peace with Erik Naggum once (at least for a while).  That is one of my proudest achievements.  I figured if I could have a civil discussion with him and find common ground, I could do it with anyone.

I was wrong.  I tried really hard with Publius for a very long time, but the claim that Kamala Harris was never a prosecutor was such a brazen falsehood that it took me by surprise, and left me to face two of the most difficult choices of my life.  The first was whether to publish the claim and its accompanying screed.  Do I contribute to the promulgation of falsehoods, or do I sign on to cancel culture and become a censor?  Neither of these options is appealing.  Both run deeply counter to my moral intuitions.  It's not even clear to me which is the lesser of the two evils.  I ultimately chose the former, not because I had a good argument for it, but just because I didn't want to face the second choice: if I published Publius's comments, should I respond to them, and if so, how?

The reason this second choice was so difficult is that responding to Publius takes a lot of effort.  It can be mentally exhausting.  He's not stupid.  If I make a mistake, I can be pretty sure he'll call me out on it.  (See for example our recent exchange about the Chinese Room.)  So I can't phone it in.  I have to think and do research and keep track of a zillion different points (because Publius is a master of the Gish Gallop).  It takes time and mental energy, both of which I have in increasingly short supply these days.  So responding to Publius has a cost.  On the other hand, not responding has a cost too.  If I publish his comments and don't reply, someone might come away with the impression that I think his claim might have some merit, and that's not a good outcome for me either.

I agonized over this for nearly two months hoping that some other alternative would present itself, but none did.  So I decided to do what I did.  I don't regret it, but I'm not proud of it either.  I find it a little harder to look at myself in the mirror now than I did before.  Before I could say that, yeah, I'm far from being perfectly rational, but I'm willing to listen to anyone and try my best to understand their position.  I can't say that any more.

But I can still say it about you, Don.  I've never censored you, and you have never said anything that would make me even consider it.  Your silence here is self-imposed.  Any time you feel like breaking that silence, either in public or privately, I'm ready and willing to listen to you, and that has always been the case.

I have failed. Now what?

Nearly two months ago now I wrote:

It's getting harder and harder to find a reason to keep doing this.  My opportunity costs are high, and writing a blog entry takes a non-trivial amount of time.  I wrote this because I needed to blow off some steam, and I wanted to get my position on the election results on the record while they were still fresh in my mind.  Whether I keep doing this in the future is now very much an open question.

I got a handful of replies to that.  Thanks to everyone who responded.

The reason it has taken me so long to post any follow-up is that I also got an extended eight-part (I think that's a new record) rant from Publius, which began:

In my replies below, I will show with facts why Kamala Harris is the true threat to the rule of law, free speech, and free elections. As a bonus, you'll also find out that Kamala Harris was never a prosecutor.

I regret to say that that defeated me.  I've spent over forty years actively seeking to understand points of view that I don't agree with in the hope of finding a way to more effectively advocate for science and rationality.  During that time, I can't recall even a single instance of successfully bringing someone into the rational fold who wasn't already there.  Worse, I've discovered to my dismay that many people profess to hew to rationality aren't nearly as rational as they fancy themselves to be, and when I've tried to point this out to them they get defensive and tribal, in other words, they react like any other human would when their core beliefs are challenged.  I don't like that, and so I've tried very very hard not to get defensive and tribal when my core beliefs are challenged.  Instead, I've tried very hard to get to the roots of disagreements, partly because it has always been important for me to make sure that my beliefs rest on firm philosophical foundations, but also because I've observed over the years that a lot of disagreements turn out to be trivial differences over the definitions of words, and that people actually agree on a lot more than they appear to.  This is not to say that there aren't real disagreements -- there certainly are.  But I've found these to be a lot more rare than most people think.  In my personal and professional life I've gotten a huge amount of leverage out of this insight, and I was hoping that I could share it with others and maybe make the world a better place.

To say that I have failed would be quite the understatement.  I have failed spectacularly.  At the risk of stating the obvious, Kamala Harris was the Attorney General of California between 2011 and 2017.  That makes her a prosecutor in my book.  But even as I write this the thought that is foremost in my mind is not that I am advancing a sound argument supported by facts, but the not-so-sudden realization that what I am doing is futile.  No matter what I say, Publius will maintain that Kamala Harris was never a prosecutor, and there is no doubt in my mind that he will be able to compose lengthy screeds to defend his position.  Maybe they will even have merit.  I don't know because I'm not going to read them.  I'm done.  I'm defeated.  Life is too short to keep dealing with this kind of bullshit.  Congratulations, Publius, you won.

In a rational world, all of this would be trivial.  Instead of a loyal following my blog has ended up with a lone heckler.  Boo-hoo, let's all feel sorry for me.  But that is not really the point.  I've had many, many failures in my life.  (Hm, maybe I should write a blog post about that.)  None of them have sent me into the kind of existential despair I'm currently feeling, and that's because Publius is not an outlier, but merely a representative sample of a much, much larger population, who just elected a convicted felon and rapist to be the President of the United States of America.  But of course, to these people, Donald Trump is not a convicted felon nor a rapist, just like Kamala Harris was never a prosecutor, and nothing you nor I nor anyone else can say will ever convince them otherwise.  Truth doesn't matter.  Reality doesn't matter.  All that matters is Donald Trump and Making America Great Again (whatever the fuck that might actually turn out to mean).

Trick is, reality is even more persistent than Publius.  You can only turn your back on reality for so long before it will reassert itself. Running a country actually requires work and expertise, and if you don't put in the work, things will fall apart.  Silencing the news media might make you feel better, but it won't produce prosperity or freedom.  Neither will tariffs.  Neither will restricting immigration.

I feel like I'm watching a train wreck approaching and nothing I say or do can stop it.  The rational thing to do is to work to prepare for the disaster, to try to anticipate the worst-case scenario and put plans in place now for how to recover afterwards.  But that is not something I can do on my own, and I've found very few people interested in even discussing it.  The message I'm getting from Democrats is: we need to write bigger checks to Democrats so we can take back the House in the mid-terms and keep Donald Trump in check.  This seems hopelessly naive to me.  The Democrats had six years to get Donald Trump under control.  They impeached him twice, indicted him I don't know how many times, convicted him on 34 felony counts, and none of that made the tiniest dent in his power or influence.  And now he's heading back to the White House with complete legal immunity for anything he does as an "official act" (again, whatever the fuck that might actually turn out to mean).  The only thing that will stop Donald Trump now is if his supporters turn on him.  Good luck with that.

In a situation like that I'm finding it incredibly hard to resist the temptation to curl up into a fetal position, or maybe move to Canada.  It's not what I want.  That is not the person I want to be.  But I'm honestly out of ideas.  I've given it my best shot for somewhere between 20 and 40 years depending on how you count.  I have failed.  I don't know where to go from here.

Merry Christmas.

The Bright Side of the Election Results

I'm writing this at 9AM Pacific standard time on November 6, the morning after the election.  Not all the dust has quite settled yet, but two things are clear: Donald Trump has won, and the Republicans have taken control of the Senate.  The House is still a toss-up, and it's still unclear whether Trump will win the popular vote, but the last time I looked at the numbers he had a pretty commanding lead.

From my political perspective this is about as bad an outcome as I could have imagined, but from my perspective as an American citizen it could have been a whole lot worse.  The scenario I really dreaded was a replay of Bush v Gore, with Trump winning because three Supreme Court justices he appointed put their thumbs on the scale for him.  That didn't happen.  In fact, none of the truly nightmarish predictions came to pass.  There was no widespread chaos or efforts to disrupt the election, just one (AFAIK) random act of arson.  There was no mass suppression of votes.  None of the outcomes turned on any court decisions.  There was no armed insurrection.  There was no widespread AI-backed misinformation campaign, just more of the same run-of-the-mill falsehoods that have become part and parcel of the Trump brand.  There will be no second Civil War, just a peaceful transfer of power on January 20.

It was, on the whole, a good day for democracy.

Don't get me wrong, I still think this is a bad outcome, and I predict it will ultimately prove to be catastrophic.  There are some who believe in democracy because they think it produces good outcomes.  I am not one of those people.  The point of democracy IMO is not that it produces good outcomes -- it often doesn't.  The point of democracy is that it, like the rule of law itself, provides a non-violent mechanism for resolving disputes.  And this time around it's going to work.  The problem is, I predict, that we are now going to learn the hard way that running a country is not actually such an easy thing to do, and Donald Trump is not quite as smart as he has portrayed himself to be.  He wasn't up to the task the last time, and I see no reason to think he's going to do any better this time.  We're going to have a peaceful transition of power on January 20.  Some time after that -- not right away, but certainly before 2028 -- we're going to have chaos and catastrophe.

Part of me hopes that the Republicans will take control of the House, not because I expect that will make things any better (it won't) but because it will make it that much harder for the Republicans to deny responsibility for the disaster they are about to create.  If we're going to do this experiment, let's fucking do it right and eliminate all the confounding factors.  It will cost more, but at least we won't need to argue about whether things are bad because we didn't go MAGA enough.

Specifically what will go wrong and when I cannot say.  One easy prediction is that if Trump actually follows through with his promise to impose import tariffs, we will see a resurgence of inflation.  That much is a no-brainer.  Whether Trump will actually be stupid enough to do this remains to be seen.  The main reason he's returning to the White House now is because inflation hit a high of 9.1% in 2022 in the wake of the pandemic.  His promised tariffs will produce much steeper price increases than that.  God only knows if that will be enough to make his supporters turn on him.

But if high inflation turns out to be the worst of the fallout from a second Trump administration we will be very, very lucky.  There is so much more at stake.  Taiwan.  Ukraine.  The Balkans (and the rest of eastern Europe).  Gaza (and the rest of the middle-east).  NATO.  Climate change -- this election has probably squandered our last hope to limit carbon emissions to non-civilization-ending levels, though that won't become apparent for quite a while yet.  And then there is the very exclusive anguish you are going to experience if you are dark-skinned, or gay, or Muslim, or female.

It's going to be bad.

Will we be able to recover from this?  I don't know.  Frankly, I wouldn't bet my life savings on it.  It's going to be very, very hard, and very, very painful, and take a very, very long time.  It's going to require a lot of people on the left to let go of some very cherished ideas, like that the only way Donald Trump could possibly win would be to cheat.  He didn't.  He won fair and square.  That is the reality we have to face.

POSTSCRIPT: I was a little surprised to see someone other than Publius posting comments on my last two posts.  It has been a long time since that has happened, and I was beginning to think that I was playing to an empty house (well, not quite empty, but it's more than a little disheartening when it seems like your audience consists of a single heckler).  So if you've been lurking, and you've been getting some value from what I write, please let me know either by leaving a comment or dropping me a line.  It's getting harder and harder to find a reason to keep doing this.  My opportunity costs are high, and writing a blog entry takes a non-trivial amount of time.  I wrote this because I needed to blow off some steam, and I wanted to get my position on the election results on the record while they were still fresh in my mind.  Whether I keep doing this in the future is now very much an open question.  I have never tried to monetize my writing.  My only reward is the knowledge that someone out there got some value out of it, and that's enough for me.  Most of the time I can sustain myself on faith, but every now and then I need actual data, and this day more than usual.

What scares me about a second Trump administration

As long as I'm getting things on the record (while I still can without too much fear of reprisal) I want to endorse a video by Legal Eagle that lays out the case against voting for Donald Trump in 18 minutes of some of the best video commentary I've ever seen.  It's well worth watching, and encouraging others to watch, but just in case you don't want to invest the time and would rather read, I'm posting a (very lightly edited) transcript here.  Yes, this is blatant copyright infringement.  I apologize.  But the stakes are high and this seemed like a risk worth taking.  The Legal Eagle team has laid out this case better than I could ever hope to do.

Ron Prognosticates: Trump is Going to Win

 I'm too depressed to elaborate much on this, but I just wanted to go on the record with this prediction before the election.  Why do I think Trump is going to win?  Because DJT stock is up and has been rising steadily since it hit an all-time low in late September.  It didn't even go down today after yesterday's disastrous MSG rally.  The polls have been static since late September, or even moving slightly in Harris's favor.  So why is DJT stock up >400% in one month? I have a hard time believing that there is that much dumb money out there.  The most likely explanation I can think of is that someone out there with deep pockets knows the fix is in and Trump is going to win.

Never before in my life have I ever wanted so much to be wrong.

[UPDATE]  DJT fell 22% today, its worst trading day ever.  Maybe it was just a Trump pump-and-dump?  God, I hope so.

Have Republicans Ever Actually Listened to the Lyrics of YMCA?

Yesterday we were treated to the sight of a major party nominee at what was supposed to be a town hall meeting suddenly stop taking questions and just dancing (badly) for the better part of an hour.   A mere 20 years ago, well within living memory, less than five seconds of screaming were enough to end Howard Dean's political career.  My, how times change.

But the truly astonishing thing, the real elephant in the room, is not just that Trump could swing his hips for 39 minutes and still have a shot at becoming President of the United States of America, it is that one of the songs on his playlist was YMCA by the Village People.  I wonder if anyone in his audience has ever paid attention to what that song is actually about.

Here are the lyrics to the first verse and chorus for easy reference:

Young man, there's no need to feel down, I said
Young man, pick yourself off the ground, I said
Young man, 'cause you're in a new town
There's no need to be unhappy
Young man, there's a place you can go, I said
Young man, when you're short on your dough you can
Stay there and I'm sure you will find
Many ways to have a good time

It's fun to stay at the Y.M.C.A.
It's fun to stay at the Y.M.C.A.
They have everything for young men to enjoy
You can hang out with all the boys
It's fun to stay at the Y.M.C.A.
It's fun to stay at the Y.M.C.A.
You can get yourself clean, you can have a good meal
You can do whatever you feel

In case it's not obvious (and apparently it isn't) this song is about young men going to the YMCA to have promiscuous sex with other men. That is what they are celebrating.  That is what Donald Trump has led them to, almost certainly without himself even realizing it.  Trump is so clueless, and his followers are so clueless, that they can't even decipher "You can do whatever you feel" when sung by a group of men dressed in homoerotic iconography.

Maybe if more people pointed this out to them the scales would start to fall from their eyes?  Probably not, but a boy can dream.

UPDATE: Heh, I just noticed that Trump has a telling Freudian slip just before he calls for YMCA to be played.  He tries to say, "We're going to make the country great again" but he slips and what actually comes out of his mouth is "We're going to make the country gay..." just before he catches himself and stops mid-aphorism.  So maybe deep down he does know.

Yes, you can have exactly-once delivery

Introduction

This post is ostensibly about an obscure technical issue in distributed systems, but it's really about human communications, and how disagreements that on the surface appear to be about technical issues can sometimes turn out to actually be disagreements about the meanings of words.  I'm taking the time to write a fairly extensive post about this for two reasons.  First, I'm hoping it will be helpful to provide a case-study about what appears on the surface to be a technical dispute that turns out to be a mere quibble over terminology.  Learning to recognize such situations can help defuse tense situations.  And second, this might provide some cover for someone early in their career who comes to the same conclusion I did about a certain emperor having no clothes.

The dispute

I was reading Hacker News a few days ago and stumbled on a comment posted by /u/solatic in response to an article entitled "Falsehoods programmers believe about TCP".  I quote that comment here in its entirety:

Related: you can get at most once delivery or at least once delivery; you cannot get exactly once delivery. If I had a dollar for every junior who thought that a lack of exactly once delivery guarantees was a bug...

This struck me as odd because I could see a straightforward way to implement exactly-once delivery on top of at-least-once delivery.  But solatic seemed sufficiently confident that I thought I might be missing something, so I posted what I thought was a cautious response:

If you can get at-least-once delivery, why can you not build exactly-once on top of that?

I won't go into detail about what happened after that.  You can go read the resulting thread if you like, but the TL;DR is that the discussion spun wildly out of control, which ultimately motivated me to write this post.

Before I dive into the details, I want to give my credentials because I am about to challenge what is apparently some pretty deeply entrenched conventional wisdom.  Unfortunately, many of the people who hold these mistaken views seem to subscribe to arguments from authority.  I got a lot of admonitions that were tantamount to, "You obviously need some remedial training in basic computer science."  That's possible, but I want to provide some information about my background so you can adjust your Bayesian prior regarding that particular hypothesis.

I have a Ph.D. in Computer Science.  My specialty is (or at least was) AI and not distributed systems, but I nonetheless know my way around a network stack.  My first startup, about 30 years ago now, was an (unsuccessful) attempt to launch a new network architecture.  We built our own hardware.  I wrote the Linux device drivers for our custom NIC.  I currently work part-time as a consultant for a hardware design team that builds network routing chips.  I know how a modern network works literally from the transistors on up.  And I know about the two generals and the muddy children.

So with that, let's look at the claim that I found questionable: "You can get at most once delivery or at least once delivery; you cannot get exactly once delivery."

What does "delivery" mean here?  The entire kerfuffle turns on the ambiguity of the meaning of this word, which none of my interlocutors ever defined.  But whatever it means, my puzzlement stemmed from the fact that I could see a trivial way to build exactly-once delivery from at-least-once delivery, under any reasonable definition of "delivery".  Specifically, if you could get at-least-once delivery, then turning that into exactly-once delivery should be (it seemed to me) a simple matter of keeping track of all the delivered messages and removing duplicates.  Doesn't that seem plausible to you?  If so, congratulations, your intuitions are serving you well because that will in fact work.  It's not a particularly good solution from an engineering point of view, and it is absolutely not how you would do it in practice (at least not without a lot more refinement) but it does refute the claim that you "cannot get exactly-once delivery".  You can.

So why do so many people deny this obvious truth?  Well, that's a good question.  I'm not 100% sure because no one in that HN thread ever presented a cogent argument or even a reference.  But one common theme that ran through the responses was that I needed to brush up on the two-generals problem.  So let's do that.

The two-generals problem

The Two Generals problem is a classic problem in theoretical computer science first studied by Leslie Lamport in 1983.  It was the first time ever someone proved that a particular problem in computer science had no solution.  Some people claim that if you could get exactly-once delivery you could solve the two-generals problem, but this is false.  The two problems have nothing to do with each other, despite having some superficial resemblance.

The two-generals problem is this: there are two generals (imagine that) leading two armies against a common enemy.  They can defeat their enemy only if they successfully join forces and coordinate a simultaneous attack.  But they are geographically separated in an era before modern telecommunications.  The only way they can communicate is by sending messages via courier thorough enemy territory, where they risk being captured and never heard from again.

It turns out that it is impossible for the generals to coordinate their attack under these conditions.  In order to coordinate the attack, each general needs to know that the other general has received and agreed to the same attack plan that they intend to undertake.  The technical term is that they need to achieve common knowledge, which turns out to be impossible using an unreliable communications channel.  The proof is easy to understand: if there were such a protocol, then that protocol must have a termination condition, some point at which consensus has been reached, after which no further messages need to be sent.  But the protocol must still work even if the last message in the protocol fails to arrive, and so sending the last message must not be necessary -- the protocol has to still work without it.  By induction, the protocol must work even if no messages are sent at all.  That is clearly impossible, and so our assumption that there exists such a protocol must be false, QED.

But achieving common knowledge has absolutely nothing to do with the question of whether exactly-once delivery is possible -- it is.  As I've already described above, the easiest way to do it is to keep track of all delivered messages and remove duplicates.  (Exercise: why does this strategy not succumb to the same proof of impossibility as the two-generals problem?)

Exactly-once delivery

Now, this is where things get tricky, because no one ever actually responded to this very straightforward suggestion.  So at this point I need to extrapolate the replies I did receive and guess what the response would be if I really put someone's feet to the fire.  My guess is that they would say that what I've provided is not exactly-once delivery but exactly once processing.  I've made the processing of incoming messages idempotent, so that it doesn't matter if I get duplicate delivery, but I have not actually eliminated duplicate delivery, or something like that.

It is possible to define "delivery" makes the claim that exactly-once delivery is impossible true.  The problem is that such a definition only serves to advance pedantry and ignores important subtleties in human communication.  Let's go back to the original claim: "You can get at most once delivery, or at least once delivery; you cannot get exactly once delivery."  There are some things implied here that are not explicitly mentioned.  Specifically, there is an implication that the two things that you (supposedly) *can* have -- at-most-once and at-least-once -- are actually things that one might choose under realistic circumstances.  Consider:

You can get at most once delivery, or at least once delivery, or you can hit yourself in the head with a hammer.

That statement is true -- hitting yourself in the head with a hammer is an option.  But I hope I don't have to persuade you that this is not very useful, and the only thing you're likely to accomplish by pointing it out is to annoy people.  When you provide a list of options, there is an implied social contract that the items on the list have at least some plausible utility.

In a similar vein, one way to get at-most-once delivery is simply to disconnect the communications channel.  That will absolutely guarantee that no message is ever delivered more than once, because every message will in fact be delivered exactly zero times.  Like hitting yourself in the head with a hammer, this is an option you can undertake, but it's not very useful, and so asking someone to actually spend brain cycles considering it can't be anything more than a deliberate attempt to waste their time or insult their intelligence.

Because of this, someone hearing you say "You can get at most once delivery, or at least once delivery; you cannot get exactly once delivery" is entitled to assume that disconnecting the network is not what you meant with regards to the first option.  That assumption is reinforced by the inclusion of at-least-once delivery as a viable option, because that is actually not possible unless the odds of a message being delivered are greater than zero.  Likewise, your interlocutor is entitled to assume that the reliability of the network is less than 100%, not because you explicitly said so, but because networks are unreliable in point of actual fact, and also because that is the only thing that makes this topic worth discussing at all, and the only thing that makes the impossibility of exactly-once delivery even remotely plausible.  In a 100% reliable network you can insure exactly-once delivery by (wait for it!) ... sending each message exactly once!

Likewise, "you cannot have exactly-once-delivery, but you can have exactly-once processing" implies that there is a meaningful distinction to be drawn between "delivery" and "processing".  There isn't.  Processing is ubiquitous in modern networks.  Every router does it, and obviously every end-point does it.  Any distinction between "delivery" and "processing" that would make exactly-once delivery impossible would necessarily beg the question by stipulating that discarding duplicate messages is "processing".  You can't make this distinction in any other way because discarding duplicates can happen anywhere in the abstraction stack, including the device driver, or even in the NIC hardware.  No one does it that way, but only because it's easier and more convenient to discard duplicates in the network stack or the application, not because it's impossible to do it anywhere else.

So yes, you can have exactly-once delivery.  That may not be what you want.  In fact, it is almost certainly not what you want for anything that is actually mission-critical, because a lot can go wrong downstream of message delivery.  But that is a different question.  If you want it, you can absolutely have it.

Just for the sake of completeness I should point out that removing duplicates at the receiver is a pretty extreme oversimplification of what you would do in practice to provide exactly-once delivery.  A complete solution would almost certainly be an example of Greenspun's Tenth Law applied to the TCP protocol rather than Common Lisp.

Final Thoughts

This post was intended to be about human communication more than distributed systems or network protocols, so I want to address one last question: How did this myth of the impossibility of exactly-once delivery get so firmly established?  This is a genuine mystery to me.  I can find no academic citation anywhere, only a few blog posts, none of which have any references.  So I can only speculate.  My guess is that it stems from the fact that distributed systems are monstrously complicated both in theory and in practice, and there are some things that you really cannot guarantee in the face of certain kinds of failures.  But that is true of non-distributed systems too.  Distributed systems just amplify the problems, they don't introduce them.  Even in a non-distributed system you always have the possibility that a cosmic ray will flip a bit somewhere.  Yes, you can have ECC, but you can also have two cosmic rays, or a manufacturing defect, or sabotage, or a bug.  In the real world you can never eliminate all risks.  The best you can hope for is to reduce them to the point where you like the odds.  That's usually good enough.

That is even the case for human endeavors like this blog post.  It's possible that I've overlooked something and that my argument is wrong.  But I've thought about it, and I've done some research, and I'm able to account for all the data, so I'm casting my lot and publishing this, fully aware of the possibility that someone might point something out in the comments that will make me slap myself on the forehead and go "duh".  But even then I will count that as a good outcome because I will learn something.

That's how the scientific method works.

Information, Data, and Knowledge

(This is part 10 in a series on the scientific method.)

In 1966, well within living memory as I write this in 2024, Digital Equipment Corporation released the PDP-10, later rebranded as the DECsystem-10 or, more colloquially, the DEC-10.  The base model cost over $100,000 in 1966 dollars, well over a million dollars today.  For that price you got 8,192 words of memory, each being 36 bits wide, or just a hair over 32 kilobytes by modern reckoning.  A top-of-the-line model gave you about 144 kilobytes (32,768 36-bit words) of memory and cost three times the base model price.  At the time, the DEC-10 was considered a "low cost" computer.  Nowadays a computer with vastly superior specifications fits on a board the size of your little finger and can be purchased at retail for less than less than one 1966 dollar.  If you buy just the chip in modest volume, you can get them for less than one 2024 dollar.

Notice that despite decades of technological improvement, the specifications of a computer's capabilities were given in essentially the same terms in 1966 as they are today, namely, by specifying the amount of memory it contains, and this will always be the case.  Memory has a utility that transcends technology and culture and the whims of fashion: it can store information.

Information is both ubiquitous and ineffable.  We live in an "information age", inundated by information around the clock, but almost no one understands what it actually is.  It's challenging to find a definition that is accessible and not circular.  Wikipedia says, "Information is an abstract concept that refers to something which has the power to inform," which doesn't seem particularly helpful or enlightening to me.

So let's set this as our Problem: there is this thing, information, that everyone talks about, that permeates our modern existence, but that no one seems to actually understand.  Why?  Is information real, an essential part of objective reality, or is it just a social construct like the law or the value of fiat currency?  What is information made of?  Can it be created and destroyed like computers and chairs, or is it conserved like mass and energy?

Let's start by taking stock of some of the (apparent) properties of this thing we're trying to explain.

Information does not exist by itself.  It is always contained in in or bound to something, like a sheet of paper or a computer memory or a DNA molecule or a brain.  This container doesn't necessarily need to be a material object made of atoms, it can be an electromagnetic wave.  But it has to be something.  There is no such thing as information in isolation, not bound to any material thing.

Despite the fact that this containment is essential, the identity of a piece of information is independent of its binding.  The document you are reading right now is in some sense "made of" information.  Most likely, you are reading it on a computer, so that information is contained in a computer memory.  It is transferred into your brain -- i.e. the information is copied -- through the act of reading.  You could likewise copy the information in this document onto a sheet of paper by printing it, or into sound waves in the atmosphere by reading it aloud.  But regardless of whether the information in this document is contained in a computer memory or a sheet of paper or your brain or the air around you, it is, at least in some sense, the same information in all four cases.

How much can information be changed and still be "the same information"?  If this document is translated into a different language, is it still "the same information"?  What about if it is compressed or encrypted?  When information is compressed or encrypted, the result bears virtually no resemblance to the original.  (In the case of encryption, that is the whole point!)  But in each case, there is still some sense in which the result is "the same" as the original.  A compressed or encrypted version of this document is still unambiguously (it would seem) a version of this document and not a version of any other document.

I'm going to put a pin in that for now and invite you to consider a different question, which will eventually lead us to the answers to the preceding ones: what is the difference between information and data?  To persuade you that there is indeed a difference, here is an example of some data:

77 72 17 56 75 22 50 76 20 49 29 16 4 61 33 71 87 65 56 92

The difference between that and all of the surrounding text should be obvious: the text conveys some sort of "meaning" while the data is "just a bunch of numbers".  There might be some meaning there, but it's not readily apparent the way it is with the surrounding text.

So the difference between "information" and "data" is that information has "meaning" and data doesn't.  But what is "meaning"?  That word seems at least as troublesome to define as "information", and so reducing "information" to "meaning" doesn't represent a lot of progress.  But in this case looks are deceiving because the meaning of "meaning" can actually be made precise.  To see how, consider another example:

55 83 56 74 55 70 55 73 56 75 54 77 54 72 55 68 54 66 52 70

At first glance, this too looks like "just a bunch of numbers" but if you look closer you might start to notice some patterns in the second list that weren't there in the first.  For example, if you consider the list as pairs of numbers, the first number is always smaller than the second.  If you look at the first number in the pair, they all fall in a fairly narrow range: 52-56.  The second number in the pairs also fall in a narrow range: 66-83.  That might be enough for you to figure out what these numbers actually are without my telling you: they are the low and high temperature forecasts for a particular location (Redwood City, California) on a particular date (September 8, 2024) according to the Apple Weather app on my iPhone.  So those numbers contain information.

The difference between data and information is that information has a referent -- it is about something.  In the case of the second example, the referent is a weather forecast.  Notice that in order to make sense of that information, to extract its meaning, requires that you know something about what numbers are, how they are represented as marks on a page (or a computer screen), and how those numbers relate to temperatures and geography and time.  But the essential feature, the thing that confers information-ness to those numbers, is that they have a referent, that they correspond to something, in this case, temperatures.

It is this correspondence that is the defining characteristic of information.  Information is a correspondence between two physical things, like marks on a page and current events, or the reading on a thermometer and the temperature of the surrounding material, or the order of base pairs in a DNA molecule and the shape of a protein.

Notice that recognizing this correspondence can require quite a bit of effort.  There is nothing at all obvious about the relationship between the shapes of numerals and temperatures, nothing about the visual appearance of "23" that corresponds to "cold", or the visual appearance of "98" that corresponds to "hot".  To see the correspondence requires going through a lot of intermediate steps, like learning what numbers are, understanding what "hot" and "cold" are, what a temperature is, and so on.  There is nothing at all obvious about the relationship of base pairs in a DNA molecule and the shapes of proteins.  That correspondence goes through a relatively straightforward mapping of base pairs to amino acids, and then a much more complicated mapping of sequences of amino acids to shapes in three-dimensional space.  There are similar processes at work in compressed and encrypted data.  To see the correspondence between compressed and uncompressed data you have to know the compression algorithm.  To see the correspondence between encrypted and unencrypted data you have to know both the algorithm and the encryption key.

There is one additional requirement for data to qualify as information: the correspondence between the information and its referent must be causal.  It can't be simply due to chance.  I could produce a plausible-looking weather forecast simply by picking numbers at random.  That forecast might even be correct if I got lucky, but it would not contain any information.  The correspondence between the forecast and the actual future temperatures has to be the result of some repeatable process, and the correlation between the information and its referent has to be better than what could be produced by picking numbers at random.

How can you be sure that some candidate information is actually information and not just a coincidence?  You can't!  The best you can do is calculate the probability that a correlation is a coincidence and decide if those odds are small enough that you are willing to ignore them.  Calculating those odds can be tricky, and it's actually quite easy to fool yourself into thinking that something is information when it's not.  For example, there is a classic scam where a stock broker will send out market predictions to (say) 1024 people.  512 of the predictions say a a stock will go up, and 512 say it will go down.  For the group that got the correct predictions he sends out a second set of predictions.  256 people get an "up" prediction, and 256 get "down".  This process gets repeated eight more times.  At the end, there will be one person who has seen the broker make correct predictions ten times in a row.  The odds of that happening by chance is 1 in 1024.  Of course it did not happen by chance, but neither did it happen because the broker actually has a way of predicting which way a stock will move.  The existence of information depends on the assumption that the universe is not trying to scam us, which seems like a reasonable assumption, but we can't actually know it for certain.

The existence of information depends on another assumption as well.  Information is a correlation of the states of two (or more) systems, but what counts as a "system"?  There are some choices that seem "natural", but only because of our particular human perspective.  A bit in a computer memory, for example, only looks like a binary system because that is how it is intended to be viewed.  In actual fact, computer memories are analog systems.  When a 0 changes to a 1, that change does not happen instantaneously.  It takes time for the electrons to move around, and while that is happening the state of that bit is neither 0 nor 1 but somewhere in between.  We generally sweep that detail under the rug when we think about computer memory, but that doesn't change the fact that the underlying reality is actually much more complicated.

Some aspects of the apparent behavior of information actually depend on this.  It appears that information can be created and destroyed.  Human creativity appears to create information, and human carelessness or malice appears to destroy it.  There's a reason that backups are a thing.  But if we view the universe as a whole, and if our current understanding of the laws of physics is correct, then information can neither be created nor destroyed.  The currently known laws of physics are deterministic and time-symmetric.  If you knew the current state of the universe, you could in theory project that forward and backward in time with perfect fidelity.  When you "destroy" information, that information isn't actually destroyed, it is merely "swept under the rug" by some simplifying assumption or other.  When you "create" new information, it is not really new, it is pre-existing information being changed into a form that we can recognize under a certain set of simplifying assumptions.

It is tempting then to say that information is in some sense not real but rather a figment of our imagination, an artifact of some arbitrary set of simplifying assumptions that we have chosen for purely economic reasons, or maybe for political or social reasons.  Choosing a different set of simplifying assumptions would lead us to completely different conclusions about what constitutes information.  And that is true.

But our simplifying assumptions are not arbitrary.  The reality of information is grounded in the fact that everything you know is bound to one particular physical system: your brain.  You can imagine taking a "God's-eye view" of the universe, but you cannot actually do it!  In order to take the God's-eye view you would need a brain capable of containing all of the information in the universe, including the information contained in your own brain.  God might be able to pull that trick off, but you can't.

Could we perhaps build a machine that can take the God's-eye view?  At first glance that would seem impossible too for the same reason: that machine would have to contain all of the information in the universe.  But the machine would itself be part of the universe, and so it would have to contain a copy of all of the information contained within the machine, including a copy of that copy, and a copy of the copy of the copy, and so on forever.

There is a clever trick we can do to get around this problem.  See if you can figure out what it is.  Here is a hint: suppose you wanted to make a backup of your hard drive, but you didn't have any extra storage.  If your drive is less than 50% full this is easy: just copy the data on the used part of the drive onto the unused portion.  But can you think of a way to do it if your drive is full?)

Notwithstanding that it might be possible in principle to build a God's-eye-view machine, there are two problems that make it impossible in practice.  First, the engineering challenges of actually collecting all of the information in the universe are pretty daunting.  You would need to know at the very least the position of every subatomic particle, including those inside planets and stars in galaxies billions of light years away.  We can't even access that information for our own planet, let alone the countless trillions of others in the universe.  And second, throughout this entire discussion I've assumed that the universe is classical, that is, that it is made of particles that exist in specific locations at specific times.  It isn't.  At root, our universe is quantum, and that changes everything in profound ways.  Quantum information has a fundamentally different character from classical information, and the biggest difference is that quantum information is impossible to copy.  If you could do it, you could build a time machine.  But that's a story for another day.

The takeaway for today is that ignorance is a fundamental part of the human condition.  Indeed, as we will see when we finally get around to talking about quantum mechanics, ignorance is actually the mechanism by which the classical world emerges from the underlying quantum reality.  (Note that this is a controversial statement, so take it with a big grain of salt for now.)  But even in a classical world, taking the God's-eye view, while possible in principle, is impossible in practice, at least for us mortals.

Can we mere mortals ever actually know anything?  No, we can't, not with absolute certainty.  We are finite beings with finite life spans and finite brains.  We can only ever be in possession of a finite amount of data, and that data will always be consistent with an infinite number of potential theories.   We can never be 100% certain of anything.  It is always possible that tomorrow we will discover that our entire existence is some kind of long con -- a simulation, maybe, running in a tiny corner of a vast data center built by some unfathomably advanced alien civilization, and the only reason that the laws of physics appear to be the same from day to day is that no one has bothered to update our software.  But who knows what tomorrow may bring?  Maybe you or I will live to see the upgrade to laws-of-physics 2.0.

But I'll give long odds against.

The Trouble With Big Numbers

This is part of my series on the scientific method, but it's a bit of a tangent, an interlude if you will, so I'm not giving it a number.  As you will see, that will turn out to be metaphorically significant.  I'm writing this because my muse Publius raised the problem of infinity in comments on earlier installments in this series, and so I thought it would be worth discussing why these are problematic for mathematics but not for science.

(BTW, the title of this post is an allusion to something I wrote five years ago, which itself was an allusion to something I wrote fifteen years ago.  I guess I'm just good at finding trouble.)

There is an old joke that goes something like this: one cave man says to another, "I'll bet you that I can name a bigger number than you."  The second cave man responds, "You're on.  What's your number?"  The first caveman says triumphantly, "Four!"  The second cave man thinks for a while and finally says, "You win."

The joke is not just that the second cave man couldn't count to five, but that it was a silly game to begin with because second player can (it would seem) always win by simply taking the number that the first player names and adding one.  It seems obvious that you should be able to do that no matter what the first player says because otherwise there would have to exist a counter-example, a number to which it is not possible to add 1, and obviously there is no such counterexample, right?

Well, sort of.  There are systems of arithmetic where four actually is the biggest number.  For example, in modulo-5 arithmetic it is possible to add 1 to 4, but the result is zero.  In modulo-5 arithmetic, four actually is the biggest number.

But this is obviously silly, notwithstanding that modular arithmetic really is a legitimate mathematical thing with lots of practical applications.  There is obviously a number greater than four, namely five, the very number we had to deploy to describe the system in which there is no number greater than four.  In fact, to describe a system of modular arithmetic whose biggest number is N we have to use a number one bigger than N.  So this argument seems self-defeating.

There is another way to construct a system of arithmetic with a biggest number, and that is to simply stipulate that there is a biggest number, and that adding one to this number is just not allowed.  Again, this might feel like cheating, but if we are using numbers to count actual physical objects, then there is already a smallest number: zero.  So why could there not be a biggest one?

But this still feels like cheating, because if we can name the number that we want to serve as the biggest number, we can obviously (it would seem) name a number that is one more than that.  So unlike zero, which is kind of a "natural" choice for a smallest number, there is no apparent "natural" choice for a biggest number.  We can try playing tricks like "one more than the biggest number that we can actually name", but that is simply a fun paradox, not an actual number.

So it would appear that logic leaves us no choice but to accept that there is no biggest number, and so we have to somehow deal with apparently inescapable fact that there are an infinite number of numbers.  But that leads to problems of its own.

Imagine that you have three buckets, each of which is capable of holding an infinite number of balls.  Bucket #1 starts out full of balls while the other two are empty.  You now proceed to execute the following procedure:

1.  Take three balls out of bucket 1 and put them in bucket 2.

2.  Take one ball out of bucket 2 and put it in bucket 3.

3.  Repeat until bucket 1 is empty.

That third step should make you a little suspicious.  I stipulated at the outset that bucket 1 starts out with an infinite number of balls, and so if you try to empty it three balls at a time it will never be empty.  But we can fix that by speeding up the process: every time you go through the loop you have to finish it in half the time you took on the previous step.  That will let you perform an infinite number of iterations in a finite amount of time.  Again, you need to suspend disbelief a little to swallow the idea of doing every step twice as fast as the previous one, but you needed to do that when I asked you to imagine a bucket that contained an infinite number of balls in the first place, so having to deploy your imagination is already part of the game.

The puzzle is: when you finish, how many balls are in bucket #2?

The "obvious" answer is that there are an infinite number of balls in bucket #2.  For every ball that gets removed from B2 and put in B3 there are two balls left behind in B2.  So after every step there must be twice as many balls in B2 as B3.  At the end there are an infinite number of balls in B3, so there must be even more -- twice as many in fact -- left behind in B2.

And this is our first hint of trouble, because there is no such thing as "twice infinity".  If you multiply the number of counting numbers by 2 -- or any other finite number -- the result is equal to (the technical term is "can be put in one-to-one correspondence with") the number of counting numbers.

But now imagine that as we take the balls out of B1 and put them in B2 we mark them to keep track of the order in which we processed them.  The first ball gets numbered 1, the second one gets numbered 2, and so on.  Now when we pull them out in step 2, we pull them out in order: ball number 1 gets pulled out first, ball #2 gets pulled next, and so on.  If we do it this way, then bucket 2 will be EMPTY at the end because every ball will have been pulled out at some point along the way!  (In fact, we can modify the procedure to leave any number of balls in bucket 2 that we want.  Details are left as an exercise.)

So clearly things get weird when we start to think about infinity.  But actually, when dealing with large numbers, things get weird long before we get anywhere close to infinity.

There is a famously large number called a googol (the name of the Google search engine is a play on this).  It is a 1 followed by 100 zeros, i.e.:

10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

Things are already getting a little unwieldy here.  Is that really 100 zeros?  Did you count them?  Are you sure you didn't miss one or count one twice?  To make things a little more manageable this number is generally written using an exponent: 10^100.  But notice that we had to pay a price for shortening a googol this way: we lost the ability to add one!  In order to write down the result of adding 1 to a googol we need to write 99 zeros:

10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001

One could argue for writing 10^100+1 instead, but that doesn't work in general.  Consider adding 1 to:

3276520609964131756207215092068230686229472208995125701975064355186510619201918716896974491640125539

which is less than a third of a googol.

But a googol is not even close to the biggest number the human mind can conjure up.  Next up is a googolplex, which is a 1 followed by a googol of zeros, i.e. 10^(10^100).  Adding one to that without cheating and just writing 10^(10^100)+1 is completely hopeless.  There are fewer than a googol elementary particles in our universe (about 10^80 in fact) so it is simply not physically possible to write out all of the digits in a googolplex.  Even if we allowed ourselves to re-use material we couldn't do it.  Our universe is only about 13 billion years old, which is less than 10^18 seconds.  The fastest conceivable physical process operates on a scale called the Planck time, which is the time it takes for light to travel the diameter of a proton, about 10^-43 seconds.  A single photon with a cycle time this short would have an energy of about 6.6 gigajoules, a little under two megawatt-hours, the energy equivalent of about one and a half tons of TNT.  So even if we could build a computer that ran this fast it could not process the digits of a googolplex in the life span of the universe.

An obvious pattern emerges here: after 10^100 (a googol) and 10^(10^100) (a googolplex) comes 10^(10^(10^100)), a one followed by a googolplex of zeros.  That number doesn't have a name, and there's not really any point in giving it one because this is clearly a Sisyphean task.  We can keep carrying on forever creating bigger and bigger "power towers": 10^(10^(10^100)), 10^(10^(10^(10^100))) and so on.

What happens if we start to write power towers with large numbers of terms, like 10^10^10^10... repeated, say, 1000 times?  To keep those from getting out of hand we have invent yet another new notation.  Designing such a notation gets deep into the mathematical weeds which I want to steer clear of, so I'm just going to adopt (a minor variant of) something invented by Donald Knuth called up-arrow notation: A↑B means A^(A^(... <== B times.  So 10↑5 means 10^(10^(10^(10^(10^10)))).  10↑10 is already too unwieldy for me to want to type out.  But even at 5 iterations it is already challenging to communicate just how vast this number is.  I can explicitly expand it out one level (10^(10^(10^(10^10000000000)))) but not two -- that would require about ten gigabytes of storage.  Expanding it out to three levels would require more resources than exist in this universe.  Four is right out.

But we are nowhere near done stretching the limits of our imagination.  Up-arrow notation can be iterated: A↑↑B means A↑(A↑(A... <== B times.  A↑↑↑B means A↑↑(A↑↑(A... <== B times, and so on.  And this allows us to arrive -- but just barely -- at one of the crown jewels of big numbers, the famous Graham's number, which uses a number of up-arrows so big that it itself needs to be described using up-arrow notation.

Graham's number is already mind-bogglingly vast, but it's a piker compared to TREE(3).  I'm not even going to try to explain that one.  If you're interested there's a lot of material about it on the web.  And even that is only just getting started.  Everything I've described so far is still computable in principle (though not in practice).  If we had a big enough computer and enough time then we could in principle actually write out all the digits of Graham's number or even TREE(3), or even TREE(TREE(3)) -- these patterns go on and on and on.  But beyond these lie even bigger numbers which are uncomputable even in principle, though it's easy to show that they must exist.  That will have to wait until I get around to discussing the halting problem.  For now you will just have to take my word for it that there are a series of numbers called busy beavers which are provably larger than any number I've described so far, or even any number that can be constructed by any combination of techniques I've described so far.  TREE(TREE(TREE.... iterated Graham's number of times?  Busy beavers are bigger, but every one of them is nonetheless finite, exactly as far away from infinity as zero is.

And it gets even crazier than that.  Let's suspend disbelief and imagine that we can actually "get to infinity" somehow.  We're still not done, not by a long shot.  It turns there are different kinds of infinity.  The smallest one is the number of counting numbers or, equivalently, the number of ways you can arrange a finite set of symbols into finite-length strings.  If you allow your strings to grow infinitely long then the number of such strings is strictly larger than the number of finite-length strings.  And you can keep playing this game forever.  The number of infinite two-dimensional arrays of symbols is strictly larger than the number of one-dimensional infinite strings of symbols, and so on and so one.

But wait, there's more!  All this is just what we get when we treat numbers as measures of quantity.  Remember the balls-and-buckets puzzle above, and how things got really strange when we allowed ourselves to paint numbers on the balls so we could distinguish one ball from another?  It turns out that if we think about adding one as not just an indicator of quantity but of position that we can squeeze different kinds of infinities in between the ones we just constructed above.  If we think of adding one as producing "the number after" rather than just the number that is "one more than" then we can introduce a number that is "the number after all the regular counting numbers".  Mathematicians call that ω (the lower-case Greek letter omega).  Then we can introduce the number after that (ω+1) and the number after that (ω+2) and so on until we get to ω+ω=2ω.  And of course we can go on from there to 2ω+1, 2ω+2... 3ω, 4ω... ω*ω, ω^3, ω^4, ω^ω, ω↑ω, ω↑↑ω and so on until we get to ω with ω up-arrows, which mathematicians call ε₀.  And then the whole game begins anew with ε₀+1, ε₀+ω, ε₀+ε₀...

These kinds of infinities are called transfinite ordinals, and they have two interesting features.  First, the "size" of each of these numbers, that is, the number of numbers between zero and any one of them, is exactly the same as the number of regular counting numbers.  If we think about numbers as referring to position, then each ordinal is "bigger" than the one before, but if we think about them as referring to quantity then each one is exactly the same "size".  And second, the game of inventing new ordinals does not have a regular pattern to it.  It requires creativity.  The bigger you get, the harder it becomes to define what "add one" means.  It gets so hard, that mathematicians who figure out how to do it have the resulting numbers named after them.

The study of big numbers, both finite and infinite, is a deep, deep rabbit hole, one that ultimately leads to Turing machines and the theory of computation, which is the deepest rabbit hole of all.  It's fascinating stuff, and well worth studying for its own sake.  But is any of this relevant for science or is it just an intellectual curiosity?

Until and unless we develop a "theory of everything" that allows us to predict the result of any experiment, we cannot rule out the possibility that this theory will involve very large numbers (by which I mean numbers that require power towers or beyond to represent), and possibly even infinities.  But so far this has not been the case.  There are only two situations in our current best theories where infinities arise, and in both of those cases there is every reason to believe that this is an indication of a problem with the theory and not a reflection of anything real.

Just in case you were wondering, those two situations are singularities inside black holes and self-interactions in quantum mechanics.  In the case of singularities, general relativity predicts their existence while at the same time predicting that they can never be observed because they always lie inside the event horizon of a black hole.  In the case of self-interactions, it just turns out that you can basically just ignore them.  If you simply throw out the infinities when they pop up, everything works out and the predictions made by the theory just turn out to be 100% accurate.  No one knows why, but that's the way it is.

But there is another situation where a kind of infinity pops up which is not so easily dismissed.

Suppose you travel exactly one mile in a straight line, then turn exactly 90 degrees and travel another mile, again in a straight line.  If you then want to return to your starting point by traveling in a straight line, how far would you have to go?

This innocuous-seeming question is the gateway to a whole series of major mathematical and philosophical problems.  You can get the answer by applying Pythagoras's theorem (which, BTW, was almost certainly not discovered by Pythagoras, but that's another story): it's the square root of 2.  The problem arises when you actually try to write this quantity out as a number.  The answer obviously has to be somewhere between 1 and 2, so it's not a whole number.  It also has to be somewhere between 14/10 and 15/10 because (14/10)^2 is 196/100, which is a little less than 2, and (15/10)^2 is 225/100, which is a little more than 2.  We can keep narrowing down this range to smaller and smaller intervals, but we can never find a ratio of two integers whose square is exactly 2.  The square root of 2 is an irrational number.  If we wanted to write it out exactly, we would need an infinite number of digits.  We haven't even gotten past the number 2 and simple multiplication, and yet somehow infinity has managed to rear its ugly head.

I opened this post with a joke, so I'll close with another: a farmer hired a physicist to design a machine to shear sheep.  After a few weeks the physicist submitted his report.  It began, "Assume a spherical sheep."

It's funny because, of course, sheep aren't spherical.  But there is a metaphorical spherical sheep hiding in our problem statement.  It's in the phrase, "exactly one mile"  (and also "exactly 90 degrees").  In order for the mathematical irrationality of the square root of 2 to matter in the physical situation I have described it really is critical to travel exactly one mile.  If you deviate from this in the slightest then the distance back to your starting point can become a rational number, representable numerically with no error in a finite number of symbols.  For example, if either leg of your journey is exactly one part in 696 longer than a mile then the return trip will be exactly 985/696 miles.

In fact, for any number you care to name, no matter how small, I can give you a smaller number such that adding that much distance to one leg of the trip will make the return distance be a rational number.  That means that if you odometer has any error at all, no matter how small, the return distance could be rational.

Of course, "could be" is not the same as "is".  It's possible that the actual underlying physical (or even metaphysical) reality is truly continuous, and actually does require an infinite number of symbols to describe.  But here is the important question: how could you ever possibly know?  What experiment could possibly demonstrate this?  In order to know whether physical reality is truly continuous you would need to somehow obtain an infinite amount of data!  To be able to tell, for example, whether our three-dimensional space is truly continuous, you would need to be able to measure a length to infinite precision.  How would you do that?  Forget the problems with actually designing such a device and how you would get around (say) thermal fluctuations and the Heisenberg uncertainty principle.  I grant you arbitrarily advanced technology and even new physics, and simply ask: what would the output of such a measuring device look like?  It can't be a digital display; those can only output a finite number of possible outputs.  So it would have to be some kind of analog output, like an old-school volt meter.  But that doesn't help either, because to read an analog gauge you still have to look at it, and your eye doesn't have infinite resolution.  And even if your eye had infinite resolution, you would still have to contend with the fact that the needle of the gauge is made of atoms which are subject to thermal fluctuations.  And if you tried to solve that problem by cooling the gauge down to absolute zero, you still ultimately have to contend with the Heisenberg uncertainty principle.  (Yes, I know I granted you new physics, but you still have to be consistent with quantum mechanics.)

The ultimate fact of the matter is that no matter how hard we try, no matter what technology we invent, no matter how many resources we deploy, we will only ever have a finite amount of data, and so we can always account for that data with finite explanations.  We can imagine infinities.  They might even pop up unbidden in our mathematical models.  But when they do, that will almost certainly be an indication that we've done something wrong because we can know for sure that neither infinite quantities nor infinite precision can ever be necessary to explain our observations.  In fact, we can calculate pretty easily the amount of data our universe can possibly contain, and it's a tiny number compared to what the human imagination is capable of conceiving.

Infinities are like wizards and unicorns.  They're fun to think about, but they aren't real.