No, Science is Not Just Another Religion

I want to debunk once and for all this idea that "science is just another religion".  It isn't, for one simple reason: all religions are based on some kind of metaphysical assumptions.  Those assumptions are generally something like the authority of some source of revealed knowledge, typically a holy text.  But it doesn't have to be that.  It can be as simple as assuming that there must be some kind of purpose or meaning to existence, assuming that the question "Why are we here?" or "Why is there something rather than nothing?" actually has a satisfying answer.  There is nothing inherently wrong with making such assumptions, but there is something wrong with trying to deny that you've made them, or assuming that everyone else has made them too.

Science makes no metaphysical assumptions.  I'll often hear people say that science assumes materialism or atheism or some such thing, but that is wrong.  Science is just the process of seeking explanations that account for observations.  It makes no assumptions.  It does not assume materialism.  It does not assume the existence of objective reality.  It does not assume that satisfactory explanations of observations even exist.  It turns out that satisfactory explanations do exist.  We know this because we have actually found them.  And it turns out that these explanations do not require deities or mind-body dualism or divine revelation.  All they require is the right math.

It turns out that with the right math we can account for all phenomena that are observed to occur here on earth.  (I hedge with "here on earth" because we do not yet have complete explanations of some cosmological phenomena.)  At the foundation of modern science is something called the Standard Model, which describes the behavior of all known fundamental particles.  The predictions made by the Standard Model agree with all observations.  Not a single observation of a phenomenon here on earth has ever disagreed with the predictions of the Standard Model.  Indeed, this has precipitated a major crisis in theoretical physics because for the last 50 years since the Standard Model was formulated, particle physicists have had no idea what to do with themselves.

None of relies on any assumptions.  It is all empirically observed fact, which you verify for yourself every time you use a piece of modern technology.  Every time you use a phone or a computer you are doing an experiment that checks the predictions of the Standard Model.

Does this prove that God does not exist, or that there is no afterlife?  No.  Nothing is ever proven in science.  Science does not produce proofs, it produces explanations.  But these explanations have one very important feature: they converge.  Throughout its history, the progress of science has been monotonic.  Even when major revolutions have happened, when radically new explanations have displaced old ones, it turns out that the old explanations were actually reasonable approximations to the new under certain circumstances.

Whatever this thing is that science is empirically observed to be converging towards is called "scientific truth", not because science makes any claim to uncover metaphysical Truth with a capital T, but simply because it's a convenient and intuitively plausible label.  We say that it is true that the earth is round, that it revolves around the sun, that it is made of atoms, despite the fact that it is possible that none of these things are metaphysically True.  Indeed, the Standard Model tells us that the statement "matter is made of atoms" is just an approximation to the truth, just as Newtonian gravity is just an approximation to the truth.

But these approximations have a very important property: they are very good approximations, in the sense that they give us the ability to predict certain future events with vastly greater accuracy than any other method ever devised by man.  Astrology, tarot, reading auras -- none of these even come close to the predictive power of the scientific method.  (Indeed, all of these methods have more or less the same predictive power, which is to say, none at all.)

The fact that naturalistic explanations can account for all phenomena that are observed to occur here on earth, and that they give us the gift of prophecy, is hard-won knowledge, pieced together bit by bit by many thousands of people working over centuries.  This progress was punctuated by major breakthroughs made by people whose names are immortalized in history: Galileo.  Copernicus.  Kepler.  Newton.  Darwin.  Einstein.  But there were countless others who toiled anonymously to bring this Promethean gift of knowledge to mankind.  To dismiss all this hard work as "just another religion", or that "atheism requires faith" is an insult to their hard work.  It is every bit as insulting as saying that Christians worship a zombie, or that Mohamed was a pedophile.  (It is actually even more insulting because both of those things are technically true!)

Science demands no faith.  In fact it is the exact opposite: science demands skepticism.  Science rejects arguments from authority.  Science has no pope, no ecclesiastical hierarchy.  Scientific academia might have such a hierarchy, but scientific academia absolutely should not be conflated with science!  Science is inherently democratic.  Anyone can do it.  This is not to say that doing science is easy.  It isn't.  It requires diligence and hard work.  But it does not require faith.

I was motivated to write this in part because I often hear Christians try to claim credit for the rise of science in Western Europe during the Enlightenment.  As evidence they point to the fact that nearly all of the Big Names in Enlightenment science were Christians.  Which is true.  Galileo, Copernicus, Kepler, and Newton were all devout Christians.  But virtually everyone in Western Europe was a Christian at that time.  Assigning the credit to Christianity is no more valid than assigning the credit to the fact that they were white men, because they were all that too.

But the history is neither here nor there.  What matters now is what is happening now.  And what is happening right now is that the Catholic Church just elected a pope who once

...lamented that Western news media and popular culture fostered “sympathy for beliefs and practices that are at odds with the gospel.” He cited the “homosexual lifestyle” and “alternative families comprised of same-sex partners and their adopted children.”

That is anti-scientific, not because of the position, but because of the justification: homosexuality is at odds with the Gospel.  That is an argument from authority, which is as violently anti-science as you can get.  So it doesn't matter if the Church was anti-science in the past or not.  What matters is that it is anti-science right now.  And the Church of course is not unique in this.  All religions are inherently anti-science because all religions are based on some metaphysical assumptions.  That is the defining characteristic of a religion.  Science is unique among human intellectual endeavors.  It is the only philosophical foundation that makes no assumptions.  It is nothing more and nothing less than the quest to explain observation.  That turns out to be an extremely powerful lever, which is why, I think, that all other philosophical traditions look on the success of science with envy and try to tear it down.  If your identity or your livelihood revolves around being a person of faith, the success of science can be bad for business.

What did you expect? Of course the Trump administration will defy court orders!

I'm not sure whether to laugh or cry over this:

Federal judges have issued President Donald Trump stinging legal rebukes in the early clashes over his blitz of executive orders, and two of his top aides have responded by suggesting that his administration defy the courts and move forward with its agenda.

There’s no indication that Trump has adopted such a strategy, although a U.S. judge in Rhode Island ruled Monday that the administration has been violating a court order to disburse billions of dollars in already approved grant funding and hinted at possible penalties.

Well, duh!  Of course Trump is not going to obey court orders.  Why on earth should he?  Who is going to make him?  (I have a vague recollection of hearing a supposedly famous quote about the court's armies but now I can't find it.)

This is the saddest part:

But the combative rhetoric by Vice President JD Vance and top adviser Elon Musk has troubled legal experts, who said there is no modern precedent for a president to ignore or defy court orders.  [Emphasis added.]

As if that matters.  In case you hadn't noticed, there is no precedent, modern or otherwise, for anything Donald Trump has done.  Donald Trump's entire brand is built on telling people where they can stick their modern precedents.

There is only one way to make Donald Trump obey a court order and that is to levy a credible threat of removing him from office through the impeachment process.  Good luck with that.

An Open Letter to Democratic Candidates and Organizations

The ink is barely dry on the (drama-free) certification of the election results and I'm already getting calls from organizations and candidates seeking donations.  (Apparently my name is on a lot of lists.)  In order to save everyone a lot of time, I thought I'd write up my current thinking so I don't have to have the same conversation dozens of times.

First and foremost: I really do appreciate the fact that you're out there in the trenches working to make the world a better place.  I have never been personally active in politics beyond writing checks (though not for lack of trying, but that's another story).  Nonetheless I get that it's a ton of shit-work, and I am grateful that you have stepped up to the plate to do it so I don't have to.  I also get that you are probably not part of the real power structure in the Democratic party, and so a lot of what I am about to say doesn't apply to you.  However, if you are soliciting a donation from me, then you are almost certainly a lot closer to that power structure than I am, and so I'm saying these things to you in the hopes that you might pass them along to those who are empowered to effect change.

As far as I can tell, the Democratic party has not yet (to put it mildly) fully taken on board the lessons of the 2024 election.  A lot of the pitches I'm getting are saying that we need to (say) re-take the House in 2026 in order to "have robust oversight on Trump's admin overreaches" (that is a direct quote).  I really don't want to insult the person who wrote that because I have a lot of respect for them, but there seems to me a pretty significant disconnect between that sentiment and reality.  What exactly are you going to do to implement this "robust oversight" that you didn't before?  You had six years to robustly oversee Trump.  You impeached him twice, indicted him I don't know how many time, and even managed to convict him of multiple felonies.  But you never laid a glove on him.  Now he is returning to the White House with blanket immunity from criminal prosecution bestowed on him by the Supreme Court.  At this point he could quite literally shoot someone on Fifth Avenue and get away with it.

So "help us re-take the House so we can reel in Donald Trump" is a non-starter for me.  That ship sailed a long time ago.  Donald Trump is going to the White House with de facto dictatorial power.  No one is going to be able to stop him.

Furthermore, even if we could stop him, I think it would be a mistake to try.  Elections have consequences, and Trump won this one fair and square.  A lot of people -- tens of millions -- are thrilled.  I think it's really important not to lose sight of that.  For these people, MAGA is not just a slogan.  They really do believe in their heart of hearts that Donald Trump is going to Make America Great Again.  I think they're wrong, and I think we are going to learn this lesson the hard way.  But I think we have to learn this lesson the hard way because we certainly haven't learned it the easy way.  Donald Trump did not win by trickery or dissembling.  Everyone who voted for him did so knowing exactly who he is and what he stands for (to the extent that he actually stands for anything).  If you believe in democracy, you have to now let the chips fall.  And who knows, maybe Trump will surprise us and do a good job.  I'll give long odds against, but you know, being wrong about that would be a good outcome, wouldn't it?

But I'm pretty sure I'm not wrong, and that the next four years will be an unmitigated disaster on multiple fronts.  But what we need to do IMO is not to try to stop it (because we can't) but to prepare now to pick up the pieces and try to rebuild from the ashes.

That is not going to be a short-term project.  This situation has been a long time in the making.  Trump is not an aberration.  He is not even really the problem.  He is a reflection of a sea-change in American politics that has been going on for at least 30 years, and arguably 50 or even 100 or more.  This is probably going to be a multi-generational project.  But we have an existence proof in the present situation that multi-generational projects of this magnitude can succeed.

But there is one pre-requisite for success that has me very pessimistic, and that is the need to acknowledge that what we progressives have been doing for the last 30 years has failed — manifestly, definitively, and spectacularly.  Nothing can guarantee success, but there is one thing that almost certainly guarantees failure, and that is to change nothing, to continue to insist that we were right all along, to just wait and bide our time until Trump finally implodes, the American people finally return to their senses, and then try to "get back to normal", back to the way things were.  That won't work, because "the way things were" is what brought us Trump in the first place.  To quote one of my favorite aphorisms of all time by Georg Christoph Lichtenberg: different is not necessarily better, but better is necessarily different.  (It reads better in the original German: Es ist nicht gesagt, daß es besser wird, wenn es anders wird. Wenn es aber besser werden soll, muß es anders werden.)

Ironically, despite being ostensibly liberal, the Democratic party has been extraordinarily resistant to change.  Our leaders have a tendency to hang on to power well past their sell-by dates.  Our standard-bearer for the last four years has been one of the longest-serving politicians in American history.  (Equally ironic for a party purporting to represent the interests of working people and the middle class, Joe Biden comes from the single most corporate-friendly state in the union.  In fact, being a corporate tax-and-liability haven is pretty much Delaware's main industry!)  Chuck Schumer is not exactly a fresh face, and before him we had Diane Feinstein.  AOC, one of the brightest young stars in the party, recently lost a bid to for a leadership role to a 74-year old with cancer.  On the Supreme Court, RBG, an 87-year-old with cancer, declined to step down and allow Barack Obama to appoint a younger and healthier successor, and now Sonia Sotomayor, a 70-year-old with diabetes, has done likewise.  If she dies in the next four years (not at all unlikely) the Supreme Court will have four justices appointed by Trump.  That may not be recoverable.  And it will be almost entirely self-inflicted wounds.

So don't pitch me on "taking back the House is 2026" because Trump is Bad.  I know Trump is Bad.  I've known it for eight years (though it apparently was news to Merrick Garland).  What I want to know is what do you propose to do about it that is different than what you've tried in the last eight years.  Because what you've tried over the last eight years hasn't worked, and I see no reason to think that it will somehow magically start to work now.

I have no illusions that this is going to be quick and easy.  I don't think there is a magic bullet, a single Brilliant Idea that will break the logjam and suddenly usher in a new golden age.  I get that politics is messy and hard.  But there is one thing I would like to see with regards to the process of figuring this all out: if you have an answer to my question, even a half-baked one, don't call me to tell me about it.  Instead, write it down and either send it to me via email or, even better, publish it and send me a link.  There are two reasons for this.  First, I can read a lot faster than you can talk, and so it's a lot more efficient if I can read a document rather than listen to a pitch.  And second, no successful political movement has ever been based on an oral tradition.  This is not to say that oration won't be important too -- it will.  But, as I said, I think picking up the post-Trump pieces will be a multi-generational project, and documents live longer than orators.  It matters that the Declaration of Independence and the Constitution and the Scriptures and Project 2025 are written down and not merely recited.  There is a reason that "it is written" is a thing.

I have some ideas of my own about what needs to change.  In fact, this article started out being about those, and I do plan to publish that eventually but I've decided to defer it for now, again for two reasons.  First, the draft of that version of the article was getting way too long, and second, this is not an area in which I have any actual expertise.  I'm a software engineer and computer scientist, not a politician.  I don't have answers, only ideas.  One of those ideas is that the process of coming up with answers ought to be a discussion, not a dictum.  I'm happy, even eager, to participate in that discussion, but I'm not the right person to lead it.  That would be you, because you are the one stepping up to the plate, not me.

So thank you again for stepping up and volunteering to do this dirty job so I don't have to.  I want to help, I really do.  But don't ask me to write a check just because you're a Democrat, or because you support Democrats, or because you're going to turn out Democratic votes, or because you're going to try to stop Trump.  We tried that.  It didn't work.  Now we need to try something else.

I am (not) a Failure: Lessons Learned From Six (and a half) Failed Startup Attempts

A while back I wrote:

I've had many, many failures in my life.  (Hm, maybe I should write a blog post about that.)

This is that post.  I'm writing it not as a lament, but rather because I've ended up in a good place in life despite my extraordinary track record of failing at just about everything I've ever tried.  If my younger self had heard these stories he might have had a less stressful life knowing that even a long string of failures in specific endeavors doesn't necessarily translate to failure in life.  Maybe some version of my younger self is still out there and will get some similar value out of it.

Intro

Growing up, I had two major life ambitions: to become a tenured university professor, and to found a successful startup company.  My role models were people like Rodney Brooks (who was on my Ph.D. thesis committee) who made a big splash in the AI/Robotics world in the early 1990s with the Subsumption Architecture, and was a co-founder of Lucid and iRobot.

I didn't accomplish either of those goals.  I never even got an interview for an academic job, let alone tenure.  I made six (and a half) attempts to found a startup.  All of them failed.

Failure #0: Professor Ron

In elementary school, my ambition was to be a doctor, specifically a brain surgeon.  That dream died in high school biology class when we had to dissect fetal pigs and I discovered that I was a lot more squeamish than I had thought.  So I traded my surgical scrubs for a computer keyboard and set my sights on AI instead.  Fast-forward to 1991, when a freshly minted PH.D. started sending out curriculum vitae (that's academic-speak for "resumes") to CS departments.  Unfortunately, my timing could not have been worse.  It was the height of AI Winter and the first Bush recession, and the few universities that were hiring were getting hundreds of applications for single openings.  My publication record was not great.  I hadn't published a single journal article.  My main conference publication (PDF is here) was well received, but (apparently) insufficiently academic for academia.  I didn't even get an interview.

So I just stayed where I was, which was JPL.  I've told that story elsewhere so I won't repeat it here.  Suffice to say, my JPL career had ups and downs, but eventually entered a long and steady decline from which it never recovered.  But the reason that my career declined had nothing to do with JPL or any external circumstance, it happened because I gradually came to realize that scientific research was not what I had imagined it to be.  In the real world, research is not a Platonic quest for objective truth. It is, first and foremost, a human endeavor, deeply intertwined with human ambitions and foibles, including my own.  The reason my career lasted as long as it did was that I learned, at least to a certain extent, how to work the system.  I was nominally successful, eventually becoming the most cited AI researcher in all of NASA according to a popular metric at the time (Citeseer).  But as time went by I became less and less proud of the work I was doing, and the entire field seemed mired in a stasis that I felt powerless to dislodge.  So I quit.

Lesson learned: academic research is no panacea.  (I think things are actually worse now than they were in my day.)

Failure #1: FlowNet

I spent the summer of 1990 as a visiting graduate student in Rodney Brooks's lab at MIT.  While there I met a brilliant hardware engineer named Mike Ciholas, who went on to found a company that today employs dozens of people and bears his name.  He pitched an idea to me for a new design for a computer network that he called FlowNet (full text is here).  We built working prototypes.  If they had gone into production they would have had more than an order of magnitude price-performance advantage over the competition at the time.  We had numerous meetings with VC's, but none of them ever gave us a term sheet, probably because they knew what we did not: Fast Ethernet was coming, and although it would be expensive at first, it would quickly get commodified and beat out any potential competition no matter how technically superior it might be.  We built our prototypes in 1993.  Fast Ethernet came out in 1995.  We folded up the tent in 1997.

Lesson learned: direct frontal assaults on infrastructure and standards are very unlikely to succeed.  More worthy players than us have learned this lesson the hard way over the years.

Interlude: Google

In 2000 I went to work for an obscure little Silicon Valley startup company.  I've told that story elsewhere so I won't tell it again, but I wanted to mention it for two reasons.  First, it sets the stage for what was to come, and second, while it was unquestionably a success, it was not my success.  In fact, on a personal level I count Google as a failure as well.  My tenure there was marked by a long series of screwups, and I had a serious case of impostor syndrome.    I lasted a year, but after that I went back to JPL to bide my time until the IPO.

Failure #2: IndieBuyer

The Google IPO allowed me to quit JPL and pursue my entrepreneurial ambitions more seriously.  With the benefit of hindsight I should have moved to Northern California and hung out at Y Combinator, but I had lived in Los Angeles for 17 years at that point, which was longer than any place else I had ever lived before by a wide margin.  By my standards at the time, I had deep roots there.  I have also always had a love for magic and movies, and Los Angeles was the epicenter of both of those industries.  So I decided to try to do something in the entertainment space.

The result of that was a company called IndieBuyer, which was going to be Netflix for independent films.  The idea was that technology was making filmmaking more and more accessible, leading to a long tail of undervalued product that couldn't find an audience because of the bottleneck presented by a distribution system designed for more capital-intensive projects.  The business model was that we would sell DVDs (which were still very much a thing back then) but with a money-back guarantee: if you bought a DVD that we recommended and you didn't like it, you could return it for a full refund.  The returns would provide the data for a recommendation engine that would, we hoped, over time become a better and better predictor of who would like what kinds of movies.  The long-term business model was to intelligently match niche products with niche audiences at scale.

It was a plausible idea, and it might have even worked if we had been better capitalized.  But I was funding it entirely from my own pocket, and after a couple of years with very little growth I lost my nerve and decided to pull the plug.  We did try to find some outside investors, but because we had already launched and weren't showing much traction, we didn't attract any interest.

Lesson learned: it's really hard to make money in the movie business.

Failure #2.5: Evryx

This is the half-failure alluded to in the title.  I count it as half a failure because, although I was actively involved, I was not a founder.  Evryx (which eventually rebranded itself as SnapNow) was founded by another engineer at JPL.  The company was an early pioneer in reverse image search.  It used an algorithm called SIFT to match an image taken with a cell phone to a library of stored images and provide information about whatever was in the image.  I was never entirely clear on what the business model was intended to be, but the technology seemed cool.  I happened to mention it in passing to a friend of mine whom I had met at a political fundraiser.  He looked into it and got sufficiently excited about it to become their CEO.  The founder became CTO.

About a year later the company had a crisis: the founder/CTO was behaving erratically, and my friend, the CEO, decided he had to fire him.  He tapped me to come aboard as acting CTO until a permanent replacement could be recruited.  The company limped along for another year or so but ultimately folded in 2007, in no small measure because of the fallout from the financial crisis that began that year.  The CEO had actually managed to secure another round of funding, but it was voted down at a shareholder meeting, with the (fired) CTO casting the deciding vote against.  The company folded a few weeks later.  The bankrupt entity was eventually acquired at a fire-sale price by a private investor, and I have no idea what happened to it after that.  It was sitting on some potentially valuable IP, so maybe they were able to do something with it.

Lesson learned: even smart people can do incredibly stupid things.  The CTO said he voted against the financing because he didn't like the terms and didn't want to be diluted.  The result was that his intact share of the company ended up being worthless.

(I actually saw a similar thing play out at another company in which I was a passive investor.  The technical co-founders basically torpedoed the company, but in their case they went on to start a new company with the exact same IP and business model.  It was blatantly illegal, but to that point the company was funded entirely by small angel investors, and no one was willing to put up the resources to try to stop them.  So they got away with it.)

Failure #3: iCab

In 2009 my wife and I took a cruise around the Pacific rim.  At one point we ended up in Saigon and needed to get back to the cruise ship, but couldn’t find a taxi.  I thought to myself: this place is chock-full of taxi drivers, all of whom have cell phones, and many of whom have smart phones.  Someone should write an app that would let me broadcast the fact that I'm standing in this location looking for a ride.  It was a brilliant idea.  Unfortunately, Garrett Camp and Travis Kalanick had the exact same idea at the exact same time.  I founded a company called iCab, and they founded a company called Ubercab, which eventually changed its name to Uber.

iCab was stillborn.  It was never even established as an actual company.  I assembled a team, mainly people I knew from Evryx, and wrote a prototype app which actually worked.  But we couldn't figure out a way to procure drivers.  We were in LA and we tried pitching the idea to cab companies, but they were not interested.  The idea of using black car limos never even occurred to me.  With no supply chain, we folded up the tent before we had even pitched it.

Lesson learned: a brilliant idea is not enough.  You have to execute.  And sometimes that requires another brilliant idea.

Failure #4: Virgin Charter (nee Smart Charter)

In 2010 my wife and I decided to get an apartment in Santa Monica so we could spend more time by the beach.  I started hanging out in the offices of a venture capital firm with an office on the Third Street Promenade.  One day a fellow named Scott Duffy walked into the office and asked if he could hire us to do due diligence on an acquisition he was considering.  I don't remember the name of the company, but it collected business intelligence on the private aviation industry.

In the course of conducting this due diligence we learned that the private jet market is incredibly fragmented and inefficient, with dozens of operators most of whom had only a small handful of planes.  At the time there were a few thousand private jets operating as charters, but there were only three operators with fleets of more than 100 planes.  The largest of these was NetJets with (if memory serves) about 250, only about 10% of the total.

To match customers with aircraft, a secondary industry of brokers had cropped up.  If you wanted to charter a jet you would call up a broker, give them your mission parameters, and then they would start making phone calls and sending emails and faxes (yes, fax was still a thing back then) to try to find a matching aircraft.  The turnaround time to get a quote was measured in hours, and actually securing a reservation could take days.

But even worse than the time involved was the fact that when an aircraft took you somewhere, it eventually had to return to its home base.  Most of the time this happened in one of two ways: you either paid to have the aircraft (and its crew) wait for you and take you back, or you had to pay for two round trips.  About 40% of legs flown by private jets were flown empty.  This led to a secondary market of brokers trying to sell these empty legs, which had already been paid for.  If you were lucky and happened to find one that was flying a route that you wanted to travel, you could pick one up for a song.  The empty flight was already paid for, so any revenue at all was pure profit.

The problem is that most of the customer base for private jets want to go where and when they want to go, not where and when a plane happens to be available, so it was rare to actually sell an empty leg.  But if you've ever wanted to fly on a private jet and you are flexible with regards to your timing and route, you could (and still can) find some very good deals.

One man's inefficiency is always another man's business opportunity.  I realized that the problem was the lack of a centralized clearing house for demand.  If we could somehow collect all the data on who wanted to go where and when, we could route the fleet much more efficiently.  We crunched the numbers and it turned out to be a billion-dollar business -- if we could somehow pull it off.  The problem was that the customers and the suppliers all depended on the brokers.  In order to succeed we wold need to wrest both away.  How to do it?

I came up with the following diabolical plan: build a web site that would let people enter their trip details, and then give access to that data to the brokers as a lead generation service for a very modest fee.  The fee was actually a red herring to disguise the fact that we were secretly plotting to put the brokers out of business.  It would be an offer they couldn't refuse: more customers for a small amount of money.  But once we reached a critical mass of users, we would start phase two: offer that lead data directly to charter operators so they could bypass the brokers.  Again, it would be an offer they could not refuse.  We would offer these leads to the operators for free, and it would be a much faster and more efficient way of getting bookings because the whole process would be automated.  As a bonus, they could pocket what would have been the broker's commission.

Finally, once we had a critical mass of operators on board, we would start phase 3, a centralized scheduling service that would dispatch the entire fleet in order to minimize the number of empty legs being flown.  Again, it would be an offer they couldn't refuse.  Anyone who didn't subscribe would have to charge significantly more for the same product in order to pay for the additional empty legs, and market forces would eventually drive all the customers to us.

To this day I'm pretty sure that plan would have worked if we had actually executed it.  It was a brilliant plan if I do say so myself.  In fact, it was so brilliant that it convinced Richard Branson to acquire the company before we launched for $10M.  We started the company as Smart Charter, but we launched as Virgin Charter.

I vehemently opposed the acquisition, but I was overridden by my two co-founders.  When I lost that fight, I resigned from the company, but I kept my shares on the off chance that my intuitions were wrong and Virgin would actually make it work.  Unfortunately, my fears were well-founded.  As soon as they took over, they changed the business model to be directly consumer-facing, because that is Virgin's strength.  Instead of a stealthy three-phase plan, they went for a direct assault on an established industry with very little understanding of the actual dynamics of that industry.  Charter jets are a very rarified market.  There's a lot of dollars involved, but not a whole lot of people, and those people don't like to be treated like ordinary consumers.

At our peak we had dozens of sales people.  They all sat around the office with almost nothing to do.  This was made all the more ironic by the fact that the original business model required fewer than ten people and probably could have been run on a single server.  Even at 100% market share, the volumes in private jets are just not that high.

Lesson learned: Be careful who you take on as a partner.  Success in one industry does not necessarily translate into success in another.  This is another lesson that others have learned the hard way.

Failure #5: Founder's Forge

After being stung so hard by the private jet fiasco, I decided to design a business that I could run entirely without partners.   This was around the time that Bitcoin first appeared, though it would still be many, many years before it broke out of hacker circles and into public awareness.  I had always been interested in cryptography, and there were parts of Bitcoin's technology that I found very appealing, particularly its use of public-key cryptography and digital signatures.  I foresaw increasing problems with credit card fraud because of the fact that the protocol was fundamentally broken: the credentials that you use to authorize a transaction are not bound to that transaction.  They can be re-used.  This is still the basis of most credit card fraud today, and it's the reason your card has a chip in it.  That chip has a secret key in it, and it computes digital signatures.  That makes it impossible to re-use the authorization for one transaction on a different transaction, and makes the credentials in general harder (but still not impossible) to steal.

The problem is that you can't use that chip for e-commerce.  It's not because this is inherently impossible, it's just that the infrastructure to do it doesn't exist.  There is no reason you could not have a card reader peripheral plugged into a USB port that would execute the same protocol that point-of-sale terminals use and provide the same protection.  In fact, there is no reason why you even need any hardware at all.  The entire protocol could be executed entirely in software on your computer or smart phone.  It would cost next to nothing and it would eliminate most credit card fraud.  So why hasn't it been done?

It is because the process is controlled by the banks, and the banks don't care.  Why don't the banks care?  Because they treat the cost of fraud as just another cost of doing business, and they pass it along to you, the consumer.  And they do it in a diabolical, stealthy way that you don't notice.  But that's another story.  Back in the day, I was not yet aware of all this.  I thought that fixing credit card fraud would be, if not welcomed, at least not actively opposed.  I was wrong.

I knew that the banks controlled the credit card system, and I knew they comprised a massive, ossified industry that was going to be next to impossible to penetrate.  So I came up with a plan that would hide the fact that I was trying to solve credit card fraud by presenting it as a completely different product addressing a completely different problem and market.

The idea was this: by this time I had a fair amount of experience starting and running companies, and one of the "pain points" was financial record-keeping.  Making sure the books were up to date was a sufficiently serious chore that most companies employed an entire team of people to do it.  (There's a reason that "the finance department" is a thing.)

The fundamental reason that record-keeping is a chore is that the records are decoupled from the transactions they record.  This is for historical reasons.  You can easily hand someone a dollar, or write them a check, or swipe your credit card without writing down that you did these things.  In the vast majority of cases you have to manually connect the record of a transaction with the actual transaction itself.

But in an age where most transactions are electronic, this doesn't have to be the case.  There is no reason why you can't build an interface in which the authorization and the record-keeping are one and the same operation, and if you do that, they cannot fall out of sync.  So the idea was to provide a service that coupled transactions and record-keeping to guarantee that they always remained in sync.  And the interface to the authorization system would quietly use digital signatures as the authorization mechanism.  The users of the system would mostly be unaware that there were digital signatures happening under the hood, just as the users of chipped credit cards are mostly unaware of this.

The company was called "Founders Forge" because the initial target market was going to be startup companies.  The name was a play on "SourceForge", the then-popular open-source hosting site.  But all of that was a red herring because the real plan was to replace credit cards with digital signatures for e-commerce.

Of course, to make all this work I needed a way to move money around, and for this I needed a bank.  I always knew that was going to be the heavy-lift so I focused on that early, long before I had even a prototype product ready to launch.  As things happened, I never even built a prototype because the banks turned out to be an impenetrable brick wall, but that impenetrability manifested itself in a very unexpected way.

I approached banks with the following pitch: I am building an accounting system for startup companies, and I want to link it with their actual financial transactions.  So I need access to an API to allow me to authorize transactions on my client's behalf.  The clients will authorize me as their agent so it will all be on the up-and-up.  The benefit for the bank is that I would funnel them a steady stream of new business in the form of the most promising Silicon Valley startup companies.

Somewhat to my surprise, this pitch was very well received.  I got positive responses and follow-up meetings from half a dozen different banks, and I thought I was on my way.

But then events took a very unexpected turn.  One by one, every bank that had initially responded positively changed their minds.  Worse, not a single one of them would tell me why.  They said, "Sorry, turns out we can't do it" and then stopped returning my calls.  In one case I actually got as high as the bank's CTO, who seemed very excited about the idea, but then he too suddenly backed off.  At that point I decided to quit, and went into a deep existential crisis for a while.  I had been working on Founders Forge for about six years.

I never got a straight answer from any of the banks about why they changed their minds.

Lesson learned: never underestimate the capacity of the banking industry to stifle innovation.  (Also, it really does appear to be run by a shadowy cabal of people you will likely never meet.)

Failure #6: Spark Innovations

I don't actually remember exactly how my last company got started.  I did this one with a partner who handled the business side of things, but was also a really good designer and front-end developer, so we made a kick-ass team.  The product was motivated by the observation that people used Microsoft Excel for purposes that it was never really intended to serve.  In particular, people tended to use it as a replacement for a SQL database.  The idea was to build a product with a spreadsheet-like UI, but with a SQL database on the back end.  We had three launch customers lined up, all of whom seemed to be very excited about the idea.  While we were building the MVP we had a previously scheduled meeting with a VC acquaintance.  We told him what we were working on, not really intending it to be a pitch, but just FYI.  In one of those legendary Silicon-Valley moments he got so excited about the idea that he asked us to let him invest just to get a foot in the door.  We didn't really want to take any money until after we launched, but he insisted, so we capitulated.

To make a long story short, we launched, and all three of our launch customers suddenly lost interest and didn't tell us why.  It was a similar situation that I had encountered previously with the banks.  No feedback, just suddenly not interested and not returning our calls.  At that point my partner had to quit because he needed a paycheck.  In a final desperate hail-mary I tried to pivot the company to something I could do on my own, but that didn't work either (obviously).  I'll spare you the details on that one.

And that's it.  My tale of six and a half failed startups plus one failed academic career, all of which somehow added up to a good life at the end of the day.  Of course, hitting the jackpot with Google didn't hurt, but I really don't think that was the whole story.  I learned a lot.  I grew a lot.  And  I think my current happiness stems mainly from the fact that I like the person I've become, someone who can fail again and again and again and again and still find a way, for the most part, to be happy.

A Reply to Don Geddis

Don Geddis left a comment on my last post.  My reply grew far longer than would reasonably fit into a comment reply so I decided to post it as an article.  Don wrote:

I wonder if you've considered that perhaps you have more in common with the people who frustrate you, than your current self-image suggests.

My reply:

I've not just considered it, I will happily concede that I am not as rational as I fancy myself to be.  In my case, I would prefer to put a slightly different spin on it and say that I am not as rational as I would like to be, or as I try to be, but whatever.  I'm a fallible human, just like everyone else.  (Duh!)  I make mistakes, and one of the mistakes I've made on more than one occasion is to be overly confident in my own abilities.  The best I can say about myself (or at least could until recently) is that I try.  I try to recognize when I make mistakes.  I try to learn from them.  I try to improve.

The failure I'm referring to in this post is not that I've failed to be sufficiently rational, or even that I've failed to persuade others to become more rational.  I only mentioned that to put my gloomy assessment of the situation into context.  My failure is that I reached the limits of my willingness to try.  I've had arguments with Protestants and Catholics and Muslims and Witnesses and Mormons and YECs and lunar landing denialists and even Republicans.  (I've never had the pleasure of interacting directly with a flat-earther, but if there are any among my audience who would be willing to engage I'd welcome the opportunity.)  The vast majority of these discussions have been civil and constructive.  I've learned a lot, and found a surprising amount of common ground.  I can steel-man just about any position (even yours, I'll bet).  In a not-insignificant number of cases I've ended up becoming friends with my interlocutor (at least for a while).  I even managed to make peace with Erik Naggum once (at least for a while).  That is one of my proudest achievements.  I figured if I could have a civil discussion with him and find common ground, I could do it with anyone.

I was wrong.  I tried really hard with Publius for a very long time, but the claim that Kamala Harris was never a prosecutor was such a brazen falsehood that it took me by surprise, and left me to face two of the most difficult choices of my life.  The first was whether to publish the claim and its accompanying screed.  Do I contribute to the promulgation of falsehoods, or do I sign on to cancel culture and become a censor?  Neither of these options is appealing.  Both run deeply counter to my moral intuitions.  It's not even clear to me which is the lesser of the two evils.  I ultimately chose the former, not because I had a good argument for it, but just because I didn't want to face the second choice: if I published Publius's comments, should I respond to them, and if so, how?

The reason this second choice was so difficult is that responding to Publius takes a lot of effort.  It can be mentally exhausting.  He's not stupid.  If I make a mistake, I can be pretty sure he'll call me out on it.  (See for example our recent exchange about the Chinese Room.)  So I can't phone it in.  I have to think and do research and keep track of a zillion different points (because Publius is a master of the Gish Gallop).  It takes time and mental energy, both of which I have in increasingly short supply these days.  So responding to Publius has a cost.  On the other hand, not responding has a cost too.  If I publish his comments and don't reply, someone might come away with the impression that I think his claim might have some merit, and that's not a good outcome for me either.

I agonized over this for nearly two months hoping that some other alternative would present itself, but none did.  So I decided to do what I did.  I don't regret it, but I'm not proud of it either.  I find it a little harder to look at myself in the mirror now than I did before.  Before I could say that, yeah, I'm far from being perfectly rational, but I'm willing to listen to anyone and try my best to understand their position.  I can't say that any more.

But I can still say it about you, Don.  I've never censored you, and you have never said anything that would make me even consider it.  Your silence here is self-imposed.  Any time you feel like breaking that silence, either in public or privately, I'm ready and willing to listen to you, and that has always been the case.

I have failed. Now what?

Nearly two months ago now I wrote:

It's getting harder and harder to find a reason to keep doing this.  My opportunity costs are high, and writing a blog entry takes a non-trivial amount of time.  I wrote this because I needed to blow off some steam, and I wanted to get my position on the election results on the record while they were still fresh in my mind.  Whether I keep doing this in the future is now very much an open question.

I got a handful of replies to that.  Thanks to everyone who responded.

The reason it has taken me so long to post any follow-up is that I also got an extended eight-part (I think that's a new record) rant from Publius, which began:

In my replies below, I will show with facts why Kamala Harris is the true threat to the rule of law, free speech, and free elections. As a bonus, you'll also find out that Kamala Harris was never a prosecutor.

I regret to say that that defeated me.  I've spent over forty years actively seeking to understand points of view that I don't agree with in the hope of finding a way to more effectively advocate for science and rationality.  During that time, I can't recall even a single instance of successfully bringing someone into the rational fold who wasn't already there.  Worse, I've discovered to my dismay that many people profess to hew to rationality aren't nearly as rational as they fancy themselves to be, and when I've tried to point this out to them they get defensive and tribal, in other words, they react like any other human would when their core beliefs are challenged.  I don't like that, and so I've tried very very hard not to get defensive and tribal when my core beliefs are challenged.  Instead, I've tried very hard to get to the roots of disagreements, partly because it has always been important for me to make sure that my beliefs rest on firm philosophical foundations, but also because I've observed over the years that a lot of disagreements turn out to be trivial differences over the definitions of words, and that people actually agree on a lot more than they appear to.  This is not to say that there aren't real disagreements -- there certainly are.  But I've found these to be a lot more rare than most people think.  In my personal and professional life I've gotten a huge amount of leverage out of this insight, and I was hoping that I could share it with others and maybe make the world a better place.

To say that I have failed would be quite the understatement.  I have failed spectacularly.  At the risk of stating the obvious, Kamala Harris was the Attorney General of California between 2011 and 2017.  That makes her a prosecutor in my book.  But even as I write this the thought that is foremost in my mind is not that I am advancing a sound argument supported by facts, but the not-so-sudden realization that what I am doing is futile.  No matter what I say, Publius will maintain that Kamala Harris was never a prosecutor, and there is no doubt in my mind that he will be able to compose lengthy screeds to defend his position.  Maybe they will even have merit.  I don't know because I'm not going to read them.  I'm done.  I'm defeated.  Life is too short to keep dealing with this kind of bullshit.  Congratulations, Publius, you won.

In a rational world, all of this would be trivial.  Instead of a loyal following my blog has ended up with a lone heckler.  Boo-hoo, let's all feel sorry for me.  But that is not really the point.  I've had many, many failures in my life.  (Hm, maybe I should write a blog post about that.)  None of them have sent me into the kind of existential despair I'm currently feeling, and that's because Publius is not an outlier, but merely a representative sample of a much, much larger population, who just elected a convicted felon and rapist to be the President of the United States of America.  But of course, to these people, Donald Trump is not a convicted felon nor a rapist, just like Kamala Harris was never a prosecutor, and nothing you nor I nor anyone else can say will ever convince them otherwise.  Truth doesn't matter.  Reality doesn't matter.  All that matters is Donald Trump and Making America Great Again (whatever the fuck that might actually turn out to mean).

Trick is, reality is even more persistent than Publius.  You can only turn your back on reality for so long before it will reassert itself. Running a country actually requires work and expertise, and if you don't put in the work, things will fall apart.  Silencing the news media might make you feel better, but it won't produce prosperity or freedom.  Neither will tariffs.  Neither will restricting immigration.

I feel like I'm watching a train wreck approaching and nothing I say or do can stop it.  The rational thing to do is to work to prepare for the disaster, to try to anticipate the worst-case scenario and put plans in place now for how to recover afterwards.  But that is not something I can do on my own, and I've found very few people interested in even discussing it.  The message I'm getting from Democrats is: we need to write bigger checks to Democrats so we can take back the House in the mid-terms and keep Donald Trump in check.  This seems hopelessly naive to me.  The Democrats had six years to get Donald Trump under control.  They impeached him twice, indicted him I don't know how many times, convicted him on 34 felony counts, and none of that made the tiniest dent in his power or influence.  And now he's heading back to the White House with complete legal immunity for anything he does as an "official act" (again, whatever the fuck that might actually turn out to mean).  The only thing that will stop Donald Trump now is if his supporters turn on him.  Good luck with that.

In a situation like that I'm finding it incredibly hard to resist the temptation to curl up into a fetal position, or maybe move to Canada.  It's not what I want.  That is not the person I want to be.  But I'm honestly out of ideas.  I've given it my best shot for somewhere between 20 and 40 years depending on how you count.  I have failed.  I don't know where to go from here.

Merry Christmas.

The Bright Side of the Election Results

I'm writing this at 9AM Pacific standard time on November 6, the morning after the election.  Not all the dust has quite settled yet, but two things are clear: Donald Trump has won, and the Republicans have taken control of the Senate.  The House is still a toss-up, and it's still unclear whether Trump will win the popular vote, but the last time I looked at the numbers he had a pretty commanding lead.

From my political perspective this is about as bad an outcome as I could have imagined, but from my perspective as an American citizen it could have been a whole lot worse.  The scenario I really dreaded was a replay of Bush v Gore, with Trump winning because three Supreme Court justices he appointed put their thumbs on the scale for him.  That didn't happen.  In fact, none of the truly nightmarish predictions came to pass.  There was no widespread chaos or efforts to disrupt the election, just one (AFAIK) random act of arson.  There was no mass suppression of votes.  None of the outcomes turned on any court decisions.  There was no armed insurrection.  There was no widespread AI-backed misinformation campaign, just more of the same run-of-the-mill falsehoods that have become part and parcel of the Trump brand.  There will be no second Civil War, just a peaceful transfer of power on January 20.

It was, on the whole, a good day for democracy.

Don't get me wrong, I still think this is a bad outcome, and I predict it will ultimately prove to be catastrophic.  There are some who believe in democracy because they think it produces good outcomes.  I am not one of those people.  The point of democracy IMO is not that it produces good outcomes -- it often doesn't.  The point of democracy is that it, like the rule of law itself, provides a non-violent mechanism for resolving disputes.  And this time around it's going to work.  The problem is, I predict, that we are now going to learn the hard way that running a country is not actually such an easy thing to do, and Donald Trump is not quite as smart as he has portrayed himself to be.  He wasn't up to the task the last time, and I see no reason to think he's going to do any better this time.  We're going to have a peaceful transition of power on January 20.  Some time after that -- not right away, but certainly before 2028 -- we're going to have chaos and catastrophe.

Part of me hopes that the Republicans will take control of the House, not because I expect that will make things any better (it won't) but because it will make it that much harder for the Republicans to deny responsibility for the disaster they are about to create.  If we're going to do this experiment, let's fucking do it right and eliminate all the confounding factors.  It will cost more, but at least we won't need to argue about whether things are bad because we didn't go MAGA enough.

Specifically what will go wrong and when I cannot say.  One easy prediction is that if Trump actually follows through with his promise to impose import tariffs, we will see a resurgence of inflation.  That much is a no-brainer.  Whether Trump will actually be stupid enough to do this remains to be seen.  The main reason he's returning to the White House now is because inflation hit a high of 9.1% in 2022 in the wake of the pandemic.  His promised tariffs will produce much steeper price increases than that.  God only knows if that will be enough to make his supporters turn on him.

But if high inflation turns out to be the worst of the fallout from a second Trump administration we will be very, very lucky.  There is so much more at stake.  Taiwan.  Ukraine.  The Balkans (and the rest of eastern Europe).  Gaza (and the rest of the middle-east).  NATO.  Climate change -- this election has probably squandered our last hope to limit carbon emissions to non-civilization-ending levels, though that won't become apparent for quite a while yet.  And then there is the very exclusive anguish you are going to experience if you are dark-skinned, or gay, or Muslim, or female.

It's going to be bad.

Will we be able to recover from this?  I don't know.  Frankly, I wouldn't bet my life savings on it.  It's going to be very, very hard, and very, very painful, and take a very, very long time.  It's going to require a lot of people on the left to let go of some very cherished ideas, like that the only way Donald Trump could possibly win would be to cheat.  He didn't.  He won fair and square.  That is the reality we have to face.

POSTSCRIPT: I was a little surprised to see someone other than Publius posting comments on my last two posts.  It has been a long time since that has happened, and I was beginning to think that I was playing to an empty house (well, not quite empty, but it's more than a little disheartening when it seems like your audience consists of a single heckler).  So if you've been lurking, and you've been getting some value from what I write, please let me know either by leaving a comment or dropping me a line.  It's getting harder and harder to find a reason to keep doing this.  My opportunity costs are high, and writing a blog entry takes a non-trivial amount of time.  I wrote this because I needed to blow off some steam, and I wanted to get my position on the election results on the record while they were still fresh in my mind.  Whether I keep doing this in the future is now very much an open question.  I have never tried to monetize my writing.  My only reward is the knowledge that someone out there got some value out of it, and that's enough for me.  Most of the time I can sustain myself on faith, but every now and then I need actual data, and this day more than usual.

What scares me about a second Trump administration

As long as I'm getting things on the record (while I still can without too much fear of reprisal) I want to endorse a video by Legal Eagle that lays out the case against voting for Donald Trump in 18 minutes of some of the best video commentary I've ever seen.  It's well worth watching, and encouraging others to watch, but just in case you don't want to invest the time and would rather read, I'm posting a (very lightly edited) transcript here.  Yes, this is blatant copyright infringement.  I apologize.  But the stakes are high and this seemed like a risk worth taking.  The Legal Eagle team has laid out this case better than I could ever hope to do.

Ron Prognosticates: Trump is Going to Win

 I'm too depressed to elaborate much on this, but I just wanted to go on the record with this prediction before the election.  Why do I think Trump is going to win?  Because DJT stock is up and has been rising steadily since it hit an all-time low in late September.  It didn't even go down today after yesterday's disastrous MSG rally.  The polls have been static since late September, or even moving slightly in Harris's favor.  So why is DJT stock up >400% in one month? I have a hard time believing that there is that much dumb money out there.  The most likely explanation I can think of is that someone out there with deep pockets knows the fix is in and Trump is going to win.

Never before in my life have I ever wanted so much to be wrong.

[UPDATE]  DJT fell 22% today, its worst trading day ever.  Maybe it was just a Trump pump-and-dump?  God, I hope so.

Have Republicans Ever Actually Listened to the Lyrics of YMCA?

Yesterday we were treated to the sight of a major party nominee at what was supposed to be a town hall meeting suddenly stop taking questions and just dancing (badly) for the better part of an hour.   A mere 20 years ago, well within living memory, less than five seconds of screaming were enough to end Howard Dean's political career.  My, how times change.

But the truly astonishing thing, the real elephant in the room, is not just that Trump could swing his hips for 39 minutes and still have a shot at becoming President of the United States of America, it is that one of the songs on his playlist was YMCA by the Village People.  I wonder if anyone in his audience has ever paid attention to what that song is actually about.

Here are the lyrics to the first verse and chorus for easy reference:

Young man, there's no need to feel down, I said
Young man, pick yourself off the ground, I said
Young man, 'cause you're in a new town
There's no need to be unhappy
Young man, there's a place you can go, I said
Young man, when you're short on your dough you can
Stay there and I'm sure you will find
Many ways to have a good time

It's fun to stay at the Y.M.C.A.
It's fun to stay at the Y.M.C.A.
They have everything for young men to enjoy
You can hang out with all the boys
It's fun to stay at the Y.M.C.A.
It's fun to stay at the Y.M.C.A.
You can get yourself clean, you can have a good meal
You can do whatever you feel

In case it's not obvious (and apparently it isn't) this song is about young men going to the YMCA to have promiscuous sex with other men. That is what they are celebrating.  That is what Donald Trump has led them to, almost certainly without himself even realizing it.  Trump is so clueless, and his followers are so clueless, that they can't even decipher "You can do whatever you feel" when sung by a group of men dressed in homoerotic iconography.

Maybe if more people pointed this out to them the scales would start to fall from their eyes?  Probably not, but a boy can dream.

UPDATE: Heh, I just noticed that Trump has a telling Freudian slip just before he calls for YMCA to be played.  He tries to say, "We're going to make the country great again" but he slips and what actually comes out of his mouth is "We're going to make the country gay..." just before he catches himself and stops mid-aphorism.  So maybe deep down he does know.

Yes, you can have exactly-once delivery

Introduction

This post is ostensibly about an obscure technical issue in distributed systems, but it's really about human communications, and how disagreements that on the surface appear to be about technical issues can sometimes turn out to actually be disagreements about the meanings of words.  I'm taking the time to write a fairly extensive post about this for two reasons.  First, I'm hoping it will be helpful to provide a case-study about what appears on the surface to be a technical dispute that turns out to be a mere quibble over terminology.  Learning to recognize such situations can help defuse tense situations.  And second, this might provide some cover for someone early in their career who comes to the same conclusion I did about a certain emperor having no clothes.

The dispute

I was reading Hacker News a few days ago and stumbled on a comment posted by /u/solatic in response to an article entitled "Falsehoods programmers believe about TCP".  I quote that comment here in its entirety:

Related: you can get at most once delivery or at least once delivery; you cannot get exactly once delivery. If I had a dollar for every junior who thought that a lack of exactly once delivery guarantees was a bug...

This struck me as odd because I could see a straightforward way to implement exactly-once delivery on top of at-least-once delivery.  But solatic seemed sufficiently confident that I thought I might be missing something, so I posted what I thought was a cautious response:

If you can get at-least-once delivery, why can you not build exactly-once on top of that?

I won't go into detail about what happened after that.  You can go read the resulting thread if you like, but the TL;DR is that the discussion spun wildly out of control, which ultimately motivated me to write this post.

Before I dive into the details, I want to give my credentials because I am about to challenge what is apparently some pretty deeply entrenched conventional wisdom.  Unfortunately, many of the people who hold these mistaken views seem to subscribe to arguments from authority.  I got a lot of admonitions that were tantamount to, "You obviously need some remedial training in basic computer science."  That's possible, but I want to provide some information about my background so you can adjust your Bayesian prior regarding that particular hypothesis.

I have a Ph.D. in Computer Science.  My specialty is (or at least was) AI and not distributed systems, but I nonetheless know my way around a network stack.  My first startup, about 30 years ago now, was an (unsuccessful) attempt to launch a new network architecture.  We built our own hardware.  I wrote the Linux device drivers for our custom NIC.  I currently work part-time as a consultant for a hardware design team that builds network routing chips.  I know how a modern network works literally from the transistors on up.  And I know about the two generals and the muddy children.

So with that, let's look at the claim that I found questionable: "You can get at most once delivery or at least once delivery; you cannot get exactly once delivery."

What does "delivery" mean here?  The entire kerfuffle turns on the ambiguity of the meaning of this word, which none of my interlocutors ever defined.  But whatever it means, my puzzlement stemmed from the fact that I could see a trivial way to build exactly-once delivery from at-least-once delivery, under any reasonable definition of "delivery".  Specifically, if you could get at-least-once delivery, then turning that into exactly-once delivery should be (it seemed to me) a simple matter of keeping track of all the delivered messages and removing duplicates.  Doesn't that seem plausible to you?  If so, congratulations, your intuitions are serving you well because that will in fact work.  It's not a particularly good solution from an engineering point of view, and it is absolutely not how you would do it in practice (at least not without a lot more refinement) but it does refute the claim that you "cannot get exactly-once delivery".  You can.

So why do so many people deny this obvious truth?  Well, that's a good question.  I'm not 100% sure because no one in that HN thread ever presented a cogent argument or even a reference.  But one common theme that ran through the responses was that I needed to brush up on the two-generals problem.  So let's do that.

The two-generals problem

The Two Generals problem is a classic problem in theoretical computer science first studied by Leslie Lamport in 1983.  It was the first time ever someone proved that a particular problem in computer science had no solution.  Some people claim that if you could get exactly-once delivery you could solve the two-generals problem, but this is false.  The two problems have nothing to do with each other, despite having some superficial resemblance.

The two-generals problem is this: there are two generals (imagine that) leading two armies against a common enemy.  They can defeat their enemy only if they successfully join forces and coordinate a simultaneous attack.  But they are geographically separated in an era before modern telecommunications.  The only way they can communicate is by sending messages via courier thorough enemy territory, where they risk being captured and never heard from again.

It turns out that it is impossible for the generals to coordinate their attack under these conditions.  In order to coordinate the attack, each general needs to know that the other general has received and agreed to the same attack plan that they intend to undertake.  The technical term is that they need to achieve common knowledge, which turns out to be impossible using an unreliable communications channel.  The proof is easy to understand: if there were such a protocol, then that protocol must have a termination condition, some point at which consensus has been reached, after which no further messages need to be sent.  But the protocol must still work even if the last message in the protocol fails to arrive, and so sending the last message must not be necessary -- the protocol has to still work without it.  By induction, the protocol must work even if no messages are sent at all.  That is clearly impossible, and so our assumption that there exists such a protocol must be false, QED.

But achieving common knowledge has absolutely nothing to do with the question of whether exactly-once delivery is possible -- it is.  As I've already described above, the easiest way to do it is to keep track of all delivered messages and remove duplicates.  (Exercise: why does this strategy not succumb to the same proof of impossibility as the two-generals problem?)

Exactly-once delivery

Now, this is where things get tricky, because no one ever actually responded to this very straightforward suggestion.  So at this point I need to extrapolate the replies I did receive and guess what the response would be if I really put someone's feet to the fire.  My guess is that they would say that what I've provided is not exactly-once delivery but exactly once processing.  I've made the processing of incoming messages idempotent, so that it doesn't matter if I get duplicate delivery, but I have not actually eliminated duplicate delivery, or something like that.

It is possible to define "delivery" makes the claim that exactly-once delivery is impossible true.  The problem is that such a definition only serves to advance pedantry and ignores important subtleties in human communication.  Let's go back to the original claim: "You can get at most once delivery, or at least once delivery; you cannot get exactly once delivery."  There are some things implied here that are not explicitly mentioned.  Specifically, there is an implication that the two things that you (supposedly) *can* have -- at-most-once and at-least-once -- are actually things that one might choose under realistic circumstances.  Consider:

You can get at most once delivery, or at least once delivery, or you can hit yourself in the head with a hammer.

That statement is true -- hitting yourself in the head with a hammer is an option.  But I hope I don't have to persuade you that this is not very useful, and the only thing you're likely to accomplish by pointing it out is to annoy people.  When you provide a list of options, there is an implied social contract that the items on the list have at least some plausible utility.

In a similar vein, one way to get at-most-once delivery is simply to disconnect the communications channel.  That will absolutely guarantee that no message is ever delivered more than once, because every message will in fact be delivered exactly zero times.  Like hitting yourself in the head with a hammer, this is an option you can undertake, but it's not very useful, and so asking someone to actually spend brain cycles considering it can't be anything more than a deliberate attempt to waste their time or insult their intelligence.

Because of this, someone hearing you say "You can get at most once delivery, or at least once delivery; you cannot get exactly once delivery" is entitled to assume that disconnecting the network is not what you meant with regards to the first option.  That assumption is reinforced by the inclusion of at-least-once delivery as a viable option, because that is actually not possible unless the odds of a message being delivered are greater than zero.  Likewise, your interlocutor is entitled to assume that the reliability of the network is less than 100%, not because you explicitly said so, but because networks are unreliable in point of actual fact, and also because that is the only thing that makes this topic worth discussing at all, and the only thing that makes the impossibility of exactly-once delivery even remotely plausible.  In a 100% reliable network you can insure exactly-once delivery by (wait for it!) ... sending each message exactly once!

Likewise, "you cannot have exactly-once-delivery, but you can have exactly-once processing" implies that there is a meaningful distinction to be drawn between "delivery" and "processing".  There isn't.  Processing is ubiquitous in modern networks.  Every router does it, and obviously every end-point does it.  Any distinction between "delivery" and "processing" that would make exactly-once delivery impossible would necessarily beg the question by stipulating that discarding duplicate messages is "processing".  You can't make this distinction in any other way because discarding duplicates can happen anywhere in the abstraction stack, including the device driver, or even in the NIC hardware.  No one does it that way, but only because it's easier and more convenient to discard duplicates in the network stack or the application, not because it's impossible to do it anywhere else.

So yes, you can have exactly-once delivery.  That may not be what you want.  In fact, it is almost certainly not what you want for anything that is actually mission-critical, because a lot can go wrong downstream of message delivery.  But that is a different question.  If you want it, you can absolutely have it.

Just for the sake of completeness I should point out that removing duplicates at the receiver is a pretty extreme oversimplification of what you would do in practice to provide exactly-once delivery.  A complete solution would almost certainly be an example of Greenspun's Tenth Law applied to the TCP protocol rather than Common Lisp.

Final Thoughts

This post was intended to be about human communication more than distributed systems or network protocols, so I want to address one last question: How did this myth of the impossibility of exactly-once delivery get so firmly established?  This is a genuine mystery to me.  I can find no academic citation anywhere, only a few blog posts, none of which have any references.  So I can only speculate.  My guess is that it stems from the fact that distributed systems are monstrously complicated both in theory and in practice, and there are some things that you really cannot guarantee in the face of certain kinds of failures.  But that is true of non-distributed systems too.  Distributed systems just amplify the problems, they don't introduce them.  Even in a non-distributed system you always have the possibility that a cosmic ray will flip a bit somewhere.  Yes, you can have ECC, but you can also have two cosmic rays, or a manufacturing defect, or sabotage, or a bug.  In the real world you can never eliminate all risks.  The best you can hope for is to reduce them to the point where you like the odds.  That's usually good enough.

That is even the case for human endeavors like this blog post.  It's possible that I've overlooked something and that my argument is wrong.  But I've thought about it, and I've done some research, and I'm able to account for all the data, so I'm casting my lot and publishing this, fully aware of the possibility that someone might point something out in the comments that will make me slap myself on the forehead and go "duh".  But even then I will count that as a good outcome because I will learn something.

That's how the scientific method works.

Information, Data, and Knowledge

(This is part 10 in a series on the scientific method.)

In 1966, well within living memory as I write this in 2024, Digital Equipment Corporation released the PDP-10, later rebranded as the DECsystem-10 or, more colloquially, the DEC-10.  The base model cost over $100,000 in 1966 dollars, well over a million dollars today.  For that price you got 8,192 words of memory, each being 36 bits wide, or just a hair over 32 kilobytes by modern reckoning.  A top-of-the-line model gave you about 144 kilobytes (32,768 36-bit words) of memory and cost three times the base model price.  At the time, the DEC-10 was considered a "low cost" computer.  Nowadays a computer with vastly superior specifications fits on a board the size of your little finger and can be purchased at retail for less than less than one 1966 dollar.  If you buy just the chip in modest volume, you can get them for less than one 2024 dollar.

Notice that despite decades of technological improvement, the specifications of a computer's capabilities were given in essentially the same terms in 1966 as they are today, namely, by specifying the amount of memory it contains, and this will always be the case.  Memory has a utility that transcends technology and culture and the whims of fashion: it can store information.

Information is both ubiquitous and ineffable.  We live in an "information age", inundated by information around the clock, but almost no one understands what it actually is.  It's challenging to find a definition that is accessible and not circular.  Wikipedia says, "Information is an abstract concept that refers to something which has the power to inform," which doesn't seem particularly helpful or enlightening to me.

So let's set this as our Problem: there is this thing, information, that everyone talks about, that permeates our modern existence, but that no one seems to actually understand.  Why?  Is information real, an essential part of objective reality, or is it just a social construct like the law or the value of fiat currency?  What is information made of?  Can it be created and destroyed like computers and chairs, or is it conserved like mass and energy?

Let's start by taking stock of some of the (apparent) properties of this thing we're trying to explain.

Information does not exist by itself.  It is always contained in in or bound to something, like a sheet of paper or a computer memory or a DNA molecule or a brain.  This container doesn't necessarily need to be a material object made of atoms, it can be an electromagnetic wave.  But it has to be something.  There is no such thing as information in isolation, not bound to any material thing.

Despite the fact that this containment is essential, the identity of a piece of information is independent of its binding.  The document you are reading right now is in some sense "made of" information.  Most likely, you are reading it on a computer, so that information is contained in a computer memory.  It is transferred into your brain -- i.e. the information is copied -- through the act of reading.  You could likewise copy the information in this document onto a sheet of paper by printing it, or into sound waves in the atmosphere by reading it aloud.  But regardless of whether the information in this document is contained in a computer memory or a sheet of paper or your brain or the air around you, it is, at least in some sense, the same information in all four cases.

How much can information be changed and still be "the same information"?  If this document is translated into a different language, is it still "the same information"?  What about if it is compressed or encrypted?  When information is compressed or encrypted, the result bears virtually no resemblance to the original.  (In the case of encryption, that is the whole point!)  But in each case, there is still some sense in which the result is "the same" as the original.  A compressed or encrypted version of this document is still unambiguously (it would seem) a version of this document and not a version of any other document.

I'm going to put a pin in that for now and invite you to consider a different question, which will eventually lead us to the answers to the preceding ones: what is the difference between information and data?  To persuade you that there is indeed a difference, here is an example of some data:

77 72 17 56 75 22 50 76 20 49 29 16 4 61 33 71 87 65 56 92

The difference between that and all of the surrounding text should be obvious: the text conveys some sort of "meaning" while the data is "just a bunch of numbers".  There might be some meaning there, but it's not readily apparent the way it is with the surrounding text.

So the difference between "information" and "data" is that information has "meaning" and data doesn't.  But what is "meaning"?  That word seems at least as troublesome to define as "information", and so reducing "information" to "meaning" doesn't represent a lot of progress.  But in this case looks are deceiving because the meaning of "meaning" can actually be made precise.  To see how, consider another example:

55 83 56 74 55 70 55 73 56 75 54 77 54 72 55 68 54 66 52 70

At first glance, this too looks like "just a bunch of numbers" but if you look closer you might start to notice some patterns in the second list that weren't there in the first.  For example, if you consider the list as pairs of numbers, the first number is always smaller than the second.  If you look at the first number in the pair, they all fall in a fairly narrow range: 52-56.  The second number in the pairs also fall in a narrow range: 66-83.  That might be enough for you to figure out what these numbers actually are without my telling you: they are the low and high temperature forecasts for a particular location (Redwood City, California) on a particular date (September 8, 2024) according to the Apple Weather app on my iPhone.  So those numbers contain information.

The difference between data and information is that information has a referent -- it is about something.  In the case of the second example, the referent is a weather forecast.  Notice that in order to make sense of that information, to extract its meaning, requires that you know something about what numbers are, how they are represented as marks on a page (or a computer screen), and how those numbers relate to temperatures and geography and time.  But the essential feature, the thing that confers information-ness to those numbers, is that they have a referent, that they correspond to something, in this case, temperatures.

It is this correspondence that is the defining characteristic of information.  Information is a correspondence between two physical things, like marks on a page and current events, or the reading on a thermometer and the temperature of the surrounding material, or the order of base pairs in a DNA molecule and the shape of a protein.

Notice that recognizing this correspondence can require quite a bit of effort.  There is nothing at all obvious about the relationship between the shapes of numerals and temperatures, nothing about the visual appearance of "23" that corresponds to "cold", or the visual appearance of "98" that corresponds to "hot".  To see the correspondence requires going through a lot of intermediate steps, like learning what numbers are, understanding what "hot" and "cold" are, what a temperature is, and so on.  There is nothing at all obvious about the relationship of base pairs in a DNA molecule and the shapes of proteins.  That correspondence goes through a relatively straightforward mapping of base pairs to amino acids, and then a much more complicated mapping of sequences of amino acids to shapes in three-dimensional space.  There are similar processes at work in compressed and encrypted data.  To see the correspondence between compressed and uncompressed data you have to know the compression algorithm.  To see the correspondence between encrypted and unencrypted data you have to know both the algorithm and the encryption key.

There is one additional requirement for data to qualify as information: the correspondence between the information and its referent must be causal.  It can't be simply due to chance.  I could produce a plausible-looking weather forecast simply by picking numbers at random.  That forecast might even be correct if I got lucky, but it would not contain any information.  The correspondence between the forecast and the actual future temperatures has to be the result of some repeatable process, and the correlation between the information and its referent has to be better than what could be produced by picking numbers at random.

How can you be sure that some candidate information is actually information and not just a coincidence?  You can't!  The best you can do is calculate the probability that a correlation is a coincidence and decide if those odds are small enough that you are willing to ignore them.  Calculating those odds can be tricky, and it's actually quite easy to fool yourself into thinking that something is information when it's not.  For example, there is a classic scam where a stock broker will send out market predictions to (say) 1024 people.  512 of the predictions say a a stock will go up, and 512 say it will go down.  For the group that got the correct predictions he sends out a second set of predictions.  256 people get an "up" prediction, and 256 get "down".  This process gets repeated eight more times.  At the end, there will be one person who has seen the broker make correct predictions ten times in a row.  The odds of that happening by chance is 1 in 1024.  Of course it did not happen by chance, but neither did it happen because the broker actually has a way of predicting which way a stock will move.  The existence of information depends on the assumption that the universe is not trying to scam us, which seems like a reasonable assumption, but we can't actually know it for certain.

The existence of information depends on another assumption as well.  Information is a correlation of the states of two (or more) systems, but what counts as a "system"?  There are some choices that seem "natural", but only because of our particular human perspective.  A bit in a computer memory, for example, only looks like a binary system because that is how it is intended to be viewed.  In actual fact, computer memories are analog systems.  When a 0 changes to a 1, that change does not happen instantaneously.  It takes time for the electrons to move around, and while that is happening the state of that bit is neither 0 nor 1 but somewhere in between.  We generally sweep that detail under the rug when we think about computer memory, but that doesn't change the fact that the underlying reality is actually much more complicated.

Some aspects of the apparent behavior of information actually depend on this.  It appears that information can be created and destroyed.  Human creativity appears to create information, and human carelessness or malice appears to destroy it.  There's a reason that backups are a thing.  But if we view the universe as a whole, and if our current understanding of the laws of physics is correct, then information can neither be created nor destroyed.  The currently known laws of physics are deterministic and time-symmetric.  If you knew the current state of the universe, you could in theory project that forward and backward in time with perfect fidelity.  When you "destroy" information, that information isn't actually destroyed, it is merely "swept under the rug" by some simplifying assumption or other.  When you "create" new information, it is not really new, it is pre-existing information being changed into a form that we can recognize under a certain set of simplifying assumptions.

It is tempting then to say that information is in some sense not real but rather a figment of our imagination, an artifact of some arbitrary set of simplifying assumptions that we have chosen for purely economic reasons, or maybe for political or social reasons.  Choosing a different set of simplifying assumptions would lead us to completely different conclusions about what constitutes information.  And that is true.

But our simplifying assumptions are not arbitrary.  The reality of information is grounded in the fact that everything you know is bound to one particular physical system: your brain.  You can imagine taking a "God's-eye view" of the universe, but you cannot actually do it!  In order to take the God's-eye view you would need a brain capable of containing all of the information in the universe, including the information contained in your own brain.  God might be able to pull that trick off, but you can't.

Could we perhaps build a machine that can take the God's-eye view?  At first glance that would seem impossible too for the same reason: that machine would have to contain all of the information in the universe.  But the machine would itself be part of the universe, and so it would have to contain a copy of all of the information contained within the machine, including a copy of that copy, and a copy of the copy of the copy, and so on forever.

There is a clever trick we can do to get around this problem.  See if you can figure out what it is.  Here is a hint: suppose you wanted to make a backup of your hard drive, but you didn't have any extra storage.  If your drive is less than 50% full this is easy: just copy the data on the used part of the drive onto the unused portion.  But can you think of a way to do it if your drive is full?)

Notwithstanding that it might be possible in principle to build a God's-eye-view machine, there are two problems that make it impossible in practice.  First, the engineering challenges of actually collecting all of the information in the universe are pretty daunting.  You would need to know at the very least the position of every subatomic particle, including those inside planets and stars in galaxies billions of light years away.  We can't even access that information for our own planet, let alone the countless trillions of others in the universe.  And second, throughout this entire discussion I've assumed that the universe is classical, that is, that it is made of particles that exist in specific locations at specific times.  It isn't.  At root, our universe is quantum, and that changes everything in profound ways.  Quantum information has a fundamentally different character from classical information, and the biggest difference is that quantum information is impossible to copy.  If you could do it, you could build a time machine.  But that's a story for another day.

The takeaway for today is that ignorance is a fundamental part of the human condition.  Indeed, as we will see when we finally get around to talking about quantum mechanics, ignorance is actually the mechanism by which the classical world emerges from the underlying quantum reality.  (Note that this is a controversial statement, so take it with a big grain of salt for now.)  But even in a classical world, taking the God's-eye view, while possible in principle, is impossible in practice, at least for us mortals.

Can we mere mortals ever actually know anything?  No, we can't, not with absolute certainty.  We are finite beings with finite life spans and finite brains.  We can only ever be in possession of a finite amount of data, and that data will always be consistent with an infinite number of potential theories.   We can never be 100% certain of anything.  It is always possible that tomorrow we will discover that our entire existence is some kind of long con -- a simulation, maybe, running in a tiny corner of a vast data center built by some unfathomably advanced alien civilization, and the only reason that the laws of physics appear to be the same from day to day is that no one has bothered to update our software.  But who knows what tomorrow may bring?  Maybe you or I will live to see the upgrade to laws-of-physics 2.0.

But I'll give long odds against.