How do we know that quantum randomness is really random?

Since the dawn of quantum physics, the Born rule has been the cause of much consternation and gnashing of dentition, with Einstein famously complaining that God doesn't play dice.    Was Einstein right?  Is the apparent randomness of quantum measurements an illusion?  Are the results of quantum measurements actually deterministic but dependent on some hidden state that we simply don't have access to?  In other words, is the apparent randomness a reflection of a fundamental truth about objective reality, or simply a reflection of our ignorance?  And how can we possibly know for sure?  After all, ignorance, by its very nature, does not yield readily to introspection.

For the purposes of this discussion let's consider an idealized quantum experiment that has only two outcomes.  You can think of this as measuring the spin of an electron or the polarization of a photon.  Again, the details don't matter.  All that matters is that there are two possible outcomes (let's call them A and B).  Let's further suppose just for the sake of simplicity that both outcomes are equally likely.

So we do a bunch of experiments and collect a bunch of data.  This data is in the form of a sequence of A's and B's, each corresponding to the outcome of one instance of the experiment.  We analyze this sequence and it looks random.  We apply a bunch of statistical tests and they call come back and say, yep, this is a random sequence.  We wrack our brains to try to come up with a way to predict the outcome of the next experiment with odds better than chance and we fail.  Does that prove that this sequence is in fact random?

No, of course not.  We can trivially reproduce the exact same sequence in a purely deterministic way simply by playing back the record of the previous outcomes!  So how can we know that the sequence wasn't generated this way the first time around?  How do we know that there isn't a deterministic process out there in the universe somewhere that has pre-ordained the outcome of every quantum measurement we ever make?

It would seem that the fact that any sequence of experimental outcomes could be generated by playing back a record of them shows that we can never be sure that it wasn't actually done that way.  But this is wrong, and in fact it is easy to see exactly how and why it is wrong.  You might want to stop here and see if you can figure it out on your own.

Here's a clue: if the sequence of experimental outcomes was the result of simply replaying a record, or indeed of any kind of deterministic computational process (of which replaying a record is just one trivial example), we should be able to find evidence of that somewhere.  In order to make a record we have to store information somewhere.  In order to make a computational process we have to make a computer.  If that information was stored in any kind of straightforward way, and in particular (no pun intended) if that information is stored in any kind of physical artifact made of atoms then we should be able to find it.  Or at least we should be able to find some evidence that it exists.

But we can't.  And if you think about it, it is absolutely impossible for such an artifact to exist.  Why? Because it would have to store the outcome not just for the experiments that we actually do, but for any experiment that we could possibly do, and it would have to store those outcomes for every particle in the universe on which we might choose to perform an experiment including the particles that make up the artifact that is supposedly storing all this information!

Now, this is not yet an ironclad proof because there is one remaining possibility.  We don't actually have to store a separate record of the results of our experiments in order to be able to reproduce the same sequence of results.  It is enough simply to hang on to the particles themselves.  Once we have made a measurement on a particle, if we make the same measurement again we will get the same result.  So it's possible that the information that determines the results of experiments performed on particles is stored in the particles themselves.

The usual way to dispense with this possibility is to invoke Bell's theorem and to point out that it rules out local hidden variables.  But there's a more elementary (and, I think, more compelling) argument.

It is true that if we do the same experiment on a single particle twice in a row we will get the same result.  But nothing constrains us from doing different experiments on a single particle.  We could, for example, measure the position of a particle, and then its momentum, and then its position again, and then its momentum again.  If we do this, every result will be (apparently) random, completely disconnected from anything that has gone before.  And (and this is key) we can do this forever.  We can perform an infinite (well, OK, unbounded) number of measurements on one particle.  For the results to be deterministic, the state of the particle would have to store an infinite (and this time I really do mean infinite) amount of information.

Well, how do we know that this is not in fact the case?  Bohmian mechanics, for example, is a theory of exactly this sort.  In Bohmian mechanics, particles have positions, and (it turns out) all of the potentially infinite information that can be read out (eventually) via quantum measurements is encoded in the initial position of the particle.  This is possible because the position of the particle is metaphysically exact, represented by an actual real number with an infinite number of digits in its expansion and hence can contain an infinite amount of information.

How do we know that this is not what is actually happening?

Well, we don't.  Bohmian mechanics reproduces the predictions of quantum mechanics exactly so there is no way to settle the question experimentally.  There are nonetheless three good reasons to reject Bohm as an adequate explanation of physical reality.

First, the way that Bohm handles spin is really weird, bordering on the perverse.  Again for those who don't know, spin is a property of certain particles (mainly electrons) that can be measured and always produces one of two results (usually called "up" and "down" even though these don't actually have any physical significance).  In Bohmian mechanics, the only metaphysically real property that a particle has is position (and hence also velocity, which is just the time derivative of the position as in classical mechanics).  Spin is not part of the metaphysically real state of a particle.  When you think you're measuring spin, you're actually measuring the particle's position (because that's all there is) but the wave function (the "pilot wave", the thing that's pushing the particle around) conspires to move the particle through spin-measurement apparatus in just such a way that it looks as if the particle has spin, even though it really doesn't.  Bohmian mechanics is quite literally a cosmic conspiracy theory.

Second, Bohmian mechanics is causally non-local.  When you do an EPR-type experiment Bohm says that the underlying metaphysical reality is different depending on the order in which you perform the two measurements.  But according to relativity, the order of space-like separated events is not well defined.  So in order to extract an unambiguous description of physical reality from Bohm you have to arbitrarily assign an order to space-like separated events (the technical term for this is choosing a foliation).  There is no way to tell which foliation is correct (if there were then you could experimentally falsify relativity) so the choice is arbitrary.  But (and this is the crucial point) the fact that it is arbitrary completely undermines the whole point of adopting Bohmian mechanics to begin with, which was to provide a complete description of physical reality that was compatible with classical intuition.  Instead of one description of reality you have an arbitrary number of them, one for each possible foliation, and there's no way to tell which one is actually correct.

Third, although Bohm hangs his hat entirely on the physical reality of particle positions, it is fundamentally impossible to know what the position of a particle actually is.  Remember, when you measure the spin of a particle, on Bohmian mechanics you are not really measuring spin, you are really measuring position (because that's all there is).  Well, it turns out that when you measure position you aren't really measuring position either.  The only kind of measurement you can make is one that tells you whether or not the particle was inside or outside a particular (no pun intended) finite region of space.  The actual position of a particle cannot be measured.  So the one thing that Bohm advances as a description of physical reality turns out to be the operational equivalent of an invisible pink unicorn (IPU) — a thing that is posited to exist but which, by its very nature, can never be measured.

There is an even simpler argument to demonstrate the non-measurability of Bohmian positions: a measurement can only ever produce a finite amount of information, but the information encoded in the particle's actual metaphysical position is necessarily infinite (because it has to encode the results of all possible future measurements).  So the results of position measurements must contain errors.

And this is the crux of the matter: it just turns out as a matter of physical fact that a single particle can produce what appears to be an unbounded amount of information.  There are only two possibilities: either that information is generated on the fly ex nihilo, (that is, it's "really random") or it is stored somewhere.  But no one has been able to identify any possible repository in the physical world where that information could be stored.  In fact, QM fundamentally depends on this not being possible!  Interference effects only manifest themselves if there is no possible way to distinguish two different states of a system.  But to contain information, a system must have distinguishable states -- that's what information means!  To be compatible with the data, a hypothetical repository of that information must necessarily be an IPU.  Any theory where the existence of such a repository was experimentally demonstrable would not be compatible with QM.