A Clean-Sheet Introduction to the Scientific Method

 About twenty years ago I inaugurated this blog by writing the following:

"I guess I'll start with the basics: I am a scientist. That is intended to be not so much a description of my profession (though it is that too) as it is a statement about my religious beliefs."

I want to re-visit that inaugural statement in light of what I've learned in the twenty years since I first wrote it.  In particular, I want to clarify what I mean when I say that being a scientist is a statement about my religious beliefs.  I thought that there would be enough consensus about the meaning of "science" and "religious belief" that this would not be necessary, but that turns out to be one of the many, many things I as wrong about back then.  In this post I'm going to try to fix that, or at least start to.

Let me start with the easy part: By "religious beliefs" I do not mean to imply that science is a religion in the usual sense.  It isn't.  Religions generally involve things like the worship of deities, respect for the authority of revealed wisdom, and the carrying out of prayer and rituals.  Science has none of that, not because science rejects these things *a priori*, but because when you pursue science you are invariably (but not inevitably!) led to the conclusion that there are no deities active in our universe, and therefore no good reason to accept the authority of revealed wisdom, and hence not much point spending valuable time on prayer and ritual (except insofar as one might find satisfaction in pursuing prayer and ritual for their own sake).

What I *do* mean by "religious beliefs" is that being a scientist -- pursuing science, engaging in the scientific method -- need not be a profession.  It can also be a *way of life*.  I believe that science provides a *complete worldview* applicable to all aspects of life, not just ones that are commonly regarded as "science-y".  Furthermore, I believe that this worldview can be practiced by anyone, not just professional scientists.  You don't even have to be good at math (though it doesn't hurt).  And I also think that if more people did this, the world would be a better place.

In particular, I believe that science can be applied to answer questions about *morality*, and I claim that if you do this properly the results are *better* than those produced by traditional religions.  I also believe that science can provide satisfactory answers to deep existential questions, like what is the meaning of life.  But that will be a very long row to hoe.  For now I want to start simply by describing what science actually *is* because it turns out that there are a lot of misconceptions about that, particularly among the religious.

But let me start at the beginning

What is science?

Science is a process, a method, for solving a particular kind of problem.  The most succinct description I have found of the scientific method is:

Find the best explanation that accounts for all the observed data, and act as if that explanation is correct until you encounter contradictory data or a better explanation.

That is obviously an extreme oversimplification.  It is roughly akin to explaining how to play golf by saying, "Swing the club in such a way that it makes the ball go into the hole."  That's not wrong, but by itself it's not very useful either.

Golf actually turns out to be a pretty good analogy.  The scientific method is a skill, just like golf, and like golf, it is something anyone can do at a beginner level, but achieving mastery takes time and effort.  Unlike golf, the scientific method is good for a lot more than just getting balls into holes.  Golf is a uni-tasker.  Science is the ultimate multi-tasker.  You can even use it to improve your golf game!

Also like golf, you can do it both for fun and for profit.  You don't have to be a professional golfer to enjoy golf and to get something out of it.  Doing science can be rewarding for it's own sake, but there is also a significant side-benefit because, as I said, science is a method for solving problems.  So not only can it be fun and challenging and engaging, it can also give you solutions to problems.  And the kinds of problems that the scientific method can be applied to is much broader than most people realize, and using the scientific method in general is easier than a lot of people realize.  In fact, you are almost certainly already doing it, possibly without even realizing it.  Let me show you.

An example

Look around you (or, if you're blind, feel around you).  You will see (or at least think you see) things -- people, tables, cars, buildings, trees.  These things (seem to) exist in three-dimensional space, and occupy specific parts of that space, that is, the world is such that it makes sense to say things like "this tree is over here" and "that car is over there (and moving in that direction)".

Moreover, you can interact with some of the things around you in very complex and interesting ways.  There are things called "humans" that you can talk to and they will talk back and the things they say to you and that you say to them seem to convey some kind of meaning that corresponds to the properties of other things.  You can say, for example, "Do you see that tree over there?" and a human might respond, "Yes.  I think it's a maple."  And this will resonate with you in a way that saying the same thing to a dog and hearing it bark will not.

How can you account for all this?  How do you explain it?  Well, the obvious way to explain it is that the things you see are real, that is, that there really are trees and cars and other humans "out there" in point of actual physical (and maybe even metaphysical) fact.  This explanation is so obvious that it is hard to even conceive of an alternative.  Some of you might even be thinking to yourselves, "Well, duh, of course trees are real.  This guy must be some kind of moron if he thinks that is a profound observation."

The explanation that the things you perceive are real is obvious and compelling, but it is not the only possible explanation.  Another possible explanation is that you are living in the Matrix, a very high quality simulation created by some advanced alien race with technology vastly superior to our own.  That might seem unlikely, but it's possible, and it's not immediately obvious how you could definitively rule it out (or even that it is actually false!)

It turns out that neither one of these explanations is actually correct.  Both of them can actually be ruled out by experiment.  But it turns out that for the most part this doesn't actually matter.  Remember, the scientific method is not "find the correct explanation", it is "find the best explanation that accounts for all the data", and "objects appear to be real because they actually are real" is a very good explanation that is consistent with if not all of the data, at least the data that most people have access to.

Notice also that the second part of the scientific method is not, "accept that this explanation is correct", it is, "act as if this explanation is correct", and then there is the final caveat, "until you encounter contradictory data or a better explanation".

So the scientific method leads you naturally, without even being aware of it, to act as if the things you perceive are real are actually real, despite the fact (and here I have to ask you to temporarily suspend your disbelief and just take my word for it) this isn't actually true.  However, despite the fact that it isn't actually true, acting as if objects are real will not steer you far wrong in day-to-day life.

---

Here is another example.  This one is taken from history.  Imagine that you are living some time before the invention of the telescope.  You look up in the night sky and you can see the sun, moon, and stars.  Most of the stars stay in the same location (relative to each other) except that they all rotate around one star -- if you happen to be living in the northern hemisphere, otherwise they will appear to turn around an imaginary point that lies below the horizon.  (Explain that, flat-earthers!)

All of this is already strange enough, but to compound the mystery there are five -- and only five -- things that look like stars but don't move in the same way as all the others.  These are called "wanderers" or "planetae" in ancient Greek.

Two of these planetae, Venus and Mercury, are only ever seen near the horizon around sunset and sunrise.  The other three -- Mars, Jupiter and Saturn -- can be seen throughout the night.  These three all move generally in the same direction (relative to the other stars) but one them, Mars, occasionally stops and moves backwards.

How do you account for all this?

That was a question that occupied the finest human minds for thousands of years and they grappled with it to varying degrees of success.  The explanation that ultimately prevailed for well over 1000 years was produced by Claudius Ptolemy, a Greek astronomer living in Alexandria in the first century CE.  The details of Ptolemy's explanation don't matter much.  The thing that matters here is that it was based on the "fact" that what goes on in the heavens is radically different from what happens here on earth.  I put "fact" in scare quotes here because with the benefit of modern knowledge we know that this is not in fact a fact.  But from the perspective of someone living before telescopes, not only is it a fact, it is an obvious one.  The earth is dirty, the heavens are clean.  Any source of light on earth eventually extinguishes itself, but the lights in the heavens burn forever.  Anything moving on earth eventually stops, but the heavenly bodies move forever without ever coming to a halt.  And finally and most obviously, the behavior of things on earth is governed by the law that "what goes up must come down."  Some things like birds and canon balls can rise above the surface of the earth, but they can only go so far, and they can only stay aloft temporarily.  Eventually the canon ball will fall and the bird will roost (or die).  But the objects in the heavens stay there forever.  With one exception.  See if you can figure out what it is before I tell you.

[Spoiler alert]

Meteorites.  Every now and then a stone would fall from the heavens.  Where they came from was a deep mystery because on the one hand they looked like ordinary rocks, but on the other hand they came from the heavens which, as everybody knew because it was just obvious, were made of very different stuff governed by very different laws than those which pertained here on earth.

It was not until Isaac Newton in 1687 that this mystery was solved.  It turned out that the "obvious fact" that what happened in the heavens were radically different from what happens on earth was actually wrong.  The heavenly bodies are in fact made of the same stuff that things on earth are made of, and are governed by the same laws.  Today we take this for granted.  In 1687 it was a radical breakthrough, the dawn of modern science.  And one of the reasons it was accepted is that it explained the previously-mysterious observation of rocks falling from the sky.

---

At this point I want to go on a small tangent to put this event in its proper historical perspective.  As I write this, in March of 2024, it has been 336 years since Newton's Principia was published.  That might sound like a long time, but it's actually not.  I am almost 60 years old, so I have been alive for almost 20% of the total history of modern science.  Some of the most fundamental scientific theories are surprisingly recent.  The existence of atoms, for example, was controversial as recently as the early 20th century.  Albert Einstein died in 1955, slightly before I was born, but well within current living memory.  Many of the pioneers of quantum mechanics were alive when I was born.  I have personally met and spoken with Freeman Dyson, who died a mere four years ago.  Many of the experiments that provided the foundation for quantum computation were done while I was in high school.  The frontiers of quantum computation and artificial intelligence are being explored even as I write this.  We are very much still in the midst of the scientific revolution.  Quantum computing and AI are today what digital computers were in 1955, what relativity was in 1905, and what thermodynamics and steam power were in 1855.

One of the things that has happened in the 336 years since Principia was first published is that science has become an industry (much like golf has).  Isaac Newton was the first modern scientist, but he was not a professional scientist.  There was no such thing back then.  What we call "science" today was called "natural philosophy" then, and it included all kinds of things that would not be considered science today, like alchemy and astrology.  If you had asked Newton to describe the "scientific method" he would have had no idea what you were talking about.

With that in mind, I invite you to consider the following question: why is science a thing?  Why are there arguments over the definition of "science" but not "astrology" or "alchemy"?  Why is there so much more prestige (and money!) surrounding science than alchemy or astrology?  Sure, there are a few people making money as astrologers, but try getting an NSA grant to find a better way of casting horoscopes and you will get laughed out of the room.

The answer is: science is more effective at producing useful results than alchemy or astrology.  If you are reading this before the coming climate apocalypse, then you are steeped in technology.  (And if you are reading it after, take this as testimony that there was a time before the climate apocalypse when technology was ubiquitous.)  Computers, internal combustion engines, air conditioning, the Internet -- all of these things grew out of science and not alchemy or astrology.  Science is a thing because it works.

Which raises the obvious question: why does it work?  Why is science so much more effective at producing useful results than alchemy or astrology, or, for that matter, any other form of human endeavor?

To answer that, I will need to describe the scientific method in a little more detail.  But before I do that I need to first explain why describing the scientific method is not as straightforward as it might seem.

Why describing the scientific method is hard

If you seek out descriptions of the scientific method on the web you will find that they do not all agree with each other.  For example, Wikipedia says:

The scientific method involves careful observation coupled with rigorous skepticism, because cognitive assumptions can distort the interpretation of the observation. Scientific inquiry includes creating a hypothesis through inductive reasoning, testing it through experiments and statistical analysis, and adjusting or discarding the hypothesis based on the results.

However, if you dig deeper, you will find that not everyone agrees with this definition.  For example, Karl Popper, a highly regarded philosopher of science, argues that induction is not part of the scientific method, that it is a myth.  As an even more extreme example, if you go to Answers in Genesis, a creationist web site, you will find a very different description:

Science means “knowledge” and refers to a process by which we learn about the natural world. There are two different kinds of science; observational and historical. Historical science deals with the past and is not directly testable or observable so it must be interpreted according to your worldview.

The Bible is the foundation for science. Non-Christians must borrow biblical ideas—such as an orderly universe that obeys laws—in order to do science. If naturalism were true—if nature is “all there is”—then why should the universe have such order? Without the supernatural, there is no basis for logical, orderly laws of nature.

How can you tell who to believe?  Specifically, why should you believe what I am about to tell you?

One possible answer is that I was once a professional scientist.  I was an AI researcher at JPL for 12 years, from 1988 to 2000.  I made my living publishing peer-reviewed papers.  I was fairly successful.  I was the most referenced CS researcher in all of NASA (according to citeseer), and I held that title for many years even after I left.  I advanced to the rank of Principal, which is "awarded to recognize sustained outstanding individual contributions in advancing scientific or technical knowledge", which came with the most coveted perk at JPL: on-lab parking.

But that is not a very good answer, for two reasons.  First, just because I was able to make my living as a scientist doesn't necessary mean I understood how the scientific method works.  Being a successful professional scientist has as much (maybe even more) to do with politics than it does with science.  In fact, when my career advancement began to turn more on politics than science, that is what made me decide to quit.

Another possible answer is that what I am about to tell you aligns with things said by even more illustrious names like Karl Popper and Richard Feynman.  Their authority is much better than mine, but it's still an argument from authority, and the bedrock principle of science is that experiment and not authority should be the final arbiter of truth.  At least that's what Feynman said, so it must be true, right?

Ironically, the right answer can be found in the Bible, in the Gospel according to Matthew, chapter 7: by their fruits ye shall know them.  Remember, the reason we care about science at all is because it is effective at producing useful results.  The reason you should believe what I am about to tell you is that it will explain this effectiveness.  It will not be a complete explanation because that would take much longer than one blog post.  A much more detailed explanation is possible.  The one I am about to give you will be oversimplified.  But it will nonetheless explain the effectiveness of the scientific method, at least to some extent.  In other words, the scientific method can be applied to itself to explain its own effectiveness.  And that is the reason you should believe it.

By the way, an important thing to keep in mind as you read the next section: the scientific method is a *natural process*.  It is a discovery, not an invention.  It is something that happens, something that people (and even animals!) do, at least to some extent, without even being aware of it.  You can, in fact you almost certainly do, engage in the scientific method instinctively, just as you can probably hit a golf ball without any training.  But you'll be a lot better at science (and golf!) with training and practice.  So let's start.

The scientific method

The scientific method consists of seven steps.  It is important to follow these steps carefully and deliberately, otherwise you'll just end up with the scientific equivalent of a wild swing.

Step 1:  Identify a Problem.  I'm capitalizing Problem because it's a term of art which has a more specific meaning that it does in common usage.  A capital-P Problem is a discrepancy between your background knowledge, everything you believe to be true at the present moment, and something you observe.  Examples of currently open scientific Problems include things like, "Galaxies appear to rotate faster than they should based on the amount of observable matter they contain", and "There is life on earth, but we don't know how it started."  But Problems don't have to be Big Scientific Questions.  They can be as prosaic as, "I'm doing a good job at work but I'm not getting promoted" or "My wife seems to be mad at me even though she doesn't have any reason to be."

Note that the existence of Problems is not a shortcoming of the scientific method.  To the contrary, identifying a Problem is the crucial first step of the process.  I mention this because a common criticism of science among creationists is to point to Problems, things that science does not yet understand, and cite them as a reason for not trusting science at all.  This argument is not just wrong, it actually betrays a profound ignorance of how science actually works.  The only way to not have Problems is to already understand everything, to be omniscient.  The existence of Problems is a feature, not a bug.

(A creationist would no doubt respond: but we have an omniscient source of knowledge: God!  To which I respond: OK, but your access to this omniscient source of knowledge doesn't seem to give you much leverage towards producing useful results.  That is a Problem!)

Step 2:  Make a list of all simplifying assumptions that you are going to make.  For example, in the vast majority of situations here on earth it is safe to ignore relativity and quantum mechanics, but it's important to keep in the back of your mind that you are ignoring them.

Step 3:  Try to come up with a plausible *hypothesis*, a *guess* at an explanation that is consistent *both* with all the data that produced your background knowledge, *and* the discrepancy that constitutes the Problem you are addressing.  At the frontiers of science you will often get stuck at this point because coming up with *any* plausible hypothesis is considered a major achievement.  Sometimes this will happen when using the scientific method in day-to-day life.  That's OK.  Getting stuck temporarily is a normal part of the process.

Note that the term "background knowledge" is a little misleading, because very often a plausible hypothesis will be that some part of your background "knowledge" is wrong.  The use of the term "scientific knowledge" is fairly common, and it implies that this knowledge is immutable and not open to question, but that is not true.  All "knowledge" in science is tentative and subject to being overturned by new data or better hypotheses.  But this doesn't mean that we don't know anything.  Some scientific results are so well established, and backed up with so much evidence, that the odds of it being wrong, while not zero, are extremely low, and the evidence needed to show that it is wrong would be truly extraordinary.  We will sometimes abbreviate that by calling it "knowledge" or "established scientific fact" even though what we really mean is "current-best explanation, one which is so well established that the odds of overturning it, while not quite zero, are so close to zero as to make no practical difference."

Step 4:  Subject your hypotheses/guesses to criticism.  In other words, try as hard as you can to show why each of your hypotheses is *wrong*.  Anything is fair game here, including asking other people to poke holes in your ideas.  In fact, that is encouraged.  You can also participate in the scientific method by helping to poke holes in other people's ideas (this is called "peer review")

Note that -- and this is very important -- you are not trying to show that your hypothesis is valid or correct!  The object here to do the exact opposite: trying to show that your hypothesis is wrong.  Of course, you are hoping that you will fail in this endeavor, but you must nonetheless try as hard as you can and in good faith to debunk yourself.

There a few rules about scientific criticism:

Rule 1: you have to separate your criticism of the hypothesis from criticism of its presentation.  The former is vastly more valuable than the latter.  The latter is ultimately important too, but it's much less important, and doing too much of the latter at the expense of the former gets really annoying.

Rule 2: you have to criticize within the bounds of the assumptions laid out in step 2.  So in this case, if you want to criticize the hypothesis I am laying out here, it's out of bounds to say, "But you've ignored quantum mechanics."  Yes, I *know* I have ignored quantum mechanics.  I *said* I was going to ignore quantum mechanics.  Your pointing that out again is not helpful.

Rule 3: you can't change the problem statement.  So, for example, you can't criticize the hypothesis I am laying out here on the grounds that science has not (yet) produced answers to various political and social problems.  The problem I'm addressing here is: why does science appear to be so effective at producing *any useful results at all* (and in producing technology in particular)?  Any criticism not having to do with that is out of bounds.  (This is the reason it is important to have a clear, explicit, and unambiguous problem statement.)

Beyond that pretty much anything is fair game.  Here are three particularly valid forms of criticism:

Valid criticism 1: The hypothesis is inconsistent with observation.  It doesn't matter how plausible or mathematically elegant your hypothesis is, if it doesn't agree with experiment (subject to the assumptions laid out in step 2) it goes in the hopper.

Valid criticism 2: The hypothesis is unfalsifiable.  It must be possible, at least in principle, to do an experiment whose outcome would show that the hypothesis is *wrong*.  If there is no possible experiment that could be done whose outcome could be at odds with the hypothesis, then it is not a valid scientific hypothesis.  (I call this the "invisible pink unicorn" or IPU rule.)

Valid criticism 3: The hypothesis contains unnecessary detail.  You can always make any hypothesis consistent with all observations by adding additional details, but a high quality theory will be parsimonious: it will account for a lot of data with as few details as possible.

(In fact, since you live in the information age, you can actually think of the whole scientific method as a data compression process: it takes a vast amount of raw data and boils it down to the minimum amount of information needed to reproduce that data.  This turns out to be more than just a casual observation, but rather a Very Deep Insight that sheds light on why the scientific method works.  But before I can get into those details I will have to talk about the theory of computation and information, and we're nowhere near ready for that.)

Step 5: Consider whether the criticism you have produced or received is valid.  If it is, go back to step 3 and try again.

Step 6: Sometimes you will come up with more than one hypothesis that withstands all of the criticism that anyone can think to throw at it.  In that case, examine the predictions that these hypotheses make and choose one that is different between them.  Then do an experiment to see which hypothesis makes the correct prediction, and discard the others.  Note that it is entirely possible that the results will eliminate all of the surviving candidates, in which case you will need to go back to step 3.  But if this doesn't happen, if one hypothesis survives, then, congratulations, your one remaining hypothesis has now been promoted to the status of a Theory!  A Theory is a hypothesis that has withstood all attempts to invalidate it.  In science, "theory" is a synonym for "knowledge" or "fact" subject to the caveats described above.

Finally, the last step is:

Step 7: Use your Theory to make more predictions and test those against experiment too.

It turns out that if you follow this process, by the time you get to step 7, it is extremely rare for the results of those subsequent experiments to contradict the predictions made by the theory.  And that is the magic.  That is the reason that science is effective at producing useful results.  It is because it produces theories with predictive power.  It literally gives you the gift of prophecy, and if you have that, you can choose your actions to more reliably produce results that you want.

This of course raises the obvious question of why this procedure works, and specifically why this particular procedure works so much better than anything else anyone has been able to come up with.  That question also turns out to have an answer, but it is a much, much longer and more complicated answer.  It involves quantum mechanics, information theory, and the theory of computation.  I'm planning future installments about all of those, but if you're impatient this story is told reasonably well in David Deutsch's book, "The Fabric of Reality" (though what he says about parallel universes needs to be taken with a grain of salt).

Does science lead to Truth?

That science produces Theories with predictive power is simply an observed empirical fact.  As time goes by, it gets harder and harder to find Problems, harder and harder to find observations that cannot be explained and predicted by existing Theories, and harder and harder to come up with new Theories that tie up the fewer and fewer remaining loose ends.  It is possible that some day we might even come up with a Theory of Everything that will tie up the last remaining loose end, and the whole project will be complete.

There is another empirical observation we can make about the scientific method: it converges.  Not only does it produce Theories with more and more predictive power, on those rare occasions when a new Theory completely overturns an old one, the old theory (uncapitalized now because it has been shown to be wrong) always turns out to be a good approximation to the new one under certain circumstances, and in particular, under the circumstances that pertain here in our solar system.  To find phenomena that cannot be explained with current scientific Theories you have to go far outside our solar system and look at neutron stars, black holes, and even other galaxies.

One possible explanation for these empirical observations is that there is an actual metaphysical Truth out there, and that the thing that the scientific method is converging towards is this metaphysical Truth.  That's a hypothesis, a possible explanation for the empirical observation that science converges, at least so far.  This hypothesis makes a prediction: that science will continue to converge, and may some day even reach the point where there are no more Problems, where it can explain all observations.  This hypothesis is falsifiable, so it's a valid scientific hypothesis.  And for the last 336 years no data has contradicted it.

Does that prove that science finds metaphysical Truth?  No.  Nothing is ever proven in science.  All knowledge is tentative and subject to being overturned by new data or a better explanation.  But it is, at the moment, the current-best explanation.

This is not to say that there are no extant Problems with the hypothesis that science converges towards metaphysical truth.  There are at least four that I can think of.  The first is the so-called "hard problem of consciousness", i.e. explaining qualia and subjective experience.  The second is "deriving ought from is", i.e. using the scientific method to obtain a theory of moral behavior.  And the last one is the problem of origins and teleology.  Why is there something rather than nothing?  How did life on earth begin?  What is the point of all this?  And finally there is the problem of religion: why do so many people believe things that are at odds with science?

I actually believe that all of these Problems have had some pretty significant dents put into them by the scientific method, much more than is generally appreciated or understood, even among scientists.  I've written about all of these things at one time or another, but usually in the context of developing my own ideas about them, and never as a coherent summary that presents the final results in a unified and organized way.  I'm going to try to remedy that in the future.  But it has already taken me a week just to get this far so I thought I'd go ahead and put this out there and subject it to criticism.