In my inaugural post I advocated that people do their own experiments. Unfortunately, designing a good experiment, and interpreting the results of experiments, are not always easy things to do.
For example, Dr. Larry Sparks fed cholesterol to rabbits and found that the rabbits had a higher incidence of Alzheimer-like symptoms than rabbits where were not fed cholesterol. Does this prove that cholesterol causes Alzheimer-like symptoms in rabbits? It would seem to be a reasonable conclusion to draw, until Sparks moved his lab from Kentucky to Arizona and found that he could no longer reproduce his results. Why should cholesterol cause Alzheimer symptoms in Kentucky but not in Arizona?
The answer turns out to be that it is not cholesterol by itself that causes the Alzheimer symptoms, but cholesterol in conjunction with copper found in tap water. In Arizona they used bottled water, which did not contain copper. When they added copper to the bottled water the Alzheimer symptoms returned.
This is why to get reliable scientific results you have to be very careful. In particular, it is important to do control experiments in order to be sure that the effects you see are the result of what you are focusing your attention on (cholesterol) and not the result of some random thing that isn't on your radar screen at all (tap water). It is also important to do proper statistical analysis of results in order to separate real effects from random coincidences. Even with all these precautions it is still possible to be mislead. The original Kentucky studies were properly controlled and produced statistically significant results, but they still did not paint a complete picture of the situation. In fact, it is entirely possible that we still don't have the complete picture, because there could be additional factors that both the Kentucky and Arizona labs have in common that all contribute to the production of Alzheimer symptoms in rabbits that just haven't been noticed yet.
Statistics can also be misleading. The usual standard for statistical significance in scientific publications is a 95% confidence level. This does not mean that there is a 95% chance that the result is "correct" or "proves your theory". It means that there is only a 5% chance (or less) that the results you see were caused purely by random chance. Conversely, it means that there is a 95% chance that the results you see were caused by something other than random chance, but by itself the 95% confidence figure does not tell you what that other thing was. Note by the way that 95% sounds like a pretty high number, but you have to compare it to the nunber of experiments being done. 95% confidence means that about 1 experiment in 20 will give you a false positive result, that is, a result that looks real but was in fact caused purely by random chance. Multiple that proportion by the tens of thousands of experiments that are being done and you see that it is a virtual certainty that some of the results floating around out there are in fact false positives. Most of these get corrected when other labs try to reproduce the results. Every time you reproduce a result the probability that this is due to random chance shrinks. It doesn't take many reproductions before this probability shrinks to insignificance.
However (and this is a big however) for this to work it is important not to cherry-pick your results. If you do an experiment 100 times and you get a positive result at a 95% significance level five times this is almost certainly due to random chance. (That's exactly what 95% signifiance means: you expect to get five false positives for every 100 experiments that you do).
All this is prelude to the $64,000 question, about which I expect to he writing quite a bit: does HIV cause AIDS? I am actually not so much interested in the answer to that question as I am in the process by which one arrives at an answer, and what one can accept as a standard of proof. I have been looking into this issue for nearly ten years now as a mostly disinterested observer in the following sense: I am not HIV positive, nor is anyone I know with the exception of Christine Maggiore, and I do not know her very well. (We have met once, and we've exchanged some emails, that's all.) So I do not have a personal stake in whether the answer to the question is yes or no. What I do care about is that the answer be correct, and from what I've been able to tell so far the consensus view on the causal relationship between HIV and AIDS is at best highly oversimplified, not unlike the simple conclusion that cholesterol causes Alzheimer's disease.
More on this (probably much more) later.
No comments:
Post a Comment