|Buy this book!|
"An individual instance may not logically refute a statistical study, but it cannot be dismissed as being irrelevant to the matter." (p. 4).
"No wonder politicians are rarely fazed by statistical data. Presented with a study that challenges their position, they simply bring into question the authors or the data." (p. 7). Because it is very easy to introduce bias into an RCT, and because any study will have weaknesses, they can always be challenged. And studies often contradict one another. I notice some of my critics apply their criteria for judging a study far differently for studies they agree with than for those they do not. You can always cherry pick studies in this manner to prove whatever argument you are making.
"The simplicity of [randomizing patients to active treatment and placebo groups in an RCT] hides the practical difficulties of selecting two groups of patients equally matched in terms of all relevant factors." (p. 13).
"Numerous errors [in technical aspects of a study] may occur that threaten the integrity of the findings yet these are far from easy to detect." (p. 19). In other words, practitioners reading the study might easily be oblivious to major weaknesses of the study.
The size of a study sample is "inversely proportional to knowledge of the subject matter, the size of the treatment effects, the value of the results to individual patients, and the overall importance of the study." (p. 23). This has to do with the difference between statistical significance and clinical significance.
Say that a given drug caused a reduction in the incidence of gallstones by only one percent or in only one percent of patients. A study may show that this difference is statistically significant, but in terms of its relative value to you when balanced against the cost and/or the side effects, it may be next to worthless. In order to find such a small statistically significant differences between one drug and another, or between a drug and a placebo, one needs a very large sample. Significant results in studies with smaller samples tend to be much more dramatic from a clinical standpoint.
"Confounding is present when both the supposed cause and the supposed effect [of a clinical result] are associated with a third factor which is responsible - in whole or in part - for the difference in outcome detected between the groups...no methods are available for correcting for the presence of unknown confounding variables." (p. 30).
"Yet, surprisingly, trials that are being reported as being randomized yield groups with equal numbers of patients more often than would be expected by chance." (p. 63). This is strong evidence that the randomization process in a lot of studies is being manipulated to influence the results.
In re subjective symptoms - like almost all symptoms seen in psychiatry: "Such changes, even with the assistance of validated scales of symptoms, depend entirely on the account given by each individual patient." (p.69). There is strong evidence that patients frequently lie to doctors for a variety of reasons. I will address this issue in more detail in a future post.
"[The pitfalls or running analyses on subgroups of subjects in a randomized controlled study was illustrated by one study in which the] ...overall results showed a reduction in mortality from myocardial infarction with aspirin yet subgroup analysis suggested that the drug was of no benefit to those born under the sign of Gemini or Libra." (p. 75).
"Strictly speaking, statistical data apply to groups, not to individual patients. But clinicians treat individual patients....In every case, we can legitimately ask whether the findings [of an RCT] are applicable to that particular individual." (p. 89).
"Estimates suggest that less than 1% of patients will be recruited to trials. Thus, from this measure alone, it's highly unlikely that participants will be representative of the broader population of patients with the disease." (p.91).
"Patients excluded from [RCT's] tend to have a worse prognosis than those who are recruited." (p. 95) and "Most clinical research is carried out in teaching hospitals by medical staff with both a particular interest in the disease and considerable expertise, supported by nurses with specialist skills and well qualified junior doctors. Under these circumstances, the standard of care is expected to be ...superior to the average general hospital." (p.96). [This second point may in many cases not be true in the United States due to the proliferation of Contract Research Organizations or CRO's]. The chances study patients will get better can be much better than for your average Joe, making the results of the study less generalizable to the population of all patients with a disease.
Another big point made by Penston concerns what he calls the relative risk deception. Let us say that a cancer drug reduces the percentage of recurrences from 5% to 4%. The authors may then claim they have reduced the chances of recurrences by 1 in 5, or 20%. That is highly misleading. This cited rate of risk reduction, as old Albert Einstein might say, is relative. The absolute risk reduction is only 1% - which may even be within the margin of error of the study - since 95% of the sample would not have had a recurrence regardless of whether or not they took the drug!