Evaluation of Medical Information

Evaluating information, medical and otherwise, on the Web or in print is difficult. It is important to decide if any given information is worth reviewing, using or remembering. Here are some general guidelines to help you evaluate medical information:

Who is the author? What are her or his credentials? What is the motive for presenting the information?

Is the information presented clearly in an unbiased manner?

Are information sources cited or footnoted so you can find the original author or are sources readily available in a standard textbook?

Is the information valid? To decide this, one has to know a bit about the scientific method.

A brief review of THE SCIENTIFIC METHOD

At the heart of the conflict between modern medicine and many health food store owners, naturopaths, some chiropractors, a few physicians, and a variety of others who attack the institutions of science, including medicine, is the question of what is acceptable as true or factual. Both sides make claims and counter claims. How can you decide who is right?

Medicine is based on scientific observation, research, and testing. That means, before a given statement or theory can be accepted, say eating green peppers cures warts, it has to be proven by a scientific study. In this case a statistically valid, controlled, double-blind study would best prove that green peppers cure warts.

STATISTICALLY VALID means a large number of people must take part to remove the likelihood of chance happenings from skewing the results. If two people are being studied for wart cures, and one of them has his wart cut off by a piece of falling glass, it is going to make a much bigger difference in the result of the study than if five hundred people are being studied and one of them loses his wart to flying glass. The larger number makes the study statistically valid.

A CONTROLLED study has two similar groups of subjects to be tested. In our example case, say there are two groups of 250 people each, all with warts. Both groups have similar ethnic mix, a similar diet, take similar medications, live in similar areas, etc., but one group eats green peppers and the other does not.  The experimental group eats peppers, the control group does not.  If the pepper group has fewer warts at the end of the study than the other group, the control group, then it probably was the peppers and not the sun or the blood pressure medicine or the high altitude that made the warts disappear.

The DOUBLE BLIND part is to eliminate the possibility that personal bias might influence the results of the warts study. Psychological influence is very important in medicine, but our question is about green peppers and warts so we have to think up a way to have the test subjects "blind" to whether they are taking peppers or not. We could disguise the peppers with a lot of other greens in salad, or grind them up and put them in capsules to be swallowed. We'd have to feed the control group pepperless salad or give them capsules filled with sugar (a placebo) so both groups are treated equally, and the test subjects in both groups would be "blind" to the knowledge of which group was getting the peppers. Now, who is going to decide when a wart is gone? Warts don't just suddenly disappear. They get smaller until there's just a little spot on the skin--which might be there for several weeks. The test subjects can't say when a wart is gone because gone to one person might not be gone to another. The guy who administered the peppers can't tell because he knows who got peppers and who didn't and his bias toward pepper eaters might influence his judgment. What is needed is someone who is "blind" to who had peppers and who didn't to look at all the warts and decide which are worse, which are unchanged and which are gone. This is the second "blind" person, the first were the test subjects, hence the term DOUBLE BLIND.

So, after the statistically valid, controlled, double-blind study is done and published, several other people will repeat the experiment to confirm the results. If they all come out with similar results, then a definite statement can be made regarding the tested question or theory. That's a lot of work, time and expense involved in this kind of ideal scientific investigation, but drug companies, universities, governmental agencies and sometimes individuals go to all this trouble because that's really the only way to prove something works or doesn't work.

However, sometimes a question or theory is impossible or impractical to test completely. Will the chicken pox vaccine require a booster shot fifty years after original series of two shots are given? Just exactly what does dioxin, that deadly contaminant in some weed killers or other petrolium products, do to the human fetus? Do heart attack victims do better in zero gravity? In these cases animal models, less ideal experimental designs, computer models, or sometimes simply a well reasoned and educated guess is the best we can do under the circumstances.

Some questions don't fit scientific investigation. Did you think the sunset last night was pretty? Will Hillary make a good  President? Will the Kurds ever become peaceful people? What is your favorite color? These are subjective questions, not measurable scientific questions and therefore not testable by the scientific method.

Another kind of evidence or argument is sometimes presented in medical circles and more often in the popular press or by non medical people. That is the anecdote or personal experience. I knew a mother who swore blowing in her baby's face would stop the hiccoughs. My cousin tried it and it worked for her, so now I recommend it to all my friends. How do we know these were normal or average babies? Was it the blowing, or the distraction or the halitosis that stopped the hiccoughs? What constitutes stopping the hiccoughs anyhow--a fifteen second pause? This kind of argument is less valid than a good scientific study with acknowledged limitations. Sometimes the question is minor and doesn't deserve the expense and time of a full-blown scientific inquiry. The best way to get a baby to take liquid medicine is to put it in a small amount of formula and let him or her suck it from a nipple. I recommend that to a lot of mothers just from my experience with children and from talking to mothers, but I have never seen a statistically valid, controlled, double blinded, scientific study on the subject.

Laetrile cures cancer. Autism is caused by childhood immunizations. Vitamin C will prevent or cure a cold. These are all statements which are scientifically testable and have been publicly proclaimed by seemingly responsible and knowledgeable people. These claims have been widely spread by the popular media and taken as fact by many people--some physicians included. The scientific and responsible approach would be to prove these claims prior to a press release. These claims are being answered by good studies, but scientific studies take time to fund, test, review and publish. Often scientific results get less media attention than the original claims.

A short review of META-ANALYSIS

Meta-Analysis is a statistical combination of different scientific studies of the same or similar questions for the purpose of further defining results. For example, there might be dozens of studies done to evaluate green peppers and warts, but the results are not consistent or maybe they are even conflicting. To help us decide on the effectiveness of green peppers for getting rid of warts we need a Meta-Analysis study of all these studies. We ideally need to include the studies that are well designed on scientific principles, do not show bias, and are similar enough in design to be comparable. But, this is not an ideal world. A wart study done including only plantar warts, or dog warts, or an endpoint when the wart is gone for ten years might not be included. A study of expert or grandmother opinions of green peppers in the treatment of warts would be excluded for lack of solid results based on scientific principles. Sometimes good studies are done but there is a single flaw in the design that makes it less reliable than another study without that flaw. For instance, maybe an investigator had a well controlled, double blind, crossover study of warts and green peppers done on army recruits. In the population of army recruits, there is an obvious sex and age bias, and also a perhaps not so obvious racial bias. In Meta-Analysis such a study may not be tossed out because it is not perfect, but the results would be statistically weighted to account for the fact the study population is biased.

So in Meta-Analysis, after including only scientifically valid results, and after all the studies are evaluated and weighted on the strength of their evidence, the results of all the remaining studies are combined to get a larger population of test subjects. A larger test population increases the reliability of the results. In Meta- Analysis, a scientifically perfect study gets more weight than a flawed but useable study, but both results can be used in the final analysis. Maybe in our green pepper example there are six studies that are included with a total experimental subject population of 3000 individuals. Those results are more reliable and applicable to the practice of medicine than any single study, no matter how well designed, that has perhaps only 35 experimental subjects.

A quick statement on EVIDENCE-BASED PRACTICE

Evidence-Based Practice is a catch phrase of the decade in medicine (or other fields such as teaching), but it also is a good shift toward more valid information on which to base medical decisions. Evidence-Based Practice uses scientific information, including Meta-Analysis studies, to decide medical questions whenever possible. Traditionally doctors made decisions largely based on their training or expert opinion. Hopefully if the doctor was well trained and talked to reliable experts, the decisions made were based in science. There is a better way. In our example of whether eating green peppers gets rid of warts, we would look at the scientific literature for our results rather than rely on our doctor's opinion. Such results are based on scientific evidence that applies to our question and therefore are more trustworthy that the opinion of one person. That is Evidence-Based Practice.

(Brief) CONCLUSION

So regardless of what you read or who you talk to about a medical question, you have to ask, where did the information come from and how was the information derived? And in turn, what do you accept as true?

For further discussion on evaluation of information on the web check the California Medical Association page on the subject.

Ted Humphry, M.D.

this information last updated 2/08

return to Home Page

return to More Resources for Information