NYU Langone Medical Center – Why Treatments Can Appear Effective Even When They Are Not

download (6)

NYU Langone Medical Center.

Observer Bias

First, researchers tend to observe what they expect to observe, a confounding factor known as observer bias. One placebo-controlled study evaluated a new treatment for multiple sclerosis. 7 The researchers behind this study added an interesting twist: while most of physicians assigned to evaluate the participants for improvement were blinded, a few were not blinded, and they knew which participants were receiving placebo. As it happened, the treatment proved to be no more effective than placebo. However, the unblinded physicians managed to “observe” a significant difference in outcome between patients on placebo and those getting the treatment under study. In other words, they imagined they saw improvement where they expected to see it. No doubt this happens frequently in the daily life of a practicing physician, who is never blinded. For this reason the professional opinions of practicing doctors are far less reliable than the outcomes of double-blind, placebo-controlled studies.

Rosenthal Effect

Not only do observers’ expectations influence their own observations, they can also cause study participants to behave in the way the observers expect. This is the Rosenthal effect, and it is startlingly powerful. In one famous set of experiments, graduate students were given several photographs and told to show them to their subjects. 3 The subjects were supposed to rate their impression of the people in the photos on a scale whose extremes were “big success in life” and “utter failure in life.” (The photos were selected from magazines and were intended to show rather normal people.)

Next, half the graduate student experimenters were informed by their professors that their subjects would rate most of the people in the photos as failures. The rest of the graduate students were led to expect their subjects to rate the photos as showing only successful people.

Almost invariably, subjects gave precisely the ratings experimenters expected. This is particularly amazing because the graduate students were only allowed to read a set speech to their subjects. They were not allowed to change a single word, and did not do so. Apparently, they managed to communicate their expectations through small changes in inflection of voice.

Reinterpretation Effect

One of the reasons study participants respond to observer expectation in medical studies is a desire to please their physician. Patients tend to stress improvements and downplay problems if that’s what they sense the doctor wants to hear. This does not necessarily involve lying. Participants may simply reinterpret their own experience to show improvement. A good example of this reinterpretation effect occurs when you take vitamin C over the winter and then decide, no matter how many colds you had, you would have gotten more if you didn’t use the vitamin C. You don’t really know this, but you may tell yourself it is true nonetheless.

Placebo Effect

An entirely different possibility is that the power of suggestion may actually improve your health. This is the concept of the placebo effect. It may be, for example, that if you expect your knee arthritis to improve, it really will improve, through the power of the mind. (The concept of the placebo effect has recently undergone serious challenge, but it probably does occur at least to some extent.)

Memory Distortion

Memory distortion effects also influence the apparent outcome of treatments. Physicians (like everyone else) have a tendency to remember their greatest successes and most extreme failures, and drop from their memory everything in between. This can lead to a very skewed recall of the effectiveness of a treatment. Suppose a surgery works dramatically 15 times, fails absolutely 5 times, and yields mediocre results in the great majority of patients. The surgeon will most likely recall the surgery as highly effective.

Cognitive Dissonance

Cognitive dissonance is another influence that makes physician impression unreliable. It is a well-established principle of experimental psychology that if you state out loud that something is true (eg, a treatment is effective), your mind will jump through hoops to make you experience the results as consistent with your beliefs. If you make your living doing something, you will similarly experience a strong tendency to believe that what you are doing really works.

Natural Course of the Disease: Illusion of Agency

Another major confusing influence is the natural course of the disease. Many diseases eventually run their course and symptoms improve on their own. This can give a false impression that a treatment has worked. However, due to a very powerful psychological tendency called the illusion of agency, a doctor will tend to feel that her efforts caused this improvement.

Regression Toward the Mean

A related effect is called regression toward the mean. This term refers to a statistical principle. Simply put, most objective measurements of the state of the body fluctuate over time. Cholesterol level is a good example. People who are admitted to a study because their cholesterol levels are high may simply have high cholesterol at the moment they were tested for the study. During the subsequent several months, their cholesterol level will naturally move up and down. Suppose they happened to have been caught at a time of particularly high levels at the beginning of the study. By the end of that study, odds are they will show a lower reading. You might object that the effect should be symmetrical, and they just as well could have been caught at a low cholesterol moment at the beginning of the study. However, if that had been the case, they wouldn’t have been allowed to participate in the study, because they wouldn’t appear to have high cholesterol. Thus, this effect tends to produce an impression of improvement when in fact what is being observed is simply the workings of chance.

Study Effect

Another influence is called the study effect. Individuals in scientific studies (or under the care of a physician generally) often feel motivated to take better care of themselves overall. If you have diabetes, for example, and you enroll in a study of a new diabetes treatment, you may subconsciously begin to take your insulin shots more religiously, control your diet more enthusiastically, and make sure that you don’t miss any doctors’ appointments. The net result may be an improvement in symptoms that has nothing to do with a specific therapy under study.

Study Drop Outs

Finally, participants with bad results may drop out of a study (or stop coming to a physician), while those with good results remain. This will tend to bias the apparent outcome toward more positive results.

All of these factors combine to make it immensely tricky to informally discover whether a treatment is effective. Neither a physician’s clinical experience nor a patient’s personal experience is particularly trustworthy. By the 1960s, researchers had begun to settle on an effective solution to this problem.

1 Comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s