Home » Uncategorized » Evaluations Need to be Controlled

Evaluations Need to be Controlled

A few weeks ago, I reviewed a pilot study for a health promotion campaign. The set-up was pretty simple. Five images were created to inform the population about the dangers of engaging in this specific health behavior. A number of adults were recruited and asked questions about their attitudes toward this behavior. They were then shown 1 of the 5 health promotion images and were asked the same attitude-related questions again. It was a standard one-group, within-subjects design evaluation.

Unsurprisingly, attitudes about the behavior decreased after viewing any of the health promotion images, exactly as the campaign creators had hoped for. But there was a problem. A very simple problem that could have been rectified before the evaluation even began.

I have no way of knowing if the health promotion images actually changed attitudes towards the behavior.

How is this possible? Because there was no control group.

Think about it this way. If you are part of a health-related study that relies on self-report, as in this study, it is fairly easy to guess what the evaluators or researchers want to hear. You are promoting a new exercise program? Great! I say I’ve doubled the number of days that I have exercised since participating. A new poster about how bad smoking is? Of course I agree that smoking is a terrible habit to have. Do I think fruits and vegetables are good foods? Why yes, yes I do.

Did you pick up on it? Depending on how an evaluation is designed, how an individual’s response is measured, and how often an individual’s response is measured, subjects may be inadvertently primed to give you the answer you want to hear, particularly if you are taking measurements at multiple time points (which you should be doing). This is what I call testing bias. By measuring something at one time point, we may inadvertently influence the measurement at a subsequent time point.

The solution to this is rather simple. Whenever we are testing individuals at multiple time points, there needs to be a control group that doesn’t receive the new program or intervention. The exact nature of the control depends on many factors. First, you must consider whether you have the ability to randomize subjects. This is what is done in true experimental trials but is often difficult when evaluating a new program. Programs often cover geographic regions, and if there is belief the program has positive health benefits, there are limited ethical arguments for refusing to provide such services. In these situations, we often create a non-randomized control group be selecting individuals in a different town or neighborhood, which is done in quasi-experimental studies, often known as natural experiments. There are even instances where different states or countries have acted as non-randomized control groups.

Second, you must consider the type of control group to create. A true control group receives no intervention. For the pilot study mentioned above, a true control group would have simply been asked questions about their attitudes toward a health behavior on two separate occasions. A placebo control group is given some other intervention that isn’t expected to influence what you are measuring. For example, if the health promotion campaign was to discourage smoking, a placebo control group could have been shown images about climate change, which would not be expected to change attitudes on smoking. Finally, the control group could be exposed to a previously tested intervention. These studies are known as comparative effectiveness studies. For our pilot study, the subjects could have viewed last year’s health promotion campaign materials. In these studies, we hope to see the new intervention work as well or better than the previous intervention.

There is an unintended consequence of using a control group in a multiple time-point evaluation. Whoever is in charge of the data must be knowledgeable enough to properly analyze the data. This isn’t a scenario that can be handled with t-tests and chi-square analysis. At a minimum, your data analyst needs to be proficient in repeated-measures ANOVA, multi-level modeling, or path analysis/structural equation modeling.

The take home message: Unless absolutely impossible, always use a control group in your evaluations. Programs, campaigns, and other interventions require considerable resources to develop, and funders will want to know that your new intervention worked. That is not possible unless there is a control group.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s


%d bloggers like this: