Most programmes are not run as trials, so most evaluations have no control group. This guide shows what you can build in its place, how to choose between the four approaches in AnalyZ Solutions, and how to word the result honestly.
A control group is a set of people who are like the people in your programme in every way except one: they did not receive it. It is the cleanest way to answer the question every funder eventually asks, which is what would have happened to these people without us?
The strongest version is a randomised control group. People are assigned to the programme or to the control group by chance, so the two groups differ only by luck. Any difference in results afterwards can be put down to the programme.
Most programmes do not have one. The programme may already be running when someone asks for an evaluation. Everyone who is eligible may be getting it, which leaves nobody to leave out. Choosing who gets help by chance may be unacceptable to the community or to your own organisation. Or the budget may not have covered the extra data collection.
That does not mean you cannot say anything credible about impact. It means you need to be careful about what you build in place of the control group, and honest about what that substitute can and cannot show.
| The shortcut | What it tells you | Why it falls short |
|---|---|---|
| Compare participants before and after | How much the outcome changed. | Everything else that changed in the same period is included. The season, prices, other programmes and ordinary growth all get credited to you. |
| Compare participants with people who did not join | How the two groups differ today. | People who join are usually different from people who do not. They may be better off, more motivated or nearer to the project office. |
| Ask participants whether the programme helped | What people believe about the programme. | It is useful evidence about how people experienced it. It does not measure how much of any change the programme caused. People tend to be polite and to remember what helped. |
Each of these has a use as supporting evidence. None of them, on its own, is an estimate of the programme's effect.
The usual substitute is a comparison group. It plays the same role as a control group, but nobody assigned it by chance. It is found, or built, from data you already have. Because chance did not make it, you have to give reasons for believing it is comparable, and you should say plainly what those reasons are.
Four approaches are built into AnalyZ Solutions. Each one finds its comparison in a different place.
| Where the comparison comes from | What it looks like | Guide |
|---|---|---|
| Similar people who were not reached | Households with the same age, education, income and location as participants, who did not take part. Matching picks the closest ones. | Propensity score matching |
| The trend in a group that did not get the programme | The same outcome measured before and after for participants and for a comparison group. The comparison group's change shows what time alone did. | Difference-in-differences |
| People just on the other side of an eligibility line | Households that scored just above the poverty cutoff compared with those that scored just below. | Regression discontinuity |
| A fair nudge towards the programme | People who were invited by lottery or by a random roll-out, compared by whether the invitation moved them to take part. | Instrumental variables |
The right approach depends on how your programme was run and what data you have, not on which method you would prefer. Work down these questions and stop at the first yes.
Sometimes two questions get a yes. That is a good position to be in. If you can use both difference-in-differences and matching, run both. When they agree, you can say the finding does not depend on one method's assumptions.
A livelihoods project gave start-up grants to 600 women in two districts. Eligibility was decided by a vulnerability score, but the project did not draw a hard line, so cutoff-based methods are out. There was a baseline survey of the 600 grantees and of 450 similar women in a neighbouring district, and an endline survey of both a year later.
The second question gets a yes: measurements before and after, for a group that did not get the programme as well. Difference-in-differences is the starting point. The team would also have background details for everyone, so they could run propensity score matching on the endline data as a second opinion. If both gave a similar effect on monthly income, the report could present the result with more confidence than either alone.
Sometimes the data simply are not there. If you are still planning, you have options. If the programme is finished, the choices are narrower, but you still have some.
The strength of the claim you can make depends on the evidence behind it. Match your wording to what you did.
| What you did | What you can say |
|---|---|
| Before and after, no comparison | Outcomes changed by this amount after the programme. The data cannot say how much of the change the programme caused. |
| Participants compared with similar non-participants, adjusted for what you measured | Participants had outcomes this much higher than similar non-participants. Characteristics that were not measured could account for part of the difference. |
| Difference-in-differences with a believable comparison group | The programme raised the outcome by this amount relative to the trend in the comparison group, assuming the two groups would have moved in step. |
| Regression discontinuity or instrumental variables | The programme raised the outcome by this amount for people near the cutoff, or for people whose participation depended on the invitation. |
In every case, give the size of the effect and its confidence interval, state the assumption the method relies on, and say how you checked it. A report that states its assumptions and shows its checks is more convincing than one that claims certainty.
Open Effect Analysis from the Quantitative Analysis home screen and choose the method that came out of the questions in section 4. Each guide walks through its form and its results.
Assuming any comparison group will do. A group that differs from participants in ways that matter for the outcome gives a misleading answer, however large it is.
Choosing the comparison group after seeing the results. Decide who is in the comparison group, and why, before you look at the outcome.
Using a group that got something similar. If the comparison group received another programme with the same aim, the comparison understates the effect.
Adjusting for things the programme changed. Background characteristics should be measured before the programme or be things it could not change.
Promising a causal result in the proposal. What you can promise depends on the data you will have. Say which approach you plan to use and what it assumes.
Leaving the comparison question until the end. The best time to plan for a comparison group is before the programme starts, when baseline data can still be collected.
Open Effect Analysis in AnalyZ Solutions. Free to sign up, free to analyze.
Get started