Measuring impact means separating what an intervention changed from what would have happened anyway. This guide explains the idea without any statistics background, shows how to read an effect and its size, and helps you choose between the four methods in AnalyZ Solutions.
An effect is the difference a programme made. Not the change you can see in the data, but the part of that change that the programme caused. Effect analysis is the set of methods for working out how big that difference was.
An example shows why the distinction matters. A school feeding programme runs in 40 schools. Before it starts, pupils attend on average 62% of school days. A year later they attend 74%. That is a rise of 12 points. Did the programme raise attendance by 12 points?
Not necessarily. Attendance may have risen for reasons that have nothing to do with feeding: a better harvest, a new road, a calmer year. To say what the programme did, you need to know what attendance would have been in those 40 schools without it. Nobody can observe that. The programme either happened or it did not.
Statisticians call the unobservable "what would have happened without the programme" the counterfactual. An effect is the gap between what did happen and the counterfactual. Everything in effect analysis is a way of finding a believable stand-in for it.
Two comparisons come to mind first, and both can go wrong.
| The comparison | Why it seems reasonable | What can go wrong |
|---|---|---|
| Before and after, for the programme group only | The same people, measured twice. | Anything else that changed over the same period is credited to the programme: the season, prices, a new policy, ordinary growth. |
| Participants and non-participants, at the same time | People in the same area, measured together. | People who join a programme usually differ from those who do not. They may be better off, more motivated or closer to the office. Part of the gap is who they are, not what the programme did. |
The second problem has a name, selection. A training programme that attracts the most motivated people will look good even if the training itself does little. A health programme that people join when they are already ill will look bad even if it helps.
The most reliable way to get a fair comparison is a randomised trial. People are assigned to the programme by chance, so the two groups differ only by luck, and any difference in outcomes can be put down to the programme. Many programmes cannot be run like that. They may already be running, assignment by chance may be ethically or politically impossible, or the money may be spent before anyone asks for an evaluation.
The methods in AnalyZ Solutions use data that already exist. Each one finds a different way to build a comparison that comes close to what a trial would have given, and each one states clearly what it has to assume to do so.
Figure 1 shows the idea with the school example. Attendance in the programme schools rose from 62% to 74%. In schools without the programme it rose from 63% to 68%, a general trend of 5 points. If the programme schools would have followed the same trend, they would have reached 67% without the programme. The effect is the gap between 74% and 67%: 7 points, not 12.
Effect Analysis in AnalyZ Solutions offers four methods. They answer the same question with different data and different assumptions.
| Method | The question it asks | What you need | What it assumes |
|---|---|---|---|
| Propensity score matching | Did participants gain compared with similar people who did not take part? | One survey with participants and non-participants, and characteristics measured before the programme. | You measured the important reasons that made people join. |
| Difference-in-differences | Did the programme group change more over time than a comparison group? | The same outcome measured before and after the programme, for both groups. | Without the programme, the two groups would have moved in step. |
| Regression discontinuity | Did people just above an eligibility cutoff do better than people just below it? | A score, index or age that decides eligibility, with many people near the cutoff. | People cannot precisely control which side of the cutoff they land on. |
| Instrumental variables | What did the programme do for people whose participation was changed by a fair nudge? | An instrument, such as a random invitation, that moves participation. | The instrument affects the outcome only by changing participation. |
Let your data and the way the programme was run decide, and ask the questions in this order.
Sometimes two methods fit. When they do, running both and reporting that they agree is stronger evidence than either alone. If your programme has no control group and you are not sure where to start, this guide walks through the options.
Every method reports an effect, together with a measure of how sure the data make us. Four things are worth reading, in this order.
The first number to read is the effect itself, in the same units as the outcome. If the outcome is attendance as a percentage, an effect of 7 means 7 percentage points. If it is monthly income in dollars, an effect of 33 means $33 a month. Say it in those words, because everyone understands them.
A number alone does not say whether the effect is large. Two comparisons help. One is the starting level: 7 points on a base of 62% is an improvement of about 11%. The other is how much outcomes vary from one person to the next, which is the standard deviation. If pupils' attendance typically varies by about 15 points, an effect of 7 points is about half a standard deviation (7 รท 15 = 0.47).
Dividing an effect by the standard deviation gives what researchers call a standardised effect size. It lets you compare programmes that measure different things. The module reports effects in the original units, so to get the standardised version, divide the estimate by the standard deviation of the outcome, which Summary Statistics will give you.
| Standardised effect | Common label |
|---|---|
| About 0.2 | Small |
| About 0.5 | Medium |
| About 0.8 | Large |
These labels come from research in psychology and are a rough guide. A small effect can be well worth having if the programme is cheap and reaches many people, and a medium effect can be disappointing if the programme is costly. Judge the size against costs and against what similar programmes have achieved.
The 95% confidence interval is the range of effects that are consistent with the data. An estimate of 7 with an interval from 5 to 9 tells you the effect is probably somewhere between 5 and 9. A wide interval means the data say little, even when the estimate looks impressive.
The p-value measures how surprising the result would be if the programme had no effect at all. A small value, conventionally below 0.05, means chance alone is an unlikely explanation. It tells you nothing about the size of the effect. A tiny effect can have a tiny p-value in a large sample, and a large effect can have a large one in a small sample.
Whichever method you use, a few things make the analysis far more credible.
Open Effect Analysis from the Quantitative Analysis home screen and choose a method. Each of the four guides linked in section 4 walks through its own form and its own results.
Calling a before-and-after change an effect. It includes everything else that changed over the same period.
Comparing participants with non-participants and stopping there. People who join differ from people who do not.
Choosing the method after seeing which gives the best answer. Decide on the method from the way the programme was run, then report whatever it gives.
Not stating the assumptions. Every method rests on one. Say what it is and why you believe it.
Reporting only the p-value. A reader needs the size of the effect and its interval.
Calling the result proof. These methods give careful estimates under stated assumptions. They are stronger than a simple comparison and weaker than a well-run trial.
Open Effect Analysis in AnalyZ Solutions. Free to sign up, free to analyze.
Get started