Quantitative Analysis/ Effect Analysis
Did the programme make a difference?

How to Measure Impact of Intervention: An Introduction to Programme Effect Measurement Approaches

Measuring impact means separating what an intervention changed from what would have happened anyway. This guide explains the idea without any statistics background, shows how to read an effect and its size, and helps you choose between the four methods in AnalyZ Solutions.

10 min read Effect Analysis Beginner

01What do we mean by an effect?

An effect is the difference a programme made. Not the change you can see in the data, but the part of that change that the programme caused. Effect analysis is the set of methods for working out how big that difference was.

An example shows why the distinction matters. A school feeding programme runs in 40 schools. Before it starts, pupils attend on average 62% of school days. A year later they attend 74%. That is a rise of 12 points. Did the programme raise attendance by 12 points?

Not necessarily. Attendance may have risen for reasons that have nothing to do with feeding: a better harvest, a new road, a calmer year. To say what the programme did, you need to know what attendance would have been in those 40 schools without it. Nobody can observe that. The programme either happened or it did not.

The missing comparison

Statisticians call the unobservable "what would have happened without the programme" the counterfactual. An effect is the gap between what did happen and the counterfactual. Everything in effect analysis is a way of finding a believable stand-in for it.

02Why the obvious comparisons mislead

Two comparisons come to mind first, and both can go wrong.

The comparisonWhy it seems reasonableWhat can go wrong
Before and after, for the programme group onlyThe same people, measured twice.Anything else that changed over the same period is credited to the programme: the season, prices, a new policy, ordinary growth.
Participants and non-participants, at the same timePeople in the same area, measured together.People who join a programme usually differ from those who do not. They may be better off, more motivated or closer to the office. Part of the gap is who they are, not what the programme did.

The second problem has a name, selection. A training programme that attracts the most motivated people will look good even if the training itself does little. A health programme that people join when they are already ill will look bad even if it helps.

03A fair comparison

The most reliable way to get a fair comparison is a randomised trial. People are assigned to the programme by chance, so the two groups differ only by luck, and any difference in outcomes can be put down to the programme. Many programmes cannot be run like that. They may already be running, assignment by chance may be ethically or politically impossible, or the money may be spent before anyone asks for an evaluation.

The methods in AnalyZ Solutions use data that already exist. Each one finds a different way to build a comparison that comes close to what a trial would have given, and each one states clearly what it has to assume to do so.

Figure 1. What happened and what would have happenedAttendance in programme schools rises from 62 to 74 percent. Schools without the programme rise from 63 to 68 percent. The dotted line shows where programme schools would have ended, at 67 percent, so the programme effect is 7 points.Programme schoolsOther schoolsWhat would have happened55%60%65%70%75%80%Before the programmeAfter the programme62%63%74%68%67%Effect of theprogramme:7 points
Figure 1. Programme schools improved by 12 points, but other schools show that about 5 of those points would have happened anyway

Figure 1 shows the idea with the school example. Attendance in the programme schools rose from 62% to 74%. In schools without the programme it rose from 63% to 68%, a general trend of 5 points. If the programme schools would have followed the same trend, they would have reached 67% without the programme. The effect is the gap between 74% and 67%: 7 points, not 12.

04The four methods at a glance

Effect Analysis in AnalyZ Solutions offers four methods. They answer the same question with different data and different assumptions.

MethodThe question it asksWhat you needWhat it assumes
Propensity score matchingDid participants gain compared with similar people who did not take part?One survey with participants and non-participants, and characteristics measured before the programme.You measured the important reasons that made people join.
Difference-in-differencesDid the programme group change more over time than a comparison group?The same outcome measured before and after the programme, for both groups.Without the programme, the two groups would have moved in step.
Regression discontinuityDid people just above an eligibility cutoff do better than people just below it?A score, index or age that decides eligibility, with many people near the cutoff.People cannot precisely control which side of the cutoff they land on.
Instrumental variablesWhat did the programme do for people whose participation was changed by a fair nudge?An instrument, such as a random invitation, that moves participation.The instrument affects the outcome only by changing participation.

Choosing between them

Let your data and the way the programme was run decide, and ask the questions in this order.

  1. Was eligibility decided by a score cutoff?
    If yes, and many people sit near the cutoff, regression discontinuity is usually the strongest option.
  2. Do you have an outcome measured before and after, for a programme group and a comparison group?
    If yes, difference-in-differences makes good use of that information.
  3. Was there a lottery or random invitation, with take-up that was not complete?
    If yes, instrumental variables can use the randomness.
  4. Do you have a single survey with participants, non-participants and good background characteristics?
    If yes, propensity score matching is the standard choice.

Sometimes two methods fit. When they do, running both and reporting that they agree is stronger evidence than either alone. If your programme has no control group and you are not sure where to start, this guide walks through the options.

05How big is the effect? Reading the result

Every method reports an effect, together with a measure of how sure the data make us. Four things are worth reading, in this order.

The estimate, in the units of the outcome

The first number to read is the effect itself, in the same units as the outcome. If the outcome is attendance as a percentage, an effect of 7 means 7 percentage points. If it is monthly income in dollars, an effect of 33 means $33 a month. Say it in those words, because everyone understands them.

The size relative to where people started and how much people vary

A number alone does not say whether the effect is large. Two comparisons help. One is the starting level: 7 points on a base of 62% is an improvement of about 11%. The other is how much outcomes vary from one person to the next, which is the standard deviation. If pupils' attendance typically varies by about 15 points, an effect of 7 points is about half a standard deviation (7 รท 15 = 0.47).

Dividing an effect by the standard deviation gives what researchers call a standardised effect size. It lets you compare programmes that measure different things. The module reports effects in the original units, so to get the standardised version, divide the estimate by the standard deviation of the outcome, which Summary Statistics will give you.

Standardised effectCommon label
About 0.2Small
About 0.5Medium
About 0.8Large

These labels come from research in psychology and are a rough guide. A small effect can be well worth having if the programme is cheap and reaches many people, and a medium effect can be disappointing if the programme is costly. Judge the size against costs and against what similar programmes have achieved.

The confidence interval

The 95% confidence interval is the range of effects that are consistent with the data. An estimate of 7 with an interval from 5 to 9 tells you the effect is probably somewhere between 5 and 9. A wide interval means the data say little, even when the estimate looks impressive.

The p-value

The p-value measures how surprising the result would be if the programme had no effect at all. A small value, conventionally below 0.05, means chance alone is an unlikely explanation. It tells you nothing about the size of the effect. A tiny effect can have a tiny p-value in a large sample, and a large effect can have a large one in a small sample.

Statistically significant is not the same as important. Report the size and the interval first, and the p-value last.

06What your data need

Whichever method you use, a few things make the analysis far more credible.

07How to run it in AnalyZ Solutions

Open Effect Analysis from the Quantitative Analysis home screen and choose a method. Each of the four guides linked in section 4 walks through its own form and its own results.

  1. Prepare the data in Data View
    Check that the treatment and group variables have two values, that the outcome and background variables have the right type, and that codes such as 99 for refused are marked as missing.
  2. Set the survey design if you have one
    If your data come from a survey with sampling weights, declare the weight, strata and clusters once in Data View.
  3. Choose the method and fill in the form
    Each method asks for the variables it needs, such as the treatment, the outcome and the background characteristics.
  4. Run it and read the checks before the effect
    Each method reports checks that tell you whether its assumptions are plausible. Read them first.
When your data come from a survey with sampling weights. All four methods use the survey weight by default when a survey design is set. Estimates are weighted, and uncertainty comes from the survey design, which accounts for the sampling weights, strata and clusters. The guide for each method describes what changes.

08Common mistakes

Calling a before-and-after change an effect. It includes everything else that changed over the same period.

Comparing participants with non-participants and stopping there. People who join differ from people who do not.

Choosing the method after seeing which gives the best answer. Decide on the method from the way the programme was run, then report whatever it gives.

Not stating the assumptions. Every method rests on one. Say what it is and why you believe it.

Reporting only the p-value. A reader needs the size of the effect and its interval.

Calling the result proof. These methods give careful estimates under stated assumptions. They are stronger than a simple comparison and weaker than a well-run trial.

Frequently Asked Questions
Is effect analysis the same as a randomised trial?
No. A trial assigns people to the programme by chance, which makes the comparison fair by design. Effect analysis uses data that were not collected that way and builds the comparison afterwards, so it rests on assumptions that a trial does not need.
Which method is the strongest?
It depends on the data. Regression discontinuity and instrumental variables with a random instrument get close to a trial under their assumptions, but only for particular groups of people. Difference-in-differences and propensity score matching apply more widely and rely on assumptions you have to defend.
Do I need advanced statistics?
No. The guides explain each method in plain language, and the module reports checks that tell you whether the method is working. What matters most is knowing how the programme was run and what could have made the groups different.
How is this different from Inferential Analysis?
Inferential tests ask whether two things are related, for example whether income differs between two groups. Effect analysis asks whether the programme caused the difference, which needs a design that deals with why the groups differed in the first place.
How many people do I need?
A few hundred is a sensible minimum for most methods. Regression discontinuity uses only the people near the cutoff, and instrumental variables throws away part of the variation, so both need more.

Find out what your programme changed

Open Effect Analysis in AnalyZ Solutions. Free to sign up, free to analyze.

Get started
Related Guides