Quantitative Analysis/ Effect Analysis
Separating a programme from the trend

What is difference-in-differences? Theory, calculation and application

Difference-in-differences compares how an outcome changed over time in a programme group and in a comparison group. This guide explains the logic in plain language, the one assumption it depends on, and how to run and read it in AnalyZ Solutions.

11 min read Effect Analysis Intermediate

01What is difference-in-differences?

Difference-in-differences compares how an outcome changed over time in a group that received a programme with how it changed in a group that did not. The programme's effect is the extra change in the programme group. The name describes the arithmetic: you take a difference over time in each group, and then take the difference between those two differences.

Take a school feeding programme. Attendance, measured as the percentage of school days a pupil attended in the month, was recorded for 900 pupils at baseline and again at endline. About half of them attend schools that received the programme, and the rest attend similar schools that did not.

Average attendance (%)BaselineEndlineChange
Programme schools66.576.7+10.2
Comparison schools71.274.0+2.8

Attendance rose 10.2 points in the programme schools. Is that the effect of feeding? Not entirely. It also rose 2.8 points in the comparison schools, where nothing was provided. Something else was pushing attendance up everywhere, perhaps better weather at endline or a bad month at baseline. Only the extra 7.4 points can be put down to the programme.

A direct comparison at endline would also mislead. The programme schools started 4.7 points lower than the comparison schools, so they have more ground to make up. Difference-in-differences avoids both traps. It does not care where each group started. It looks only at how much each group moved.

What you get out of it

02When to use it

Use difference-in-differences when you measured the same outcome at two points in time, once before the programme and once after, for both a programme group and a comparison group. The programme should have started between the two rounds, and the comparison group should not have received it.

The two rounds can follow the same people (often called panel data) or can be fresh samples drawn from the same schools or villages (repeated cross-sections). Both work.

It is not the right tool in a few situations. With only one round of data, propensity score matching is the usual choice. If eligibility depended on a score cutoff, regression discontinuity is stronger. And if the programme started before your first round, the baseline is no longer a clean before.

03The idea: the comparison group shows the trend

The central idea is simple. If the programme schools had not received the programme, their attendance would probably still have moved. The comparison schools show how much. Difference-in-differences takes the comparison group's change and treats it as the change the programme group would have had anyway.

Figure 1. The difference-in-differences logicAverage attendance at baseline and endline. Comparison schools go from 71.2 to 74.0. Programme schools go from 66.5 to 76.7. A dotted line shows programme schools would have reached 69.3 without the programme, leaving an effect of 7.4 points.Programme schoolsComparison schoolsWithout the programme (estimate)6468727680BaselineEndline71.266.576.774.069.3Effect:7.4 points
Figure 1. Average attendance in programme and comparison schools. The gap between the solid and dotted lines at the endline is the estimate

In Figure 1 the dotted line starts at the programme schools' baseline (66.5) and rises by the comparison group's change (2.8). It ends at 69.3. The programme schools actually reached 76.7. The gap between the two, 7.4 points, is the estimate.

The assumption behind it

Everything rests on one assumption, called parallel trends: without the programme, the two groups would have changed by the same amount. The groups do not need to start at the same level. They need to be moving in step.

This cannot be proved, and with only two rounds of data it cannot be tested either. What you can do is ask whether it is believable. Were the two sets of schools similar in location, size and the kind of families they serve? Did anything else happen during the period to one group and not the other, such as a new road, a conflict, a teacher strike or a different programme? If you have earlier rounds, plot the trends before the programme and check that the lines ran side by side.

The comparison group must be untouched. If the comparison schools were affected by the programme, for example because pupils moved from one school to the other, the comparison no longer shows what would have happened without it.

04How it is calculated

The simple version

Work out the average outcome in each of the four cells, then do two subtractions.

Effect = ( programme endline - programme baseline ) - ( comparison endline - comparison baseline )

In the example: (76.70 - 66.51) - (74.01 - 71.19) = 10.19 - 2.82 = 7.37.

The regression version

The module gets the same answer with a regression, which also lets you adjust for other characteristics and gives a standard error. The regression has three main terms.

Outcome = a + b1 × Group + b2 × Round + b3 × ( Group × Round ) + other covariates
TermWhat it measures
Group (b1)How different the programme group was from the comparison group at baseline.
Round (b2)How much the comparison group changed between baseline and endline. This is the background trend.
Group × Round (b3)The extra change in the programme group. This is the difference-in-differences effect.

Group takes the value 1 for the programme group and 0 for the comparison group. Round takes 1 for the endline and 0 for the baseline. Group × Round is 1 only for programme-group observations at endline, which is why its coefficient captures the programme.

How sure are we?

The module reports robust standard errors, known as HC1. They allow the outcome to vary by different amounts for different pupils, which is closer to real data than assuming one common spread. They do not allow for the fact that pupils in the same school tend to resemble each other. If you sampled whole schools, treat the confidence interval as somewhat too narrow, or set up the survey design with school as the cluster (see the note in section 6).

Yes or no outcomes

If the outcome is binary, such as whether a child attended school at all, the module also fits a logistic regression. The interaction coefficient from that model is on the log-odds scale, and it is not the effect on the probability of the outcome. So the module reports a separate estimate called the average marginal effect, which is the difference-in-differences in probability. Use that one in your report.

05Reading the output

Model summary

The number of observations, the two groups and the two rounds as the module understands them, the number of covariates and the type of standard error.

Group × Round means

The four average outcomes, with the number of observations in each. This is the table you can check by hand.

Regression coefficients

One row for each term, with the coefficient, robust standard error, t statistic, p-value and 95% confidence interval. The row labelled Group × Round (DiD) is the effect. The intercept and the covariate rows are there for completeness and are usually not what you report.

Model fit

R-squared and adjusted R-squared tell you how much of the variation between individual pupils the model explains. The F test checks that the terms together explain something. Neither says anything about whether the programme effect is credible. A difference-in-differences with a low R-squared can be perfectly sound.

06How to run it in AnalyZ Solutions

Open Effect Analysis and choose Difference-in-Differences.

  1. Choose your data structure
    Choose Already have both rounds if one dataset has a column for the round. Choose Upload a second round if the baseline is your current dataset and the endline is a separate file.
  2. Set up the rounds
    For the first option, select the round variable, which must have exactly two values. For the second, select the ID variable or variables that match each person across the two files, then upload the endline file. It can be CSV, TSV, Excel, JSON, Stata or SPSS, and it needs the same column names as round 1. If one column is not unique, select several together, such as county, sub-county and serial number.
  3. Select the group variable
    The binary variable that separates programme from comparison, as recorded in round 1.
  4. Select the outcome variable
    Numeric, or yes or no coded 1 and 0.
  5. Select covariates (optional)
    Characteristics to adjust for, such as the pupil's age.
  6. Click Run DiD
    You get the model summary, the table of group averages, the regression coefficients, the model fit and, for a yes or no outcome, the logistic results and the average marginal effect.
Check which value counts as the later round and the programme group. When the round or group is stored as text, the module treats the value that comes last in alphabetical order as the endline and as the programme group. Baseline and Endline, or Comparison and Programme, work as expected. If your labels do not sort that way, recode them to 0 and 1 first. The subsample filter does not apply when you upload a second round.
When your data come from a survey with sampling weights. The effect comes from weighted regression with standard errors from the survey design, and the table of group averages shows weighted means. If you sampled schools or villages, set the cluster in the survey design so the standard errors allow for pupils in the same school resembling each other. The weight can be used when both rounds are in one dataset. It cannot be used when the second round is uploaded as a separate file, because that file carries no weights, strata or clusters, so put both rounds in one dataset instead.

07Interpreting the result

The figures below are invented to show the steps, and they come from running the module on a made-up dataset. 1,800 observations come from 900 pupils measured at baseline and endline. The outcome is the percentage of school days attended in the month, the group variable separates programme from comparison schools, and the covariate is the age of the pupil.

N observationsGroup AGroup BRound ARound BCovariates
1,800ComparisonProgrammeBaselineEndline1
GroupRoundMeanN
ComparisonBaseline71.19429
ComparisonEndline74.01429
ProgrammeBaseline66.51471
ProgrammeEndline76.70471
TermCoefficientSEp-value
Intercept76.5821.492< 0.001
Group-4.7300.590< 0.001
Round2.8240.577< 0.001
Group × Round (DiD)7.3680.818< 0.001
age-0.5130.135< 0.001
R²Adjusted R²F-statisticF p-value
0.1700.16892.0< 0.001

Reading it step by step

  1. Start with the table of means
    The programme schools began 4.7 points below the comparison schools and ended 2.7 points above them. In the comparison schools attendance rose 2.8 points. That is the background trend.
  2. Read the effect line
    The Group × Round coefficient is 7.37 with a standard error of 0.82. The 95% confidence interval runs from 5.8 to 9.0, and the p-value is below 0.001. The estimate matches the hand calculation in section 4. That is expected here, because each pupil's age is the same in both rounds, so adjusting for it shifts the levels but leaves the difference-in-differences where it was.
  3. Compare with a before-and-after view
    Looking only at the programme schools, attendance rose 10.2 points. Relying on that would have overstated the effect by almost 40%, because 2.8 points of it happened anyway.
  4. Leave the baseline gap alone
    The Group coefficient, -4.7, repeats the baseline gap. It does not matter for the effect. What matters is that the two groups would have moved in step, which is an assumption to defend in your report.
  5. Keep model fit in proportion
    R-squared is 0.17. Most of the variation in attendance is between individual pupils, which no model of group and round could capture. This is normal and does not weaken the estimate.

In plain words

Attendance rose by 7.4 percentage points more in the schools that received the feeding programme than in comparable schools that did not (95% confidence interval 5.8 to 9.0). The estimate assumes that, without the programme, attendance in the two sets of schools would have followed the same trend.

08Common mistakes

Taking parallel trends for granted. It is the whole basis of the method. State it, say why you believe it, and use earlier rounds to check it if you have them.

Ignoring other events. A road, a conflict or a second programme that reached only one group in the same period will be counted as the programme effect.

Comparing different kinds of people across rounds. If the endline sample covers a different mix of pupils from the baseline, such as only those who stayed in school, the change reflects who was measured as well as what happened to them.

Using a comparison group the programme touched. Spillovers make the comparison group look better than it would have been, which shrinks the estimate.

Starting with a ceiling. If the programme group already attends 95% of days at baseline, it cannot rise as much as a group that starts at 65%. The groups cannot follow parallel trends near a limit.

Reading a short confidence interval as certainty. Pupils in the same school resemble each other, so the true uncertainty is larger than a standard error that treats pupils as independent suggests.

Frequently Asked Questions
Do the two groups have to start at the same level?
No. That is the strength of the method. The groups can start at different levels as long as they would have moved by the same amount without the programme.
Can I test parallel trends?
Not with only two rounds. If you have an earlier round, compare the change between the earlier round and the baseline in the two groups. If they moved together before the programme, the assumption is more believable. The module treats it as an assumption you are making, not something it can check.
What if my outcome is yes or no?
Use the average marginal effect that the module reports under the logistic results. It is the difference-in-differences in probability, so you can say that the share of children attending rose by a number of percentage points.
Do I need the same people in both rounds?
No. The two rounds can be the same people followed over time, or separate samples from the same groups. In either case the groups should be comparable in the two rounds.
Can I use it with survey data and sampling weights?
Yes, when both rounds are in one dataset. Set the survey weight in Data View and leave Apply survey weight ticked. It cannot be used when the second round is uploaded as a separate file.

Measure the change a programme caused

Open Effect Analysis in AnalyZ Solutions. Free to sign up, free to analyze.

Get started
Related Guides