Quantitative Analysis/ Effect Analysis
Using a cutoff as a natural experiment

What is regression discontinuity design? Theory, calculation and application

Regression discontinuity estimates a programme effect by comparing people just above and just below an eligibility cutoff. This guide explains the idea in plain language, the checks that make it credible, and how to run and read it in AnalyZ Solutions.

12 min read Effect Analysis Intermediate

01What is regression discontinuity design?

Regression discontinuity design estimates the effect of a programme by comparing people just either side of an eligibility cutoff. If everyone who scores 60 or above gets a scholarship and everyone below does not, then students who scored 61 and students who scored 59 are almost the same kind of student. What happened to them afterwards can be compared fairly, because the only systematic difference between them is the scholarship.

The name describes what the method looks for. Regression means a line fitted to the data. Discontinuity means a sudden break. The method fits a line through the outcomes on each side of the cutoff and measures the break between the two lines where they meet.

Take a vocational scholarship awarded to every student who scores at least 60 on an entrance test. You follow 1,600 students and record their monthly income two years later. Income tends to rise with the test score, because stronger students do better anyway, so you expect a gentle upward slope. If the scholarship helps, there will also be a step up at 60.

A few terms

What you get out of it

02When to use it

Use regression discontinuity when a programme's eligibility is decided by a threshold on a continuous measure. Typical cases include a poverty score below which households receive a cash transfer, an age at which a benefit starts, a test score for a scholarship, or an income limit for a subsidy.

It works well when:

Sharp and fuzzy designs

DesignWhen it appliesWhat the module estimates
SharpThe cutoff decides treatment completely. Everyone at or above it gets the programme and nobody below does.The jump in the outcome at the cutoff.
FuzzyCrossing the cutoff raises the chance of getting the programme but does not settle it. Some eligible people do not take it up, or some ineligible people get it anyway.The jump in the outcome divided by the jump in the share who actually received the programme.

03The idea: close to the cutoff, luck decides

Whether a student scores 59.5 or 60.5 depends partly on chance: a question that happened to suit them, a good night's sleep, a marker's judgement. So students on either side of the line differ little in ability, motivation or family background. If incomes jump at the line, the best explanation is the scholarship.

Figure 1. Income by entrance test scoreDots show average monthly income for groups of students by entrance test score. Income rises gently with the score and then jumps at the cutoff of 60, where the scholarship starts.$100$125$150$175$2004050607080Cutoff: score 60Jump: $29Entrance test scoreMonthly income two years later
Figure 1. Average income by score. The lines are fitted within 8.0 points either side of the cutoff, the window the module chose

Figure 1 shows the example. Each dot is the average income of students in a two-point band of scores. Incomes rise slowly with the score, then step up at 60. The two green lines are fitted to students within 8 points of the cutoff, and the gap between them at 60 is the estimate, about $29.

An effect for people near the cutoff

The estimate applies to students close to the cutoff. It says what the scholarship did for students scoring around 60. It does not say what it would do for a student who scored 95 or one who scored 30. This limitation is the price of a very credible comparison.

04How it is calculated

  1. Choose a window
    Only students within the bandwidth of the cutoff are used. The module can pick the width for you or you can set it.
  2. Fit a line on each side
    A straight line through the incomes of students below the cutoff, and another through those above it. The lines may have different slopes.
  3. Measure the gap
    The effect is the difference between the two lines at the cutoff itself.
Outcome = a + b1 × ( score - cutoff ) + effect × ( score at or above cutoff ) + b3 × ( score - cutoff ) × ( score at or above cutoff )

The coefficient called effect is the gap between the lines at the cutoff. Within the window, every student counts equally.

Choosing the bandwidth

A narrow window gives a fairer comparison, because the students are more alike, but it leaves fewer students and a noisier estimate. A wide window gives more precision, but it brings in students who are less alike and whose incomes may follow a curve that a straight line misses. The module offers two ways to choose.

Whatever you choose, the module also re-runs the estimate at half, three-quarters, one and a half times and twice the width, so you can see whether the answer depends on the choice.

Fuzzy designs

Suppose crossing the cutoff raises the share who actually receive the programme from 10% to 90%. The outcome jump then reflects the programme in only 80% of cases, so it is divided by 0.8. The module does this with a two-stage method, the same one used for instrumental variables, and reports the first-stage jump so you can see how strong the link between the cutoff and the programme is.

How sure are we?

Each estimate comes with a robust standard error, a t statistic, a p-value and a 95% confidence interval. Because only the students inside the window are used, the standard error is usually larger than in other methods. That is the cost of a comparison this clean.

05Reading the output

Model summary

The design, the number of students inside the bandwidth and on each side of the cutoff, the bandwidth and the cutoff. If one side has very few students, the estimate rests on little data.

Treatment effect at the cutoff

The estimate in the units of the outcome, with its standard error, t statistic, p-value and 95% confidence interval. For a fuzzy design this is labelled as a local average treatment effect.

Model fit

R-squared and an F test for the local regression inside the window. As with any design that compares groups, a modest R-squared does not weaken the estimate.

First stage (fuzzy designs only)

The jump in the probability of receiving the programme at the cutoff. A jump near zero means the cutoff barely changes who gets the programme, and the fuzzy estimate will be unreliable.

The design checks

CheckWhat it asksWhat you want to see
Manipulation check (density)Are there more or fewer people just above the cutoff than just below?No significant difference. A jump in the number of people at the line suggests that scores were moved.
Sorting check (counts)Are the numbers of observations on the two sides of the window similar?No significant difference.
Covariate balanceDo characteristics measured before the programme, such as age, jump at the cutoff?No significant jump. A jump suggests something other than the programme changes at the line.
Bandwidth sensitivityDoes the estimate hold across a range of window widths?Estimates of similar size and sign.
Placebo cutoffsDoes the method find an effect at fake cutoffs, one below and one above the real one?No significant effect. A significant placebo result is a warning sign.

06How to run it in AnalyZ Solutions

Open Effect Analysis and choose Regression Discontinuity.

  1. Choose the design
    Sharp if the cutoff decides treatment completely. Fuzzy if it only influences it.
  2. Select the running variable
    The score, index or age that decides eligibility.
  3. Select the outcome variable
    Numeric, or a yes or no coded 1 and 0.
  4. For a fuzzy design, select the treatment actually received
    A binary variable that records whether each person really got the programme.
  5. Set the cutoff value
    It starts at the median of the running variable. Replace it with your real threshold.
  6. Choose the bandwidth
    Automatic selection by cross-validation, or manual, entered in the units of the running variable.
  7. Select covariates (optional)
    Characteristics measured before the programme, used for the balance check.
  8. Apply a subsample filter if needed
  9. Click Run RDD
    You get the model summary, the treatment effect, the model fit, the first stage for a fuzzy design, and the manipulation, sorting, covariate balance, bandwidth sensitivity and placebo checks, with a chart of the data.
Check the cutoff and which side is treated. The module treats observations at or above the cutoff as the upper group, and the effect is the upper group minus the lower group. If your programme goes to people below the threshold, such as a poverty score under 50, the sign is reversed, so a negative estimate means the programme raised the outcome. Say clearly in your report which side received the programme.
When your data come from a survey with sampling weights. The effect comes from a weighted local linear regression inside the window, with standard errors from the survey design. The bandwidth search, the manipulation check, the covariate balance, the bandwidth sensitivity and the placebo cutoffs all use the weights. Observations outside the window stay in the design as a subpopulation, so the standard errors still reflect the whole survey. Weighted N inside the bandwidth appears in the model summary.

07Interpreting the result

The figures below are invented to show the steps, and they come from running the module on a made-up dataset of 1,600 students. The running variable is the entrance test score, the cutoff is 60, and the outcome is monthly income in dollars two years later. The design is sharp, and the covariate is age. For this example the window was set by hand to 8 points either side of the cutoff.

DesignN in windowN leftN rightBandwidthCutoff
Sharp6363133238.0 (manual)60
EstimandEstimateSEtp-value95% CI lower95% CI upper
RD Treatment Effect29.134.396.63< 0.00120.5237.74
CheckResultp-value
Density just left of cutoff0.0632
Density just right of cutoff0.06720.7989
Counts in the window313 left, 323 right0.721
Age: jump at cutoff0.7240.1224
BandwidthWindow widthEstimateSEp-value
0.5x426.305.96< 0.001
0.75x631.205.01< 0.001
1.0x829.134.39< 0.001
1.5x1232.383.69< 0.001
2.0x1632.813.24< 0.001
Placebo cutoffCutoff valueEstimatep-value
Below true cutoff49.0-8.760.1000
Above true cutoff70.1-2.180.6621

Reading it step by step

  1. Read the effect
    The estimate is 29.1. Students who just reached the cutoff earned about $29 more per month two years later than students who just missed it. The 95% confidence interval runs from $21 to $38, and the p-value is below 0.001.
  2. Check for sorting
    The density on the two sides of the cutoff is close (0.063 and 0.067, p = 0.80), and the window holds 313 students on the left and 323 on the right. Nothing suggests that scores were moved across the line.
  3. Check the covariate
    Age differs by 0.7 years across the cutoff, which is small and not significant (p = 0.12). Students just above and below the line look alike on a characteristic that the scholarship could not have changed.
  4. Check the window
    Across windows from 4 to 16 points the estimate stays between 26 and 33, always positive and significant. The answer does not depend on one choice of width.
  5. Check the placebos
    At fake cutoffs of 49 and 70, the estimates are -8.8 and -2.2, and neither is significant. The first has a p-value of 0.10, so it is not exactly zero, but with a few hundred students per window an estimate of that size is within what chance produces.
  6. Remember who it covers
    The result is for students scoring close to 60. It should not be quoted as the effect for every scholarship recipient.

In plain words

Two years after the entrance test, students who scored just above the scholarship cutoff earned about $29 more per month than students who scored just below it (95% confidence interval $21 to $38). The checks found no sign that scores were manipulated or that the two sides differed in age, and the result held across windows from 4 to 16 points.

08Common mistakes

Leaving the default cutoff. The module starts with the median of the running variable. If you forget to enter the real threshold you will be estimating a jump where nothing happened.

Ignoring signs of manipulation. If staff round scores up to help someone pass, or applicants report an income just under the limit, the two sides of the cutoff are no longer comparable. A jump in the density is the warning.

Quoting the effect for everyone. The estimate holds near the cutoff. Students far above it may respond quite differently.

Choosing the window to get the answer you want. Decide the bandwidth first, and report the sensitivity table whichever way it comes out.

Using a running variable with few values. A score from 1 to 10 gives ten points to fit lines to. Many distinct values near the cutoff are needed.

Mixing up sharp and fuzzy. If some people above the cutoff did not take the programme, a sharp analysis estimates the effect of being offered it, not the effect of receiving it.

Frequently Asked Questions
How wide should the window be?
There is no universal answer. Start with the automatic choice, then look at the sensitivity table. If the estimate and its sign hold across half to double the width, the choice is not driving the result. A window that holds fewer than about 50 observations on either side is thin.
What if some students above the cutoff did not use the scholarship?
Choose the fuzzy design and select the variable that records who actually received it. The module then estimates the effect for students whose take-up was changed by crossing the cutoff.
Can I use it if scores are rounded to whole numbers?
Yes, as long as there are enough distinct values near the cutoff, and as long as the rounding was not done in a way that piles many people onto the cutoff itself.
Does it tell me the effect for everybody?
No. It tells you the effect for people close to the cutoff. Whether it also holds further away depends on whether the programme works the same way for different people, which the design cannot test.
Can I use it with survey data and sampling weights?
Yes. Set the survey weight in Data View and leave Apply survey weight ticked. The estimate and all the checks then use the weights, and the standard errors reflect the survey design.

Use an eligibility cutoff to measure a programme effect

Open Effect Analysis in AnalyZ Solutions. Free to sign up, free to analyze.

Get started
Related Guides