Quantitative Analysis/ Effect Analysis
When people choose their own treatment

What is instrumental variables analysis? Theory, calculation and application

Instrumental variables analysis estimates the effect of a treatment that people chose for themselves by using a nudge that is unrelated to the hidden reasons for choosing. This guide explains the idea in plain language and shows how to run and read it in AnalyZ Solutions.

12 min read Effect Analysis Advanced

01What is instrumental variables analysis?

Instrumental variables analysis estimates the effect of a treatment when the people who receive it differ from those who do not in ways you cannot measure. It borrows a source of variation that is unrelated to those hidden differences, called an instrument, and uses only that part of the variation to learn what the treatment does.

Here is a typical case. A training course is offered to 1,000 people, and invitations are handed out by lottery. About half are invited. Not everyone invited attends, and some people who were not invited find a way in. You recorded who attended and each person's monthly income afterwards.

Average monthly income ($)Number of people
Attended the course126.0404
Did not attend82.4596

People who attended earn about $44 more. Is that the value of the course? Probably not. The people who attend a course are often the more motivated ones, and motivation raises income without any course at all. Comparing attendees with non-attendees mixes the effect of the course with the effect of the kind of person who attends. You cannot measure motivation, so you cannot adjust for it.

The lottery gets around this. Who was invited was settled by chance, so it has nothing to do with motivation. And an invitation can only affect income by getting people to attend. Using the lottery as the instrument isolates the attendance that the lottery caused, and that part is free of the motivation problem.

A few terms

What you get out of it

02When to use it

Use instrumental variables when you cannot reach a fair comparison with the other methods, because people chose or were steered into the treatment for reasons you cannot measure, and you have a credible instrument. Good instruments are rare. The usual ones are:

The hard part is the instrument, not the software. Most variables that look like instruments fail the second requirement in section 3. If you cannot argue convincingly that the instrument reaches the outcome only through the treatment, one of the other methods is the safer choice.

03The idea: borrowing the lottery's randomness

Figure 1. How an instrument worksDiagram. A lottery invitation moves attendance at a training course, and attendance affects income. Motivation, which is not measured, pushes both attendance and income up. The invitation has no direct route to income.Lottery invitationthe instrumentAttended trainingthe treatmentIncomethe outcomeMotivationnot measuredmoveseffect we wantpushes attendance uppushes income upno direct route allowed
Figure 1. The invitation is useful because it changes attendance without being tied to motivation or reaching income by any other route

Figure 1 shows the logic. Motivation, which you cannot measure, pushes both attendance and income up. That is why a plain comparison misleads. The invitation is different. It moves attendance, but it is not connected to motivation and has no other route to income. So any difference in income between the invited and the not invited must have travelled through attendance.

Three requirements

RequirementIn plain wordsCan you check it?
RelevanceThe instrument really moves the treatment. Invited people attend much more often than people who were not invited.Yes. The module reports a first-stage strength test.
As good as randomThe instrument is unrelated to the hidden things that affect the outcome. A lottery guarantees this.By design, not by test. You argue for it from how the instrument arose.
Only through the treatmentThe instrument has no effect on the outcome except by changing the treatment. An invitation that also came with a free toolkit would break this.Not with a single instrument. You argue for it from knowledge of the programme.

Who the estimate describes

Only some people respond to an instrument. Among the 1,000 people, 64% of those invited attended and 15% of those not invited did. Some people would attend either way, some would never attend, and some attend only if invited. The method learns about the last group, those whose attendance changed because of the invitation. The estimate is the effect of the course for them. It is called a local average treatment effect. It may not hold for people who would always attend, or who would never attend.

04How it is calculated

Two stages

  1. First stage
    Predict who attends from the instrument and any covariates. This gives each person an attendance prediction that depends only on the lottery and the covariates, and not on hidden motivation.
  2. Second stage
    Regress income on that predicted attendance. The coefficient is the effect of attendance, and the method is called two-stage least squares, or 2SLS.

The idea in one division

With one instrument and one treatment, the result can be written as a ratio.

Effect = ( effect of the instrument on the outcome ) / ( effect of the instrument on the treatment )

In the example, being invited raised income by $12.20 on average. But the invitation moved attendance by only 0.49 of a person, that is, 49 percentage points. So the $12.20 came from 49% of people changing their behaviour, and the effect per person whose attendance changed is 12.20 ÷ 0.49 = 24.9. That is the 2SLS estimate.

Weak instruments

If the instrument barely moves the treatment, the denominator is close to zero and the ratio swings wildly. The module tests the strength of the first stage with an F statistic. A common rule of thumb is that F should be above 10. Below that the instrument is called weak, and the estimate should be treated with caution.

The tests that go with it

How sure are we?

The module reports robust standard errors (HC1) with a t statistic, p-value and 95% confidence interval for both the plain and the instrumental estimate. The instrumental interval is wider, because the method uses only part of the variation in the data.

05Reading the output

Model summary

The number of observations, the treatment, the number of instruments and covariates, the outcome type and whether the model is just-identified (one instrument per treatment) or over-identified (more).

First-stage strength

The F statistic for the instruments in the first stage, with a warning if it falls below 10. This answers the relevance question and nothing else. A strong first stage does not make an instrument valid.

First-stage coefficients

How much each instrument moves the treatment. In the example, being invited raises the chance of attending by 0.49, or 49 percentage points.

OLS versus 2SLS

The plain (OLS) estimate and the instrumental (2SLS) estimate side by side, each with its confidence interval. The gap between them shows how much hidden differences were distorting the plain comparison.

Model fit

The first-stage R-squared is a real fit measure for the model that predicts the treatment. The R-squared for the 2SLS model is shown for reference only. It can be low or even negative and is not a guide to whether the effect estimate is any good.

Reduced form

The effect of the instrument directly on the outcome. It needs no assumptions beyond randomness of the instrument. If the reduced form shows nothing, there is nothing to scale up, whatever the 2SLS estimate says.

Endogeneity test and over-identification test

The tests described above. The over-identification test appears only when there is more than one instrument.

06How to run it in AnalyZ Solutions

Open Effect Analysis and choose Instrumental Variables (2SLS).

  1. Select the outcome variable
    Numeric, or a yes or no coded 1 and 0.
  2. Select the treatment variable
    The endogenous variable, whose effect you want. It is usually a yes or no. When it is stored as text, the module treats the last value in alphabetical order as 1, so check that the label you care about sorts last or code it as 1 and 0 yourself.
  3. Select the instrument or instruments
    Choose the variable or variables that move the treatment but have no direct effect on the outcome.
  4. Select exogenous covariates (optional)
    Characteristics that are not affected by the treatment, such as age. They can make the estimate more precise.
  5. Apply a subsample filter if needed
  6. Click Run 2SLS
    You get the model summary, the first-stage strength test and coefficients, the OLS and 2SLS estimates, the model fit, the reduced form, the endogeneity test and, with several instruments, the over-identification test.
The module can say the instrument is strong. It cannot say it is valid. Validity comes from how the instrument arose and what you know about the programme. Write down the argument in your report, and expect a reader to challenge it.
When your data come from a survey with sampling weights. The first stage, the 2SLS and plain estimates and the reduced form are weighted, with standard errors from the survey design. The first-stage F statistic is a design-based Wald F, and the endogeneity test uses the same errors. The over-identification test is not shown when a survey weight is applied, because it has no accepted design-based version. Weighted N appears in the model summary.

07Interpreting the result

The figures below are invented to show the steps, and they come from running the module on a made-up dataset of 1,000 people. The outcome is monthly income in dollars, the treatment is attending the training course (1 for attended), the instrument is the lottery invitation, and the covariate is age.

N observationsTreatmentInstrumentsCovariatesOutcome typeIdentification
1,000attended11ContinuousJust-identified
First-stage Fp-valueInstrumentCoefficientSEt
330.1< 0.001invited0.49020.026618.40
EstimatorEstimateSEtp-value95% CI lower95% CI upper
OLS (naive, ignores endogeneity)43.012.5516.88< 0.00138.0148.00
2SLS (instrumented)24.895.274.72< 0.00114.5535.22
Reduced form: instrumentCoefficientSEtp-value
invited12.202.834.32< 0.001
Endogeneity testCoefficientSEtp-value
Residual from first stage24.125.974.04< 0.001
First-stage R²Adjusted R²2SLS R² (reference only)
0.2500.2480.205

Reading it step by step

  1. Check the instrument is strong
    The first-stage F is 330, far above 10. Being invited raised the chance of attending by 0.49, from 15% to 64%. The invitation clearly moves attendance.
  2. Compare the two estimates
    The plain OLS estimate is $43 and the 2SLS estimate is $25. The plain comparison overstated the effect of the course by about 70%, which is the work done by motivation and other hidden differences.
  3. Check the endogeneity test
    The p-value is below 0.001, so the two estimates differ by more than chance would explain. The plain regression cannot be trusted here, and the instrumental estimate is the one to report.
  4. Look at the reduced form
    The invitation alone raised income by about $12, with a p-value below 0.001. Because the lottery was random, this comparison has no selection problem. It confirms that something real is happening before any scaling up.
  5. Read the effect
    The 2SLS estimate is 24.9. Among people whose attendance depended on the invitation, the course is estimated to have raised monthly income by about $25. The 95% confidence interval runs from $15 to $35.
  6. Notice the price of honesty
    The 2SLS interval is more than twice as wide as the plain one (21 dollars against 10). That is normal. The method uses only the part of attendance that the lottery caused, so it has less information to work with.

In plain words

People whose decision to attend depended on the lottery invitation earned about $25 more per month because of the training (95% confidence interval $15 to $35). Comparing attendees with non-attendees directly would have suggested $43, mostly because attendees were more motivated to begin with. The result applies to people whose attendance was changed by the invitation and relies on the invitation affecting income only through attendance.

08Common mistakes

Using a weak instrument. If the first-stage F is below 10, the estimate can be far off and its confidence interval misleading. Look for a stronger instrument or do not rely on the result.

Using an instrument with a direct route to the outcome. An invitation that also came with a toolkit, or distance to a clinic that also tracks wealth, breaks the key assumption. The software cannot detect this.

Choosing a variable because it is available. An instrument needs an argument, not just a correlation with the treatment.

Reading the estimate as the effect for everyone. It describes people whose treatment was changed by the instrument.

Judging the model by its R-squared. The 2SLS R-squared is not a measure of quality and can be negative.

Concluding the plain regression is wrong just because the two estimates differ. The difference points to hidden differences only if the instrument is valid. If it is not, the instrumental estimate can be the worse of the two.

Frequently Asked Questions
What makes a good instrument?
It moves the treatment a lot, it was determined by something unrelated to the hidden differences between people, and it affects the outcome only by changing the treatment. Random assignment satisfies the second condition by design. Everything else needs an argument.
Does it have to be a lottery?
No, but randomness makes the argument easy. Other instruments, such as the timing of a rollout or a rule that sorts people, need more careful justification and often a note on what could go wrong.
What if my first-stage F is below 10?
Treat the 2SLS estimate with caution. A weak instrument makes the estimate unstable and its confidence interval unreliable. Look for a stronger instrument, add one that is also valid, or use a different method.
Why is the instrumental estimate less precise?
Because it throws away the variation in the treatment that came from people choosing for themselves, and uses only the variation caused by the instrument. Less information gives wider intervals. A bigger sample helps.
Can I use it with survey data and sampling weights?
Yes. Set the survey weight in Data View and leave Apply survey weight ticked. The estimates and standard errors then reflect the survey design. The over-identification test is not shown with a survey weight.

Estimate a treatment effect when people choose their own treatment

Open Effect Analysis in AnalyZ Solutions. Free to sign up, free to analyze.

Get started
Related Guides