Instrumental variables analysis estimates the effect of a treatment that people chose for themselves by using a nudge that is unrelated to the hidden reasons for choosing. This guide explains the idea in plain language and shows how to run and read it in AnalyZ Solutions.
Instrumental variables analysis estimates the effect of a treatment when the people who receive it differ from those who do not in ways you cannot measure. It borrows a source of variation that is unrelated to those hidden differences, called an instrument, and uses only that part of the variation to learn what the treatment does.
Here is a typical case. A training course is offered to 1,000 people, and invitations are handed out by lottery. About half are invited. Not everyone invited attends, and some people who were not invited find a way in. You recorded who attended and each person's monthly income afterwards.
| Average monthly income ($) | Number of people | |
|---|---|---|
| Attended the course | 126.0 | 404 |
| Did not attend | 82.4 | 596 |
People who attended earn about $44 more. Is that the value of the course? Probably not. The people who attend a course are often the more motivated ones, and motivation raises income without any course at all. Comparing attendees with non-attendees mixes the effect of the course with the effect of the kind of person who attends. You cannot measure motivation, so you cannot adjust for it.
The lottery gets around this. Who was invited was settled by chance, so it has nothing to do with motivation. And an invitation can only affect income by getting people to attend. Using the lottery as the instrument isolates the attendance that the lottery caused, and that part is free of the motivation problem.
Use instrumental variables when you cannot reach a fair comparison with the other methods, because people chose or were steered into the treatment for reasons you cannot measure, and you have a credible instrument. Good instruments are rare. The usual ones are:
Figure 1 shows the logic. Motivation, which you cannot measure, pushes both attendance and income up. That is why a plain comparison misleads. The invitation is different. It moves attendance, but it is not connected to motivation and has no other route to income. So any difference in income between the invited and the not invited must have travelled through attendance.
| Requirement | In plain words | Can you check it? |
|---|---|---|
| Relevance | The instrument really moves the treatment. Invited people attend much more often than people who were not invited. | Yes. The module reports a first-stage strength test. |
| As good as random | The instrument is unrelated to the hidden things that affect the outcome. A lottery guarantees this. | By design, not by test. You argue for it from how the instrument arose. |
| Only through the treatment | The instrument has no effect on the outcome except by changing the treatment. An invitation that also came with a free toolkit would break this. | Not with a single instrument. You argue for it from knowledge of the programme. |
Only some people respond to an instrument. Among the 1,000 people, 64% of those invited attended and 15% of those not invited did. Some people would attend either way, some would never attend, and some attend only if invited. The method learns about the last group, those whose attendance changed because of the invitation. The estimate is the effect of the course for them. It is called a local average treatment effect. It may not hold for people who would always attend, or who would never attend.
With one instrument and one treatment, the result can be written as a ratio.
In the example, being invited raised income by $12.20 on average. But the invitation moved attendance by only 0.49 of a person, that is, 49 percentage points. So the $12.20 came from 49% of people changing their behaviour, and the effect per person whose attendance changed is 12.20 ÷ 0.49 = 24.9. That is the 2SLS estimate.
If the instrument barely moves the treatment, the denominator is close to zero and the ratio swings wildly. The module tests the strength of the first stage with an F statistic. A common rule of thumb is that F should be above 10. Below that the instrument is called weak, and the estimate should be treated with caution.
The module reports robust standard errors (HC1) with a t statistic, p-value and 95% confidence interval for both the plain and the instrumental estimate. The instrumental interval is wider, because the method uses only part of the variation in the data.
The number of observations, the treatment, the number of instruments and covariates, the outcome type and whether the model is just-identified (one instrument per treatment) or over-identified (more).
The F statistic for the instruments in the first stage, with a warning if it falls below 10. This answers the relevance question and nothing else. A strong first stage does not make an instrument valid.
How much each instrument moves the treatment. In the example, being invited raises the chance of attending by 0.49, or 49 percentage points.
The plain (OLS) estimate and the instrumental (2SLS) estimate side by side, each with its confidence interval. The gap between them shows how much hidden differences were distorting the plain comparison.
The first-stage R-squared is a real fit measure for the model that predicts the treatment. The R-squared for the 2SLS model is shown for reference only. It can be low or even negative and is not a guide to whether the effect estimate is any good.
The effect of the instrument directly on the outcome. It needs no assumptions beyond randomness of the instrument. If the reduced form shows nothing, there is nothing to scale up, whatever the 2SLS estimate says.
The tests described above. The over-identification test appears only when there is more than one instrument.
Open Effect Analysis and choose Instrumental Variables (2SLS).
The figures below are invented to show the steps, and they come from running the module on a made-up dataset of 1,000 people. The outcome is monthly income in dollars, the treatment is attending the training course (1 for attended), the instrument is the lottery invitation, and the covariate is age.
| N observations | Treatment | Instruments | Covariates | Outcome type | Identification |
|---|---|---|---|---|---|
| 1,000 | attended | 1 | 1 | Continuous | Just-identified |
| First-stage F | p-value | Instrument | Coefficient | SE | t |
|---|---|---|---|---|---|
| 330.1 | < 0.001 | invited | 0.4902 | 0.0266 | 18.40 |
| Estimator | Estimate | SE | t | p-value | 95% CI lower | 95% CI upper |
|---|---|---|---|---|---|---|
| OLS (naive, ignores endogeneity) | 43.01 | 2.55 | 16.88 | < 0.001 | 38.01 | 48.00 |
| 2SLS (instrumented) | 24.89 | 5.27 | 4.72 | < 0.001 | 14.55 | 35.22 |
| Reduced form: instrument | Coefficient | SE | t | p-value |
|---|---|---|---|---|
| invited | 12.20 | 2.83 | 4.32 | < 0.001 |
| Endogeneity test | Coefficient | SE | t | p-value |
|---|---|---|---|---|
| Residual from first stage | 24.12 | 5.97 | 4.04 | < 0.001 |
| First-stage R² | Adjusted R² | 2SLS R² (reference only) |
|---|---|---|
| 0.250 | 0.248 | 0.205 |
People whose decision to attend depended on the lottery invitation earned about $25 more per month because of the training (95% confidence interval $15 to $35). Comparing attendees with non-attendees directly would have suggested $43, mostly because attendees were more motivated to begin with. The result applies to people whose attendance was changed by the invitation and relies on the invitation affecting income only through attendance.
Using a weak instrument. If the first-stage F is below 10, the estimate can be far off and its confidence interval misleading. Look for a stronger instrument or do not rely on the result.
Using an instrument with a direct route to the outcome. An invitation that also came with a toolkit, or distance to a clinic that also tracks wealth, breaks the key assumption. The software cannot detect this.
Choosing a variable because it is available. An instrument needs an argument, not just a correlation with the treatment.
Reading the estimate as the effect for everyone. It describes people whose treatment was changed by the instrument.
Judging the model by its R-squared. The 2SLS R-squared is not a measure of quality and can be negative.
Concluding the plain regression is wrong just because the two estimates differ. The difference points to hidden differences only if the instrument is valid. If it is not, the instrumental estimate can be the worse of the two.
Open Effect Analysis in AnalyZ Solutions. Free to sign up, free to analyze.
Get started