Quantitative Analysis/ Effect Analysis
Comparing like with like

What is propensity score matching? Theory, calculation and application

Propensity score matching estimates what a programme did by comparing participants with similar people who did not take part. This guide explains the idea in plain language and shows how to run the analysis, check the match and read the result in AnalyZ Solutions.

12 min read Effect Analysis Intermediate

01What is propensity score matching?

Propensity score matching is a way of estimating what a programme did when the people in it were not picked at random. It builds a fair comparison group out of the people who did not take part, choosing those who look most like the participants, and then compares the two groups.

Here is the kind of problem it is made for. A vocational training programme runs in a district, and a survey covers 1,500 households. Of these, 544 took part. Participating households earn about $230 a month and the others about $180, a gap of $51. Is that what the training is worth?

Probably not, because of who took part. The participants were younger, had stayed in school longer and already earned more before the programme started. Those things raise income whether or not anyone attends a course. Some of the gap is the training. Some of it is the kind of household that signs up for training. Matching is a way of separating the two.

Average forParticipantsNon-participants
Age of household head (years)36.139.1
Years of schooling7.76.4
Household size4.95.0
Monthly income before the programme ($)117104
Distance to market (km)8.49.4
Monthly income now ($)230180

A few terms

What you get out of it

Matching can only correct for what you measured. If households joined because of something that is not in your data, such as ambition or a connection to the programme staff, matching cannot remove its influence. Section 5 explains the sensitivity check that helps you judge how much this matters.

02When to use it

Use propensity score matching when you have a single dataset, the programme was not assigned at random, and you recorded most of the reasons that made some people join and others stay out. It suits a one-off household or beneficiary survey that includes both participants and non-participants.

Three things need to be in place.

If you have measurements from before and after the programme for both groups, difference-in-differences usually makes better use of that information. If eligibility was decided by a score cutoff, use regression discontinuity. If people were invited at random but not everyone attended, use instrumental variables.

03The idea: comparing like with like

Suppose you want to compare participants with non-participants of the same age. That is easy. Add schooling, then household size, then distance to the market, and soon you are looking for a non-participant who is 36, has seven years of schooling, lives in a household of five and is eight kilometres from the market. Real data rarely contains such a twin.

The propensity score

The propensity score squeezes all those characteristics into one number: the estimated chance that a household with those characteristics takes part in the programme. A household that looks like the typical participant gets a high score, say 0.6. A household that looks like the typical non-participant gets a low one, say 0.2.

Two households with the same score had the same chance of joining. Judging only by what you measured, one of them joined and the other did not by luck. That is what makes them a fair pair. Comparing their incomes tells you something close to what the programme did, because the usual reasons for a difference have been evened out.

Overlap

Figure 1. Propensity scores of participants and non-participantsBar chart comparing the estimated chance of joining for participants and non-participants. Participants have higher scores on average, but the two groups overlap across most of the range.0%10%20%0.0-0.10.1-0.20.2-0.30.3-0.40.4-0.50.5-0.60.6-0.70.7-0.80.8-0.9Propensity score: the estimated chance of joining the programmeNon-participants (956)Participants (544)Share of the group
Figure 1. How likely each household was to join, before matching. The two groups overlap across most of the range

Matching only works where the two groups overlap. Figure 1 shows the scores in the example. Participants score higher on average (0.43 against 0.33), but both groups cover most of the range. At the top end a few participants have scores above 0.7 and almost no comparison households do. Those participants have no fair partner, so the matching leaves them out. This is called a lack of common support, and it is one reason the effect is reported only for households that could be matched.

04How it is calculated

The module works in four steps.

  1. Estimate the propensity score
    A logistic regression predicts participation from the covariates. Each household's predicted chance is its propensity score. The module works with the scores on the logit scale (the log of the odds), which stretches out the extremes.
  2. Build the comparison group
    Choose one of four techniques, described below.
  3. Compare outcomes
    In the matched or weighted sample, subtract the comparison outcome from the participant outcome and average.
  4. Check the result
    Test whether the groups now look alike, and ask how much a hidden factor would have to matter to change the answer.

The four techniques

TechniqueHow it worksGood to know
Nearest neighbor matching (1:1, caliper)Each participant is paired with the one non-participant whose score is closest, as long as the two are within a set distance called the caliper. No comparison household is used twice.Easy to follow and to check. Participants without a close partner are dropped.
Kernel matching (Gaussian)Each participant is compared with a weighted average of all non-participants, with closer scores counting for more.Keeps every participant. The bandwidth controls how quickly the weights fade with distance.
Inverse probability weightingNo pairs. Each household is weighted by the inverse of the chance of being in its own group, so non-participants who look like participants count for more.Uses everyone. Very high or very low scores can produce extreme weights.
Doubly robust (AIPW)Combines the weighting with a second model that predicts the outcome.The estimate holds if either of the two models is right.

If you are unsure, start with nearest neighbor matching, because you can see the pairs, and then check that kernel matching or the doubly robust estimate gives a similar answer. When different techniques agree, the result does not hang on one technical choice.

A small example of the arithmetic

Take three participants, each paired with the non-participant whose propensity score is closest. Their monthly incomes in dollars are:

PairParticipantMatched comparisonDifference
124020535
219817127
331028129

The average difference is (35 + 27 + 29) ÷ 3 = 30.3. The module does the same for every matched participant, 483 of them in the example.

Effect on the treated (ATT) = average of ( participant outcome - matched comparison outcome )

Which effect do you want?

The module reports one of three effects, chosen under Estimand.

How sure are we?

Each effect comes with a standard error, a 95% confidence interval and a p-value. For nearest neighbor and kernel matching, the standard error comes from the spread of the matched differences. For inverse probability weighting and the doubly robust estimate, the module repeats the whole analysis on resampled data, 300 times by default, and uses the spread of those results. This is called a bootstrap.

Checking the match

Standardised difference = ( average in participants - average in comparison ) / combined standard deviation

A value of 0 means the groups are identical on that characteristic. The usual rule is that anything between -0.1 and 0.1 counts as balanced. Rubin's B and R look at the propensity scores themselves. B should be below 25, and R, the ratio of the spreads, should be between 0.5 and 2.

05Reading the output

Matching summary

This shows how many participants and comparison households were in the data, the technique and the estimand. For matching techniques it also shows how many participants found a match. If many did not, the effect describes only the ones that did.

Propensity score model fit

A pseudo R-squared for the participation model. It tells you how well your characteristics separate participants from non-participants. It is not a score for the effect. A low value is not a problem for matching, because it means participation was hard to predict from what you measured. A high value, above about 0.4, means the two groups are easy to tell apart, so look at the overlap carefully.

Treatment effect estimate

The estimate is the effect in the units of the outcome. The table also shows the standard error, a t statistic, the p-value, the 95% confidence interval and two E-values, which are explained below.

Covariate balance and matching quality

The balance table lists each characteristic with its standardised difference before and after matching. The matching quality line counts how many are within 0.1 and gives the average reduction in the differences.

Sensitivity analysis

Matching cannot see what you did not measure, so the module asks how strong a hidden factor would have to be. The E-value is the minimum strength of association that a hidden factor would need with both participation and the outcome to explain the whole effect away. The second E-value does the same for the edge of the confidence interval closest to zero. A larger E-value means the finding is harder to overturn. The bias grid beneath it shows how the estimate would shrink under hypothetical hidden factors of increasing strength.

06How to run it in AnalyZ Solutions

Open Effect Analysis and choose Propensity Score Matching.

  1. Select the treatment variable
    The list shows variables with two values. The module treats the last value in alphabetical order as the treated group, so with labels such as Participant and Non-participant the treated group is Participant. If you are not sure, code treated as 1 and not treated as 0.
  2. Select the outcome variable
    Choose the measure you want to see change. It can be numeric, or a yes or no coded 1 and 0.
  3. Select the covariates
    Choose the characteristics measured before the programme that you think influenced both participation and the outcome.
  4. Choose the estimation technique
    Nearest neighbor matching, kernel matching, inverse probability weighting or doubly robust.
  5. Choose the estimand
    ATT, ATC or ATE.
  6. Adjust the options if needed
    The defaults are a caliper of 0.2, a kernel bandwidth of 0.06 and 300 bootstrap replications. Inverse probability weighting also offers stabilised weights, which tame extreme weights.
  7. Apply a subsample filter if needed
  8. Click Run matching
    You get the matching summary, the propensity score model fit, the effect estimate, Rubin's B and R, the matching quality summary, the covariate balance table, and the sensitivity analysis.
Read the balance table before the effect. A small p-value on top of a poor match is not a result. If balance is poor, change the covariates, the caliper or the technique and run it again.
When your data come from a survey with sampling weights. The propensity model is fitted with the survey weights, and the matching, balance checks and effect estimates carry the weights, so results describe the population the survey represents. Nearest neighbor matching uses the design variance of the weighted mean of matched differences. Kernel matching, inverse probability weighting and the doubly robust estimate use a bootstrap that resamples the survey's clusters within strata. Weighted N for the treated and comparison groups appears in the matching summary.

07Interpreting the result

The figures below are invented to show the steps. They come from running the module on a made-up dataset of 1,500 households, of which 544 took part in a vocational training programme and 956 did not. The outcome is monthly household income in dollars. The covariates are the age of the household head, years of schooling, household size, income before the programme and distance to the market. The technique is nearest neighbor matching and the estimand is ATT.

N treatedN controlN matchedCaliperPseudo R²
5449564830.20.078
EstimandEstimateSEtp-value95% CI lower95% CI upperE-valueE-value (CI)
ATT33.403.509.54< 0.00126.5440.264.323.48
CovariateSMD before matchingSMD after matching
Age of household head-0.334-0.013
Years of schooling0.472-0.014
Household size-0.018-0.015
Income before the programme0.2400.001
Distance to market-0.210-0.014
Rubin's B beforeRubin's B afterRubin's R beforeRubin's R after
68.050.311.041.01
Figure 2. Covariate balance before and after matchingDot plot of the standardised difference between participants and comparison households for five characteristics. Before matching, age, schooling, income and distance differ clearly; after matching all five sit inside the plus or minus 0.1 band.Good balance (within 0.1)-0.4-0.2+0.2+0.40Age of household headYears of schoolingHousehold sizeIncome before the programmeDistance to marketBefore matchingAfter matchingStandardised difference between participants and comparison households
Figure 2. Before matching the groups differed on four of five characteristics. After matching every difference is close to zero

Reading it step by step

  1. Check the match first
    Before matching, four of the five characteristics differed by more than 0.1, and schooling by almost half a standard deviation (0.47). After matching all five are within 0.02 of zero, and Rubin's B fell from 68 to 0.3. The comparison group now looks like the participants on everything that was measured.
  2. See how many were matched
    483 of 544 participants found a partner within the caliper. The other 61 did not, so the estimate is for the 483 who did, which is most of the group.
  3. Read the effect
    The ATT is 33.4. Among the households that took part, the training is estimated to have raised monthly income by about $33. The 95% confidence interval runs from $27 to $40, and the p-value is below 0.001, so an effect of zero is not plausible.
  4. Compare it with the raw gap
    The simple difference between participants and non-participants was $51. Matching removed about $17 of that, which is the part explained by who took part. Without matching, the programme would have been credited with about half as much again as the data support.
  5. Ask how fragile it is
    The E-value is 4.3, and 3.5 for the confidence interval. A hidden factor would have to be tied to both joining and income with that strength to explain the result away. That is a strong association. A good test is to ask whether you can name a factor, missing from your data, that would matter that much.

In plain words

Households that joined the vocational training earned about $33 more per month than similar households that did not, once age, schooling, household size, earlier income and distance to the market were taken into account (95% confidence interval $27 to $40). A direct comparison of participants and non-participants overstated the gain by about $17. Because matching adjusts only for measured characteristics, this is strong evidence of a positive effect and not proof of one.

08Common mistakes

Matching on things the programme changed. If the training raised income, then matching on income after the programme removes part of the effect you are trying to measure. Use only characteristics that were fixed or measured before the programme.

Leaving out an important reason for joining. If households with a link to the programme staff joined more often and you did not record it, matching cannot fix that. Think about why people joined before you choose the covariates.

Reading the p-value without reading the balance table. A significant effect on a poorly matched comparison group tells you very little.

Not saying how many participants were left out. If a large share finds no partner, the result describes a narrower group than you may intend. Report the numbers.

Treating the propensity model as a prediction exercise. The aim is balance, not a model that predicts participation well. Covariates that predict participation but have nothing to do with the outcome can make matching harder without improving the answer.

Treating a matched estimate as proof of cause. It is a careful comparison of similar households. It is only as good as the list of characteristics behind it.

Frequently Asked Questions
How many covariates should I include?
Include the characteristics that plausibly influenced both participation and the outcome and that were measured before the programme. Five to fifteen is common. Adding everything in the dataset is not safer, because covariates unrelated to the outcome can make matching harder, and covariates affected by the programme can bias the result.
Which technique should I choose?
Nearest neighbor matching is the easiest to explain because you can see the pairs. Kernel matching keeps every participant. Doubly robust is the most forgiving if your models are not perfect. Running two or three and comparing is good practice. If they agree, report one and mention the others.
What does the caliper do?
The caliper is the largest gap in propensity score allowed between a participant and the household matched to them. It is measured in standard deviations of the logit propensity score, and the default is 0.2. A smaller caliper gives closer matches but leaves more participants unmatched.
What is a good E-value?
There is no fixed cut-off. Judge it against the strength of the relationships you can see in your own data, and ask whether a missing factor could plausibly be tied to participation and the outcome that strongly. A value close to 1 means a weak hidden factor could explain the result.
Can I use it with survey data and sampling weights?
Yes. Set the survey weight in Data View and leave Apply survey weight ticked. The results then describe the population the survey represents, and the standard errors reflect the survey design.

Compare participants with a fair comparison group

Open Effect Analysis in AnalyZ Solutions. Free to sign up, free to analyze.

Get started
Related Guides