Quantitative Analysis/ Structural Equation Modelling
Effects through middle steps, for any kind of outcome

Counterfactual Mediation: How to split an effect into the part that goes through middle steps and the part that does not

Counterfactual mediation splits an effect into the part that goes through one or more middle steps and the part that does not. It works with yes or no outcomes, covariates, serial mediators and groups, and it checks how fragile the result is. This guide covers the definitions, the method, every panel, a worked example and how to report it.

26 min read Structural equation modelling Advanced

01What is counterfactual mediation?

Counterfactual mediation answers the same question as mediation analysis: how much of an effect goes through a middle step? It does it in a way that works when your outcome or your middle step is a yes or no, or a set of ordered categories, and when you want to take other variables into account.

The word counterfactual means "what would have happened otherwise". The method compares the average result in situations that you can describe in words, such as "everyone trained" and "no one trained". The size of each effect is given in the units of the outcome. For a yes or no outcome, that is percentage points.

An example

A programme offered job training to 900 young adults. Half took part, and taking part was voluntary. Six months later you know whether each person is employed (yes or no), and you have a confidence score and a job-search skills score. You also know age and sex. About 22% of the group are employed. You think training helps people into work by raising confidence and job-search skills. You want to know four things. How many percentage points does training add to the chance of being employed? How much of that goes through confidence, and how much through job-search skills? Is there any effect left that neither explains? And how sure can you be if some unmeasured factor is at work?

What you get out of it

  1. Total, direct and indirect effect
    In percentage points for a yes or no outcome, with intervals.
  2. Effects by route
    The indirect effect split by mediator, and a test of whether two routes differ.
  3. Share mediated
    The part of the total effect that goes through the mediators.
  4. Sensitivity
    How strong an unmeasured factor would need to be to change the conclusion.
  5. Options
    Several mediators, a categorical exposure, and results for each group separately.

02When to use it

Use it when your outcome or mediator is yes or no or an ordered scale, when you want to adjust for other variables such as age and sex, when you have up to four mediators, or when you want the result in plain percentage points. It also covers an exposure with several categories, and results by group.

If your three variables are all numeric and you need nothing else, plain mediation analysis is quicker. For longer chains, use path analysis.

This is not a way to prove cause. The results describe causal effects only if nothing important is left out and the order in time is right. The sensitivity panel helps you judge how much that matters.

03What you need before you start

Requirements and what happens if they are not met
PointWhat is neededIf it is not met
ExposureNumeric, yes or no, or a category variable with 3 to 10 categories. For categories, you pick a reference category.Other types are not accepted.
MediatorsOne to four. Numeric, yes or no, or ordered. Parallel if they do not affect each other, serial if they follow each other.Wrong order in a serial set gives a wrong split.
OutcomeNumeric, yes or no, or an ordered scale.Choose the type that matches the answers.
CovariatesVariables that could affect the mediator and outcome, such as age and sex. They are held fixed.Missing important covariates leaves hidden common causes.
Sample sizeSeveral hundred cases, and more for a yes or no outcome with few cases in one category. The example uses 900.Intervals widen and some resamples fail. The module reports failed resamples.
OrderExposure first, then mediators in order, then the outcome. Covariates are measured before the exposure.A mediator measured after the outcome cannot mediate.

04The idea in a picture

Figure 1 shows the example. Training may raise confidence and job-search skills. Each of those may raise the chance of employment. Training may also help in ways the two mediators do not capture, which is the direct arrow. Age and sex are held constant in every step.

Figure 1. Two mediators in parallelJob training points to confidence and to job-search skills. Each of those points to employment. Job training also points straight to employment. Age and sex are held constant throughout.Job training(yes / no)ConfidenceJob-searchskillsEmployed(yes / no)direct
Figure 1. Two mediators in parallel between training and employment

The method splits the total effect by comparing three situations, shown in Figure 2.

Figure 2. The three situations comparedThree boxes in a row. No training, then training with mediators kept at their no-training levels, then training with mediators at their trained levels. The first step is the direct effect and the second is the indirect effect.No trainingTraining, mediatorsas without trainingTraining, mediatorsas with trainingdirect effectindirect effect
Figure 2. The three situations compared
  1. Nobody trained. The average chance of employment, with everyone at their no-training levels of confidence and skills.
  2. Training, with mediators kept as they would be without it. Training is switched on, but confidence and skills stay at the levels people would have had without training.
  3. Training, with mediators as they would be with it. Training is on, and confidence and skills rise as training makes them rise.

The step from the first to the second is the direct effect. It is what training does with the mediators held back. The step from the second to the third is the indirect effect. It is what changes only because the mediators change. The two together are the total effect.

05How it is calculated

Four steps

  1. Model each mediator
    The mediator is predicted from the exposure and the other variables you adjust for. A numeric mediator uses a linear model. A yes or no mediator uses a probit model. An ordered mediator uses an ordered probit model.
  2. Model the outcome
    The outcome is predicted from the exposure, the mediators and the other variables. A yes or no outcome uses a probit model.
  3. Simulate the three situations
    For every person in your data, the models predict the outcome in each of the three situations. The averages across people are compared.
  4. Resample for intervals
    The whole procedure is repeated on many resamples of your data. The middle 95% of the results forms the interval.

In the example, the average chance of employment is about 15.2% with nobody trained, about 21.9% with training and mediators held back, and about 28.1% with training and mediators as they actually change. The direct effect is the first step, 21.9 - 15.2 = 6.7 points. The indirect effect is the second step, 28.1 - 21.9 = 6.2 points. The total is 12.9 points.

Why not just multiply two coefficients? With a yes or no outcome, the numbers in the two models are on different scales, so a product does not mean what it seems to. Simulating the situations and comparing averages avoids that problem.

Sensitivity to unmeasured factors

Imagine something you did not measure, such as motivation. It could make people more likely to join training and also more likely to find work. Then part of what looks like an effect of training is really motivation. The E-value says how strong that link would have to be to explain the result away.

risk ratio = chance with training / chance without training
E-value = risk ratio + square root of ( risk ratio × (risk ratio - 1) )

For the total effect, the chance of employment goes from 15.2% to 28.1%, a risk ratio of 1.85. The E-value is 1.85 + √(1.85 × 0.85) = 3.09. An unmeasured factor would need to be linked to both training and employment by a risk ratio of about 3.1 each, beyond age and sex, to explain the total effect away. To reach an interval that includes zero, a factor of 2.38 would be enough.

06The model in more detail

Definitions

Let X be the exposure (trained or not), M the mediator (confidence) and Y the outcome (employed). Write Y(x, m) for the outcome a person would have if X were set to x and M set to m. Write M(x) for the mediator they would have if X were set to x. We never see both versions for one person. The method asks what would happen on average if everyone were in each situation.

total effect = average of Y(1, M(1)) - Y(0, M(0))
natural direct effect = average of Y(1, M(0)) - Y(0, M(0))
natural indirect effect = average of Y(1, M(1)) - Y(1, M(0))
total = direct + indirect

In words: the direct effect is the change in outcome if the exposure changes while the mediator stays where it would have been without the exposure. The indirect effect is the change if the exposure is held at its trained level and only the mediator moves from the level it would have without the exposure to the level it would have with it.

How the simulation works

  1. Fit a model for each mediator
    Mediator = exposure + covariates. Numeric mediators use a linear model, yes or no a probit model, and ordered an ordered-probit model.
  2. Fit a model for the outcome
    Outcome = exposure + mediators + covariates (and exposure × mediator if you allow an interaction).
  3. Predict for every person in three situations
    Not trained, trained with the mediator that comes with training, and trained with the mediator that comes without training. The mediator values are simulated from the fitted models, which is why this is called g-computation by simulation, and the results are averaged over people.
  4. Average and subtract
    The three averages give the total, direct and indirect effects.
  5. Resample
    The whole procedure runs on many resamples of your data. The middle 95% is the interval. Failed resamples are counted and reported.

Why the unit is percentage points

For a yes or no outcome, effects are differences in the share with the outcome. In the example 15.2% of untrained people would have had a job. The total effect is 12.9 percentage points, so 28.1% of everyone would have a job if all were trained. Of the 12.9, 6.7 points are direct and 6.2 points go through confidence, so about 48% is indirect.

Several mediators

Parallel mediators each carry part of the effect and do not affect each other. The module gives the effect through each, and tests whether two routes differ. Serial mediators follow an order, such as training, then knowledge, then confidence. The order matters, because the model assumes each mediator can affect the ones after it. The module splits the indirect effect by route. Put the mediators in the order you believe in and say why. With the exposure-mediator interaction switched on, the effect of the mediator may differ for trained and untrained people.

Categorical exposure and results by group

With a categorical exposure, each category is compared with the reference category, and the module gives total, direct and indirect effects for each comparison. With results by group (two to six groups), the whole analysis runs inside each group. The groups are compared using paired bootstrap resamples, so the difference in effects has its own interval.

Sensitivity to hidden factors

The result needs no hidden factor that affects both the mediator and the outcome. The sensitivity panel shows how fragile the result is to that assumption in two ways.

E-value for a risk ratio RR (with RR above 1) = RR + square root of ( RR × (RR - 1) )

The E-value is the minimum strength of link, on the risk ratio scale, that a hidden factor would need with both the exposure and the outcome to fully explain the effect away. In the example, the E-value is 3.09 for the total effect and 1.89 for the indirect effect (2.38 and 1.57 for the interval limits closest to no effect). A hidden factor would need to be linked to both training and employment by a risk ratio of about 3 to explain away the total effect. For the indirect effect, a risk ratio of about 1.9 would do it, so the indirect result is less sturdy than the total.

The hidden-correlation curve shows the indirect effect as a function of the correlation between the errors of the mediator and outcome models. The point where the curve crosses zero tells how much hidden correlation would erase the indirect effect.

Assumptions

07Reading the output

Effects table and chart

The table lists the total, direct and indirect effects, with the indirect effect split by mediator. Each has an interval from the resampling. The chart shows the same as a forest plot. An interval that does not include zero is a clear effect.

Comparing routes

With several mediators, the module tests whether two routes differ in size. If the interval of the difference includes zero, you cannot say one mediator matters more than the other.

Share mediated

This is the indirect effect divided by the total effect, with an interval. Treat it as a rough guide, and be careful when the total effect is small.

The models behind the numbers

The mediator and outcome models are shown, so you can check the direction of every link.

Sensitivity panel

A collapsed panel gives E-values for the total, direct and indirect effects. When there is one continuous mediator and a continuous outcome, it also shows how the indirect effect changes if the unmeasured factors behind the mediator and the outcome are correlated.

A categorical exposure, and results by group

If the exposure has three to ten categories, the module compares each category with a reference category you choose. It can also run the model separately for each group, such as women and men, and test whether the effects differ between groups.

08How to run it in AnalyZ Solutions

Open Structural Equation Modelling and choose Counterfactual Mediation. The module is marked Beta.

  1. Choose the exposure (X)
    Yes or no, numeric, or a categorical variable with up to ten categories.
  2. Choose the mediator (M) and any more
    Add up to three more. Choose how they relate: parallel if they do not affect each other, serial if the first affects the second.
  3. Choose the outcome (Y)
    Numeric, yes or no, or an ordered scale.
  4. Add covariates
    Optional. Variables such as age and sex that you want to hold constant.
  5. Choose what to show
    Everyone together, a categorical exposure compared with a reference category, or each group separately.
  6. Set the size of the exposure change
    For a numeric exposure, one standard deviation or one unit.
  7. Click Run Counterfactual Mediation
    You get the effects, the chart, the models, the sensitivity panel and a plain-language interpretation.
When your data come from a survey with sampling weights. With a survey weight set, every model and average uses the weights, the intervals follow the survey design, and the weighted number of cases is shown.

09Reading the output, panel by panel

Every part of the Counterfactual Mediation output
PanelWhat it showsWhat to look for
Effects table and chartTotal, direct and indirect effects, with intervals, in percentage points or in the outcome's units.An interval that avoids zero.
Effects by routeThe indirect effect through each mediator, and tests of differences.Which route carries most.
Share mediatedIndirect divided by total, with a note when it is not meaningful.Shown only when the effects point the same way.
The modelsThe coefficients of the mediator and outcome models.Check that signs are sensible.
SensitivityE-values and the hidden-correlation curve.Values close to 1 mean the result is fragile.
By groupEffects for each group and the differences.Whether the difference interval avoids zero.
InterpretationA plain-language summary.A first draft.

10Interpreting the result

The figures below are invented to show the steps. They come from a real run of the module on a simulated dataset of 900 young adults, with 1,000 resamples. The exposure is training (yes or no), the mediators are confidence and job-search skills, and age and sex are covariates. The outcome is employed (yes or no).

Effects of training on the chance of being employed (percentage points)
EffectEstimate95% CIp-value
Total effect12.97.7 to 18.20.001
Direct effect6.71.0 to 12.10.020
Indirect effect6.23.3 to 9.00.001
through confidence3.81.8 to 5.90.001
through job-search skills2.40.7 to 4.50.002

Step 1: read the total effect. Training adds about 12.9 percentage points to the chance of being employed (interval 7.7 to 18.2). That is from 15.2% without training to about 28.1% with it.

Step 2: split it. About 6.2 points are indirect and 6.7 points are direct. Both intervals stay above zero. Confidence carries 3.8 points and job-search skills 2.4.

Step 3: compare the routes. The difference between the two routes is 1.3 points (interval -1.4 to 4.1). That includes zero, so the data do not show one mediator to be more important than the other.

Step 4: read the share mediated. About 48% of the total effect goes through the two mediators (interval 25% to 86%). The interval is wide, so the share is only a rough guide.

Sensitivity to an unmeasured factor
EffectEstimate (points)E-valueE-value to reach an interval that includes zero
Total effect12.93.092.38
Indirect effect6.21.891.57

Step 5: check the sensitivity. To explain the total effect away, a hidden factor would need to be linked to both training and employment by a risk ratio of about 3.1 each. For the indirect effect, the figure is 1.9. That is a weaker link, so the indirect result is easier to challenge than the total. Whether such a factor is likely depends on what you know about how people joined the programme.

In plain words

Young adults who took the job training were about 13 percentage points more likely to be employed six months later. About half of that gain went through higher confidence and better job-search skills. The rest was not explained by these two. The data cannot say which of the two mediators matters more. These results would need a fairly strong hidden factor to disappear, but the indirect part is less secure than the total. As the training was not assigned at random, differences in motivation could still play a part.

11How to report the result

Example write-up. We estimated whether the effect of a job-skills training on employment works through confidence, in 900 participants, adjusting for age and sex. Without training, 15.2% would be employed. Training raised employment by 12.9 percentage points (95% bootstrap interval in the table). 6.7 points were a direct effect and 6.2 points went through confidence, so about half of the effect was indirect. An unmeasured factor would need a risk ratio of about 3.1 with both training and employment to explain away the total effect, and about 1.9 to explain away the indirect effect. The indirect result should therefore be read with more caution than the total.

12A checklist before you trust the result

  1. Exposure, mediators and outcome are in a clear order.
  2. Covariates cover the main common causes.
  3. For serial mediators, you can defend the order.
  4. You report the number of resamples and any that failed.
  5. You report effects in the right unit: percentage points for a yes or no outcome.
  6. You report E-values and read them.
  7. You say "would be" about counterfactual figures, not "was".

13Common mistakes

Reading the result as proof of cause. It is an estimate of a causal effect under assumptions. Use the sensitivity panel and say so in your report.

Adjusting for the wrong things. Include variables that come before the exposure. Do not adjust for variables that the exposure changes, because that removes part of the effect you want to measure.

Ignoring the order of mediators. In a serial model with a yes or no variable, the split of routes can depend on the order, so choose the order from your theory.

Too many mediators for the sample. Each adds a model. Keep to what your theory needs.

Treating a wide interval as no effect. It may mean too little data to tell.

Comparing groups without checking group sizes. Small groups give wide, unstable intervals.

Frequently Asked Questions
What is the difference between this and the simpler mediation analysis?
The simpler analysis multiplies regression coefficients and works only with numeric variables. Counterfactual mediation also handles yes or no and ordered variables, covariates, several mediators, categorical exposures and groups, and reports effects in outcome units.
What does counterfactual mean here?
It means a comparison of what would happen under different situations, such as everyone trained against no one trained, estimated from your data and models.
What are percentage points?
The difference between two percentages. Going from 15% to 28% is a rise of 13 percentage points, not 13%.
What is an E-value?
It is the minimum strength of link an unmeasured factor would need with both the exposure and the outcome to explain an effect away. Larger values mean the result is harder to explain away.
Can I use it with a numeric exposure such as the number of sessions?
Yes. You choose whether the effect is for a one standard deviation increase or a one unit increase.
Why would I show results by group?
To see whether the direct and indirect effects differ, for example between women and men. The module also tests the difference between groups.
Is the indirect effect "natural" or "controlled"?
Natural. It lets the mediator take the value it would have had anyway. A controlled effect fixes the mediator at one value for everyone. The natural version answers how much of the effect goes through the mediator.
Why does the order of serial mediators matter?
Because the model assumes the first affects the second. Reversing them changes the effects.
Why are the numbers in percentage points?
Because for a yes or no outcome the effect is a difference between two shares.
What does an E-value of 1.9 tell me?
A hidden factor would need a risk ratio of 1.9 with both exposure and outcome to explain the result away. It is not a verdict. Judge whether such a factor is plausible in your field.
Why would some resamples fail?
In a yes or no model, a resample may have too few cases in a category. The module reports how many were dropped. Many failures mean the model is too large for the sample.

Split an effect into its routes

Open Structural Equation Modelling in AnalyZ Solutions. Free to sign up, free to analyze.

Get started
Related Guides