Quantitative Analysis/ Structural Equation Modelling
Can two groups be compared on the same scale?

Measurement Invariance: How to check that a questionnaire means the same thing for different groups

Before comparing women and men, regions or languages on a questionnaire score, you must check that the questions work the same way in every group. This guide covers the logic, the calculations, the rules for step failure, what to do about items that differ, every panel of the output, a worked example and how to report it.

24 min read Structural equation modelling Advanced

01What is measurement invariance?

Before you compare two groups on a scale, you need to know that the scale means the same thing to both. If women and men read the questions differently, a difference in scores may reflect that, and not a real difference in the topic. Measurement invariance is the check for this.

The same tools also answer a second question: do the effects in a model differ between groups? Does a link that holds for women hold equally for men? That is a test of equal paths. The Compare Groups module does both.

An example

You surveyed 274 men and 226 women about service quality (four questions) and trust in providers (four questions). Three more questions measure use of services. You want to know two things. Can you fairly compare women and men on quality and trust? And is the link from trust to use of services the same for both?

What you get out of it

  1. Fit at each level
    How well a model fits when the groups are forced to be more and more alike.
  2. A verdict at each step
    Whether the groups can be treated as alike at that level, or not.
  3. The items that differ
    Which questions behave differently across groups, so you can handle them.
  4. Differences in group averages
    Allowed once the scale is shown to work alike.
  5. Path comparisons
    The model's effects in each group, and tests of whether they differ.

02When to use it

Use it when you compare groups on a scale, for example countries, sexes, age groups, languages, or a programme group and a comparison group. Do it before you compare averages, and before you pool groups into one analysis.

Use the path comparison when your model has effects that might work differently in different groups, such as whether trust matters more for use of services among women than among men.

You need enough cases in each group. A rough guide is at least 100 in each group, and more for larger models. The module accepts two to eight groups. If you have not yet checked that the scale works within each group, start with confirmatory factor analysis.

03What you need before you start

Requirements and what happens if they are not met
PointWhat is neededIf it is not met
Grouping variableTwo to eight groups, with at least 30 cases in each. 100 or more in each is a safer aim.The module stops if a group is too small.
A scale that already holdsConfirmatory factor analysis should work in the whole sample first, with a good structure.If the structure fails overall, invariance results are not meaningful.
Same questions in every groupThe same items, in the same order and wording, for each group.Differences in wording may look like group differences.
Sample sizeOverall, 10 cases for every estimated number across the groups, and more for ordinal items.Differences between steps become unreliable.
Item scaleNumbers, or ordered categories for rating scales.Choose the type that matches the answers.

04The idea in a picture

Measurement invariance is tested in steps. Each step asks for more equality between groups than the last. Figure 1 shows the three main ones.

Figure 1. Three levels of measurement invarianceThree boxes in a row joined by arrows: same structure, then same loadings, then same starting points. Each level adds a stricter equality across groups.1. Same structure(configural)2. Same loadings(metric)3. Same starting points(scalar)addadd
Figure 1. Three levels of measurement invariance
  1. Same structure (configural). The same questions measure the same topics in every group. Everything else is free to differ. If this fails, the scale is not the same thing in the groups, and comparing them makes no sense.
  2. Same loadings (metric). Each question is tied to its topic with the same strength in every group. If this holds, you can compare how topics relate to other things across groups.
  3. Same starting points (scalar). People with the same level of the topic give the same answers, whatever their group. If this holds, you can compare group averages.

A fourth, stricter level asks for equal error sizes. The module offers it as an option, and it is not needed for most comparisons.

An everyday picture. Imagine measuring height with two rulers. They agree if their markings are the same distance apart (loadings), and if both start at zero (starting points). If one ruler starts at 3, you cannot compare readings, even if the markings are right.

05How it is calculated

Fitting the same model, with more and more equality

The software fits the model to all groups at once. At the first level, each group has its own numbers. At the second, the loadings are forced to be equal across groups. At the third, the starting points are forced equal as well. Each time, it records how well the model fits.

If forcing equality makes the fit much worse, the groups are not alike at that level.

How much worse is too much?

The chi-square test of the change in fit is one guide, but with larger samples it flags tiny differences. So the module also uses the change in three other measures, which are common benchmarks.

  1. CFI must not fall by more than 0.010.
  2. RMSEA must not rise by more than 0.015.
  3. SRMR must not rise by more than 0.030 when testing equal loadings, or 0.015 when testing equal starting points.

A step is supported when all three stay within these limits.

When one item differs

Often most items work alike and one or two do not. You can then allow those items to differ between groups and keep the rest equal. This is partial invariance. If most items are equal, it is generally accepted as enough to compare group averages.

Equal paths

For paths, the model is fitted twice: once with each group having its own effects, and once with the effects forced to be equal. The change in chi-square tells you whether the effects differ overall. The module also tests each path on its own.

06The model in more detail

The levels, built up step by step

What each level holds equal across groups
LevelHeld equalWhat it means if it holds
ConfiguralNothing. The same structure, with free numbers in each group.The groups understand the scale as made of the same topics and questions.
Metric (weak)LoadingsA one-point change in the topic gives the same change in each question in every group. Associations between the topic and other variables can be compared.
Scalar (strong)Loadings and question starting levels (intercepts or thresholds)Groups with the same level of the topic give the same average answers. Averages of the topic can be compared.
Strict (optional)Loadings, starting levels and error sizesThe noise is the same in every group. Needed only when comparing observed sum scores.

Each step is tested against the one before. Equal loadings must hold before equal starting levels make sense.

How a step is judged

Each step makes the model stricter, so fit can only get worse. The question is by how much. A chi-square difference test is too sensitive in large samples, so the module also uses three differences. A step fails if the difference in CFI is a drop of 0.010 or more, or the RMSEA rises by more than 0.015, or the SRMR rises by more than 0.030 (0.015 for the scalar step). CFI is the main guide.

change in CFI = CFI of the stricter model - CFI of the looser model
step holds if change in CFI is above -0.010, change in RMSEA is at most 0.015 and change in SRMR is within its limit

Counting what each level frees

With the example (8 questions, 2 topics, 2 groups), configural estimates every number separately in each group. The metric step forces the 6 free loadings to be equal, which adds 6 degrees of freedom. The scalar step forces the 8 question starting levels to be equal across the groups, adding 8 more. More degrees of freedom mean more checks the model can fail.

When some items differ

If the metric or scalar step fails, find the items that differ most. The module shows the loadings by group and a list of the items whose equality causes most misfit. You can allow those items to differ and re-run. This gives partial invariance. A common view is that a scale is usable if at least two items per topic stay equal. Report which items you freed and why.

Testing group means

When the scalar step holds, the module gives the difference in the average level of each topic between groups, with the first group as reference. Without the scalar step you cannot say that a difference in averages is a real difference. It may only be a difference in how the questions work.

Ordered answers

For rating scales, the module uses thresholds, the cut-points between answer categories, in place of intercepts. The steps become configural, then thresholds and loadings together, then more. The same changes in fit are used with the adjusted tests.

Equal paths

A different question is whether effects in a model are the same in each group, for example the effect of trust on use. The module fits the model once with the paths free and once with them held equal. If holding them equal makes the chi-square clearly worse, the paths differ. It can then tell which. This only makes sense once the measurement part is invariant, because otherwise a difference in a path may reflect a difference in measurement.

07Reading the output

Fit at each level

A table and a chart show the fit at each level. A chart of CFI makes it easy to see whether it drops from one level to the next.

Comparison between levels

This table lists the change in fit between levels and says whether each step is supported. Read it from top to bottom. Stop at the first step that fails.

Loadings by group

A chart shows the standardised loadings of every item in each group. Bars that differ a lot point to the items that cause trouble.

Which items differ most

When a level fails, the module shows what happens to fit when each item is freed on its own. The item with the largest improvement is the first to check.

Allow to differ and re-run

You can free the items you chose and run again. The results are then labelled as partial invariance.

Differences in group averages

When the starting points are equal, or partly so, the module shows the difference in each factor's average between the first group and each other group, with its error and p-value, and an effect size.

Path comparison

For equal paths, the module shows each path by group, with intervals, and the test of whether it differs between groups.

Do not compare averages before invariance holds. If the loadings or starting points differ and you ignore it, a gap between groups may be nothing more than a difference in how the questions work.

08How to run it in AnalyZ Solutions

Open Structural Equation Modelling and choose Compare Groups. The module is marked Beta.

  1. Choose the grouping variable
    The variable that defines the groups. It can have two to eight groups, with at least 30 cases in each.
  2. Choose what to test
    Measurement invariance, to see whether a scale means the same thing in every group. Or equal paths, to see whether effects differ between groups.
  3. Define the factors
    Name each factor and choose its items. For equal paths, also add the equations of your model.
  4. Say how answers are treated
    All continuous, or the mix of binary, ordinal and continuous.
  5. Click Run Compare Groups
    You get the fit at each level, the comparison between levels, the loadings by group, the items that differ and a plain-language interpretation.
  6. If a level fails, free the items that differ
    Use Allow to differ and re-run. Then read the results as partial invariance.
When your data come from a survey with sampling weights. With a survey weight set, every group is estimated with the weights, and the weighted number of cases is shown for each group.

09Reading the output, panel by panel

Every panel in the Compare Groups output
PanelWhat it showsWhat to look for
Fit at each levelFit of configural, metric, scalar (and strict)Good fit at the configural level first.
Comparison between levelsChanges in CFI, RMSEA and SRMR, with a verdictA step that holds is marked supported.
Loadings by groupEach loading in each groupItems where the loadings differ most.
Which items differ mostItems ranked by how much equality hurts fitThe first candidates to free.
Allow to differ and re-runChoose items to freeFree the fewest items you can defend.
Differences in group averagesDifferences in topic means, at the scalar levelA difference only counts if scalar holds.
Path comparisonPaths in each group, the test, and which differA significant difference in a few paths.
InterpretationA plain-language summaryA first draft.

10Interpreting the result

The figures below are invented to show the steps. They come from a real run of the module on a simulated survey of 500 people. The scales are service quality and trust in providers, as in the confirmatory factor analysis guide. Women answered one trust question, "Staff will help me", in a different way.

Part 1: can women and men be compared?

Fit at each level
LevelChi-squaredfCFIRMSEASRMR
Configural33.4361.0000.0000.024
Metric47.9420.9960.0240.046
Scalar53.0480.9970.0210.047
Change in fit between levels (chi-square, CFI, RMSEA and SRMR show the change)
StepChi-sq.dfpCFIRMSEASRMRSupported
Configural → Metric14.460.025-0.0040.0240.023No
Metric → Scalar5.260.5200.001-0.0030.001Yes

Step 1: structure. The first model fits well in both groups (CFI 1.000), so the same two topics are present for women and men.

Step 2: loadings. Forcing equal loadings worsens the fit by 14.4 chi-square points on 6 degrees of freedom (p = 0.025). CFI drops only a little, but RMSEA rises by 0.024, which is above the 0.015 limit. This step is not supported.

Standardised loadings of the trust questions, by group
ItemMenWomen
trust10.900.82
trust20.790.78
trust30.710.72
trust40.700.37

Step 3: find the cause. The list of items that differ points to trust4, "Staff will help me". The loading is 0.70 for men and 0.37 for women. Freeing this one loading alone improves the fit by 10.0 points (p = 0.002). In plain terms, this question tells you much less about trust for women than for men.

Step 4: allow trust4 to differ and run again. With only that loading free, the other seven loadings are equal across groups.

Fit at each level, with trust4 allowed to differ
LevelChi-squaredfCFIRMSEASRMR
Configural33.4361.0000.0000.024
Metric37.9411.0000.0000.030
Scalar42.9471.0000.0000.032
Change in fit between levels, with trust4 allowed to differ
StepChi-sq.dfpCFIRMSEASRMRSupported
Configural → Metric4.450.4880.0000.0000.006Yes
Metric → Scalar5.160.5350.0000.0000.002Yes

Step 5: read the new result. Both steps are now supported. Equal loadings hold for all items except trust4, and the starting points are equal too. This is partial invariance, and it is enough to compare group averages.

Step 6: compare the averages. Women score 0.04 higher than men on trust (error 0.10) and 0.11 higher on service quality (error 0.09). Both differences are small compared with their errors. There is no sign that the groups differ in average trust or quality.

Part 2: does trust lead to use equally for women and men?

Standardised paths by group
PathMenWomenChi-square for the differencep-value
Service quality → Trust0.590.630.010.918
Trust → Use of services0.480.687.790.005
Service quality → Use of services0.13-0.010.690.406

Step 7: read the paths. Quality has the same link to trust in both groups. The link from trust to use is much stronger for women (0.68) than for men (0.48). The test of that single path gives a p-value of 0.005. Making all paths equal worsens fit by 10.6 points on 3 degrees of freedom (p = 0.014), so the paths do differ overall, mostly through this one.

In plain words

The scale worked the same way for women and men, except for one question about whether staff will help. For women that question said less about trust. With that one difference allowed for, women and men could be compared fairly, and they did not differ in average trust or service quality. The link between trust and use of services was clearly stronger for women than for men. The results describe this sample, and they do not explain why the difference exists.

11How to report the result

Example write-up. We tested whether the trust and quality scales work alike for women and men (500 respondents). The configural model fitted well in both groups. Holding loadings equal worsened fit beyond the limits (change in RMSEA 0.024), mainly because trust4 loaded 0.70 for men and 0.37 for women. With the loading of trust4 allowed to differ, metric and scalar invariance both held, so partial invariance was supported and group means could be compared. Equal paths were then tested. Allowing the paths to differ improved the chi-square by 10.6 on 3 degrees of freedom. The effect of trust on use was stronger for women (0.68) than for men (0.48).

12A checklist before you trust the result

  1. The scale was checked by confirmatory factor analysis in the whole sample first.
  2. Every group has at least 30 cases, ideally 100 or more.
  3. You tested configural before metric and metric before scalar.
  4. You report the change in CFI, RMSEA and SRMR at each step.
  5. Freed items are few, listed by name, and have a reason.
  6. You compare group means only at the scalar level (full or partial).
  7. You test equal paths only after measurement holds.

13Common mistakes

Comparing group averages without checking invariance. A gap may reflect how the questions work and not the topic.

Treating the chi-square test as the only guide. With large samples it flags tiny differences. Look at the change in CFI, RMSEA and SRMR too.

Freeing many items. If many items differ, the scale is not the same across groups, and partial invariance will not rescue it.

Freeing items only to get a pass. Check the wording of each item and ask why it might differ.

Very small groups. They give unstable results. Check the number of cases in each.

Reading a difference in paths as a cause. It shows that effects differ. It does not say why.

Frequently Asked Questions
What is the difference between measurement invariance and equal paths?
Invariance asks whether a scale means the same thing in each group. Equal paths asks whether effects in a model, such as the effect of trust on use, are the same in each group.
What if the scale fails at the first level?
Then the groups do not share the same structure, and comparing them on this scale is not meaningful. Consider whether the scale needs revising or translating for one of the groups.
Is partial invariance good enough?
Generally, when most items are equal across groups, yes. Report which items you freed and why. If many items differ, the scale is not comparable.
How many groups can I compare?
From two to eight. Each group needs at least 30 cases, and 100 or more is a safer aim.
Can I use it with yes or no items?
Yes. Choose the mix of binary, ordinal and continuous items. The module then tests invariance for ordered categories.
What are the fit change limits based on?
They are common benchmarks from the research literature. A drop in CFI of more than 0.01 or a rise in RMSEA of more than 0.015 signals a meaningful loss of fit. They are guides, not rules.
Why not just compare the averages of women and men?
If a question works differently for the two groups, the averages are not on the same scale. A difference in averages may reflect the questions, not a real difference.
What is the difference between metric and scalar invariance?
Metric means each question relates to the topic equally strongly in all groups. Scalar means that in addition people with the same level of the topic give the same average answer. You need scalar to compare averages.
How many items can I free?
Keep at least two per topic equal. If you free more, the scale is not comparable in practice.
Why use changes in fit and not only the chi-square test?
The chi-square difference is significant in large samples for tiny, unimportant differences. Changes in CFI, RMSEA and SRMR judge how much worse the fit gets.
Does failing invariance mean the scale is bad?
Not necessarily. It means it does not work identically in the groups. It can be a finding in itself, for example when a translated item is read differently.

Compare groups fairly

Open Structural Equation Modelling in AnalyZ Solutions. Free to sign up, free to analyze.

Get started
Related Guides