Before comparing women and men, regions or languages on a questionnaire score, you must check that the questions work the same way in every group. This guide covers the logic, the calculations, the rules for step failure, what to do about items that differ, every panel of the output, a worked example and how to report it.
Before you compare two groups on a scale, you need to know that the scale means the same thing to both. If women and men read the questions differently, a difference in scores may reflect that, and not a real difference in the topic. Measurement invariance is the check for this.
The same tools also answer a second question: do the effects in a model differ between groups? Does a link that holds for women hold equally for men? That is a test of equal paths. The Compare Groups module does both.
You surveyed 274 men and 226 women about service quality (four questions) and trust in providers (four questions). Three more questions measure use of services. You want to know two things. Can you fairly compare women and men on quality and trust? And is the link from trust to use of services the same for both?
Use it when you compare groups on a scale, for example countries, sexes, age groups, languages, or a programme group and a comparison group. Do it before you compare averages, and before you pool groups into one analysis.
Use the path comparison when your model has effects that might work differently in different groups, such as whether trust matters more for use of services among women than among men.
You need enough cases in each group. A rough guide is at least 100 in each group, and more for larger models. The module accepts two to eight groups. If you have not yet checked that the scale works within each group, start with confirmatory factor analysis.
| Point | What is needed | If it is not met |
|---|---|---|
| Grouping variable | Two to eight groups, with at least 30 cases in each. 100 or more in each is a safer aim. | The module stops if a group is too small. |
| A scale that already holds | Confirmatory factor analysis should work in the whole sample first, with a good structure. | If the structure fails overall, invariance results are not meaningful. |
| Same questions in every group | The same items, in the same order and wording, for each group. | Differences in wording may look like group differences. |
| Sample size | Overall, 10 cases for every estimated number across the groups, and more for ordinal items. | Differences between steps become unreliable. |
| Item scale | Numbers, or ordered categories for rating scales. | Choose the type that matches the answers. |
Measurement invariance is tested in steps. Each step asks for more equality between groups than the last. Figure 1 shows the three main ones.
A fourth, stricter level asks for equal error sizes. The module offers it as an option, and it is not needed for most comparisons.
The software fits the model to all groups at once. At the first level, each group has its own numbers. At the second, the loadings are forced to be equal across groups. At the third, the starting points are forced equal as well. Each time, it records how well the model fits.
If forcing equality makes the fit much worse, the groups are not alike at that level.
The chi-square test of the change in fit is one guide, but with larger samples it flags tiny differences. So the module also uses the change in three other measures, which are common benchmarks.
A step is supported when all three stay within these limits.
Often most items work alike and one or two do not. You can then allow those items to differ between groups and keep the rest equal. This is partial invariance. If most items are equal, it is generally accepted as enough to compare group averages.
For paths, the model is fitted twice: once with each group having its own effects, and once with the effects forced to be equal. The change in chi-square tells you whether the effects differ overall. The module also tests each path on its own.
| Level | Held equal | What it means if it holds |
|---|---|---|
| Configural | Nothing. The same structure, with free numbers in each group. | The groups understand the scale as made of the same topics and questions. |
| Metric (weak) | Loadings | A one-point change in the topic gives the same change in each question in every group. Associations between the topic and other variables can be compared. |
| Scalar (strong) | Loadings and question starting levels (intercepts or thresholds) | Groups with the same level of the topic give the same average answers. Averages of the topic can be compared. |
| Strict (optional) | Loadings, starting levels and error sizes | The noise is the same in every group. Needed only when comparing observed sum scores. |
Each step is tested against the one before. Equal loadings must hold before equal starting levels make sense.
Each step makes the model stricter, so fit can only get worse. The question is by how much. A chi-square difference test is too sensitive in large samples, so the module also uses three differences. A step fails if the difference in CFI is a drop of 0.010 or more, or the RMSEA rises by more than 0.015, or the SRMR rises by more than 0.030 (0.015 for the scalar step). CFI is the main guide.
With the example (8 questions, 2 topics, 2 groups), configural estimates every number separately in each group. The metric step forces the 6 free loadings to be equal, which adds 6 degrees of freedom. The scalar step forces the 8 question starting levels to be equal across the groups, adding 8 more. More degrees of freedom mean more checks the model can fail.
If the metric or scalar step fails, find the items that differ most. The module shows the loadings by group and a list of the items whose equality causes most misfit. You can allow those items to differ and re-run. This gives partial invariance. A common view is that a scale is usable if at least two items per topic stay equal. Report which items you freed and why.
When the scalar step holds, the module gives the difference in the average level of each topic between groups, with the first group as reference. Without the scalar step you cannot say that a difference in averages is a real difference. It may only be a difference in how the questions work.
For rating scales, the module uses thresholds, the cut-points between answer categories, in place of intercepts. The steps become configural, then thresholds and loadings together, then more. The same changes in fit are used with the adjusted tests.
A different question is whether effects in a model are the same in each group, for example the effect of trust on use. The module fits the model once with the paths free and once with them held equal. If holding them equal makes the chi-square clearly worse, the paths differ. It can then tell which. This only makes sense once the measurement part is invariant, because otherwise a difference in a path may reflect a difference in measurement.
A table and a chart show the fit at each level. A chart of CFI makes it easy to see whether it drops from one level to the next.
This table lists the change in fit between levels and says whether each step is supported. Read it from top to bottom. Stop at the first step that fails.
A chart shows the standardised loadings of every item in each group. Bars that differ a lot point to the items that cause trouble.
When a level fails, the module shows what happens to fit when each item is freed on its own. The item with the largest improvement is the first to check.
You can free the items you chose and run again. The results are then labelled as partial invariance.
When the starting points are equal, or partly so, the module shows the difference in each factor's average between the first group and each other group, with its error and p-value, and an effect size.
For equal paths, the module shows each path by group, with intervals, and the test of whether it differs between groups.
Open Structural Equation Modelling and choose Compare Groups. The module is marked Beta.
| Panel | What it shows | What to look for |
|---|---|---|
| Fit at each level | Fit of configural, metric, scalar (and strict) | Good fit at the configural level first. |
| Comparison between levels | Changes in CFI, RMSEA and SRMR, with a verdict | A step that holds is marked supported. |
| Loadings by group | Each loading in each group | Items where the loadings differ most. |
| Which items differ most | Items ranked by how much equality hurts fit | The first candidates to free. |
| Allow to differ and re-run | Choose items to free | Free the fewest items you can defend. |
| Differences in group averages | Differences in topic means, at the scalar level | A difference only counts if scalar holds. |
| Path comparison | Paths in each group, the test, and which differ | A significant difference in a few paths. |
| Interpretation | A plain-language summary | A first draft. |
The figures below are invented to show the steps. They come from a real run of the module on a simulated survey of 500 people. The scales are service quality and trust in providers, as in the confirmatory factor analysis guide. Women answered one trust question, "Staff will help me", in a different way.
| Level | Chi-square | df | CFI | RMSEA | SRMR |
|---|---|---|---|---|---|
| Configural | 33.4 | 36 | 1.000 | 0.000 | 0.024 |
| Metric | 47.9 | 42 | 0.996 | 0.024 | 0.046 |
| Scalar | 53.0 | 48 | 0.997 | 0.021 | 0.047 |
| Step | Chi-sq. | df | p | CFI | RMSEA | SRMR | Supported |
|---|---|---|---|---|---|---|---|
| Configural → Metric | 14.4 | 6 | 0.025 | -0.004 | 0.024 | 0.023 | No |
| Metric → Scalar | 5.2 | 6 | 0.520 | 0.001 | -0.003 | 0.001 | Yes |
Step 1: structure. The first model fits well in both groups (CFI 1.000), so the same two topics are present for women and men.
Step 2: loadings. Forcing equal loadings worsens the fit by 14.4 chi-square points on 6 degrees of freedom (p = 0.025). CFI drops only a little, but RMSEA rises by 0.024, which is above the 0.015 limit. This step is not supported.
| Item | Men | Women |
|---|---|---|
| trust1 | 0.90 | 0.82 |
| trust2 | 0.79 | 0.78 |
| trust3 | 0.71 | 0.72 |
| trust4 | 0.70 | 0.37 |
Step 3: find the cause. The list of items that differ points to trust4, "Staff will help me". The loading is 0.70 for men and 0.37 for women. Freeing this one loading alone improves the fit by 10.0 points (p = 0.002). In plain terms, this question tells you much less about trust for women than for men.
Step 4: allow trust4 to differ and run again. With only that loading free, the other seven loadings are equal across groups.
| Level | Chi-square | df | CFI | RMSEA | SRMR |
|---|---|---|---|---|---|
| Configural | 33.4 | 36 | 1.000 | 0.000 | 0.024 |
| Metric | 37.9 | 41 | 1.000 | 0.000 | 0.030 |
| Scalar | 42.9 | 47 | 1.000 | 0.000 | 0.032 |
| Step | Chi-sq. | df | p | CFI | RMSEA | SRMR | Supported |
|---|---|---|---|---|---|---|---|
| Configural → Metric | 4.4 | 5 | 0.488 | 0.000 | 0.000 | 0.006 | Yes |
| Metric → Scalar | 5.1 | 6 | 0.535 | 0.000 | 0.000 | 0.002 | Yes |
Step 5: read the new result. Both steps are now supported. Equal loadings hold for all items except trust4, and the starting points are equal too. This is partial invariance, and it is enough to compare group averages.
Step 6: compare the averages. Women score 0.04 higher than men on trust (error 0.10) and 0.11 higher on service quality (error 0.09). Both differences are small compared with their errors. There is no sign that the groups differ in average trust or quality.
| Path | Men | Women | Chi-square for the difference | p-value |
|---|---|---|---|---|
| Service quality → Trust | 0.59 | 0.63 | 0.01 | 0.918 |
| Trust → Use of services | 0.48 | 0.68 | 7.79 | 0.005 |
| Service quality → Use of services | 0.13 | -0.01 | 0.69 | 0.406 |
Step 7: read the paths. Quality has the same link to trust in both groups. The link from trust to use is much stronger for women (0.68) than for men (0.48). The test of that single path gives a p-value of 0.005. Making all paths equal worsens fit by 10.6 points on 3 degrees of freedom (p = 0.014), so the paths do differ overall, mostly through this one.
The scale worked the same way for women and men, except for one question about whether staff will help. For women that question said less about trust. With that one difference allowed for, women and men could be compared fairly, and they did not differ in average trust or service quality. The link between trust and use of services was clearly stronger for women than for men. The results describe this sample, and they do not explain why the difference exists.
Comparing group averages without checking invariance. A gap may reflect how the questions work and not the topic.
Treating the chi-square test as the only guide. With large samples it flags tiny differences. Look at the change in CFI, RMSEA and SRMR too.
Freeing many items. If many items differ, the scale is not the same across groups, and partial invariance will not rescue it.
Freeing items only to get a pass. Check the wording of each item and ask why it might differ.
Very small groups. They give unstable results. Check the number of cases in each.
Reading a difference in paths as a cause. It shows that effects differ. It does not say why.
Open Structural Equation Modelling in AnalyZ Solutions. Free to sign up, free to analyze.
Get started