Confirmatory factor analysis tests whether the questions in your survey group into the topics you planned, and how well each question and each topic is measured. This guide covers the ideas, the calculations, every part of the output in AnalyZ Solutions, a worked example, and how to report it.
Confirmatory factor analysis checks whether your questionnaire measures what you designed it to measure. You decide in advance which questions belong together. The analysis then tells you whether the answers people gave behave that way.
Some things cannot be asked about directly. Nobody can answer a single question called "trust". So a questionnaire asks several questions that each touch on trust, and the answers are combined. The hidden thing behind the answers is called a factor. The questions are called items.
Suppose you surveyed 500 people in a host community about the services they receive. You wrote eight statements, rated from 1 (strongly disagree) to 5 (strongly agree). Four are about service quality, for example "Staff are respectful". Four are about trust in providers, for example "Staff keep my data private". You expect two factors.
Before you add the four answers together into a score, you want to know three things. Do the four quality items really move together? Are they separate from the trust items? And is any single item a poor fit? Confirmatory factor analysis answers all three.
Use it when you have a questionnaire with a planned structure and you want evidence that the structure holds in your data. Typical cases are a scale you wrote yourself, a published scale used in a new country or language, and a set of scores you plan to use later in a larger model such as a structural equation model.
It needs reasonable numbers. As a rule, you need 10 cases for every number the model estimates and never fewer than 100. With very small samples the fit numbers become unstable.
If you only want to know whether the items of one scale hang together, Cronbach's alpha is a lighter check. Confirmatory analysis tells you more, because it looks at each item and at how factors relate.
The analysis is quick to run, but the result is only as good as the set-up. Check these points first.
| Point | What is needed | If it is not met |
|---|---|---|
| Structure | Each factor and its items are decided from theory or from how the questionnaire was designed, not from this data. | The fit then describes a model built to fit, and it will probably fit worse on new data. |
| Items per factor | At least two items per factor. Three or more is much safer. A model with a single factor needs at least three items. | The module stops with a message, or the model cannot be estimated. |
| One factor per item | Each item belongs to one factor. | The module stops. To let an item load on a second factor, use the cross-loading change after a first run. |
| Sample size | At least 10 cases for every estimated number, and never fewer than 100 cases. The example below estimates 17 numbers, so 170 is the floor and 500 is comfortable. | Fit values and loadings become unstable, and the chance of a failed run rises. |
| Item scale | Numeric scores, or ordered categories such as a 1 to 5 agree scale. | Choose the variable type that matches your items (see the next section). |
| Direction of items | Items worded in reverse (for example "I do not trust staff") are recoded first so a high score always means more of the topic. | An unrecoded item shows a negative loading and drags down reliability. |
| Missing answers | A case with a missing answer on any item in the model is left out. The number of cases used is shown. | If many cases drop out, check the missing values in Data View first. |
| Near-duplicate items | Items should not be almost the same variable. | The module stops when two items are perfectly or almost perfectly correlated. Remove one. |
Figure 1 shows the model for the example. Each oval is a factor. Each box is a question. An arrow from a factor to a question says "this question is driven by this factor". The number on the arrow is the loading.
Read a loading like a correlation between the question and the factor. A loading of 0.75 means the factor explains a large part of how people answered. A loading of 0.30 means the question is mostly something else.
What the factor does not explain is called the error of the item. It includes real noise, such as a respondent hurrying, and anything else the item picks up. The curved arrow between the ovals is the correlation between the two factors. Factors are allowed to be related. They just should not be the same thing.
If two items share one factor, the model predicts how strongly they correlate. It is the product of their two loadings. Quality item 1 loads 0.748 and quality item 3 loads 0.671, so the model predicts a correlation of 0.748 × 0.671 = 0.50 between them.
The software tries many sets of loadings and error sizes and keeps the set that makes the predicted correlations closest to the real ones. The gaps that remain are called residuals. Small gaps everywhere mean a good fit. A large gap between two particular items means the model is missing something about that pair.
Four numbers are usually reported. None is perfect alone, so read them together.
Two numbers are worked out from the loadings of each factor. Composite reliability says how consistent the items are as a group. Average variance extracted (AVE) says how much of the items' variation the factor captures on average.
For trust, the four loadings are 0.863, 0.782, 0.713, 0.550. They add up to 2.908, and the squared sum is 8.46. The error parts, 1 minus each squared loading, add up to 1.83. So composite reliability is 8.46 ÷ (8.46 + 1.83) = 0.82. The average of the squared loadings gives an AVE of 0.54. These match the table in the example below.
Each answer is split into two parts. One part comes from the factor. The other part is error, which is everything else.
Two things follow. Items on the same factor should correlate with each other, and the strength of that correlation is set by their loadings. Items on different factors should correlate less, and the strength is set by their loadings and by the correlation between the factors.
A model is only testable if the data hold more information than the model uses. With 8 items there are 8×(8+1)/2 = 36 variances and covariances to explain. The first run of the model estimates 17 numbers: 6 loadings (the first item of each factor is fixed, see below), 8 error variances, 2 factor variances and 1 factor correlation. The difference, 36 - 17 = 19, is the degrees of freedom. It is the number of independent checks the model can fail. If the degrees of freedom were zero, the model would fit perfectly by design and there would be nothing to test. If they were negative, the model could not be estimated, and the module tells you to add items or remove paths.
A factor is invisible, so it has no natural unit. The module gives it one by fixing the loading of the first item of each factor, the marker item, at 1. The factor then runs on the same scale as that item. This only sets the unit. It does not change fit, and the standardised loadings you read are the same whichever item comes first. It is still worth putting your clearest item first.
For numeric items, the software uses maximum likelihood. It tries values for all the estimated numbers, works out the variances and covariances the model would produce, and compares them with the real ones. It keeps the values that make the gap smallest. The size of the leftover gap, multiplied by the number of cases, is the chi-square. The standard errors come from how sharply the gap grows when a number is moved away from its best value.
For ordered items, such as a 1 to 5 scale or yes or no answers, the module uses the approach that is standard for them. It treats each answer as a rough reading of an underlying continuous feeling, estimates the correlations between those underlying feelings (polychoric correlations) and the cut-points between answer categories (thresholds), and then fits the model with weighted least squares. The chi-square is adjusted for the departure from a perfect model, and the standard errors are robust. This avoids treating a 5-point scale as if the distance between each point were equal.
With a survey weight set, the numbers are estimated from the weighted correlations, and the standard errors come from the survey design. The fit statistics then use the effective sample size, which is smaller than the weighted total when the weights vary a lot.
For the first run in the example, chi-square is 40.4 on 19 degrees of freedom. The baseline model, in which no items are related, has a chi-square of 1575.8 on 28. So CFI = 1 - (40.4 - 19) / (1575.8 - 28) = 0.986. RMSEA = √((40.4 - 19) / (19 × 499)) = 0.047. The values match the table in the worked example.
CFI and TLI ask how much better the model is than a model that assumes nothing is related. RMSEA asks how big the leftover misfit is per degree of freedom, so it rewards simple models. SRMR is a plain average of the leftover gaps in correlation terms, so it is easy to picture: 0.035 means the model misses the real correlations by about 0.035 on average. The module also gives a 90% interval for RMSEA.
For every parameter that the model fixes at zero, the software can estimate how much the chi-square would fall if that parameter were freed. This is a modification index. It is reported with the expected change, which is how large the new number would be. The module lists two kinds. An error covariance means two items share something beyond their factor, such as similar wording. A cross-loading means an item also reflects a second factor.
A modification index above about 10 is a flag worth reading. In the example the first one is 33.4, for the error covariance between quality1 and quality2. Adding it is the same as letting those two items share something. Each change you add is listed in the "Changes added to the model" box, and you can reset to the original model at any time.
Convergent validity asks whether the items of a factor hang together, and the average variance extracted (AVE) measures it. Discriminant validity asks whether factors are distinct from each other. The module uses the heterotrait-monotrait ratio, a ratio of how much items on different factors correlate to how much items on the same factor correlate. Values near 1 mean the factors are hard to tell apart. Above 0.85 is a warning.
AnalyZ Solutions shows the chi-square, CFI, TLI, RMSEA with its range, and SRMR, and marks each against the benchmarks above. The summary also gives AIC and BIC. These are for comparing two models fitted to the same data. A lower value is better, and they have no benchmark on their own.
The loadings chart is colour coded. 0.70 or higher is strong, 0.50 to 0.69 is reasonable, 0.30 to 0.49 is weak, and below 0.30 is poor. Standardised loadings are used so items on different scales can be compared.
Composite reliability of 0.70 or higher and AVE of 0.50 or higher are the usual targets. To check that two factors are really different, the module shows the heterotrait-monotrait ratio. It is a measure of how alike two factors are. Values above 0.85 suggest people do not tell the two factors apart.
The module lists the biggest unexplained correlations between items. It also lists items that seem to belong partly to another factor. Each suggestion shows how much better the fit would get. You choose whether to add it, and the changes you add are listed so you can undo them.
Five-point answers are ranked categories, not true numbers. If your items have five or more categories and are not heavily lopsided, treating them as numbers usually works. For yes or no items, or items with only three categories or so, choose the option for a mix of binary, ordinal and continuous items. The analysis then uses a method made for ordered answers.
| Panel | What it shows | What to look for |
|---|---|---|
| Model fit | Chi-square, df, p-value, CFI, TLI, RMSEA with its range, SRMR, AIC and BIC. Each is marked good, acceptable or poor. | CFI and TLI at 0.95 or higher, RMSEA at 0.06 or lower, SRMR at 0.08 or lower. Do not rely on the chi-square alone. |
| Standardised loadings | A chart of every loading, coloured by size. | Green (0.70 or more) and teal (0.50 to 0.69) are fine. Amber (0.30 to 0.49) is weak. Red (below 0.30) is poor. |
| Factor loadings table | Raw and standardised loadings with standard errors, intervals and p-values. | A raw loading with an interval that includes zero means the item does not follow the factor. |
| Reliability and convergent validity | Composite reliability and AVE for each factor, with a verdict. | Reliability 0.70 or more. AVE 0.50 or more. AVE is stricter and often lower. |
| Factor correlations | The correlation between each pair of factors. | Very high values (above about 0.85) suggest the factors are not distinct. |
| Discriminant validity | The heterotrait-monotrait ratio for each pair. | Above 0.85 is a warning. |
| Largest unexplained correlations | The pairs of items whose real correlation the model misses most, with a suggested change. | Look for pairs of items with similar wording or placement. |
| Changes added to the model | The changes you have added, with a Reset changes button. | Report every change in your write-up. |
| Interpretation | A plain-language summary of the above. | Use it as a first draft. Check it against the tables. |
| Message | What it means | What to do |
|---|---|---|
| The error variance of an item is at or near zero | The factor explains almost all of the item. Often a sign the item is almost a copy of the factor, or the model asks too much of it. | Check the item. With two items on a factor, add a third. Consider whether another item is nearly identical. |
| A standardised loading is above 1 | The solution is improper. It can come from a small sample, too few items, or a badly specified factor. | Simplify the model. Check for a near-duplicate item. |
| The model did not fully converge | The search for the best numbers did not finish cleanly. | Treat the results with caution. Try a simpler model, fewer changes, or more cases. |
| The model is not identified | Some numbers cannot be separated from each other. | Give every factor at least three items, or remove redundant paths. |
| More estimates than the data can support | The model has too many numbers for the items you gave it. | Add items or remove changes. |
| Two items are almost perfectly correlated | The items carry the same information. | Remove one. |
Suggested changes are tempting because each one improves the fit. A sound routine looks like this.
Open Structural Equation Modelling and choose Confirmatory Factor Analysis. The module is marked Beta.
The figures below are invented to show the steps. They come from a real run of the module on a simulated survey of 500 people. The eight items are the ones in Figure 1.
| Chi-square | df | p-value | CFI | TLI | RMSEA (90% CI) | SRMR |
|---|---|---|---|---|---|---|
| 40.4 | 19 | 0.003 | 0.986 | 0.980 | 0.047 (0.027 to 0.068) | 0.035 |
Step 1: check the fit. CFI (0.986), TLI (0.980), RMSEA (0.047) and SRMR (0.035) all meet the good benchmarks. The chi-square test is significant, which is common and not alarming with 500 cases. Still, the largest unexplained correlation stands out. It is between quality1 and quality2, and fixing it would improve the chi-square by about 33.
Those two items are "Staff are respectful" and "Services are reliable". They were written one after the other on the form, and people who rush tend to answer neighbouring items alike. That is a reason you can put in words, so the correlation was added to the model.
| Chi-square | df | p-value | CFI | TLI | RMSEA (90% CI) | SRMR |
|---|---|---|---|---|---|---|
| 10.3 | 18 | 0.923 | 1.000 | 1.008 | 0.000 (0.000 to 0.013) | 0.013 |
Step 2: read the loadings. All eight standardised loadings are above 0.50. The weakest is trust4, "Staff will help me", at 0.55. It is acceptable, but it is the first item to look at if you ever shorten the scale.
| Item | Factor | Loading |
|---|---|---|
| quality1 | Service quality | 0.75 |
| quality2 | Service quality | 0.67 |
| quality3 | Service quality | 0.67 |
| quality4 | Service quality | 0.69 |
| trust1 | Trust in providers | 0.86 |
| trust2 | Trust in providers | 0.78 |
| trust3 | Trust in providers | 0.71 |
| trust4 | Trust in providers | 0.55 |
Step 3: check reliability. Both factors have composite reliability near or above 0.70, so the items of each factor are consistent. Trust has an AVE of 0.54. Service quality has an AVE of 0.48, just under the 0.50 target. That is a borderline result, not a failure. It says the quality items share a little less than half of their variation, so it is worth wording them more tightly next time.
| Factor | Composite reliability | AVE |
|---|---|---|
| Service quality | 0.79 | 0.48 |
| Trust in providers | 0.82 | 0.54 |
Step 4: check that the factors are separate. The two factors correlate at 0.60. The heterotrait-monotrait ratio is 0.58, well below 0.85. People did distinguish between the quality of services and trust in providers.
The eight questions behave as two separate topics, service quality and trust in providers. Every question is reasonably linked to its topic. The trust scale is consistent. The quality scale is consistent enough to use, but its items overlap a little less than ideal, and two neighbouring items share some wording. The two-topic structure is supported by this sample.
A short, complete report gives the sample, the method, the fit, the loadings, the reliability and the changes. Here is a model paragraph for the example.
| Factor and item | Standardised loading |
|---|---|
| Service quality (reliability 0.79, AVE 0.48) | |
| Staff are respectful | 0.75 |
| Services are reliable | 0.66 |
| Waiting times are fair | 0.67 |
| Complaints are taken seriously | 0.69 |
| Trust in providers (reliability 0.82, AVE 0.54) | |
| Staff are honest | 0.86 |
| Staff keep my data private | 0.78 |
| Staff act fairly | 0.71 |
| Staff will help me | 0.55 |
In the module, you can send the result to Canvas as a chart or to Sense Making alongside other analyses, and download the charts as images.
Using the same data to find the structure and to confirm it. If you changed the model until it fitted, the fit tells you little. Test the final structure on a fresh sample when you can.
Adding error correlations with no reason. Every one makes fit look better. Add only those you can explain.
Judging by one fit number. Read the chi-square, CFI, RMSEA and SRMR together, and also read the loadings.
Treating lopsided or very short answer scales as numbers. Use the ordered-answers option for these.
Factors with only two items. They can be estimated, but they are fragile. Three or more items per factor is safer.
Treating a good fit as proof the scale is valid. Fit shows that the answers are consistent with your structure. It does not show that the scale measures what you think it measures.
Open Structural Equation Modelling in AnalyZ Solutions. Free to sign up, free to analyze.
Get started