Quantitative Analysis/ Structural Equation Modelling
Checking a questionnaire against your design

Confirmatory Factor Analysis: How to check that your questionnaire measures what you designed it to measure

Confirmatory factor analysis tests whether the questions in your survey group into the topics you planned, and how well each question and each topic is measured. This guide covers the ideas, the calculations, every part of the output in AnalyZ Solutions, a worked example, and how to report it.

22 min read Structural equation modelling Intermediate

01What is confirmatory factor analysis?

Confirmatory factor analysis checks whether your questionnaire measures what you designed it to measure. You decide in advance which questions belong together. The analysis then tells you whether the answers people gave behave that way.

Some things cannot be asked about directly. Nobody can answer a single question called "trust". So a questionnaire asks several questions that each touch on trust, and the answers are combined. The hidden thing behind the answers is called a factor. The questions are called items.

An example

Suppose you surveyed 500 people in a host community about the services they receive. You wrote eight statements, rated from 1 (strongly disagree) to 5 (strongly agree). Four are about service quality, for example "Staff are respectful". Four are about trust in providers, for example "Staff keep my data private". You expect two factors.

Before you add the four answers together into a score, you want to know three things. Do the four quality items really move together? Are they separate from the trust items? And is any single item a poor fit? Confirmatory factor analysis answers all three.

What you get out of it

  1. Fit
    One set of numbers that says how well your planned structure matches the answers.
  2. Loadings
    How strongly each item reflects its factor. A weak item is a candidate to drop or reword.
  3. Reliability
    Whether the items of a factor are consistent enough to add into one score.
  4. Separation
    Whether two factors are really different, or whether people answer them as one thing.
It tests a structure you choose. If you do not yet know how your items group, start with exploratory factor analysis. Confirmatory analysis comes after, ideally on a new sample.

02When to use it

Use it when you have a questionnaire with a planned structure and you want evidence that the structure holds in your data. Typical cases are a scale you wrote yourself, a published scale used in a new country or language, and a set of scores you plan to use later in a larger model such as a structural equation model.

It needs reasonable numbers. As a rule, you need 10 cases for every number the model estimates and never fewer than 100. With very small samples the fit numbers become unstable.

If you only want to know whether the items of one scale hang together, Cronbach's alpha is a lighter check. Confirmatory analysis tells you more, because it looks at each item and at how factors relate.

03What you need before you start

The analysis is quick to run, but the result is only as good as the set-up. Check these points first.

Requirements and what happens if they are not met
PointWhat is neededIf it is not met
StructureEach factor and its items are decided from theory or from how the questionnaire was designed, not from this data.The fit then describes a model built to fit, and it will probably fit worse on new data.
Items per factorAt least two items per factor. Three or more is much safer. A model with a single factor needs at least three items.The module stops with a message, or the model cannot be estimated.
One factor per itemEach item belongs to one factor.The module stops. To let an item load on a second factor, use the cross-loading change after a first run.
Sample sizeAt least 10 cases for every estimated number, and never fewer than 100 cases. The example below estimates 17 numbers, so 170 is the floor and 500 is comfortable.Fit values and loadings become unstable, and the chance of a failed run rises.
Item scaleNumeric scores, or ordered categories such as a 1 to 5 agree scale.Choose the variable type that matches your items (see the next section).
Direction of itemsItems worded in reverse (for example "I do not trust staff") are recoded first so a high score always means more of the topic.An unrecoded item shows a negative loading and drags down reliability.
Missing answersA case with a missing answer on any item in the model is left out. The number of cases used is shown.If many cases drop out, check the missing values in Data View first.
Near-duplicate itemsItems should not be almost the same variable.The module stops when two items are perfectly or almost perfectly correlated. Remove one.

04The idea in a picture

Figure 1 shows the model for the example. Each oval is a factor. Each box is a question. An arrow from a factor to a question says "this question is driven by this factor". The number on the arrow is the loading.

Figure 1. Two factors measured by eight questionsTwo ovals, service quality and trust in providers, each with four arrows pointing to four survey questions. The numbers on the arrows are standardised loadings. A curved double arrow joins the two ovals.Service qualityTrust in providersStaff arerespectfulStaff arehonestServices arereliableStaff keep mydata privateWaiting timesare fairStaff actfairlyComplaints aretaken seriouslyStaff willhelp me0.750.860.670.780.670.710.690.550.60
Figure 1. Two factors measured by eight questions, with the loadings from the example below

Read a loading like a correlation between the question and the factor. A loading of 0.75 means the factor explains a large part of how people answered. A loading of 0.30 means the question is mostly something else.

What the factor does not explain is called the error of the item. It includes real noise, such as a respondent hurrying, and anything else the item picks up. The curved arrow between the ovals is the correlation between the two factors. Factors are allowed to be related. They just should not be the same thing.

05How it is calculated

Matching the correlations

If two items share one factor, the model predicts how strongly they correlate. It is the product of their two loadings. Quality item 1 loads 0.748 and quality item 3 loads 0.671, so the model predicts a correlation of 0.748 × 0.671 = 0.50 between them.

The software tries many sets of loadings and error sizes and keeps the set that makes the predicted correlations closest to the real ones. The gaps that remain are called residuals. Small gaps everywhere mean a good fit. A large gap between two particular items means the model is missing something about that pair.

Fit numbers

Four numbers are usually reported. None is perfect alone, so read them together.

  1. Chi-square test. Asks whether the model matches the data exactly. With large samples it nearly always says no for tiny reasons, so it is a first look, not a verdict.
  2. CFI and TLI. Compare your model with a model in which nothing is related. 0.95 or higher is good, 0.90 or higher is acceptable.
  3. RMSEA. The size of the average misfit, with a range around it. 0.06 or lower is good, 0.08 or lower is acceptable.
  4. SRMR. The average size of the residuals. 0.08 or lower is good, 0.10 or lower is acceptable.

Reliability and average variance extracted

Two numbers are worked out from the loadings of each factor. Composite reliability says how consistent the items are as a group. Average variance extracted (AVE) says how much of the items' variation the factor captures on average.

composite reliability = (sum of loadings)² / ( (sum of loadings)² + sum of (1 - loading²) )
AVE = average of (loading²)

For trust, the four loadings are 0.863, 0.782, 0.713, 0.550. They add up to 2.908, and the squared sum is 8.46. The error parts, 1 minus each squared loading, add up to 1.83. So composite reliability is 8.46 ÷ (8.46 + 1.83) = 0.82. The average of the squared loadings gives an AVE of 0.54. These match the table in the example below.

06The model in more detail

The equations

Each answer is split into two parts. One part comes from the factor. The other part is error, which is everything else.

answer to item = loading × factor score + error
loading = how strongly the item follows the factor

Two things follow. Items on the same factor should correlate with each other, and the strength of that correlation is set by their loadings. Items on different factors should correlate less, and the strength is set by their loadings and by the correlation between the factors.

correlation of two items on the same factor = loading 1 × loading 2
correlation of two items on different factors = loading 1 × loading 2 × correlation of the factors

Counting what can be estimated

A model is only testable if the data hold more information than the model uses. With 8 items there are 8×(8+1)/2 = 36 variances and covariances to explain. The first run of the model estimates 17 numbers: 6 loadings (the first item of each factor is fixed, see below), 8 error variances, 2 factor variances and 1 factor correlation. The difference, 36 - 17 = 19, is the degrees of freedom. It is the number of independent checks the model can fail. If the degrees of freedom were zero, the model would fit perfectly by design and there would be nothing to test. If they were negative, the model could not be estimated, and the module tells you to add items or remove paths.

Giving a factor a scale

A factor is invisible, so it has no natural unit. The module gives it one by fixing the loading of the first item of each factor, the marker item, at 1. The factor then runs on the same scale as that item. This only sets the unit. It does not change fit, and the standardised loadings you read are the same whichever item comes first. It is still worth putting your clearest item first.

How the numbers are found

For numeric items, the software uses maximum likelihood. It tries values for all the estimated numbers, works out the variances and covariances the model would produce, and compares them with the real ones. It keeps the values that make the gap smallest. The size of the leftover gap, multiplied by the number of cases, is the chi-square. The standard errors come from how sharply the gap grows when a number is moved away from its best value.

For ordered items, such as a 1 to 5 scale or yes or no answers, the module uses the approach that is standard for them. It treats each answer as a rough reading of an underlying continuous feeling, estimates the correlations between those underlying feelings (polychoric correlations) and the cut-points between answer categories (thresholds), and then fits the model with weighted least squares. The chi-square is adjusted for the departure from a perfect model, and the standard errors are robust. This avoids treating a 5-point scale as if the distance between each point were equal.

With a survey weight set, the numbers are estimated from the weighted correlations, and the standard errors come from the survey design. The fit statistics then use the effective sample size, which is smaller than the weighted total when the weights vary a lot.

The fit measures, with a calculation

chi-square = number of cases × size of the gap between the model and the data
CFI = 1 - (chi-square - df) / (chi-square of the no-relations model - its df)
TLI = ( baseline chi-square/df - model chi-square/df ) / ( baseline chi-square/df - 1 )
RMSEA = square root of ( (chi-square - df) / ( df × (cases - 1) ) )
SRMR = square root of the average squared difference between observed and predicted correlations

For the first run in the example, chi-square is 40.4 on 19 degrees of freedom. The baseline model, in which no items are related, has a chi-square of 1575.8 on 28. So CFI = 1 - (40.4 - 19) / (1575.8 - 28) = 0.986. RMSEA = √((40.4 - 19) / (19 × 499)) = 0.047. The values match the table in the worked example.

CFI and TLI ask how much better the model is than a model that assumes nothing is related. RMSEA asks how big the leftover misfit is per degree of freedom, so it rewards simple models. SRMR is a plain average of the leftover gaps in correlation terms, so it is easy to picture: 0.035 means the model misses the real correlations by about 0.035 on average. The module also gives a 90% interval for RMSEA.

Suggested changes

For every parameter that the model fixes at zero, the software can estimate how much the chi-square would fall if that parameter were freed. This is a modification index. It is reported with the expected change, which is how large the new number would be. The module lists two kinds. An error covariance means two items share something beyond their factor, such as similar wording. A cross-loading means an item also reflects a second factor.

A modification index above about 10 is a flag worth reading. In the example the first one is 33.4, for the error covariance between quality1 and quality2. Adding it is the same as letting those two items share something. Each change you add is listed in the "Changes added to the model" box, and you can reset to the original model at any time.

Reliability and validity from the loadings

Convergent validity asks whether the items of a factor hang together, and the average variance extracted (AVE) measures it. Discriminant validity asks whether factors are distinct from each other. The module uses the heterotrait-monotrait ratio, a ratio of how much items on different factors correlate to how much items on the same factor correlate. Values near 1 mean the factors are hard to tell apart. Above 0.85 is a warning.

07Reading the output

Model fit

AnalyZ Solutions shows the chi-square, CFI, TLI, RMSEA with its range, and SRMR, and marks each against the benchmarks above. The summary also gives AIC and BIC. These are for comparing two models fitted to the same data. A lower value is better, and they have no benchmark on their own.

Loadings

The loadings chart is colour coded. 0.70 or higher is strong, 0.50 to 0.69 is reasonable, 0.30 to 0.49 is weak, and below 0.30 is poor. Standardised loadings are used so items on different scales can be compared.

Reliability and separation

Composite reliability of 0.70 or higher and AVE of 0.50 or higher are the usual targets. To check that two factors are really different, the module shows the heterotrait-monotrait ratio. It is a measure of how alike two factors are. Values above 0.85 suggest people do not tell the two factors apart.

Suggested changes

The module lists the biggest unexplained correlations between items. It also lists items that seem to belong partly to another factor. Each suggestion shows how much better the fit would get. You choose whether to add it, and the changes you add are listed so you can undo them.

Do not add changes only to improve fit. A suggestion is worth taking only when you can explain it in plain words, for example two items that are almost the same sentence. Chasing fit numbers on one sample often produces a model that does not hold on the next.

Types of answers

Five-point answers are ranked categories, not true numbers. If your items have five or more categories and are not heavily lopsided, treating them as numbers usually works. For yes or no items, or items with only three categories or so, choose the option for a mix of binary, ordinal and continuous items. The analysis then uses a method made for ordered answers.

08Reading the output, panel by panel

Every panel in the Confirmatory Factor Analysis output
PanelWhat it showsWhat to look for
Model fitChi-square, df, p-value, CFI, TLI, RMSEA with its range, SRMR, AIC and BIC. Each is marked good, acceptable or poor.CFI and TLI at 0.95 or higher, RMSEA at 0.06 or lower, SRMR at 0.08 or lower. Do not rely on the chi-square alone.
Standardised loadingsA chart of every loading, coloured by size.Green (0.70 or more) and teal (0.50 to 0.69) are fine. Amber (0.30 to 0.49) is weak. Red (below 0.30) is poor.
Factor loadings tableRaw and standardised loadings with standard errors, intervals and p-values.A raw loading with an interval that includes zero means the item does not follow the factor.
Reliability and convergent validityComposite reliability and AVE for each factor, with a verdict.Reliability 0.70 or more. AVE 0.50 or more. AVE is stricter and often lower.
Factor correlationsThe correlation between each pair of factors.Very high values (above about 0.85) suggest the factors are not distinct.
Discriminant validityThe heterotrait-monotrait ratio for each pair.Above 0.85 is a warning.
Largest unexplained correlationsThe pairs of items whose real correlation the model misses most, with a suggested change.Look for pairs of items with similar wording or placement.
Changes added to the modelThe changes you have added, with a Reset changes button.Report every change in your write-up.
InterpretationA plain-language summary of the above.Use it as a first draft. Check it against the tables.

Warnings and what to do

Messages you may see
MessageWhat it meansWhat to do
The error variance of an item is at or near zeroThe factor explains almost all of the item. Often a sign the item is almost a copy of the factor, or the model asks too much of it.Check the item. With two items on a factor, add a third. Consider whether another item is nearly identical.
A standardised loading is above 1The solution is improper. It can come from a small sample, too few items, or a badly specified factor.Simplify the model. Check for a near-duplicate item.
The model did not fully convergeThe search for the best numbers did not finish cleanly.Treat the results with caution. Try a simpler model, fewer changes, or more cases.
The model is not identifiedSome numbers cannot be separated from each other.Give every factor at least three items, or remove redundant paths.
More estimates than the data can supportThe model has too many numbers for the items you gave it.Add items or remove changes.
Two items are almost perfectly correlatedThe items carry the same information.Remove one.

Changing the model without fooling yourself

Suggested changes are tempting because each one improves the fit. A sound routine looks like this.

  1. Run the model you planned
    Do not change it before you have seen the result.
  2. Read fit, loadings and reliability first
    If they are good, you may not need any change.
  3. Take a suggestion only with a reason in words
    "These two items are almost the same sentence" is a reason. "The number is large" is not.
  4. Add one change at a time
    Each one affects the others. Re-run after each.
  5. Stop early
    Two or three well-reasoned changes is usually the limit. If you need many, the structure is wrong, and you should rethink it or test it on new data.
  6. Report every change
    Someone reading your work should be able to rebuild the final model from your description.

09How to run it in AnalyZ Solutions

Open Structural Equation Modelling and choose Confirmatory Factor Analysis. The module is marked Beta.

  1. Name your first factor
    Give it a short name, such as Service quality.
  2. Choose its items
    Pick the questions that belong to the factor. Each factor needs at least two. Add more factors as needed.
  3. Say how the answers are treated
    Choose all continuous, or the mix of binary, ordinal and continuous.
  4. Click Run Confirmatory Factor Analysis
    You get the fit table, the loadings chart, reliability, factor correlations, the separation check, suggested changes, and a plain-language interpretation.
  5. Review the suggestions
    Add a change only if you can justify it. The analysis re-runs with it.
When your data come from a survey with sampling weights. With a survey weight set, the model is estimated with the weights and the weighted number of cases is shown next to the sample size.

10Interpreting the result

The figures below are invented to show the steps. They come from a real run of the module on a simulated survey of 500 people. The eight items are the ones in Figure 1.

Model fit, first run
Chi-squaredfp-valueCFITLIRMSEA (90% CI)SRMR
40.4190.0030.9860.9800.047 (0.027 to 0.068)0.035

Step 1: check the fit. CFI (0.986), TLI (0.980), RMSEA (0.047) and SRMR (0.035) all meet the good benchmarks. The chi-square test is significant, which is common and not alarming with 500 cases. Still, the largest unexplained correlation stands out. It is between quality1 and quality2, and fixing it would improve the chi-square by about 33.

Those two items are "Staff are respectful" and "Services are reliable". They were written one after the other on the form, and people who rush tend to answer neighbouring items alike. That is a reason you can put in words, so the correlation was added to the model.

Model fit, after allowing quality1 and quality2 to be related
Chi-squaredfp-valueCFITLIRMSEA (90% CI)SRMR
10.3180.9231.0001.0080.000 (0.000 to 0.013)0.013

Step 2: read the loadings. All eight standardised loadings are above 0.50. The weakest is trust4, "Staff will help me", at 0.55. It is acceptable, but it is the first item to look at if you ever shorten the scale.

Standardised loadings
ItemFactorLoading
quality1Service quality0.75
quality2Service quality0.67
quality3Service quality0.67
quality4Service quality0.69
trust1Trust in providers0.86
trust2Trust in providers0.78
trust3Trust in providers0.71
trust4Trust in providers0.55

Step 3: check reliability. Both factors have composite reliability near or above 0.70, so the items of each factor are consistent. Trust has an AVE of 0.54. Service quality has an AVE of 0.48, just under the 0.50 target. That is a borderline result, not a failure. It says the quality items share a little less than half of their variation, so it is worth wording them more tightly next time.

Reliability and convergent validity
FactorComposite reliabilityAVE
Service quality0.790.48
Trust in providers0.820.54

Step 4: check that the factors are separate. The two factors correlate at 0.60. The heterotrait-monotrait ratio is 0.58, well below 0.85. People did distinguish between the quality of services and trust in providers.

In plain words

The eight questions behave as two separate topics, service quality and trust in providers. Every question is reasonably linked to its topic. The trust scale is consistent. The quality scale is consistent enough to use, but its items overlap a little less than ideal, and two neighbouring items share some wording. The two-topic structure is supported by this sample.

11How to report the result

A short, complete report gives the sample, the method, the fit, the loadings, the reliability and the changes. Here is a model paragraph for the example.

Example write-up. A two-factor confirmatory factor analysis was run on the answers of 500 respondents using maximum likelihood. The planned model fitted well (chi-square = 40.4, df = 19, p = .003; CFI = .986; TLI = .980; RMSEA = .047, 90% CI .027 to .068; SRMR = .035). The largest suggested change was a correlation between the errors of quality1 and quality2, which were written one after the other with a similar sentence. Adding it improved the fit further (chi-square = 10.3, df = 18; CFI = 1.000; RMSEA = .000; SRMR = .013). Standardised loadings ranged from .55 to .86. Composite reliability was .79 for service quality and .82 for trust in providers, and average variance extracted was .48 and .54. The two factors correlated at .60 (heterotrait-monotrait ratio = .58), so they were treated as distinct.
A table for your report
Factor and itemStandardised loading
Service quality (reliability 0.79, AVE 0.48)
Staff are respectful0.75
Services are reliable0.66
Waiting times are fair0.67
Complaints are taken seriously0.69
Trust in providers (reliability 0.82, AVE 0.54)
Staff are honest0.86
Staff keep my data private0.78
Staff act fairly0.71
Staff will help me0.55

In the module, you can send the result to Canvas as a chart or to Sense Making alongside other analyses, and download the charts as images.

12A checklist before you trust the result

  1. The structure was fixed from theory or design before looking at this data.
  2. Every factor has at least three items, or you can explain why not.
  3. Reverse-worded items were recoded.
  4. You chose the variable type to match the items: numbers, or ordered categories.
  5. You counted the cases used and checked that it meets 10 cases per estimated number and at least 100.
  6. The fit is good on several measures, not only one.
  7. All loadings are at least 0.50, or you can explain each lower one.
  8. Reliability is at least 0.70 and AVE is close to or above 0.50.
  9. Factors are distinct: the heterotrait-monotrait ratio is at most 0.85.
  10. Every change you added has a reason in words, and you report it.
  11. You say that fit does not prove the scale measures what you intend.

13Common mistakes

Using the same data to find the structure and to confirm it. If you changed the model until it fitted, the fit tells you little. Test the final structure on a fresh sample when you can.

Adding error correlations with no reason. Every one makes fit look better. Add only those you can explain.

Judging by one fit number. Read the chi-square, CFI, RMSEA and SRMR together, and also read the loadings.

Treating lopsided or very short answer scales as numbers. Use the ordered-answers option for these.

Factors with only two items. They can be estimated, but they are fragile. Three or more items per factor is safer.

Treating a good fit as proof the scale is valid. Fit shows that the answers are consistent with your structure. It does not show that the scale measures what you think it measures.

Frequently Asked Questions
What is the difference between exploratory and confirmatory factor analysis?
Exploratory analysis asks the data how items group. Confirmatory analysis tests a grouping you decided on beforehand. Use exploratory first when the structure is unknown, then confirm it on new data. If you explore and confirm on the same data, the confirmation is weak.
What is a good loading?
0.70 or higher is strong. 0.50 or higher is usually reasonable. Between 0.30 and 0.49 is weak. Below 0.30 the item says very little about the factor and is a candidate to drop or reword.
My fit is poor. What do I do?
Read the largest unexplained correlations and the loadings. Often one or two items cause most of the problem. Reword or remove them, or add a change you can justify in words. Do not keep adding changes until the numbers look good. If many changes are needed, the structure is not right.
How many items does a factor need?
At least two, and three or four is much safer. With only two items, a factor depends on the rest of the model to be estimated and is fragile.
Why is the chi-square test almost always significant?
The test asks whether the model is exactly right. With hundreds of cases, even tiny and unimportant misfit is detected. That is why CFI, TLI, RMSEA and SRMR are read alongside it.
Can I use it with yes or no questions or a 1 to 5 scale?
Yes. Choose the mix of binary, ordinal and continuous so the items are treated as ordered categories. For a 5-point scale with a fairly even spread of answers, treating items as numbers often gives similar results, but the ordered option is the safer choice, especially for short or lopsided scales.
What does the average variance extracted (AVE) mean, and why is it often below 0.50?
It is the average share of an item's variation that the factor explains. With loadings around 0.70 it is about 0.50. Since a loading of 0.70 is already strong, AVE near but under 0.50 is common and is a borderline result, not a failure. Reliability above 0.70 with AVE just below 0.50 is a typical pattern.
Do I need to run this before Cronbach's alpha?
No. Alpha is a lighter check of consistency. Confirmatory analysis tells you more, because it tests the structure and each item, and it handles several scales together.
How is this different from the structural equation model?
Confirmatory factor analysis is the measurement part only. A structural equation model adds arrows between the factors. Check the measurement part first, then build the full model.
Can I use survey weights?
Yes. With a survey weight set, the model uses the weights, the standard errors follow the survey design, and the weighted number of cases is shown.

Check your questionnaire structure

Open Structural Equation Modelling in AnalyZ Solutions. Free to sign up, free to analyze.

Get started
Related Guides