Cross-Tabulation Survey Analysis: How to Read a Crosstab Without Fooling Yourself

Tabulation vocabulary is consistent across most research firms. Greenbook's "Anatomy of a Crosstab" says the banner (or header) "consists of the subgroups for data analysis" and that the stub "contains the table rows", usually the answer options plus summary rows, nets and statistics (Greenbook).

Keep the banner short. Every extra column adds comparisons, and every comparison adds a chance of a false alarm.

The two differ only in the denominator. A column percentage divides each cell by its column total. A row percentage divides each cell by its row total. Sawtooth Software defines both this way in its crosstab help (Sawtooth).

Use column percentages when the banner holds the groups you want to compare: "Of people aged 18-24, how many said yes?" Use row percentages when you want to profile an answer: "Of those who said yes, how many are aged 18-24?" The University of Oregon's banner-table guide says statements "should be phrased in terms of the denominator", for example "of all 280 people who said..." (University of Oregon). Its tables put the answer categories in the columns and the groups in the rows, the reverse of the layout here, so what matters is the denominator, not the direction.

Check what your tool shows by default. Sawtooth shows counts and column percentages first, and other tools may default to row percentages, so the same table can be read two ways without anyone noticing.

Multi-select questions need extra care. One person can appear in several rows, so rows do not add to the column total, and a chi-square test (below) is not valid on such a table without adjustment.

The counts below are invented to show the arithmetic. They are not survey results. Imagine 350 category buyers asked: "Would you buy this new pack?"

Column percentages for "Yes" are 45.0% (18-24), 38.3% (25-34) and 18.2% (35-44). The row percentage for "Yes" in the 18-24 column is also 45.0% here, but only because the row and column totals happen to match at 120. In the 35-44 column the row percentage is 16.7%, so the two readings answer different questions.

The younger groups look more positive. Is that real or noise?

The chi-square test of independence asks whether the answer and the group are unrelated. For each cell, the expected count under independence is row total × column total ÷ grand total. The statistic is the sum of (observed − expected)² ÷ expected over all cells, with (rows − 1) × (columns − 1) degrees of freedom (Wikipedia, chi-squared test).

For the "Yes" row in our example, the expected counts are 120 × 120 ÷ 350 = 41.14, 41.14 and 37.71. Summing all nine cells gives a chi-square of 26.26 with (3 − 1) × (3 − 1) = 4 degrees of freedom. The upper-tail 5% critical value for 4 degrees of freedom is 9.488 (NIST critical values table). Because 26.26 is well above 9.488, the data are inconsistent with age and answer being unrelated.

That is all the test says. It does not say which age groups differ, nor by how much, and a large sample can make a trivial gap significant.

To compare one answer between two columns, use a two-proportion z-test. Pool the two groups: p = (x₁ + x₂) ÷ (n₁ + n₂). The standard error is √[p(1 − p)(1/n₁ + 1/n₂)] and z is the difference in proportions divided by it (Wikipedia, two-proportion z-test). Tabulation software prints the result as letters under the cells.

At the usual 5% level a two-sided test needs |z| above 1.96. The first two pass and the third does not. We ran three comparisons, though, and the chance of at least one false positive across m independent tests is 1 − (1 − α)^m, which is 1 − 0.95³ = 14.3% here (Wikipedia, multiple comparisons). The Bonferroni correction tests each comparison at α ÷ m, so 0.05 ÷ 3 = 0.0167, which needs |z| above about 2.39. Our first two still pass.

In a big table, treat isolated "significant" letters as leads to check, not findings.

No single number is right for every table, and we have not found an authoritative universal rule. The published guidance is about expected and observed counts:

• Chi-square: the test can mislead when expected frequencies are very small, and a minimum expected count of at least 5 is often recommended (Wikipedia, chi-squared test, above). Our smallest expected count was 34.57.
• Two-proportion z-test: Wikipedia states the conditions as n₁p̂₁, n₁(1 − p̂₁), n₂p̂₂ and n₂(1 − p̂₂) all above 10, and recommends exact tests such as Fisher's when counts fall below 5.
• Percentages on a small base: a percentage built on a handful of people moves a lot with one answer. Decide before analysis the smallest base you will report, and flag anything below it.

Plan for this at design stage: quotas must be large enough for the cell, not just the total. Our sample size guide shows how margins shrink with base size.

1. Find the base under each column and note any that are small.
2. Confirm whether the percentages are column or row, and say the base aloud in each sentence.
3. Run the overall test (chi-square) before hunting for pairs.
4. Test only the comparisons you chose before looking, and correct for how many you ran.
5. Report the size of the gap, not only the letters. A significant 3-point gap may not change a decision.

If you want crosstabs without building them by hand: Poseidon, the Hercules analytics engine, produces cross-tabs, significance tests and coded open-ended answers from the cleaned responses (panel methodology). Whichever tool you use, apply the checks above to what it prints. The Free plan includes 30 responses, enough to test a questionnaire and see the table layout before you plan banner columns.