# Statistical Significance Testing for Survey Results: Which Test, Worked by Hand, and the Traps

Significance testing for survey data: which test to use, a worked two-proportion z-test and chi-square crosstab, multiple comparisons, weights and sample size.

Published 2026-08-25 · Updated 2026-10-12
Canonical: https://hercules.works/blogs/statistical-significance-testing-india

A significance test asks how surprising a difference would be if there were no real difference in the population. For two percentages from separate groups, use a two-proportion z-test; for a crosstab, a chi-square test; for averages, a t-test or ANOVA. A p-value below 0.05 is not proof and says nothing about size or importance. Weighting, clustering and running many tests all make real-looking differences easier to find by chance, so adjust for them. Hercules Works lists significance tests among the outputs of its analysis.

## Is 46% Really Bigger Than 39%?

Two concepts are tested with 400 people each. Concept A scores 46% top-two-box purchase intent and Concept B 39%. The marketing team wants to launch A. Before anyone does, someone should ask whether a 7-point gap could appear by chance between two groups of 400 drawn from the same population.

That is the job of a significance test. The worked examples below can be done with a calculator. Margins of error on a single percentage are covered elsewhere: see the per-cell table in our [Gabor-Granger guide](/blogs/gabor-granger-pricing-survey/) and the cell-size discussion in [pricing research](/blogs/pricing-research-platform-india/). This page is about comparing.

## What a p-Value Does and Does Not Say

The American Statistical Association's 2016 statement on p-values sets out six principles ([ASA, 7 March 2016](https://www.amstat.org/asa/files/pdfs/p-valuestatement.pdf)). Three matter most for survey readers:

- "P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone."
- "Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold."
- "A p-value, or statistical significance, does not measure the size of an effect or the importance of a result."

In practice: report the difference and its interval, then the p-value. A 2-point gap can be significant with a huge sample and still not matter. A 9-point gap can miss significance with small groups and still be worth a follow-up study.

## Choosing the Test

| You are comparing | Groups | Test |
|---|---|---|
| Two percentages | Two separate groups of people | Two-proportion z-test (or chi-square on the 2 × 2 table) |
| A categorical answer across groups | Two or more groups | Chi-square test of independence |
| Two percentages | The same people, two items (for example, liking concept A and concept B) | McNemar's test |
| Averages | Two separate groups | Two-sample t-test |
| Averages | Three or more groups | ANOVA, then pairwise tests with a correction |
| A small 2 × 2 table | Any | Fisher's exact test |

Watch the same-people row. When every respondent rates both concepts, the two percentages are not independent, and a test for separate groups gives the wrong answer.

## Worked Example 1: Two-Proportion z-Test

The figures are illustrative. Concept A: 184 of 400 (46%). Concept B: 156 of 400 (39%). Each respondent saw one concept. The NIST/SEMATECH e-Handbook gives the pooled test statistic ([NIST](https://www.itl.nist.gov/div898/handbook/prc/section3/prc33.htm)):

| Step | Calculation | Result |
|---|---|---|
| Pooled share | (184 + 156) ÷ 800 | 0.425 |
| Standard error | √[0.425 × 0.575 × (1/400 + 1/400)] | 0.0350 |
| z | (0.46 − 0.39) ÷ 0.0350 | 2.00 |
| Two-sided p | from the normal distribution | about 0.045 |

So the gap is significant at the 5% level, just. Now suppose the data were weighted and the weights carry a design effect of 1.3. Multiply the standard error by √1.3: it becomes 0.0399, z falls to 1.76, and p rises to about 0.08. The same 7 points are no longer significant. Check whether your crosstab tool allows for weights when it tests. Our [sampling guide](/blogs/sampling-methods-india/) explains where design effects come from.

## Worked Example 2: Chi-Square on a Small Crosstab

The figures are illustrative. Three hundred buyers say which pack size they would choose, split by city tier.

| | Small | Medium | Large | Total |
|---|---|---|---|---|
| Tier 1 | 30 | 70 | 50 | 150 |
| Tier 2 and 3 | 50 | 65 | 35 | 150 |
| Total | 80 | 135 | 85 | 300 |

Expected count in each cell = row total × column total ÷ 300. Because both rows have 150, the expected counts are 40, 67.5 and 42.5 in each row. Sum (observed − expected)² ÷ expected over all six cells:

| Cell | Small | Medium | Large |
|---|---|---|---|
| Tier 1 | (30 − 40)² ÷ 40 = 2.50 | 0.09 | 1.32 |
| Tier 2 and 3 | 2.50 | 0.09 | 1.32 |

Chi-square = 7.83 with (2 − 1) × (3 − 1) = 2 degrees of freedom, giving p ≈ 0.02. Pack preference differs by tier. The test does not say where; the cell contributions do. Small packs supply 5.0 of the 7.83: Tier 2 and 3 buyers lean to small packs, and Tier 1 buyers to large.

Small cells weaken the test. SciPy's documentation cites "an often quoted guideline" that it be used only if observed and expected frequencies in each cell are at least 5 ([SciPy](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.chi2_contingency.html)). For a sparse 2 × 2 table, use Fisher's exact test, or merge categories before testing.

## Many Tests, Many False Alarms

A banner of crosstabs with significance letters can run hundreds of tests. At a 5% level, each test with no real difference behind it still has a 1 in 20 chance of flagging one. Across 20 such tests, the chance of at least one false flag is 1 − 0.95²⁰, about 64%.

The Bonferroni method controls this by testing each of g comparisons at α ÷ g; NIST describes building each interval with confidence coefficient 1 − α/g ([NIST](https://www.itl.nist.gov/div898/handbook/prc/section4/prc473.htm)). For 20 comparisons at 5%, each needs p below 0.0025. It is strict, so use it for the comparisons you did not plan, and name the few you did plan before fieldwork.

## How Many Respondents a Comparison Needs

Size the sample from the smallest difference worth acting on. With the usual approximation, the number per group to detect a change from p₁ to p₂ at 5% significance with 80% power is (1.96 + 0.84)² × [p₁(1 − p₁) + p₂(1 − p₂)] ÷ (p₁ − p₂)².

| Difference to detect | Per group |
|---|---|
| 40% vs 50% | about 385 |
| 40% vs 45% | about 1,530 |

Halving the difference you want to see roughly quadruples the sample. If the budget allows 400 per cell, say plainly that gaps under about 10 points may be missed.

## A Caveat for Online Panel Data

These tests assume each group behaves like a random sample from its population. AAPOR's task force on non-probability sampling agrees that the margin of sampling error is "not appropriate for non-probability samples", and suggests that model-based measures of precision use different terms ([AAPOR, 2013](https://aapor.org/wp-content/uploads/2022/11/NPS_TF_Report_Final_7_revised_FNL_6_22_13-2.pdf)).

Tests remain useful for comparing randomly assigned cells inside one panel study, as in Example 1, because random assignment makes the cells comparable with each other. Be more careful when the claim is about the population: a significant difference between Tier 1 and Tier 3 panel members tells you about connected consumers in those tiers.

## Running This Study on Hercules

Our [panel methodology](/panel-methodology) page says the analysis produces cross-tabs, significance tests and coded open-ended answers. It does not say which tests or corrections are applied, so ask before relying on the letters in a table, especially for weighted data. Clients receive individual answers without identity data, so you can rerun any test above yourself.

The two-concept test is 800 completes, 400 per concept: 800 × ₹120 = ₹96,000 on a Pro bundle ([pricing](/pricing)). Detecting the 5-point gap in the sizing table would take about 3,060 completes, so settle which gap matters before you buy. With the Free plan's 30 responses, check that each respondent is routed to one concept only.

## Frequently asked questions

### What does statistically significant mean in a survey?

That a difference this large would be unlikely if there were no real difference in the population, usually judged at the 5% level. It does not mean the difference is large, important or certain.

### Which test compares two percentages?

For separate groups of people, a two-proportion z-test, which gives the same result as a chi-square test on the 2 × 2 table. If the same people answered both items, use McNemar's test.

### When should I use a chi-square test?

When you compare how a categorical answer, such as brand or pack size, is distributed across two or more groups. Watch for cells with small expected counts.

### Do I need to adjust significance tests for weighting?

Yes. Weights reduce the effective sample size. Dividing by the design effect, or multiplying the standard error by its square root, gives an honest test; ignoring it makes more differences look significant.

### What is the Bonferroni correction?

A way to keep the overall false-alarm rate at α when you make g comparisons, by testing each at α ÷ g. With 20 comparisons at 5%, each needs p below 0.0025.

### Is p < 0.05 enough to make a decision?

No. The American Statistical Association says decisions should not rest only on whether a p-value passes a threshold. Look at the size of the difference, its interval, the sample and the cost of being wrong.
