What the Hypothesis Test Selector does
This decision guide suggests a suitable statistical test from four facts about your study: what you want to find out, the type of outcome, how many groups there are and whether the same people appear in more than one. It names the test, gives sensible alternatives, lists the assumptions you must check and shows the decision path, so you can explain the choice in a methods section.
The rules are the standard textbook mapping taught in introductory statistics - t-tests, ANOVA, Mann-Whitney, Wilcoxon, Kruskal-Wallis, Friedman, chi-square, Fisher, McNemar, correlation and regression. They are written out openly rather than hidden behind a score, and they run in your browser.
How to use it
- Say what you want to do: compare groups, measure a relationship, or compare one sample with a known value.
- Choose the outcome type. A test score is continuous; a 1-5 agreement rating is ordinal; passed or failed is binary; favourite brand is nominal.
- Describe the design: how many groups and whether they are independent people or the same people measured more than once (before and after, three conditions per person).
- Answer the assumption questions honestly. If you are not sure the data are roughly normal, say so - the guide then prefers a rank-based test and tells you how to check.
- Read the suggested test, the alternatives and the assumption checklist, then copy the summary into your analysis plan.
Reading the results
The suggestion is a standard, defensible default for the design you described, not the only valid choice. Where statisticians commonly prefer a variant - Welch's t-test over Student's, for example - the guide says so and why.
The assumption checklist is the part that matters most after the choice. A test used when its assumptions fail can give misleading p-values, so check each item with a plot or a diagnostic before relying on the result.
The decision path shows every answer that led to the test, which is useful when a supervisor or reviewer asks why you chose it.
Worked example: three teaching methods with skewed scores
A teacher compares reading-test scores for three independent classes, each taught with a different method. The scores are continuous, but one class has a long tail of very low scores, so she answers "No - skewed or outliers" to the normality question.
The path is: compare groups, continuous outcome, three or more groups, independent groups, not normal. The guide suggests the Kruskal-Wallis H test, with Dunn's post-hoc tests and a correction for multiple comparisons as the follow-up, and lists the assumptions: independent observations, and similar distribution shapes if she wants to talk about medians.
Had the scores been roughly normal with similar spreads, the same answers would have led to one-way ANOVA with Tukey post-hoc tests; with clearly different spreads, to Welch's ANOVA.
Parametric or non-parametric?
Parametric tests such as t-tests and ANOVA assume roughly normal data (or residuals) and use the actual values, which makes them more powerful when the assumption holds. Rank-based tests such as Mann-Whitney and Kruskal-Wallis make weaker assumptions and cope with skew and outliers, at some cost in power.
With larger samples - roughly 30 or more per group - the central limit theorem makes t-tests fairly robust to non-normality, which is why the question mentions sample size. Heavy outliers are still a problem at any size.
Limitations: what the result does not prove
- It covers common single-outcome designs. Mixed designs, multilevel or clustered data, survival times, time series, equivalence testing and Bayesian analyses need specialist advice.
- The guide cannot see your data. Whether an assumption holds is your judgement, based on plots and diagnostics.
- Choosing a test after looking at several results and picking the one that is significant invalidates the p-value. Decide in advance.
- It does not calculate the test. Use statistical software, or the A/B test significance calculator for two conversion rates.
Privacy: where your data goes
Everything you paste, type or drop is processed in this browser tab. It is not uploaded, logged, stored or sent to analytics. Session recording and tag-manager scripts are switched off on this page.
Standards and sources
- NIST/SEMATECH e-Handbook of Statistical Methods - checked 19 Sep 2026
- NIST/SEMATECH e-Handbook of Statistical Methods - product and process comparisons
- NIST/SEMATECH e-Handbook - comparisons based on data from two processes
Frequently asked questions
What is the difference between paired and independent samples?
Paired or repeated measures means each observation in one group is linked to one in another - the same person before and after, or matched twins. Independent means different, unrelated people in each group. Using the wrong one gives wrong standard errors.
Why does it suggest Welch's t-test instead of Student's t-test?
Welch's version does not assume equal variances. When variances are equal it performs almost identically, and when they differ it keeps the false-positive rate correct, so many statisticians now use it by default.
Can I use a t-test on Likert-scale data?
For a single Likert item, a rank-based test such as Mann-Whitney is the safer default because the gaps between points are not guaranteed equal. Scores summed across many items behave more like continuous data and are often analysed with t-tests or ANOVA.
When should I use Fisher's exact test instead of chi-square?
Use Fisher's exact test when any expected cell count is below 5, which usually happens with small samples. Chi-square relies on a large-sample approximation that breaks down in that situation.
What should I do after a significant ANOVA?
ANOVA only says that some means differ. Use post-hoc tests - Tukey HSD for equal variances, Games-Howell otherwise - or planned contrasts you specified in advance, to find which groups differ, and report effect sizes.
How do I check normality?
Look at a histogram and a Q-Q plot of each group or of the residuals. Formal tests such as Shapiro-Wilk flag trivial departures in large samples and miss real ones in small samples, so plots are usually more informative.
Last reviewed by the A2Z.Tools team against the sources listed above.
