RESEARCH STATISTICS GUIDE

How to Choose the Right Statistical Test for Your Study

Choosing a statistical test should start with the scientific question and study design—not with a normality test or a list of test names. A good analysis plan identifies the effect you want to estimate, the independent experimental unit, the outcome type, and any pairing, repeated measurements or clustering before selecting a model.

Start with the question, not the test

Researchers often approach statistical planning by asking whether they should use a t test, ANOVA, Mann–Whitney test or mixed model. That reverses the ideal order. The first question is what effect the study is designed to estimate.

Examples include a difference between treatment groups at one prespecified follow-up, a difference in change over time, an overall longitudinal trajectory, an association between two variables, an adjusted association, a risk or rate contrast, or a time-to-event effect. The same dataset can support more than one scientifically sensible question, but the primary analysis should be aligned with the primary question that was defined in advance.

Identify the experimental unit

The experimental unit is the independent unit that was assigned to a condition or represents an independent biological replicate. This is important because the number of measurements is not always the number of independent observations.

For example, several assay wells measured from the same donor sample are technical replicates rather than independent biological replicates. Likewise, animals allocated to treatment by cage may not be independent treatment units if the cage is the unit of allocation. Treating dependent measurements as independent can inflate the apparent sample size and produce overly precise inference.

Ask whether observations are independent, paired, repeated or clustered

Dependence is one of the main reasons a simple test can be inappropriate. Pre/post values from the same participant are paired. Multiple follow-up measurements from one participant are repeated. Patients within a clinic, pupils within a school, or animals within a cage can be clustered. Matched case-control studies create an explicit matching structure.

The analysis should preserve that structure. A paired t test can be appropriate for a simple two-occasion continuous pre/post comparison. A longitudinal mixed-effects model may be more appropriate when there are several repeated measurements. Cluster-randomised studies need analyses that respect the randomisation cluster.

Match the model family to the outcome

The type of primary outcome constrains the reasonable model families. Continuous outcomes often use mean-based linear models. Binary outcomes commonly use binary-regression models. Counts and rates can require Poisson or negative-binomial models. Ordered outcomes can require ordinal methods, and time-to-event outcomes require survival-analysis methods.

This is why multiplicity or clustering should not replace the underlying outcome model. For example, a multi-arm trial with a binary outcome still needs a binary-outcome model before prespecified contrasts and multiplicity control are added.

Baseline adjustment and change scores answer related but different questions

For a randomised trial with a continuous baseline measurement and one primary follow-up, baseline-adjusted regression or ANCOVA is often an efficient way to estimate the between-group treatment effect at follow-up while accounting for baseline values. A change-score analysis can also be scientifically defensible in some settings, but it should not be confused with separate within-group pre/post tests.

Testing whether one group changes significantly and another group does not is not a valid test of whether the groups differ from each other. The between-group effect should be tested directly.

Do not let a normality test choose the analysis

A common workflow is to run a normality test and use its P value to decide between a “parametric” and “nonparametric” test. This is too simplistic.

For t tests, ANOVA, ANCOVA and linear regression, the relevant assumptions concern the model errors, variance structure, influential observations and—in regression models—the functional form of continuous predictors. The raw outcome itself does not need to look perfectly bell-shaped for every linear-model analysis to be valid.

A statistically significant Shapiro–Wilk test should therefore not automatically trigger a switch to Mann–Whitney, Wilcoxon or Kruskal–Wallis. Instead, fit the prespecified model, inspect the diagnostics that are relevant to that model, and judge whether any violation is material to the planned estimand and inference.

Plan first, then check the fitted model

Assumption checking is still important—it simply belongs in the right place. The primary analysis should be planned from the study design and scientific question. After the data have been collected and the model fitted, examine the diagnostics appropriate to that method.

For a t test or linear model, that can include influential observations, residual behavior and heteroscedasticity. For a mixed model, it also includes the covariance and random-effects structure. Logistic regression requires attention to sparse data, separation and functional form. Poisson models require dispersion and exposure checks. Cox regression requires assessment of proportional hazards. Bayesian models require computational diagnostics, posterior predictive checks and sensitivity to prior choices.

If assumptions are materially problematic, the response depends on the problem. Options can include robust or heteroscedastic inference, a scientifically interpretable transformation, a generalized model, an appropriate rank-based estimand, a different covariance structure or a sensitivity analysis. The response should be justified rather than triggered mechanically by a single diagnostic P value.

Think about multiplicity after defining the primary model

When several confirmatory claims are planned, multiplicity control should be layered onto the analysis rather than treated as a separate replacement for the model. A useful sequence is:

1. choose the outcome-appropriate primary model;
2. define the prespecified contrasts or hypotheses;
3. apply a multiplicity strategy appropriate to that family of claims.

The correct approach can differ for several treatments compared with a common control, all pairwise comparisons, ordered hypotheses, repeated confirmatory time points, multiple subgroups or high-dimensional features.

Report the effect, not only the P value

A statistical analysis should identify the primary effect estimate and its uncertainty. For a simple continuous comparison, that may be a mean difference with a confidence interval. For a binary outcome it may be a risk difference, risk ratio or odds ratio. For survival data it may be a hazard ratio or another prespecified survival estimand.

Where frequentist hypothesis testing is used, the exact P value can be reported alongside the effect estimate and confidence interval. For Bayesian analyses, report the posterior effect estimate and credible interval, together with prespecified posterior probabilities or decision thresholds where they are scientifically meaningful.

Know when specialist analysis is the right answer

Not every study should be forced into an automatic recommendation. Complex crossover designs, competing risks, high-dimensional data, complicated ordinal or clustered outcomes, adaptive trials and other specialised designs can require modelling choices that depend on study-specific details.

In those settings, identifying the reason specialist input is needed is itself a useful planning result. The goal is not to produce a test name at any cost; it is to produce a defensible analysis plan.

Further reading and related resources

• Blazon Scientific research resources
• Blazon Research Tools