Choose the T-Test From the Data Structure
A t-test evaluates a difference in means relative to the uncertainty in that difference. The first decision is not the significance level or the Python function; it is whether the data represent one sample, two independent groups, or paired observations. That relationship comes from data collection, not from how two columns happen to appear in a table.
Read the t-statistic and p-value separately
- Null hypothesis (
H₀): the population mean or mean difference equals the value being tested, commonly zero. - Alternative hypothesis (
H₁): the population mean or mean difference is different from that value. - t-statistic: the observed difference measured in standard-error units. A larger absolute value is less compatible with the null hypothesis.
- p-value: assuming the null hypothesis and model assumptions are true, the probability of obtaining a result at least as extreme as the observed one.
If p < α, where α is chosen before looking at the result, reject H₀. This decision does not tell us whether the difference is large enough to matter in practice.
A complete result should therefore report the estimated mean difference and a confidence interval alongside the p-value. The interval shows both direction and plausible magnitude; the p-value alone answers neither.
Check the assumptions for the test you selected
The assumptions apply to the observations or differences being modeled:
- Numeric outcome: the measured variable should be quantitative.
- Independent observations: required between subjects; paired measurements are handled as within-subject differences.
- Approximate normality: the sample values for a one-sample test, each group for a small independent test, or the pairwise differences for a paired test should be approximately normal. A Q–Q plot can reveal severe skewness and outliers. A Shapiro–Wilk result should not be used as an automatic switch because large samples can detect departures too small to matter.
- Equal variances: required by the pooled independent t-test, but not by Welch’s t-test. When equal variance is not defensible, use Welch’s version.
The three tests answer different questions
There are three main types of T-tests depending on the structure of your data and what you want to compare:
One-sample t-test: compare one sample with a reference mean
-
Formula:
t = (x̄ - μ) / (s / √n)(x̄= sample mean,μ= reference mean,s= sample standard deviation,n= sample size) -
Example: Test whether the mean transaction-processing time from a sample differs from a
2-second reference value.
Independent t-test: compare two unrelated groups
-
Formula under equal variances:
t = (x̄₁ - x̄₂) / (sₚ × √(1/n₁ + 1/n₂))(sₚis the pooled standard deviation.) -
Example: Compare mean card spending between customers randomly assigned to a campaign and an independent control group. If assignment was not random, the test describes a difference but does not by itself establish that the campaign caused it.
Paired t-test: test the within-pair differences
-
Formula:
t = d̄ / (s_d / √n)(d̄= mean paired difference,s_d= standard deviation of the differences.) -
Example: Compare the monthly balance of the same customers before and after a product change. The test operates on each customer’s within-pair difference, not on two unrelated columns of balances.
Run all three tests with SciPy
The examples use reproducible synthetic data and match each data structure to its corresponding SciPy function.
import numpy as np
from scipy import stats
# Set random seed to generate reproducible dummy data
np.random.seed(42)
# ==========================================
# 1. One-Sample T-test
# ==========================================
print("--- 1. One-Sample T-test ---")
# Hypothesis: Is the sample mean significantly different from 170?
sample_heights = np.random.normal(loc=172, scale=5, size=30)
theoretical_mean = 170
t_stat, p_value = stats.ttest_1samp(sample_heights, theoretical_mean)
print(f"T-statistic: {t_stat:.4f}, p-value: {p_value:.4f}")
if p_value < 0.05:
print("=> Reject Null Hypothesis: The mean is significantly different from 170.\n")
else:
print("=> Fail to Reject Null Hypothesis: No significant difference.\n")
# ==========================================
# 2. Independent Two-Sample T-test
# ==========================================
print("--- 2. Independent Two-Sample T-test ---")
# Hypothesis: Are the means of Class A and Class B different?
class_a_heights = np.random.normal(loc=168, scale=5, size=30)
class_b_heights = np.random.normal(loc=175, scale=6, size=35)
# By default, ttest_ind assumes equal population variances.
# Because standard deviations are 5 and 6 (slightly different), we use Welch's t-test by setting equal_var=False
t_stat, p_value = stats.ttest_ind(class_a_heights, class_b_heights, equal_var=False)
print(f"T-statistic: {t_stat:.4f}, p-value: {p_value:.4f}")
if p_value < 0.05:
print("=> Reject Null Hypothesis: There is a significant difference between Class A and Class B.\n")
else:
print("=> Fail to Reject Null Hypothesis: No significant difference.\n")
# ==========================================
# 3. Paired Sample T-test
# ==========================================
print("--- 3. Paired Sample T-test ---")
# Hypothesis: Is there a significant difference in weight before and after the diet?
weight_before = np.random.normal(loc=85, scale=10, size=20)
# Assume the diet causes an average weight loss of 3kg per person
weight_after = weight_before - np.random.normal(loc=3, scale=2, size=20)
t_stat, p_value = stats.ttest_rel(weight_before, weight_after)
print(f"T-statistic: {t_stat:.4f}, p-value: {p_value:.4f}")
if p_value < 0.05:
print("=> Reject Null Hypothesis: The weight before and after the diet are significantly different.\n")
else:
print("=> Fail to Reject Null Hypothesis: No significant difference.\n")
Choose from the relationship between observations
- Use a one-sample t-test for one group against a reference value.
- Use an independent t-test, preferably Welch’s when equal variance is uncertain, for two unrelated groups.
- Use a paired t-test when the same subjects are measured twice or observations are explicitly matched.
The pairing or independence comes from how the data were collected. It cannot be inferred from the two columns after the fact.
Statistical significance is not business significance
With a large sample, a small and operationally irrelevant difference can produce a small p-value. With a small sample, a meaningful difference may remain uncertain. Report the sample sizes, mean difference, confidence interval, and an effect size such as Cohen’s d when its assumptions are appropriate.
If many segments, products, or time windows are tested, the chance of at least one false positive increases. Those comparisons should be planned in advance or handled with a multiple-testing procedure rather than selecting only the smallest p-value afterward.
The useful sequence is: define the comparison, identify whether observations are independent or paired, choose the test, inspect the effect and uncertainty, and only then interpret the p-value.