Equivalence Testing¶
Classical hypothesis tests ask "are these two groups different?" Equivalence testing flips the question: are these two groups similar enough to be considered practically the same?
This is critical in manufacturing (batch-to-batch consistency), bioequivalence studies (generic vs. brand-name drugs), and any domain where you need to demonstrate that a change or substitution has no meaningful effect on the functional response.
The TOST framework¶
The functional equivalence test in fdars implements a Two One-Sided Tests (TOST) procedure adapted for functional data:
- Define an equivalence margin \(\delta > 0\).
- Test \(H_0^-: \|\mu_1 - \mu_2\|_\infty \ge \delta\) against \(H_1^-: \|\mu_1 - \mu_2\|_\infty < \delta\).
- If \(H_0^-\) is rejected at level \(\alpha\), the two groups are declared equivalent within margin \(\delta\).
Equivalently, the test constructs a simultaneous confidence band (SCB) for the mean difference \(\mu_1 - \mu_2\) and declares equivalence when the entire band sits inside the corridor \([-\delta, \delta]\). The null distribution of the test statistic is estimated via a Gaussian multiplier bootstrap.
Equivalence is concluded when the whole simultaneous confidence band lies inside the corridor, i.e.
where \(c_\alpha\) is the \((1-\alpha)\) quantile of the bootstrap null distribution and \(\hat\sigma(t)\) the pointwise standard error; equivalently \(T < \delta - c_\alpha\hat\sigma\).
Returned fields
equivalence_test returns equivalent (bool), p_value, and test_statistic
(the observed sup-norm \(T\)). The R reference additionally prints a critical value
and the SCB range in its summary; the Python binding exposes the decision, the
p-value, and the statistic. Compare \(T\) against \(\delta\) directly, or sweep \(\delta\)
(below) to locate the decision threshold, which is where \(c_\alpha\) effectively sits.
Visually, equivalence holds when the second group's mean stays inside the \(\pm\delta\) corridor drawn around the first group's mean. The left panel shows two groups that remain within the margin; the right panel shows a shifted group that escapes it.
In the left panel mean B stays comfortably inside the shaded \(\pm\delta\) corridor around mean A, so the sup-norm \(T\) is small and equivalence is plausible; in the right panel the 5-unit shift pushes mean B far outside the corridor, driving \(T\) well past \(\delta\) and ruling equivalence out.
Usage¶
import numpy as np
from fdars import Fdata
from fdars.simulation import simulate
from fdars.tolerance import equivalence_test
argvals = np.linspace(0, 1, 100)
# Two groups with very similar means
fd_a = Fdata(simulate(50, argvals, n_basis=5, seed=1), argvals=argvals)
fd_b = Fdata(simulate(50, argvals, n_basis=5, seed=2) + 0.2, argvals=argvals) # small offset
result = equivalence_test(
data1=fd_a.data,
data2=fd_b.data,
delta=1.0, # equivalence margin
alpha=0.05, # significance level
nb=1000, # bootstrap replicates
seed=42,
)
Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
data1 |
ndarray (n1, m) |
-- | First group of functional observations |
data2 |
ndarray (n2, m) |
-- | Second group of functional observations |
delta |
float |
-- | Equivalence margin (\(\delta > 0\)) |
alpha |
float |
0.05 |
Significance level |
nb |
int |
1000 |
Number of bootstrap replicates |
seed |
int |
42 |
Random seed |
Returns a dictionary:
| Key | Type | Description |
|---|---|---|
equivalent |
bool |
True if equivalence is established at level \(\alpha\) |
p_value |
float |
Bootstrap p-value |
test_statistic |
float |
Observed sup-norm of the mean difference |
print(f"Equivalent: {result['equivalent']}")
print(f"p-value: {result['p_value']:.4f}")
print(f"Sup-norm: {result['test_statistic']:.4f}")
Choosing the margin \(\delta\)¶
The margin \(\delta\) is the maximum allowable pointwise difference between the two mean functions. It should be set before looking at the data, based on domain knowledge:
Do not choose \(\delta\) from the data
Setting \(\delta\) to be just larger than the observed difference inflates the Type I error. Always specify \(\delta\) based on what constitutes a practically meaningful difference in your application.
| Domain | Typical \(\delta\) guidance |
|---|---|
| Manufacturing | Specification tolerance / 2 |
| Bioequivalence | 20 % of the reference mean (FDA guidance) |
| Environmental monitoring | Regulatory action threshold |
Example -- equivalent vs. non-equivalent groups¶
import numpy as np
from fdars import Fdata
from fdars.simulation import simulate
from fdars.tolerance import equivalence_test
argvals = np.linspace(0, 1, 100)
delta = 2.0
# ── Case 1: Similar groups (should be equivalent) ────────────
fd_a = Fdata(np.asarray(simulate(40, argvals, n_basis=5, seed=10)), argvals=argvals)
fd_b = Fdata(np.asarray(simulate(40, argvals, n_basis=5, seed=20)) + 0.1, argvals=argvals)
r1 = equivalence_test(fd_a.data, fd_b.data, delta=delta, alpha=0.05, nb=2000, seed=42)
print(f"Case 1 — Equivalent: {r1['equivalent']} p={r1['p_value']:.4f}")
# ── Case 2: Different groups (should NOT be equivalent) ──────
fd_c = Fdata(np.asarray(simulate(40, argvals, n_basis=5, seed=10)), argvals=argvals)
fd_d = Fdata(np.asarray(simulate(40, argvals, n_basis=5, seed=20)) + 5.0, argvals=argvals) # large shift
r2 = equivalence_test(fd_c.data, fd_d.data, delta=delta, alpha=0.05, nb=2000, seed=42)
print(f"Case 2 — Equivalent: {r2['equivalent']} p={r2['p_value']:.4f}")
Case 1 — Equivalent: True p=0.0345 Case 2 — Equivalent: False p=1.0000
Case 1's near-identical means sit well inside the margin and are declared equivalent with a small p-value, whereas Case 2's 5-unit shift blows past \(\delta = 2\) and is correctly declared not equivalent -- the test cleanly separates a practically-irrelevant offset from a meaningful one.
Choosing \(\delta\) relative to sampling uncertainty
Equivalence requires the entire \((1-\alpha)\) simultaneous confidence band for
\(\mu_1 - \mu_2\) to sit inside the \(\pm\delta\) corridor — and that band has a
half-width of roughly \(c_\alpha \cdot \mathrm{SE}\), driven by the sample size,
not by the raw mean difference. So two samples from the same distribution
are not automatically equivalent: if \(\delta\) is smaller than the band
half-width, the verdict is (correctly) False. Here the groups differ by only
\(0.1\), yet \(\delta\) must clear the band's reach — delta = 1.0 returns False,
while delta = 2.0 returns True. Always choose \(\delta\) from a
practically-meaningful tolerance and sanity-check it against the band width.
Sensitivity to \(\delta\)¶
Because \(\delta\) is a modelling choice, it is worth sweeping over a range of margins to locate the decision threshold -- the value of \(\delta\) at which the verdict flips from "not equivalent" to "equivalent". Plotting the decision as a step function makes the threshold obvious.
import numpy as np
from docs_fig import fig, render
from fdars.simulation import simulate
from fdars.tolerance import equivalence_test
rng = np.random.default_rng(1)
t = np.linspace(0, 1, 80)
X1 = np.array([np.sin(2 * np.pi * t) + rng.normal(0, 0.3, t.size) for _ in range(30)])
X2 = np.array([np.sin(2 * np.pi * t) + 0.3 + rng.normal(0, 0.3, t.size) for _ in range(25)])
deltas = np.arange(0.1, 1.01, 0.05)
decided = np.array([
equivalence_test(X1, X2, delta=float(d), nb=400, seed=42)["equivalent"]
for d in deltas
])
f, ax = fig()
ax.step(deltas, decided.astype(int), where="post", color="#3f51b5", lw=1.6)
ax.scatter(deltas[decided], np.ones(decided.sum()), color="#198754", zorder=3,
label="equivalent")
ax.scatter(deltas[~decided], np.zeros((~decided).sum()), color="#dc3545", zorder=3,
label="not equivalent")
ax.set(title="Decision as a function of the equivalence margin δ",
xlabel="δ", ylabel="equivalence declared", yticks=[0, 1],
yticklabels=["No", "Yes"])
ax.legend(loc="center right")
print(render(f))
The step from "No" to "Yes" marks the smallest margin under which these two groups are declared equivalent -- a compact summary of how much difference the data can tolerate.
How the verdict tracks the true shift¶
Fixing \(\delta\) and instead sweeping the actual vertical offset between the two groups shows the test doing its job: the observed sup-norm statistic \(T\) should rise roughly linearly with the shift, and equivalence should hold only while \(T\) stays below the margin.
import numpy as np
from docs_fig import fig, render
from fdars.simulation import simulate
from fdars.tolerance import equivalence_test
t = np.linspace(0, 1, 80)
delta = 1.0
base = np.asarray(simulate(40, t, n_basis=5, seed=10))
other0 = np.asarray(simulate(40, t, n_basis=5, seed=20))
shifts = np.linspace(0.0, 2.5, 14)
stats, decided = [], []
for s in shifts:
r = equivalence_test(base, other0 + s, delta=delta, nb=400, seed=42)
stats.append(r["test_statistic"])
decided.append(r["equivalent"])
stats = np.asarray(stats); decided = np.asarray(decided)
f, ax = fig(figsize=(7.4, 3.8))
ax.plot(shifts, stats, "-o", color="#3f51b5", lw=1.6, ms=4, label="sup-norm T")
ax.axhline(delta, color="#dc3545", ls="--", lw=1.4, label=f"margin δ = {delta}")
ax.scatter(shifts[decided], stats[decided], color="#198754", zorder=4, s=55,
label="declared equivalent")
ax.set(title="Test statistic vs. true group shift", xlabel="applied shift", ylabel="T")
ax.legend(loc="upper left")
print(render(f))
The statistic climbs almost linearly with the imposed shift, and the green markers -- the runs declared equivalent -- sit exactly where \(T\) dips below the dashed margin, confirming the decision rule behaves monotonically in the underlying difference.
The band that drives the decision¶
Equivalence is ultimately a statement about the simultaneous confidence band for the mean
difference \(\mu_1 - \mu_2\): the verdict is True only when that entire band nests inside
the \(\pm\delta\) corridor. Plotting the mean difference with a bootstrap-style band against
the corridor makes the geometry of the decision explicit.
import numpy as np
from docs_fig import fig, render
from fdars.simulation import simulate
t = np.linspace(0, 1, 100)
delta = 1.0
rng = np.random.default_rng(3)
A = np.asarray(simulate(40, t, n_basis=5, seed=10))
B = np.asarray(simulate(40, t, n_basis=5, seed=20)) + 0.35
diff = A.mean(0) - B.mean(0)
se = np.sqrt(A.var(0, ddof=1) / A.shape[0] + B.var(0, ddof=1) / B.shape[0])
band = 2.5 * se # simultaneous-style half-width
f, ax = fig(figsize=(7.4, 3.8))
ax.axhspan(-delta, delta, color="#198754", alpha=0.12, label=f"±δ corridor ({delta})")
ax.fill_between(t, diff - band, diff + band, color="#3f51b5", alpha=0.25,
label="confidence band")
ax.plot(t, diff, color="#3f51b5", lw=2.0, label="mean difference")
ax.axhline(0, color="0.5", lw=0.8)
ax.set(title="Mean difference and its band vs. the ±δ corridor", xlabel="t",
ylabel="μ₁(t) − μ₂(t)")
ax.legend(loc="upper right", fontsize=8)
print(render(f))
Here the shaded band stays inside the green corridor across the whole domain, so the sup-norm never reaches \(\delta\) and equivalence is declared; a band that poked above \(+\delta\) or below \(-\delta\) at any \(t\) would flip the verdict to non-equivalent.
One-sample test¶
equivalence_test_one_sample tests whether a single sample's mean is equivalent to a
known reference function \(\mu_0\) -- for example, checking that a new production run
matches a fixed specification curve. The hypotheses and TOST machinery are identical;
only the second group is replaced by the fixed target.
import numpy as np
from fdars.tolerance import equivalence_test_one_sample
rng = np.random.default_rng(42)
t = np.linspace(0, 1, 80)
X = np.array([np.sin(2 * np.pi * t) + rng.normal(0, 0.3, t.size) for _ in range(30)])
mu0 = np.sin(2 * np.pi * t) # reference / specification curve
res = equivalence_test_one_sample(X, mu0, delta=0.5, alpha=0.05, nb=1000, seed=42)
print(f"Equivalent to reference: {res['equivalent']}")
print(f"Sup-norm |mean - mu0|: {res['test_statistic']:.4f}")
print(f"p-value: {res['p_value']:.4f}")
Equivalent to reference: True Sup-norm |mean - mu0|: 0.1258 p-value: 0.0000
Parameters (equivalence_test_one_sample)
| Parameter | Type | Default | Description |
|---|---|---|---|
data |
ndarray (n, m) |
-- | Sample of functional observations |
mu0 |
ndarray (m,) |
-- | Reference / target mean function |
delta |
float |
-- | Equivalence margin (\(\delta > 0\)) |
alpha |
float |
0.05 |
Significance level |
nb |
int |
1000 |
Bootstrap replicates |
seed |
int |
42 |
Random seed |
Returns the same equivalent / p_value / test_statistic dictionary as the
two-sample test.
Visually, the one-sample test asks whether the sample mean stays within the \(\pm\delta\) corridor drawn around the fixed reference curve \(\mu_0\). The panel below overlays the sample mean on the specification and shades the corridor, so the sup-norm statistic is simply the largest vertical gap between the two lines.
import numpy as np
from docs_fig import fig, render
from fdars.tolerance import equivalence_test_one_sample
t = np.linspace(0, 1, 80)
delta = 0.5
rng = np.random.default_rng(42)
X = np.array([np.sin(2 * np.pi * t) + rng.normal(0, 0.3, t.size) for _ in range(30)])
mu0 = np.sin(2 * np.pi * t)
res = equivalence_test_one_sample(X, mu0, delta=delta, alpha=0.05, nb=1000, seed=42)
xbar = X.mean(0)
gap = np.abs(xbar - mu0)
i_max = int(gap.argmax())
f, ax = fig(figsize=(7.4, 3.8))
ax.fill_between(t, mu0 - delta, mu0 + delta, color="#198754", alpha=0.12,
label=f"μ₀ ± δ ({delta})")
ax.plot(t, mu0, color="#198754", lw=2.0, label="reference μ₀")
ax.plot(t, xbar, color="#e8710a", lw=2.0, label="sample mean")
ax.vlines(t[i_max], min(xbar[i_max], mu0[i_max]), max(xbar[i_max], mu0[i_max]),
color="#dc3545", lw=2.2, label=f"sup-norm T = {res['test_statistic']:.2f}")
ax.set(title=f"One-sample equivalence to μ₀ (equivalent={res['equivalent']})",
xlabel="t", ylabel="X(t)")
ax.legend(loc="upper right", fontsize=8)
print(render(f))
The sample mean hugs the reference and the red segment -- the point of maximum deviation, i.e. the statistic \(T\) -- stays well inside the corridor, so the run is declared equivalent to specification.
See also¶
- Tolerance bands -- the confidence and tolerance bands that underlie the SCB used here.
References¶
- Dette, H., Kokot, K. (2021). Detecting relevant differences in the covariance operators of functional time series. Biometrika, 108(4), 895–913.
- Wellek, S. (2010). Testing Statistical Hypotheses of Equivalence and Noninferiority, 2nd ed. Chapman & Hall/CRC.
- Degras, D.A. (2011). Simultaneous confidence bands for nonparametric regression with functional data. Statistica Sinica, 21(4), 1735–1765.