lucid.rodeo · incerto · tails

The tail gauge

An instrument, not an oracle: paste a series, read a file locally, or draw from a seeded generator — then measure its tail three ways: the survival curve, the Hill plot, and whether the moments look finite. All of it computed here, in your browser.

Exact — instrument The instrument is exact — the standard estimators computed on the actual numbers — and it runs only in your browser: nothing you paste or load ever leaves your machine. And reading a tail is measurement, not a signal. the fence →
read locally by your browser (FileReader) — the file never leaves your machine
1 · the survival curve (CCDF), log-log
2 · the Hill plot — α̂ across a log-spaced sweep of k (10 → n/4)
k = — band: α̂ ± 1.96·α̂⁄√k (asymptotic — it ignores the bias from slowly-varying corrections)

α̂ = — (generate or load data)

Choosing k is judgement, not automation — the plot shows the whole sweep of k so the choice stays visible. A tail exponent that only exists at one k is not a tail exponent.

3 · do the moments exist? (max-over-sum)
p = 1 (mean) · —
p = 2 (variance) · —
p = 3 (skewness) · —
p = 4 (kurtosis) · —

Rₙ(p) = max|Xᵢ|ᵖ ⁄ Σ|Xᵢ|ᵖ — a suggestive diagnostic, not a hypothesis test. Near 0, the pth moment is plausibly finite; large, and that moment is in doubt. The law behind it: Rₙ(p) → 0 almost surely if and only if E|X|ᵖ < ∞, while a tail exponent below p leaves the ratio at a nondegenerate random limit — visibly alive. At the boundary α = p the ratio does drift to 0 in probability, but only on ln n time, so a middling reading at these sample sizes proves nothing either way.

n used
skipped
α̂ @ chosen k
chosen k
naive CCDF slope
Rₙ(1..4)
source
seed
CCDF axes
Hill axes
generate or load data

The estimator: with the sample sorted downward, the Hill estimate at k uses the top k+1 order statistics X₍₁₎ ≥ … ≥ X₍ₖ₊₁₎, summing over the top k against the threshold X₍ₖ₊₁₎: α̂ = k ⁄ Σᵢ₌₁..ₖ ln(X₍ᵢ₎ ⁄ X₍ₖ₊₁₎), with asymptotic standard error ≈ α̂/√k. For a Pareto with minimum xₘ the survival function is P(X > x) = (x/xₘ)⁻ᵅ for x ≥ xₘ — a straight line of slope −α on log-log axes (the generators here use inverse-transform sampling with xₘ = 1, guarding U away from 0). The dashed NAIVE line is deliberately the wrong tool, shown so you can see it be wrong: least-squares on a log-log CCDF is biased and its standard errors are invalid, because the points of an empirical survival curve are mechanically correlated — the honest route is maximum likelihood plus goodness-of-fit, which is the whole argument of Clauset–Shalizi–Newman.

What the gauge cannot say: it never certifies a power law. Every reading is a property of this finite sample — the band is asymptotic, k-sensitivity is information rather than noise, and a plateau you have to squint for is not a plateau. The lognormal is the standing trap: all its moments are finite, yet its Hill plot never flattens — it slides steadily toward 0 as k grows, and at practical sample sizes the mid-range sits at a plausible-looking α ≈ 2–3, indistinguishable from a genuine fat tail without the full CSN machinery. Where the Student-t(ν) genuinely carries tail exponent α = ν, the Gaussian carries none at all — its estimate just keeps sliding. Run all four generators and watch the difference; that side-by-side is the honest education this page offers.

On the wing's tiers: this page earns the MEASURED pill only when a dataset with clean redistribution rights is committed beside it — until then, bring your own. A reader's own return series is theirs to measure, locally, with nothing uploaded anywhere. And a measured tail is a property of the past sample, not a forecast.

The estimator is Bruce M. Hill, A Simple General Approach to Inference About the Tail of a Distribution, Annals of Statistics 3(5), 1163–1174 (1975). The case against least-squares fits — and the honest fitting recipe — is Clauset, Shalizi & Newman, Power-law distributions in empirical data, SIAM Review 51(4), 661–703 (2009). The max-to-sum diagnostic is "Test 4: MS Plot" among the convergence diagnostics in Taleb's Statistical Consequences of Fat Tails (STEM Academic Press, 2020; free full text on arXiv); the strong law beneath it is in Embrechts, Klüppelberg & Mikosch, Modelling Extremal Events: for Insurance and Finance (Springer, 1997).

the war file →