# The forecasting equations behind predictions.md

Version 1, 2026-10-22. Every formula the site's tools use, with worked examples to test your own implementation against. The same code answers at https://www.predictions.md/api/v1 and ships as one dependency-free module at https://www.predictions.md/math/predictions-math.mjs. Test vectors: https://www.predictions.md/math/vectors.json.

MIT licensed. For information only: nothing here is betting, trading, investment, financial, legal or tax advice. A forecast is a probability, not a promise. Every name and number in this file is made up (Example Rovers, agent:news, q1). Terms: https://www.predictions.md/terms.

## How to use this in your own model

1. Port the formula. Each equation gives the math in plain ASCII, its variables, when to use it and the pitfalls the code guards against (clamping, floors, rounding, small n). Match the pitfalls too: most disagreements between implementations come from a clamp or a rounding step.
2. Run the vectors. `vectors.json` lists inputs and the exact outputs the code returns, with a tolerance; each worked example names its vector. `function` is `namespace.name` (a module function, called with the input's values in order) or a tool name (called with the input as the request). `expected` is checked key by key; prose fields are not vectors.
3. Cross-check against the site. Every tool named below answers `GET` or `POST https://www.predictions.md/api/v1/<tool>` with the same code: no key, nothing stored, nothing fetched. Send your own numbers and compare.

Conventions: `p` is a probability in [0, 1]; "points" means percentage points; `ln` is the natural log; `round(x, d)` is `Math.round(x * 10^d) / 10^d` (a half rounds up), and numbers are shown rounded unless marked unrounded. `logit`, `sigmoid` and `clamp` are defined in 1.1; the floor `f` is 0.001 unless a tool's `floor_pct` says otherwise.

## 1. Probability and odds

#### 1.1 Log-odds and the logistic function

Formula:

    logit(p) = ln(p / (1 - p))
    sigmoid(x) = 1 / (1 + exp(-x))
    clamp(p, f) = min(1 - f, max(f, p))

Variables: p strictly inside (0, 1); x any real number; f the floor.

Use when: averaging, correlating, calibrating or updating forecasts. Work on log-odds, then convert back.

Pitfalls: logit(0) and logit(1) are infinite, so clamp first. sigmoid(logit(p)) returns p only up to floating point; rounded log-odds lose precision on the way back.

Tool: `convert_odds` (`log_odds`, 4 decimals).

Worked example: logit(0.61) = 0.44731221804366483 (vector `logit_061`); logit(0.45) = -0.20067069546215124 (`logit_045`); sigmoid(1) = 0.7310585786300049 (`sigmoid_1`); sigmoid(0.4473) = 0.6099970933235057 (`sigmoid_04473`), which is 0.61 only to 4 decimals.

#### 1.2 Odds formats

Formula:

    decimal = 1 / p                       p = 1 / decimal
    fractional = (1 - p) / p              p = den / (num + den)      for "num/den"
    odds for = p / (1 - p)                p = a / (a + b)            for "a:b"
    American: p > 0.5: -100 p / (1 - p);  else +100 (1 - p) / p
              p = 100 / (A + 100) for A > 0;   p = -A / (-A + 100) for A < 0

Variables: p the probability; A the American figure.

Use when: reading a quote in any form. All of these describe one probability.

Pitfalls: decimal odds must be above 1; American odds are +100 or more, or -100 or less; "evens" is decimal 2; p = 0.5 shows as "+100 (even)". Fractions are the simplest with a denominator up to 100 (continued fractions): 0.61 gives 39/61. Auto reading: "61%" percent, "3/2" fractional, "3:2" odds for, a signed number of 100 or more American, a bare number in (0, 1) a probability, else decimal.

Tool: `convert_odds` with `value` and an optional `from`.

Worked example (`read_american_plus_150`, `read_american_minus_156`, `read_fractional_3_2`, `read_decimal_2_5`, `read_odds_for_3_2`, `read_percent_61`): "+150" reads as 0.4; "-156" as 156/256 = 0.609375; "3/2" as 0.4; "2.5" as 0.4; "3:2" as 0.6; "61%" as 0.61. (`convert_061`): 0.61 gives decimal 1.6393, "39/61", "-156.41", "61:39", log-odds 0.4473. (`convert_040`): 0.4 gives 2.5, "3/2", "+150", "2:3", -0.4055.

#### 1.3 Odds ratio, risk ratio and the difference in points

Formula:

    odds ratio = [p1 / (1 - p1)] / [p2 / (1 - p2)]
    risk ratio = p1 / p2
    difference = 100 (p1 - p2)      (points)

Variables: p1 the first probability; p2 the one it is compared with.

Use when: saying how two forecasts, or a forecast and a quote, differ.

Pitfalls: the odds ratio is undefined at 0 or 1 (null within 1e-9 of either end); the risk ratio at p2 = 0. Rounded to 4, 4 and 2 decimals.

Tool: `convert_odds` with `compare_to`.

Worked example (`compare_040_055`, `convert_odds_tool_example`): 0.4 against 0.55: odds ratio 0.5455, risk ratio 0.7273, difference -15 points; `{ "value": "+150", "compare_to": "55%" }` returns the same.

#### 1.4 Implied probabilities and the overround

Formula:

    q_i = 1 / decimal_i;   total = sum q_i;   overround = 100 (total - 1)      (points)
    proportional: p_i = q_i / total
    additive:     p_i = q_i - (total - 1) / n
    power:        p_i = q_i ^ k,  with k chosen so that sum p_i = 1 (bisection on k)

Variables: n outcomes of one event, exactly one of which happens; k > 0.

Use when: quoted odds cover every outcome and you want probabilities that add up to 1. Quotes are someone else's forecast plus a margin, never a fact.

Pitfalls: proportional takes the same share from each outcome; additive takes the same points from each (a larger share from long shots) and can push a small q_i below 0; power trims long shots harder but keeps every p_i above 0. Rounded to 6 decimals, the overround to 3.

Tool: `implied_probabilities` (proportional). Additive and power: none yet, and no vector, since the site has no code for them.

Worked example (`implied_three_outcomes`): q = [0.5, 0.3, 0.25]: total 1.05, 5 points over, proportional [0.47619, 0.285714, 0.238095]. (`implied_probabilities_tool_example`): decimal odds 2.10, 3.40, 3.60: implied 0.47619, 0.294118, 0.277778, total 1.048086 (4.809 points over), scaled 0.454343, 0.280624, 0.265033.

## 2. Bayes

#### 2.1 Bayes' rule in odds form

Formula:

    prior odds = p0 / (1 - p0)
    posterior odds = prior odds x LR
    p1 = posterior odds / (1 + posterior odds)

Variables: p0 the prior; LR the likelihood ratio of one piece of evidence; p1 the posterior.

Use when: you have a starting probability (section 3) and one piece of evidence.

Pitfalls: a prior of 0 or 1 can't move; the site reads it as 0.001 or 0.999 and says so (`clamped: true`). LR must be above 0. Steps are shown to 6 decimals, computed unrounded.

Tool: `bayes_update` with `prior` and `evidence`.

Worked example (`bayes_two_steps`): prior 0.45 (odds 0.818182); LR 1.4 gives odds 1.145455 and p 0.533898; then LR 0.6 gives odds 0.687273 and p 0.407328. (`bayes_prior_1_clamped`): a prior of 1 is read as 0.999 (odds 999); LR 0.5 gives odds 499.5 and p 0.998002, `clamped: true`.

#### 2.2 Likelihood ratio from two conditional probabilities

Formula:

    LR = P(E | yes) / P(E | no)

Variables: the chance of seeing the evidence E if the answer is yes, and if it is no.

Use when: judging how likely the evidence is under each answer is easier than judging the ratio.

Pitfalls: P(E | no) must be above 0. An LR of 1 is no evidence. The LR is your judgment; the math can't check it.

Tool: `bayes_update` with `p_if_yes` and `p_if_no`.

Worked example (`bayes_update_lr_and_p_if`): p_if_yes 0.3 and p_if_no 0.5 give LR 0.6 and the steps of 2.1: posterior 0.407328. (`bayes_update_tool_example`): the site's example uses p_if_no 0.45: LR 0.666667, posterior 0.43299.

#### 2.3 Sequential updates and log-odds additivity

Formula:

    posterior odds = prior odds x LR1 x LR2 x ... x LRk
    logit(p_k) = logit(p0) + ln LR1 + ln LR2 + ... + ln LRk

Variables: as in 2.1, for k pieces of evidence.

Use when: several pieces of evidence arrive. The order doesn't matter; the product does.

Pitfalls: this assumes the pieces are conditionally independent given the answer. Two pieces that say the same thing count twice: shrink their LRs or merge them.

Tool: `bayes_update` with the evidence in order.

Worked example (`logit_045`): logit(0.45) = -0.200671; ln 1.4 = 0.336472; ln 0.6 = -0.510826; the sum is -0.375024, and sigmoid(-0.375024) = 0.407328, the posterior of 2.1.

## 3. Base rates

#### 3.1 Beta posterior mean for k of n cases

Formula:

    estimate = (k + a) / (n + a + b)
    Laplace: a = b = 1;   Jeffreys: a = b = 0.5;   none: a = b = 0

Variables: k yes cases among n similar cases; a and b the Beta prior.

Use when: a reference class of past cases should give a starting probability that never hits 0% or 100%.

Pitfalls: with no prior, all-yes or all-no cases give 100% or 0%, which a forecast never is (the site refuses). The reference class is your choice, and it decides the answer. Rounded to 4 decimals.

Tool: `base_rate` with `yes`, `total` and an optional `prior` (laplace by default; jeffreys; none) or `prior_a`, `prior_b`.

Worked example (`cases_laplace_11_of_30`): 11 of 30 with Laplace: raw 0.3667, estimate 12 / 32 = 0.375. (`cases_jeffreys_11_of_30`): Jeffreys gives 11.5 / 31 = 0.371. (`base_rate_tool_example`): the paste line reads "- base_rate: 38% (11 of 30, Laplace; 90% interval 24% to 52%; class: ...)".

#### 3.2 Beta quantile interval

Formula:

    posterior ~ Beta(k + a, n - k + b)
    interval = [BetaInv(tail, k + a, n - k + b), BetaInv(1 - tail, k + a, n - k + b)],  tail = (1 - level) / 2

Variables: level 0.9 by default; BetaInv the inverse of the regularized incomplete beta function.

Use when: saying how uncertain the base rate is.

Pitfalls: BetaInv is found by bisection to 1e-15, so any correct implementation agrees to 6 decimals. Null unless both parameters are above 0.

Tool: `base_rate` (`interval`).

Worked example (`beta_inv_005_12_20`, `beta_inv_095_12_20`): for Beta(12, 20) the 5% quantile is 0.24077485846609248 and the 95% quantile 0.5182492532388516, so 11 of 30 with Laplace has the 90% interval [0.2408, 0.5182].

#### 3.3 Wilson interval

Formula:

    center = (k + z^2 / 2) / (n + z^2)
    half = z sqrt(k (n - k) / n + z^2 / 4) / (n + z^2)
    interval = [max(0, center - half), min(1, center + half)],   z = NormInv(1 - (1 - level) / 2)

Variables: k successes in n; z the standard normal quantile.

Use when: a frequentist interval for a rate: base rates, calibration bins, coverage of numeric intervals.

Pitfalls: null for n = 0. The site's NormInv is a rational approximation accurate to about 1e-9.

Tool: `base_rate` (`wilson`); `score_forecasts` (`observed_interval`); `score_distribution` (`rate_interval`).

Worked example (`wilson_11_of_30_z_1645`): k = 11, n = 30, z = 1.645: [0.23864709853367255, 0.5167496517592782], shown as [0.2387, 0.5167]. (`norm_inv_0975`): NormInv(0.975) = 1.959963986120195.

#### 3.4 Gamma-Poisson rate for k events over a period

Formula:

    rate ~ Gamma(shape = k + a, rate = T + b)          (T and b in days)
    estimate per unit = (k + a) / (T + b) x days per unit
    interval = [GammaInv(tail, k + a), GammaInv(1 - tail, k + a)] / (T + b) x days per unit

Variables: k events in T days; a, b the Gamma prior (Jeffreys a = 0.5, b = 0 by default); days per unit 1, 7, 30.4375 or 365.25.

Use when: the reference class is a count of events over time, not a count of cases.

Pitfalls: with no prior and no events the rate is 0 (the site refuses). Rates are shown to 4 significant digits. GammaInv is bisection on the regularized lower incomplete gamma.

Tool: `base_rate` with `events`, `over` and `unit`.

Worked example (`events_rate_3_in_20_years`, `base_rate_events_tool`): 3 events in 20 years (7305 days) with Jeffreys: estimate 3.5 / 7305 x 365.25 = 0.175 a year, 90% interval 0.05418 to 0.3517, raw rate 0.15.

#### 3.5 Chance of at least one event in a window

Formula:

    P(at least one in t days) = 1 - ((T + b) / (T + b + t)) ^ (k + a)      (exact under Gamma uncertainty)
    plug-in = 1 - exp(-k t / T)

Variables: t the days left in the window; the rest as in 3.4. With `count_elapsed` (the default), quiet days already gone are added to T.

Use when: a "will it happen by date X" question whose base rate is an event count.

Pitfalls: compute it as -expm1(-(k + a) log1p(t / (T + b))) to keep precision for small chances. It assumes a constant rate and independent events.

Tool: `base_rate` with `events`, `over`, `unit` and `window_days` (or `from` and `to`).

Worked example (`chance_at_least_one_180_days`): shape 3.5, T = 7305, t = 180: 0.08166855685003568, shown as 0.081669; the plug-in is 0.071256. (`chance_at_least_one_after_90_quiet_days`): 90 quiet days later, T + 90 = 7395 and t = 90: 0.041455; the tool's schedule shows day 179 at 0.000468.

#### 3.6 Constant-hazard decay of a by-a-date forecast

Formula:

    p(t) = 1 - (1 - p_now) ^ ((D - t) / D)

Variables: p_now the forecast today; D the days to the resolve date, inclusive; t the days from now.

Use when: nothing has happened and nothing else has changed. It says what the forecast drifts to on its own, not what it should be.

Pitfalls: new evidence should move the forecast instead. Rounded to 6 decimals.

Tool: `base_rate` with `p_now`, `by` and `as_of`.

Worked example (`decay_30pct_over_10_days`, `base_rate_decay_tool`): p_now 0.3 with D = 10 (2026-12-22 to 2026-12-31): day 5 gives 1 - 0.7^0.5 = 0.16334; day 9 gives 1 - 0.7^0.1 = 0.035039.

## 4. Pooling

#### 4.1 Mean, median and the geometric mean of odds

Formula:

    mean = sum w_i p_i / sum w_i
    median = the middle value of p (unweighted; the average of the two middle values for an even count)
    L = sum w_i logit(clamp(p_i, f)) / sum w_i;   geo_odds = sigmoid(L)
    spread = 100 (max p - min p)

Variables: p_i the forecasts; w_i their weights (1 by default); f the floor.

Use when: several forecasters answered the same yes/no question; the skill's default is geo_odds.

Pitfalls: forecasts outside [f, 1 - f] are clamped for the log-odds average only (`clamped: true`); the mean and median use the raw values. A weight of 0 drops a forecast; all weights 0 is an error. Rounded to 6 decimals, the spread to 2.

Tool: `combine_forecasts` (`combined`).

Worked example (`combine_three_equal_weights`): 0.58, 0.61 and 0.55: geo_odds 0.580198, mean 0.58, median 0.58, spread 6 points. (`combine_three_weighted_2_1_1`): weights 2, 1, 1: geo_odds 0.580148. (`combine_floor_clamps_0_0001`): 0.0001 and 0.6: the first is read as 0.001 for the log-odds: geo_odds 0.037304, mean 0.30005, `clamped: true`.

#### 4.2 Trimmed average of log-odds

Formula:

    k = floor(G x trim / 100) if G >= 5, else 0
    drop the k smallest and the k largest log-odds, then take the weighted mean of the rest

Variables: G the number of votes (groups); trim the percent dropped from each end (10 by default).

Use when: one wild vote should not drag the pool.

Pitfalls: nothing is trimmed under 5 votes. The trim is on the log-odds scale.

Tool: `combine_forecasts` with `method: trimmed_geo_odds`.

Worked example (`trimmed_logit_10_values_trim_10`): log-odds -2, -1, 0, 1, 2, 3, 4, 5, 6, 20 with trim 10: k = 1, the mean of the 8 kept is 2.5, 2 dropped. (`trimmed_logit_under_5_values`): -2, 0, 20: nothing dropped, mean 6.

#### 4.3 Groups count once

Formula:

    L_g = weighted mean of logit(clamp(p_i, f)) within group g
    L = sum W_g L_g / sum W_g,   W_g = the mean weight of the group's members
    samples: sd in points (n - 1), se = sd / sqrt(n), samples for a target se = ceil((sd / target)^2)

Variables: groups from the `group` field, or with `group_samples` from a name ending in #n.

Use when: one agent sampled several times, or near-copies of one model, should not outvote the others.

Pitfalls: a group's vote carries the mean of its members' weights, not their sum. Identical samples add nothing and are flagged.

Tool: `combine_forecasts` with `group` fields, or `group_samples: true`.

Worked example (`combine_forecasts_groups_count_once`): agent:ratings#1 0.6 and agent:ratings#2 0.64 in group "ratings", agent:form 0.5: the group pools to 0.620204, then 2 votes give geo_odds 0.560997; the group's sd is 2.83 points, its se 2, and 3 samples would bring the se under 2. (`pool_checks_samples_target_se`): samples 0.6, 0.64, 0.56: sd 4 points, se 2.31, 4 samples for a 2-point se.

#### 4.4 Extremizing

Formula:

    geo_odds = sigmoid(extremize x L)
    mean, median: sigmoid(extremize x logit(clamp(value, f)))

Variables: extremize a factor from 0.5 to 3 (1 = no change).

Use when: a pool of independent, under-confident forecasters sits too close to 50%. 6.6 (`slope_only.factor`) gives the factor that would have fit past outcomes.

Pitfalls: extremizing correlated forecasters doubles down on their shared error (section 5).

Tool: `combine_forecasts` with `extremize`.

Worked example (`combine_extremize_1_5`): 0.58, 0.61 and 0.55 with factor 1.5: geo_odds 0.619017, mean and median 0.61873.

#### 4.5 Influence of one forecast

Formula:

    influence_i = 100 (pool - pool without i)      (points)
    leave_one_out_range = [min, max] of the pools without each forecast in turn

Variables: pool the lead method's result.

Use when: one forecast alone moves the pool by more than `max_influence_pts` (10 by default): check it before you report the combination.

Pitfalls: a flagged forecast can be the right one. The flag is a check, not a verdict.

Tool: `combine_forecasts` (`influence`, `leave_one_out_range`).

Worked example (`combine_forecasts_tool_example`): agent:form 0.58 moves the pool by -0.01 points, agent:ratings 0.61 by +1.51, person 0.55 by -1.49; the leave-one-out range is [0.56506, 0.595089].

#### 4.6 Outliers by modified z with the MAD

Formula:

    x_i = logit(clamp(p_i, f));  med = median(x);  MAD = median(|x_i - med|)
    modified z_i = 0.6745 (x_i - med) / MAD;   outlier when |z_i| > 3.5
    MAD = 0: any x_i that differs from med is flagged

Variables: as above.

Use when: 5 or more forecasts; under 5 no outlier check runs.

Pitfalls: the check is on log-odds, so 0.05 among values near 0.55 is far.

Tool: `combine_forecasts` (`outliers`).

Worked example (`pool_checks_outlier_modified_z`): 0.55, 0.58, 0.6, 0.57 and 0.05: agent:e has modified z -26.81 and alone moves the pool by -16.09 points (flagged); the pool is 0.414174.

#### 4.7 Slip detection

Formula:

    m = median of the other forecasts' log-odds
    "0.6% meant as 60%?":  p <= 0.01 and |logit(100 p) - m| <= 0.4 and |logit(p) - m| >= 2
    "the chance of no?":   |logit(1 - p) - m| <= 0.4 and |logit(p) - m| >= 1.5
    also flagged: sent as the number 1 (read as 100%); outside the floor

Variables: as above, every log-odds clamped at f.

Use when: a number sits far from the others while its complement, or 100 times it, sits with them.

Pitfalls: needs at least 2 other forecasts. Flags only; nothing is dropped.

Tool: `combine_forecasts` (`suspects`).

Worked example (`pool_checks_slip_0_6_percent`): 0.58, 0.61, 0.55 and 0.006: agent:d is flagged "0.6% meant as 60%?". (`pool_checks_slip_chance_of_no`): 0.8, 0.82, 0.78 and 0.2: "20%: the chance of no? 80% would sit with the others."

## 5. Correlation and independence

#### 5.1 Pearson correlation of log-odds

Formula:

    x_i = logit(clamp(p_ai, f)),  y_i = logit(clamp(p_bi, f))      over the questions both answered
    r = sum (x_i - mean x)(y_i - mean y) / sqrt(sum (x_i - mean x)^2 x sum (y_i - mean y)^2)

Variables: forecasters a and b; f the floor.

Use when: asking whether two forecasters move together across questions.

Pitfalls: null under 2 shared points, or when either side is constant (a sum of squares under 1e-18); r is clipped to [-1, 1]. Pairs under `min_shared` questions (5 by default) are not reported.

Tool: `correlate_forecasters` (`pairs[].r_log_odds`, 4 decimals).

Worked example (`pearson_hand_case`): x = [-1, 0, 1, 2, 3], y = [0, 1, 1, 3, 5]: sxy = 12, sxx = 10, syy = 16, r = 12 / sqrt(160) = 0.9486832980505138. (`pearson_constant_side_is_null`): x all 1 gives null.

#### 5.2 Spearman correlation

Formula:

    rank(x): 1 for the smallest; tied values share the average of the ranks they span
    Spearman = Pearson(rank(x), rank(y))

Variables: as in 5.1, on the raw probabilities (ranks don't change under logit).

Use when: you care about order, not spacing.

Pitfalls: ties use average ranks, so the shortcut 1 - 6 sum d^2 / (n (n^2 - 1)) is wrong with ties.

Tool: `correlate_forecasters` (`spearman`).

Worked example (`ranks_with_a_tie`): [3, 1, 2, 2] ranks as [4, 1, 2.5, 2.5]. (`spearman_hand_case`): the data of 5.1 has one tie in y: Spearman 0.9746794344808964.

#### 5.3 Fisher z interval for r

Formula:

    z_r = atanh(r);  se = 1 / sqrt(n - 3)
    interval = [tanh(z_r - z se), tanh(z_r + z se)],   z = NormInv(1 - (1 - confidence) / 2)

Variables: r the correlation on n shared questions.

Use when: n >= 10. The interval says how little a small n pins r down.

Pitfalls: null under 10 points. r is clipped to +/-0.999999999 before atanh.

Tool: `correlate_forecasters` (`r_interval`).

Worked example (`fisher_interval_r_06_n_12`): r = 0.6 on 12 points at 95%: [0.039804809295699115, 0.8732171697474119]. (`fisher_interval_under_10_is_null`): 9 points gives null.

#### 5.4 Error correlation

Formula:

    e_ai = p_ai - outcome_i;   r_errors = Pearson(e_a, e_b)      over resolved shared questions
    r_brier = Pearson(e_a^2, e_b^2)

Variables: outcome_i 1 for yes, 0 for no.

Use when: 3 or more shared questions have resolved.

Pitfalls: across questions with both outcomes every error shares the outcome's sign, so r_errors runs high for any two forecasters; r_brier is the better guide to being off on the same questions.

Tool: `correlate_forecasters` (`r_errors`, `r_brier`, `n_resolved`).

Worked example (`correlate_two_forecasters_resolved`): agent:form 0.6, 0.3, 0.7, 0.45, 0.8 and agent:news 0.45, 0.55, 0.4, 0.65, 0.5 on five resolved questions (yes, no, yes, no, yes): r of log-odds -0.5723, Spearman -0.6, 24 points apart, agreement 0, and still r_errors 0.9826 and r_brier 0.7565.

#### 5.5 Mean pairwise correlation and the effective number of forecasters

Formula:

    rho_bar = sum (n_shared_ab x r_ab) / sum n_shared_ab      over the reported pairs
    n_eff = n / (1 + (n - 1) rho_bar),  kept in [1, n];  rho_bar <= -1 / (n - 1) counts as n
    variance ratio = 1 / n_eff;   one more adds = n_eff(n + 1) - n_eff(n)

Variables: n forecasters; r_ab each pair's correlation of log-odds.

Use when: deciding whether another similar forecaster is worth adding, or how much agreement is worth.

Pitfalls: n_eff treats every pair as equally correlated at rho_bar: a summary of the pool, not a model of these forecasters. Under equal variances and correlations, var(mean of n) = (1 + (n - 1) rho) / n times one forecaster's.

Tool: `correlate_forecasters` (`overall`).

Worked example (`n_effective_3_rho_05`, `n_effective_4_rho_05`): 3 forecasters at rho 0.5: 3 / (1 + 2 x 0.5) = 1.5; 4 give 1.6, so one more adds 0.1 of a voice. (`correlate_forecasters_tool_example`): three forecasters on 8 questions: form and ratings r 0.993 (near-duplicates, 2.25 points apart), form and news -0.0418, ratings and news 0.0081; rho_bar 0.3198, n_eff 1.83, variance ratio 0.5465, one more adds 0.212.

## 6. Scoring yes/no forecasts

#### 6.1 Brier score and log score

Formula:

    brier_i = (p_i - o_i)^2;   Brier = mean brier_i            (0 is perfect, 0.25 is always saying 50%)
    log loss_i = -ln(clamp(p_i, f)) on a yes, -ln(1 - clamp(p_i, f)) on a no;   log score = -mean log loss_i

Variables: o_i 1 for yes, 0 for no; f the floor (`floor_pct` / 100).

Use when: questions have resolved. Score each forecaster's latest forecast made before the cutoff.

Pitfalls: the log score needs the floor or one 0% is minus infinity; `clamp` reports how many rows sat at or past the floor and their share of the loss. Scores are rounded to 4 decimals.

Tool: `score_forecasts` (`overall.brier`, `overall.log_score`); `check_predictions_md` for a whole file.

Worked example (`brier_loss_082_yes`): 0.82 on a yes: 0.0324 (the code returns 0.03240000000000002, hence the tolerance). (`brier_loss_065_no`): 0.65 on a no: 0.4225. (`log_loss_082_yes`): -ln 0.82 = 0.19845093872383832. (`log_loss_clamped_at_floor`): 0.0005 on a yes is read as 0.001: 6.907755278982137.

#### 6.2 Skill against a baseline

Formula:

    skill = 1 - score / baseline score
    baselines: always 50% (Brier 0.25, log loss ln 2); the stated base_rate of each question; the hindsight base rate b = yes / n (Brier b (1 - b), log loss the entropy of b)
    diff = mean (own loss_i - baseline loss_i), with a t interval (6.3) on the paired differences

Variables: as in 6.1.

Use when: a bare score means little; skill says how much better than a baseline.

Pitfalls: hindsight skill uses the outcomes and is undefined when every outcome is the same. The stated baseline needs `base_rate` on the rows. The card's skill uses the rounded score, the overall field the unrounded one, so the last decimal can differ.

Tool: `score_forecasts` (`overall.baselines`, `brier_skill_vs_base_rate`, `card.skill`).

Worked example (`score_forecasts_tool_example`): six forecasts with Brier 0.1462: skill against 50% is 1 - 0.1462 / 0.25 = 0.4152 on the card, 0.4151 unrounded against the hindsight base rate of 0.5; the Brier difference against 50% is -0.1038, 95% interval [-0.2595, 0.0519].

#### 6.3 t interval for a mean

Formula:

    se = sd / sqrt(n),  sd with n - 1
    interval = mean +/- t(n - 1, 1 - (1 - confidence) / 2) x se

Variables: n rows; confidence 0.95 by default.

Use when: n >= 5. The site gives no interval below that.

Pitfalls: it assumes independent rows (see 6.4). Score intervals are clipped to the values the score can take ([0, 1] for Brier). The t quantile is bisection on the t CDF.

Tool: `score_forecasts` (`brier_interval`); `score_history`; `score_distribution`.

Worked example (`mean_interval_five_values`): 0.1, 0.2, 0.3, 0.4, 0.5: mean 0.3, se 0.07071067811865475, t(4) = 2.776445105198036 (`t_inv_0975_df_4`), interval [0.10367568385222714, 0.49632431614777284]. (`t_two_sided_p_2_df_11`): the two-sided p of t = 2 on 11 df is 0.07080395506803462.

#### 6.4 Cluster-robust interval, design effect and n_effective

Formula:

    S_g = sum over the rows i in cluster g of (x_i - mean)
    se^2 = [G / (G - 1)] sum_g S_g^2 / n^2,   df = G - 1
    design effect = se^2 / se_iid^2;   n_effective = n / design effect

Variables: clusters by `group`, else by question when several forecasters share one, else every row alone (which gives 6.3).

Use when: rows are not independent: several forecasters on one question, or questions that share a cause.

Pitfalls: null with a single cluster. With every key distinct it equals sd / sqrt(n) on n - 1 df.

Tool: `score_forecasts` (`n_effective`); `compare_forecasters`; `score_history`.

Worked example (`mean_interval_clustered_3_groups`): 0.1 to 0.6 in three pairs: mean 0.35, se 0.11547005383792514 on 2 df, design effect 2.285714285714285, n_effective 2.6250000000000004, interval [-0.14682754235009943, 0.8468275423500995].

#### 6.5 Calibration bins with Wilson intervals

Formula:

    bin b holds p in [b / B, (b + 1) / B), the last bin closed at 1
    mean_forecast = mean p in the bin;   observed = yes / n in the bin;   observed_interval = Wilson(yes, n, z)
    mean_outside_interval = mean_forecast outside that interval

Variables: B bins (10 by default).

Use when: asking whether 70% forecasts came true about 70% of the time.

Pitfalls: empty bins are left out; bins under 5 forecasts are marked `under_5` and say little.

Tool: `score_forecasts` (`overall.calibration`, `bins`).

Worked example (`score_forecasts_tool_example`, 5 bins): the 20-40% bin holds 2 forecasts, mean 0.25, observed 0, interval [0, 0.6576]; the 60-80% bin holds 2, mean 0.675, observed 0.5, interval [0.0945, 0.9055]; every bin is under 5.

#### 6.6 Calibration slope and intercept by logistic fit

Formula:

    P(yes | p) = sigmoid(a + b logit(clamp(p, 0.001))),  fitted by maximum likelihood (Newton's method)
    se from the inverse of the Fisher information;   interval = estimate +/- z se
    slope_only: a fixed at 0, so b is the extremize factor that would have fit these outcomes

Variables: a the intercept; b the slope.

Use when: 30 or more forecasts with at least 5 of each outcome.

Pitfalls: skipped under that size, or when the outcomes are separated too cleanly to converge. Slope below 1: more extreme than the outcomes supported; above 1: less extreme. Intercept above 0: yes came more often than forecast.

Tool: `score_forecasts` (`overall.calibration_fit`).

Worked example (`logistic_fit_twelve_points`): x = [-2, -1.5, -1, -0.5, 0, 0.5, 1, 1.5, 2, 2.5, -0.25, 0.25], o = [0, 0, 0, 1, 0, 1, 1, 1, 1, 1, 0, 1]: a = 0.4596553059290784, b = 3.03102538424095, se_a 0.9849910661306104, se_b 1.9712056675416032. (`score_forecasts_thirty_rows_calibration_fit`): 30 made-up rows: intercept 0.1716 [-0.5663, 0.9094], slope 0.3313 [-0.2328, 0.8953], slope-only factor 0.3207.

#### 6.7 Spiegelhalter's z

Formula:

    z = sum (o_i - p_i)(1 - 2 p_i) / sqrt(sum (1 - 2 p_i)^2 p_i (1 - p_i))
    p-value = erfc(|z| / sqrt(2))

Variables: as in 6.1.

Use when: a one-number calibration test that needs no bins.

Pitfalls: null when every p is 0.5 (the denominator is 0). The site's erfc is accurate to about 1e-7.

Tool: `score_forecasts` (`overall.spiegelhalter`).

Worked example (`score_forecasts_tool_example`): z -0.713, p 0.476. (`score_forecasts_thirty_rows_calibration_fit`): z 2.306, p 0.0211. (`erfc_1`): erfc(1) = 0.15729921223999332.

#### 6.8 Isotonic (PAV) decomposition

Formula:

    f = PAV(p, o): the non-decreasing fit of the outcomes against the forecasts, equal p pooled first
    miscalibration = mean (p_i - o_i)^2 - mean (f_i - o_i)^2
    discrimination = b (1 - b) - mean (f_i - o_i)^2,   b = yes / n;   uncertainty = b (1 - b)
    Brier = miscalibration - discrimination + uncertainty      (log loss the same way, with -ln and the entropy of b)

Variables: PAV merges neighbouring blocks whose means decrease until the fit is non-decreasing.

Use when: splitting a score into what calibration costs and what sorting earns, without bins.

Pitfalls: with few rows the split is rough. Recalibrated values at 0 or 1 are read as at least 1e-300 for the log loss.

Tool: `score_forecasts` (`overall.decomposition`).

Worked example (`pav_one_violation`): p = [0.2, 0.4, 0.6, 0.8], o = [0, 1, 0, 1]: fitted [0, 0.5, 0.5, 1]. (`score_forecasts_tool_example`): Brier 0.1462 = miscalibration 0.0629 - discrimination 0.1667 + uncertainty 0.25, to rounding.

#### 6.9 AUC

Formula:

    AUC = (sum of the ranks of the yes rows - n_yes (n_yes + 1) / 2) / (n_yes n_no);   ties in p share the average rank

Variables: n_yes and n_no the counts of each outcome.

Use when: asking how well the forecasts sort yes from no, calibration aside.

Pitfalls: null without both outcomes. 0.5 is no sorting.

Tool: `score_forecasts` (`overall.auc`).

Worked example (`auc_four_forecasts`): p = [0.2, 0.4, 0.6, 0.8], o = [0, 1, 0, 1]: 0.75. (`auc_tie_counts_half`): p = [0.5, 0.5, 0.9], o = [0, 1, 1]: 0.75.

#### 6.10 Forecasts needed for a half-width

Formula:

    half-width now = (hi - lo) / 2
    forecasts for half-width h = ceil((z sd / h)^2 x design effect)

Variables: sd of the per-row losses; z for the confidence level.

Use when: saying how many more resolved forecasts a tighter interval needs.

Pitfalls: it assumes the spread of losses stays the same. Under 5 forecasts there is no interval and no estimate.

Tool: `score_forecasts` (`overall.precision`).

Worked example (`score_forecasts_tool_example`): half-width 0.1557 now; 212 forecasts for +/-0.02 and 846 for +/-0.01.

#### 6.11 The cite line and its digest

Formula:

    rows sorted by q, then who, then p; each as {"q","who","p" (6 decimals),"outcome" (1 or 0)} in compact JSON
    input_sha256 = SHA-256 of that JSON text, hex
    cite = "<Rule> <value, 3 decimals> (<level> <lo>-<hi>), <n> forecasts, 50% baseline <baseline>, score/2, sha256 <first 12 hex>"

Variables: Rule is Brier or Log loss; with no interval the parenthesis says so.

Use when: quoting a score. The digest shows only that these numbers come from these rows under score/2, not when or by whom the forecasts were made. It is never a certificate.

Pitfalls: any change to a row, to the order of keys or to the rounding changes the digest. Use exactly the JSON form above.

Tool: `score_forecasts` (`cite`, `card.input_sha256`).

Worked example (`rows_digest_two_rows`): rows q2 / agent:news / 0.3 / no and q1 / agent:news / 0.82 / yes sort to q1 first and digest to b8c5c79ff593777b854b8c9049720336bee7d0cf061d5062ec6b2a4c495ce454. (`score_forecasts_tool_example`): cite "Brier 0.146 (95% 0.000-0.302), 6 forecasts, 50% baseline 0.250, score/2, sha256 bd30e7b616fc".

## 7. Comparing forecasters

#### 7.1 Paired differences on shared questions

Formula:

    d_q = loss_A(q) - loss_B(q)      over the questions both answered
    mean_diff = mean d_q;   interval = mean +/- t(df) se      (6.3, or 6.4 with groups)
    relative skill = 1 - mean loss_A / mean loss_B
    wins_A = count of d_q < 0;  wins_B = count of d_q > 0;  ties the rest (within 1e-12)

Variables: loss the Brier loss or the log loss per question (`rule`).

Use when: asking whether a new agent, prompt or model did better, rather than comparing two separate scores.

Pitfalls: under 10 shared questions: counts and means only, no interval, no test. A negative mean_diff means A had the lower loss. `flips_if_largest_dropped` says whether the sign would change without the biggest difference.

Tool: `compare_forecasters` with `a` and `b` (`comparisons[0]`).

Worked example (`compare_forecasters_tool_example`): agent:v7 against agent:v6 on 12 questions: losses 0.0942 and 0.1996, mean difference -0.1054 with 95% interval [-0.122, -0.0888], 12 wins to 0, relative skill 0.5282. (`compare_forecasters_log_rule`): with `rule: log`, losses 0.361 and 0.5903, difference -0.2294 [-0.2575, -0.2012].

#### 7.2 Exact sign test

Formula:

    m = wins_A + wins_B;   k = min(wins_A, wins_B)
    p = min(1, 2 sum over j = 0..k of C(m, j) / 2^m)

Variables: C(m, j) the binomial coefficient, computed with log-gamma.

Use when: a test that assumes nothing about the size of the differences.

Pitfalls: ties are left out; null when m = 0.

Tool: `compare_forecasters` (`sign_test_p`).

Worked example (`sign_test_9_wins_3_losses`): 9 wins to 3: 2 x (1 + 12 + 66 + 220) / 4096 = 0.14599609374999933. (`sign_test_no_decisions_is_null`): 0 and 0 give null.

#### 7.3 t test and Holm's adjustment

Formula:

    t = mean_diff / se;   p = the two-sided tail of t on df
    Holm: sort the p-values ascending; adjusted p_(k) = max over j <= k of min(1, (m - j + 1) p_(j))

Variables: m the number of comparisons in one call.

Use when: several comparisons in one call: every forecaster against a reference, or the contribution mode.

Pitfalls: separate calls get no adjustment, so count every variant you tried. A comparison counts as separated only when its interval excludes 0 and, with several, its Holm p is below 1 - confidence.

Tool: `compare_forecasters` (`t_test_p`, `holm_p`).

Worked example (`holm_three_p_values`): [0.01, 0.04, 0.03] adjust to [0.03, 0.06, 0.06]. (`compare_forecasters_ref_half_holm`): three forecasters against always 50%: Holm p 0.019, 0 and 0.0024, all separated.

#### 7.4 Questions needed for 80% power

Formula:

    n = ceil(((z + 0.8416) sd(d) / |mean d|)^2 x design effect)

Variables: z for the confidence level; 0.8416 the normal quantile for 80% power.

Use when: a gap looks real but the interval covers 0: how many shared questions a gap this size would need.

Pitfalls: null when mean d is 0. It assumes the gap and the spread stay the same.

Tool: `compare_forecasters` (`questions_for_80pct_power`).

Worked example (`compare_forecasters_ref_half_holm`): agent:v6 against 50% (mean difference -0.0504) would need about 13 shared questions; agent:v8 (-0.0748) about 6.

#### 7.5 Leave-one-out contribution to the pool

Formula:

    c_q = loss(pool without k, q) - loss(pool with everyone, q)
    mean_contribution = mean c_q over the questions k answered

Variables: the pool as in section 4 (`pool_method`, `extremize`, `trim_pct`).

Use when: asking whether the pool scored better with a forecaster in it.

Pitfalls: contributions don't add up to the pool's skill. Interval and tests from 10 shared questions; Holm across the forecasters.

Tool: `compare_forecasters` with `contribution: true`.

Worked example (`compare_forecasters_contribution`): agent:v7 +0.0598 [0.0513, 0.0684] (the pool scored better with it); agent:v6 -0.0456 [-0.0538, -0.0374] (better without it).

#### 7.6 The hindsight-single baseline

Formula:

    pick the one other forecaster with the lowest mean loss on the questions it shares with A, after the outcomes are known

Variables: A the forecaster under test; the candidates every other forecaster.

Use when: asking whether A beat the best single alternative. The pick uses the outcomes, which flatters it; the label says so.

Pitfalls: among many candidates the best one by luck looks good; treat a narrow win over it as weak evidence.

Tool: `compare_forecasters` with `b: hindsight_single`.

Worked example (`compare_forecasters_hindsight_single`): among agent:v6 and agent:v8, agent:v8 (Brier 0.1752) is picked; agent:v7 still has the lower loss, difference -0.081 [-0.1094, -0.0527].

## 8. Coherence

#### 8.1 The relations

Formula:

    one_of:          sum p_i = 1                            off = 100 |sum - 1|
    at_most_one_of:  sum p_i <= 1                           off = 100 max(0, sum - 1)
    implies:         p(if) <= p(then)                       off = 100 max(0, p(if) - p(then))
    chain:           p_1 <= p_2 <= ..., narrowest first     off = 100 x the largest adjacent violation
    complement:      p_a + p_b = 1                          off = 100 |sum - 1|

Variables: one forecaster's numbers on the linked questions.

Use when: questions are linked by logic: exactly one of several dates, "by June" implies "by December", a question and its negation.

Pitfalls: the tolerance is half a point per question in the relation, at least 1, to absorb rounding. A wrongly declared relation gives a confident wrong repair.

Tool: `check_coherence` (`forecasters[].checks`).

Worked example (`check_coherence_tool_example_one_of`): 0.55, 0.3 and 0.35 on a one_of set: 20 points off against a tolerance of 1.5.

#### 8.2 Repair by projection onto the simplex with a floor

Formula:

    v_i = x_i - f;   room = 1 - n f
    sort v descending; rho = the largest j with v_(j) - (sum_{i <= j} v_(i) - room) / j > 0
    theta = (sum_{i <= rho} v_(i) - room) / rho;   y_i = max(v_i - theta, 0) + f

Variables: x the forecasts; y the repair; f the floor (0.01 in `check_coherence` by default, the skill's floor).

Use when: a one_of or complement set, or an at_most_one_of set whose clipped total is over 1 (otherwise clipping each value to [f, 1 - f] is enough).

Pitfalls: with n f > 1 no repair exists (lower the floor). The repair is the nearest point in squared distance; its margin is m = sum (x_i - y_i)^2.

Tool: `check_coherence` (`repaired`, `incoherence_m`).

Worked example (`check_coherence_tool_example_one_of`): 0.55, 0.3 and 0.35 (sum 1.2) each move by -6.67 points to 0.4833, 0.2333 and 0.2833; m = 0.013333. (`check_coherence_complement`): 0.6 and 0.5 become 0.55 and 0.45. (`check_coherence_at_most_one_of_over`): 0.5, 0.4 and 0.3 become 0.4333, 0.3333 and 0.2333.

#### 8.3 Repair of implies and chains by PAV

Formula:

    y = PAV over the chain order, narrowest first; then clamp each value to [f, 1 - f]

Variables: as in 6.8.

Use when: a single implies or chain relation.

Pitfalls: PAV pools the violating neighbours to their mean, so both sides move.

Tool: `check_coherence`.

Worked example (`check_coherence_implies`): by-june 0.6 and by-december 0.5 both become 0.55. (`check_coherence_chain`): 0.6, 0.5 and 0.7 become 0.55, 0.55 and 0.7.

#### 8.4 Overlapping relations: Dykstra's alternating projections

Formula:

    repeat: project onto each relation's set in turn, with Dykstra's increments, then onto the box [f, 1 - f]
    stop when nothing moves by more than 1e-11, or after 5000 rounds

Variables: one equality or inequality per relation; a chain gives one per adjacent pair.

Use when: two or more relations share a question, so no single projection applies.

Pitfalls: relations that can't all hold inside the floor are reported as a problem, not forced. Questions linked by any relation form a component, repaired and checked on its own.

Tool: `check_coherence` (`method` lists "Dykstra's alternating projections").

Worked example (`check_coherence_overlapping_dykstra`): one_of {a, b, c} = 0.5, 0.3, 0.3 and "a implies d" with d = 0.4: a 0.44, b 0.28, c 0.28, d 0.44; m = 0.006.

#### 8.5 The Brier-dominance claim and when the floor breaks it

Formula:

    for every yes/no outcome vector o the relations allow:  change(o) = sum (y_i - o_i)^2 - sum (x_i - o_i)^2
    worst_change = max over o;   best_change = min over o
    without a floor, change(o) <= -m for every allowed o (projection onto a convex set that contains them all)

Variables: as in 8.2.

Use when: saying the repair lowers the total Brier score whatever happens. The site lists the allowed outcomes (up to 16 linked questions) and claims it only when worst_change <= 0.

Pitfalls: the floor's box can break it. The site then says so and reports the worst rise.

Tool: `check_coherence` (`brier`).

Worked example (`check_coherence_tool_example_one_of`): worst and best change -0.0133, so the repair lowers the total Brier in every allowed outcome. (`check_coherence_with_outcomes`): with outcomes no, yes, no the total goes from 0.915 to 0.9017. (`check_coherence_floor_breaks_dominance`): 0.995 and 0.005 on a one_of pair with a 1% floor sum to 1 but sit outside the floor and move to 0.99 and 0.01: in the outcome (yes, no) the total rises by 0.0002, in (no, yes) it falls by 0.0199.

#### 8.6 Pooling several forecasters on one set

Formula:

    p_k proportional to (prod_i p_ik) ^ (extremize / n)      over the set's options, plus "none of these" for at_most_one_of

Variables: n forecasters who each gave every question of the set; p_ik forecaster i's value on option k, clamped at f.

Use when: several forecasters each gave a full distribution over a one_of set.

Pitfalls: needs 2 or more forecasters with every question of the set.

Tool: `check_coherence` (`pooled`).

Worked example (`check_coherence_pooled_set`): agent:x 0.6 / 0.4 and agent:y 0.4 / 0.6 pool to 0.5 / 0.5.

## 9. Time

#### 9.1 Time-averaged Brier over a forecast's history

Formula:

    p(t) = the forecaster's latest line at or before t
    time_avg_brier = integral over the window of (p(t) - o)^2 dt / span
    skip (default): span = cutoff - the later of the first line and the window start, integrating from there
    half: span = cutoff - window start, with p = 0.5 (Brier 0.25) before the first line
    window start = opened_at, else the first forecast by anyone;   cutoff = the earlier of resolved_at and the end of the resolve_by day

Variables: o the outcome; the lines dated forecasts.

Use when: crediting a call made early, not just the last line.

Pitfalls: the integral is exact over the constant pieces. A date without a time is 00:00. Lines at or after the cutoff are not scored. Say which start you used when you quote this.

Tool: `score_history` (`questions[].forecasters[].time_avg_brier`).

Worked example (`score_history_one_question_days`): a window from 2026-05-01 to the end of 2026-06-30 (61 days); agent:news says 0.7 for 40 days (Brier 0.09) then 0.92 for 21 days (0.0064): (0.09 x 40 + 0.0064 x 21) / 61 = 0.0612, against a last-line Brier of 0.0064. (`score_history_tool_example_skip`): 0.7 then 0.82 from day 40 gives 0.0702.

#### 9.2 Coverage share and the before-first rule

Formula:

    coverage_share = (cutoff - the later of the first line and the window start) / (cutoff - window start)

Variables: as in 9.1.

Use when: a forecaster who joined late covers less of the window; `skip` averages over what they covered, `half` counts the time before as 50%.

Pitfalls: a late first line can look better under `skip` than an early one; `half` penalizes coming late.

Tool: `score_history` with `before_first`.

Worked example (`score_history_late_first_line_half`): one line of 0.9 on day 20 of a 61-day window: coverage 41 / 61 = 0.6721; with `half`, (0.25 x 20 + 0.01 x 41) / 61 = 0.0887. (`score_history_tool_example_half`): agent:ratings covers 20.75 of 30.75 days on coffee-index; `half` lifts the average from 0.1225 to 0.164 and the overall from 0.1177 to 0.128.

#### 9.3 Lead time

Formula:

    lead_days = (cutoff - the earliest t from which |p(t) - o| <= within_pts / 100 held through to the cutoff) / 1 day

Variables: within_pts 10 by default.

Use when: saying how long before the cutoff the forecast was within 10 points of the outcome and stayed there.

Pitfalls: null when the last line is not within `within_pts` of the outcome.

Tool: `score_history` (`lead_days`, `early_calls`).

Worked example (`score_history_one_question_days`): 0.92 from 2026-06-10 to the cutoff at the start of 2026-07-01: 21 days. (`score_history_tool_example_skip`): agent:ratings' 0.9 from 2026-06-20: 11 days; agent:news' 0.82 is 18 points off: null.

## 10. Numeric and interval forecasts

#### 10.1 Pinball loss and the quantile score

Formula:

    pinball(tau, q, y) = tau (y - q) if y >= q, else (1 - tau)(q - y)
    quantile score = mean pinball over the levels sent;   crps_q = 2 x quantile score

Variables: q the forecast's tau-quantile; y the outcome.

Use when: a forecast is given as a few quantiles.

Pitfalls: crps_q approximates CRPS and depends on the levels sent; more, evenly spread levels bring it closer. Values must not fall as the level rises.

Tool: `score_distribution` with `quantiles`.

Worked example (`pinball_tau_01_q_40_y_55`, `pinball_tau_05_q_50_y_55`, `pinball_tau_09_q_65_y_55`): quantiles 40 (0.1), 50 (0.5) and 65 (0.9) with outcome 55: 1.5, 2.5 and 1 (the code returns 0.9999999999999998); the quantile score is 1.6667 and crps_q 3.3333 (`score_distribution_tool_example`).

#### 10.2 Interval (Winkler) score and coverage

Formula:

    alpha = 1 - level
    score = (hi - lo) + (2 / alpha)(lo - y) if y < lo, + (2 / alpha)(y - hi) if y > hi
    coverage rate = covered / n, with a Wilson interval, against the mean nominal level

Variables: [lo, hi] a central interval at `level`; y the outcome.

Use when: a forecast is a central interval with a stated level.

Pitfalls: the interval score is reported on its own; it is not a CRPS. A few intervals say little about coverage.

Tool: `score_distribution` with `interval`.

Worked example (`interval_score_covered`): [40, 65] at 80% with outcome 55: 25, covered. (`interval_score_missed_above`): outcome 70: 25 + 10 x 5 = 75. (`score_distribution_coverage_wilson`): five 80% intervals, 4 held the outcome: rate 0.8, Wilson [0.3755, 0.9638], consistent with the level; mean score 29.8.

#### 10.3 CRPS of a normal forecast

Formula:

    z = (y - mean) / sd
    crps = sd [z (2 Phi(z) - 1) + 2 phi(z) - 1 / sqrt(pi)]

Variables: Phi the standard normal CDF; phi its density.

Use when: a forecast is a mean and a standard deviation.

Pitfalls: sd must be above 0. The site's Phi uses an erfc accurate to about 1e-7, so these vectors carry a tolerance of 1e-5.

Tool: `score_distribution` with `normal`.

Worked example (`crps_normal_mean_50_sd_8_y_55`): mean 50, sd 8, outcome 55: 3.077203755762034 (shown as 3.0772, z 0.625). (`crps_normal_outcome_at_mean`): outcome 50: 1.869559818040873, which is sd (2 phi(0) - 1 / sqrt(pi)).

#### 10.4 Empirical CRPS for samples

Formula:

    crps = mean |x_i - y| - (1/2) mean over all ordered pairs i, j of |x_i - x_j|
    pair term from the sorted sample: sum over i < j of (x_(j) - x_(i)) = sum_k x_(k) (2k - n - 1)

Variables: x the n equally likely draws; y the outcome.

Use when: a forecast is an ensemble of draws.

Pitfalls: the all-pairs form is slightly harsher on small ensembles than the fair form. 2 to 500 samples.

Tool: `score_distribution` with `samples`.

Worked example (`crps_samples_five`): 41, 47, 52, 58 and 63 with outcome 55: mean |x - y| = 7.2, mean |x_i - x_j| over all ordered pairs = 8.8, crps = 7.2 - 4.4 = 2.8. (`crps_samples_point_mass`): 50 and 50 with outcome 55: 5, the absolute error.

#### 10.5 Multi-option Brier, log loss and the ranked probability score

Formula:

    brier = sum_k (p_k - 1[k = outcome])^2            (0 to 2)
    log loss = -ln(max(p_outcome, floor))
    rps = sum_k (P_k - O_k)^2,  P and O the cumulative sums in option order      (0 to options - 1)

Variables: p_k the probability of option k; 1[.] the indicator.

Use when: a question has several options; RPS when the options are ordered, first to last.

Pitfalls: probabilities must add up to 1 within 2 points (then they are scaled). RPS depends on the order of the options list.

Tool: `score_distribution` with `options` and `probs`.

Worked example (`brier_multi_three_options`, `rps_three_options`, `score_options_draw_happened`): 0.5, 0.3 and 0.2 with the second option happening: Brier 0.25 + 0.49 + 0.04 = 0.78; log loss -ln 0.3 = 1.2039728043259361; RPS (0.5 - 0)^2 + (0.8 - 1)^2 + 0 = 0.29. (`rps_three_options_last_happened`): the last option happening gives 0.89.

#### 10.6 Means with intervals and the lead score

Formula:

    lead = crps if any row has one, else pinball, else interval score
    each score: mean, t interval (6.3, cluster-robust by question when several rows share one), n_effective

Variables: as in 6.3 and 6.4.

Use when: summarizing many numeric forecasts. Every score is a loss in the outcome's units: lower is better.

Pitfalls: no interval under 5 forecasts. Compare forecasters only on the same questions and units.

Tool: `score_distribution` (`overall`, `by_forecaster`).

Worked example (`score_distribution_tool_example`): six forecasts on two questions: mean CRPS 3.7243 over 3, pinball 2.25 over 2 (crps_q 4.5), interval score 25 over 1, absolute error 5.6 over 5 with interval [0, 40.1609] and n_effective 1.9.

## Porting checklist

- Clamp before every logit: 0.001 by default, `floor_pct` / 100 where a tool takes one, 0.01 in `check_coherence`.
- Round only for display, and at the same places: 4 decimals for scores, correlations and intervals, 6 for probabilities in pools and Bayes steps, 2 for points.
- Use n - 1 in standard deviations, G - 1 degrees of freedom with clusters, no intervals under 5 rows, and 10 shared questions before a comparison is tested.
- Ties: average ranks in Spearman and AUC; equal p pooled first in PAV.
- Dates: a date without a time is 00:00; the resolve_by day ends at 23:59:59 plus one second; times without a zone are UTC, or the file's `timezone`.
- Match the digest's JSON form exactly before you trust a cite line.
- Run `vectors.json`: 1e-6 absolute on every number unless the vector says otherwise; prose fields are not vectors.

## Changes

- 2026-10-22, version 1: first release. Sections 1 to 10 match the tools at site version 1.2 (score/2, compare/1, coherence/1, correlate/1, history/1, distribution/1).