Do AI forecasts beat prediction markets?

The experiment

Markets like Polymarket put a real-money price on future events — an implied probability that aggregates thousands of traders. The open question: does a general-purpose AI model add any information beyond that price? Most claims either way are unfalsifiable because nobody keeps receipts. We do:

• Hourly, the AI (gemini-2.5-flash) forecasts a rotating set of active markets: an outcome and a probability.
• Each forecast is stored with the contemporaneous market price — same timestamp, so the comparison is look-ahead-free by construction.
• At resolution, every receipt is scored with the Brier score — (1 − probability assigned to the eventual winner)² — for the AI and for the market price. Lower is better.
• All math is deterministic; no LLM touches the numbers. Unscorable receipts are counted by reason (8480 so far), never silently dropped. Score records are immutable.

Full methodology on the live scoreboard. Brier is computed only for two-outcome markets; a winner requires one outcome priced ≥0.95 at resolution.

Headline result: the market is hard to beat

That is the honest baseline, and it should update your prior about both AI hype and market efficiency: a frontier-family model, prompted with the market's own price, still cannot out-forecast that price on average. But the aggregate hides two findings that are more interesting.

Finding 1 — the AI knows when it knows, except in the middle

At the extremes the AI is well calibrated. In the middle bands it is systematically overconfident — events it rates as likely happen far less often than stated (red). This mirrors a known LLM failure mode: verbal probability estimates cluster toward confident-sounding mid-range values that aren't earned by evidence.

Finding 2 — the edge pocket that died under scrutiny

Our first pass at this breakdown found something exciting: in moderately-favored markets, the AI appeared to beat the price. We briefly reported it as "the AI's one edge pocket." Then we applied the test that kills most published forecasting edges — and it killed ours.

Descriptive breakdown only — see the caveat below before reading the green cell as an edge.

The look-ahead trap: this table buckets by the eventual winner's price — information nobody has at forecast time. "Winner priced 60–80%" silently means "the favorite was priced 60–80% and won", which smuggles the outcome into the conditioning. Re-conditioned on what is actually knowable ex-ante (the favorite's price), the pocket collapses: one sub-band still looks good, its neighbor flips to zero, and the combined band improves Brier by under 2% — the signature of noise, not structure.

This is the same leak that invalidates a large share of published trading "edges", and catching it in our own headline finding is exactly what this project is for. The descriptive table stays; the claim is retired.

The strictest test: sweep every mixture weight

The cleanest single question the data can answer: over all 5575 scored binary forecasts, is there any mixture w·AI + (1−w)·market that beats the market price alone?

This sweep is in-sample; the live blend variants above are the standing out-of-sample check of the same question, and this page updates nightly as both accrue. The strongest claim our data currently supports is the null — which is itself the finding: a frontier-family model, handed the market's own price, cannot improve on it. If that ever changes, this section will say so with receipts.

The self-correction experiment

After the calibration analysis surfaced the mid-range overconfidence, the system deployed corrections along two axes. Deterministic blends mix the AI's probability with the market price (50/50, 25/75) — a dose-response sweep with the raw model as one endpoint. Prompt-v2 (gemini-2.5-flash-cal2) is the more interesting one: the same model re-prompted with its own audited failure modes as hard rules — defer to the price absent a concrete stated reason, don't round uncertainty up to confident-sounding numbers, don't fade extreme prices. The machine trying to fix itself, with receipts. All variants are scored publicly, side by side, on the same markets and hours as the original.

Early result: the blend cuts the raw model's deficit roughly in half. It has not yet crossed the market. The experiment continues automatically; this table is live.

The dose-response curve

The whole experiment in one table: how far each mixture sits from the market's own Brier, ordered by market weight. Out-of-sample, live, updated nightly.

The deficit shrinks monotonically toward zero as market weight rises and never goes negative: every step away from the price costs accuracy, in proportion to the step. That asymptote-from-above IS the null result, drawn as a curve.

Get the data

Why publish losses?

Because unfalsifiable forecasting claims are worthless. The value of this record is that it cannot be curated after the fact: receipts are written before resolution, scores are immutable, and the aggregate includes every forecast the system ever made. When the AI eventually improves, the improvement will be credible for the same reason today's deficit is.

Market-implied odds, not financial advice. Forecasts are experimental model output.