VoxOdds Research · updated nightly · live experiment
Do AI forecasts beat prediction markets?
Every hour, an AI model forecasts real-money prediction markets. We store each forecast next to the market price captured at the same instant — an append-only receipt — and score both at resolution. No cherry-picking, no look-ahead, losses published. This page renders directly from the live results: 2104 scored forecasts across 337 markets so far, 2527 more awaiting resolution.
The experiment
Markets like Polymarket put a real-money price on future events — an implied probability that aggregates thousands of traders. The open question: does a general-purpose AI model add any information beyond that price? Most claims either way are unfalsifiable because nobody keeps receipts. We do:
- Hourly, the AI (gemini-2.5-flash) forecasts a rotating set of active markets: an outcome and a probability.
- Each forecast is stored with the contemporaneous market price — same timestamp, so the comparison is look-ahead-free by construction.
- At resolution, every receipt is scored with the Brier score — (1 − probability assigned to the eventual winner)² — for the AI and for the market price. Lower is better.
- All math is deterministic; no LLM touches the numbers. Unscorable receipts are counted by reason (6 so far), never silently dropped. Score records are immutable.
Full methodology on the live scoreboard. Brier is computed only for two-outcome markets; a winner requires one outcome priced ≥0.95 at resolution.
Headline result: the market is hard to beat
AI Brier 0.190 vs market Brier 0.147 across 2104 scored forecasts.
The AI is right 74% of the time (market: 81%) and is the closer forecaster on 61% of individual forecasts — but when it misses, it misses bigger. On average, the price wins.
That is the honest baseline, and it should update your prior about both AI hype and market efficiency: a frontier-family model, prompted with the market's own price, still cannot out-forecast that price on average. But the aggregate hides two findings that are more interesting.
Finding 1 — the AI knows when it knows, except in the middle
| AI says | n | Actually happens | AI Brier | Market Brier |
|---|---|---|---|---|
| 20-30% | 1 | 100% | 0.640 | 0.922 |
| 50-60% | 89 | 70% | 0.247 | 0.124 |
| 60-70% | 264 | 46% | 0.259 | 0.230 |
| 70-80% | 463 | 67% | 0.219 | 0.167 |
| 80-90% | 462 | 52% | 0.308 | 0.270 |
| 90-100% | 825 | 89% | 0.078 | 0.042 |
At the extremes the AI is well calibrated. In the middle bands it is systematically overconfident — events it rates as likely happen far less often than stated (red). This mirrors a known LLM failure mode: verbal probability estimates cluster toward confident-sounding mid-range values that aren't earned by evidence.
Finding 2 — the one pocket where the AI wins
| Market line (winner's price at forecast time) | n | AI Brier | Market Brier | AI edge |
|---|---|---|---|---|
| mkt 95%+ | 514 | 0.054 | 0.001 | -0.053 |
| mkt 80-95% | 516 | 0.057 | 0.018 | -0.039 |
| mkt 60-80% | 394 | 0.083 | 0.098 | +0.016 |
| mkt 40-60% | 353 | 0.325 | 0.205 | -0.119 |
| mkt <40% (upset) | 327 | 0.597 | 0.576 | -0.021 |
The AI's one edge pocket: mkt 60-80% markets, where its Brier 0.083 beat the market's 0.098 across 394 forecasts. Everywhere else the price wins — near-certainties are priced almost perfectly, and coin-flips beat the AI's overconfident mid-range calls. Moderately-favored territory appears to be where markets leave the most residual uncertainty for a model to exploit.
The self-correction experiment
After the calibration analysis surfaced the mid-range overconfidence, the system deployed a deterministic correction: a second model variant that blends the AI's probability 50/50 with the market price at forecast time. Both variants are scored publicly, side by side. The question it answers: does the AI add information beyond the price? If the blend's Brier drops below the market's, the answer is yes.
| Model | n | Brier | Market Brier | Deficit vs market |
|---|---|---|---|---|
| blend-mkt50-v1 | 444 | 0.181 | 0.158 | +0.023 |
| gemini-2.5-flash | 2104 | 0.190 | 0.147 | +0.043 |
Early result: the blend cuts the raw model's deficit roughly in half. It has not yet crossed the market. The experiment continues automatically; this table is live.
Get the data
Everything on this page is reproducible from the raw score records (CC BY 4.0, cite VoxOdds):
- Full dataset (CSV) — one row per scored forecast: prediction, contemporaneous price, resolution, Brier scores, model variant.
- Rollup JSON · Live scoreboard · MCP tool
get_track_recordatvoxodds.com/mcpfor AI agents.
Why publish losses?
Because unfalsifiable forecasting claims are worthless. The value of this record is that it cannot be curated after the fact: receipts are written before resolution, scores are immutable, and the aggregate includes every forecast the system ever made. When the AI eventually improves, the improvement will be credible for the same reason today's deficit is.
Market-implied odds, not financial advice. Forecasts are experimental model output.