How every model is tested
The same test for v1, v2 and v3. Freeze the clock at the end of a past year, let the model see only what was published until then, and train it only on outcomes already known. It ranks every crossing of a subfield with a method the subfield does not use yet. Then we count what really happened in the next three years.
One rule, for everyone
A forecast counts as true only if at least three works of that subfield use the method within three years. The same automatic rule judges the model, the popularity bet and chance.
Sealed in advance
Before each confirmation the criteria were written in a document, fingerprinted with SHA-256 and committed. Then the test was run once, with code, vocabulary and filters frozen.
The failures stay
Every year is shown, including the weak ones, and so are the ideas we tried and discarded. The examples below are the first of each list, hits and misses as they are.
Year by year, every model on one chart
Share of the top 100 forecasts on untouched crossings that came true, for each frozen year. v3 reads all of science (OpenAlex); v1 and v2 read physics, mathematics and computer science (arXiv). Popularity: always bet on the most used methods. Chance: a crossing picked at random.
One model, four questions
“AI will reach every field” is true and tells you nothing. So we ask v3 four questions of increasing difficulty: everything; no generative AI; classic methods only; classic methods entering a field where nobody uses them. Each was tested on a year the model had never seen: forecasts made at the end of 2023, judged on 2024–2025, with the rules written down before running.
The higher it ranks, the more often it comes true
Share of forecasts that came true at each depth of the ranking, on untouched crossings. A ranking that knows what it is doing goes down smoothly from the top. It does, for every model and every view.
Three years is our rule. History goes further.
The official test is strict on purpose: at least three works, within three years. A forecast that comes true in year five, or that a first work confirms, still saw the future before anyone else. The same top 100 forecasts on untouched crossings, judged four ways, from the strictest to the most generous.
When the first work arrives
For the top 100 forecasts on untouched crossings, the year in which the first work using the method appeared in the subfield. Most arrive in the very first year after the forecast; very few never arrive.
It works in every domain of science
v3, top 100 forecasts on untouched crossings ranked within each domain. Large: the sealed confirmation on 2017–2022 (three years). Below: the fresh test on 2023 (two years). Further down, the same check for v2 on the areas of arXiv.
v2 on arXiv: top 1,000 of each year by area, 2012–2022
Bright: came true within three years. Soft: at least one work by 2026.
What it saw, what happened
Pick a model and a year. You see what the model had in front of it with the clock frozen, its first forecasts on untouched crossings and the works per year that followed. Nothing is chosen by hand.
Each year is a replay with the clock frozen: the model sees only the past and is trained only on outcomes already known. For v3, 2008–2016 are the design years and 2017–2022 the sealed confirmation, run once; 2023 was a second fresh test for the four views.
v1 and v2 were built and tested on arXiv (151 fields, about 3 million papers); v3 on OpenAlex (252 subfields, 163 million works). Percentages from different libraries are not identical tests, but each model faced the same rule, and each beat chance and popularity by a wide margin.