A trading AI model can have an impressive testing history and a very different result once it starts working on new data. Yet even that gap is easy to misread.
At the end of September 2026, the paper Can AI Make Money in Crypto? Measuring the Gap from Backtests to Real Markets was published. Its value is not yet another ranking of algorithms. The authors compared how AI trading methods perform at three stages: historical backtesting, virtual trading on real exchange infrastructure, and trading with real money.
Three stages, three different questions
A backtest shows how an algorithm would have behaved on past data. It is useful for a first assessment of an idea, but it offers no protection against overfitting: if you keep adjusting the model while looking at the validation period, you end up tuning it to the past.
Paper trading, or virtual trading, tests whether a pattern survives when the future is still unknown. It is a more realistic check, but the capital remains virtual and real execution conditions are only partly reproduced.
Real trades add the actual execution price, liquidity and slippage. Even if the algorithm itself has not changed, the result can.
Each stage tests a new set of assumptions. Success at one stage does not guarantee success at the next.
What the researchers found
The authors examined 32 methods: classical machine learning, reinforcement learning, large language models (LLMs) and trading agents.
On historical data for 2025, 23 of the 24 ML methods delivered positive returns. The other families did not.
Methods were then selected for virtual trading from June 4 to September 21, 2026, and the picture reversed: of the eight ML methods, only two stayed profitable, while all six methods from the other families finished the period in the black.
It is tempting to draw a headline conclusion here: “classical ML loses to LLMs.” That would be premature. The backtest and virtual-trading periods are different, and the second stage only included methods that had already performed well in the first. A table like this mixes three factors: a change in the market, a selection effect and execution conditions.
The key comparison: the same period
To remove at least the “different markets” factor, the authors compared backtesting and virtual trading for 14 methods over one and the same interval — June 4 to September 21, 2026.
| Method | Backtest | Paper trading |
|---|---|---|
| XGBoost | +33.95% | +27.53% |
| QWEN | +11.35% | +37.77% |
For XGBoost, virtual trading came out worse than the simulation. For QWEN it was the other way round — 26.42 percentage points better. What is more, 10 of the 14 methods achieved a higher result in paper trading than in the backtest.
This is an important detail: the problem is not that backtests always overstate. They can be wrong in either direction — on returns and on risk alike.
Why paper trading is not yet the real market
Four methods made it to real money. They were tested from August 24 to September 21, 2026.
Only one of the four made a profit. And none of them outperformed simply holding Bitcoin over the same period.
Execution is especially telling: the average price deviation rose from 0.31‰ in virtual trading to 1.86‰ in live trading — almost six times as much.
But the conclusions have to be kept in proportion. Four methods and four weeks are far too few to declare an entire family of algorithms unfit. And beating Bitcoin is not a universal yardstick: strategies differ in their risks and objectives.
Another problem: how participants are selected
If only the historical winners from a large pool of algorithms advance to the next stage, there is a risk of bias. Some of them may have won not because of a durable pattern but because of a lucky combination of market conditions. On re-testing, that edge can disappear.
The problem has been known for a long time. Back in 2010, Gavin Cawley and Nicola Talbot showed in On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation that overfitting happens not only when a model is trained but also when it is selected by a metric. That is why it is not enough to control training — the mechanism for picking winners has to be controlled as well.
How this relates to Tantoryn AI
For us, this is a practical question. In the Tantoryn AI project, ML model research, the evaluation of trading candidates and the operation of the virtual trading loop are kept separate.
In our report for October 5–9 we described the first results of our search for standalone signals based on market state. Several candidates did not pass validation, and an error in data preparation was found and corrected. We did not present those results as proof that a model is ready to trade.
The current public trading loop runs on virtual trades. Real orders are disabled. Its observations therefore cannot be treated as proven returns on real capital. Conversely, the failure of an individual research candidate does not by itself define the quality of the whole loop — these are different objects of evaluation.
The new external study neither confirms nor refutes Tantoryn AI's models. It was carried out by other authors, on other methods and in a different environment. For us, its value lies in a more precise question: what exactly must each next level of validation prove before stronger claims can be based on it?
How to read claims about successful trading AI
When a developer says a model has shown high accuracy or profit, looking at the headline number is not enough.
You need to understand when the model and its parameters were chosen, whether test results were used for later changes, and how independent the final check was. And whether the conditions are comparable: the market period, costs and execution rules.
Above all, distinguish the status of the evidence. Backtesting, virtual trading and real trades are not three names for the same test. Each stage adds new assumptions and has its own limitations.
A strong result is not just a high metric. It is the ability to explain precisely what that metric proves, under which conditions it was obtained and which questions remain open.
This article is about research methodology. It is not investment advice and makes no claims about the profitability of Tantoryn AI.
Sources
1. Xingtong Yu et al. Can AI Make Money in Crypto? Measuring the Gap from Backtests to Real Markets. September 28, 2026. Scientific preprint. The main source of the numerical results and of the description of the three validation stages.
2. Gavin Cawley, Nicola Talbot. On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation. Journal of Machine Learning Research, 2010. A study of selection bias in model selection.
3. Tantoryn AI: Five Days of ML Research — Errors, Checks and the First Negative Results. October 10, 2026. The project's official report on its research work.
4. Tantoryn AI official website. The public status of the project and of its virtual trading loop.