Over five days we tested several approaches to finding trading signals. We rejected three candidates, had to re-check one because of a data error, and one more did not pass validation for statistical reasons.
The main result of the week is not a new trading model. We do not yet have a candidate that gives sufficient grounds to use its signals. But we now understand better which approaches fail validation, and why.
Training a model and getting predictions does not yet prove that those predictions are fit for financial decisions.
Four different outcomes that must not be mixed up
This week made it clear that a single “the model failed” status is too coarse. It hides different situations, and they call for different actions.
Invalid validation — the result cannot be used because of an error in the data or the procedure.
Insufficient evidence — an advantage has not been confirmed with the required confidence.
No admissible option — none of the tested ways of applying the model meets the admission conditions.
Candidate rejected — the model failed validation and is closed for the current cycle.
A data error means: fix it and re-check. Insufficient evidence is not the same as a proven absence of signal. No admissible option is a negative verdict on fitness for use, not on the statistics. And a rejected candidate should not be brought back into work just because the result is inconvenient.
Below is how these four outcomes looked in practice.
From evaluating ready-made signals to finding them
Previously, a large part of our ML research was about evaluating trading candidates that had already been formed. A separate component identified a market situation, and an ML model then helped assess its quality.
The new direction asks a different question: can a model detect a trading opportunity directly from the state of the market, without a pre-formed candidate?
This is a different task. Results from a model that evaluates ready-made signals cannot be automatically transferred to a model that has to find them on its own.
We split the work into two scales — Intra (more frequent decisions over a short horizon) and Swing (less frequent decisions over a longer one). Each is validated separately: success or failure on one scale is not treated as evidence for the other.
The research set includes simple linear models, rule-extraction methods, gradient boosting and other approaches. The goal is not to pick the most complex algorithm, but to understand which approach provides convincing evidence of usefulness.
Step one: define what exactly the model predicts
The first question was the training target.
You can predict the direction of the market. You can estimate the potential return of a move. You can predict whether a trading target will be reached. These are different tasks, and success in one does not guarantee success in another.
For the new direction we chose to estimate the potential outcome of a market move over a fixed horizon, net of trading costs.
At the same time, we separated two things: the outcome the model learns to predict, and the outcome an actual trading decision would produce once execution and position-management rules are applied.
This distinction proved material already in the preliminary check. The data contained cases where the direction of a move looked attractive, but a possible position would have closed differently because of its path. The market may end up higher, but first go through an adverse move that would have knocked the position out.
A correctly predicted direction does not necessarily mean a successful trade.
An error in data preparation
An error occurred during the initial preparation of the research data. Some checks formally completed successfully even though the data contained no usable observations. The technical status “done” did not reflect whether the result was actually correct.
We did not accept that dataset. We fixed the error, added a check that genuinely admissible observations are present, and rebuilt the data. The corrected result passed a repeat check.
The lesson is not that a pipeline can contain an error — that is well known. The more important conclusion is that a check must confirm the presence of a correct result, not just the absence of a software exception. If every observation has become unusable, a program finishing successfully must not turn into a positive research verdict.
First candidate: a simple linear model
The first practical candidate was a linear model for Intra.
This was a deliberate choice. A simple model helps check whether the data contains a stable signal before moving on to more complex algorithms. It also sets a point of comparison: a future complex model must show value beyond what a simple method can detect.
The research cycle included data preparation, training and validation on a separate time period. The first validation ran into the same data-processing error; its result was declared invalid and the run was repeated after the fix.
The outcome is negative: for the linear Intra model we could not find an admissible decision option that meets the selection conditions.
This does not necessarily mean the model detects no statistical patterns. But between detecting a pattern and being allowed to use it for a trading decision there is an additional boundary. The linear Intra candidate did not cross it.
Second candidate: a linear model for Swing
We then tested the same task on a longer scale. The reason for the negative result turned out to be different.
The linear Swing model completed validation, but the evidence was insufficient for a positive decision.
Here it is important to distinguish two outcomes. In the first, the model has been validated, but no admissible way of using it meets the requirements. In the second, there is not enough data to confirm an advantage with confidence.
These are not the same. In the first case, the procedure leads to a negative decision on fitness for use. In the second, statistical uncertainty remains: the available data gives no grounds for admission, but on its own does not prove that a useful signal is entirely absent.
We got exactly the second result. The candidate was not admitted for further use, but we do not turn insufficient evidence into a claim that there are no patterns at all, nor into a reason to declare the model a success.
Third approach: rule-based models
The next direction was rule-based models — methods that identify combinations of conditions associated with a particular outcome, instead of a fully opaque prediction.
This is potentially useful for analysing market behaviour: you can see which patterns the algorithm finds. But interpretability is not reliability. A rule can look economically sensible, be easy to explain, and still have no stable statistical advantage.
So we applied the same validation principles to these models as to the linear ones.
The results were again negative, but for different reasons. One candidate was rejected at the stage that assesses whether the statistical evidence is sufficient. The second did not gather enough evidence for admission.
Both cycles are closed. These results do not prove that rule-based methods are useless in general — they apply to the specific candidates tested and the conditions that were set. But they show why an interpretable model cannot be considered more reliable just because its decisions are easier to explain. Explainability helps you study a model. It does not replace testing its predictive and economic value.
Moving to more complex models
After linear approaches and rule-based models, we moved on to structural gradient boosting — a method that can describe non-linear dependencies and feature interactions.
This extends the algorithm's expressive power, but creates new risks: the model can find complex patterns that do not hold on other time periods. That is why added complexity is not, by itself, considered progress.
For Intra, the research configuration was prepared and accepted, and then data preparation was carried out. For Swing, by the end of the week we had the result of a separate structural boosting validation: the evidence was insufficient for admission.
Moving to a more complex method did not automatically remove the problem the previous candidates had. Work on the remaining approaches continues. Their future results cannot be assumed in advance to be either positive or negative.
Why we do not retrain every failed model until it gives a positive result
After a negative result there is a natural urge to change the settings, pick different features, adjust the selection conditions, try a few more variants. Sometimes such changes are justified.
But there is a danger: if the research is continually adapted to the same validation period, that period gradually becomes part of training. Formally, the model never receives those observations as training data — but the researcher uses the validation results to choose the next steps. This is how overfitting arises at the level of the whole process.
That is why we separate development, model selection and the subsequent independent validation. A sequential admission procedure applies to new candidates. A failed result does not grant automatic permission to keep changing the model until a suitable metric appears.
In financial machine learning this is especially important: weak patterns, noise and a large number of tuning options easily produce results that are convincing but unstable.
How this connects to our separate article
We have also prepared a separate article for HackerNoon — “When a Trading AI Chooses Silence: Caution or an Illusion of Accuracy?” It is currently in editorial review; we will add the link here once it is published.
It deals with a different but related problem: a model can show very high accuracy if it declines to predict in difficult situations. Research shows that this kind of metric improvement does not always mean that a more reliable prediction has appeared.
More detail will be in that article.
What changed this week
First, the new direction of ML research moved from design to practical validation.
Second, negative results for several candidates were obtained and recorded, and the reasons for non-admission differed.
Third, a data-preparation error was found and fixed, and the checks were strengthened so that a formally successful but actually unusable result is not accepted.
Fourth, validation of more complex models began — without assuming that added complexity will automatically produce a better result.
What comes next
We continue to test the remaining approaches under the established procedure.
At this point, the results of the candidates described here give no grounds to declare a new trading model ready for use. The profitability of the new direction has not been confirmed either. This does not mean the research is stopping — it means that preliminary expectations do not substitute for evidence.
Over five days we did not get grounds to present a new model as successful. What we did get is a more precise understanding of which approaches are not yet passing validation, why that happens, and where research discipline must be maintained.
The work is not finished. Model training continues, and we are confident that among the remaining approaches there will be a candidate that passes validation and provides grounds for use.