Beyond Mean Squared Error: Rethinking Regression Objectives for Tabular Deep Learning

Date: October 9, 2026
Subject: Breakthrough Research in Tabular Predictive Modeling
Source: Yandex Research / arXiv:2610.13909

The landscape of machine learning is often dominated by breakthroughs in computer vision and large language models, yet the bedrock of enterprise data science remains tabular data. From financial risk assessment to supply chain forecasting, the majority of industrial AI applications rely on structured datasets. Despite this, the mathematical foundation for these models—specifically the objective functions used for regression—has remained surprisingly stagnant. A new paper, “Regression Objectives for Tabular Deep Learning,” released by researchers at Yandex, suggests that the industry’s reliance on the standard Mean Squared Error (MSE) may be masking significant performance potential.

The State of Tabular Regression: Challenging the MSE Status Quo

For decades, Mean Squared Error (MSE) has served as the default objective function for regression tasks. It is mathematically convenient, computationally efficient, and generally effective. However, as deep learning models for tabular data (such as TabNet, FT-Transformer, and MLP-based architectures) have matured, the limitations of MSE have become increasingly apparent.

The core issue, as identified by the researchers, is that MSE assumes a uniform distribution of error, which rarely reflects the reality of complex, real-world tabular data. Real-world data is often heteroskedastic—meaning the variance of the error is not constant across the range of the input variables. While researchers have proposed various alternatives to MSE, these proposals have historically been fragmented, tested on disparate datasets, and evaluated using inconsistent protocols. This fragmentation has left practitioners in the dark regarding which objective functions actually provide a tangible advantage in a production environment.

Chronology of the Investigation

The research team, led by Nikolay Kartashev, embarked on a comprehensive, systematic re-evaluation of regression objectives. Their study, submitted on October 9, 2026, followed a structured three-phase methodology:

Benchmarking the Baseline: The team curated a broad collection of established, publicly available tabular datasets. They then trained multiple neural backbones—ranging from simple MLPs to state-of-the-art transformer-based tabular models—using the standard MSE baseline to establish a control group.
Comparative Analysis of Alternatives: The researchers introduced a wide range of proposed loss functions, including various robust regression objectives, to measure their performance against the MSE baseline. This phase was designed to strip away the "hype" associated with specialized loss functions and isolate their empirical contribution to model accuracy.
Identification of Systematic Limitations: During the evaluation, the team noticed a persistent issue: even when an objective function was theoretically capable of handling heteroskedasticity, the models struggled to reach their full potential. This led to the discovery of the "Early Stopping Conflict"—a phenomenon where different regions of the feature space require different training durations to reach optimal convergence.

Supporting Data and Empirical Findings

The empirical results were striking. The team found that many "novel" loss functions touted in recent literature offered only marginal or inconsistent gains over a properly tuned MSE baseline. In many cases, the complexity introduced by these functions was not justified by the minor, often statistically insignificant, improvements in RMSE (Root Mean Square Error) or MAE (Mean Absolute Error).

However, one exception stood out: Quantile Loss.

The Quantile Advantage

The researchers observed that quantile loss consistently provided aggregate improvements across a diverse range of backbones. By optimizing for specific quantiles rather than the mean, the models were able to capture the conditional distribution of the target variable more effectively. This allows the model to provide not just a point estimate, but a probabilistic range, which is critical for decision-making in high-stakes environments like insurance underwriting or algorithmic trading.

The Heteroskedasticity Gap

Despite the success of quantile loss, the researchers identified a fundamental bottleneck. In datasets with varying levels of target noise—where some regions of the data are inherently more volatile than others—the models exhibited conflicting convergence rates. Specifically:

Low-noise regions: Models converged quickly and began overfitting if training continued.
High-noise regions: Models required longer training periods to capture the underlying patterns effectively.

Because most training regimes use a global early-stopping mechanism, the model is often forced to compromise. It stops training either too early for the high-noise regions (leading to underfitting) or too late for the low-noise regions (leading to overfitting). This "global stopping" constraint effectively limits the model’s ability to express heteroskedastic distributions, even when the underlying objective function is theoretically capable of doing so.

The Proposed Solution: Heteroskedasticity-Aware Label Smoothing

To address the early-stopping conflict, the Yandex team introduced a novel extension of the quantile loss: Input-Dependent Label Smoothing.

Instead of applying a static target to the entire dataset, this method adjusts the label smoothing parameters based on the input features. By acknowledging that different regions of the feature space carry different levels of uncertainty, the model can effectively "dampen" the gradient signals in high-noise areas while remaining sensitive in high-confidence areas.

The results demonstrated that this method outperformed standard quantile loss across several architectures. By providing the model with a mechanism to account for spatial uncertainty, the researchers were able to mitigate the negative impact of global early stopping, allowing the models to converge more effectively across the entire data manifold.

Implications for Data Science and Industry

The implications of this research are far-reaching for both academic research and industrial machine learning deployment.

1. The Myth of the "One-Size-Fits-All" Objective

The study serves as a stern warning against the blind adoption of complex loss functions. For many practitioners, the study suggests that a well-tuned MSE baseline, or a simple quantile loss, is often sufficient. The "marginal gains" reported in many papers may simply be artifacts of specific dataset choices or suboptimal hyperparameter tuning rather than true architectural breakthroughs.

2. The Critical Need for "Local" Training Dynamics

Perhaps the most significant takeaway is the identification of the Early Stopping Conflict. This suggests that the next generation of tabular models should move away from global training parameters. Future research will likely focus on "dynamic" training regimes, where learning rates or stopping criteria are modulated based on local data density or predicted variance.

3. Advancing Uncertainty Estimation

By moving toward heteroskedasticity-aware objectives, companies can build more robust systems. In industries like medicine or finance, knowing how confident a model is in its prediction is often as valuable as the prediction itself. The research provides a practical roadmap for moving from point-estimation models to distribution-aware systems.

Conclusion and Future Outlook

The Yandex Research team has provided a much-needed "sanity check" for the field of tabular deep learning. By systematically dismantling the assumptions surrounding regression objectives, they have clarified that while there is no "magic bullet," there is a clear path forward for more granular, uncertainty-aware modeling.

The release of the accompanying code at https://github.com/yandex-research/tabular-regression ensures that these findings can be integrated into existing pipelines immediately. As the industry continues to refine its approach to tabular data, the lessons learned here—that training dynamics must be as nuanced as the data they process—will likely become the new standard for model development.

Ultimately, this study reminds us that in the race for ever-larger models, the fundamental mathematics of how we teach those models to "learn" remains the most critical variable in the equation. For researchers and engineers alike, the message is clear: stop looking for a more complex objective, and start looking for a more intelligent way to manage the training process.