From Scratch to Scikit: A Deep Dive into the Mechanics of Machine Learning

In an era where high-level libraries like scikit-learn, TensorFlow, and PyTorch abstract away the complexities of mathematical optimization, a growing cohort of developers is returning to the roots of the field. Hariom Lohar, a software engineer and open-source contributor, recently embarked on an ambitious project: recreating the foundational algorithms of machine learning—Logistic Regression, Decision Trees, Random Forests, and K-Means—entirely from scratch using only Python’s standard library.

The resulting repository, titled minisklearn, provides more than just a coding exercise; it serves as a rigorous benchmark comparing manual implementations against industry-standard optimized libraries. The results of his experiment reveal profound insights into how algorithmic design, hyperparameter sensitivity, and mathematical precision dictate performance in real-world scenarios.

The Genesis of the "Mini-Sklearn" Project

The primary motivation behind Lohar’s project was pedagogical: to demystify the "black box" nature of machine learning libraries. By bypassing high-level APIs, a developer is forced to confront the underlying linear algebra, calculus, and logic structures that power predictive modeling.

Lohar chose the iconic Titanic dataset—a staple in the data science community—as his testing ground. With 889 rows and seven distinct features, the dataset offers a manageable but nuanced challenge for binary classification. By holding the testing parameters constant—using a fixed test_ratio of 0.2 and a consistent seed=0—Lohar ensured that any variance in performance could be attributed directly to the algorithmic implementation rather than data shuffling or sample bias.

Chronology: Building the Foundation

The development process was not merely about writing code; it was an iterative cycle of implementation, testing, and debugging. The chronology of the project followed a specific trajectory:

Defining the Metrics: Before building the models, Lohar developed his own confusion-matrix logic. This ensured that the performance metrics (Accuracy, Precision, Recall, and F1-score) were calculated consistently across all models, preventing discrepancies arising from library-specific scoring conventions.
Developing the Classifiers: Lohar implemented the models using standard Python. The challenge was significant, particularly for the Decision Tree and Random Forest, which require complex recursive logic and entropy/Gini impurity calculations to determine optimal data splits.
The Benchmarking Phase: Once the models were operational, they were subjected to the same training data as their scikit-learn counterparts.
Analysis and Troubleshooting: Upon observing the initial performance gaps, the final phase involved a forensic analysis of why the custom models were underperforming compared to the highly optimized scikit-learn implementations.

Supporting Data: The Discrepancy in Performance

The quantitative results of this experiment highlight the massive chasm between a baseline implementation and a production-ready, highly optimized library.

Comparative Performance Table

Model	Custom Implementation (Acc/P/R/F1)	Scikit-Learn (Acc/P/R/F1)	Performance Gap
Logistic Regression	0.684 / 0.714 / 0.294 / 0.417	0.847 / 0.806 / 0.794 / 0.800	0.164
Decision Tree	0.706 / 0.682 / 0.441 / 0.536	0.780 / 0.716 / 0.706 / 0.711	0.073
Random Forest	0.695 / 0.652 / 0.441 / 0.526	0.831 / 0.828 / 0.706 / 0.762	0.136
K-Means	Inertia: 742,754.6	Inertia: 742,754.6	0.0

The K-Means Anomaly

The most striking finding was the K-Means result. While the supervised classifiers showed significant performance gaps, the K-Means clustering algorithm matched the scikit-learn inertia (the sum of squared distances of samples to their closest cluster center) to the exact decimal. This suggests that the "Lloyd’s algorithm"—the standard iterative approach to K-Means—is mathematically robust and easier to implement correctly from scratch than the nuanced optimization paths required for regression and tree-based classification.

Implications: Why the Classifiers "Failed"

The systematic under-prediction of "survived" cases in the custom classifiers—evidenced by significantly lower recall scores—provides a masterclass in why modern libraries are so difficult to replicate.

1. The Sensitivity of Hyperparameters

Lohar noted that his models suffered from "too-tiny learning rates." In scikit-learn, hyperparameters are often tuned through sophisticated solvers (like L-BFGS or Liblinear) that automatically adjust step sizes and handle regularization. Manually implementing these optimization loops requires a deep understanding of gradient descent, which is notoriously difficult to calibrate without automated adaptive learning rate schedulers.

I built Logistic Regression, Decision Trees, Random Forests, and K-Means from scratch :.. here's what surprised me

2. Feature Scaling

The Titanic dataset contains a mix of categorical and continuous variables. scikit-learn pipelines often incorporate implicit feature scaling (such as StandardScaling or Min-Max normalization). Lohar’s custom implementation likely lacked the robust scaling required to handle the variance in raw input data, leading to skewed weights in the logistic regression model.

3. Tree Fragmentation

In the Decision Tree implementation, the issue of "fragmenting continuous inputs" suggests that the custom algorithm lacked efficient binning strategies or pruning mechanisms. Without proper pruning, trees become overfit to training noise and fail to generalize, explaining why the recall for the "survived" class was significantly hindered.

Professional Perspectives on "From-Scratch" Learning

Industry experts often argue that building models from scratch is the single most effective way to transition from a "user" of AI to an "architect" of AI. When a developer understands the mechanics of entropy or the nuances of the sigmoid function, they gain a tactical advantage when debugging production models that exhibit "drift" or poor convergence.

Lohar’s conclusion is a measured one: while he acknowledges that these foundational models are not always the answer, he advocates for their use in specific scenarios where massive neural networks may be overkill. "There are problems that can’t be solved by these tiny models," he noted, "but they perform surprisingly well on specific cases where neural networks struggle."

This sentiment echoes a growing trend in "Small Data" movements. In scenarios with limited compute resources, high interpretability requirements, or very small datasets, the traditional machine learning algorithms—if implemented and tuned correctly—often outperform the high-variance, data-hungry deep learning models currently dominating the discourse.

Future Outlook: A Shift in Methodology

For developers looking to emulate Lohar’s project, the lesson is clear: the path to mastery is paved with the realization that libraries are not magic. They are collections of mathematical choices.

Moving forward, Lohar suggests that he will prioritize these foundational models for initial problem-solving before gravitating toward neural networks. By establishing a "baseline of simplicity," developers can determine whether a complex deep learning architecture is actually required or if a well-tuned Decision Tree or Logistic Regression model is sufficient.

The minisklearn repository stands as a testament to the fact that while we stand on the shoulders of giants like the scikit-learn core team, we must occasionally climb down to the ground level to understand the terrain. Whether it is matching K-Means inertia to the decimal or debugging the subtle failures of a custom-built Random Forest, the process of building from scratch remains the ultimate crucible for any serious data scientist.

As the machine learning industry continues to evolve toward higher abstractions, projects like this serve as a vital reminder: the beauty of the field lies not in the final prediction, but in the mathematical rigor that makes that prediction possible.