A Field Guide to Machine Learning Algorithms: Assumptions, Failure Modes and When We Use Each

Choosing an algorithm is less about leaderboards than about assumptions. Every model encodes beliefs about the data: that effects add up, that similar points have similar labels, that clusters are round, that the future looks like the past. When those beliefs match the problem, simple models win. When they do not, no amount of tuning helps. This guide covers the algorithms we use most in client work, with the maths that matters, the assumptions each makes, how each fails, and when we reach for it.

How should you think about model choice?

Three ideas frame every choice below.

Bias–variance. Expected squared error decomposes into bias² + variance + irreducible noise. High-bias models (linear models, shallow trees) underfit complex structure. High-variance models (deep trees, 1-nearest-neighbour) fit noise. Regularisation, ensembling and more data all move you along this trade-off.

Inductive bias. With finite data, a model can only generalise by preferring some functions over others. Linear models prefer additive effects. Trees prefer axis-aligned interactions. k-NN prefers local smoothness. Convolutional networks prefer translation-invariant patterns. Pick the bias that matches the problem's structure.

The decision, not the metric. Optimise what the business acts on: precision at a fixed review capacity, expected cost, calibrated probability, ranking quality. Then choose the simplest model that achieves it, because simple models are cheaper to run, explain and maintain.

Linear and logistic regression

What it is. Linear regression predicts $\hat y = w^\top x + b$ by minimising squared error. Logistic regression models $P(y=1 \mid x) = \sigma(w^\top x + b)$ and minimises log-loss. It is a linear model in log-odds space.

Regularisation is what makes it practical:

$$\min_w \; \frac{1}{n}\sum_i L(y_i, w^\top x_i) + \lambda\left(\alpha \lVert w\rVert_1 + \tfrac{1-\alpha}{2}\lVert w\rVert_2^2\right)$$

L2 (ridge) shrinks correlated coefficients together and stabilises the solution. L1 (lasso) drives coefficients to exactly zero, which gives feature selection. Elastic net ($0<\alpha<1$) combines both, and behaves better than lasso when features are correlated. Always standardise features before regularising. The penalty is scale-dependent, so unscaled features are penalised unequally.

Assumptions. Effects are additive and linear in the (transformed) features. For inference rather than prediction: independent errors and no severe multicollinearity.

Failure modes. Missed interactions and non-linearities, unless you engineer them (splines, binning, crosses). Unstable, uninterpretable coefficients under multicollinearity. Perfect separation in logistic regression, where coefficients diverge without regularisation.

When we use it. As the baseline for every tabular problem. Also for credit scorecards and other regulated decisions that need transparent, monotone and auditable effects; on very high-dimensional sparse data such as bag-of-words; and when calibrated probabilities matter. Logistic regression tends to be well calibrated when its assumptions roughly hold and it is not heavily regularised.

k-nearest neighbours

What it is. Predict from the labels of the $k$ closest training points under a distance metric. There is no training, only a stored dataset and an index.

Assumptions. Points that are close in feature space have similar labels, and the distance metric captures "close".

Failure modes. The curse of dimensionality: in high dimensions, distances concentrate, so the nearest and farthest neighbours become almost equally far and "nearest" loses meaning. It is sensitive to feature scaling and irrelevant features. Prediction cost grows with the dataset unless you use approximate nearest-neighbour indexes (HNSW, IVF).

When we use it. Rarely as a classifier on raw tabular features. Constantly in its modern form: nearest-neighbour search over learned embeddings. That powers semantic search, retrieval-augmented generation, deduplication and "similar items" recommendations. The algorithm is the same; good embeddings make the distance meaningful.

Support vector machines

What it is. A maximum-margin classifier. The soft-margin primal problem is:

$$\min_{w,b}\; \tfrac12 \lVert w\rVert^2 + C\sum_i \max\left(0,\, 1 - y_i (w^\top x_i + b)\right)$$

The hinge loss ignores points that are correctly classified beyond the margin, so the solution depends only on the support vectors. The kernel trick replaces dot products with a kernel $k(x, x')$, such as the RBF kernel $\exp(-\gamma\lVert x-x'\rVert^2)$. That fits non-linear boundaries without computing the feature map explicitly.

Failure modes. Kernel SVM training scales roughly quadratically to cubically with the number of rows, which is impractical beyond a few hundred thousand. Its outputs are margins, not probabilities, and need calibration (Platt scaling). It is sensitive to $C$, $\gamma$ and feature scaling.

When we use it. Linear SVMs remain strong for high-dimensional sparse text classification. Kernel SVMs fit small, clean datasets with complex boundaries. For most tabular work, gradient boosting has replaced them.

Decision trees and random forests

A single tree recursively splits the feature space to maximise purity gain (Gini impurity or entropy for classification, variance reduction for regression). It is interpretable but high-variance: small data changes produce different trees.

A random forest averages many deep trees, each trained on a bootstrap sample and choosing each split from a random subset of features (max_features). Averaging $B$ trees with pairwise correlation $\rho$ and variance $\sigma^2$ gives an ensemble variance of:

$$\rho\,\sigma^2 + \frac{1-\rho}{B}\,\sigma^2$$

Adding trees removes the second term. Feature subsampling reduces $\rho$, attacking the first. That is the whole idea. Rows left out of each bootstrap give a free out-of-bag error estimate.

Strengths. Robust defaults, little tuning, resistance to overfitting as trees are added, and easy parallelism.

Failure modes. Impurity-based importance is biased towards high-cardinality and continuous features, so use permutation importance or SHAP. Forests cannot extrapolate beyond the training target range. With careful tuning, gradient boosting usually scores higher.

When we use it. As a fast, robust second baseline; when we need a good model with almost no tuning; and as a building block (isolation forests, below). For maximum accuracy on tabular data we move to gradient boosting, covered in depth in Gradient-Boosted Trees from First Principles.

Neural networks for tabular data

Deep learning dominates images, audio and language. On typical business tables, carefully tuned GBDTs remain very hard to beat; benchmark studies have repeatedly found tree ensembles ahead on medium-sized tabular datasets. Neural networks earn their place on tabular problems when:

• there are very large datasets with high-cardinality categorical features, using learned embeddings per category;
• the model must combine tables with text, images or sequences in one architecture;
• the task needs multi-task outputs or end-to-end training with other components;
• event histories are long, and sequence models over the history outperform hand-built aggregates.

A pragmatic hybrid: use pretrained models to turn unstructured inputs into embeddings, and feed those embeddings into a GBDT alongside the tabular features.

k-means and clustering

What it is. k-means minimises within-cluster squared distance:

$$\min_{C_1..C_k} \sum_{j=1}^{k} \sum_{x \in C_j} \lVert x - \mu_j \rVert^2$$

It alternates assignment and centroid updates (Lloyd's algorithm). Use k-means++ initialisation and several restarts, because the objective is non-convex.

Assumptions. Clusters are roughly spherical, similar in size and density, and well described by their mean in Euclidean space.

Failure modes. Elongated or nested clusters, very different densities, outliers that drag centroids, and categorical data, where you need k-prototypes or a different distance. Choosing $k$ is a judgement. The elbow method, silhouette scores and business usefulness all inform it.

Alternatives. DBSCAN / HDBSCAN find arbitrarily shaped, density-based clusters and label noise points. Gaussian mixture models give soft, elliptical clusters with probabilities.

When we use it. Customer and behavioural segmentation, typically on standardised RFM-style features (recency, frequency, monetary value) or on embeddings. The output is validated with marketing teams: a statistically tidy segment nobody can act on has no value.

Isolation forest for anomaly detection

What it is. Build random trees by picking a random feature and a random split value at each node. Anomalies are few and different, so they get isolated in fewer splits. The anomaly score is based on the average path length across trees, normalised by the expected path length for a dataset of that size.

Why it works well in practice. It needs no distance metric or density estimate, scales linearly, and copes with high dimensions better than distance-based methods.

Failure modes. The contamination parameter (expected anomaly share) sets the threshold, and it is often a guess. Anomalies that are only unusual in combination, with each feature normal on its own, can be missed. And "unusual" is not "bad": anomaly scores need review workflows and feedback before they become automated decisions.

When we use it. Transaction and claims screening, data quality monitoring (unusual pipeline volumes or value distributions), and IoT telemetry. It is usually a first stage that ranks cases for human review, whose labels later train a supervised model.

Time-series forecasting

Forecasting has its own failure modes: trends, seasonality, holidays, structural breaks, and the fact that you can only validate on the future.

• Statistical models (exponential smoothing / ETS, ARIMA and seasonal ARIMA) model each series' level, trend and seasonality explicitly. They are strong with few, long, regular series, and they give principled prediction intervals.
• Global machine-learning models. One gradient-boosting model trained across all series, using lag features, rolling statistics, calendar and promotion features, and series identifiers. It shares strength across thousands of related series (stores × products), and often wins in retail and demand planning.
• Deep and foundation models for forecasting are maturing fast. They are worth benchmarking for large portfolios of series, but they need the same rigorous back-testing.

Validation is the part people get wrong. Use rolling-origin back-testing: train up to time $t$, forecast $t+1 \dots t+h$, move $t$ forward and repeat. Compute every lag and rolling feature only from data available at each forecast origin. Report errors per horizon and per segment, not only overall, and compare against a seasonal naive baseline. A surprising number of sophisticated models fail to beat "same as last year".

Recommender systems: matrix factorisation

What it is. Approximate the sparse user–item interaction matrix $R$ as $R \approx U V^\top$, with low-dimensional user and item factors. For implicit feedback (clicks, views, purchases, with no explicit ratings), alternating least squares with confidence weighting (Hu, Koren and Volinsky, 2008) is the classic approach. Spark MLlib's ALS supports it with implicitPrefs=True.

Failure modes. Cold start: new users and items have no factors. Popularity bias. Feedback loops, where the model recommends what it already recommended.

When we use it. As a strong candidate generator, combined with content-based features for cold start and a re-ranking model (often gradient boosting) that brings in context, business rules and diversity. Offline metrics (precision@k, NDCG) must be confirmed with online A/B tests, because offline gains do not always transfer.

A decision table

Situation	Start with	Then try
Tabular classification or regression	Regularised logistic or linear regression	Gradient-boosted trees
Regulated decision needing transparency	Logistic regression or GAM with monotone effects	GBDT with monotone constraints + SHAP
High-dimensional sparse text	Linear model or linear SVM	Fine-tuned transformer
Similarity search, deduplication, RAG	Embeddings + approximate nearest neighbours	Re-ranking model
Segmentation	k-means on standardised features	HDBSCAN or GMM; embeddings
Unlabelled anomaly screening	Isolation forest	Supervised model once labels exist
Few long regular time series	ETS / ARIMA	Global GBDT with lag features
Many related time series	Global GBDT with lag and calendar features	Deep / foundation forecasting models
Implicit-feedback recommendations	ALS matrix factorisation	Two-stage retrieval + GBDT re-ranker

What matters more than the algorithm?

In our experience, the same four things decide success far more often than the algorithm:

1. The target definition. Predicting the thing the business will actually act on, at the moment it acts.
2. Leak-free, point-in-time-correct features, built once and reused for training and serving.
3. Validation that mirrors deployment: time-based, group-aware, with an untouched test set.
4. Operations: monitoring, retraining and a clear owner. See how we productionise data and AI.

Get those right and a well-regularised baseline often reaches most of the achievable value. Get them wrong and the most sophisticated model will fail quietly in production.

References

• T. Hastie, R. Tibshirani and J. Friedman, The Elements of Statistical Learning, 2nd ed., Springer, 2009.
• L. Breiman, Random Forests, Machine Learning, 2001.
• F. T. Liu, K. M. Ting and Z.-H. Zhou, Isolation Forest, ICDM 2008.
• Y. Hu, Y. Koren and C. Volinsky, Collaborative Filtering for Implicit Feedback Datasets, ICDM 2008.
• R. J. Hyndman and G. Athanasopoulos, Forecasting: Principles and Practice, 3rd ed., OTexts.
• L. Grinsztajn, E. Oyallon and G. Varoquaux, Why do tree-based models still outperform deep learning on tabular data?, NeurIPS Datasets and Benchmarks, 2022.