At the heart of every machine learning algorithm lies a cost function (also called a loss function or objective function). It is the compass that guides learning: a mathematical measure of how far the model's predictions are from the true targets. Closely related to cost functions are distance metrics, which quantify the similarity or dissimilarity between data points. While cost functions are used to train models, distance metrics are used for clustering, retrieval, and as building blocks for many cost functions. This article presents a comprehensive taxonomy of both cost functions and distance metrics, from the simplest Euclidean distance to complex, task-specific objectives used in state-of-the-art deep learning, with real-world examples and references throughout.
What are Cost Functions and Distance Metrics?
A cost function is a function that maps a set of predictions and corresponding ground-truth labels to a scalar value that represents the "cost" or "error" of the model. The learning algorithm then adjusts the model's parameters to minimize this cost. A distance metric, on the other hand, is a function that quantifies the similarity or dissimilarity between two data points. Many cost functions are built on top of distance metrics; for example, the Mean Squared Error is the average of squared Euclidean distances between predictions and targets.
The diagram in Fig 1 illustrates the relationship between distance metrics and cost functions in the learning pipeline. Distance metrics provide the raw similarity measures, while cost functions aggregate these measures to produce a scalar error that can be optimized.
This diagram illustrates the fundamental relationship between distance metrics and cost functions. The input data flows through the model (with parameters θ) to produce a prediction ŷ. The distance metric measures the dissimilarity between the prediction and the true target y. The cost function aggregates these distance measures across all samples to produce a scalar error. This error is then backpropagated through the model, and the gradients are used to update the parameters in a direction that reduces the cost. The choice of distance metric determines what "similarity" means for the task, while the cost function determines how these similarities are aggregated and optimized.
The mathematical backbone of both cost functions and distance metrics is rooted in metric spaces and information theory. A proper distance metric must satisfy four properties: non-negativity, identity of indiscernibles, symmetry, and the triangle inequality. Many cost functions, such as MSE and Cross-Entropy, are derived from distance metrics but may not satisfy all metric properties, as they are designed for optimization rather than geometry. The following sections explore the full taxonomy of both distance metrics and cost functions.
Foundations: Properties of Distance Metrics
A function d(x, y) is a valid distance metric if it satisfies the following four properties for all points x, y, and z in the space:
- Non-negativity: d(x, y) ≥ 0, and d(x, y) = 0 if and only if x = y.
- Symmetry: d(x, y) = d(y, x).
- Triangle Inequality: d(x, z) ≤ d(x, y) + d(y, z).
These properties ensure that the distance function behaves intuitively and can be used to define a metric space. Many common distance metrics, such as Euclidean and Manhattan distances, satisfy all four properties. However, some useful measures, such as the Kullback-Leibler divergence and cosine distance, violate one or more of these properties (e.g., KL divergence is not symmetric and does not satisfy the triangle inequality). These are often called divergences or dissimilarity measures rather than metrics.