Error, loss functions, and why they are needed
Case 4. Same accuracy – different log loss
Accuracy answers only one question: how often the model predicts the correct class at a chosen threshold (for example, 0.5). But it does not capture how confident the model is in its predictions.
In this case we will use the same true labels and two models that yield the same class labels at the 0.5 threshold. However, one model outputs probabilities closer to 0 and 1 and achieves a lower log loss, while the other stays closer to 0.5 and gets penalized with a higher log loss.
For binary classification, log loss is defined as:
$\text{LogLoss} = -\frac{1}{n} \sum_{i=1}^{n} [y_i \log(p_i) + (1 - y_i) \log(1 - p_i)]$, where $y_i$ is the true label (0 or 1) and $p_i$ is the predicted probability of class 1.
Example of use:
<?php
function logLoss(array $y, array $p, float $eps = 1e-15): float {
$loss = 0.0;
$n = count($y);
for ($i = 0; $i < $n; $i++) {
$pi = max($eps, min(1 - $eps, $p[$i]));
$loss += $y[$i] * log($pi) + (1 - $y[$i]) * log(1 - $pi);
}
return -$loss / $n;
}