Search for a command to run...
No comments yet. Be the first to comment.
Ready to embark on an exciting journey into the world of AI and ML? Perfect! Let's start at the very beginning with a concept that's absolutely essential - data training. Now, you might be thinking, "Data training... that doesn't sound too complicate...
I recently watched a short video by Carly Taylor on gamer mindset. In that 1 minute video, she talks about Chris Sawyer and how he wrote the awesome game of Roller Coaster Tycoon entirely in assembly (all by himself) because the compute power at the ...
I recently published an article on how to build an MCP server in C# (https://tjgokken.com/building-your-first-model-context-protocol-server-in-c). On that article, we built a simple MCP-style server. It remembered things, stored context in memory. Th...
Because APIs Aren't Confusing Enough Already
Or how to tame the container beast without selling your soul to YAML
Bringing It All Together: K8s Hackathon
Welcome to the first installment of our "ML Fundamentals: Loss Functions" series! Today, we're diving into Mean Bias Error (MBE), the unsung hero of bias detection in machine learning models.
When building machine learning models, particularly regression models, it's crucial to assess how well your model's predictions align with the actual values. One way to measure this alignment is by using a loss function. Among the many loss functions available, Mean Bias Error (MBE) is one of the simpler ones, although it's not as commonly used in practice as others like Mean Squared Error (MSE) or Mean Absolute Error (MAE).
Mean Bias Error (MBE) is like that brutally honest friend who tells you whether you're consistently aiming too high or too low. In technical terms, it measures the average difference between your predicted values and the actual values.
A positive MBE means your model is getting a bit carried away with optimism—consistently predicting values higher than reality (over-predicting). A negative MBE suggests your model is a pessimist—consistently predicting values lower than reality (under-predicting).
Mathematically, MBE is defined as:
$$MBE = \frac{1}{N} \sum_{i=1}^{N} (y_i - \hat{y}_i)$$
Where:
yi: The actual observed value for the i-th data point.
hat y is the predicted value
N is the number of data points
In plain English: Take all the differences between actual and predicted values, add them up, and divide by how many you have. That's it! Now, that wasn’t so hard, was it? Actually, everything is hard until you understand it.
MBE points you in the direction your model is drifting, kind of like a coach for an athlete. It grounds you if you get over- or under-confident.
In real life it is used in:
Detecting Systematic Bias: Is your model consistently overshooting or undershooting?
Model Validation: Not so much for training, but fantastic for checking if your model has a directional problem.
Simple Interpretability: The value directly tells you the average amount your predictions are off by (and in which direction).
However, MBE has an Achilles' heel—it allows positive and negative errors to cancel each other out. Your model could be wildly off in both directions, but if the errors balance out, MBE might look deceivingly close to zero. That's why it's often paired with other metrics like Mean Squared Error (MSE) or Mean Absolute Error (MAE).
Here's a simple example to calculate MBE using Python:
import numpy as np
# Actual values (what really happened)
y_true = np.array([3, -0.5, 2, 7])
# Predicted values (what our model thought would happen)
y_pred = np.array([2.5, 0.0, 2, 8])
# Calculate Mean Bias Error
mbe = np.mean(y_true - y_pred)
print(f"Mean Bias Error (MBE): {mbe}")
# Output: Mean Bias Error (MBE): -0.25
In this case, our MBE is -0.25, suggesting a slight tendency to over-predict. But remember, this is a tiny example—in real-world datasets, these patterns become much more meaningful.
Unfortunately, ML.NET doesn't have a built-in method for calculating MBE directly, so we’ll need to write a custom implementation.
using System;
using System.Linq;
class Program
{
static void Main()
{
double[] yTrue = { 3.0, -0.5, 2.0, 7.0 };
double[] yPred = { 2.5, 0.0, 2.0, 8.0 };
// Calculate Mean Bias Error
double mbe = yTrue.Zip(yPred, (actual, predicted) => actual - predicted).Average();
Console.WriteLine($"Mean Bias Error (MBE): {mbe}");
// Output: Mean Bias Error (MBE): -0.25
}
}
When working with Azure Machine Learning, tracking custom metrics like MBE can be extremely valuable for several reasons:
Experiment Tracking: Azure ML automatically captures standard metrics, but MBE isn't included by default. Logging it helps you compare model bias across different experiment runs.
Model Monitoring: After deployment, tracking MBE can alert you to potential data drift if your model starts showing different bias patterns in production.
Team Collaboration: When working in teams, having MBE logged alongside other metrics makes it easier for colleagues to understand the complete performance profile of your model.
Here's how to implement it:
from azureml.core import Run
import numpy as np
# Assume we already have our predictions and actual values
y_true = np.array([3, -0.5, 2, 7])
y_pred = np.array([2.5, 0.0, 2, 8])
# Calculate Mean Bias Error
mbe = np.mean(y_true - y_pred)
# Log the metric to Azure ML
run = Run.get_context()
run.log("Mean Bias Error", mbe)
# You can also log it at regular intervals during training
# for epoch in range(num_epochs):
# # training code here
# current_mbe = calculate_mbe_on_validation_set()
# run.log("Mean Bias Error", current_mbe, description="Validation MBE")
This code integrates with Azure ML's experiment tracking system, allowing you to visualize how your model's bias changes during training or across different model versions.
Imagine you're building a model to predict daily temperatures. After training, you calculate an MBE of -2.5°C/F. This tells you that, on average, your model is predicting temperatures 2.5°C/F higher than the actual temperatures.
This insight is valuable because you might decide to:
Add a simple correction factor to your predictions
Investigate why your model has this warm bias.
Collect more diverse training data to help address this bias
OK, OK, I know, it still does not help when the weather forecasts says no rain and you are looking out and it is pouring down but hopefully you get the idea. Maybe, the weather is too unpredictable, but I can say with a huge confidence that tomorrow is going to be 25°C or 77°F plus or minus 20°.
See what I did there?
While MBE is useful, it has its limitations.
As I mentioned earlier, positive and negative results cancel each other out. On top of that, since it is taking average values, extremes can significantly influence the results. These reasons are also why it is not usually a good idea to use it alone when training models.
That's why in practice, MBE is typically used alongside other metrics like Mean Absolute Error (MAE) or Root Mean Squared Error (RMSE), which we'll explore in upcoming articles.
Mean Bias Error might not be the rockstar of loss functions, but it plays a critical role in understanding directional bias in your predictions. While it has limitations, its simplicity and interpretability make it a useful addition to your model evaluation toolkit.
Other loss functions like Mean Absolute Error (MAE) or Root Mean Squared Error (RMSE) complement MBE by focusing on error magnitude rather than direction. Together, these metrics provide a more holistic view of how well your model is performing.
Remember that no single metric tells the complete story about your model. However, if there is a bias in my predictions, I would like to know about it. Wouldn’t you?
Have questions about MBE or want to share how you use it in your models? Drop a comment below!