=================================================================================

In machine learning, estimation error, ε(h^) - ε:(h*), refers to the difference between the predicted or estimated value and the true value of the quantity we are trying to predict. It is a measure of how well the machine learning model generalizes to new, unseen data.

When we train a machine learning model, we use a dataset to teach the model to make predictions. The model learns patterns and relationships within that training data. However, the true test of the model's effectiveness is its ability to make accurate predictions on new, previously unseen data.

Estimation error can arise for various reasons:

1. Bias (Underfitting): If the model is too simple and cannot capture the underlying patterns in the data, it may have high bias. This leads to underfitting, and the model may perform poorly on both the training and test datasets.

2. Variance (Overfitting): If the model is too complex and captures noise or random fluctuations in the training data, it may have high variance. This leads to overfitting, where the model performs well on the training data but poorly on new, unseen data.

3. Noise in the Data: The presence of random noise or outliers in the data can also contribute to estimation error. If the model learns to fit the noise in the training data, it may not generalize well to new data.

4. Insufficient Data: If the amount of training data is limited, the model may not learn the underlying patterns effectively, leading to estimation error.

Figure 3765a shows the expected risk (error), approximation error, and estimation error.

Figure 3765a. Expected risk (error), approximation error, and the estimation error. [1]

The expected risk (error) of a hypothesis h_s ∈H, which is selected based on the training dataset  S  from a hypothesis class H, can be decomposed into the approximation error, ε_app, and the estimation error, ε_est, as following,

          L_D(h_s) = ε_app + ε_est ------------------------------------- [3765]

Figure 3765b shows the relationship between these terms in Equation 3765. The red points are specific hypotheses.  The best hypothesis (the Bayes hypothesis) lies outside the chosen hypothesis class H. The distance between the risk of  h^ and the risk of h* is the estimation error, while the distance between  ℎ* and Bayes hypothesis is the approximation error.

Some properties are:

• The larger  H is, the smaller this error is, because it's more likely that a larger hypothesis class contains the actual hypothesis we are looking for. Therefore, if  H does not contain the actual hypothesis we are searching for, then this error could not be zero.

• This error does not depend on the training data since in Equation 3765, there's no S (the training dataset).

============================================

[1] www.medium.com.