Difference between Estimation and Approximation Errors  Python Automation and Machine Learning for ICs   An Online Book  

Python Automation and Machine Learning for ICs http://www.globalsino.com/ICs/  


Chapter/Index: Introduction  A  B  C  D  E  F  G  H  I  J  K  L  M  N  O  P  Q  R  S  T  U  V  W  X  Y  Z  Appendix  
================================================================================= Table 3764. Difference between estimation and approximation errors.
The expected risk (error) of a hypothesis h_{s} ∈H, which is selected based on the training dataset S from a hypothesis class H and is the output of the ERM learner (under the hypothesis class H ) (ERM_{H}), can be decomposed into the approximation error, ε_{app}, and the estimation error, ε_{est}, as following, L_{D}(h_{s}) = ε_{app} + ε_{est}  [3764a] = ε_{app} + (L_{D}(h_{s})  ε_{app}) [3764b]  [3764c] where, represents the empirical risk of a hypothesis on a dataset . Empirical risk is the average loss over the dataset, where the loss is a measure of how well the hypothesis approximates the true distribution of the data. is the minimum empirical risk over all hypotheses ℎ in the hypothesis space . In other words, it represents the best achievable empirical risk among all possible hypotheses in the given hypothesis space. represents the excess risk. The excess risk is the difference between the empirical risk of the specific hypothesis ℎ and the best achievable empirical risk. With the theory of the hypothesis class (page3982), we have,  [3764d] where, L(h^) represents the expected loss of the learned hypothesis ℎ^ on unseen data. L(h*) represents the expected loss of the best possible hypothesis ℎ* in the hypothesis class on unseen data. The term on the righhand side is a term related to the complexity of the hypothesis class and the sample size. Here: is the size of the hypothesis class (the number of possible hypotheses). is a parameter representing the confidence level. is the sample size. Figure 3764a shows the relationship between these terms in Equation 3764a. The red points are specific hypotheses. The best hypothesis (the Bayes hypothesis) lies outside the chosen hypothesis class H. The distance between the risk of h^ and the risk of h* is the estimation error, while the distance between ℎ* and Bayes hypothesis is the approximation error. Some properties are:
Figure 3764b shows the estimation and approximation errors with noisy data. The estimation error refers to the difference between the true function and the estimated function. The estimated function is the one which is obtained based on the hypothesis or model. The difference between the true function and the observed (noisy) data (blue scattered points) represents the approximation error. Both estimation error and approximation error can be positive or negative.
============================================


=================================================================================  

