Your Last Test Goaltest Time 200accuracy 0 40andrew You Can Practice Again or Test

A Simple Intuition for Overfitting, or Why Testing on Grooming Data is a Bad Idea

Last Updated on Baronial 21, 2016

When yous first start out with machine learning you lot load a dataset and effort models. You might call up to yourself, why can't I just build a model with all of the data and evaluate information technology on the same dataset?

It seems reasonable. More data to train the model is better, right? Evaluating the model and reporting results on the same dataset will tell you how good the model is, correct?

Incorrect.

In this mail service you volition discover the difficulties with this reasoning and develop an intuition for why it is important to exam a model on unseen data.

Train and Examination on the Same Dataset

If you have a dataset, say the iris flower dataset, what is the best model of that dataset?

The best model is the dataset itself. If you have a given data instance and ask for information technology's classification, y'all can look that instance up in the dataset and report the correct result every time.

This is the problem yous are solving when yous train and examination a model on the same dataset.

You are asking the model to make predictions to information that it has "seen" earlier. Information that was used to create the model. The best model for this problem is the wait-upwards model described above.

Descriptive Model

There are some circumstances where you do want to train a model and evaluate it with the aforementioned dataset.

You may want to simplify the explanation of a predictive variable from information. For example, you may want a fix of simple rules or a decision tree that best describes the observations you have collected.

In this case, y'all are building a descriptive model.

These models tin can be vey useful and tin can aid you in your projection or your business to amend understand how the attributes relate to the predictive value. You can add meaning to the results with the domain expertise that yous have.

The important limitation of a descriptive model is that it is limited to describing the data on which it was trained. You have no idea how authentic a predictive the model it is.

Modeling a Target Function

Consider a made upwards classification problem that goal of which is to classify data instances equally either red or green.

Modeling a Target Office
Photo by seantoyer, some rights reserved.

For this problem, assume that there exists a perfect model, or a perfect part that tin correctly discriminate any data instance from the domain as ruby or green. In the context of a specific problem, the perfect bigotry function very likely has profound meaning in the problem domain to the domain experts. We want to think nigh that and try to tap into that perspective. We want to deliver that result.

Our goal when making a predictive model for this trouble is to best approximate this perfect bigotry function.

We build our approximation of the perfect discrimination function using sample data collected from the domain. Information technology's not all the possible data, it'south a sample or subset of all possible data. If we had all the data, there would exist no need to brand predictions because the answers could just be looked upwards.

The data we use to build our approximate model contains structure within it pertaining the the platonic discrimination function. Your goal with information grooming is to best betrayal that structure to the modeling algorithm. The data also contains things that are irrelevant to the discrimination role such as biases from the selection of the information and random racket that perturbs and hides the structure. The model y'all select to approximate the function must navigate these obstacles.

The framework helps us empathise the deeper difference between a descriptive and predictive model.

Descriptive vs Predictive Models

The descriptive model is merely concerned with modeling the structure in the observed data. It makes sense to train and evaluate it on the same dataset.

The predictive model is attempting a much more hard problem, approximating the truthful bigotry function from a sample of data. Nosotros want to use algorithms that do not pick out and model all of the noise in our sample. We practise want to chose algorithms that generalize beyond the observed data. It makes sense that nosotros could just evaluate the power of the model to generalize from a data sample on data that it had not see before during grooming.

The best descriptive model is accurate on the observed data. The best predictive model is accurate on unobserved data.

Overfitting

The flaw with evaluating a predictive model on training data is that it does not inform you on how well the model has generalized to new unseen information.

A model that is selected for its accuracy on the training dataset rather than its accuracy on an unseen test dataset is very likely have lower accuracy on an unseen examination dataset. The reason is that the model is non as generalized. It has specalized to the construction in the preparation dataset. This is called overfitting, and information technology's more insidious than y'all think.

For case, you may desire to terminate training your model once the accurateness stops improving. In this state of affairs, in that location will exist a point where the accuracy on the training set continues to improve but the accuracy on unseen data starts to degrade.

You lot may be thinking to yourself: "and so I'll railroad train on the training dataset and peek at the examination dataset as I go". A fine idea, but at present the exam dataset is no longer unseen data as information technology has been involved and influenced the preparation dataset.

Tackling Overfitting

You must test your model on unseen data to counter overfitting.

A split of data 66%/34% for training to examination datasets is a good start. Using cantankerous validation is ameliorate, and using multiple runs of cross validation is ameliorate again. You want to spend the fourth dimension and get the best estimate of the models accurate on unseen information.

You lot can increase the accurateness of your model by decreasing its complexity.

In the case of determination trees for example, y'all tin can prune the tree (delete leaves) afterwards grooming. This will subtract the amount of specialisation in the specific training dataset and increment generalisation on unseen data. If you are using regression for example, you can utilize regularisation to constrain the complication (magnitude of the coefficients) during the training process.

Summary

In this post you learned the important framework of phrasing the evolution of a predictive model as an approximation of an unknown ideal discrimination function.

Under this framework y'all learned that evaluating the model on grooming data alone is insufficient. Yous learned that the best and most meaningful way to evaluate the ability of a predictive model to generalize is to evaluate information technology on unseen data.

This intuition provided the basis for why information technology is critical to use railroad train/exam split tests, cross validation and ideally multiple cantankerous validation in your examination harness when evaluating predictive models.

murphymuresind.blogspot.com

Source: https://machinelearningmastery.com/a-simple-intuition-for-overfitting/

Share this post