1. Recap: Estimating E(MSE) and cross-validation
2. Linear regression
3. Which variables shall I include in my model?
   • Feature / subset selection
   • Shrinkage / Regularization methods: Lasso and Ridge
4. Conclusions

Suppose that you’re a teacher writing an exam for some students [models]. If you want to evaluate their skills, will you give them exercises [observations] that they have already seen [train set], and that they still have on their desks, or new exercises, inspired by what they learned, but different from them? [different set] (jpl, stackoverflow)

See: https://javier.science/panel_bias_variance/

• Complex models will tend to overfit, and have high variance
• Simple models will tend to underfit, and have high bias

Which model has high bias?

• Comparing statistical methods (e.g. linear regression vs knn regression)
• Comparing models with different predictors included (e.g. linear regression including predictors [\(X_1\), \(X_2\)] vs [\(X_1\), \(X_2\), \(X_3\)] )
• Comparing two models with different hyperparameters (e.g. KNN regression using the closest 3 vs 10 neighbors)

\(y = f(x) + \epsilon\)

• Data
  • \(Y\): Observed outcomes (dependent variable)
  • \(X = {x_1, ..., x_p}\): p predictors (also called features, or independent variables)
  • \(f\): function to estimate

Goal: Find the parameters \(\theta\) that minimize the loss/cost/MSE

\(\mathcal{L(\theta)} = Cost(\theta) = MSE(\theta) = \frac{1}{n}\sum^{n}_{i = 1}{(y_i - h_\theta(x_i))^2}\)

• \(y_i\) = observed value \(i\)
• \(h_\theta(x_i)\) –> Prediction from our model for observation \(i\) (also seen as \(\hat{f}(x_i)\) or \(\hat{y_i}\))

^{_{Source: Eric Eaton, Applied Machine Learning course}}

• Goal: Minimize MSE (\(\mathcal{L(\theta)}\))

• \(\mathcal{L(\theta)} = Cost(\theta) = MSE(\theta) = \frac{1}{n}\sum^{n}_{i = 1}{(y_i - h_\theta(x_i))^2}\)

^{_{Source: Eric Eaton, Applied Machine Learning course}}

• Goal: Minimize MSE (\(\mathcal{L(\theta)}\))
• \(\mathcal{L(\theta)} = \frac{1}{n}\sum^{n}_{i = 1}{(y_i - h_\theta(x_i))^2}\)

^{_{Source: Eric Eaton, Applied Machine Learning course}}

• Goal: Minimize MSE (\(\mathcal{L(\theta)}\))
• \(\mathcal{L(\theta)} = \frac{1}{n}\sum^{n}_{i = 1}{(y_i - h_\theta(x_i))^2}\)

^{_{Source: Eric Eaton, Applied Machine Learning course}}

• Goal: Minimize MSE (\(\mathcal{L(\theta)}\))
• \(\mathcal{L(\theta)} = \frac{1}{n}\sum^{n}_{i = 1}{(y_i - h_\theta(x_i))^2}\)

^{_{Source: Eric Eaton, Applied Machine Learning course}}

^{_{Source: Andrew Ng, Machine Learning course}}

• Option 1: Using algebra
• Option 2: Using gradient descent –> optimization technique used in many machine learning methods

• Initiate parameters \(\theta\) randomly
• Predict \(h_\theta(x)\) (i.e., \(\hat{y}\)) and calculate \(\mathcal{L(\theta)}\)
• Move to a lower point in the curve
• Repeat

^{_{Source: Andrew Ng, Machine Learning course}}

^{_{Source: Andrew Ng, Machine Learning course}}

^{_{Source: Andrew Ng, Machine Learning course}}

• When we get to the bottom, we stop

^{_{Source: Andrew Ng, Machine Learning course}}

• Calculus!
  • We want to know where is “down” in the curve—i.e. the gradient of the curve (\(\mathcal{L(\theta)}\)) .
  • The gradient is the slope of the tangent line—i.e., the derivative of the MSE (\(\mathcal{L(\theta)}\)).
• In practice: For each coefficient in the model, \(\theta_j\)
  • New \(\theta_j\) = \(\theta_j - \alpha \frac{\partial \mathcal{L(\theta)}}{\partial \theta_j}\)
    • alpha = learning rate (a small value, can be adaptive)

• We estimate the generalization error using out-of-sample prediction error (e.g. on a validation or test dataset; or through cross-validation)
• We can fit a linear regression using gradient descent

Now:

• Which variables shall I include in my model?
  • Feature (subset) selection
  • Shrinkage / Regularization methods: Lasso and Ridge
• Conclusions

• Including too few variables: Model with high bias
• Including too many variables: Model with high variance

^{_{Source: Eric Eaton, Applied Machine Learning course}}

• Feature selection:
  • Pick the best \(p\) predictors
  • How: Best subset, forward selection, backward selection
• Regularization:
  • Constrain the sum of squared cofficients (\(L2\)) or absolute sum of coefficients (\(L1\)) to be below \(s\)
  • How: Adapt the loss function (e.g. MSE) to penalize including variables
• Dimensionality reduction:
  • Combine variables that correlate (covered in aDAV II)
  • How: Unsupervised learning

• For each level of complexity (number of predictors):
  • Fit x models of equal complexity –> Keep the best using e.g. MSE or \(R^2\)
• Estimate E(MSE) for models of different complexity using cross-validation –> Select best model
• Estimate E(MSE) of the best model using test data

1. Let \(M_0\) denote the null model, which contains no predictors. This model simply predicts the sample mean for each observation.
2. For \(k = 1, 2, ..., p\) (\(p\) = total number of predictors):
   • Fit all \(({p \atop k})\) models that contain exactly \(k\) predictors
     • fit all \(p\) models that contain exactly one predictor
     • fit all \(({p \atop 2}) = p(p - 1)/2\) models that contain exactly two predictors
     • and so forth
   • Pick the best among these \(({p \atop k})\) models, and call it \(M_k\) (using e.g., smallest MSE or highest R2)
3. Select a single best model from among \(M_0, ..., M_p\) using:
   • A measure that takes into consideration the number of predictors (e.g. AIC, BIC, adjusted-R^2)
   • Or better, cross-validated prediction error, e.g., MSE, or (1-R^2)

Train MSE in the credit data set

Careful! Train MSEs decrease with the complexity. We need to select using the validation MSE, not the train MSE!

1. Let \(M_0\) denote the null model, which contains no predictors. This model simply predicts the sample mean for each observation.
2. For \(k = 0, 1, 2, ..., p - 1\):
   • Consider all \(p-k\) models that augment the predictors in \(M_k\) with one additional predictor
   • Pick the best among these \(({p - k})\) models, and call it \(M_{k+1}\) (using e.g., smallest MSE or highest R2)
3. Select a single best model from among \(M_0, ..., M_p\) using:
   • A measure that takes into consideration the number of predictors (e.g. AIC, BIC, adjusted-R^2)
   • Or better, cross-validated prediction error, e.g., MSE or (1-R^2)

• Forward stepwise selection’s has a computational advantage (~\(p^2/2\) vs \(2^p\) models)

• Though forward stepwise tends to do well in practice, it is not guaranteed to find the best possible model out of all \(2^p\) models containing subsets of the p predictors.

• For instance, suppose that in a given data set with p = 3 predictors, the best possible one-variable model contains X1, and the best possible two-variable model instead contains X2 and X3. Then forward stepwise selection will fail to select the best possible two-variable model, because M1 will contain X1, so M2 must also contain X1 together with one additional variable.

1. Let \(M_p\) denote the full model, which contains all \(p\) predictors.
2. For \(k = p, p - 1, ..., 1\):
   • Consider all \(k\) models that contain all but one of the predictors in \(M_k\), for a total of \(k-1\) predictors
   • Pick the best among these \(k\) models, and call it \(M_{k-1}\) (using e.g., smallest RSS or highest R2)
3. Select a single best model from among \(M_0, ..., M_p\) using:
   • A measure that takes into consideration the number of predictors (e.g. AIC, BIC, adjusted-R^2)
   • Or better, cross-validated prediction error, e.g., MSE or (1-R^2)

Number of models fitted at each step, example for a dataset with 20 predictors:

• Best subset
  \(+\) Finds the best subset, as advertised – when there is enough data to find it
  \(−\) Need to fit \(2^p\) models. With e.g., 20 predictors that is 1,048,576 regressions to run and evaluate. Not even mentioning squares, cubes, products, etc.

  
  

• Forward/backward
  \(+\) Much more efficient, \(O(p^2)\) instead of \(O(2^p)\), e.g. 211 models for 20 predictors
  \(−\) Not guaranteed to find the best subset

• Backward sometimes not even possible (e.g. p > n)
• Forward can be fooled, especially when two variables work together but do nothing alone:
  • Backward considers performance of variable together with others.

    
    

• Both backward and forward are well-known to be bad at finding ‘true’ subset of predictors
  \(\to\) Reveled in several fields (e.g. social science);
• For prediction, we do not care about the ‘true’ subset.

• We want to fit the training data (estimate the weights of the coefficients)
• Make the model behave ‘regularly’ by penalizing the purchase of ‘too many’ coefficients
• Extremely efficient way to approximately solve the best subset problem: Variable selection + regression in one step
• Often yields very good results

• If you are interested in prediction and not inference (i.e. if identifying the relevant features is not a primary goal of the analysis), regularization will usually be better

Usually, find the \(\theta_j\) (or sometimes we use the notation \(\beta_j\)) that minimizes

\(\mathcal{L(\theta)} = MSE = \frac{1}{n}\sum^{n}_{i = 1}{(y_i - h_\theta(x_i))^2}\)

Now, find the \(\theta_j\) that minimizes

\(\mathcal{L(\theta)} = MSE + \lambda \cdot Penalty\)

where the penalty is:

• \(\sum_{j>0}{|\theta_j|}\) –> (L1 Lasso) Tends to set some coefficients to zero (great for interpretability)
• \(\sum_{j>0}{\theta_j^2}\) –> (L2, Ridge) Tends to keep all coefficients

Equivalent, minimize \(MSE\) subject to:

• \(\sum_{j>0}{|\theta_j|} < s\) –> (L1 Lasso) Tends to set some coefficients to zero (great for interpretability)
• \(\sum_{j>0}{\theta_j^2} < s\) –> (L2, Ridge) Tends to keep all coefficients

\(s\) = budget, “buying” coefficients cannot cost more than that

LASSO:

fit <- glmnet(x, y, alpha = 1, lambda = 1.5)

Ridge:

fit <- glmnet(x, y, alpha = 0, lambda = 1.5)

• LASSO and ridge have a tuning parameter \(\lambda\)
• The usual least squares is \(\lambda = 0\)
• Higher \(\lambda \to\) stricter penalty \(\to\) smaller budget \(s\)
• Higher \(\lambda\) ‘shrinks’ coefficients to 0
• Can select \(\lambda\) using cross-validation (cv.glmnet())

• Gradient descent is a very efficient way to fit/train models

• By using feature selection or regularization, we can obtain better prediction accuracy and model interpretability

• Feature selection includes best subset, forward and backward selection

  • Best subset selection performs best, but it comes at a prize

• Regularization includes LASSO and Ridge

  • LASSO shrinks unimportant parameters to truly zero, while Ridge shrinks them to small values