Machine Learning 1, SS22 Homework 2

PCA. Neural Networks.

Contents

1. Neural Networks [6 points + 3* points]                                              2
   1. PCA and Classification [16 points]............................................................... 2
   2. Model selection using GridSearchCV from sklearn [3* points]................. 3
2. Regression with Neural Networks [9 points]                                               3

 

General remarks

Your submission will be graded based on:

• Correctness (Is your code doing what it should be doing? Is your derivation correct?)
• The depth of your interpretations (Usually, only a couple of lines are needed.)
• The quality of your plots (Is everything clearly visible in the print-out? Are axes labeled?............................................................................................................................... )
• Your submission should run with Python 3.5+.

For this assignment, we will be using an implementation of Multilayer Perceptron from scikit-learn. The documentation for this is available at the scikit website. The two relevant multi-layer perceptron classes are

– MLPRegressor for regression and MLPClassifier for classification.

For both classes (and all scikit-learn model implementations), calling the fit method trains the model, and calling the predict method with the training or testing data set gives the predictions for that data set, which you can use to calculate the training and testing errors.

 

 

 

 

 

1

 

 

2

 

1. Neural Networks [6 points + 3* points]
   1. PCA and Classification [16 points]

 

∗

PCA can be used as a data preprocessing technique, to reduce the dimensionality of data. In this task, we will use the Sign Language dataset. It consists of images of hands that should be classified into 10 different classes, as shown in Fig. 1. The images are of size (64, 64) pixels. That means, the input dimension to the neural network that we want to train to classify those images would be quite large (64 64 = 4096), and hence we want to reduce their dimension by means of PCA. By doing this, the benefit will be that the training time of the network will be shorter.

Tasks:

1. 1. 1. PCA for dimensionality reduction. Load the dataset (features and targets). Use PCA from sklearn.decomposition to reduce the dimensionality of the data. Creating an instance of PCA class. Choose n components (number of principal components) such that about 85% of variance is explained. In the report, state n components that you used, and the exact percentage of variance explained that you get.

Hints: You will need to fit the model, and apply the dimensionality reduction of the original features, in order to obtain the data with a reduced dimension: (n  samples, n  components).  Check the Attributes of this class, and find which one to use to get the percentage of variance explained. To narrow your search, the number of principal components should be: 100 < n components < 200.

1. 1. 2. 

∈ {               }

Varying  the  number  of  hidden  neurons.   We  will  use  the  data  with  reduced  dimension  (from the previous step) to train a neural network to perform classification. We will vary the number of neurons in one hidden layer of the neural network: n hidden      5, 100, 200 .   For this task, we will use MLPClassifier from sklearn.neural network. Create an instance of MLPClassifier.  Set max iter  to 500, random state to 0, hidden layer sizes to be n hid (one of values from n hidden), and all the other parameters should have their default values. (If the warning about the optimization not converging appears, you can ignore it. No need to do anything about it.)

For each n hid, report accuracy on the train and test set, and the current loss.

Answer the questions: How do we know (in general) if the model does not have enough capacity (the case of underfitting)? How do we know (in general) if the model starts to overfit? Does that happen with some architectures/models? (If so, say with what number of neurons that happens). Which model would you choose here and why?

1. 1. 3. 

∈ {              }

To prevent overfitting, we could use a few approaches, for example, introducing regularization and/or early stopping. Copy the code from the previous task. Try out (a) alpha = 1.0 (b) early stopping = True, (c) alpha = 1.0 and early stopping = True. Choose (a), (b), or (c) that you think works best (write a sentence saying what your choice was). Then, report the train and test accuracy, and the loss for n hidden       5, 100, 200 . Does this improve the results from the previous step? Which model would you choose now?

1. 1. 4. Variability of the performance. Choose the best performing parameters that you  found  in  the previous task,  and vary the seed (parameter random state) with 5 different values of your choice. (Note: When the seed is fixed, we can reproduce our results, and we avoid getting different results for every run.) When changing the seed, what exactly changes? Report minimum and maximum accuracy, and mean accuracy over 5 different runs and standard deviation i.e., (mean ± std).
      5. Using a model with any (fixed) seed of your choice, plot the loss curve (loss over iteration). Hint: Check the Attributes of the classifier.
      6. Using a model with any (fixed) seed of your choice, calculate predictions on the test set. In the code, print the classification report and confusion matrix, and include either a screenshot of both of them, or copy the values to the report. How could you calculate yourself recall from support and confusion matrix entries? Explain in words what is  recall.  What  is  the  most  misclassified  image? State which class (digit) it was, and how you concluded that.