Homework answers / question archive / CSE 6363: Machine Learning University of Texas at Arlington Summer 2022 Assignment 2 This assignment covers neural networks, backpropagation, and Support Vector Machines

CSE 6363: Machine Learning University of Texas at Arlington Summer 2022 Assignment 2 This assignment covers neural networks, backpropagation, and Support Vector Machines

Business

Share With

---

CSE 6363: Machine Learning

University of Texas at Arlington

Summer 2022

Assignment 2

This assignment covers neural networks, backpropagation, and Support Vector Machines.

1 Neural Network Library

In this first part of this assignment, you will create a neural network library. The library will be made up of documented classes and functions that allow users to easily construct a neural network with an arbitrary number of layers and nodes. Through implementing this library, you will understand more clearly the atomic components that make up a basic neural network.

1.1 The Layer Class

For the layers you will create in this assignment, it is worth it to create a parent class named Layer which defined the forward and backward functions that are used by all layers.

In this way, we can take advantage of polymorphism to easily compute the forward and backward passes of the entire network.

1.2 Linear Layer

Create a class that implements a linear layer. The class should inherit the Layer class and implement both forward and backward.

For a given input, the forward pass is computed as

f(x; w) = w

T x.

The backward pass should compute the gradient with respect to the weights.

d

dw

f(x; w) = x

This is then multiplied with the gradients computed by the layer ahead of this one.

Since there may be multiple layers, it should additionally compute df

dx

to complete a chain

of backward passes.

1.3 Sigmoid Function

Create a class that implements the logistic sigmoid function. The class should inherit the

Layer class and implement both forward and backward.

CSE 6363: Assignment 2 Dillhoff

1.4 Hyperbolic Tangent Function

Create a class that implements the hyperbolic tangent function. The class should inherit the Layer class and implement both forward and backward.

1.5 Softmax

Create a class that implements the softmax function. The class should inherit the Layer class and implement both forward and backward.

The softmax function is defined as

pk =

e

fk

P

j

e

fj

.

The backward pass should compute the gradient with respect to the weights.

d

dw

f(x; w) = x

1.6 Cross-Entropy Loss

Create a class that implements cross-entropy loss. The class should inherit the Layer

class and implement both forward and backward.

Feel free to use the code defined in the class examples when integrating this with your library.

1.7 The Sequential Class

In order to create a clean interface that includes multiple layers, you will need to create

a class that contains a list of layers which make up the network. The Sequential class will contain a list of layers. New layers can be added to it by appending them to the current list. This class will also inherit from the Layer class so that it can call forward and backward as required.

1.8 Saving and Loading

Implement a weight saving and loading feature for a constructed network such that all

model weights can be saved to and loaded form a file. This will enable trained models to

be stored and shared.

2 Testing your library

Construct a neural network with 1 hidden layer of 2 nodes in order to solve the XOR

problem. Construct the input using numpy. You can reference the code we used for multilayer perceptrons in class to help. Train and verify that your model can solve the XOR

problem.

This may take many attempts to converge to a solution depending on your architecture,

choice of activation function, learning rate, and other factors. Attempt to solve this

2

CSE 6363: Assignment 2 Dillhoff

problem with the architecture described above using sigmoid activations and then again

using hyperbolic tangent activations. In your report, describe which one was easier to

train for this problem.

Save the weights as XOR_solved.w.

3 Handwritten Digit Recognition

In the second part of the assignment, you will use your neural network library to construct

several networks for handwritten digit recognition. There are 60000 training images and

10000 testing images. You should randomly select 10% of the training images to use as a

validation set.

In this part, you can experiment with the number of layers and nodes per layer as you wish.

Use the loss of the validation set to guide your selection of hyperparameters. Experiment

with at least 3 configurations of hyperparameters, plotting the training and validation

loss as you train each configuration. Stop training when the loss does not improve after 5

steps (early stopping). In your report, include the training/validation plots with each

choice of hyperparameters.

Once you have trained at least 3 different models, evaluate each one on the test set.

Include the test accuracy with your report.

4 Playing with Hyperparameters

The model behaves differently depending on many factors. Hyperparameters play a huge

role in this. Experiment with the following scenarios and include a written section describing your observations during training under each scenario.

• Initialize the model with all zeros.

• Initialize the model with random values between -10 and 10.

• Train on MNIST using a learning rate of 1 and then again with a learning rate of

0.001. Plot the training loss curve in both instances.

5 Support Vector Machines

To be released on June 29, 2022.

Submission

Create a zip file that includes all of your code as well as your report. The TA should

be able to easily run the code to reproduce all plots and results. Include any additional

instructions, if necessary.

3