CS7840 Spring 21/22 Soft Computing Assignment 1 4

CS7840 Spring 21/22

Soft Computing Assignment 1

4. 10 Points
Generate a sample of data for N = 50 points and variance σ = 50.
a. Plot the points.
b. Calculate the gradient descent for a line through the sample data points with a
learning rate η = 0.00001 for 10,000 iterations.
c. Plot the points and the best fit line that was calculated using Gradient Descent.
d. Repeat the process for N = 100, 200, and 500 and for σ = 100, 50, 300.
e. What happens when you increase the learning rate from η = 0.00001 to η = 0.0001,
then again to η = 0.001?
f. Plot the results at the 10th, 100th, and 1000th iterations.

5. 10 Points
Generate a surface with the equation:
Z = 0.1X3
+ Y 2
For values of X, Y = [−2, 2].
a. Plot the surface.
b. Implement the Steepest Descent algorithm to fine the minimum on this surface with
an initial starting point of X0 = 1.5 and Y0 = 1.8.
c. What are the values of (X1, Y1) and X2, Y2.
d. How many iterations does it take for the values to converge such that ? < 0.0001
where ? is the change in the value between Xn, Yn and Xn−1, Yn−1.

6. 10 Points
Load the MNIST dataset again, or use the previously loaded training and testing data
for MNIST. Select N = 15,000 sample points from either the training or testing data
or a combination of both.
a. Calculate the Eigenvalue-Eigenvector pairs of the variance-covariance matrix of the
sample data X.
Variance-Covariance Matrix
The Variance-Covariance Matrix of a dataset X is?.
Σ = V ar(X) = E[(X − µ)(X − µ)T
] (10)
Eigenvalue-Eigenvector Pair
The Eigenvalue-Eigenvector Pair of Σ are the eigenvectors normalized by their
eigenvalues.
(λ, e) = ([λ1, λ2, ...λp], [[e1, e2, ..., ep]) (11)
They are ordered so that λ1 ≥ λ2 ≥ ... ≥ λp ≥ 0.
b. What is the shape of the variance-covariance matrix?
c. Express the MNIST data using the top two principal components and show a plot
of the 1st PC against the 2nd PC.
d. Is this sufficient to represent the MNIST data?

7. 10 Points
Load the MNIST dataset again, or use the previously loaded training and testing data
for MNIST. Select N = 15,000 sample points from either the training or testing data
or a combination of both.
a. Use the Logistic Hyperbolic Cosine function as the linear transformation function W
to calculate the independent components of the selected MNIST data. Use a tolerance
of 0.00001.
Independent Component Analysis
Independent Component Analysis finds the linear transform W = (w1, w2, ..., wp)
for a dataset X so that S = WX. The independent components are A =
(a1, a2, ..., ap) and the dataset X is expressed as:
Xi = ai,1S1 = ai,2S2 + ... + ai,pSp (12)
b. Express the MNIST data using the top two independent components and show a
plot of the 1st IC against the 2nd IC.
c. How long does it take the ICA algorithm to run, compared to PCA?
d. Is there better separation for the MNIST data when using ICA?
e. Is two components sufficient to represent the MNIST dataset when using ICA?

8. 20 Points
Use the separated MNIST training and testing data with the model created in Problem
1.
a. Calculate the PCA for the training and testing data and run the model from Problem
1 using the Sparse Categorical Crossentropy as the loss function.
b. Calculate the confusion matrix of this data after 3 epochs.
c. Repeat part a. but use the ICA dimension reduction method.
d. Calculate the confusion matrix for this data after 3 epochs.
e. Are there parameters you can tune to improve results for the data after using PCA
or ICA?
f. Use the k-Means clustering algorithm implemented in Problem 3 to cluster the
MNIST data into 10 clusters.
g. Display the data assigned to each cluster.
h. Repeat the process with the data after PCA.
i. Repeat the process with the data after ICA.
j. What are some of the parameters you can tune in each step to get better results.