COMP615 Assignment Two page 1 of 4
COMP615 – Foundations of Data Science
This assignment allows you to address a real-world problem using the machine learning
workbench. The analysis and justifications of your answers carry a high proportion of the marks
awarded. Please make sure you read through the entire assignment before you start.
This assignment should be completed in pairs (maximum of two students). You will be given
a dataset once you have emailed and notified us of the name of your partner.
Project Report – Basic Structure
Your final report must include the following sections and information.
Use provided cover page on Canvas
Parer Code, Paper Name, and Semester
Students’ Names and IDs
Table of Contents, List of Figures and Tables
1 Introduction [5 marks]
You are expected to provide information on the dataset you are assigned to use for your
assignment. Provide a statement of the problem, outlining the problem that your dataset
addresses.
2 Data Exploration [10 marks]
In this section of your report, you are required to discuss the dataset and any missing or
outliers found in the dataset.
How many features (attributes), instances, and data types
Provide summary statistics of your dataset.
Illustrate the features of your dataset using meaningful visualisation (eg. boxplots,
histograms, etc.).
Explain the data pre-processing steps taken to make the data ready for performing
classification.
3 Decision Tree Classifier
You are required to build a model using the Decision Tree Classifier and answer the
following questions based on the model built. In building the model, use the 10-fold crossvalidation option for testing. Your answers need to be supported by suitable evidence,
wherever appropriate. Some examples of suitable evidence are Confusion Matrices, Model
Visualizations, and Model Summary Report.
a) Now build a model using the Decision Tree algorithm. Adjust two suitable parameters
(one at a time) to reduce the size of the tree to improve the accuracy of your model.
Report the accuracy score for each parameter using the plots. Provide the final
optimized classification tree and describe its structure. [12 marks]
COMP615 Assignment Two page 3 of 4
b) Describe the role of the two parameters in the model building that you used in a) above.
Do you expect using the same values obtained for this dataset will improve the accuracy
for other types of datasets Justify your answer. [8 marks]
c) Generate and examine the Confusion Matrix carefully and explain your findings.
Provide the model summary report and discuss the metrics (accuracy, precision, recall,
and F1-score).
[10 marks]
d) Find the feature importance based on the final classification model and explain your
findings. [ 5 mark
s]
4 Artificial Neural Network (ANN)
In this part, you are required to explore various architectures for building an Artificial
Neural Network (ANN). In building the model, use the 10-fold cross-validation option for
testing.
a) Use an appropriate feature selection method to identify the top five most significant
features. State the method used and list the features produced. Compare the list
produced in the previous section by the Decision Tree model. Identify similarities and
differences. Discuss any differences. [10 marks]
b) Use the sklearn.MLPClassifier with default values for parameters and a single hidden
layer with k neurons (k <=25). Use default values for all parameters other than the
number of iterations. Determine and report the best number for iteration that gives the
highest accuracy. Use this classification accuracy as a baseline for comparison in later
parts of this question. [5 marks]
c) Enable the loss value to be shown on the training segment and track the loss as a
function of the iteration count. You will observe that even when the loss value decreases
the error value increases between consecutive iterations. Conversely, the error value
may decrease when the loss increases between consecutive iterations. How do you
explain this [5 marks]
d) Experiment with two hidden layers and experimentally determine the split of the
number of neurons across each of the two layers that gives the highest classification
accuracy. In part 1, we had all k neurons in a single layer, in this part we will transfer
neurons from the first hidden layer to the second iteratively in step size of 1. Thus, for
example in the first iteration, the first hidden layer will have k-1 neurons whilst the
second layer will have 1, in the second iteration k-2 neurons will be in the first layer
with 2 in the second and so on. Summarise your classification accuracy results in a 25
by 2 table with the first column specifying the combination of neurons used (e.g., 12,
13) and the second column specifying the classification accuracy. [10 marks]
e) From the table created in part d, you will observe the accuracy variation with the split
of neurons across the two layers. Give explanations for some possible reasons for this
variation. [5 marks]
COMP615 Assignment Two page 4 of 4
5 Performance Comparison
Compare the performance of the Decision Tree and MLP Classifiers on your dataset. Choose
the best-performing model for your dataset and explain why you have chosen it. Discuss the
findings from your experiments and provide your opinion about these two classifiers.
[10 marks]
There will be 5 marks for the presentation of the assignment including spelling and grammar,
layout, formatting, and readability of the figures.
Good luck!