"You will be graded on parts that are marked with **\\#TODO** comments. Read the comments in the code to make sure you don't miss any.\n",
"You will be graded on parts that are marked with **TODO** comments. Read the comments in the code to make sure you don't miss any.\n",
"\n",
"### Mandatory for 478 & 878:\n",
"\n",
...
...
@@ -69,10 +69,9 @@
"| 1 | Implement `distance` | |\n",
"| 2 | Implement `k-NN` methods | |\n",
"| 3 | Model evaluation | |\n",
"| 4 | Learning curve | |\n",
"| 5 | Optimizing *k* | |\n",
"| 6 | ROC curve analysis | |\n",
"| 7 | Assess suitability of *k*-NN| |\n",
"| 4 | Optimizing *k* | |\n",
"| 5 | ROC curve analysis | |\n",
"| 6 | Assess suitability of *k*-NN| |\n",
"\n",
""
]
...
...
@@ -129,8 +128,8 @@
"metadata": {},
"source": [
"### Rubric:\n",
"* Euclidean +5, +5\n",
"* Manhattan +5, +5"
"* Euclidean +7.5, +7.5\n",
"* Manhattan +7.5, +7.5"
]
},
{
...
...
%% Cell type:markdown id: tags:
# *k*-Nearest Neighbor
We'll implement *k*-Nearest Neighbor (*k*-NN) algorithm for this assignment. We will use the **madelon** dataset as in Programming Assignment 0.
A skeleton of a general supervised learning model is provided in "model.ipynb". The functions that will be implemented there will be indicated in this notebook.
### Assignment Goals:
In this assignment, we will:
* implement 'Euclidean' and 'Manhattan' distance metrics
* use the validation dataset to find a good value for *k*
* evaluate our model with respect to performance measures:
* accuracy, generalization error and ROC curve
* try to assess if *k*-NN is suitable for the dataset you used
### Note:
You are not required to follow this exact template. You can change what parameters your functions take or partition the tasks across functions differently. However, make sure there are outputs and implementation for items listed in the rubric for each task. Also, indicate in code with comments which task you are attempting.
%% Cell type:markdown id: tags:
# GRADING
You will be graded on parts that are marked with **\#TODO** comments. Read the comments in the code to make sure you don't miss any.
You will be graded on parts that are marked with **TODO** comments. Read the comments in the code to make sure you don't miss any.
### Mandatory for 478 & 878:
| | Tasks | 478 | 878 |
|---|----------------------------|-----|-----|
| 1 | Implement `distance` | 15 | 15 |
| 2 | Implement `k-NN` methods | 35 | 30 |
| 3 | Model evaluation | 25 | 20 |
| 5 | ROC curve analysis | 25 | 25 |
### Mandatory for 878, bonus for 478
| | Tasks | 478 | 878 |
|---|----------------|-----|-----|
| 4 | Optimizing *k* | 10 | 10 |
### Bonus for 478/878
| | Tasks | 478 | 878 |
|---|----------------|-----|-----|
| 6 | Assess suitability of *k*-NN | 10 | 10 |
Points are broken down further below in Rubric sections. The **first** score is for 478, the **second** is for 878 students. There are a total of 100 points in this assignment and extra 20 bonus points for 478 students and 10 bonus points for 878 students.
%% Cell type:markdown id: tags:
# YOUR GRADE
### Group Members:
| | Tasks | Points |
|---|----------------------------|-----|
| 1 | Implement `distance` | |
| 2 | Implement `k-NN` methods | |
| 3 | Model evaluation | |
| 4 | Learning curve | |
| 5 | Optimizing *k* | |
| 6 | ROC curve analysis | |
| 7 | Assess suitability of *k*-NN| |
| 4 | Optimizing *k* | |
| 5 | ROC curve analysis | |
| 6 | Assess suitability of *k*-NN| |
%% Cell type:markdown id: tags:
You can use numpy for array operations and matplotlib for plotting for this assignment. Please do not add other libraries.
%% Cell type:code id: tags:
``` python
importnumpyasnp
importmatplotlib.pyplotasplt
```
%% Cell type:markdown id: tags:
Following code makes the Model class and relevant functions available from model.ipynb.
%% Cell type:code id: tags:
``` python
%run'model.ipynb'
```
%% Cell type:markdown id: tags:
## TASK 1: Implement `distance` function
%% Cell type:markdown id: tags:
Choice of distance metric plays an important role in the performance of *k*-NN. Let's start with implementing a distance method in the "distance" function in **model.ipynb**. It should take two data points and the name of the metric and return a scalar value.
We can start implementing our *k*-NN classifier. *k*-NN class inherits Model class. Use the "distance" function you defined above. "fit" method takes *k* as an argument. "predict" takes as input an *mxd* array containing *d*-dimensional *m* feature vectors for examples and outputs the predicted class and the ratio of positive examples in *k* nearest neighbors.
%% Cell type:markdown id: tags:
### Rubric:
* correct implementation of fit method +5, +5
* correct implementation of predict method +20, +15
%% Cell type:code id: tags:
``` python
classkNN(Model):
'''
Inherits Model class. Implements the k-NN algorithm for classification.
Now that we have the true labels and the predicted ones from our model, we can build a confusion matrix and see how accurate our model is. Implement the "conf_matrix" function (in model.ipynb) that takes as input an array of true labels (*true*) and an array of predicted labels (*pred*). It should output a numpy.ndarray. You do not need to change the value of the threshold parameter yet.
# For now, we will consider a data point as predicted in a class if more than 0.5
# of its k-neighbors are in that class.
threshold=0.5
# convert predicted ratios to predicted labels
pred_labels=None
# show the distribution of predicted and true labels in a confusion matrix
confusion=conf_matrix(...)
confusion
```
%% Cell type:markdown id: tags:
Evaluate your model on the test data and report your **accuracy**. Also, calculate and report the 95% confidence interval on the generalization **error** estimate.
%% Cell type:code id: tags:
``` python
# TODO
# Calculate and report accuracy and generalization error with confidence interval here. Show your work in this cell.
A learning curve shows how error changes as the training set size increases. For more information, see [learning curves](https://www.dataquest.io/blog/learning-curves-machine-learning/).
We'll plot the error values for training and validation data while varying the size of the training set. Report a good size for training set for which there is a good balance between bias and variance.
%% Cell type:markdown id: tags:
### Rubric:
* Correct training error calculation for different training set sizes +8, +8
* Correct validation error calculation for different training set sizes +8, +8
* Reasonable learning curve +4, +4
%% Cell type:code id: tags:
``` python
# train using %10, %20, %30, ..., 100% of training data
* Accuracies reported with various *k* values +5, +5
* Confusion matrices shown for various *k* values +5, +5
%% Cell type:markdown id: tags:
We can use the validation set to come up with a *k* value that results in better performance in terms of accuracy.
Below calculate the accuracies for different values of *k* using the validation set. Report a good *k* value and use it in the analyses that follow this section. Report confusion matrix for the new value of *k*.
%% Cell type:code id: tags:
``` python
# TODO
# Change values of k.
# Calculate accuracies for the validation set.
# Report a good k value.
# Calculate the confusion matrix for new k.
```
%% Cell type:markdown id: tags:
## TASK 6: ROC curve analysis
* Correct implementation +20, +20
%% Cell type:markdown id: tags:
ROC curves are a good way to visualize sensitivity vs. 1-specificity for varying cut off points. Now, implement, in **model.ipynb**, a "ROC" function. "ROC" takes a list containing different threshold values to try and returns two arrays; one where each entry is the sensitivity at a given threshold and the other where entries are 1-specificities.
%% Cell type:markdown id: tags:
Use the *k* value you found above, if you completed TASK 5, else use *k* = 10 to plot the ROC curve for values between 0.1 and 1.0.
## TASK 7: Assess suitability of *k*-NN to your dataset
%% Cell type:markdown id: tags:
Use this cell to write about your understanding of why *k*-NN performed well if it did or why not if it didn't. What properties of the dataset could have affected the performance of the algorithm?