Skip to content
Snippets Groups Projects
Commit fa1511cc authored by Zeynep Hakguder's avatar Zeynep Hakguder
Browse files

pa1

parent 13cc1ff9
No related branches found
No related tags found
No related merge requests found
Expand all Show whitespace changes
Inline Side-by-side
ProgrammingAssignment_1/ProgrammingAssignment1.ipynb
+ 89
− 32
View file @ fa1511cc
%% Cell type:markdown id: tags:
# *k*-Nearest Neighbor
We'll implement *k*-Nearest Neighbor (*k*-NN) algorithm for this assignment. We recommend using [Madelon](https://archive.ics.uci.edu/ml/datasets/Madelon) dataset, although it is not mandatory. If you choose to use a different dataset, it should meet the following criteria:
* dependent variable should be binary (suited for binary classification)
* number of features (attributes) should be at least 50
* number of examples (instances) should be between 1,000 - 5,000
A skeleton of a general supervised learning model is provided in "model.ipynb". Please look through it and complete the "preprocess" and "partition" methods.
A skeleton of a general supervised learning model is provided in "model.ipynb". The functions that will be implemented there will be indicated in this notebook.
### Assignment Goals:
In this assignment, we will:
* learn to split a dataset into training/validation/test partitions
* we'll implement 'Euclidean' and 'Manhattan' distance metrics
* use the validation dataset to find a good value for *k*
* Having found the "best" *k*, we'll obtain final performance measures:
* Evaluate our model with respect to performance measures:
* accuracy, generalization error and ROC curve
%% Cell type:markdown id: tags:
# GRADING
You will be graded on parts that are marked with **\#TODO** comments. Read the comments in the code to make sure you don't miss any.
### Mandatory for 478 & 878:
| | Tasks | 478 | 878 |
|---|----------------------------|-----|-----|
| 1 | Implement `distance` | 10 | 10 |
| 2 | Implement `k-NN` methods | 25 | 20 |
| 3 | Model evaluation | 25 | 20 |
| 4 | Learning curve | 20 | 20 |
| 6 | ROC curve analysis | 20 | 20 |
### Mandatory for 878, bonus for 478
| | Tasks | 478 | 878 |
|---|----------------|-----|-----|
| 5 | Optimizing *k* | 10 | 10 |
### Bonus for 478/878
Points are broken down further below in Rubric sections. The **first** score is for 478, the **second** is for 878 students. There a total of 100 points in this assignment and extra 10 bonus points for 478 students.
| | Tasks | 478 | 878 |
|---|----------------|-----|-----|
| 7 | Assess suitability of *k*-NN | 10 | 10 |
Points are broken down further below in Rubric sections. The **first** score is for 478, the **second** is for 878 students. There a total of 100 points in this assignment and extra 20 bonus points for 478 students and 10 bonus points for 878 students.
%% Cell type:markdown id: tags:
You can use numpy for array operations and matplotlib for plotting for this assignment. Please do not add other libraries.
%% Cell type:code id: tags:
``` python
import numpy as np
import matplotlib.pyplot as plt
```
%% Cell type:markdown id: tags:
Following code makes the Model class and relevant functions available from model.ipynb.
%% Cell type:code id: tags:
``` python
%run 'model.ipynb'
```
%% Cell type:markdown id: tags:
## TASK 1: Implement `distance` function
%% Cell type:markdown id: tags:
Choice of distance metric plays an important role in the performance of *k*-NN. Let's start with implementing a distance method in the "distance" function in **model.ipynb**. It should take two data points and the name of the metric and return a scalar value.
Choice of distance metric plays an important role in the performance of *k*-NN. Let's start with implementing a distance method in the "distance" function in **model.ipynb**. It should take two data points and the name of the metric and return a scalar value.
%% Cell type:markdown id: tags:
### Rubric:
* Euclidean +5, +5
* Hamming +5, +5
* Manhattan +5, +5
%% Cell type:markdown id: tags:
### Test `distance`
%% Cell type:code id: tags:
``` python
x = np.array(range(100))
y = np.array(range(100, 200))
dist_euclidean = distance(x, y, 'Euclidean')
dist_hamming = distance(x, y, 'Hamming')
dist_manhattan = distance(x, y, 'Manhattan')
print('Euclidean distance: {}, Manhattan distance: {}'.format(dist_euclidean, dist_manhattan))
```
%% Cell type:markdown id: tags:
## TASK 2: Implement $k$-NN Class Methods
%% Cell type:markdown id: tags:
We can start implementing our *k*-NN classifier. *k*-NN class inherits Model class. You'll need to implement "fit" and "predict" methods. Use the "distance" function you defined above. "fit" method takes *k* as an argument. "predict" takes as input an *mxd* array containing *d*-dimensional *m* feature vectors for examples and outputs the predicted class and the ratio of positive examples in *k* nearest neighbors.
We can start implementing our *k*-NN classifier. *k*-NN class inherits Model class. Use the "distance" function you defined above. "fit" method takes *k* as an argument. "predict" takes as input an *mxd* array containing *d*-dimensional *m* feature vectors for examples and outputs the predicted class and the ratio of positive examples in *k* nearest neighbors.
%% Cell type:markdown id: tags:
### Rubric:
* correct implementation of fit method +5, +5
* correct implementation of predict method +20, +15
%% Cell type:code id: tags:
``` python
class kNN(Model):
'''
Inherits Model class. Implements the k-NN algorithm for classification.
'''
def fit(self, k, distance_f, **kwargs):
def fit(self, training_features, training_labels, k, distance_f,**kwargs):
'''
Fit the model. This is pretty straightforward for k-NN.
Args:
training_features: ndarray
training_labels: ndarray
k: int
distance_f: function
kwargs: dict
Contains keyword arguments that will be passed to distance_f
'''
# TODO
# set self.k, self.distance_f, self.distance_metric
# set self.train_features, self.train_labels, self.k, self.distance_f, self.distance_metric
raise NotImplementedError
return
def predict(self, test_indices):
def predict(self, test_features):
'''
Args:
test_features: ndarray
mxd array containing features for the points to be predicted
Returns:
ndarray
'''
raise NotImplementedError
pred = []
# TODO
# for each point in test points
# for each point in test_features
# use your implementation of distance function
# distance_f(..., distance_metric)
# to find the labels of k-nearest neighbors.
# Find the ratio of the positive labels
# and append to pred with pred.append(ratio).
# when calculating learning curve you can make use of
# self.learning_curve and self.training_proportion
return np.array(pred)
```
%% Cell type:markdown id: tags:
## TASK 3: Build and Evaluate the Model
%% Cell type:markdown id: tags:
### Rubric:
* Reasonable accuracy values +10, +5
* Reasonable confidence intervals on the error estimate +10, +10
* Reasonable confusion matrix +5, +5
%% Cell type:markdown id: tags:
Remember you need to provide values to *t*, *v* parameters for "partition" function and to *feature_file* and *label_file* for "preprocess" function.
Preprocess the data files and partition the data.
%% Cell type:code id: tags:
``` python
# populate the keyword arguments dictionary kwargs
kwargs = {'t': 0.3, 'v': 0.1, 'feature_file': ..., 'label_file': ...}
# initialize the model
my_model = kNN(preprocessor_f=preprocess, partition_f=partition, **kwargs)
my_model = kNN()
# obtain features and labels from files
features, labels = preprocess(feature_file=..., label_file=...)
# partition the data set
val_indices, test_indices, train_indices = partition(size=..., t = 0.3, v = 0.1)
```
%% Cell type:markdown id: tags:
Assign a value to *k* and fit the *k*-NN model.
%% Cell type:code id: tags:
``` python
# pass the training features and labels to the fit method
kwargs_f = {'metric': 'Euclidean'}
my_model.fit(k = 10, distance_f=distance, **kwargs_f)
my_model.fit(training_features=..., training_labels-..., k=10, distance_f=..., **kwargs_f)
```
%% Cell type:markdown id: tags:
Evaluate your model on the test data and report your **accuracy**. Also, calculate and report the confidence interval on the generalization **error** estimate.
### Computing the confusion matrix for *k* = 10
Now that we have the true labels and the predicted ones from our model, we can build a confusion matrix and see how accurate our model is. Implement the "conf_matrix" function (in model.ipynb) that takes as input an array of true labels (*true*) and an array of predicted labels (*pred*). It should output a numpy.ndarray. You do not need to change the value of the threshold parameter yet.
%% Cell type:code id: tags:
``` python
final_labels = my_model.predict(my_model.test_indices)
# TODO
# get model predictions
pred_ratios = my_model.predict(my_model.features[my_model.test_indices])
# For now, we will consider a data point as predicted in the positive class if more than 0.5
# of its k-neighbors are positive.
threshold = 0.5
# convert predicted ratios to predicted labels
pred_labels = None
# TODO
# Calculate and report accuracy and generalization error with confidence interval here. Show your work in this cell.
# obtain true positive, true negative,
#false positive and false negative counts using conf_matrix
tp,tn, fp, fn = conf_matrix(...)
```
%% Cell type:markdown id: tags:
### Computing the confusion matrix for *k* = 10
Now that we have the true labels and the predicted ones from our model, we can build a confusion matrix and see how accurate our model is. Implement the "conf_matrix" function (in model.ipynb) that takes as input an array of true labels (*true*) and an array of predicted labels (*pred*). It should output a numpy.ndarray. You do not need to change the value of the threshold parameter yet.
Evaluate your model on the test data and report your **accuracy**. Also, calculate and report the 95% confidence interval on the generalization **error** estimate.
%% Cell type:code id: tags:
``` python
# TODO
conf_matrix(my_model.labels[my_model.test_indices], final_labels, threshold = 0.5)
# Calculate and report accuracy and generalization error with confidence interval here. Show your work in this cell.
print('Accuracy: {}'.format(accuracy))
print('Confidence interval: {}-{}'.format(lower_bound, upper_bound))
```
%% Cell type:markdown id: tags:
## TASK 4: Plotting a learning curve
A learning curve shows how error changes as the training set size increases. For more information, see [learning curves](https://www.dataquest.io/blog/learning-curves-machine-learning/).
We'll plot the error values for training and validation data while varying the size of the training set. Report a good size for training set for which there is a good balance between bias and variance.
%% Cell type:markdown id: tags:
### Rubric:
* Correct training error calculation for different training set sizes +8, +8
* Correct validation error calculation for different training set sizes +8, +8
* Reasonable learning curve +4, +4
%% Cell type:code id: tags:
``` python
# try sizes 50, 100, 150, 200, ..., up to the largest multiple of 50 >= train_size
training_sizes = np.arange(50, my_model.train_size + 1, 50)
# train using %10, %20, %30, ..., 100% of training data
training_proportions = np.arange(0.10, 1.01, 0.10)
train_size = len(train_indices)
training_sizes = np.int(np.ceil(size*proportion))
# TODO
error_train = []
error_val = []
# Calculate error for each entry in training_sizes
# for training and validation sets and populate
# error_train and error_val arrays. Each entry in these arrays
# should correspond to each entry in training_sizes.
# For each size in training_sizes
for size in training_sizes:
# fit the model using "size" data porint
# Calculate error for training and validation sets
# populate error_train and error_val arrays.
# Each entry in these arrays
# should correspond to each entry in training_sizes.
plt.plot(training_sizes, error_train, 'r', label = 'training_error')
plt.plot(training_sizes, error_val, 'g', label = 'validation_error')
plt.legend()
plt.show()
```
%% Cell type:markdown id: tags:
## TASK 5: Determining *k*
%% Cell type:markdown id: tags:
### Rubric:
* Increased accuracy with new *k* +5, +5
* Improved confusion matrix +5, +5
%% Cell type:markdown id: tags:
We can use the validation set to come up with a *k* value that results in better performance in terms of accuracy.
Below calculate the accuracies for different values of *k* using the validation set. Report a good *k* value and use it in the analyses that follow this section. Report confusion matrix for the new value of *k*.
%% Cell type:code id: tags:
``` python
# TODO
# Change values of k.
# Calculate accuracies for the validation set.
# Report a good k value.
# Calculate the confusion matrix for new k.
```
%% Cell type:markdown id: tags:
## TASK 6: ROC curve analysis
* Correct implementation +20, +20
%% Cell type:markdown id: tags:
### ROC curve and confusion matrix for the final model
ROC curves are a good way to visualize sensitivity vs. 1-specificity for varying cut off points. Now, implement, in "model.ipynb", a "ROC" function that predicts the labels of the test set examples using different *threshold* values in "predict" and plot the ROC curve. "ROC" takes a list containing different *threshold* parameter values to try and returns two arrays; one where each entry is the sensitivity at a given threshold and the other where entries are 1-specificities.
%% Cell type:markdown id: tags:
We can finally create the confusion matrix and plot the ROC curve for our optimal *k*-NN classifier. Use the *k* value you found above, if you completed TASK 5, else use *k* = 10. We'll plot the ROC curve for values between 0.1 and 1.0.
%% Cell type:code id: tags:
``` python
# TODO
# ROC curve
roc_sens, roc_spec_ = ROC(my_model, my_model.test_indices, np.arange(0.1, 1.0, 0.1))
plt.plot(roc_sens, roc_spec_)
plt.show()
```
%% Cell type:markdown id: tags:
## TASK 7: Assess suitability of *k*-NN to your dataset
%% Cell type:markdown id: tags:
Use this cell to write about your understanding of why *k*-NN performed well if it did or why not if it didn't.
......
ProgrammingAssignment_1/ProgrammingAssignment1_solution.ipynb 0 → 100644
+ 581
− 0
View file @ fa1511cc
This diff is collapsed.
ProgrammingAssignment_1/model.ipynb
+ 55
− 102
View file @ fa1511cc
%% Cell type:markdown id: tags:
# JUPYTER NOTEBOOK TIPS
Each rectangular box is called a cell.
* ctrl+ENTER evaluates the current cell; if it contains Python code, it runs the code, if it contains Markdown, it returns rendered text.
* alt+ENTER evaluates the current cell and adds a new cell below it.
* If you click to the left of a cell, you'll notice the frame changes color to blue. You can erase a cell by hitting 'dd' (that's two "d"s in a row) when the frame is blue.
%% Cell type:markdown id: tags:
# Supervised Learning Model Skeleton
We'll use this skeleton for implementing different supervised learning algorithms. Please complete "preprocess" and "partition" methods below.
We'll use this skeleton for implementing different supervised learning algorithms.
%% Cell type:code id: tags:
``` python
class Model:
# preprocess_f and partition_f expect functions
# use kwargs to pass arguments to preprocessor_f and partition_f
# kwargs is a dictionary and should contain t, v, feature_file, label_file
# e.g. {'t': 0.3, 'v': 0.1, 'feature_file': 'some_file_name', 'label_file': 'some_file_name'}
def __init__(self, preprocessor_f, partition_f, **kwargs):
self.features, self.labels = preprocessor_f(kwargs['feature_file'], kwargs['label_file'])
self.size = len(self.labels) # number of examples in dataset
self.feat_dim = self.features.shape[1] # number of features
self.val_indices, self.test_indices = partition_f(self.size, kwargs['t'], kwargs['v'])
self.val_size = len(self.val_indices)
self.test_size = len(self.test_indices)
self.train_indices = np.delete(np.arange(self.size), np.append(self.test_indices, self.val_indices), 0)
self.train_size = len(self.train_indices)
def fit(self):
raise NotImplementedError
def predict(self, test_points):
raise NotImplementedError
```
%% Cell type:code id: tags:
``` python
#TODO: GIVE!
def preprocess(feature_file, label_file):
'''
Args:
feature_file: str
file containing features
label_file: str
file containing labels
Returns:
features: ndarray
nxd features
labels: ndarray
nx1 labels
'''
# You might find np.genfromtxt useful for reading in the file. Be careful with the file delimiter,
# e.g. for comma-separated files use delimiter=',' argument.
# TODO
raise NotImplementedError
# read in features and labels
features = np.genfromtxt(feature_file)
labels = np.genfromtxt(label_file)
return features, labels
```
%% Cell type:code id: tags:
``` python
#TODO: GIVE!
def partition(size, t, v = 0):
'''
Args:
size: int
number of examples in the whole dataset
t: float
proportion kept for test
v: float
proportion kept for validation
Returns:
test_indices: ndarray
1D array containing test set indices
val_indices: ndarray
1D array containing validation set indices
'''
# np.random.permutation might come in handy. Do not sample with replacement!
# Be sure not to use the same indices in test and validation sets!
# number of test and validation examples
t_size = np.int(np.ceil(size*t))
v_size = np.int(np.ceil(size*v))
# shuffle the indices
permuted = np.random.permutation(size)
# spare the first t_size for test
test_indices = permuted[:t_size]
# and the next v_size for validation
val_indices = permuted[t_size+1:t_size+v_size+1]
# use the first np.ceil(size*t) for test,
# the following np.ceil(size*v) for validation set.
# TODO
raise NotImplementedError
return test_indices, val_indices
```
%% Cell type:markdown id: tags:
In cases, where data is not abundantly available, we resort to getting an error estimate from average of error on different splits of error. In this case, every fold of data is used for testing and for training in turns, i.e. assuming we split our data into 3 folds, we'd
* train our model on fold-1+fold-2 and test on fold-3
* train our model on fold-1+fold-3 and test on fold-2
* train our model on fold-2+fold-3 and test on fold-1.
We'd use the average of the error we obtained in three runs as our error estimate. Implement function "kfold" below.
%% Cell type:code id: tags:
``` python
# TODO: Programming Assignment 2
def kfold(indices, k):
'''
Args:
indices: ndarray
1D array with integer entries containing indices
k: int
Number of desired splits in data.(Assume test set is already separated.)
Returns:
fold_dict: dict
A dictionary with integer keys corresponding to folds. Values are (training_indices, val_indices).
val_indices: ndarray
1/k of training indices randomly chosen and separates them as validation partition.
train_indices: ndarray
Remaining 1-(1/k) of the indices.
e.g. fold_dict = {0: (train_0_indices, val_0_indices),
1: (train_0_indices, val_0_indices), 2: (train_0_indices, val_0_indices)} for k = 3
'''
return fold_dict
```
## TASK 1: Implement `distance` function
%% Cell type:markdown id: tags:
Choice of distance metric plays an important role in the performance of *k*-NN. Let's start with implementing a distance method in the "distance" function below. It should take two data points and the name of the metric and return a scalar value.
"distance" function will be used in calculating cost of *k*-NN. It should take two data points and the name of the metric and return a scalar value.
%% Cell type:code id: tags:
``` python
#TODO: Programming Assignment 1
def distance(x, y, metric):
'''
Args:
x: ndarray
1D array containing coordinates for a point
y: ndarray
1D array containing coordinates for a point
metric: str
Euclidean, Hamming
Euclidean, Manhattan
Returns:
dist: float
'''
if metric == 'Euclidean':
raise NotImplementedError
elif metric == 'Hammming':
elif metric == 'Manhattan':
raise NotImplementedError
else:
raise ValueError('{} is not a valid metric.'.format(metric))
return dist # scalar distance btw x and y
```
%% Cell type:markdown id: tags:
We will extend "Model" class while implementing supervised learning algorithms.
%% Cell type:code id: tags:
``` python
class Model:
# preprocess_f and partition_f expect functions
# use kwargs to pass arguments to preprocessor_f and partition_f
# kwargs is a dictionary and should contain t, v, feature_file, label_file
# e.g. {'t': 0.3, 'v': 0.1, 'feature_file': 'some_file_name', 'label_file': 'some_file_name'}
def __init__(self, preprocessor_f, partition_f, **kwargs):
self.features, self.labels = preprocessor_f(kwargs['feature_file'], kwargs['label_file'])
self.size = len(self.labels) # number of examples in dataset
self.feat_dim = self.features.shape[1] # number of features
self.val_indices, self.test_indices = partition_f(self.size, kwargs['t'], kwargs['v'])
self.val_size = len(self.val_indices)
self.test_size = len(self.test_indices)
self.train_indices = np.delete(np.arange(self.size), np.append(self.test_indices, self.val_indices), 0)
self.train_size = len(self.train_indices)
def fit(self):
raise NotImplementedError
def predict(self, test_points):
raise NotImplementedError
```
%% Cell type:markdown id: tags:
## General supervised learning performance related functions
### (To be implemented later when it is indicated in other notebooks)
%% Cell type:markdown id: tags:
Implement the "conf_matrix" function that takes as input an array of true labels (*true*) and an array of predicted labels (*pred*). It should output a numpy.ndarray.
%% Cell type:code id: tags:
``` python
# TODO: Programming Assignment 1
def conf_matrix(true, pred):
'''
Args:
true: ndarray
nx1 array of true labels for test set
pred: ndarray
nx1 array of predicted labels for test set
Returns:
ndarray
'''
raise NotImplementedError
tp = tn = fp = fn = 0
# calculate true positives (tp), true negatives(tn)
# false positives (fp) and false negatives (fn)
# returns the confusion matrix as numpy.ndarray
return np.array([tp,tn, fp, fn])
```
%% Cell type:markdown id: tags:
ROC curves are a good way to visualize sensitivity vs. 1-specificity for varying cut off points. Now, implement a "ROC" function that predicts the labels of the test set examples using different *threshold* values in "predict" and plot the ROC curve. "ROC" takes a list containing different *threshold* parameter values to try and returns two arrays; one where each entry is the sensitivity at a given threshold and the other where entries are 1-specificities.
%% Cell type:code id: tags:
``` python
# TODO: Programming Assignment 1
def ROC(model, indices, value_list):
'''
Args:
model: a fitted supervised learning model
indices: ndarray
1D array containing indices
value_list: ndarray
1D array containing different threshold values
Returns:
sens: ndarray
1D array containing sensitivities
spec_: ndarray
1D array containing 1-specifities
'''
# use predict method to obtain predicted labels at different threshold values
# use conf_matrix to calculate tp, tn, fp, fn
# calculate sensitivity, 1-specificity
# return two arrays
raise NotImplementedError
return sens, spec_
```
......
ProgrammingAssignment_1/model_solution.ipynb
+ 77
− 116
View file @ fa1511cc
%% Cell type:markdown id: tags:
# JUPYTER NOTEBOOK TIPS
Each rectangular box is called a cell.
* ctrl+ENTER evaluates the current cell; if it contains Python code, it runs the code, if it contains Markdown, it returns rendered text.
* alt+ENTER evaluates the current cell and adds a new cell below it.
* If you click to the left of a cell, you'll notice the frame changes color to blue. You can erase a cell by hitting 'dd' (that's two "d"s in a row) when the frame is blue.
%% Cell type:markdown id: tags:
# Supervised Learning Model Skeleton
We'll use this skeleton for implementing different supervised learning algorithms. Please complete "preprocess" and "partition" methods below.
We'll use this skeleton for implementing different supervised learning algorithms.
%% Cell type:code id: tags:
``` python
class Model:
def fit(self):
raise NotImplementedError
def predict(self, test_points):
raise NotImplementedError
```
%% Cell type:code id: tags:
``` python
def preprocess(feature_file, label_file):
'''
Args:
feature_file: str
file containing features
label_file: str
file containing labels
Returns:
features: ndarray
nxd features
labels: ndarray
nx1 labels
'''
# You might find np.genfromtxt useful for reading in the file. Be careful with the file delimiter,
# e.g. for comma-separated files use delimiter=',' argument.
# TODO
raise NotImplementedError
# read in features and labels
features = np.genfromtxt(feature_file)
labels = np.genfromtxt(label_file)
return features, labels
```
%% Cell type:code id: tags:
``` python
#TODO: GIVE!
def partition(size, t, v = 0):
'''
Args:
size: int
number of examples in the whole dataset
t: float
proportion kept for test
v: float
proportion kept for validation
Returns:
test_indices: ndarray
1D array containing test set indices
val_indices: ndarray
1D array containing validation set indices
'''
# np.random.permutation might come in handy. Do not sample with replacement!
# Be sure not to use the same indices in test and validation sets!
# use the first np.ceil(size*t) for test,
# the following np.ceil(size*v) for validation set.
# number of test and validation examples
t_size = np.int(np.ceil(size*t))
v_size = np.int(np.ceil(size*v))
# TODO
# shuffle the indices
permuted = np.random.permutation(size)
raise NotImplementedError
# spare the first t_size for test
test_indices = permuted[:t_size]
# and the next v_size for validation
val_indices = permuted[t_size+1:t_size+v_size+1]
train_indices = np.delete(np.arange(size), np.append(test_indices, val_indices), 0)
return test_indices, val_indices
return test_indices, val_indices, train_indices
```
%% Cell type:markdown id: tags:
In cases, where data is not abundantly available, we resort to getting an error estimate from average of error on different splits of error. In this case, every fold of data is used for testing and for training in turns, i.e. assuming we split our data into 3 folds, we'd
* train our model on fold-1+fold-2 and test on fold-3
* train our model on fold-1+fold-3 and test on fold-2
* train our model on fold-2+fold-3 and test on fold-1.
We'd use the average of the error we obtained in three runs as our error estimate. Implement function "kfold" below.
%% Cell type:code id: tags:
``` python
# TODO: Programming Assignment 2
def kfold(indices, k):
'''
Args:
indices: ndarray
1D array with integer entries containing indices
k: int
Number of desired splits in data.(Assume test set is already separated.)
Returns:
fold_dict: dict
A dictionary with integer keys corresponding to folds. Values are (training_indices, val_indices).
val_indices: ndarray
1/k of training indices randomly chosen and separates them as validation partition.
train_indices: ndarray
Remaining 1-(1/k) of the indices.
e.g. fold_dict = {0: (train_0_indices, val_0_indices),
1: (train_0_indices, val_0_indices), 2: (train_0_indices, val_0_indices)} for k = 3
'''
return fold_dict
```
## TASK 1: Implement `distance` function
%% Cell type:markdown id: tags:
Choice of distance metric plays an important role in the performance of *k*-NN. Let's start with implementing a distance method in the "distance" function below. It should take two data points and the name of the metric and return a scalar value.
"distance" function will be used in calculating cost of *k*-NN. It should take two data points and the name of the metric and return a scalar value.
%% Cell type:code id: tags:
``` python
#TODO: Programming Assignment 1
def distance(x, y, metric):
'''
Args:
x: ndarray
1D array containing coordinates for a point
y: ndarray
1D array containing coordinates for a point
metric: str
Euclidean, Hamming
Returns:
dist: float
'''
if metric == 'Euclidean':
raise NotImplementedError
elif metric == 'Hammming':
raise NotImplementedError
dist = np.sqrt(np.sum(np.square((x-y))))
elif metric == 'Manhattan':
dist = np.sum(abs(x-y))
else:
raise ValueError('{} is not a valid metric.'.format(metric))
return dist # scalar distance btw x and y
```
%% Cell type:markdown id: tags:
We will extend "Model" class while implementing supervised learning algorithms.
%% Cell type:code id: tags:
``` python
class Model:
# preprocess_f and partition_f expect functions
# use kwargs to pass arguments to preprocessor_f and partition_f
# kwargs is a dictionary and should contain t, v, feature_file, label_file
# e.g. {'t': 0.3, 'v': 0.1, 'feature_file': 'some_file_name', 'label_file': 'some_file_name'}
def __init__(self, preprocessor_f, partition_f, **kwargs):
self.features, self.labels = preprocessor_f(kwargs['feature_file'], kwargs['label_file'])
self.size = len(self.labels) # number of examples in dataset
self.feat_dim = self.features.shape[1] # number of features
self.val_indices, self.test_indices = partition_f(self.size, kwargs['t'], kwargs['v'])
self.val_size = len(self.val_indices)
self.test_size = len(self.test_indices)
self.train_indices = np.delete(np.arange(self.size), np.append(self.test_indices, self.val_indices), 0)
self.train_size = len(self.train_indices)
def fit(self):
raise NotImplementedError
def predict(self, test_points):
raise NotImplementedError
```
%% Cell type:markdown id: tags:
## General supervised learning performance related functions
### (To be implemented later when it is indicated in other notebooks)
%% Cell type:markdown id: tags:
Implement the "conf_matrix" function that takes as input an array of true labels (*true*) and an array of predicted labels (*pred*). It should output a numpy.ndarray.
%% Cell type:code id: tags:
``` python
# TODO: Programming Assignment 1
def conf_matrix(true, pred):
'''
Args:
true: ndarray
nx1 array of true labels for test set
pred: ndarray
nx1 array of predicted labels for test set
Returns:
ndarray
'''
raise NotImplementedError
tp = tn = fp = fn = 0
# calculate true positives (tp), true negatives(tn)
# false positives (fp) and false negatives (fn)
size = len(true)
for i in range(size):
if true[i]==1:
if pred[i] > 0:
tp += 1
else:
fn += 1
else:
if pred[i] == 0:
tn += 1
else:
fp += 1
# returns the confusion matrix as numpy.ndarray
return np.array([tp,tn, fp, fn])
```
%% Cell type:markdown id: tags:
ROC curves are a good way to visualize sensitivity vs. 1-specificity for varying cut off points. Now, implement a "ROC" function that predicts the labels of the test set examples using different *threshold* values in "predict" and plot the ROC curve. "ROC" takes a list containing different *threshold* parameter values to try and returns two arrays; one where each entry is the sensitivity at a given threshold and the other where entries are 1-specificities.
%% Cell type:code id: tags:
``` python
# TODO: Programming Assignment 1
def ROC(model, indices, value_list):
def ROC(true_labels, preds, value_list):
'''
Args:
model: a fitted supervised learning model
indices: ndarray
1D array containing indices
true_labels: ndarray
1D array containing true labels
preds: ndarray
1D array containing thresholded value (e.g. proportion of positive neighbors in kNN)
value_list: ndarray
1D array containing different threshold values
Returns:
sens: ndarray
1D array containing sensitivities
spec_: ndarray
1D array containing 1-specifities
'''
# use predict method to obtain predicted labels at different threshold values
# use conf_matrix to calculate tp, tn, fp, fn
# calculate sensitivity, 1-specificity
# return two arrays
sens = []
spec_ = []
for threshold in value_list:
pred_labels = [1 if x >= threshold else 0 for x in pred_ratios]
tp,tn, fp, fn = conf_matrix(true_labels, pred_labels)
se = tp/(tp+fn)
sens.append(se)
spec = tn/(tn+fp)
spec_.append(1 - spec)
raise NotImplementedError
return sens, spec_
return np.array(sens), np.array(spec_)
```
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment