Skip to content
Snippets Groups Projects
Commit 22dd3471 authored by Zeynep Hakguder's avatar Zeynep Hakguder
Browse files

added Naive Bayes

parent 763d501b
Branches
No related tags found
No related merge requests found
Show whitespace changes
Inline Side-by-side
ProgrammingAssignment1.ipynb
+ 1
− 1
View file @ 22dd3471
%% Cell type:markdown id: tags:
# $k$-Nearest Neighbor
We'll implement $k$-Nearest Neighbor ($k$-NN) algorithm for this assignment. We recommend using [Madelon](https://archive.ics.uci.edu/ml/datasets/Madelon) dataset, although it is not mandatory. If you choose to use a different dataset, it should meet the following criteria:
* dependent variable should be binary (suited for binary classification)
* number of features (attributes) should be at least 50
* number of examples (instances) should be at least 1,000
A skeleton of a general supervised learning model is provided in "model.ipynb". Please look through it and complete the "preprocess" and "partition" methods.
### Assignment Goals:
In this assignment, we will:
* learn to split a dataset into training/validation/test partitions
* use the validation dataset to find a good value for $k$
* Having found the "best" $k$, we'll obtain final performance measures:
* accuracy, generalization error and ROC curve
%% Cell type:markdown id: tags:
You can use numpy for array operations and matplotlib for plotting for this assignment. Please do not add other libraries.
%% Cell type:code id: tags:
``` python
import numpy as np
import matplotlib.pyplot as plt
```
%% Cell type:markdown id: tags:
Following code makes the Model class and relevant functions available from model.ipynb.
%% Cell type:code id: tags:
``` python
%run 'model.ipynb'
```
%% Cell type:markdown id: tags:
Choice of distance metric plays an important role in the performance of $k$-NN. Let's start with implementing a distance method in the "distance" function below. It should take two data points and the name of the metric and return a scalar value.
%% Cell type:code id: tags:
``` python
def distance(x, y, metric):
'''
x: a 1xd array
y: a 1xd array
metric: Euclidean, Hamming, etc.
'''
raise NotImplementedError
return dist # scalar distance btw x and y
```
%% Cell type:markdown id: tags:
### $k$-NN Class Methods
%% Cell type:markdown id: tags:
We can start implementing our $k$-NN classifier. $k$-NN class inherits Model class. You'll need to implement "fit" and "predict" methods. Use the "distance" function you defined above. "fit" method takes $k$ as an argument. "predict" takes as input an $mxd$ array containing $d$-dimensional $m$ feature vectors for examples and outputs the predicted class and the ratio of positive examples in $k$ nearest neighbors.
%% Cell type:code id: tags:
``` python
class kNN(Model):
'''
Inherits Model class. Implements the k-NN algorithm for classification.
'''
def fit(self, k, distance_f, **kwargs):
'''
Fit the model. This is pretty straightforward for k-NN.
'''
# set self.k, self.distance_f, self.distance_metric
raise NotImplementedError
return
def predict(self, test_indices):
raise NotImplementedError
pred = []
# for each point in test points
# use your implementation of distance function
# distance_f(..., distance_metric)
# to find the labels of k-nearest neighbors.
# Find the ratio of the positive labels
# and append to pred with pred.append(ratio).
return np.array(pred)
```
%% Cell type:markdown id: tags:
### Build and Evaluate the Model (Accuracy, Confidence Interval, Confusion Matrix)
%% Cell type:markdown id: tags:
It's time to build and evaluate our model now. Remember you need to provide values to $p$, $v$ parameters for "partition" function and to $file\_path$ for "preprocess" function.
%% Cell type:code id: tags:
``` python
# populate the keyword arguments dictionary kwargs
kwargs = {'p': 0.3, 'v': 0.1, seed: 123, 'file_path': 'madelon_train'}
# initialize the model
my_model = kNN(preprocessor_f=preprocess, partition_f=partition, **kwargs)
```
%% Cell type:markdown id: tags:
Assign a value to $k$ and fit the kNN model.
%% Cell type:code id: tags:
``` python
kwargs_f = {'metric': 'Euclidean'}
my_model.fit(k = 10, distance_f=distance, **kwargs_f)
```
%% Cell type:markdown id: tags:
Evaluate your model on the test data and report your **accuracy**. Also, calculate and report the confidence interval on the generalization **error** estimate.
%% Cell type:code id: tags:
``` python
final_labels = my_model.predict(my_model.test_indices)
# For now, We will consider a data point as predicted in the positive class if more than 0.5
# of its k-neighbors are positive.
threshold = 0.5
# Calculate accuracy and generalization error with confidence interval here.
```
%% Cell type:markdown id: tags:
### Plotting a learning curve
A learning curve shows how error changes as the training set size increases. For more information, see [learning curves](https://www.dataquest.io/blog/learning-curves-machine-learning/).
We'll plot the error values for training and validation data while varying the size of the training set. Report a good size for training set for which there is a good balance between bias and variance.
%% Cell type:code id: tags:
``` python
# try sizes 0, 100, 200, 300, ..., up to the largest multiple of 100 >= train_size
training_sizes = np.xrange(0, my_model.train_size + 1, 100)
# Calculate error for each entry in training_sizes
# for training and validation sets and populate
# error_train and error_val arrays. Each entry in these arrays
# should correspond to each entry in training_sizes.
plt.plot(training_sizes, error_train, 'r', label = 'training_error')
plt.plot(training_sizes, error_val, 'g', label = 'validation_error')
plt.legend()
plt.show()
```
%% Cell type:markdown id: tags:
### Computing the confusion matrix for $k = 10$
Now that we have the true labels and the predicted ones from our model, we can build a confusion matrix and see how accurate our model is. Implement the "conf_matrix" function (in model.ipynb) that takes as input an array of true labels ($true$) and an array of predicted labels ($pred$). It should output a numpy.ndarray. You do not need to change the value of the threshold parameter yet.
%% Cell type:code id: tags:
``` python
conf_matrix(my_model.labels[my_model.test_indices], final_labels, threshold = 0.5)
```
%% Cell type:markdown id: tags:
### Finding a good value for $k$
We can use the validation set to come up with a $k$ value that results in better performance in terms of accuracy. Additionally, in some cases, predicting examples from a certain class correctly is more critical than other classes. In those cases, we can use the confusion matrix to find a good trade off between correct and wrong predictions and allow more wrong predictions in some classes to predict more examples correctly in a that class.
Below calculate the accuracies for different values of $k$ using the validation set. Report a good $k$ value and use it in the analyses that follow this section.
%% Cell type:code id: tags:
``` python
# Change values of $k.
# Calculate accuracies for the validation set.
# Report a good k value that you'll use in the following analyses.
```
%% Cell type:markdown id: tags:
### ROC curve and confusion matrix for the final model
ROC curves are a good way to visualize sensitivity vs. 1-specificity for varying cut off points. Now, implement, in "model.ipynb", a "ROC" function that predicts the labels of the test set examples using different $threshold$ values in "predict" and plot the ROC curve. "ROC" takes a list containing different $threshold$ parameter values to try and returns two arrays; one where each entry is the sensitivity at a given threshold and the other where entries are 1-specificities.
%% Cell type:markdown id: tags:
We can finally create the confusion matrix and plot the ROC curve for our optimal $k$-NN classifier. (Use the $k$ value you found above.)
We can finally create the confusion matrix and plot the ROC curve for our optimal $k$-NN classifier. (Use the $k$ value you found above.) We'll plot the ROC curve for values between 0.1 and 1.0.
%% Cell type:code id: tags:
``` python
# confusion matrix
conf_matrix(true_classes, predicted_classes)
```
%% Cell type:code id: tags:
``` python
# ROC curve
roc_sens, roc_spec_ = ROC(my_model, my_model.test_indices, np.arange(0.1, 1.0, 0.1))
plt.plot(roc_sens, roc_spec_)
plt.show()
```
......
......
ProgrammingAssignment2.ipynb
+ 245
− 48
View file @ 22dd3471
%% Cell type:markdown id: tags:
# Linear Regression & Naive Bayes
We'll implement linear regression & Naive Bayes algorithms for this assignment. Please modify the "preprocess" and "partition" methods in "model.ipynb" to suit your datasets for this assignment. In this assignment, we have a small dataset available to us. We won't have examples to spare for validation set, instead we'll use cross-validation to tune hyperparameters.
We'll implement linear regression & Naive Bayes algorithms for this assignment. Please modify the "preprocess" in this notebook and "partition" method in "model.ipynb" to suit your datasets for this assignment. In the linear regression part of this assignment, we have a small dataset available to us. We won't have examples to spare for validation set, instead we'll use cross-validation to tune hyperparameters. In our Naive Bayes implementation, we will not use validation set or crossvalidation.
### Assignment Goals:
In this assignment, we will:
* implement linear regression
* use gradient descent for optimization
* use residuals to decide if we need a polynomial model
* change our model to quadratic/cubic regression and use cross-validation to find the "best" polynomial degree
* implement regularization techniques
* $l_1$/$l_2$ regularization
* use cross-validation to find a good regularization parameter $\lambda$
* implement Naive Bayes
* address sparse data problem with **pseudocounts** (**$m$-estimate**)
%% Cell type:markdown id: tags:
You can use numpy for array operations and matplotlib for plotting for this assignment. Please do not add other libraries.
%% Cell type:code id: tags:
``` python
import numpy as np
import matplotlib.pyplot as plt
```
%% Cell type:markdown id: tags:
Following code makes the Model class and relevant functions available from "model.ipynb".
%% Cell type:code id: tags:
``` python
%run 'model.ipynb'
```
%% Cell type:markdown id: tags:
We'll implement the "preprocess" function and "kfold" function for $k$-fold cross-validation in "model.ipynb". 5 and 10 are commonly used values for $k$. You can use either one of them.
%% Cell type:code id: tags:
``` python
def mse(y_pred, y_true):
def preprocess(file_path):
'''
y_hat: values predicted by our method
y_true: true y values
file_path: where to read the dataset from
Returns:
features: ndarray
nxd array containing `float` feature values
labels: ndarray
1D array containing `float` label
'''
# You might find np.genfromtxt useful for reading in the file. Be careful with the file delimiter,
# e.g. for comma-separated files use delimiter=',' argument.
raise NotImplementedError
# returns mean squared error between y_pred and y_true
return cost
return features, labels
```
%% Cell type:markdown id: tags:
We'll start by implementing a partition function for $k$-fold cross-validation. $5$ and $10$ are commonly used values for $k$. You can use either one of them.
We'll need to use mean squared error (mse) for linear regression. Next, implement "mse" function that takes predicted and true y values, and returns the "mse" between them.
%% Cell type:code id: tags:
``` python
def kfold(k):
def mse(y_pred, y_true):
'''
k: number of desired splits in data.
Assume test set is already separated.
This function chooses 1/k of training indices randomly and separates them as validation partition.
It returns the 1/k selected indices as val_indices and the remaining 1-(1/k) of the indices as train_indices
Args:
y_hat: ndarray
1D array containing data with `float` type. Values predicted by our method
y_true: ndarray
1D array containing data with `float` type. True y values
Returns:
cost: float
A single value. Mean squared error between y_pred and y_true.
'''
raise NotImplementedError
return cost
return train_indices, val_indices
```
%% Cell type:markdown id: tags:
We can define our linear_regression model class now. Implement the "fit" and "predict" methods. Keep the default values for now, later we'll change the $polynomial\_degree$. If your "kfold" implementation works as it should, each call to fit and predict
%% Cell type:code id: tags:
``` python
class linear_regression(Model):
def __init__(self, preprocessor_f, partition_f, **kwargs):
super().__init__(preprocessor_f, partition_f, **kwargs)
theta =
if k_fold:
self.data_dict = kfold(self.train_indices, k = kwargs['k'])
# counter for train fold
self.i = 0
# counter for test fold
self.j = 0
# You can disregard polynomial_degree and regularizer in your first pass
def fit(self, learning_rate = 0.001, epochs = 1000, regularizer=None, polynomial_degree=1, **kwargs):
train_features = self.train_features[self.data_dict[self.i]]
train_labels = self.train_labels[self.data_dict[self.i]]
#initialize theta_cur randomly
# for each epoch
# compute y_hat array which holds model predictions for training examples
# compute model predictions for training examples
y_hat = None
# use mse function to find the cost
cost = None
# calculate gradients wrt theta
grad_theta = None
# update theta
theta_curr = None
raise NotImplementedError
return theta
if regularizer = None:
# use mse function to find the cost
cost = None
# calculate gradients wrt theta
grad_theta = None
# update theta
theta_curr = None
raise NotImplementedError
else:
# take regularization into account
raise NotImplementedError
# update the model parameters to be used in predict method
self.theta = theta_curr
# increment counter for next fold
self.i += 1
def predict(self, indices):
# obtain test features for current fold
test_features = self.train_features[self.data_dict[self.j]]
raise NotImplementedError
# increment counter for next fold
self.j += 1
return y_hat
```
%% Cell type:code id: tags:
``` python
# populate the keyword arguments dictionary kwargs
kwargs = {'p': 0.3, 'v': 0.0, 'file_path': 'mnist_test.csv', 'k': 5}
# p: proportion for test data
# k: parameter for k-fold crossvalidation
kwargs = {'p': 0.3, 'v': 0.1, 'file_path': 'madelon', 'k': 1}
# initialize the model
my_model = linear_regression(preprocessor_f=preprocess, partition_f=partition, k_fold=True, **kwargs)
```
%% Cell type:code id: tags:
``` python
# use fit_kwargs to pass arguments to regularization function
# fit_kwargs is empty for now since we are not applying
# regularization yet
fit_kwargs = {}
my_model.fit(**fit_kwargs)
```
%% Cell type:markdown id: tags:
Residuals are the differences between the predicted value $y_{hat}$ and the true value $y$ for each example. Predict $y_{hat}$ for the validation set. Calculate and plot residuals.
Residuals are the differences between the predicted value $y_{hat}$ and the true value $y$ for each example. Predict $y_{hat}$ for the validation set.
%% Cell type:code id: tags:
``` python
y_hat_val = my_model.predict(my_model.features[my_model.val_indices])
residuals = my_model.labels[my_model.val_indices] - y_hat_val
plt.plot(residuals)
plt.show()
```
%% Cell type:markdown id: tags:
If the data is better suited for quadratic/cubic regression, regions of positive and negative residuals will alternate in the plot. Regardless, modify fit" and "predict" in the class definition to raise the feature values to $polynomial\_degree$. You can directly make the modification in the above definition, do not repeat. Use the validation set to find the degree of polynomial that results in lowest "mse".
If the data is better suited for quadratic/cubic regression, regions of positive and negative residuals will alternate in the plot. Regardless, modify fit" and "predict" in the class definition to raise the feature values to $polynomial\_degree$. You can directly make the modification in the above definition, do not repeat. Use the validation set to find the degree of polynomial that results in lowest _mse_.
%% Cell type:code id: tags:
``` python
# calculate mse for linear model
kwargs = {'p': 0.3, 'file_path': 'madelon', 'k': 5}
# initialize the model
my_model = linear_regression(preprocessor_f=preprocess, partition_f=partition, k_fold=True, **kwargs)
fit_kwargs = {}
my_model.fit(polynomial_degree = 1 ,**fit_kwargs)
pred_3 = my_model.predict(my_model.features[my_model.val_indices])
mse_1 = mse(pred_2, my_model.labels[my_model.val_indices])
# calculate mse for quadratic model
my_model.fit(polynomial_degree = 2 ,**fit_kwargs)
pred_2 = my_model.predict(my_model.features[my_model.val_indices])
mse_2 = mse(pred_2, my_model.labels[my_model.val_indices])
# calculate mse for cubic model
my_model.fit(polynomial_degree = 3 ,**fit_kwargs)
pred_3 = my_model.predict(my_model.features[my_model.val_indices])
mse_3 = mse(pred_2, my_model.labels[my_model.val_indices])
# calculate mse for each of linear model, quadratic and cubic models
# and append to mses_for_models
mses_for_models = []
for i in range(1,4):
kfold_mse = 0
for k in range(5):
my_model.fit(polynomial_degree = i ,**fit_kwargs)
pred = my_model.predict(my_model.features[my_model.val_indices], fold = k)
k_fold_mse += mse(pred, my_model.labels[my_model.val_indices])
mses_for_models_for_models.append(k_fold_mse/k)
```
%% Cell type:markdown id: tags:
Define "regularization" function which implements $l_1$ and $l_2$ regularization. You'll use this function in "fit" method of "linear_regression" class.
%% Cell type:code id: tags:
``` python
def regularization(method):
def regularization(weights, method):
'''
Args:
weights: ndarray
1D array with `float` entries
method: str
Returns:
value: float
A single value. Regularization term that will be used in cost function in fit.
'''
if method == "l1":
value = None
raise NotImplementedError
elif method == "l2":
value = None
raise NotImplementedError
return value
```
%% Cell type:markdown id: tags:
Using crossvalidation and the value of $polynomial_{degree}$ you found above, try different values of $\lambda$ to find a a good value that results in low _mse_. Report the best values you found for hyperparameters and the resulting _mse_.
%% Cell type:markdown id: tags:
## Naive Bayes Spam Classifier
This part is independent of the above part. We will use the Enron spam/ham dataset. You will need to decompress the provided "enron.tar.gz" folder. The two subfolders contain spam and ham emails.
The features for Naive Bayes algorithm will be word counts. Number of features will be equal to the unique words seen in the whole dataset. The "preprocess" function will be more involved this time. You'll need to remove pucntuation marks (you may find string.punctuation useful), tokenize text to words (remember to lowercase all) and count the number of words.
%% Cell type:code id: tags:
``` python
def preprocess_bayes(folder_path):
'''
Args:
folder_path: str
Where to read the dataset from.
Returns:
features: ndarray
nxd array with n emails, d words. features_ij is the count of word_j in email_i
labels: ndarray
1D array of labels (1: spam, 0: ham)
'''
# remove punctutaion marks
# tokenize, lowercase
# count number of words in each email
raise NotImplementedError
return features, labels
```
%% Cell type:markdown id: tags:
Implement the "fit" and "predict" methods for Naive Bayes. Use $m$-estimate to address missing attribute values (also called **Laplace smoothing** when $m$ = 1). In general, $m$ values should be small. We'll use $m$ = 1.
%% Cell type:code id: tags:
``` python
class naive_bayes(Model):
def __init__(self, preprocessor_f, partition_f, **kwargs):
super().__init__(preprocessor_f, partition_f, **kwargs)
def fit(self, m, **kwargs):
self.ham_word_counts = np.zeros(self.feat_dim)
self.spam_word_counts = np.zeros(self.feat_dim)
# find class prior probabilities
self.ham_prior = None
self.spam_prior = None
# find the number of words(counting repeats) summed across all emails in a class
n = None
# find the number of each word summed across all emails in a class
# populate self.ham_word_counts and self.spam_word_counts
# find the likelihood of a word_i in each class
# 1D ndarray
self.ham_likelihood = None
self.spam_likelihood = None
def predict(self, indices):
'''
Returns:
preds: ndarray
1D binary array containing predicted labels
'''
raise NotImplementedError
return preds
```
%% Cell type:markdown id: tags:
We can fit our model and see how accurately it predicts spam emails now. We won't use a validation set or crossvalidation this time.
%% Cell type:code id: tags:
``` python
# populate the keyword arguments dictionary kwargs
# p: proportion for test data
# k: parameter for k-fold crossvalidation
kwargs = {'p': 0.3, 'file_path': 'enron'}
# initialize the model
my_model = linear_regression(preprocessor_f=preprocess_bayes, partition_f=partition, **kwargs)
```
%% Cell type:markdown id: tags:
Using the validation set, and the value of $polynomial_{degree}$ you found above, try different values of $\lambda$ to find a a good value that results in low $mse$. You can
We can use the "conf_matrix" function we defined before to see how error is distributed.
%% Cell type:code id: tags:
``` python
preds = my_model.predict(my_model.test_indices)
tp,tn, fp, fn = conf_matrix(true = my_model.features[my_model.test_indices], pred = preds)
```
......
......
model.ipynb
+ 45
− 3
View file @ 22dd3471
%% Cell type:markdown id: tags:
# JUPYTER NOTEBOOK TIPS
Each rectangular box is called a cell.
* ctrl+ENTER evaluates the current cell; if it contains Python code, it runs the code, if it contains Markdown, it returns rendered text.
* alt+ENTER evaluates the current cell and adds a new cell below it.
* If you click to the left of a cell, you'll notice the frame changes color to blue. You can erase a cell by hitting 'dd' (that's two "d"s in a row) when the frame is blue.
%% Cell type:markdown id: tags:
# Supervised Learning Model Skeleton
We'll use this skeleton for implementing different supervised learning algorithms. Please complete "preprocess" and "partition" methods below.
%% Cell type:markdown id: tags:
You might need to preprocess your dataset depending on which dataset you are using. This step is for reading the dataset and for extracting features and labels. The "preprocess" function should return an $n \times d$ features array, and an $n \times 1$ labels array, where $n$ is the number of examples and $d$ is the number of features in the dataset. In cases where there is a big difference between the scales of feautures, we want to normalize the features to have values in the same range [0,1]. If that is the case with your dataset, output normalized features to get better prediction results.
%% Cell type:code id: tags:
``` python
def preprocess(file_path):
'''
file_path: where to read the dataset from
returns nxd features, nx1 labels
'''
# You might find np.genfromtxt useful for reading in the file. Be careful with the file delimiter,
# e.g. for comma-separated files use delimiter=',' argument.
raise NotImplementedError
return features, labels
```
%% Cell type:markdown id: tags:
Next, you'll need to split your dataset into training and validation and test sets. The "split" function should take as input the size of the whole dataset and randomly sample a proportion $p$ of the dataset as test partition and a proportion of $v$ as validation partition. The remaining will be used as training data. For example, to keep 30% of the examples as test and %10 as validation, set $p=0.3$ and $v=0.1$. You should choose these values according to the size of the data available to you. The "split" function should return indices of the training, validation and test sets. These will be used to index into the whole training set.
%% Cell type:code id: tags:
``` python
def partition(size, p, v):
def partition(size, p, v = 0):
'''
size: number of examples in the whole dataset
p: proportion kept for test
v: proportion kept for validation
'''
# np.random.choice might come in handy. Do not sample with replacement!
# Be sure to not use the same indices in test and validation sets!
# use the first np.ceil(size*p) for test,
# the following np.ceil(size*v) for validation set.
raise NotImplementedError
# return two 1d arrays: one keeping validation set indices, the other keeping test set indices
return val_indices, test_indices
```
%% Cell type:markdown id: tags:
In cases, where data is not abundantly available, we resort to getting an error estimate from average of error on different splits of error. In this case, every fold of data is used for testing and for training in turns, i.e. assuming we split our data into 3 folds, we'd
* train our model on fold-1+fold-2 and test on fold-3
* train our model on fold-1+fold-3 and test on fold-2
* train our model on fold-2+fold-3 and test on fold-1.
We'd use the average of the error we obtained in three runs as our error estimate. Implement function "kfold" below.
%% Cell type:code id: tags:
``` python
def kfold(indices, k):
'''
Args:
indices: ndarray
1D array with integer entries containing indices
k: int
Number of desired splits in data.(Assume test set is already separated.)
Returns:
fold_dict: dict
A dictionary with integer keys corresponding to folds. Values are (training_indices, val_indices).
val_indices: ndarray
1/k of training indices randomly chosen and separates them as validation partition.
train_indices: ndarray
Remaining 1-(1/k) of the indices.
e.g. fold_dict = {0: (train_0_indices, val_0_indices),
1: (train_0_indices, val_0_indices), 2: (train_0_indices, val_0_indices)} for k = 3
'''
return fold_dict
```
%% Cell type:code id: tags:
``` python
class Model:
# set the preprocessing function, partition_function
# use kwargs to pass arguments to preprocessor_f and partition_f
# kwargs is a dictionary and should contain p, v and file_path
# e.g. {'p': 0.3, 'v': 0.1, 'file_path': some_path}
def __init__(self, preprocessor_f, partition_f, **kwargs):
self.features, self.labels = preprocessor_f(kwargs['file_path'])
self.size = len(self.labels) # number of examples in dataset
self.feat_dim = self.features.shape[1] # number of features
self.val_indices, self.test_indices = partition_f(self.size, kwargs['p'], kwargs['v'])
self.val_size = len(self.val_indices)
self.test_size = len(self.test_indices)
self.train_indices = np.delete(np.arange(self.size), np.append(self.test_indices, self.val_indices), 0)
self.train_size = len(self.train_indices)
def fit(self):
raise NotImplementedError
def predict(self, testpoint):
def predict(self, indices):
raise NotImplementedError
```
%% Cell type:markdown id: tags:
## General supervised learning related functions
### (To be implemented later when it is indicated in other notebooks)
%% Cell type:markdown id: tags:
Implement the "conf_matrix" function that takes as input an array of true labels ($true$) and an array of predicted labels ($pred$). It should output a numpy.ndarray.
%% Cell type:code id: tags:
``` python
def conf_matrix(true, pred):
'''
true: nx1 array of true labels for test set
pred: nx1 array of predicted labels for test set
'''
raise NotImplementedError
tp = tn = fp = fn = 0
# calculate true positives (tp), true negatives(tn)
# false positives (fp) and false negatives (fn)
# returns the confusion matrix as numpy.ndarray
return np.array([tp,tn, fp, fn])
```
%% Cell type:markdown id: tags:
ROC curves are a good way to visualize sensitivity vs. 1-specificity for varying cut off points. Now, implement a "ROC" function that predicts the labels of the test set examples using different $threshold$ values in "predict" and plot the ROC curve. "ROC" takes a list containing different $threshold$ parameter values to try and returns two arrays; one where each entry is the sensitivity at a given threshold and the other where entries are 1-specificities.
%% Cell type:code id: tags:
``` python
def ROC(model, indices, value_list):
'''
model: a fitted supervised learning model
indices: for data points to predict
value_list: array containing different threshold values
Calculate sensitivity and 1-specificity for each point in value_list
Return two nX1 arrays: sens (for sensitivities) and spec_ (for 1-specificities)
'''
# use predict method to obtain predicted labels at different threshold values
# use conf_matrix to calculate tp, tn, fp, fn
# calculate sensitivity, 1-specificity
# return two arrays
raise NotImplementedError
return sens, spec_
```
......
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment