"* Ctrl+ENTER evaluates the current cell; if it contains Python code, it runs the code, if it contains Markdown, it returns rendered text.\n",
"* Alt+ENTER evaluates the current cell and adds a new cell below it.\n",
"* If you click to the left of a cell, you'll notice the frame changes color to blue. You can erase a cell by hitting 'dd' (that's two \"d\"s in a row) when the frame is blue."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# GRADING\n",
"\n",
"You will be graded on parts that are marked with **\\#TODO** comments. Read the comments in the code to make sure you don't miss any.\n",
"Points are broken down further below in Rubric sections. The **first** score is for 478, the **second** is for 878 students. There a total of 25 points in this assignment and extra 5 bonus points for 478 students."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Supervised Learning Model Skeleton\n",
"\n",
"We'll use this skeleton for implementing different supervised learning algorithms. For this first assignment, we'll read and partition the [\"madelon\" dataset](http://archive.ics.uci.edu/ml/datasets/madelon). Features and labels for the first two examples are listed below. Please complete \"preprocess\" and \"partition\" methods. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We'll use numpy library for this assignment. Please do not import any other libraries."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"# import necessary libraries\n",
"import numpy as np"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The 500 features in the \"madelon\" dataset have integer values:"
"This step is for reading the dataset and for extracting features and labels. The \"preprocess\" function should return an $n \\times d$ \"features\" array, and an $n \\times 1$ \"labels\" array, where $n$ is the number of examples and $d$ is the number of features in the dataset. In cases where there is a big difference between the scales of features, we want to normalize the features to have values in the same range [0,1]. Since this is not the case with this dataset, we will not do normalization."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"def preprocess(feature_file, label_file):\n",
" '''\n",
" Args:\n",
" feature_file: str \n",
" file containing features\n",
" label_file: str\n",
" file containing labels\n",
" Returns:\n",
" features: ndarray\n",
" nxd features\n",
" labels: ndarray\n",
" nx1 labels\n",
" '''\n",
" # You might find np.genfromtxt useful for reading in the file. Be careful with the file delimiter, \n",
" # e.g. for comma-separated files use delimiter=',' argument.\n",
"# TODO: Output the dimension of both features and labels."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## TASK 2: Implement `partition`"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next, you'll need to split your dataset into training, validation and test sets. The \"partition\" function should take as input the size of the whole dataset and randomly sample a proportion $t$ of the dataset as test partition and a proportion of $v$ as validation partition. The remaining will be used as training data. For example, to keep 30% of the examples as test and %10 as validation, set $t=0.3$ and $v=0.1$. You should choose these values according to the size of the data available to you. The \"split\" function should return indices of the training, validation and test sets. These will be used to index into the whole training set."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def partition(size, t, v = 0):\n",
" '''\n",
" Args:\n",
" size: int\n",
" number of examples in the whole dataset\n",
" t: float\n",
" proportion kept for test\n",
" v: float\n",
" proportion kept for validation\n",
" Returns:\n",
" test_indices: ndarray\n",
" 1D array containing test set indices\n",
" val_indices: ndarray\n",
" 1D array containing validation set indices\n",
" '''\n",
" \n",
" # np.random.permutation might come in handy. Do not sample with replacement!\n",
" # Be sure not to use the same indices in test and validation sets!\n",
" \n",
" # use the first np.ceil(size*t) for test, \n",
" # the following np.ceil(size*v) for validation set.\n",
" \n",
" # TODO\n",
" \n",
" raise NotImplementedError\n",
" \n",
" return test_indices, val_indices"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Rubric:\n",
"* Correct length of test indices +5, +2.5\n",
"* Correct length of validation indices +5, +2.5"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Test `partition`"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# TODO\n",
"# Pass the correct size argument (number of examples in the whole dataset)\n",
"test_indices, val_indices = partition(size=..., t = 0.3, v = 0.1)\n",
"\n",
"# Output the size of both features and labels."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## TASK 3: Putting things together"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The model definition is given below. We'll extend this class for different supervised classification algorithms. Specifically, we'll implement \"fit\" and \"predict\" methods for these algorithms. For this assignment, you are not asked to implement these methods. Run the cells below and make sure each piece of code fits together and works as expected."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"class Model:\n",
" # preprocess_f and partition_f expect functions\n",
" # use kwargs to pass arguments to preprocessor_f and partition_f\n",
" # kwargs is a dictionary and should contain t, v, feature_file, label_file\n",
"We will use a keyword arguments dictionary that conveniently passes arguments to functions that are themselves passed as arguments during object initialization. Please do not change these calls in this and the following assignments."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# TODO\n",
"# pass the correct arguments to preprocessor_f and partition_f\n",
"Modify `preprocess` function such that the output features take values in the range [0, 1]. Initialize a new model with this function and check the values of the features."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Rubric:\n",
"* Correct range for feature values +5, +10"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Test Normalization"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# TODO\n",
"# args is a placeholder for the parameters of the function\n",
"# Check that the range of each feature in the training set is in range [0, 1]"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.4"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
%% Cell type:markdown id: tags:
# JUPYTER NOTEBOOK TIPS
Each rectangular box is called a cell.
* Ctrl+ENTER evaluates the current cell; if it contains Python code, it runs the code, if it contains Markdown, it returns rendered text.
* Alt+ENTER evaluates the current cell and adds a new cell below it.
* If you click to the left of a cell, you'll notice the frame changes color to blue. You can erase a cell by hitting 'dd' (that's two "d"s in a row) when the frame is blue.
%% Cell type:markdown id: tags:
# GRADING
You will be graded on parts that are marked with **\#TODO** comments. Read the comments in the code to make sure you don't miss any.
Points are broken down further below in Rubric sections. The **first** score is for 478, the **second** is for 878 students. There a total of 25 points in this assignment and extra 5 bonus points for 478 students.
%% Cell type:markdown id: tags:
# Supervised Learning Model Skeleton
We'll use this skeleton for implementing different supervised learning algorithms. For this first assignment, we'll read and partition the ["madelon" dataset](http://archive.ics.uci.edu/ml/datasets/madelon). Features and labels for the first two examples are listed below. Please complete "preprocess" and "partition" methods.
%% Cell type:markdown id: tags:
We'll use numpy library for this assignment. Please do not import any other libraries.
%% Cell type:code id: tags:
``` python
# import necessary libraries
importnumpyasnp
```
%% Cell type:markdown id: tags:
The 500 features in the "madelon" dataset have integer values:
This step is for reading the dataset and for extracting features and labels. The "preprocess" function should return an $n \times d$ "features" array, and an $n \times 1$ "labels" array, where $n$ is the number of examples and $d$ is the number of features in the dataset. In cases where there is a big difference between the scales of features, we want to normalize the features to have values in the same range [0,1]. Since this is not the case with this dataset, we will not do normalization.
%% Cell type:code id: tags:
``` python
defpreprocess(feature_file,label_file):
'''
Args:
feature_file: str
file containing features
label_file: str
file containing labels
Returns:
features: ndarray
nxd features
labels: ndarray
nx1 labels
'''
# You might find np.genfromtxt useful for reading in the file. Be careful with the file delimiter,
# e.g. for comma-separated files use delimiter=',' argument.
Next, you'll need to split your dataset into training, validation and test sets. The "partition" function should take as input the size of the whole dataset and randomly sample a proportion $t$ of the dataset as test partition and a proportion of $v$ as validation partition. The remaining will be used as training data. For example, to keep 30% of the examples as test and %10 as validation, set $t=0.3$ and $v=0.1$. You should choose these values according to the size of the data available to you. The "split" function should return indices of the training, validation and test sets. These will be used to index into the whole training set.
%% Cell type:code id: tags:
``` python
defpartition(size,t,v=0):
'''
Args:
size: int
number of examples in the whole dataset
t: float
proportion kept for test
v: float
proportion kept for validation
Returns:
test_indices: ndarray
1D array containing test set indices
val_indices: ndarray
1D array containing validation set indices
'''
# np.random.permutation might come in handy. Do not sample with replacement!
# Be sure not to use the same indices in test and validation sets!
# use the first np.ceil(size*t) for test,
# the following np.ceil(size*v) for validation set.
# TODO
raiseNotImplementedError
returntest_indices,val_indices
```
%% Cell type:markdown id: tags:
### Rubric:
* Correct length of test indices +5, +2.5
* Correct length of validation indices +5, +2.5
%% Cell type:markdown id: tags:
### Test `partition`
%% Cell type:code id: tags:
``` python
# TODO
# Pass the correct size argument (number of examples in the whole dataset)
The model definition is given below. We'll extend this class for different supervised classification algorithms. Specifically, we'll implement "fit" and "predict" methods for these algorithms. For this assignment, you are not asked to implement these methods. Run the cells below and make sure each piece of code fits together and works as expected.
%% Cell type:code id: tags:
``` python
classModel:
# preprocess_f and partition_f expect functions
# use kwargs to pass arguments to preprocessor_f and partition_f
# kwargs is a dictionary and should contain t, v, feature_file, label_file
# e.g. {'t': 0.3, 'v': 0.1, 'feature_file': 'some_file_name', 'label_file': 'some_file_name'}
We will use a keyword arguments dictionary that conveniently passes arguments to functions that are themselves passed as arguments during object initialization. Please do not change these calls in this and the following assignments.
%% Cell type:code id: tags:
``` python
# TODO
# pass the correct arguments to preprocessor_f and partition_f
Modify `preprocess` function such that the output features take values in the range [0, 1]. Initialize a new model with this function and check the values of the features.
%% Cell type:markdown id: tags:
### Rubric:
* Correct range for feature values +5, +10
%% Cell type:markdown id: tags:
### Test Normalization
%% Cell type:code id: tags:
``` python
# TODO
# args is a placeholder for the parameters of the function
"* ctrl+ENTER evaluates the current cell; if it contains Python code, it runs the code, if it contains Markdown, it returns rendered text.\n",
"* alt+ENTER evaluates the current cell and adds a new cell below it.\n",
"* If you click to the left of a cell, you'll notice the frame changes color to blue. You can erase a cell by hitting 'dd' (that's two \"d\"s in a row) when the frame is blue."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Supervised Learning Model Skeleton\n",
"\n",
"We'll use this skeleton for implementing different supervised learning algorithms. Please complete \"preprocess\" and \"partition\" methods below."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This step is for reading the dataset and for extracting features and labels. The \"preprocess\" function should return an *n x d* \"features\" array, and an *n x 1* \"labels\" array, where *n* is the number of examples and *d* is the number of features in the dataset. In cases where there is a big difference between the scales of features, we want to normalize the features to have values in the same range [0,1]. Since this is not the case with this dataset, we will not do normalization."
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [],
"source": [
"def preprocess(feature_file, label_file):\n",
" '''\n",
" Args:\n",
" feature_file: str \n",
" file containing features\n",
" label_file: str\n",
" file containing labels\n",
" Returns:\n",
" features: ndarray\n",
" nxd features\n",
" labels: ndarray\n",
" nx1 labels\n",
" '''\n",
" # You might find np.genfromtxt useful for reading in the file. Be careful with the file delimiter, \n",
" # e.g. for comma-separated files use delimiter=',' argument.\n",
" \n",
" # TODO \n",
" \n",
" raise NotImplementedError\n",
"\n",
" \n",
" return features, labels"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next, you'll need to split your dataset into training, validation and test sets. The \"partition\" function should take as input the size of the whole dataset and randomly sample a proportion *t* of the dataset as test partition and a proportion of *v* as validation partition. The remaining will be used as training data. For example, to keep 30% of the examples as test and %10 as validation, set *t* = 0.3 and *v* = 0.1. You should choose these values according to the size of the data available to you. The \"split\" function should return indices of the training, validation and test sets. These will be used to index into the whole training set."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"#TODO: GIVE!\n",
"def partition(size, t, v = 0):\n",
" '''\n",
" Args:\n",
" size: int\n",
" number of examples in the whole dataset\n",
" t: float\n",
" proportion kept for test\n",
" v: float\n",
" proportion kept for validation\n",
" Returns:\n",
" test_indices: ndarray\n",
" 1D array containing test set indices\n",
" val_indices: ndarray\n",
" 1D array containing validation set indices\n",
" '''\n",
" \n",
" # np.random.permutation might come in handy. Do not sample with replacement!\n",
" # Be sure not to use the same indices in test and validation sets!\n",
" \n",
" # use the first np.ceil(size*t) for test, \n",
" # the following np.ceil(size*v) for validation set.\n",
" \n",
" # TODO\n",
" \n",
" raise NotImplementedError\n",
" \n",
" return test_indices, val_indices"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In cases, where data is not abundantly available, we resort to getting an error estimate from average of error on different splits of error. In this case, every fold of data is used for testing and for training in turns, i.e. assuming we split our data into 3 folds, we'd\n",
"* train our model on fold-1+fold-2 and test on fold-3\n",
"* train our model on fold-1+fold-3 and test on fold-2\n",
"* train our model on fold-2+fold-3 and test on fold-1.\n",
"\n",
"We'd use the average of the error we obtained in three runs as our error estimate. Implement function \"kfold\" below.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# TODO: Programming Assignment 2\n",
"\n",
"def kfold(indices, k):\n",
"\n",
" '''\n",
" Args:\n",
" indices: ndarray\n",
" 1D array with integer entries containing indices\n",
" k: int \n",
" Number of desired splits in data.(Assume test set is already separated.)\n",
" Returns:\n",
" fold_dict: dict\n",
" A dictionary with integer keys corresponding to folds. Values are (training_indices, val_indices).\n",
" \n",
" val_indices: ndarray\n",
" 1/k of training indices randomly chosen and separates them as validation partition.\n",
" train_indices: ndarray\n",
" Remaining 1-(1/k) of the indices.\n",
" \n",
" e.g. fold_dict = {0: (train_0_indices, val_0_indices), \n",
" 1: (train_0_indices, val_0_indices), 2: (train_0_indices, val_0_indices)} for k = 3\n",
" '''\n",
" \n",
" return fold_dict"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Choice of distance metric plays an important role in the performance of *k*-NN. Let's start with implementing a distance method in the \"distance\" function below. It should take two data points and the name of the metric and return a scalar value."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"#TODO: Programming Assignment 1\n",
"def distance(x, y, metric):\n",
" '''\n",
" Args:\n",
" x: ndarray \n",
" 1D array containing coordinates for a point\n",
" y: ndarray\n",
" 1D array containing coordinates for a point\n",
" metric: str\n",
" Euclidean, Hamming \n",
" Returns:\n",
" \n",
" '''\n",
" if metric == 'Euclidean':\n",
" raise NotImplementedError\n",
" elif metric == 'Hammming':\n",
" raise NotImplementedError\n",
" else:\n",
" raise ValueError('{} is not a valid metric.'.format(metric))\n",
" return dist # scalar distance btw x and y"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We will extend \"Model\" class while implementing supervised learning algorithms."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"class Model:\n",
" # preprocess_f and partition_f expect functions\n",
" # use kwargs to pass arguments to preprocessor_f and partition_f\n",
" # kwargs is a dictionary and should contain t, v, feature_file, label_file\n",
"## General supervised learning performance related functions \n",
"### (To be implemented later when it is indicated in other notebooks)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Implement the \"conf_matrix\" function that takes as input an array of true labels (*true*) and an array of predicted labels (*pred*). It should output a numpy.ndarray."
" # false positives (fp) and false negatives (fn)\n",
" \n",
" # returns the confusion matrix as numpy.ndarray\n",
" return np.array([tp,tn, fp, fn])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"ROC curves are a good way to visualize sensitivity vs. 1-specificity for varying cut off points. Now, implement a \"ROC\" function that predicts the labels of the test set examples using different *threshold* values in \"predict\" and plot the ROC curve. \"ROC\" takes a list containing different *threshold* parameter values to try and returns two arrays; one where each entry is the sensitivity at a given threshold and the other where entries are 1-specificities."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# TODO: Programming Assignment 1\n",
"\n",
"def ROC(model, indices, value_list):\n",
" '''\n",
" Args:\n",
" model: a fitted supervised learning model\n",
" indices: ndarray\n",
" 1D array containing indices\n",
" value_list: ndarray\n",
" 1D array containing different threshold values\n",
" Returns:\n",
" sens: ndarray\n",
" 1D array containing sensitivities\n",
" spec_: ndarray\n",
" 1D array containing 1-specifities\n",
" '''\n",
" \n",
" # use predict method to obtain predicted labels at different threshold values\n",
" # use conf_matrix to calculate tp, tn, fp, fn\n",
" # calculate sensitivity, 1-specificity\n",
" # return two arrays\n",
" \n",
" raise NotImplementedError\n",
" \n",
" return sens, spec_"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.4"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
%% Cell type:markdown id: tags:
# JUPYTER NOTEBOOK TIPS
Each rectangular box is called a cell.
* ctrl+ENTER evaluates the current cell; if it contains Python code, it runs the code, if it contains Markdown, it returns rendered text.
* alt+ENTER evaluates the current cell and adds a new cell below it.
* If you click to the left of a cell, you'll notice the frame changes color to blue. You can erase a cell by hitting 'dd' (that's two "d"s in a row) when the frame is blue.
%% Cell type:markdown id: tags:
# Supervised Learning Model Skeleton
We'll use this skeleton for implementing different supervised learning algorithms. Please complete "preprocess" and "partition" methods below.
%% Cell type:markdown id: tags:
This step is for reading the dataset and for extracting features and labels. The "preprocess" function should return an *n x d* "features" array, and an *n x 1* "labels" array, where *n* is the number of examples and *d* is the number of features in the dataset. In cases where there is a big difference between the scales of features, we want to normalize the features to have values in the same range [0,1]. Since this is not the case with this dataset, we will not do normalization.
%% Cell type:code id: tags:
``` python
defpreprocess(feature_file,label_file):
'''
Args:
feature_file: str
file containing features
label_file: str
file containing labels
Returns:
features: ndarray
nxd features
labels: ndarray
nx1 labels
'''
# You might find np.genfromtxt useful for reading in the file. Be careful with the file delimiter,
# e.g. for comma-separated files use delimiter=',' argument.
# TODO
raiseNotImplementedError
returnfeatures,labels
```
%% Cell type:markdown id: tags:
Next, you'll need to split your dataset into training, validation and test sets. The "partition" function should take as input the size of the whole dataset and randomly sample a proportion *t* of the dataset as test partition and a proportion of *v* as validation partition. The remaining will be used as training data. For example, to keep 30% of the examples as test and %10 as validation, set *t* = 0.3 and *v* = 0.1. You should choose these values according to the size of the data available to you. The "split" function should return indices of the training, validation and test sets. These will be used to index into the whole training set.
%% Cell type:code id: tags:
``` python
#TODO: GIVE!
defpartition(size,t,v=0):
'''
Args:
size: int
number of examples in the whole dataset
t: float
proportion kept for test
v: float
proportion kept for validation
Returns:
test_indices: ndarray
1D array containing test set indices
val_indices: ndarray
1D array containing validation set indices
'''
# np.random.permutation might come in handy. Do not sample with replacement!
# Be sure not to use the same indices in test and validation sets!
# use the first np.ceil(size*t) for test,
# the following np.ceil(size*v) for validation set.
# TODO
raiseNotImplementedError
returntest_indices,val_indices
```
%% Cell type:markdown id: tags:
In cases, where data is not abundantly available, we resort to getting an error estimate from average of error on different splits of error. In this case, every fold of data is used for testing and for training in turns, i.e. assuming we split our data into 3 folds, we'd
* train our model on fold-1+fold-2 and test on fold-3
* train our model on fold-1+fold-3 and test on fold-2
* train our model on fold-2+fold-3 and test on fold-1.
We'd use the average of the error we obtained in three runs as our error estimate. Implement function "kfold" below.
%% Cell type:code id: tags:
``` python
# TODO: Programming Assignment 2
defkfold(indices,k):
'''
Args:
indices: ndarray
1D array with integer entries containing indices
k: int
Number of desired splits in data.(Assume test set is already separated.)
Returns:
fold_dict: dict
A dictionary with integer keys corresponding to folds. Values are (training_indices, val_indices).
val_indices: ndarray
1/k of training indices randomly chosen and separates them as validation partition.
train_indices: ndarray
Remaining 1-(1/k) of the indices.
e.g. fold_dict = {0: (train_0_indices, val_0_indices),
1: (train_0_indices, val_0_indices), 2: (train_0_indices, val_0_indices)} for k = 3
'''
returnfold_dict
```
%% Cell type:markdown id: tags:
Choice of distance metric plays an important role in the performance of *k*-NN. Let's start with implementing a distance method in the "distance" function below. It should take two data points and the name of the metric and return a scalar value.
%% Cell type:code id: tags:
``` python
#TODO: Programming Assignment 1
defdistance(x,y,metric):
'''
Args:
x: ndarray
1D array containing coordinates for a point
y: ndarray
1D array containing coordinates for a point
metric: str
Euclidean, Hamming
Returns:
'''
ifmetric=='Euclidean':
raiseNotImplementedError
elifmetric=='Hammming':
raiseNotImplementedError
else:
raiseValueError('{} is not a valid metric.'.format(metric))
returndist# scalar distance btw x and y
```
%% Cell type:markdown id: tags:
We will extend "Model" class while implementing supervised learning algorithms.
%% Cell type:code id: tags:
``` python
classModel:
# preprocess_f and partition_f expect functions
# use kwargs to pass arguments to preprocessor_f and partition_f
# kwargs is a dictionary and should contain t, v, feature_file, label_file
# e.g. {'t': 0.3, 'v': 0.1, 'feature_file': 'some_file_name', 'label_file': 'some_file_name'}
## General supervised learning performance related functions
### (To be implemented later when it is indicated in other notebooks)
%% Cell type:markdown id: tags:
Implement the "conf_matrix" function that takes as input an array of true labels (*true*) and an array of predicted labels (*pred*). It should output a numpy.ndarray.
ROC curves are a good way to visualize sensitivity vs. 1-specificity for varying cut off points. Now, implement a "ROC" function that predicts the labels of the test set examples using different *threshold* values in "predict" and plot the ROC curve. "ROC" takes a list containing different *threshold* parameter values to try and returns two arrays; one where each entry is the sensitivity at a given threshold and the other where entries are 1-specificities.
%% Cell type:code id: tags:
``` python
# TODO: Programming Assignment 1
defROC(model,indices,value_list):
'''
Args:
model: a fitted supervised learning model
indices: ndarray
1D array containing indices
value_list: ndarray
1D array containing different threshold values
Returns:
sens: ndarray
1D array containing sensitivities
spec_: ndarray
1D array containing 1-specifities
'''
# use predict method to obtain predicted labels at different threshold values
