diff --git a/ProgrammingAssignment_0/GettingFamiliar.ipynb b/ProgrammingAssignment_0/GettingFamiliar.ipynb
index df3b633ca9bcd716b8478883d902afb5153dd757..2cab2a4415f10e41a858d1751dc2d5ec9a3540aa 100644
--- a/ProgrammingAssignment_0/GettingFamiliar.ipynb
+++ b/ProgrammingAssignment_0/GettingFamiliar.ipynb
@@ -22,18 +22,17 @@
"\n",
"### Mandatory for 478 & 878:\n",
"\n",
- "| Tasks | 478 | 878 |\n",
- "|----------------------------|-----|-----|\n",
- "| Implement `preprocess` | 10 | 5 |\n",
- "| Implement `partition` | 10 | 5 |\n",
- "| Putting the model together | 5 | 5 |\n",
- "\n",
+ "| | Tasks | 478 | 878 |\n",
+ "|---|----------------------------|-----|-----|\n",
+ "| 1 | Implement `preprocess` | 10 | 5 |\n",
+ "| 2 | Implement `partition` | 10 | 5 |\n",
+ "| 3 | Putting the model together | 5 | 5 |\n",
"\n",
"### Mandatory for 878, bonus for 478\n",
"\n",
- "| Tasks | 478 | 878 |\n",
- "|---------------------------------------|-----|-----|\n",
- "| Modify `preprocess` for normalization | 5 | 10 |\n",
+ "| | Tasks | 478 | 878 |\n",
+ "|---|---------------------------------------|-----|-----|\n",
+ "|4 | Modify `preprocess` for normalization | 5 | 10 |\n",
"\n",
"\n",
"Points are broken down further below in Rubric sections. The **first** score is for 478, the **second** is for 878 students. There a total of 25 points in this assignment and extra 5 bonus points for 478 students."
@@ -352,6 +351,13 @@
"* Correct range for feature values +5, +10"
]
},
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Test Normalization"
+ ]
+ },
{
"cell_type": "code",
"execution_count": null,
@@ -402,4 +408,3 @@
"nbformat": 4,
"nbformat_minor": 2
}
-
diff --git a/ProgrammingAssignment_1/ProgrammingAssignment1.ipynb b/ProgrammingAssignment_1/ProgrammingAssignment1.ipynb
index da45e04933867f555cea589193027d67a511e628..eec35ad057da7ed8d77db2ed23a43fe8064b8ccc 100644
--- a/ProgrammingAssignment_1/ProgrammingAssignment1.ipynb
+++ b/ProgrammingAssignment_1/ProgrammingAssignment1.ipynb
@@ -21,6 +21,34 @@
" * accuracy, generalization error and ROC curve\n"
]
},
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# GRADING\n",
+ "\n",
+ "You will be graded on parts that are marked with **\\#TODO** comments. Read the comments in the code to make sure you don't miss any.\n",
+ "\n",
+ "### Mandatory for 478 & 878:\n",
+ "\n",
+ "| | Tasks | 478 | 878 |\n",
+ "|---|----------------------------|-----|-----|\n",
+ "| 1 | Implement `distance` | 10 | 10 |\n",
+ "| 2 | Implement `k-NN` methods | 25 | 20 |\n",
+ "| 3 | Model evaluation | 25 | 20 |\n",
+ "| 4 | Learning curve | 20 | 20 |\n",
+ "| 6 | ROC curve analysis | 20 | 20 |\n",
+ "\n",
+ "### Mandatory for 878, bonus for 478\n",
+ "\n",
+ "| | Tasks | 478 | 878 |\n",
+ "|---|----------------|-----|-----|\n",
+ "| 5 | Optimizing $k$ | 10 | 10 |\n",
+ "\n",
+ "\n",
+ "Points are broken down further below in Rubric sections. The **first** score is for 478, the **second** is for 878 students. There a total of 100 points in this assignment and extra 10 bonus points for 478 students."
+ ]
+ },
{
"cell_type": "markdown",
"metadata": {},
@@ -58,7 +86,30 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "Choice of distance metric plays an important role in the performance of $k$-NN. Let's start with implementing a distance method in the \"distance\" function below. It should take two data points and the name of the metric and return a scalar value."
+ "## TASK 1: Implement `distance` function"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Choice of distance metric plays an important role in the performance of $k$-NN. Let's start with implementing a distance method in the \"distance\" function in **model.ipynb**. It should take two data points and the name of the metric and return a scalar value."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Rubric:\n",
+ "* Euclidean +5, +5\n",
+ "* Hamming +5, +5"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Test `distance`"
]
},
{
@@ -67,32 +118,17 @@
"metadata": {},
"outputs": [],
"source": [
- "def distance(x, y, metric):\n",
- " '''\n",
- " Args:\n",
- " x: ndarray \n",
- " 1D array containing coordinates for a point\n",
- " y: ndarray\n",
- " 1D array containing coordinates for a point\n",
- " metric: str\n",
- " Euclidean, Hamming \n",
- " Returns:\n",
- " \n",
- " '''\n",
- " if metric == 'Euclidean':\n",
- " raise NotImplementedError\n",
- " elif metric == 'Hammming':\n",
- " raise NotImplementedError\n",
- " else:\n",
- " raise ValueError('{} is not a valid metric.'.format(metric))\n",
- " return dist # scalar distance btw x and y"
+ "x = np.array(range(100))\n",
+ "y = np.array(range(100, 200))\n",
+ "dist_euclidean = distance(x, y, 'Euclidean')\n",
+ "dist_hamming = distance(x, y, 'Hamming')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "### $k$-NN Class Methods"
+ "## TASK 2: Implement $k$-NN Class Methods"
]
},
{
@@ -102,6 +138,15 @@
"We can start implementing our $k$-NN classifier. $k$-NN class inherits Model class. You'll need to implement \"fit\" and \"predict\" methods. Use the \"distance\" function you defined above. \"fit\" method takes $k$ as an argument. \"predict\" takes as input an $mxd$ array containing $d$-dimensional $m$ feature vectors for examples and outputs the predicted class and the ratio of positive examples in $k$ nearest neighbors."
]
},
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Rubric:\n",
+ "* correct implementation of fit method +5, +5\n",
+ "* correct implementation of predict method +20, +15"
+ ]
+ },
{
"cell_type": "code",
"execution_count": null,
@@ -117,6 +162,7 @@
" '''\n",
" Fit the model. This is pretty straightforward for k-NN.\n",
" '''\n",
+ " # TODO\n",
" # set self.k, self.distance_f, self.distance_metric\n",
" raise NotImplementedError\n",
" \n",
@@ -128,6 +174,8 @@
" raise NotImplementedError\n",
" \n",
" pred = []\n",
+ " # TODO\n",
+ " \n",
" # for each point in test points\n",
" # use your implementation of distance function\n",
" # distance_f(..., distance_metric)\n",
@@ -145,14 +193,24 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "### Build and Evaluate the Model (Accuracy, Confidence Interval, Confusion Matrix)"
+ "## TASK 3: Build and Evaluate the Model"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Rubric:\n",
+ "* Reasonable accuracy values +10, +5\n",
+ "* Reasonable confidence intervals on the error estimate +10, +10\n",
+ "* Reasonable confusion matrix +5, +5"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "It's time to build and evaluate our model now. Remember you need to provide values to $p$, $v$ parameters for \"partition\" function and to $file\\_path$ for \"preprocess\" function."
+ "Remember you need to provide values to *t*, *v* parameters for \"partition\" function and to *feature_file* and *label_file* for \"preprocess\" function."
]
},
{
@@ -162,7 +220,7 @@
"outputs": [],
"source": [
"# populate the keyword arguments dictionary kwargs\n",
- "kwargs = {'p': 0.3, 'v': 0.1, seed: 123, 'file_path': 'madelon_train'}\n",
+ "kwargs = {'t': 0.3, 'v': 0.1, 'feature_file': ..., 'label_file': ...}\n",
"# initialize the model\n",
"my_model = kNN(preprocessor_f=preprocess, partition_f=partition, **kwargs)"
]
@@ -199,30 +257,62 @@
"source": [
"final_labels = my_model.predict(my_model.test_indices)\n",
"\n",
- "# For now, We will consider a data point as predicted in the positive class if more than 0.5 \n",
+ "# For now, we will consider a data point as predicted in the positive class if more than 0.5 \n",
"# of its k-neighbors are positive.\n",
"threshold = 0.5\n",
- "# Calculate accuracy and generalization error with confidence interval here."
+ "\n",
+ "# TODO\n",
+ "# Calculate and report accuracy and generalization error with confidence interval here. Show your work in this cell."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- " ### Plotting a learning curve\n",
+ "### Computing the confusion matrix for $k = 10$\n",
+ "Now that we have the true labels and the predicted ones from our model, we can build a confusion matrix and see how accurate our model is. Implement the \"conf_matrix\" function (in model.ipynb) that takes as input an array of true labels ($true$) and an array of predicted labels ($pred$). It should output a numpy.ndarray. You do not need to change the value of the threshold parameter yet."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# TODO\n",
+ "conf_matrix(my_model.labels[my_model.test_indices], final_labels, threshold = 0.5)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ " ## TASK 4: Plotting a learning curve\n",
" \n",
"A learning curve shows how error changes as the training set size increases. For more information, see [learning curves](https://www.dataquest.io/blog/learning-curves-machine-learning/).\n",
"We'll plot the error values for training and validation data while varying the size of the training set. Report a good size for training set for which there is a good balance between bias and variance."
]
},
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Rubric:\n",
+ "* Correct training error calculation for different training set sizes +8, +8\n",
+ "* Correct validation error calculation for different training set sizes +8, +8\n",
+ "* Reasonable learning curve +4, +4"
+ ]
+ },
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
- "# try sizes 0, 100, 200, 300, ..., up to the largest multiple of 100 >= train_size\n",
- "training_sizes = np.xrange(0, my_model.train_size + 1, 100)\n",
+ "# try sizes 50, 100, 150, 200, ..., up to the largest multiple of 50 >= train_size\n",
+ "training_sizes = np.arange(50, my_model.train_size + 1, 50)\n",
+ "\n",
+ "# TODO\n",
"\n",
"# Calculate error for each entry in training_sizes\n",
"# for training and validation sets and populate\n",
@@ -239,64 +329,62 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "### Computing the confusion matrix for $k = 10$\n",
- "Now that we have the true labels and the predicted ones from our model, we can build a confusion matrix and see how accurate our model is. Implement the \"conf_matrix\" function (in model.ipynb) that takes as input an array of true labels ($true$) and an array of predicted labels ($pred$). It should output a numpy.ndarray. You do not need to change the value of the threshold parameter yet."
+ "## TASK 5: Determining $k$ "
]
},
{
- "cell_type": "code",
- "execution_count": null,
+ "cell_type": "markdown",
"metadata": {},
- "outputs": [],
"source": [
- "conf_matrix(my_model.labels[my_model.test_indices], final_labels, threshold = 0.5)"
+ "### Rubric:\n",
+ "* Increased accuracy with new $k$ +5, +5\n",
+ "* Improved confusion matrix +5, +5"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "### Finding a good value for $k$\n",
+ "We can use the validation set to come up with a $k$ value that results in better performance in terms of accuracy.\n",
"\n",
- "We can use the validation set to come up with a $k$ value that results in better performance in terms of accuracy. Additionally, in some cases, predicting examples from a certain class correctly is more critical than other classes. In those cases, we can use the confusion matrix to find a good trade off between correct and wrong predictions and allow more wrong predictions in some classes to predict more examples correctly in a that class.\n",
- "\n",
- "Below calculate the accuracies for different values of $k$ using the validation set. Report a good $k$ value and use it in the analyses that follow this section."
+ "Below calculate the accuracies for different values of $k$ using the validation set. Report a good $k$ value and use it in the analyses that follow this section. Report confusion matrix for the new value of $k$."
]
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
- "# Change values of $k. \n",
+ "# TODO\n",
+ "\n",
+ "# Change values of k. \n",
"# Calculate accuracies for the validation set.\n",
- "# Report a good k value that you'll use in the following analyses."
+ "# Report a good k value.\n",
+ "# Calculate the confusion matrix for new k."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "### ROC curve and confusion matrix for the final model\n",
- "ROC curves are a good way to visualize sensitivity vs. 1-specificity for varying cut off points. Now, implement, in \"model.ipynb\", a \"ROC\" function that predicts the labels of the test set examples using different $threshold$ values in \"predict\" and plot the ROC curve. \"ROC\" takes a list containing different $threshold$ parameter values to try and returns two arrays; one where each entry is the sensitivity at a given threshold and the other where entries are 1-specificities."
+ "## TASK 6: ROC curve analysis\n",
+ "* Correct implementation +20, +20"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "We can finally create the confusion matrix and plot the ROC curve for our optimal $k$-NN classifier. (Use the $k$ value you found above.) We'll plot the ROC curve for values between 0.1 and 1.0."
+ "### ROC curve and confusion matrix for the final model\n",
+ "ROC curves are a good way to visualize sensitivity vs. 1-specificity for varying cut off points. Now, implement, in \"model.ipynb\", a \"ROC\" function that predicts the labels of the test set examples using different $threshold$ values in \"predict\" and plot the ROC curve. \"ROC\" takes a list containing different $threshold$ parameter values to try and returns two arrays; one where each entry is the sensitivity at a given threshold and the other where entries are 1-specificities."
]
},
{
- "cell_type": "code",
- "execution_count": null,
+ "cell_type": "markdown",
"metadata": {},
- "outputs": [],
"source": [
- "# confusion matrix\n",
- "conf_matrix(true_classes, predicted_classes)"
+ "We can finally create the confusion matrix and plot the ROC curve for our optimal $k$-NN classifier. Use the $k$ value you found above, if you completed TASK 5, else use $k$ = 10. We'll plot the ROC curve for values between 0.1 and 1.0."
]
},
{
diff --git a/model.ipynb b/model.ipynb
index 03b6a2f9dbc97f0ebae63efb4ce7b8959cf29cd1..0d6d853813c2d7bd7be9394b3ee391e71a524a5c 100644
--- a/model.ipynb
+++ b/model.ipynb
@@ -25,7 +25,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "You might need to preprocess your dataset depending on which dataset you are using. This step is for reading the dataset and for extracting features and labels. The \"preprocess\" function should return an $n \\times d$ features array, and an $n \\times 1$ labels array, where $n$ is the number of examples and $d$ is the number of features in the dataset. In cases where there is a big difference between the scales of feautures, we want to normalize the features to have values in the same range [0,1]. If that is the case with your dataset, output normalized features to get better prediction results."
+ "This step is for reading the dataset and for extracting features and labels. The \"preprocess\" function should return an $n \\times d$ \"features\" array, and an $n \\times 1$ \"labels\" array, where $n$ is the number of examples and $d$ is the number of features in the dataset. In cases where there is a big difference between the scales of features, we want to normalize the features to have values in the same range [0,1]. Since this is not the case with this dataset, we will not do normalization."
]
},
{
@@ -34,14 +34,25 @@
"metadata": {},
"outputs": [],
"source": [
- "def preprocess(file_path):\n",
+ "#TODO: GIVE!\n",
+ "def preprocess(feature_file, label_file):\n",
" '''\n",
- " file_path: where to read the dataset from\n",
- " returns nxd features, nx1 labels\n",
+ " Args:\n",
+ " feature_file: str \n",
+ " file containing features\n",
+ " label_file: str\n",
+ " file containing labels\n",
+ " Returns:\n",
+ " features: ndarray\n",
+ " nxd features\n",
+ " labels: ndarray\n",
+ " nx1 labels\n",
" '''\n",
" # You might find np.genfromtxt useful for reading in the file. Be careful with the file delimiter, \n",
" # e.g. for comma-separated files use delimiter=',' argument.\n",
" \n",
+ " # TODO \n",
+ " \n",
" raise NotImplementedError\n",
"\n",
" \n",
@@ -52,7 +63,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "Next, you'll need to split your dataset into training and validation and test sets. The \"split\" function should take as input the size of the whole dataset and randomly sample a proportion $p$ of the dataset as test partition and a proportion of $v$ as validation partition. The remaining will be used as training data. For example, to keep 30% of the examples as test and %10 as validation, set $p=0.3$ and $v=0.1$. You should choose these values according to the size of the data available to you. The \"split\" function should return indices of the training, validation and test sets. These will be used to index into the whole training set."
+ "Next, you'll need to split your dataset into training, validation and test sets. The \"partition\" function should take as input the size of the whole dataset and randomly sample a proportion $t$ of the dataset as test partition and a proportion of $v$ as validation partition. The remaining will be used as training data. For example, to keep 30% of the examples as test and %10 as validation, set $t=0.3$ and $v=0.1$. You should choose these values according to the size of the data available to you. The \"split\" function should return indices of the training, validation and test sets. These will be used to index into the whole training set."
]
},
{
@@ -61,23 +72,34 @@
"metadata": {},
"outputs": [],
"source": [
- "def partition(size, p, v = 0):\n",
+ "#TODO: GIVE!\n",
+ "def partition(size, t, v = 0):\n",
" '''\n",
- " size: number of examples in the whole dataset\n",
- " p: proportion kept for test\n",
- " v: proportion kept for validation\n",
+ " Args:\n",
+ " size: int\n",
+ " number of examples in the whole dataset\n",
+ " t: float\n",
+ " proportion kept for test\n",
+ " v: float\n",
+ " proportion kept for validation\n",
+ " Returns:\n",
+ " test_indices: ndarray\n",
+ " 1D array containing test set indices\n",
+ " val_indices: ndarray\n",
+ " 1D array containing validation set indices\n",
" '''\n",
" \n",
- " # np.random.choice might come in handy. Do not sample with replacement!\n",
- " # Be sure to not use the same indices in test and validation sets!\n",
+ " # np.random.permutation might come in handy. Do not sample with replacement!\n",
+ " # Be sure not to use the same indices in test and validation sets!\n",
" \n",
- " # use the first np.ceil(size*p) for test, \n",
+ " # use the first np.ceil(size*t) for test, \n",
" # the following np.ceil(size*v) for validation set.\n",
" \n",
+ " # TODO\n",
+ " \n",
" raise NotImplementedError\n",
" \n",
- " # return two 1d arrays: one keeping validation set indices, the other keeping test set indices \n",
- " return val_indices, test_indices"
+ " return test_indices, val_indices"
]
},
{
@@ -98,7 +120,10 @@
"metadata": {},
"outputs": [],
"source": [
+ "# TODO: Programming Assignment 2\n",
+ "\n",
"def kfold(indices, k):\n",
+ "\n",
" '''\n",
" Args:\n",
" indices: ndarray\n",
@@ -121,6 +146,48 @@
" return fold_dict"
]
},
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Choice of distance metric plays an important role in the performance of $k$-NN. Let's start with implementing a distance method in the \"distance\" function below. It should take two data points and the name of the metric and return a scalar value."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "#TODO: Programming Assignment 1\n",
+ "def distance(x, y, metric):\n",
+ " '''\n",
+ " Args:\n",
+ " x: ndarray \n",
+ " 1D array containing coordinates for a point\n",
+ " y: ndarray\n",
+ " 1D array containing coordinates for a point\n",
+ " metric: str\n",
+ " Euclidean, Hamming \n",
+ " Returns:\n",
+ " \n",
+ " '''\n",
+ " if metric == 'Euclidean':\n",
+ " raise NotImplementedError\n",
+ " elif metric == 'Hammming':\n",
+ " raise NotImplementedError\n",
+ " else:\n",
+ " raise ValueError('{} is not a valid metric.'.format(metric))\n",
+ " return dist # scalar distance btw x and y"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We will extend \"Model\" class while implementing supervised learning algorithms."
+ ]
+ },
{
"cell_type": "code",
"execution_count": 1,
@@ -128,18 +195,18 @@
"outputs": [],
"source": [
"class Model:\n",
- " # set the preprocessing function, partition_function\n",
+ " # preprocess_f and partition_f expect functions\n",
" # use kwargs to pass arguments to preprocessor_f and partition_f\n",
- " # kwargs is a dictionary and should contain p, v and file_path\n",
- " # e.g. {'p': 0.3, 'v': 0.1, 'file_path': some_path}\n",
+ " # kwargs is a dictionary and should contain t, v, feature_file, label_file\n",
+ " # e.g. {'t': 0.3, 'v': 0.1, 'feature_file': 'some_file_name', 'label_file': 'some_file_name'}\n",
" \n",
" def __init__(self, preprocessor_f, partition_f, **kwargs):\n",
" \n",
- " self.features, self.labels = preprocessor_f(kwargs['file_path'])\n",
+ " self.features, self.labels = preprocessor_f(kwargs['feature_file'], kwargs['label_file'])\n",
" self.size = len(self.labels) # number of examples in dataset \n",
" self.feat_dim = self.features.shape[1] # number of features\n",
" \n",
- " self.val_indices, self.test_indices = partition_f(self.size, kwargs['p'], kwargs['v'])\n",
+ " self.val_indices, self.test_indices = partition_f(self.size, kwargs['t'], kwargs['v'])\n",
" self.val_size = len(self.val_indices)\n",
" self.test_size = len(self.test_indices)\n",
" \n",
@@ -158,7 +225,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "## General supervised learning related functions \n",
+ "## General supervised learning performance related functions \n",
"### (To be implemented later when it is indicated in other notebooks)"
]
},
@@ -175,10 +242,17 @@
"metadata": {},
"outputs": [],
"source": [
+ "# TODO: Programming Assignment 1\n",
+ "\n",
"def conf_matrix(true, pred):\n",
" '''\n",
- " true: nx1 array of true labels for test set\n",
- " pred: nx1 array of predicted labels for test set\n",
+ " Args: \n",
+ " true: ndarray\n",
+ " nx1 array of true labels for test set\n",
+ " pred: ndarray \n",
+ " nx1 array of predicted labels for test set\n",
+ " Returns:\n",
+ " ndarray\n",
" '''\n",
" raise NotImplementedError\n",
" \n",
@@ -203,13 +277,21 @@
"metadata": {},
"outputs": [],
"source": [
+ "# TODO: Programming Assignment 1\n",
+ "\n",
"def ROC(model, indices, value_list):\n",
" '''\n",
- " model: a fitted supervised learning model\n",
- " indices: for data points to predict\n",
- " value_list: array containing different threshold values\n",
- " Calculate sensitivity and 1-specificity for each point in value_list\n",
- " Return two nX1 arrays: sens (for sensitivities) and spec_ (for 1-specificities)\n",
+ " Args:\n",
+ " model: a fitted supervised learning model\n",
+ " indices: ndarray\n",
+ " 1D array containing indices\n",
+ " value_list: ndarray\n",
+ " 1D array containing different threshold values\n",
+ " Returns:\n",
+ " sens: ndarray\n",
+ " 1D array containing sensitivities\n",
+ " spec_: ndarray\n",
+ " 1D array containing 1-specifities\n",
" '''\n",
" \n",
" # use predict method to obtain predicted labels at different threshold values\n",