diff --git a/ProgrammingAssignment_1/ProgrammingAssignment1.ipynb b/ProgrammingAssignment_1/ProgrammingAssignment1.ipynb
index ef2dfceeaa732e83a318dd2a6f415cd755673e1b..c731ed8fe8368e101057b632a72d41638a19d1de 100644
--- a/ProgrammingAssignment_1/ProgrammingAssignment1.ipynb
+++ b/ProgrammingAssignment_1/ProgrammingAssignment1.ipynb
@@ -11,13 +11,13 @@
     "* number of features (attributes) should be at least 50\n",
     "* number of examples (instances) should be between 1,000 - 5,000\n",
     "\n",
-    "A skeleton of a general supervised learning model is provided in \"model.ipynb\". Please look through it and complete the \"preprocess\" and \"partition\" methods. \n",
+    "A skeleton of a general supervised learning model is provided in \"model.ipynb\". The functions that will be implemented there will be indicated in this notebook. \n",
     "\n",
     "### Assignment Goals:\n",
     "In this assignment, we will:\n",
-    "* learn to split a dataset into training/validation/test partitions \n",
+    "* we'll implement 'Euclidean' and 'Manhattan' distance metrics \n",
     "* use the validation dataset to find a good value for *k*\n",
-    "* Having found the \"best\" *k*, we'll obtain final performance measures:\n",
+    "* Evaluate our model with respect to performance measures:\n",
     "    * accuracy, generalization error and ROC curve\n"
    ]
   },
@@ -45,8 +45,13 @@
     "|---|----------------|-----|-----|\n",
     "| 5 | Optimizing *k* | 10  | 10  |\n",
     "\n",
+    "### Bonus for 478/878\n",
     "\n",
-    "Points are broken down further below in Rubric sections. The **first** score is for 478, the **second** is for 878 students. There a total of 100 points in this assignment and extra 10 bonus points for 478 students."
+    "|   | Tasks          | 478 | 878 |\n",
+    "|---|----------------|-----|-----|\n",
+    "| 7 | Assess suitability of *k*-NN | 10  | 10  |\n",
+    "\n",
+    "Points are broken down further below in Rubric sections. The **first** score is for 478, the **second** is for 878 students. There a total of 100 points in this assignment and extra 20 bonus points for 478 students and 10 bonus points for 878 students."
    ]
   },
   {
@@ -93,7 +98,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Choice of distance metric plays an important role in the performance of *k*-NN. Let's start with implementing a distance method in the \"distance\" function in **model.ipynb**. It should take two data points and the name of the metric and return a scalar value."
+    "Choice of distance metric plays an important role in the performance of *k*-NN. Let's start with implementing a distance method  in the \"distance\" function in **model.ipynb**. It should take two data points and the name of the metric and return a scalar value."
    ]
   },
   {
@@ -102,7 +107,7 @@
    "source": [
     "### Rubric:\n",
     "* Euclidean +5, +5\n",
-    "* Hamming +5, +5"
+    "* Manhattan +5, +5"
    ]
   },
   {
@@ -121,7 +126,8 @@
     "x = np.array(range(100))\n",
     "y = np.array(range(100, 200))\n",
     "dist_euclidean = distance(x, y, 'Euclidean')\n",
-    "dist_hamming = distance(x, y, 'Hamming')"
+    "dist_manhattan = distance(x, y, 'Manhattan')\n",
+    "print('Euclidean distance: {}, Manhattan distance: {}'.format(dist_euclidean, dist_manhattan))"
    ]
   },
   {
@@ -135,7 +141,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "We can start implementing our *k*-NN classifier. *k*-NN class inherits Model class. You'll need to implement \"fit\" and \"predict\" methods. Use the \"distance\" function you defined above. \"fit\" method takes *k* as an argument. \"predict\" takes as input an *mxd* array containing *d*-dimensional *m* feature vectors for examples and outputs the predicted class and the ratio of positive examples in *k* nearest neighbors."
+    "We can start implementing our *k*-NN classifier. *k*-NN class inherits Model class. Use the \"distance\" function you defined above. \"fit\" method takes *k* as an argument. \"predict\" takes as input an *mxd* array containing *d*-dimensional *m* feature vectors for examples and outputs the predicted class and the ratio of positive examples in *k* nearest neighbors."
    ]
   },
   {
@@ -158,25 +164,39 @@
     "    Inherits Model class. Implements the k-NN algorithm for classification.\n",
     "    '''\n",
     "       \n",
-    "    def fit(self, k, distance_f, **kwargs):\n",
+    "    def fit(self, training_features, training_labels, k, distance_f,**kwargs):\n",
     "        '''\n",
     "        Fit the model. This is pretty straightforward for k-NN.\n",
+    "        Args:\n",
+    "            training_features: ndarray\n",
+    "            training_labels: ndarray\n",
+    "            k: int\n",
+    "            distance_f: function\n",
+    "            kwargs: dict\n",
+    "                Contains keyword arguments that will be passed to distance_f\n",
     "        '''\n",
     "        # TODO\n",
-    "        # set self.k, self.distance_f, self.distance_metric\n",
-    "        raise NotImplementedError\n",
+    "        # set self.train_features, self.train_labels, self.k, self.distance_f, self.distance_metric\n",
     "        \n",
+    "        raise NotImplementedError\n",
+    "\n",
     "        return\n",
     "    \n",
     "    \n",
-    "    def predict(self, test_indices):\n",
-    "        \n",
+    "    def predict(self, test_features):\n",
+    "        '''\n",
+    "        Args:\n",
+    "            test_features: ndarray\n",
+    "                mxd array containing features for the points to be predicted\n",
+    "        Returns: \n",
+    "            ndarray\n",
+    "        '''\n",
     "        raise NotImplementedError\n",
     "        \n",
     "        pred = []\n",
     "        # TODO\n",
     "        \n",
-    "        # for each point in test points\n",
+    "        # for each point in test_features\n",
     "        # use your implementation of distance function\n",
     "        #  distance_f(..., distance_metric)\n",
     "        # to find the labels of k-nearest neighbors. \n",
@@ -184,6 +204,8 @@
     "        # Find the ratio of the positive labels\n",
     "        # and append to pred with pred.append(ratio).\n",
     "        \n",
+    "        # when calculating learning curve you can make use of\n",
+    "        # self.learning_curve and self.training_proportion\n",
     "\n",
     "        return np.array(pred)\n",
     "    "
@@ -210,7 +232,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Remember you need to provide values to *t*, *v* parameters for \"partition\" function and to *feature_file* and *label_file* for \"preprocess\" function."
+    "Preprocess the data files and partition the data."
    ]
   },
   {
@@ -219,10 +241,12 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# populate the keyword arguments dictionary kwargs\n",
-    "kwargs = {'t': 0.3, 'v': 0.1, 'feature_file': ..., 'label_file': ...}\n",
     "# initialize the model\n",
-    "my_model = kNN(preprocessor_f=preprocess, partition_f=partition, **kwargs)"
+    "my_model = kNN()\n",
+    "# obtain features and labels from files\n",
+    "features, labels = preprocess(feature_file=..., label_file=...)\n",
+    "# partition the data set\n",
+    "val_indices, test_indices, train_indices = partition(size=..., t = 0.3, v = 0.1)"
    ]
   },
   {
@@ -238,15 +262,17 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "# pass the training features and labels to the fit method\n",
     "kwargs_f = {'metric': 'Euclidean'}\n",
-    "my_model.fit(k = 10, distance_f=distance, **kwargs_f)"
+    "my_model.fit(training_features=..., training_labels-..., k=10, distance_f=..., **kwargs_f)"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Evaluate your model on the test data and report your **accuracy**. Also, calculate and report the confidence interval on the generalization **error** estimate."
+    "### Computing the confusion matrix for *k* = 10\n",
+    "Now that we have the true labels and the predicted ones from our model, we can build a confusion matrix and see how accurate our model is. Implement the \"conf_matrix\" function (in model.ipynb) that takes as input an array of true labels (*true*) and an array of predicted labels (*pred*). It should output a numpy.ndarray. You do not need to change the value of the threshold parameter yet."
    ]
   },
   {
@@ -255,22 +281,28 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "final_labels = my_model.predict(my_model.test_indices)\n",
+    "# TODO\n",
+    "\n",
+    "\n",
+    "# get model predictions\n",
+    "pred_ratios = my_model.predict(my_model.features[my_model.test_indices])\n",
     "\n",
     "# For now, we will consider a data point as predicted in the positive class if more than 0.5 \n",
     "# of its k-neighbors are positive.\n",
     "threshold = 0.5\n",
+    "# convert predicted ratios to predicted labels\n",
+    "pred_labels = None\n",
     "\n",
-    "# TODO\n",
-    "# Calculate and report accuracy and generalization error with confidence interval here. Show your work in this cell."
+    "# obtain true positive, true negative,\n",
+    "#false positive and false negative counts using conf_matrix\n",
+    "tp,tn, fp, fn = conf_matrix(...)"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### Computing the confusion matrix for *k* = 10\n",
-    "Now that we have the true labels and the predicted ones from our model, we can build a confusion matrix and see how accurate our model is. Implement the \"conf_matrix\" function (in model.ipynb) that takes as input an array of true labels (*true*) and an array of predicted labels (*pred*). It should output a numpy.ndarray. You do not need to change the value of the threshold parameter yet."
+    "Evaluate your model on the test data and report your **accuracy**. Also, calculate and report the 95% confidence interval on the generalization **error** estimate."
    ]
   },
   {
@@ -280,7 +312,10 @@
    "outputs": [],
    "source": [
     "# TODO\n",
-    "conf_matrix(my_model.labels[my_model.test_indices], final_labels, threshold = 0.5)"
+    "# Calculate and report accuracy and generalization error with confidence interval here. Show your work in this cell.\n",
+    "\n",
+    "print('Accuracy: {}'.format(accuracy))\n",
+    "print('Confidence interval: {}-{}'.format(lower_bound, upper_bound))"
    ]
   },
   {
@@ -309,15 +344,22 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# try sizes 50, 100, 150, 200, ..., up to the largest multiple of 50 >= train_size\n",
-    "training_sizes = np.arange(50, my_model.train_size + 1, 50)\n",
+    "# train using %10, %20, %30, ..., 100% of training data\n",
+    "training_proportions = np.arange(0.10, 1.01, 0.10)\n",
+    "train_size = len(train_indices)\n",
+    "training_sizes = np.int(np.ceil(size*proportion))\n",
     "\n",
     "# TODO\n",
+    "error_train = []\n",
+    "error_val = []\n",
     "\n",
-    "# Calculate error for each entry in training_sizes\n",
-    "# for training and validation sets and populate\n",
-    "# error_train and error_val arrays. Each entry in these arrays\n",
-    "# should correspond to each entry in training_sizes.\n",
+    "# For each size in training_sizes\n",
+    "for size in training_sizes:\n",
+    "    # fit the model using \"size\" data porint\n",
+    "    # Calculate error for training and validation sets\n",
+    "    # populate error_train and error_val arrays. \n",
+    "    # Each entry in these arrays\n",
+    "    # should correspond to each entry in training_sizes.\n",
     "\n",
     "plt.plot(training_sizes, error_train, 'r', label = 'training_error')\n",
     "plt.plot(training_sizes, error_val, 'g', label = 'validation_error')\n",
@@ -393,11 +435,26 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "# TODO\n",
     "# ROC curve\n",
     "roc_sens, roc_spec_ = ROC(my_model, my_model.test_indices, np.arange(0.1, 1.0, 0.1))\n",
     "plt.plot(roc_sens, roc_spec_)\n",
     "plt.show()"
    ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## TASK 7: Assess suitability of *k*-NN to your dataset"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Use this cell to write about your understanding of why *k*-NN performed well if it did or why not if it didn't."
+   ]
   }
  ],
  "metadata": {
diff --git a/ProgrammingAssignment_1/ProgrammingAssignment1_solution.ipynb b/ProgrammingAssignment_1/ProgrammingAssignment1_solution.ipynb
new file mode 100644
index 0000000000000000000000000000000000000000..c3535482ded9e01f384f660abad8e0d6d88e74e8
--- /dev/null
+++ b/ProgrammingAssignment_1/ProgrammingAssignment1_solution.ipynb
@@ -0,0 +1,581 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# *k*-Nearest Neighbor\n",
+    "\n",
+    "We'll implement *k*-Nearest Neighbor (*k*-NN) algorithm for this assignment. We recommend using [Madelon](https://archive.ics.uci.edu/ml/datasets/Madelon) dataset, although it is not mandatory. If you choose to use a different dataset, it should meet the following criteria:\n",
+    "* dependent variable should be binary (suited for binary classification)\n",
+    "* number of features (attributes) should be at least 50\n",
+    "* number of examples (instances) should be between 1,000 - 5,000\n",
+    "\n",
+    "A skeleton of a general supervised learning model is provided in \"model.ipynb\". The functions that will be implemented there will be indicated in this notebook. \n",
+    "\n",
+    "### Assignment Goals:\n",
+    "In this assignment, we will:\n",
+    "* we'll implement 'Euclidean' and 'Manhattan' distance metrics \n",
+    "* use the validation dataset to find a good value for *k*\n",
+    "* Evaluate our model with respect to performance measures:\n",
+    "    * accuracy, generalization error and ROC curve\n",
+    "* Try to assess if *k*-NN is suitable for the dataset you used\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# GRADING\n",
+    "\n",
+    "You will be graded on parts that are marked with **\\#TODO** comments. Read the comments in the code to make sure you don't miss any.\n",
+    "\n",
+    "### Mandatory for 478 & 878:\n",
+    "\n",
+    "|   | Tasks                      | 478 | 878 |\n",
+    "|---|----------------------------|-----|-----|\n",
+    "| 1 | Implement `distance`       |  10 |  10 |\n",
+    "| 2 | Implement `k-NN` methods   |  25 |  20 |\n",
+    "| 3 | Model evaluation           |  25 |  20 |\n",
+    "| 4 | Learning curve             |  20 |  20 |\n",
+    "| 6 | ROC curve analysis         |  20 |  20 |\n",
+    "\n",
+    "### Mandatory for 878, bonus for 478\n",
+    "\n",
+    "|   | Tasks          | 478 | 878 |\n",
+    "|---|----------------|-----|-----|\n",
+    "| 5 | Optimizing *k* | 10  | 10  |\n",
+    "\n",
+    "### Bonus for 478/878\n",
+    "\n",
+    "|   | Tasks          | 478 | 878 |\n",
+    "|---|----------------|-----|-----|\n",
+    "| 7 | Assess suitability of *k*-NN | 10  | 10  |\n",
+    "\n",
+    "Points are broken down further below in Rubric sections. The **first** score is for 478, the **second** is for 878 students. There a total of 100 points in this assignment and extra 20 bonus points for 478 students and 10 bonus points for 878 students."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You can use numpy for array operations and matplotlib for plotting for this assignment. Please do not add other libraries."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 119,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "import matplotlib.pyplot as plt"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Following code makes the Model class and relevant functions available from model.ipynb."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 134,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%run 'model_solution.ipynb'"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## TASK 1: Implement `distance` function"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Choice of distance metric plays an important role in the performance of *k*-NN. Let's start with implementing a distance method  in the \"distance\" function in **model.ipynb**. It should take two data points and the name of the metric and return a scalar value."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Rubric:\n",
+    "* Euclidean +5, +5\n",
+    "* Manhattan +5, +5"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Test `distance`"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 136,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Euclidean distance: 1000.0, Manhattan distance: 10000\n"
+     ]
+    }
+   ],
+   "source": [
+    "x = np.array(range(100))\n",
+    "y = np.array(range(100, 200))\n",
+    "dist_euclidean = distance(x, y, 'Euclidean')\n",
+    "dist_manhattan = distance(x, y, 'Manhattan')\n",
+    "print('Euclidean distance: {}, Manhattan distance: {}'.format(dist_euclidean, dist_manhattan))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## TASK 2: Implement $k$-NN Class Methods"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We can start implementing our *k*-NN classifier. *k*-NN class inherits Model class. Use the \"distance\" function you defined above. \"fit\" method takes *k* as an argument. \"predict\" takes as input an *mxd* array containing *d*-dimensional *m* feature vectors for examples and outputs the predicted class and the ratio of positive examples in *k* nearest neighbors."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Rubric:\n",
+    "* correct implementation of fit method +5, +5\n",
+    "* correct implementation of predict method +20, +15"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 137,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "class kNN(Model):\n",
+    "    '''\n",
+    "    Inherits Model class. Implements the k-NN algorithm for classification.\n",
+    "    '''\n",
+    "       \n",
+    "    def fit(self, training_features, training_labels, k, distance_f, **kwargs):\n",
+    "        '''\n",
+    "        Fit the model. This is pretty straightforward for k-NN.\n",
+    "        Args:\n",
+    "            training_features: ndarray\n",
+    "            training_labels: ndarray\n",
+    "            k: int\n",
+    "            distance_f: function\n",
+    "            kwargs: dict\n",
+    "                Contains keyword arguments that will be passed to distance_f\n",
+    "        '''\n",
+    "        # TODO\n",
+    "        # set self.train_features, self.train_labels,self.k, self.distance_f, self.distance_metric\n",
+    "        self.train_features = training_features \n",
+    "        self.train_labels = training_labels\n",
+    "        self.k = k\n",
+    "        self.distance_f = distance_f\n",
+    "        self.distance_metric = kwargs['metric']\n",
+    "    \n",
+    "        return\n",
+    "    \n",
+    "    \n",
+    "    def predict(self, test_features):\n",
+    "            \n",
+    "        test_size = len(test_features)\n",
+    "        train_size = len(self.train_labels)\n",
+    "        \n",
+    "        pred = np.empty(len(test_features))\n",
+    "        # TODO\n",
+    "        # for each point in test points\n",
+    "        for idx in range(test_size):\n",
+    "            point = test_features[idx]    \n",
+    "            distances = []\n",
+    "            labels = []\n",
+    "            \n",
+    "\n",
+    "            for tr_idx in range(train_size):\n",
+    "                train_example = self.train_features[tr_idx]\n",
+    "                train_label = self.train_labels[tr_idx]\n",
+    "                dist = self.distance_f(point, train_example, metric = self.distance_metric)\n",
+    "                distances.append(dist)\n",
+    "                labels.append(train_label)\n",
+    "            \n",
+    "            # get the order of distances\n",
+    "            dist_order = np.argsort(distances)\n",
+    "            # get the labels of k points that are closest to test point\n",
+    "            k_labels = list(np.array(labels)[dist_order[::-1]][:self.k])\n",
+    "            \n",
+    "            # get number of positive labels in k neighbours\n",
+    "            b = k_labels.count(1)\n",
+    "            \n",
+    "            pred[idx] = b/self.k\n",
+    "    \n",
+    "        return pred\n",
+    "    "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## TASK 3: Build and Evaluate the Model"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Rubric:\n",
+    "* Reasonable accuracy values +10, +5\n",
+    "* Reasonable confidence intervals on the error estimate +10, +10\n",
+    "* Reasonable confusion matrix +5, +5"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Preprocess the data files and partition the data."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 138,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# initialize the model\n",
+    "my_model = kNN()\n",
+    "# obtain features and labels from files\n",
+    "features, labels = preprocess('../data/madelon.data', '../data/madelon.labels')\n",
+    "# partition the data set\n",
+    "val_indices, test_indices, train_indices = partition(features.shape[0], t = 0.3, v = 0.1)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Assign a value to *k* and fit the *k*-NN model."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 139,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# pass the training features and labels to the fit method\n",
+    "kwargs_f = {'metric': 'Euclidean'}\n",
+    "my_model.fit(features[train_indices], labels[train_indices], k=10, distance_f=distance, **kwargs_f)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Computing the confusion matrix for *k* = 10\n",
+    "Now that we have the true labels and the predicted ones from our model, we can build a confusion matrix and see how accurate our model is. Implement the \"conf_matrix\" function (in model.ipynb) that takes as input an array of true labels (*true*) and an array of predicted labels (*pred*). It should output a numpy.ndarray. You do not need to change the value of the threshold parameter yet."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 140,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# TODO\n",
+    "\n",
+    "# get model predictions\n",
+    "pred_ratios = my_model.predict(features[test_indices])\n",
+    "# For now, we will consider a data point as predicted in the positive class if more than 0.5 \n",
+    "# of its k-neighbors are positive.\n",
+    "threshold = 0.5\n",
+    "# convert predicted ratios to predicted labels\n",
+    "pred_labels = [1 if x >= threshold else 0 for x in pred_ratios]\n",
+    "tp,tn, fp, fn = conf_matrix(labels[test_indices], pred_labels)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Evaluate your model on the test data and report your **accuracy**. Also, calculate and report the 95% confidence interval on the generalization **error** estimate."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 142,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Accuracy: 0.475\n",
+      "Confidence interval: 0.49110132745961876-0.5588986725403813\n"
+     ]
+    }
+   ],
+   "source": [
+    "# TODO\n",
+    "# Calculate and report accuracy and generalization error with confidence interval here. Show your work in this cell.\n",
+    "accuracy = (tp+tn)/len(test_indices)\n",
+    "error = 1 - accuracy\n",
+    "diff = 0.96 * np.sqrt((error * (1 - error)) / len(test_indices))\n",
+    "lower_bound = error - diff\n",
+    "upper_bound = error + diff\n",
+    "print('Accuracy: {}'.format(accuracy))\n",
+    "print('Confidence interval: {}-{}'.format(lower_bound, upper_bound))\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    " ## TASK 4: Plotting a learning curve\n",
+    " \n",
+    "A learning curve shows how error changes as the training set size increases. For more information, see [learning curves](https://www.dataquest.io/blog/learning-curves-machine-learning/).\n",
+    "We'll plot the error values for training and validation data while varying the size of the training set. Report a good size for training set for which there is a good balance between bias and variance."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Rubric:\n",
+    "* Correct training error calculation for different training set sizes +8, +8\n",
+    "* Correct validation error calculation for different training set sizes +8, +8\n",
+    "* Reasonable learning curve +4, +4"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 144,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XlcVWX+wPHPwyaoqAiKu6CQO7iQOmUumQuUW1pqq1ZTWma2zExNM+3za89qsnGqsUVNMwu33Cot09REu6BiKuaGCiLuuLA9vz+eC4GCXODCgXu/79frvuSe89x7v4eD33Pu8zzne5TWGiGEEO7Bw+oAhBBCVB5J+kII4UYk6QshhBuRpC+EEG5Ekr4QQrgRSfpCCOFGJOkLIYQbkaQvhBBuRJK+EEK4ES+rA7hUUFCQDgkJsToMIYSoVjZv3nxMa92gpHZVLumHhIQQFxdndRhCCFGtKKX2O9JOuneEEMKNSNIXQgg3IklfCCHciCR9IYRwI5L0hRDCjUjSF0IINyJJXwgh3EiVm6cvhKhgubkwdy6kpkLz5ubRrBk0agSenlZHJyqYJH0h3MnevXDPPfDDD5ev8/KCJk3+OAjkHRAKPm/YEDykg6A6k6QvhDvQGj74AJ54ApSCjz6C4cMhOdk8Dh7845GcDJs3w4IFcPFi4ffx9jYHgOIOCs2bQ1CQ+QxRJUnSF8LVHTgA994L330HN9wA//sftGhh1gUGQmRk0a/TGo4dK/qgcPAgrF8PX34JWVmFX1ejxpUPCs2bQ0CAHBgsIklfCFelNcyYAY8+avrxp0+H++93PNkqBQ0amEeXLkW3yc2FtLTCB4WCB4Yff4RDhyAnp/DratY0B4LWrWHUKPOoU6d82yscorTWVsdQSFRUlJaCa0KUU3Iy/PnPsHw59O1rkn9oqDWx5OSYQeOiDgq//gpJSeDra7qb7rwTBg404wuiVJRSm7XWUSW1k9+sEK5Ea/jsM3jkEdPt8u9/w4MPWjv46ulpBoibNIEePQqv0xo2bTIxz51rHsHBMHasOQB06SLdQE4mw/BCuIojR2DoUBg3Djp1gvh4mDSpas+2UQq6d4f33oPDh83gca9e8P770K2b2Y5XXzXfDIRTVOG/BiGEQ7SG2bOhQwczWDt1qulLDwuzOrLS8fGBYcNg/nxISTFjEHXrwpNPmoHnG24w3wjOnrU60mpNkr4Q1VlqKowcCXfcAW3bgs0GU6ZU7bN7RwQEwAMPwLp1sHs3PPOMucbg7rtN98+dd8LKlZcPEIsSVfO/DCHc2Lx55ux+6VJ4/XX46Sdo08bqqJwvLAyee84M+K5daxL+kiUwaJD5BvCXv8DWrVZHWW1I0heiuklLg1tvhdGjoVUrMwPmiSdcv4SCUnDttabb58gR0w0UFQVvvw0REdC5M7z5plkniiVJX4jq5Ouvzdn9ggXwf/8HP/8M7dpZHVXl8/U13VoLF5ok/9575qKwJ54w8/8HD4bPP4dz56yOtMqRpC9EdZCeDrfdZhJd8+amTMJTT8l8djBlHx56CDZuhN9+M7+X336D2283/f/jx8OqVeZCMuFY0ldKDVZK7VRKJSmlnixi/TilVJpSymZ/3Gdf3lkptV4ptV0plaCUGu3sDRDC5S1aBB07mpIHL7wAGzaYqYzicm3awEsvwe+/m6Jyo0ebb0f9+0NIiDkgJCZaHaWlSrwiVynlCewCBgDJwCZgrNY6sUCbcUCU1nrSJa+9CtBa691KqSbAZqCd1vpkcZ8nV+QKYXfihJmJ89lnps/6009Nv7UonfPnzYHzs89gxQoz46dbN7jrLhgzxlQOdQGOXpHryJl+dyBJa/271joTmAsMcyQIrfUurfVu+8+HgaNAA0deK4RbW7bMnN3Png3//Ke5alUSftn4+Zkz/m++MXWApk411zY88oi5Svimm8w6N+FI0m8KHCzwPNm+7FIj7V0485VSzS9dqZTqDvgAe8oUaTVx4vwJJi6ZSHxKvNWhiOro1Cm47z6IiTFz1TduNF06Pj5WR+YagoPNt6fNm2HbNjPwa7OZxD91qtXRVQpHkn5RhS8u7RNaDIRorSOA74BPC72BUo2BmcB4rfVloylKqfuVUnFKqbi0tDTHIq+iluxawvTN0+n+UXfe2fAOVa2gnajCvv3W9NV//LHpe9682XRDiIrRoQO88orp/x85Eh57zDx3cY4k/WSg4Jl7M+BwwQZa63Stdd7dFj4E8v9SlVJ1gG+Af2itNxT1AVrrD7TWUVrrqAYNqnfvjy3Fhq+XL4NaD2LKiinc+PmNpJ5NtTosUZWdOQMTJpjqkrVqmTr1//d/ZgqiqHg+PqbQ29ix5mD7/POm+8dFOZL0NwHhSqlQpZQPMAZYVLCB/Uw+z1Bgh325DxALfKa1/tI5IVdttlQbnRp2YuGYhUyLmcbqfauJmB7B8qTlVocmqqJVq8zZfd5drbZsMQXIROXy8oKZM02xuueeg6efdtnEX2LS11pnA5OAFZhkPk9rvV0p9YJSaqi92WT7tMx4YDIwzr78VqA3MK7AdE6XHY3SWmNLsdG5UWeUUjx49YNs+vMmGtZqSPTsaB5d/igXsy+W/EbC9Z09aypg9u9vzjTXrjWlFPz8rI7MfXl6mruK3X8/vPyyOQi7YuLXWlepR7du3XR1deDkAc1z6Gm/TCu0/FzmOf3w0oc1z6Ej/xOpE48mWhShqBJ+/FHrVq20VkrrKVO0zsiwOiJRUG6u1pMmaQ1aP/SQ1jk5VkfkECBOO5Bj5YpcJ7Kl2ADo3Kjwlxk/bz/ejX6XxWMXc+jMIbp90I3/xv1XBnndTU6OOXvs29c8/+EHM2OkZk0roxKXUgrefRcefxymTTPVPl3oal5J+k4Un2qmaXZqWPTVkjdddRMJExLo1aIXE76ZwMh5I0k/l16ZIQqr5OTAPfeYgmATJkBCAvTubXVUojhKme62p5+Gjz4ypRxcpIyzJH0nsqXYCKsfhn8N/2LbNPZvzPI7lvPGgDdYsmsJkdMjWb13dSVGKSpdXsL/7DMzM+T9980sHVG1KWVKOrzwgtl3d9xhbkFZzUnSd6K8QdySeCgPHr/mcTbct4FaPrXo/1l//v7938nKqf5/UOISOTnmLPGzz0zyeOYZqyMSpfXPf5r5+3PnmrINmZlWR1QukvSd5PTF0+w5sYfOwY5PTurauCtb7t/CvV3u5eW1L3PtjGtJOp5UgVGKSpWX8GfOhBdfNMlDVE9/+5sZf/n6a3Mh14ULVkdUZpL0nSQhNQG4fBC3JLV8avHh0A/58pYv2X18N13+24XP4j+TQd7qLifHzPmeOdN0EfzjH1ZHJMpryhTTNbdkibmX7/nzVkdUJpL0naS4mTuOGtV+FAkTEujauCt3L7ib27++nVMXTjkzRFFZcnLMvVxnzTIJ/+mnrY5IOMvEiWZg99tv4cYbISPD6ohKTZK+k9hSbATVDKKJf5Myv0fzus1ZddcqXur3EvO2z6Pzfzuz/uB6J0YpKlxewp89G/71L0n4rujee80YzY8/mjt0nT5tdUSlIknfSeJT4/OvxC0PTw9Pnu79NGvvWYtCcd3H1/Hijy+Sk+sa08VcWna2qdE+e7apnfP3v1sdkagod9xhbse4fr2pmXSy2FuEVDmS9J0gOzebralbiQyOdNp79mzWE9sEG2M6juGZH56h36f9OHDqgNPeXzhZdrY5w//8c3MJ/1NPWR2RqGijR5u7mW3ZYspppFePa24k6TvBzmM7uZhzscz9+cWpU6MOs26excwRM7Gl2IicHsmX292ibl31kneGn5fwn7zsjqLCVY0YAbGxsH07XH89HD1qdUQlkqTvBOUdxC3JHRF38OsDv9ImsA23zr+Vexfey9nMsxXyWaKUsrPhzjthzhwzl1sSvvu58UZzO8bdu6FfPzhyxOqIrkiSvhPYUmzU8KxBm8A2FfYZreu35qfxP/H0dU/zse1jun3Qjc2HN1fY5wkH5CX8uXPh1VfNXG7hngYOhKVLYf9+U1spOdnqiIolSd8JbKk2OjbsiLend4V+jrenNy9d/xKr717Nuaxz/Ol/f+KNn98g9/KbkYmKlp1tBvPmzoXXXoO//tXqiITV+vY1N14/cgT69DEHgCpIkn456QI19CtLn5A+xE+IZ0ibIfzl278waNYgjpyp2l8pXUpewv/iC5Pw//IXqyMSVcW118J338Hx46ag3p6qd0twSfrldOTsEY6dO1Y46ScmVnhhpvp+9Zl/y3w+uOkD1h1YR8T0CBbvXFyhnykwCf/2203Cf/11Sfjict27w/ffmxvl9O4NO3daHVEhkvTLKW8QN3+65vLl5obLzZub2ulbt1bYZyul+HO3P7PlgS00q9OMoXOHMmnpJM5nVc/Lw6u8rCy47TaYNw/eeMPsXyGK0rWruV9CVpbp6tm+3eqI8knSL6e8pB8RHGEWzJsHderANdfAO+9ARARERcF771XYPN62QW3ZcO8GHuv5GNM2TePqD69ma2rFHWzcUlaWOcP/8ktTE//xx62OSFR1nTqZq3Y9PEx/f3y81REBkvTLzZZio1VAK+r61jVf/RctgptuMtX4Dh82iT83Fx5+GBo3NhX6Fi92evdPDa8avDnoTZbfvpxj545x9YdX894v70nhNmfIO8PPS/iPPWZ1RKK6aNfOJH5fXzOdMy7O6ogk6ZdXoUHctWvN2fyIEeZ5gwYwebK5Ys9mMzfC/uknGDoUmjUzZ4tO7v4ZFDaIhIkJ9G/Vn4eXPczXO7526vu7nawsGDsW5s+Ht96ShC9KLzwc1qwxPQD9+8OGDZaGI0m/HM5cPEPS8aQ/aujHxkKNGqYI06UiI03SOHQIFi40o/zvvmu6f7p1g3//G44dc0pcDWs1ZNGYRQT6BbJg5wKnvKdbykv4X31l9t2jj1odkaiuQkNN4m/QAAYMMCd/FpGkXw5bj25Fo82ZvtawYIHZobVrF/8ib29zpl+w+0dr842gSROndf94engyOGwwy5OWyzz+ssjKMndJ+uorc/MMSfiivFq0MIm/aVNzYvj995aEIUm/HAqVX/j1Vzhw4I+uHUdUcPdPTHgMx84dI+6w9f2I1Upewv/6a3j7bXPzDCGcoUkT08ffqpUZ+1uxotJDkKRfDvEp8dT3q0+zOs1M146HBwwZUrY3K9j9s2gR9OplunzK0f0zqPUgFIqlu5eWLSZ3lJlpqifmJfxHHrE6IuFqgoNh9Wpo29ac4C2u3OtrJOmXgy3VRmRwpKmhv2ABXHedOXsvD29vc+D46ivT/fPuu2Z5Gbp/AmsG0qNZD5YlLStfTO4iL+HHxppuN0n4oqIEBZnunYgIuPlm8/+9kkjSL6Ps3GwSUhNM105SEmzbBsOHO/dDgoLMVM/Nm80c34cfNjOEStH9ExMWw6ZDmziaUfVLvloqL+EvWGAOtJMnWx2RcHX165uSDVdfbf725syplI+VpF9Gu9N3cyH7gkn6sbFmobOTfkEREWaOeHJyqbp/YsJj0GhWJFV+32G1kZkJt95qEv6//20OrkJUhrp1Tb/+tdeai/8+/bTCP1JVtYt3oqKidFwVuIChJHO2zuG2r28jfkI8ETdPhPPnzYBsZTp2zJwdfPKJ+ey8rqFx48zsAG9vcnUujd9szPWh1zNnpINnErm5ZnvOnTOPgj8XfBS1vKhltWqZA9PVV5urk8vbBeZMeQl/4UKT8CdNsjoi4Y4yMmDYMLh40ZRv8PQs9VsopTZrraNKaufl4JsNBt4BPIGPtNavXLJ+HPA6cMi+6D2t9Uf2dXcD/7Avf0lrXfGHskpgS7Hh4+lD2+x65j6Zzz1X+UHkdf88/DAkJJizhFmzzCBkw4YwejQeAQFE5zRgUUIsOUvuxvPchZIT9oULZYvHzw9q1vzj37zHoUNmHCLvBKNFC5P88w4C3bpBQIDzfi+OysyEW24x35zeew8eeqjyYxACzIlR3lhdGRJ+aZR4pq+U8gR2AQOAZGATMFZrnVigzTggSms96ZLX1gfigChAA5uBblrrE8V9XnU50x80axBpGWls4QGYMMEk3U6drA7L/NEsX27O/u1/RPO6+DB6WCbrlgRzzZl6hRNyUUm6LMt8fc3speKcPm2mtcbFmcemTYXLzrZuXfhA0LUr+PtX3O8pMxNGjTK/I0n4wgU480y/O5Cktf7d/sZzgWFA4hVfZQwCvtVaH7e/9ltgMFA5IxYVKD4lnpjwGJi+wCSsjh2tDsnI6+IZMsTUAvL0ZMCFk3i8HsTSV+/jmutfsiauOnVMtcE+ff5YduKEGaTOOxCsX29KFgMoBW3aFD4QdO5sDjDldfGiOcNfvBimTYMHHyz/ewpRTTiS9JsCBws8TwZ6FNFupFKqN+ZbwaNa64PFvLZpGWOtMlLOppCakUpk3avg+1lmpodSVod1OS+zewP8Arim+TUsS1rGS1Yl/aIEBMANN5hHnqNHCx8Ivv/edFmB+SbRocMfB4GoKDOQXaOG45958aI5w1+yBN5/HyZOdO42CVHFOZL0i8pml/YJLQbmaK0vKqUmAJ8C1zv4WpRS9wP3A7Ro0cKBkKyVfyXunnOmO6U0V+FaJCYshr+v+jtHzhyhsX9jq8MpXsOGEB1tHnkOHy7cLbRoEcyYYdZ5e5tutYIHgg4dzPJLXbxornP45hv4z39Mt5wQbsaRKZvJQPMCz5sBhws20Fqna60v2p9+CHRz9LX213+gtY7SWkc1qEozO4qRf+OU77aaq+t69rQ4opLFhMcAsDxpucWRlEGTJubahBdegGXLzLeBfftM5cvHHzffGObOhT//Gbp0MV1Jf/qTGeD+9FNzA4tz5yThC4FjZ/qbgHClVChmds4Y4LaCDZRSjbXWeTdpHQrssP+8Avg/pVTe1IyBwFPljtpithQbIXVbUm/Jd6YKYwWPtjtDRHAETfybsCxpGeO7jLc6nPJRClq2NI+RI80yrc3AcMFvBJ98YgZpweyjnByYPh0eeMCy0IWwWolJX2udrZSahEngnsAMrfV2pdQLQJzWehEwWSk1FMgGjgPj7K89rpR6EXPgAHghb1C3OrOl2OisGsPZ/dWiawfMrRWjw6KZnzifrJwsvD2L6P6ozpSCsDDzGDPGLMvJgV27zEFgyxZzQVveQUIINyUXZ5VSRmYG/i/78+ypzjz7URKkpZVuINFCX+/4mpHzRvLjuB/p3bK31eEIIZzI0SmbUoahlPJr6K/bAzfeWG0SPsANrW7Ay8OLZbulAJsQ7kqSfinFp5ibG3feebpia+1UgDo16tCrRS+WJkmpZSHclST9UrKl2Kina9DivHfhaYXVRExYDAmpCSSfTrY6FCGEBSTpl5ItxUZkqkLdMMBMDaxmosPNgapaTt0UQpSbJP1SyMnNISHFRud9F6pd106eDg060LxOc7mblhBuSpJ+KSQdT+JczgU6p2AuFqqGlFLEhMfw7e/fkpmTaXU4QohKJkm/FArdCD042OJoyi4mPIazmWdZe2Ct1aEIISqZJP1SsO38Ae8caN9/jNWhlMv1odfj4+kjUzeFcEOS9EshfscPtE8Dn5tvsTqUcqntU5veLXvL1E0h3JAk/VKwnfudzhcDoFUrq0Mpt5iwGBLTEtl/cr/VoQghKpEkfQel7tvGkRqZRDa/2upQnCJv6uayJOniEcKdSNJ3UPyS/wHQudcoiyNxjjaBbQitFypTN4VwM5L0HWSzmYuZIq+92eJInCNv6ub3e7/nQnYZb4QuhKh2JOk74swZbKd30SLXn/o1A62Oxmmiw6I5l3WOn/b/ZHUoQohKIknfEcuWYWuYS+eGEVZH4lT9QvtRw7OGdPEI4UYk6Tvg/ML57AyEzu36WR2KU9X0rkm/0H4ydVMINyJJvySZmWzbtJRcD+jcuKvV0ThddFg0u9J3sef4HqtDEUJUAkn6JVm1Cpt/BgCRjSItDsb58m6YLlM3hXAPkvRLEhuLrbk3dXzqEFIvxOponC6sfhjh9cOlX18INyFJ/0pyc2HhQmxt6hDZKBIP5Zq/rpjwGFbvW835rPNWhyKEqGCumcWcZcMGco+mEl/rrKms6aKiw6K5kH2BH/b9YHUoQogKJkn/SmJj2dPAiwx90aWTfp+QPvh5+UkXjxBuQJJ+cbQ2/fn9OwC4dNL39fKlf6v+LE1aitba6nCEEBVIkn5xtm+HPXuI79oYLw8v2jdob3VEFSo6LJrfT/zO7uO7rQ5FCFGBJOkXJzYWlMIWmE27oHb4evlaHVGFig4zVTeli0cI1yZJvzgLFkDPnthO7HDJ+fmXCg0IpV1QO0n6Qrg4SfpF2b8ftmwhbdgNHDpziM7BrtufX1B0WDQ/7v+RjMwMq0MRQlQQSfpFWbAAgPhrzB2yXHkQt6CY8BgyczJZtXeV1aEIISqIQ0lfKTVYKbVTKZWklHryCu1GKaW0UirK/txbKfWpUmqrUmqHUuopZwVeoRYsgA4dsHkeA1yz/EJRerXoRW2f2tLFI4QLKzHpK6U8gWlANNAeGKuUumwqi1LKH5gMbCyw+Baghta6E9ANeEApFVL+sCvQsWOwZg2MGIEtxUazOs0IqhlkdVSVooZXDW5odQPLkpbJ1E0hXJQjZ/rdgSSt9e9a60xgLjCsiHYvAq8BBW/DpIFaSikvwA/IBE6XL+QKtnixKb8wfDjxqfFu07WTJzosmv2n9rPj2A6rQxFCVABHkn5T4GCB58n2ZfmUUl2A5lrrJZe8dj6QARwBDgBvaK2Plz3cSrBgAbRowYWI9uxI2+E2g7h5ZOqmEK7NkaSviliW/91fKeUBTAUeL6JddyAHaAKEAo8rpVpd9gFK3a+UilNKxaWlpTkUeIXIyICVK2H4cLanJZKjc9ymPz9P87rN6dSwk5RaFsJFOZL0k4HmBZ43Aw4XeO4PdAR+UErtA3oCi+yDubcBy7XWWVrro8A6IOrSD9Baf6C1jtJaRzVo0KBsW+IMy5fDhQswfDi2FBvgPjN3CooOi+an/T9x+mLV7okTQpSeI0l/ExCulApVSvkAY4BFeSu11qe01kFa6xCtdQiwARiqtY7DdOlcr4xamAPCb07fCmeJjYXAQLjuOmwpNmr71KZVwGVfTFxeTHgMWblZfP/791aHIoRwshKTvtY6G5gErAB2APO01tuVUi8opYaW8PJpQG1gG+bg8bHWOqGcMVeMrCxYsgSGDAEvL2ypNiKDXbeG/pVc0/wa6tSoI/36QrggL0caaa2XAksvWfZMMW37Fvj5LGbaZtX3ww9w6hQMH06uziU+JZ67Iu+yOipLeHt6M6DVgPypm0oVNawjhKiO3O80tjixsVCzJgwcyN4TezmTecYt+/PzxITHcOjMIbYe3Wp1KEIIJ5KkD/m3RWTwYPDzIz41HnDPQdw8g8MGAzJ1UwhXI0kfYNMmOHwYhg8HwJZiw1N50qFBB4sDs04T/yZ0adRFpm4K4WIk6YPp2vHygptuAkzSbxPUBj9vP4sDs1Z0WDTrDqzj5IWTVocihHASSfpgrsLt2xcCAgCT9N25aydPTHgMOTqHb/d8a3UoQggnkaS/Ywfs3AkjRgCQfi6dg6cPul35haL0aNaDAN8A6eIRwoVI0o+NNf8OMzXkZBD3D14eXgxsPZBlScvI1blWhyOEcAJJ+gsWQPfu0NTUkMsrv+BuNXeKExMeQ8rZlPzfixCienPvpJ+cbGbu2Lt2wJzpN/FvQsNaDS0MrOrIm7q5bLd08QjhCtw76dtvi5g3VRNkEPdSDWs1JKpJFEuTZL6+EK5Akn7btuYBXMy+SGJaogziXiImLIYNyRs4fr5q3wpBCFEy9036x4+bejsFunYS0xLJzs2W/vxLxITHkKtzWblnpdWhCCHKyX2T/pIlkJNzWdcOyMydS0U1iSLQL1BKMgjhAtw36S9YYGbsRP1xTxdbio1a3rVoHdDawsCqHk8PTwaHDZapm0K4APdM+ufOmbtkDR8OHn/8CmypNiKCI/D08LQwuKopJjyGY+eOEXc4zupQhBDl4J5Jf+VKOH++UNeO1pr4lHjp2inGoNaDUCiZuilENeeeST821tTZ6dMnf9H+U/s5dfGUJP1iBNYMpEezHjJ1U4hqzv2SfnY2LF5sKmp6e+cvlkHcksWExbDp0CbSMtKsDkUIUUbul/TXrIETJwpN1QST9D2UBx0bdrQosKovJjwGjWbFnhVWhyKEKCP3S/qxseDrCwMHFlpsS7FxVeBV1PSuaVFgVV+Xxl1oWKuhTN0Uohpzr6SvtZmqOWgQ1KpVaJWUXyiZh/IgOiyaFXtWkJObY3U4QogycK+kv3mzKbJ2SdfOifMn2H9qv5RfcEBMeAzHzx/nl0O/WB2KEKIM3Cvpx8aCp2f+bRHzSA19xw1oNQAP5SFdPEJUU+6V9BcsgN69ITCw0OL4FEn6jgrwC+Ca5tfI1E0hqin3Sfq7dkFi4mVdO2CuxG1UuxHBtYMtCKz6iQmLYcuRLaScTbE6FCFEKblP0r/ktogFySBu6cSExwCwPGm5xZEIIUrLfZL+ggXQrRu0aFFocWZOJtuPbicyWMopOyoiOIIm/k2kX1+Iasg9kv7hw7BhQ5FdOzvSdpCVmyVn+qWglCI6LJqVe1aSnZttdThCiFJwKOkrpQYrpXYqpZKUUk9eod0opZRWSkUVWBahlFqvlNqulNqqlPJ1RuClsnCh+bdAgbU8Un6hbGLCYzh18RTrD663OhQhRCmUmPSVUp7ANCAaaA+MVUq1L6KdPzAZ2FhgmRcwC5igte4A9AWynBJ5aSxYAOHh0P6ysLGl2PDz8iO8fnilh1Wd3dDqBrw8vKSLR4hqxpEz/e5Aktb6d611JjAXuHw0FF4EXgMuFFg2EEjQWscDaK3TtdaVeynnyZOwapXp2lHqstVSQ79s6tSoQ68WvWTqphDVjCNJvylwsMDzZPuyfEqpLkBzrfWSS157FaCVUiuUUluUUn8t6gOUUvcrpeKUUnFpaU6u4PjNN6ayZhFdO1JDv3xiwmJISE3g0OlDVocihHCQI0n/8tNj0PkrlfIApgKPF9HOC+gF3G7/d4RSqv9lb6b1B1rrKK11VIMGDRwK3GGxsdC4MfTocdmqg6cPcuLCCUkjlTgiAAAgAElEQVT6ZRQdHg3AsiS5sYoQ1YUjST8ZaF7geTPgcIHn/kBH4Ael1D6gJ7DIPpibDPyotT6mtT4HLAW6OiNwh5w/b26LOGxYodsi5skbxJXpmmXToUEHmtdpLv36QlQjjiT9TUC4UipUKeUDjAEW5a3UWp/SWgdprUO01iHABmCo1joOWAFEKKVq2gd1+wCJTt+K4nz3HWRkFDlVE0zSVyg6BXeqtJBciVKKmPAYvvv9OzJzMq0ORwjhgBKTvtY6G5iESeA7gHla6+1KqReUUkNLeO0J4C3MgcMGbNFaf1P+sB0UGwt160LfvkWutqXYCA8Mp7ZP7UoLydXEhMdwJvMM6w6sszoUIYQDvBxppLVeiumaKbjsmWLa9r3k+SzMtM3KlZ0NixbBjTeCj0+RTWwpNq5uenUlB+Zarg+9Hh9PH5buXkq/0H5WhyOEKIHrXpG7bh2kpxfbtXPqwin2ntwrNfTLqbZPbXq37C2DuUJUE66b9GNjoUYNGDy4yNUJqQmAXInrDDFhMWxP287+k/utDkUIUQKHuneqnbzbIg4YALWL7q+X8gvOEx0ezWMrH2NZ0jImRE2wOpwqLSMzA6WU3Iu5CsnKyWLfyX3oP2aiW8bXy5cWdVuU3LAcXDPp22ywfz88U+Swg2mSYqNhrYY0qt2oEgNzTW0C2xBaL1SSfgk2JG9g4MyBnM08S3hgOBHBEUQ0jDD/BkfQsl5LPJTrfvmuSrTW/HLoF2YmzGTutrmkn0+3OiQAejTtwYb7NlToZ7hm0o+NNfPyhwwptokt1UZkcCSqiNIMonTypm5+bPuYi9kXqeFVw+qQqpzEtERu/PxGGtZqyGN/eoytR7diS7HxVeJX+WeY/j7+dAruVOhA0Cm4E3Vq1LE4etex98ReZiXMYtbWWexK34Wvly/D2gxjcNhgfDyLnvBRmQL9AktuVE6umfQXLIBevaCYq3uzcrLYdnQbj/R4pJIDc13RYdFM2zSNNfvXMKD1AKvDqVL2n9zPwJkD8fH0YeWdK2kV0Cp/3dnMs2w/up2E1ATzOJrAnG1zmL55en6bkHohl30rCKsfJvWiHHTi/AnmbZ/HzISZrDtophb3DenL3679GyPbjaSub12LI6xcrpf09+yBrVth6tRim/x27DcyczKlP9+J+oX2o4ZnDZbuXipJv4C0jDQGzjJdOmvGrymU8MHMfurRrAc9mv1RJkRrzcHTB/84ENgfS3YtIVfnAuDn5UeHhh0KHQgigiMIrFnxZ4rVQWZOJkt3L2VmwkyW7FpCZk4m7Ru05+X+L3Nbp9sqvN+8KnO9pJ93W8QiCqzlkUFc56vpXZN+of1YlrSMqRR/wHUnZy6eIXp2NAdOHeDbO78lIjjCodcppWhRtwUt6rbgpqtuyl9+IfsCiWmJhQ4Ei3YtYoZtRn6bpv5NCx0EIoIjaBPYBm9Pb6dvX1WjtWZ98npmxs9kXuI8jp8/TnCtYB6MepA7I++kS6Mu0p2LKyb9BQugc2cICSm2SXxqPL5evlwVeFXlxeUGosOieWT5I+w5vofW9VtbHY6lLmRfYPgXw7Gl2Fg4ZiG9WvQq93v6evnStXFXujb+o3yV1prUjNTLvhV89/t3ZOWaW1d4e3jTvkH7yw4GwbWCXSIJJh1PYmb8TGZtncXvJ37Hz8uPEe1GcGfEnfn3fRB/cK3fRmoq/PwzPPfcFZvZUmx0athJ/hicLCY8hkeWP8KypGVM6j7J6nAsk5Obw+1f386qvauYOWImN151Y4V9llKKRrUb0ah2Iwa2Hpi/PCsni53pOwsdCFbtXcXMhJn5bRrUbEBko0gigyOJCI4gMjiSdg3aVYkBzZKkn0vni+1fMDNhJhuSN6BQXB96Pc/0foab292Mfw1/q0Osslwr6y1caOboF3MVLpgzI1uKjZvb3VyJgbmHsPphhNcPd+ukr7VmwpIJfL3ja6YOmsodEXdYEoe3pzcdG3akY8OO3Nbptvzl6efS2Xp0K/Ep8SSkJhCfGs97v7zHxZyL5nUe3rRr0C7/IBAZHElko0ga1mpoyXYUdCH7Akt2LWFWwiyW7l5KVm4WnRp24rUbXmNsp7E0q9PM6hCrBddK+gsWQKtW0LFjsU0OnTlE+vl06c+vIDHhMfx38385n3UeP28/q8OpdE+vepqPfv2Ip697mik9p1gdzmUCawbSN6QvfUP65i/Lzs1mV/oucxBIiSc+NZ5Ve1cxK+GPklmNaje67EBQGWMFuTqXdQfWMTNhJl8mfsnJCydpXLsxk3tM5s6IO4lsJGXRS8t1kv7p0/D99/Dww0XeFjGP1NCvWNFh0byz8R1+2PdD/k1W3MVb69/i5bUv80C3B3ix34tWh+MwLw8v2jdoT/sG7RnTcUz+8mPnjhU6EMSnxvPOxnfyy2j7ePrQvkH7QgeCyOBIp8wg2nlsZ/58+n0n91HTuyY3t7uZOyPupH9of5muWg6uk/QzM2HKFLj11is2y0v6js6kEKXTJ6QPfl5+LEta5lZJ/7P4z3h85eOMaj+KaTHTXGKANKhmENeHXs/1odfnL8sbKyh4IFixZwWfxn+a36aJf5PLDgThgeEljqGlZaQxd9tcZibMZNPhTXgoD25odQMv9nuR4W2HSwl0J1FaW19voqCoqCgdFxdXYe8/at4o4lPj2f3w7gr7DHc3ZM4QEtMSSXo4ySWSX0kW71zMiC9G0DekL9/c9o1bXpF8NONooQNBQmoCiWmJZOdmA2bmUYcGHfIPBHldRb5evizetZiZCTNZnrSc7NxsIoMjuTPiTsZ2GksT/yYWb1n1oZTarLWOKqmd65zpO8iWYqNL4y5Wh+HSosOiWbJrCbuP73b5abE/7f+JW+ffStfGXYkdHeuWCR+gYa2GDGg9oNCFeZk5mexI25F/EIhPjWfxrsWFrivw8fQhMyeTpv5NeaznY9wRcYfcya6CuVXSP3PxDHtO7GF85/FWh+LSosPsN0zfvcylk358SjxD5gyhZd2WLL19qUwTvISPp4/p3ikw2Kq1JuVsSv5B4MiZI9x01U30Dekr/fSVxK2SvtTQrxyhAaG0C2rH0qSlPNLTNesb7Tm+h0GzBuFfw5+Vd64kqGaQ1SFVC0opGvs3prF/YwaFDbI6HLfkVnVcpfxC5YkOi+aHfT+QkZlhdShOd+TMEQbOGkh2bjYr71jp1nVcRPXjdkk/0C9QBocqQUx4DJk5mazet9rqUJzq5IWTDJ49mNSzqSy9fSntGrSzOiQhSsW9kn6qjc6NOrvFjBKr9WrRi9o+tVm6e6nVoTjNuaxzDJkzhB1pO4gdHUv3pt2tDkmIUnObpJ+dm83W1K3StVNJanjV4IZWN7B091Kq2rTgssjKyWL0/NGsO7CO2TfPlvLRotpym6S/89hOLuZclKRfiaLDotl/aj87ju2wOpRyydW53LvoXpbsWsL7N77PLR1usTokIcrMbZJ+fGo8IIO4lang1M3qSmvNEyufYGbCTF7s96LcA1hUe26T9G0pNmp41qBNYBurQ3Ebzes2p1PDTnyW8Bm70ndZHU6ZvLL2FaZumMrk7pN5+rqnrQ5HiHJzq6TfsWFHt7iDUFXyxDVPsPPYTtq+15YRX4xg3YF11aaP/8PNH/L3VX/n9k63M3XwVJkAIFyCWyT9vBr60rVT+e6KvIv9U/bzj97/YM3+NfT6uBfXzLiGrxK/Iic3x+rwivVV4ldM+GYC0WHRfDzsYzyUW/xXEW7ALf6Sj5w9Qtq5NCmnbJHg2sG80O8FDkw5wLSYaaRlpDHqy1G0ea8N036ZVuUu4Fq1dxW3fX0bPZv1ZP6t8+XboXApDiV9pdRgpdROpVSSUurJK7QbpZTSSqmoS5a3UEqdVUo9Ud6Ay0KuxK0aavnU4sGrH2TnpJ3Mv2U+QTWDmLRsEi3ebsE/V/2T1LOpVodI3OE4hs0dxlWBV7Fk7BJqete0OiQhnKrEpK+U8gSmAdFAe2CsUqp9Ee38gcnAxiLeZipg2RQOqaFftXh6eDKy/UjW37uetePX0rtlb/71079o+XZL/rzoz+xIs2aK585jO4meHU1QzSBW3LGCAL8AS+IQoiI5cqbfHUjSWv+utc4E5gLDimj3IvAacKHgQqXUcOB3YHs5Yy0zW4qNVgGtqOtb16oQRBGUUlzb4lpiR8fy26TfGN95PLO2zqL9++0ZMmcIP+77sdIGfZNPJzNg5gA8lAff3vmtlOoQLsuRpN8UOFjgebJ9WT6lVBegudZ6ySXLawF/A56/0gcope5XSsUppeLS0tIcCrw04lPjpWunirsq8Cr+c9N/ODDlAM/1eY4NyRvo+2lfun/UnS+2fZF/M46KkH4unYEzB3Lq4imW376csPphFfZZQljNkaRf1Dy1/NMvpZQHpvvm8SLaPQ9M1VqfvdIHaK0/0FpHaa2jGjRo4EBIjjubeZbd6bvpHCxJvzpoUKsBz/Z9lgNTDjD9xumcunCKMV+NIezdMN7Z8A5nM6/4p1RqZzPPcuPnN/L7id9ZNGaR3GBHuDxHkn4y0LzA82bA4QLP/YGOwA9KqX1AT2CRfTC3B/CaffkU4O9KqUlOiNthW1O3otFypl/N+Hn78UDUA/w26TcWjF5A87rNmbJiCs2nNuep757i8JnDJb9JCTJzMhk5bySbDm/ii1Ff0CekjxMiF6JqcyTpbwLClVKhSikfYAywKG+l1vqU1jpIax2itQ4BNgBDtdZxWuvrCix/G/g/rfV7zt+M4uUN4ha8e4+oPjyUB8PaDuOn8T+x/t713NDqBl77+TVC3g5h/MLxbD9atqGinNwc7oq9i5V7VvLRkI8Y1raoYSohXE+Jd87SWmfbz85XAJ7ADK31dqXUC0Cc1nrRld/BWrYUGwG+ATSv07zkxqJK69msJ1/e8iV7ju/h7Q1vM8M2g09snxAdFs0T1zxBv5B+Dl01q7Vm8rLJfLH9C14f8Drju7jH7TOzsrJITk7mwoULJTcWVZavry/NmjXD27ts14+oqnZJfFRUlI6Li3Pa+/X4qAe1vGux6u5VTntPUTWkn0tnetx0/v3Lv0nNSKVLoy48cc0T3NL+liteUPXs6md5Yc0L/PWav/LqgFcrMWJr7d27F39/fwIDA6WkRDWltSY9PZ0zZ84QGhpaaJ1SarPWOqqYl+Zz6Styc3JzpIa+CwusGcjTvZ9m35R9fDTkI85nn+f2r2+n9buteWv9W5y+ePqy1/x74795Yc0L3NP5Hl654RULorbOhQsXJOFXc0opAgMDy/VtzaWT/u7juzmffV6Svovz9fLl3q73sv3B7SwZu4TW9Vvz+MrHaT61OX/99q8kn04G4POtnzN5+WSGtx3Of4f81y2Tnztus6sp7z4ssU+/OpPyC+7FQ3lw41U3cuNVNxJ3OI4317/JW+vfYuqGqQy5agiLdy2mb0hf5oycg5eHS//pC1Eslz7Tt6XY8PH0oW1QW6tDEZUsqkkUc0bOIWlyEpOunsTKPSuJCI5g4ZiF+Hr5Wh2e2zp58iTvv/9+qV8XExPDyZMnr9jmmWee4bvvvitraG7DpQdyB88azNGMo2x5YItT3k9UXxmZGXh5eFHDq4bVoVhmx44dtGvXztIY9u3bx0033cS2bdsKLc/JycHT09OiqMrn0tgd3Zbs7Gy8vMr2jbOofenoQK5Lf8e1pdiIDo+2OgxRBdTyqWV1CFXLlClgszn3PTt3hrffvmKTJ598kj179tC5c2e8vb2pXbs2jRs3xmazkZiYyPDhwzl48CAXLlzgkUce4f777wcgJCSEuLg4zp49S3R0NL169eLnn3+madOmLFy4ED8/P8aNG8dNN93EqFGjCAkJ4e6772bx4sVkZWXx5Zdf0rZtW9LS0rjttttIT0/n6quvZvny5WzevJmgoKAi4501axbvvvsumZmZ9OjRg/fffx9PT09q167NY489xooVK3jzzTe54447uOeee1i5ciWTJk2ibdu2TJgwgXPnztG6dWtmzJhBQEAAffv25ZprrmHdunUMHTqUxx8vqpBBxXLZ7p2UsymkZqRK+QUhqpBXXnmF1q1bY7PZeP311/nll1/417/+RWJiIgAzZsxg8+bNxMXF8e6775Kenn7Ze+zevZuHHnqI7du3U69ePb766qsiPysoKIgtW7YwceJE3njjDQCef/55rr/+erZs2cKIESM4cOBAsbHu2LGDL774gnXr1mGz2fD09GT27NkAZGRk0LFjRzZu3EivXr0AM39+7dq1jBkzhrvuuotXX32VhIQEOnXqxPPP/1F+7OTJk/z444+WJHxw4TN9GcQV4gpKOCOvLN27dy803/zdd98lNjYWgIMHD7J7924CAwMLvSY0NJTOnc3/627durFv374i3/vmm2/Ob/P1118DsHbt2vz3Hzx4MAEBxZfP/v7779m8eTNXX301AOfPn6dhw4YAeHp6MnLkyELtR48eDcCpU6c4efIkffqYsh533303t9xyy2XtrOLySV/KLwhRddWq9Ue32w8//MB3333H+vXrqVmzJn379i1yPnqNGn+My3h6enL+/Pki3zuvnaenJ9nZpkpracYwtdbcfffdvPzyy5et8/X1vazfvuC2XImj7SqKy3bvxKfGE1IvhHq+9awORQhh5+/vz5kzZ4pcd+rUKQICAqhZsya//fYbGzZscPrn9+rVi3nz5gGwcuVKTpw4UWzb/v37M3/+fI4ePQrA8ePH2b9/f4mfUbduXQICAvjpp58AmDlzZv5Zf1Xg0mf60rUjRNUSGBjItddeS8eOHfHz8yM4ODh/3eDBg5k+fToRERG0adOGnj17Ov3zn332WcaOHcsXX3xBnz59aNy4Mf7+/kW2bd++PS+99BIDBw4kNzcXb29vpk2bRsuWLUv8nE8//TR/ILdVq1Z8/PHHzt6UMnPJKZsZmRn4v+zPs32e5dm+zzopMiGqt6owZdNqFy9exNPTEy8vL9avX8/EiROxOXsWUyWQKZuX2HZ0m9TQF0Jc5sCBA9x6663k5ubi4+PDhx9+aHVIlc4lk74M4gohihIeHs6vv/5aaFl6ejr9+/e/rO33339/2cwhV+CySb9ujbq0rFty35sQwr0FBgZWyy6esnLJ2Tu2VDOIKxUFhRCiMJdL+lJDXwghiudySX/PiT1kZGVI0hdCiCK4XNKX8gtCCFE8l0z63h7etG/Q3upQhBDlVLt2bQAOHz7MqFGjimzTt29fSrq25+233+bcuXP5zx2pz++qXDLpt2vQDh9PH6tDEUI4SZMmTZg/f36ZX39p0l+6dCn16lVuiZa8+j/FPXf0deXlclM2bSk2BrQeYHUYQlRpU5ZPye8KdZbOjTrz9uArV+/829/+RsuWLXnwwQcBeO6551BKsWbNGk6cOEFWVhYvvfQSw4YNK/S6gjdfOX/+POPHjycxMZF27doVKrg2ceJENm3axPnz5xk1ahTPP/887777LocPH6Zfv34EBQWxevXq/Pr8QUFBvPXWW8yYMQOA++67jylTprBv375i6/YXZc+ePTz00EOkpaVRs2ZNPvzwQ9q2bcu4ceOoX78+v/76K127dsXf35/Dhw+zb98+goKCmDFjBhMnTiQuLg4vLy/eeust+vXrxyeffMI333zDhQsXyMjIYNWqVeXZNYW4VNI/mnGUI2ePSA19IaqoMWPGMGXKlPykP2/ePJYvX86jjz5KnTp1OHbsGD179mTo0KHFTrn+z3/+Q82aNUlISCAhIYGuXbvmr/vXv/5F/fr1ycnJoX///iQkJDB58mTeeustVq9efdnNUjZv3szHH3/Mxo0b0VrTo0cP+vTpQ0BAALt372bOnDl8+OGH3HrrrXz11VfccccdRcZ0//33M336dMLDw9m4cSMPPvhgfqLetWsX3333HZ6enjz33HNs3ryZtWvX4ufnx5tvvgnA1q1b+e233xg4cCC7du0CYP369SQkJFC/fv3y/dIv4VJJPz4lHpBBXCFKUtIZeUXp0qULR48e5fDhw6SlpREQEEDjxo159NFHWbNmDR4eHhw6dIjU1FQaNWpU5HusWbOGyZMnAxAREUFERET+unnz5vHBBx+QnZ3NkSNHSExMLLT+UmvXrmXEiBH55Y5vvvlmfvrpJ4YOHepw3f6zZ8/y888/F6qZf/Hixfyfb7nllkJlmIcOHZr/jWHt2rU8/PDDALRt25aWLVvmJ/0BAwY4PeGDiyV9Kb8gRNU3atQo5s+fT0pKCmPGjGH27NmkpaWxefNmvL29CQkJKbKOfkFFfQvYu3cvb7zxBps2bSIgIIBx48aV+D5XKjjpaN3+3Nxc6tWrV+xVvZfWzy/4/EqfX1F1911qINeWaqNF3RbU93P+0VEI4Rxjxoxh7ty5zJ8/n1GjRnHq1CkaNmyIt7c3q1evLrFmfe/evfNvW7ht2zYSEhIAOH36NLVq1aJu3bqkpqaybNmy/NcUV8e/d+/eLFiwgHPnzpGRkUFsbCzXXXddqbanTp06hIaG8uWXXwImkcfHxzv02oLbsmvXLg4cOECbNm1K9fml5VpJX2roC1HldejQgTNnztC0aVMaN27M7bffTlxcHFFRUcyePZu2bdte8fUTJ07k7NmzRERE8Nprr9G9e3cAIiMj6dKlCx06dOCee+7h2muvzX/N/fffT3R0NP369Sv0Xl27dmXcuHF0796dHj16cN9999GlS5dSb9Ps2bP53//+R2RkJB06dGDhwoUOve7BBx8kJyeHTp06MXr0aD755JNC3zAqgkP19JVSg4F3AE/gI631K8W0GwV8CVyttY5TSg0AXgF8gEzgL1rrKw5Dl7We/vms89R+uTb/uO4fPN/v+ZJfIISbkXr6rqNC6+krpTyBacAAIBnYpJRapLVOvKSdPzAZ2Fhg8TFgiNb6sFKqI7ACaFrSZ5bF6YunGd1hNL1b9q6ItxdCCJfgyEBudyBJa/07gFJqLjAMSLyk3YvAa8ATeQu01gULV28HfJVSNbTWF3Gy4NrBfD7yc2e/rRBC5HvooYdYt25doWWPPPII48ePtyii0nMk6TcFDhZ4ngz0KNhAKdUFaK61XqKUeoKijQR+rYiEL4QQlWHatGlWh1BujiT9oq6QyB8IUEp5AFOBccW+gVIdgFeBgcWsvx+4H6BFixYOhCSEKAuttdxnopor733NHZm9kww0L/C8GXC4wHN/oCPwg1JqH9ATWKSUigJQSjUDYoG7tNZ7ivoArfUHWusorXVUgwYNSr8VQogS+fr6kp6eXu6kIayjtSY9PR1fX98yv4cjZ/qbgHClVChwCBgD3FYgiFNA/rXNSqkfgCfss3fqAd8AT2mtC3eECSEqVbNmzUhOTiYtLc3qUEQ5+Pr60qxZszK/vsSkr7XOVkpNwsy88QRmaK23K6VeAOK01ouu8PJJQBjwT6XUP+3LBmqtj5Y5YiFEmXh7exMaGmp1GMJiDs3Tr0xlnacvhBDuzNF5+i51Ra4QQogrk6QvhBBupMp17yil0oArV1yyRhDmCmNX5urbKNtX/bn6NpZn+1pqrUuc/ljlkn5VpZSKc6S/rDpz9W2U7av+XH0bK2P7pHtHCCHciCR9IYRwI5L0HfeB1QFUAlffRtm+6s/Vt7HCt0/69IUQwo3Imb4QQrgRSfp2SqnmSqnVSqkdSqntSqlH7MvrK6W+VUrttv8bYF+ulFLvKqWSlFIJSqmu1m6BY5RSnkqpX5VSS+zPQ5VSG+3b94VSyse+vIb9eZJ9fYiVcTtCKVVPKTVfKfWbfT/+yQX336P2v89tSqk5Sinf6rwPlVIzlFJHlVLbCiwr9T5TSt1tb79bKXW3FdtSnGK28XX732mCUirWXqcsb91T9m3cqZQaVGD5YPuyJKXUk2UOSGstD9PF1Rjoav/ZH9gFtMfcGOZJ+/IngVftP8cAyzClp3sCG63eBge38zHgc2CJ/fk8YIz95+nARPvPDwLT7T+PAb6wOnYHtu1T4D77zz5APVfaf5h7W+wF/Arsu3HVeR8CvYGuwLYCy0q1z4D6wO/2fwPsPwdYvW0lbONAwMv+86sFtrE9EA/UAEKBPZiaZ572n1vZ/7bjgfZlisfqX0hVfQALMbeI3Ak0ti9rDOy0//xfYGyB9vntquoDUxb7e+B6YIn9P8+xAn98fwJW2H9eAfzJ/rOXvZ2yehuusG117AlRXbLclfZf3g2N6tv3yRJgUHXfh0DIJQmxVPsMGAv8t8DyQu2qwuPSbbxk3Qhgtv3npzBVifPWrbDv0/z9WlS70jyke6cI9q/BXTD3+w3WWh8BsP/b0N6sqDuKVcj9f53obeCvQK79eSBwUmudbX9ecBvyt8++/pS9fVXVCkgDPrZ3X32klKqFC+0/rfUh4A3gAHAEs0824zr7ME9p91m125eXuAfzDQYqYRsl6V9CKVUb+AqYorU+faWmRSyrslOhlFI3AUe11psLLi6iqXZgXVXkhfkK/R+tdRcgA9M1UJzqtn3Y+7aHYb72NwFqAdFFNK2u+7AkxW1Ptd1OpdTTQDYwO29REc2cuo2S9AtQSnljEv5srfXX9sWpSqnG9vWNgbx7AZR0R7Gq5lpgqDJ3N5uL6eJ5G6inlMq7r0LBbcjfPvv6usDxygy4lJKBZK31Rvvz+ZiDgKvsP4AbgL1a6zStdRbwNXANrrMP85R2n1XHfYl9wPkm4HZt77OhErZRkr6dUkoB/wN2aK3fKrBqEZA3G+BuTF9/3vK77DMKegKn8r6SVkVa66e01s201iGYQb1VWuvbgdXAKHuzS7cvb7tH2dtX2bMnrXUKcFAp1ca+qD+QiIvsP7sDQE+lVE3732veNrrEPiygtPtsBTBQKRVg/zY00L6sylJKDQb+BgzVWp8rsGoRMMY+8yoUCAd+ocAdDO2zs8bY25ae1QMcVeUB9MJ8XUoAbPZHDKYP9Htgt/3f+vb2CpiGGVHfCkRZve7nUKkAAACvSURBVA2l2Na+/DF7p5X9jyoJ+BKoYV/ua3+eZF/fyuq4HdiuzkCcfR8uwMzkcKn9BzwP/AZsA2ZiZnlU230IzMGMT2RhzmbvLcs+w/SLJ9kf463eLge2MQnTR5+Xa6YXaP+0fRt3AtEFlsdgZhXuAZ4uazxyRa4QQrgR6d4RQgg3IklfCCHciCR9IYRwI5L0hRDCjUjSF0IINyJJXwgh3IgkfSGEcCOS9IUQwo38P5f6OLTWeCINAAAAAElFTkSuQmCC\n",
+      "text/plain": [
+       "<Figure size 432x288 with 1 Axes>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "600 is a good training size\n"
+     ]
+    }
+   ],
+   "source": [
+    "# try sizes 50, 100, 150, 200, ..., up to the largest multiple of 50 >= train_size\n",
+    "training_proportions = np.arange(0.10, 1.01, 0.10)\n",
+    "\n",
+    "# TODO\n",
+    "\n",
+    "# Calculate error for each entry in training_sizes\n",
+    "# for training and validation sets and populate\n",
+    "# error_train and error_val arrays. Each entry in these arrays\n",
+    "# should correspond to each entry in training_sizes.\n",
+    "\n",
+    "error_train = []\n",
+    "error_val = []\n",
+    "training_sizes = []\n",
+    "for proportion in training_proportions:\n",
+    "    \n",
+    "    size = len(train_indices)\n",
+    "    size_avail = np.int(np.ceil(size*proportion))\n",
+    "    training_sizes.append(size_avail)\n",
+    "    idx_avail = train_indices[:size_avail]\n",
+    "    \n",
+    "    kwargs_f = {'metric': 'Euclidean'}\n",
+    "    my_model.fit(features[idx_avail], labels[idx_avail], k = 10, distance_f=distance, **kwargs_f)\n",
+    "    \n",
+    "    val_pred_ratios = my_model.predict(features[val_indices])\n",
+    "    val_pred_labels = [1 if x >= threshold else 0 for x in val_pred_ratios]\n",
+    "    tp,tn, fp, fn = conf_matrix(labels[val_indices], val_pred_labels)\n",
+    "    val_accuracy = (tp+tn)/len(val_indices)\n",
+    "    val_error = 1 - val_accuracy\n",
+    "    error_val.append(val_error)\n",
+    "    \n",
+    "    train_pred_ratios = my_model.predict(features[train_indices])\n",
+    "    train_pred_labels = [1 if x >= threshold else 0 for x in train_pred_ratios]\n",
+    "    tp,tn, fp, fn = conf_matrix(labels[idx_avail], train_pred_labels)\n",
+    "    train_accuracy = (tp+tn)/size_avail\n",
+    "    train_error = 1 - train_accuracy\n",
+    "    error_train.append(train_error)\n",
+    "    \n",
+    "plt.plot(training_sizes, error_train, 'r', label = 'training_error')\n",
+    "plt.plot(training_sizes, error_val, 'g', label = 'validation_error')\n",
+    "plt.legend()\n",
+    "plt.show()\n",
+    "print('{} is a good training size'.format(600))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## TASK 5: Determining *k*"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Rubric:\n",
+    "* Increased accuracy with new *k* +5, +5\n",
+    "* Improved confusion matrix +5, +5"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We can use the validation set to come up with a *k* value that results in better performance in terms of accuracy.\n",
+    "\n",
+    "Below calculate the accuracies for different values of *k* using the validation set. Report a good *k* value and use it in the analyses that follow this section. Hint: Try values both smaller and larger than 10."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 157,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[0.49666666666666665, 0.55, 0.545, 0.505, 0.495, 0.465, 0.445]"
+      ]
+     },
+     "execution_count": 157,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "# TODO\n",
+    "k_accuracies = []\n",
+    "# Change values of k. \n",
+    "for k in [1, 5,  10, 50, 100, 150, 200]:\n",
+    "    # Calculate accuracies for the validation set.\n",
+    "    my_model.fit(features[train_indices], labels[train_indices], k=k, distance_f=distance, **kwargs_f)\n",
+    "    pred_ratios = my_model.predict(features[val_indices])\n",
+    "    pred_labels = [1 if x >= threshold else 0 for x in pred_ratios]\n",
+    "    tp,tn, fp, fn = conf_matrix(labels[val_indices], pred_labels)\n",
+    "    accuracy = (tp+tn)/len(val_indices)\n",
+    "    k_accuracies.append(accuracy)\n",
+    "k_accuracies\n",
+    "# Report a good k value."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## TASK 6: ROC curve analysis\n",
+    "* ROC curve has correct shape +20, +20"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### ROC curve and confusion matrix for the final model\n",
+    "ROC curves are a good way to visualize sensitivity vs. 1-specificity for varying cut off points. Now, implement, in \"model.ipynb\", a \"ROC\" function that predicts the labels of the test set examples using different *threshold* values in \"predict\" and plot the ROC curve. \"ROC\" takes a list containing different *threshold* parameter values to try and returns two arrays; one where each entry is the sensitivity at a given threshold and the other where entries are 1-specificities."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We can finally create the confusion matrix and plot the ROC curve for our optimal *k*-NN classifier. Use the *k* value you found above, if you completed TASK 5, else use *k* = 10. We'll plot the ROC curve for values between 0.1 and 1.0."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 135,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl4VOXd//H3l0DY9wRZw46AyBpZLAJWbXGpqKUVEBVFcO3++GgfvWofu7dPa7WCCoIoylKXKrW0dJFFxQBBdlAMYQuLSVjCGrJ9f3/MkF/EkAyQyWQyn9d15cqcM/eZ+Z5sn9z3Oec+5u6IiIgA1Ih0ASIiUnUoFEREpJhCQUREiikURESkmEJBRESKKRRERKSYQkFERIopFEREpJhCQUREitWMdAHnKiEhwTt06BDpMkREosrq1auz3T2xvHZRFwodOnQgNTU10mWIiEQVM9sZSjsNH4mISDGFgoiIFFMoiIhIMYWCiIgUUyiIiEixsIWCmc00s0wz23iW583MnjGzNDNbb2b9w1WLiIiEJpw9hVnAyDKevxboGvyYDDwXxlpERCQEYQsFd18GHCyjySjgFQ9IAZqYWatw1SMiEq2OnSrgD//8lHW7D4f9vSJ5TKENsLvEckZw3ZeY2WQzSzWz1KysrEopTkSkqjiRV8Az76WxYU9O2N8rkqFgpazz0hq6+zR3T3b35MTEcq/SFhGpNo7m5vP+1uxKe79ITnORAbQrsdwW2BuhWkREqoSiImfzviMs3ZrF0q1ZfLzzEAVFToPaNenesmHY3z+SobAAeMjM5gGDgBx33xfBekREIiL72Ck++CybpVuzeP+zLLKP5QFwSetGTBrWieHdEumf1JT4muEf3AlbKJjZXGAEkGBmGcATQC0Ad38eWAhcB6QBJ4C7wlWLiEhVkl9YxJpdh1kW7A2cPlbQrH48V3RNYHi3RIZ2TaBFwzqVXlvYQsHdx5bzvAMPhuv9RUSqkoxDJ1i2NZulWzNZnnaAo6cKiKth9E9qwo+u6cbwixPp1boxNWqUdri18kTd1NkiItEgN7+QFdsPsvTTLJZuzWRb1nEAWjeuww19WjGsayKXd0mgcd1aEa70ixQKIiIVwN3ZlnWMJZ9mseyzbFakH+BUQRHxNWswqGMzxg5MYsTFiXRObIBZZHsDZVEoiIicpyO5+SxPCxwgXrY1mz2HTwLQObE+tw1qz7BuCQzq2Jy68XERrjR0CgURkRAVFTkb9+YUHyD+eNdhCouchrVrcnmX5jx4ZReGdUugbdN6kS71vCkURETKkHX0FO9/lsWyrVm8/1k2B44HThe9tE1j7hveieHdWtAvqQm14qrHpNMKBRGREvILi/h456HAkNBnWWzccwSA5vXjGdYtsfh00YQGtSNcaXgoFEQk5u0+eIJln2Wx9NMslm87wLHg6aIDkpry8NcvZni3RHq2ahTx00Urg0JBRGLOybxCUrYfKD42kB48XbRNk7rc2Ld18HTR5jSqU7VOF60MCgURiQl7D59k4YZ9LN2axYrtB8krKKJ2zRoM7tSc8YPaM/ziRDol1K/Sp4tWBoWCiFRr6VnHeG7JNv6yZg8FRU7XFg24fXB7hndLZGDHZtSpFT2ni1YGhYKIVEub9x5hypI0Fm7YR+2aNRg/uD0Th3akXbPoPV20MigURKRa+XjXIaa8l8Z/PsmkYe2a3D+8M3cP7VhtzxaqaAoFEYl67s7ybQd49r00Pko/QNN6tfjRNd244/IOVW5uoapOoSAiUcvd+c+WTJ5dnMba3Ye5qFFtHr++B2MHJlG/tv68nQ991UQk6hQWOX/bsI+pi9P4ZP9R2jWryy9u7sXoAW2pXVMHji+EQkFEokZeQRFvr9nDc0u3sT37OF1aNOCpW/vwjd6tqVlNppmINIWCiFR5ufmFzFu5i2nL0tmbk0uvNo14fnx/vtazZUxcZVyZFAoiUmUdzc3n1ZRdzPggnexjeVzWoSm/vOVShndLjPmLzMJFoSAiVc6h43m89OF2Zi3fwZHcAoZ1S+ShK7swsGOzSJdW7SkURKTKyDySy/T303ltxS5O5BUy8pKWPHBlZ3q3bRLp0mKGQkFEIm73wRO8sGwbf07NoLDIubFPax4Y0ZmuFzWMdGkxR6EgIhGTlnmMqUvSeGftXuLM+OaAttw/vDNJzTUVRaQoFESk0m3ck8OUxWn8Y9N+6tSMY8LlHZh0RSdaNq4T6dJinkJBRCpN6o6DPLs4jSWfZtGwTk0eHNGFu77Sgeaal6jKUCiISFi5Ox+kZfPse2ms2H6QZvXjefjrF3P7kPYxeRObqk6hICJhUVTk/GvL50xdnMa6jBxaNqrDT27oydiBSdSN11QUVZVCQUQqVEFhEe+u38fUJWls/fwYSc3q8atbLuWW/m00L1EUUCiISIU4VVDIWx/v4fml29h54ATdLmrA02P6cv2lrTQvURRRKIjIBTmZV8jc4LxE+4/k0rttY164fQDX9LhI8xJFIYWCiJyXI7n5zP5oJzM+2M7B43kM6tiM332rN0O7JGheoigW1lAws5HA00Ac8KK7//qM55OAl4EmwTaPuvvCcNYkIhfm4PE8Zn6wnZc/2sHR3AJGXByYlyi5g+Ylqg7CFgpmFgdMAa4BMoBVZrbA3TeXaPY48Gd3f87MegILgQ7hqklEzt/+nFymLUtn7spd5BYUcm2vljwwogu92jSOdGlSgcLZUxgIpLl7OoCZzQNGASVDwYFGwceNgb1hrEdEzsPOA8d5fmk6b67OoNCdUX0D8xJ1aaF5iaqjcIZCG2B3ieUMYNAZbX4K/NPMvgPUB64OYz0icg62fn6UqYvTWLBuLzXjavDty9py77DOtGumeYmqs3CGQmlHmvyM5bHALHf/vZkNAWabWS93L/rCC5lNBiYDJCUlhaVYEQnYkJHDs4s/Y9Gmz6kXH8fEoR2ZdEUnWjTSvESxIJyhkAG0K7Hcli8PD00ERgK4+0dmVgdIADJLNnL3acA0gOTk5DODRUQqwIr0A0xZso1lW7NoVKcm372qK3dd3oGm9eMjXZpUonCGwiqgq5l1BPYAY4BxZ7TZBVwFzDKzHkAdICuMNYlICe7O0q1ZTFmcxqodh0hoEM8jI7szfnASDTUvUUwKWyi4e4GZPQQsInC66Ux332RmTwKp7r4A+BEw3cx+QGBoaYK7qycgEmZFRc6iTfuZsiSNjXuO0LpxHX76jZ7cepnmJYp1Fm1/g5OTkz01NTXSZYhEpYLCIhas28vUJdtIyzxGx4T63D+8Mzf1a0N8TU1FUZ2Z2Wp3Ty6vna5oFokBufmFvPlxBs8v3cbugyfp3rIhz4ztx/WXtiJOU1FICQoFkWrsRF4Bc1YE5iXKPHqKvu2a8MQNl3BVjxaaikJKpVAQqYZyTubz8vIdvPThdg6dyGdIp+Y8dWtfLu/cXGEgZVIoiFQj2cdOMeOD7cz+aCfHThVwVfcWPHBlFwa0bxrp0iRKKBREqoG9h08Wz0uUV1jEdZe24sERXejZulH5G4uUoFAQiWI7so/z3JJtvLUmA3e4uV8b7hvRmc6JDSJdmkQphYJIFPpk/xGmLt7Gu+sD8xKNHZjE5GGdaNtU8xLJhVEoiESRtbsP8+x7afx7y+fUj49j0rBOTBzakRYNNS+RVAyFgkgV5+58lH6AqYu38UFaNo3r1uL7V3dlwuUdaFJP8xJJxVIoiFRR7s7iTzN59r00Pt51mIQGtfnxtd25bXB7GtTWr66Eh36yRKqYwiLn7xv3MWXxNrbsO0KbJnX52ahL+FZyO+rU0rxEEl4KBZEqIr+wiLfX7OG5pdtIzzpOp4T6/G50b27q14ZacZqXSCqHQkEkwnLzC3k9dTfPL01nz+GT9GjViCnj+jOyV0vNSySVTqEgEiHHThXwWspOpr+/nexjp+if1ISf3XQJV16seYkkchQKIpXs8Ik8Zi3fwUsf7iDnZD5DuyTwwJV9GdJJ8xJJ5CkURCpJ5tFcZry/nVdTdnI8r5Cre1zEg1d2pl+S5iWSqkOhIBJmGYdOMG1ZOvNX7Sa/sIgberfmgSs7072l5iWSqkehIBIm27KO8dySbby9Zg9mcEu/ttw3ojMdE+pHujSRs1IoiFSwzXuPMGVJGgs37CM+rgbjB7dn8rBOtG5SN9KliZRLoSBSQVbvPMSUxWm890kmDWrX5L7hnZk4tCMJDWpHujSRkCkURC6Au7N82wGefS+Nj9IP0KReLX54TTfuHNKBxvVqRbo8kXOmUBA5D+7Ov7dkMmVxGmt3H6ZFw9o8fn0Pxg5Mor7mJZIopp9ekXNQWOT8bcM+pi5O45P9R2nbtC4/v6kXowe01bxEUi0oFERCkFdQxF/WZPDckm3sOHCCLi0a8Idv9+EbfVprXiKpVhQKImXIzS9k3spdTFuWzt6cXC5p3YjnbuvP1y9pSQ3NSyTVkEJBpBRHc/OZnbKTmR9sJ/tYHsntm/KLWy5lRLdETUUh1ZpCQaSEQ8fzeOnD7cxavoMjuQVc0TWBh67swqBOzSNdmkilUCiIAJ8fyeXF99N5bcUuTuQV8vVLLuKBEV3o065JpEsTqVQKBYlpuw+e4Pml23g9NYOCoiJu7NOaB67sQreLGka6NJGIUChITErLPMrUJdt4Z+1eahiMHtCW+4Z3pn1zzUsksS2kUDCzXu6+MdzFiITb7oMn+OXCLfxj035q16zBnUM6MGlYR1o11rxEIhB6T+F5M4sHZgFz3P1wKBuZ2UjgaSAOeNHdf11Km28DPwUcWOfu40KsSeScbM8+zrjpKRw5mc8DIzpz91c60lzzEol8QUih4O5DzawrcDeQamYrgZfc/V9n28bM4oApwDVABrDKzBa4++YSbboCPwa+4u6HzKzFBeyLyFltyzrG2GkpFBQ5r993OT1b614GIqUJ+VJMd/8MeBx4BBgOPGNmn5jZLWfZZCCQ5u7p7p4HzANGndFmEjDF3Q8F3yPzXHdApDyffX6UW19IociduZMGKxBEyhBSKJhZbzN7CtgCfBX4hrv3CD5+6iybtQF2l1jOCK4rqRvQzcw+NLOU4HBTae8/2cxSzSw1KysrlJJFAPhk/xHGTEvBDOZNHszFLXVWkUhZQu0pPAt8DPRx9wfd/WMAd99LoPdQmtIu+/QzlmsCXYERwFjgRTP70onh7j7N3ZPdPTkxMTHEkiXWbdqbw9hpKdSMM+ZNHkyXFgoEkfKEGgpvuftsdz95eoWZfQ/A3WefZZsMoF2J5bbA3lLavOPu+e6+HfiUQEiIXJANGTmMm76CurXimD95CJ0TG0S6JJGoEGoo3FHKugnlbLMK6GpmHYNnLo0BFpzR5m3gSgAzSyAwnJQeYk0ipVq7+zDjXkyhQe2azL93CB10T2SRkJV59pGZjQXGAR3NrOQf9IbAgbK2dfcCM3sIWETglNSZ7r7JzJ4EUt19QfC5r5nZZqAQeNjdy3xdkbKs3nmICTNX0rR+PHMmDaJt03qRLkkkqpR3SupyYB+QAPy+xPqjwPryXtzdFwILz1j3kxKPHfhh8EPkgqzacZAJM1eS2LA2cycP1gVpIuehzFBw953ATmBI5ZQjcn5S0g9w96xVtGxch7mTBnNRozqRLkkkKpV5TMHMPgh+PmpmR0p8HDWzI5VTokjZPkzLZsJLK2nTpC7zJisQRC5EeT2FocHPOpdPqqSlW7OY/EoqHRPq8+o9g0jQtBUiFyTUi9cGm1nDEssNzGxQ+MoSKd/iTzKZ9HIqnRIbMGfSYAWCSAUI9ZTU54BjJZZPBNeJRMS/Nn/O5NmpdGvZgLmTBtGsfnykSxKpFkINBQueKQSAuxehezFIhPxj4z7uf3U1PVs14rWJg2lST4EgUlFCDYV0M/uumdUKfnwPXWQmEfDu+r08OGcNvds2ZvY9g2hcr1akSxKpVkINhfuAy4E9BKamGARMDldRIqV5Z+0evjt3Df2TmvDKxEE0qqNAEKlood5PIZPANBUiEfHm6gwefmMdAzs2Y8adl1G/tkYvRcKhvGku/tvdf2tmf+LLM5zi7t8NW2UiQX9etZtH3lrP5Z2b8+Idl1E3Pi7SJYlUW+X9u3X6Lmmp4S5EpDSvrdjJY3/ZyLBuiUy7fQB1aikQRMKpvFC4FXgXaOLuT1dCPSLFXvloBz95ZxNf7d6Cqbf1VyCIVILyDjQPMLP2wN1m1tTMmpX8qIwCJTbN+GA7P3lnE9f0vIjnxisQRCpLeT2F54F/AJ2A1XzxbmoeXC9SoV5Yuo1f/f0TRl7SkmfG9iO+Zsi3EheRC1Te3EfPAM+Y2XPufn8l1SQxbMriNH636FOu792KP97al1pxCgSRylTe2UeN3P0I8Fhpw0XufjBslUnMefrfn/HUv7cyqm9rfv+tPtRUIIhUuvKGj+YANxAYOnI0fCRh4O784V9b+dN7aXyzf1t+O7o3cTWs/A1FpMKVN3x0Q/Bzx8opR2KNu/PbRZ/y3JJt3Jrcjl/dcik1FAgiERPq1Nk3m1njEstNzOym8JUlscDd+eXCLTy3ZBu3DUpSIIhUAaEO2j7h7jmnF9z9MPBEeEqSWODuPPnuZqa/v507h7Tn5zf1UiCIVAGhTiBTWnho8hk5L0VFzhMLNjE7ZScTh3bk8et7YKZAEKkKQu0ppJrZH8yss5l1MrOnCBx8FjknRUXOY29vYHbKTu4d3kmBIFLFhBoK3wHygPnAn4GTwIPhKkqqp8Ii55E31zN35W4eurILj47srkAQqWJCnTr7OPComTVw92PlbiByhsIi5+HX1/HWmj18/+qufO+qrgoEkSoo1LOPLjezzQRnTTWzPmY2NayVSbVRUFjE9+ev5a01e/jRNd34/tXdFAgiVVSow0dPAV8HDgC4+zpgWLiKkuojv7CI785bw1/X7eWRkd35zlVdI12SiJQh5DOI3H33Gf/dFVZ8OVKd5BUU8Z25H7No0+c8fn0P7rlCF8CLVHWhhsJuM7sccDOLB74LbAlfWRLtThUU8uBrH/PvLZk88Y2e3PUVXRQvEg1CHT66j8DZRm2APUBfdPaRnEVufiH3zl7Nv7dk8rObeikQRKJIqGcfZQO3hbkWqQZO5hUyeXYqH6Rl86tbLmXswKRIlyQi5yDUs486mdlfzSzLzDLN7B0zK3eA2MxGmtmnZpZmZo+W0W60mbmZJZ9L8VK1nMgr4O5Zq/ggLZvffrO3AkEkCoU6fDSHwEVrrYDWwOvA3LI2MLM4YApwLdATGGtmPUtp15DAMYoVoZctVc2xUwVMeGkVK7Yf4A/f7sO3kttFuiQROQ+hhoK5+2x3Lwh+vErgfgplGQikuXu6u+cB84BRpbT7GfBbIDfkqqVKOZqbz50zV7J65yH+OKYfN/drG+mSROQ8hRoKi83sUTPrYGbtzey/gb+ZWbPS7sgW1AbYXWI5I7iumJn1A9q5+7vnXLlUCTkn87l9xkrW7T7Mn8b248Y+rSNdkohcgFBPSb01+Hly8PPpCxbu5ux3YCvtktXi3oWZ1SBwUdyE8t7czCaffu+kJI1TVxWHT+Rxx8yVbNl3hCm39efrl7SMdEkicoHK7CmY2WVm1tLdOwbvvva/wEbgr8CA4PqzHXDOAEoOLLcF9pZYbgj0ApaY2Q5gMLCgtIPN7j7N3ZPdPTkxMTHUfZMwOnQ8j3HTV/DJvqM8P36AAkGkmihv+OgFArOjYmbDgF8BLwM5wLRytl0FdDWzjsEL3sYAC04/6e457p7g7h3cvQOQAtzo7qnntSdSaQ4cO8XY6SmkZR1j2h0DuKrHRZEuSUQqSHnDR3HufjD4+FZgmru/CbxpZmvL2tDdC8zsIWAREAfMdPdNZvYkkOruC8raXqqmrKOnuO3FFHYeOMGMO5O5oqt6biLVSbmhYGY13b0AuIr/f0whlG1x94XAwjPW/eQsbUeU93oSWZlHchk7PYW9h3N56a7LuLxzQqRLEpEKVt4f9rnAUjPLJnBjnfcBzKwLgSEkiRH7c3IZNz2F/UdymXXXZQzq1DzSJYlIGJQZCu7+CzP7D4GL1v7p7qfPHqpB4G5sEgP2HD7JuOkpHDiWx+yJAxnQ/mxnIYtItAtlCCillHVbw1OOVDW7D55g7PQUck7mM3viQPolNY10SSISRiHfT0Fiz84Dxxk3fQXHThXw2j2D6N22SaRLEpEwUyhIqbZnH2fstBROFRTy2j2D6NWmcaRLEpFKoFCQL0nLPMa46SkUFDlzJg2mR6tGkS5JRCqJQkG+YOvnRxk3PTBh7bzJg+l2UcMIVyQilSnUCfEkBmzZd4Sx01KoYQoEkVilUBAANu3NYdz0FGrF1WDe5MF0adEg0iWJSARo+EjYkJHD+BkrqB8fx9zJg2nfvH6kSxKRCFEoxLg1uw5xx8yVNK5bi7mTBtOuWb1IlyQiEaRQiGGrdx7kzpmraFY/nrmTB9OmSd1IlyQiEaZjCjFq5faD3DFjJYkNazP/XgWCiAQoFGLQ8m3Z3DlzJS0b12He5MG0aqxAEJEADR/FmA8+y+aeV1bRrmk95kwaTGLD2pEuSUSqEPUUYsiSTzO5++VVdGhen3mTFQgi8mXqKcSI/2z5nPtf/ZguLRrw6j2DaFY/PtIliUgVpJ5CDFi0aT/3vbqai1s2ZM4kBYKInJ16CtXcwg37+O7cNfRq05iX7x5I47q1Il2SiFRhCoVq7K/r9vL9+Wvp264Js+66jIZ1FAgiUjYNH1VTb6/Zw/fmrWFAUlNevnugAkFEQqKeQjX0xuoMHn5jHYM7NmfGhGTqxevbLCKh0V+Lamb+ql08+tYGhnZJYNrtydSNj4t0SSISRTR8VI28mrKTR97cwLCuiUy/Q4EgIudOPYVq4uXlO3hiwSau6t6CqeP7U7umAkFEzp1CoRp48f10fv63LXyt50U8O64/8TXVARSR86NQiHLPL93Gr//+Cddd2pKnx/SjVpwCQUTOn0Ihij373mf83z+38o0+rXnq232oqUAQkQukUIhC7s7T//mMP/77M27u14bfje6tQBCRCqFQiDLuzu//uZVnF6cxekBbfvPN3sTVsEiXJSLVhEIhirg7v/7HJ7ywNJ0xl7XjlzdfSg0FgohUoLCOOZjZSDP71MzSzOzRUp7/oZltNrP1ZvYfM2sfznqimbvz879t4YWl6YwfnKRAEJGwCFsomFkcMAW4FugJjDWznmc0WwMku3tv4A3gt+GqJ5q5O//7183M+GA7Ey7vwM9G9VIgiEhYhLOnMBBIc/d0d88D5gGjSjZw98XufiK4mAK0DWM9UamoyHn87Y3MWr6De4Z25Ilv9MRMgSAi4RHOUGgD7C6xnBFcdzYTgb+X9oSZTTazVDNLzcrKqsASq7aiIud//rKB11bs4r7hnXns+h4KBBEJq3CGQml/vbzUhmbjgWTgd6U97+7T3D3Z3ZMTExMrsMSqq7DIefiN9cxbtZvvfLULj4y8WIEgImEXzrOPMoB2JZbbAnvPbGRmVwOPAcPd/VQY64kaBYVF/Nfr63h77V5+cHU3vnd110iXJCIxIpw9hVVAVzPraGbxwBhgQckGZtYPeAG40d0zw1hL1MgvLOL789fy9tq9PPz1ixUIIlKpwtZTcPcCM3sIWATEATPdfZOZPQmkuvsCAsNFDYDXg0Mju9z9xnDVVNXlFRTxvXlr+PvG/fz42u7cO7xzpEsSkRgT1ovX3H0hsPCMdT8p8fjqcL5/NDlVUMhDc9bwr82f8/j1Pbjnik6RLklEYpCuaK4CcvMLeeC1j3nvk0z+98ZLuPPyDpEuSURilEIhwnLzC7l39mqWbs3iFzf34rZBuqhbRCJHoRBBJ/MKmfRKKh9uy+Y337yUWy9LinRJIhLjFAoRciKvgImzUknZfoDfje7D6AG6mFtEIk+hEAHHThVw90urSN15kD/e2pdRfcu60FtEpPIoFCrZ0dx8Jry0irW7D/PM2H7c0Lt1pEsSESmmUKhEOSfzuWPmSjbtyeHZsf249tJWkS5JROQLFAqV5PCJPG6fsZJP9h9h6m39+dolLSNdkojIlygUKsHB43mMf3EFaZnHeOH2AXy1+0WRLklEpFQKhTDLPnaK8S+uYHv2cabfmczwbrExy6uIRCeFQhhlHs3ltukr2H3oBDPuvIyhXRMiXZKISJkUCmHy+ZFcxk5PYd/hXF6aMJAhnZtHuiQRkXIpFMJgX85Jxk1fQeaRXF6+eyADOzaLdEkiIiFRKFSwjEMnGDd9BQeP5/HKxIEMaK9AEJHooVCoQBv35HDv7NUcyc3n1XsG0bddk0iXJCJyTsJ557WY4e7M/mgHt0xdTpE7c+4ZrEAQkaiknsIFOpKbz4/f3MDfNuzjyosT+cO3+9K0fnykyxIROS8KhQuwcU8OD875mIxDJ/nxtd2ZdEUnatSwSJclInLeFArnwd2ZnbKTn7+7heYN4vnzvYN1QFlEqgWFwjk6kpvPo2+uZ+GG/Xy1ewt+/60+Gi4SkWpDoXAONmQEhov2HNZwkYhUTwqFELg7r3y0k1/8bQsJGi4SkWpMoVCOI7n5PPLGev6+UcNFIlL9KRTKUHK46H+u6849QzVcJCLVm0KhFBouEpFYpVA4g4aLRCSWKRRKWJ9xmIfmrGGvhotEJEYpFAgMF728fAe/WLiFxAa1mX/vEAa0bxrpskREKl3Mh0LOycBw0T827eeq7i34Pw0XiUgMi+lQWJ9xmAfnfMy+w7k8dl0P7rmiI2YaLhKR2BXWqbPNbKSZfWpmaWb2aCnP1zaz+cHnV5hZh3DWc5q789KH2/nmc8spLHTm3zuEScM6KRBEJOaFradgZnHAFOAaIANYZWYL3H1ziWYTgUPu3sXMxgC/AW4NV00QGC767zfWsWjT51zdIzBc1KSehotERCC8w0cDgTR3Twcws3nAKKBkKIwCfhp8/AbwrJmZu3s4Clq3+zAPzQ0MFz1+fQ8mDtVwkYhISeEMhTbA7hLLGcCgs7Vx9wIzywGaA9kVXczrqbv5n79soEXDOvz5viH0T9LZRSIiZwpnKJT2L/iZPYBQ2mBmk4HJAElJSedVTKfE+lzV/SJ+/c1LNVwkInIW4QyFDKBdieW2wN6ztMkws5pAY+DgmS/k7tOAaQASfpYiAAAHQElEQVTJycnnNbQ0oH0zBtyuqSpERMoSzrOPVgFdzayjmcUDY4AFZ7RZANwZfDwaeC9cxxNERKR8YespBI8RPAQsAuKAme6+ycyeBFLdfQEwA5htZmkEeghjwlWPiIiUL6wXr7n7QmDhGet+UuJxLvCtcNYgIiKhC+vFayIiEl0UCiIiUkyhICIixRQKIiJSTKEgIiLFLNouCzCzLGDneW6eQBim0KjitM+xQfscGy5kn9u7e2J5jaIuFC6EmaW6e3Kk66hM2ufYoH2ODZWxzxo+EhGRYgoFEREpFmuhMC3SBUSA9jk2aJ9jQ9j3OaaOKYiISNliracgIiJlqJahYGYjzexTM0szs0dLeb62mc0PPr/CzDpUfpUVK4R9/qGZbTaz9Wb2HzNrH4k6K1J5+1yi3WgzczOL+jNVQtlnM/t28Hu9yczmVHaNFS2En+0kM1tsZmuCP9/XRaLOimJmM80s08w2nuV5M7Nngl+P9WbWv0ILcPdq9UFgmu5tQCcgHlgH9DyjzQPA88HHY4D5ka67Evb5SqBe8PH9sbDPwXYNgWVACpAc6bor4fvcFVgDNA0ut4h03ZWwz9OA+4OPewI7Il33Be7zMKA/sPEsz18H/J3AnSsHAysq8v2rY09hIJDm7unungfMA0ad0WYU8HLw8RvAVWZW2q1Bo0W5++zui939RHAxhcCd8KJZKN9ngJ8BvwVyK7O4MAllnycBU9z9EIC7Z1ZyjRUtlH12oFHwcWO+fIfHqOLuyyjlDpQljAJe8YAUoImZtaqo96+OodAG2F1iOSO4rtQ27l4A5ADNK6W68Ahln0uaSOA/jWhW7j6bWT+gnbu/W5mFhVEo3+duQDcz+9DMUsxsZKVVFx6h7PNPgfFmlkHg/i3fqZzSIuZcf9/PSVhvshMhpf3Hf+YpVqG0iSYh74+ZjQeSgeFhrSj8ytxnM6sBPAVMqKyCKkEo3+eaBIaQRhDoDb5vZr3c/XCYawuXUPZ5LDDL3X9vZkMI3M2xl7sXhb+8iAjr36/q2FPIANqVWG7Ll7uTxW3MrCaBLmdZ3bWqLpR9xsyuBh4DbnT3U5VUW7iUt88NgV7AEjPbQWDsdUGUH2wO9Wf7HXfPd/ftwKcEQiJahbLPE4E/A7j7R0AdAnMEVVch/b6fr+oYCquArmbW0cziCRxIXnBGmwXAncHHo4H3PHgEJ0qVu8/BoZQXCARCtI8zQzn77O457p7g7h3cvQOB4yg3untqZMqtEKH8bL9N4KQCzCyBwHBSeqVWWbFC2eddwFUAZtaDQChkVWqVlWsBcEfwLKTBQI6776uoF692w0fuXmBmDwGLCJy5MNPdN5nZk0Cquy8AZhDoYqYR6CGMiVzFFy7Eff4d0AB4PXhMfZe73xixoi9QiPtcrYS4z4uAr5nZZqAQeNjdD0Su6gsT4j7/CJhuZj8gMIwyIZr/yTOzuQSG/xKCx0meAGoBuPvzBI6bXAekASeAuyr0/aP4ayciIhWsOg4fiYjIeVIoiIhIMYWCiIgUUyiIiEgxhYKIiBRTKEjMMLPHgjOHrjeztWY2qAJfe3nwcwczG1difbKZPVPOtveZ2R3BxxPMrHVF1SVyrnRKqsSE4PQHfwBGuPup4IVd8e5eoZOnmdkI4L/c/Ybz3H5JcPtovshOoph6ChIrWgHZp6f3cPdsd99rZgPMbKmZrTazRadnmzSzJWb2GzNbaWZbzeyK4PpLguvWBnscXYPrjwXf59fAFcHnf2BmI8zsXTOrYWY7zKzJ6YKC8+FfZGY/NbP/MrPRBOalei24/fVm9pcS7a8xs7cq5aslMUuhILHin0C74B/4qWY23MxqAX8CRrv7AGAm8IsS29R094HA9wlcVQpwH/C0u/cl8Ac844z3eRR43937uvtTp1cGJ2d7B7gZIDh0tcPdPy/R5g0gFbgt+PoLgR5mlhhschfw0gV/JUTKoFCQmODux4ABwGQC8+LMB+4lMGnev8xsLfA4X7zPxOn/ylcDHYKPPwL+x8weAdq7+8lzKGM+cGvw8Zjgclk1OzCbwLTQTYAhRP+U51LFVbu5j0TOxt0LgSUEZk7dADwIbHL3IWfZ5PRMsoUEf1fcfY6ZrQCuBxaZ2T3u/l6IJXwEdAn+538T8PMQtnkJ+CuBmwS9Hrz/h0jYqKcgMcHMLj49/h/UF9gCJAYPQmNmtczsknJepxOQ7u7PEJitsvcZTY4SmLb7S4L/+f+FwAHvLWeZqO4L2wcPhO8l0IuZVVZtIhVBPQWJFQ2APwWHYQoIzDA5mcD9fZ8xs8YEfh/+CGwq43VuJTCckw/sB5484/n1QIGZrSPwR3zNGc/PJzAd9ISzvP4s4HkzOwkMCQ5PvQYkuvvm8ndT5MLolFSRKs7MngXWuPuMSNci1Z9CQaQKM7PVwHHgmmpwtzyJAgoFEREppgPNIiJSTKEgIiLFFAoiIlJMoSAiIsUUCiIiUkyhICIixf4fVoozmjfQ8YYAAAAASUVORK5CYII=\n",
+      "text/plain": [
+       "<Figure size 432x288 with 1 Axes>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "# TODO\n",
+    "# ROC curve\n",
+    "my_model.fit(features[train_indices], labels[train_indices], k=100, distance_f=distance, **kwargs_f)\n",
+    "pred_ratios = my_model.predict(features[test_indices])\n",
+    "\n",
+    "roc_sens, roc_spec_ = ROC(labels[test_indices], pred_ratios, np.arange(0.1, 1.0, 0.1))\n",
+    "plt.plot(roc_sens, roc_spec_)\n",
+    "plt.xlabel('Sensitivity')\n",
+    "plt.ylabel('Specificity')\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## TASK 7: Assess suitability of *k*-NN to your dataset"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Use this cell to write about your understanding of why *k*-NN performed well if it did or why not if it didn't."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.6.4"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/ProgrammingAssignment_1/model.ipynb b/ProgrammingAssignment_1/model.ipynb
index 11f5f312809f9c848e95f3af250be014be25b6bf..62a4e1553f3b72b3fa3db9b91ef742f7e2a9ed8a 100644
--- a/ProgrammingAssignment_1/model.ipynb
+++ b/ProgrammingAssignment_1/model.ipynb
@@ -18,16 +18,48 @@
    "source": [
     "# Supervised Learning Model Skeleton\n",
     "\n",
-    "We'll use this skeleton for implementing different supervised learning algorithms. Please complete \"preprocess\" and \"partition\" methods below."
+    "We'll use this skeleton for implementing different supervised learning algorithms."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 14,
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "class Model:\n",
+    "    # preprocess_f and partition_f expect functions\n",
+    "    # use kwargs to pass arguments to preprocessor_f and partition_f\n",
+    "    # kwargs is a dictionary and should contain t, v, feature_file, label_file\n",
+    "    # e.g. {'t': 0.3, 'v': 0.1, 'feature_file': 'some_file_name', 'label_file': 'some_file_name'}\n",
+    "    \n",
+    "    def __init__(self, preprocessor_f, partition_f, **kwargs):\n",
+    "        \n",
+    "        self.features, self.labels = preprocessor_f(kwargs['feature_file'], kwargs['label_file'])\n",
+    "        self.size = len(self.labels) # number of examples in dataset       \n",
+    "        self.feat_dim = self.features.shape[1] # number of features\n",
+    "        \n",
+    "        self.val_indices, self.test_indices = partition_f(self.size, kwargs['t'], kwargs['v'])\n",
+    "        self.val_size = len(self.val_indices)\n",
+    "        self.test_size = len(self.test_indices)\n",
+    "        \n",
+    "        self.train_indices = np.delete(np.arange(self.size), np.append(self.test_indices, self.val_indices), 0)\n",
+    "        self.train_size = len(self.train_indices)\n",
+    "        \n",
+    "    def fit(self):\n",
+    "        \n",
+    "        raise NotImplementedError\n",
+    "    \n",
+    "    def predict(self, test_points):\n",
+    "        raise NotImplementedError"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
    "metadata": {},
    "outputs": [],
    "source": [
-    "#TODO: GIVE!\n",
     "def preprocess(feature_file, label_file):\n",
     "    '''\n",
     "    Args:\n",
@@ -41,13 +73,10 @@
     "        labels: ndarray\n",
     "            nx1 labels\n",
     "    '''\n",
-    "    # You might find np.genfromtxt useful for reading in the file. Be careful with the file delimiter, \n",
-    "    # e.g. for comma-separated files use delimiter=',' argument.\n",
-    "    \n",
-    "    # TODO \n",
     "    \n",
-    "    raise NotImplementedError\n",
-    "\n",
+    "    # read in features and labels\n",
+    "    features = np.genfromtxt(feature_file)\n",
+    "    labels = np.genfromtxt(label_file)\n",
     "    \n",
     "    return features, labels"
    ]
@@ -58,7 +87,6 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "#TODO: GIVE!\n",
     "def partition(size, t, v = 0):\n",
     "    '''\n",
     "    Args:\n",
@@ -74,16 +102,19 @@
     "        val_indices: ndarray\n",
     "            1D array containing validation set indices\n",
     "    '''\n",
+    " \n",
+    "    # number of test and validation examples\n",
+    "    t_size = np.int(np.ceil(size*t))\n",
+    "    v_size = np.int(np.ceil(size*v))\n",
+    "\n",
+    "    # shuffle the indices\n",
+    "    permuted = np.random.permutation(size)\n",
     "    \n",
-    "    # np.random.permutation might come in handy. Do not sample with replacement!\n",
-    "    # Be sure not to use the same indices in test and validation sets!\n",
-    "    \n",
-    "    # use the first np.ceil(size*t) for test, \n",
-    "    # the following np.ceil(size*v) for validation set.\n",
-    "    \n",
-    "    # TODO\n",
+    "    # spare the first t_size for test\n",
+    "    test_indices =  permuted[:t_size]\n",
+    "    # and the next v_size for validation\n",
+    "    val_indices = permuted[t_size+1:t_size+v_size+1]\n",
     "    \n",
-    "    raise NotImplementedError\n",
     "    \n",
     "    return test_indices, val_indices"
    ]
@@ -92,51 +123,14 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "In cases, where data is not abundantly available, we resort to getting an error estimate from average of error on different splits of error. In this case, every fold of data is used for testing and for training in turns, i.e. assuming we split our data into 3 folds, we'd\n",
-    "* train our model on fold-1+fold-2 and test on fold-3\n",
-    "* train our model on fold-1+fold-3 and test on fold-2\n",
-    "* train our model on fold-2+fold-3 and test on fold-1.\n",
-    "\n",
-    "We'd use the average of the error we obtained in three runs as our error estimate. Implement function \"kfold\" below.\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# TODO: Programming Assignment 2\n",
-    "\n",
-    "def kfold(indices, k):\n",
-    "\n",
-    "    '''\n",
-    "    Args:\n",
-    "        indices: ndarray\n",
-    "            1D array with integer entries containing indices\n",
-    "        k: int \n",
-    "            Number of desired splits in data.(Assume test set is already separated.)\n",
-    "        Returns:\n",
-    "        fold_dict: dict\n",
-    "            A dictionary with integer keys corresponding to folds. Values are (training_indices, val_indices).\n",
-    "        \n",
-    "        val_indices: ndarray\n",
-    "            1/k of training indices randomly chosen and separates them as validation partition.\n",
-    "        train_indices: ndarray\n",
-    "            Remaining 1-(1/k) of the indices.\n",
-    "            \n",
-    "            e.g. fold_dict = {0: (train_0_indices, val_0_indices), \n",
-    "            1: (train_0_indices, val_0_indices), 2: (train_0_indices, val_0_indices)} for k = 3\n",
-    "    '''\n",
-    "    \n",
-    "    return fold_dict"
+    "## TASK 1: Implement `distance` function"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Choice of distance metric plays an important role in the performance of *k*-NN. Let's start with implementing a distance method in the \"distance\" function below. It should take two data points and the name of the metric and return a scalar value."
+    "\"distance\" function will be used in calculating cost of *k*-NN. It should take two data points and the name of the metric and return a scalar value."
    ]
   },
   {
@@ -154,13 +148,13 @@
     "        y: ndarray\n",
     "            1D array containing coordinates for a point\n",
     "        metric: str\n",
-    "            Euclidean, Hamming \n",
+    "            Euclidean, Manhattan \n",
     "    Returns:\n",
-    "        \n",
+    "        dist: float\n",
     "    '''\n",
     "    if metric == 'Euclidean':\n",
     "        raise NotImplementedError\n",
-    "    elif metric == 'Hammming':\n",
+    "    elif metric == 'Manhattan':\n",
     "        raise NotImplementedError\n",
     "    else:\n",
     "        raise ValueError('{} is not a valid metric.'.format(metric))\n",
@@ -171,48 +165,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "We will extend \"Model\" class while implementing supervised learning algorithms."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "class Model:\n",
-    "    # preprocess_f and partition_f expect functions\n",
-    "    # use kwargs to pass arguments to preprocessor_f and partition_f\n",
-    "    # kwargs is a dictionary and should contain t, v, feature_file, label_file\n",
-    "    # e.g. {'t': 0.3, 'v': 0.1, 'feature_file': 'some_file_name', 'label_file': 'some_file_name'}\n",
-    "    \n",
-    "    def __init__(self, preprocessor_f, partition_f, **kwargs):\n",
-    "        \n",
-    "        self.features, self.labels = preprocessor_f(kwargs['feature_file'], kwargs['label_file'])\n",
-    "        self.size = len(self.labels) # number of examples in dataset       \n",
-    "        self.feat_dim = self.features.shape[1] # number of features\n",
-    "        \n",
-    "        self.val_indices, self.test_indices = partition_f(self.size, kwargs['t'], kwargs['v'])\n",
-    "        self.val_size = len(self.val_indices)\n",
-    "        self.test_size = len(self.test_indices)\n",
-    "        \n",
-    "        self.train_indices = np.delete(np.arange(self.size), np.append(self.test_indices, self.val_indices), 0)\n",
-    "        self.train_size = len(self.train_indices)\n",
-    "        \n",
-    "    def fit(self):\n",
-    "        \n",
-    "        raise NotImplementedError\n",
-    "    \n",
-    "    def predict(self, test_points):\n",
-    "        raise NotImplementedError"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## General supervised learning performance related functions \n",
-    "### (To be implemented later when it is indicated in other notebooks)"
+    "## General supervised learning performance related functions "
    ]
   },
   {
diff --git a/ProgrammingAssignment_1/model_solution.ipynb b/ProgrammingAssignment_1/model_solution.ipynb
index 8e1f78b9b8ba4c77fe134126e7c004459a988f82..a4582f3c30f305943118e6756d059094d013902c 100644
--- a/ProgrammingAssignment_1/model_solution.ipynb
+++ b/ProgrammingAssignment_1/model_solution.ipynb
@@ -18,12 +18,28 @@
    "source": [
     "# Supervised Learning Model Skeleton\n",
     "\n",
-    "We'll use this skeleton for implementing different supervised learning algorithms. Please complete \"preprocess\" and \"partition\" methods below."
+    "We'll use this skeleton for implementing different supervised learning algorithms."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 14,
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "class Model:\n",
+    "        \n",
+    "    def fit(self):\n",
+    "        \n",
+    "        raise NotImplementedError\n",
+    "    \n",
+    "    def predict(self, test_points):\n",
+    "        raise NotImplementedError"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -40,13 +56,10 @@
     "        labels: ndarray\n",
     "            nx1 labels\n",
     "    '''\n",
-    "    # You might find np.genfromtxt useful for reading in the file. Be careful with the file delimiter, \n",
-    "    # e.g. for comma-separated files use delimiter=',' argument.\n",
-    "    \n",
-    "    # TODO \n",
     "    \n",
-    "    raise NotImplementedError\n",
-    "\n",
+    "    # read in features and labels\n",
+    "    features = np.genfromtxt(feature_file)\n",
+    "    labels = np.genfromtxt(label_file)\n",
     "    \n",
     "    return features, labels"
    ]
@@ -57,7 +70,6 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "#TODO: GIVE!\n",
     "def partition(size, t, v = 0):\n",
     "    '''\n",
     "    Args:\n",
@@ -74,73 +86,39 @@
     "            1D array containing validation set indices\n",
     "    '''\n",
     "    \n",
-    "    # np.random.permutation might come in handy. Do not sample with replacement!\n",
-    "    # Be sure not to use the same indices in test and validation sets!\n",
-    "    \n",
-    "    # use the first np.ceil(size*t) for test, \n",
-    "    # the following np.ceil(size*v) for validation set.\n",
-    "    \n",
-    "    # TODO\n",
+    "    # number of test and validation examples\n",
+    "    t_size = np.int(np.ceil(size*t))\n",
+    "    v_size = np.int(np.ceil(size*v))\n",
+    "\n",
+    "    # shuffle the indices\n",
+    "    permuted = np.random.permutation(size)\n",
     "    \n",
-    "    raise NotImplementedError\n",
+    "    # spare the first t_size for test\n",
+    "    test_indices =  permuted[:t_size]\n",
+    "    # and the next v_size for validation\n",
+    "    val_indices = permuted[t_size+1:t_size+v_size+1]\n",
+    "    train_indices = np.delete(np.arange(size), np.append(test_indices, val_indices), 0)\n",
     "    \n",
-    "    return test_indices, val_indices"
+    "    return test_indices, val_indices, train_indices"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "In cases, where data is not abundantly available, we resort to getting an error estimate from average of error on different splits of error. In this case, every fold of data is used for testing and for training in turns, i.e. assuming we split our data into 3 folds, we'd\n",
-    "* train our model on fold-1+fold-2 and test on fold-3\n",
-    "* train our model on fold-1+fold-3 and test on fold-2\n",
-    "* train our model on fold-2+fold-3 and test on fold-1.\n",
-    "\n",
-    "We'd use the average of the error we obtained in three runs as our error estimate. Implement function \"kfold\" below.\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# TODO: Programming Assignment 2\n",
-    "\n",
-    "def kfold(indices, k):\n",
-    "\n",
-    "    '''\n",
-    "    Args:\n",
-    "        indices: ndarray\n",
-    "            1D array with integer entries containing indices\n",
-    "        k: int \n",
-    "            Number of desired splits in data.(Assume test set is already separated.)\n",
-    "        Returns:\n",
-    "        fold_dict: dict\n",
-    "            A dictionary with integer keys corresponding to folds. Values are (training_indices, val_indices).\n",
-    "        \n",
-    "        val_indices: ndarray\n",
-    "            1/k of training indices randomly chosen and separates them as validation partition.\n",
-    "        train_indices: ndarray\n",
-    "            Remaining 1-(1/k) of the indices.\n",
-    "            \n",
-    "            e.g. fold_dict = {0: (train_0_indices, val_0_indices), \n",
-    "            1: (train_0_indices, val_0_indices), 2: (train_0_indices, val_0_indices)} for k = 3\n",
-    "    '''\n",
-    "    \n",
-    "    return fold_dict"
+    "## TASK 1: Implement `distance` function"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Choice of distance metric plays an important role in the performance of *k*-NN. Let's start with implementing a distance method in the \"distance\" function below. It should take two data points and the name of the metric and return a scalar value."
+    "\"distance\" function will be used in calculating cost of *k*-NN. It should take two data points and the name of the metric and return a scalar value."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
+   "execution_count": 8,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -155,12 +133,12 @@
     "        metric: str\n",
     "            Euclidean, Hamming \n",
     "    Returns:\n",
-    "        \n",
+    "        dist: float\n",
     "    '''\n",
     "    if metric == 'Euclidean':\n",
-    "        raise NotImplementedError\n",
-    "    elif metric == 'Hammming':\n",
-    "        raise NotImplementedError\n",
+    "        dist = np.sqrt(np.sum(np.square((x-y))))\n",
+    "    elif metric == 'Manhattan':\n",
+    "        dist = np.sum(abs(x-y))\n",
     "    else:\n",
     "        raise ValueError('{} is not a valid metric.'.format(metric))\n",
     "    return dist # scalar distance btw x and y"
@@ -170,48 +148,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "We will extend \"Model\" class while implementing supervised learning algorithms."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "class Model:\n",
-    "    # preprocess_f and partition_f expect functions\n",
-    "    # use kwargs to pass arguments to preprocessor_f and partition_f\n",
-    "    # kwargs is a dictionary and should contain t, v, feature_file, label_file\n",
-    "    # e.g. {'t': 0.3, 'v': 0.1, 'feature_file': 'some_file_name', 'label_file': 'some_file_name'}\n",
-    "    \n",
-    "    def __init__(self, preprocessor_f, partition_f, **kwargs):\n",
-    "        \n",
-    "        self.features, self.labels = preprocessor_f(kwargs['feature_file'], kwargs['label_file'])\n",
-    "        self.size = len(self.labels) # number of examples in dataset       \n",
-    "        self.feat_dim = self.features.shape[1] # number of features\n",
-    "        \n",
-    "        self.val_indices, self.test_indices = partition_f(self.size, kwargs['t'], kwargs['v'])\n",
-    "        self.val_size = len(self.val_indices)\n",
-    "        self.test_size = len(self.test_indices)\n",
-    "        \n",
-    "        self.train_indices = np.delete(np.arange(self.size), np.append(self.test_indices, self.val_indices), 0)\n",
-    "        self.train_size = len(self.train_indices)\n",
-    "        \n",
-    "    def fit(self):\n",
-    "        \n",
-    "        raise NotImplementedError\n",
-    "    \n",
-    "    def predict(self, test_points):\n",
-    "        raise NotImplementedError"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## General supervised learning performance related functions \n",
-    "### (To be implemented later when it is indicated in other notebooks)"
+    "## General supervised learning performance related functions "
    ]
   },
   {
@@ -223,7 +160,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 13,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -239,12 +176,28 @@
     "    Returns:\n",
     "        ndarray\n",
     "    '''\n",
-    "    raise NotImplementedError\n",
-    "    \n",
+    "        \n",
     "    tp = tn = fp = fn = 0\n",
     "    # calculate true positives (tp), true negatives(tn)\n",
     "    # false positives (fp) and false negatives (fn)\n",
     "    \n",
+    "    size = len(true)\n",
+    "    for i in range(size):\n",
+    "        if true[i]==1:\n",
+    "            if pred[i] > 0:\n",
+    "               \n",
+    "                tp += 1\n",
+    "            else:\n",
+    "               \n",
+    "                fn += 1\n",
+    "        else:\n",
+    "            if pred[i] == 0:\n",
+    "               \n",
+    "                tn += 1    \n",
+    "            else:\n",
+    "               \n",
+    "                fp += 1                            \n",
+    "    \n",
     "    # returns the confusion matrix as numpy.ndarray\n",
     "    return np.array([tp,tn, fp, fn])"
    ]
@@ -258,18 +211,19 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 11,
    "metadata": {},
    "outputs": [],
    "source": [
     "# TODO: Programming Assignment 1\n",
     "\n",
-    "def ROC(model, indices, value_list):\n",
+    "def ROC(true_labels, preds, value_list):\n",
     "    '''\n",
     "    Args:\n",
-    "        model: a fitted supervised learning model\n",
-    "        indices: ndarray\n",
-    "            1D array containing indices\n",
+    "        true_labels: ndarray\n",
+    "            1D array containing true labels\n",
+    "        preds: ndarray\n",
+    "            1D array containing thresholded value (e.g. proportion of positive neighbors in kNN)\n",
     "        value_list: ndarray\n",
     "            1D array containing different threshold values\n",
     "    Returns:\n",
@@ -283,10 +237,17 @@
     "    # use conf_matrix to calculate tp, tn, fp, fn\n",
     "    # calculate sensitivity, 1-specificity\n",
     "    # return two arrays\n",
-    "    \n",
-    "    raise NotImplementedError\n",
-    "    \n",
-    "    return sens, spec_"
+    "    sens = []\n",
+    "    spec_ = []\n",
+    "    for threshold in value_list:\n",
+    "        pred_labels = [1 if x >= threshold else 0 for x in pred_ratios]\n",
+    "        tp,tn, fp, fn = conf_matrix(true_labels, pred_labels)        \n",
+    "        se = tp/(tp+fn)\n",
+    "        sens.append(se)\n",
+    "        spec = tn/(tn+fp)\n",
+    "        spec_.append(1 - spec)\n",
+    "        \n",
+    "    return np.array(sens), np.array(spec_)"
    ]
   }
  ],