ProgrammingAssignment_1/ProgrammingAssignment1.ipynb

 %% Cell type:markdown id: tags:
 
 # *k*-Nearest Neighbor
 
 We'll implement *k*-Nearest Neighbor (*k*-NN) algorithm for this assignment. We will use the **madelon** dataset as in Programming Assignment 0.
 
 A skeleton of a general supervised learning model is provided in "model.ipynb". The functions that will be implemented there will be indicated in this notebook.
 
 ### Assignment Goals:
 In this assignment, we will:
 * implement 'Euclidean' and 'Manhattan' distance metrics
 * use the validation dataset to find a good value for *k*
 * evaluate our model with respect to performance measures:
-    * accuracy, generalization error and ROC curve
-* try to assess if *k*-NN is suitable for the dataset you used
+    * accuracy, generalization error
+    * confusion matrix
+    *  Receiver Operating Characteristic (ROC) curve
+* try to assess if *k*-NN is suitable for the dataset
 
 ## Note:
 
 You are not required to follow this exact template. You can change what parameters your functions take or partition the tasks across functions differently. However, make sure there are outputs and implementation for items listed in the rubric for each task. Also, indicate in code with comments which task you are attempting.
 
 %% Cell type:markdown id: tags:
 
 # GRADING
 
 You will be graded on parts that are marked with **TODO** comments. Read the comments in the code to make sure you don't miss any.
 
 ### Mandatory for 478 & 878:
 
 |   | Tasks                      | 478 | 878 |
 |---|----------------------------|-----|-----|
 | 1 | Implement `distance`       |  15 |  15 |
 | 2 | Implement `k-NN` methods   |  35 |  30 |
 | 3 | Model evaluation           |  25 |  20 |
 | 5 | ROC curve analysis         |  25 |  25 |
 
 ### Mandatory for 878, bonus for 478
 
 |   | Tasks          | 478 | 878 |
 |---|----------------|-----|-----|
 | 4 | Optimizing *k* | 10  | 10  |
 
 ### Bonus for 478/878
 
 |   | Tasks          | 478 | 878 |
 |---|----------------|-----|-----|
 | 6 | Assess suitability of *k*-NN | 10  | 10  |
 
 Points are broken down further below in Rubric sections. The **first** score is for 478, the **second** is for 878 students. There are a total of 100 points in this assignment and extra 20 bonus points for 478 students and 10 bonus points for 878 students.
 
 %% Cell type:markdown id: tags:
 
 # YOUR GRADE
 
 ### Group Members:
 
 |   | Tasks                      | Points |
 |---|----------------------------|-----|
 | 1 | Implement `distance`       |     |
 | 2 | Implement `k-NN` methods   |     |
 | 3 | Model evaluation           |     |
 | 4 | Optimizing *k*             |     |
 | 5 | ROC curve analysis         |     |
 | 6 | Assess suitability of *k*-NN|    |
 
 
 
 %% Cell type:markdown id: tags:
 
 You can use numpy for array operations and matplotlib for plotting for this assignment. Please do not add other libraries.
 
 %% Cell type:code id: tags:
 
 ``` python
 import numpy as np
 import matplotlib.pyplot as plt
 ```
 
 %% Cell type:markdown id: tags:
 
 Following code makes the Model class and relevant functions available from model.ipynb.
 
 %% Cell type:code id: tags:
 
 ``` python
 %run 'model.ipynb'
 ```
 
 %% Cell type:markdown id: tags:
 
 ## TASK 1: Implement `distance` function
 
 %% Cell type:markdown id: tags:
 
 Choice of distance metric plays an important role in the performance of *k*-NN. Let's start with implementing a distance method  in the "distance" function in **model.ipynb**. It should take two data points and the name of the metric and return a scalar value.
 
 %% Cell type:markdown id: tags:
 
 ### Rubric:
 * Euclidean +7.5, +7.5
 * Manhattan +7.5, +7.5
 
 %% Cell type:markdown id: tags:
 
 ### Test `distance`
 
 %% Cell type:code id: tags:
 
 ``` python
 x = np.array(range(100))
 y = np.array(range(100, 200))
 dist_euclidean = distance(x, y, 'Euclidean')
 dist_manhattan = distance(x, y, 'Manhattan')
 print('Euclidean distance: {}, Manhattan distance: {}'.format(dist_euclidean, dist_manhattan))
 ```
 
 %% Cell type:markdown id: tags:
 
 ## TASK 2: Implement *k*-NN Class Methods
 
 %% Cell type:markdown id: tags:
 
 We can start implementing our *k*-NN classifier. *k*-NN class inherits Model class. Use the "distance" function you defined above. "fit" method takes *k* as an argument. "predict" takes as input an *mxd* array containing *d*-dimensional *m* feature vectors for examples and for each input point outputs the predicted class and the ratio of positive examples in *k* nearest neighbors.
 
 %% Cell type:markdown id: tags:
 
 ### Rubric:
 * correct implementation of fit method +10, +10
 * correct implementation of predict method +25, +20
 
 %% Cell type:code id: tags:
 
 ``` python
 class kNN(Model):
     '''
     Inherits Model class. Implements the k-NN algorithm for classification.
     '''
 
-    def fit(self, training_features, training_labels, classes, k, distance_f,**kwargs):
+    def fit(self, training_features, training_labels, k, distance_f,**kwargs):
         '''
         Fit the model. This is pretty straightforward for k-NN.
         Args:
             training_features: ndarray
             training_labels: ndarray
-            classes: ndarray
-                1D array containing unique classes in the dataset
             k: int
             distance_f: function
             kwargs: dict
                 Contains keyword arguments that will be passed to distance_f
         '''
         # TODO
-        # set self.train_features, self.train_labels, self.classes, self.k, self.distance_f, self.distance_metric
+        # set self.train_features, self.train_labels, self.k, self.distance_f, self.distance_metric
 
         raise NotImplementedError
 
         return
 
 
     def predict(self, test_features):
         '''
         Args:
             test_features: ndarray
                 mxd array containing features for the points to be predicted
         Returns:
             preds: ndarray
-                mx2 array containing predicted class and proportion for each test point
+                mx1 array containing proportion of positive class among k nearest neighbors of each test point
         '''
         raise NotImplementedError
 
         # TODO
 
-        # for each point in test_features
+        # for each point
         # use your implementation of distance function
         #  distance_f(..., distance_metric)
         # to find the labels of k-nearest neighbors.
 
-        # you'll need proportion of the dominant class
+        # You'll need proportion of positive examples
         # in k nearest neighbors
 
         return preds
 
 ```
 
 %% Cell type:markdown id: tags:
 
 ## TASK 3: Build and Evaluate the Model
 
 %% Cell type:markdown id: tags:
 
 ### Rubric:
 * Reasonable accuracy values +10, +5
 * Reasonable confidence intervals on the error estimate +10, +10
 * Reasonable confusion matrix +5, +5
 
 %% Cell type:markdown id: tags:
 
 Preprocess the data files and partition the data.
 
 %% Cell type:code id: tags:
 
 ``` python
 # initialize the model
 my_model = kNN()
 # obtain features and labels from files
 features, labels = preprocess(feature_file=..., label_file=...)
 # get class names (unique entries in labels)
 classes = np.unique(labels)
 # partition the data set
 val_indices, test_indices, train_indices = partition(size=..., t = 0.3, v = 0.1)
 ```
 
 %% Cell type:markdown id: tags:
 
 Assign a value to *k* and fit the *k*-NN model.
 
 %% Cell type:code id: tags:
 
 ``` python
 # pass the training features and labels to the fit method
 kwargs_f = {'metric': 'Euclidean'}
-my_model.fit(training_features=..., training_labels-..., classes, k=10, distance_f=..., **kwargs_f)
+my_model.fit(training_features=..., training_labels-..., k=10, distance_f=..., **kwargs_f)
 ```
 
 %% Cell type:markdown id: tags:
 
 ### Computing the confusion matrix for *k* = 10
 Now that we have the true labels and the predicted ones from our model, we can build a confusion matrix and see how accurate our model is. Implement the "conf_matrix" function (in model.ipynb) that takes as input an array of true labels (*true*) and an array of predicted labels (*pred*). It should output a numpy.ndarray. You do not need to change the value of the threshold parameter yet.
 
 %% Cell type:code id: tags:
 
 ``` python
 # TODO
 
 # get model predictions
 pred_ratios = my_model.predict(features[test_indices])
 
-# For now, we will consider a data point as predicted in a class if more than 0.5
-# of its k-neighbors are in that class.
+# For now, we will consider a data point as predicted in positive class if more than 0.5
+# of its k-neighbors are positive examples.
 threshold = 0.5
 # convert predicted ratios to predicted labels
 pred_labels = None
 
 # show the distribution of predicted and true labels in a confusion matrix
 confusion = conf_matrix(...)
 confusion
 ```
 
 %% Cell type:markdown id: tags:
 
 Evaluate your model on the test data and report your **accuracy**. Also, calculate and report the 95% confidence interval on the generalization **error** estimate.
 
 %% Cell type:code id: tags:
 
 ``` python
 # TODO
 # Calculate and report accuracy and generalization error with confidence interval here. Show your work in this cell.
 
 print('Accuracy: {}'.format(accuracy))
 print('Confidence interval: {}-{}'.format(lower_bound, upper_bound))
 ```
 
 %% Cell type:markdown id: tags:
 
- ## TASK 4: Plotting a learning curve
-
-A learning curve shows how error changes as the training set size increases. For more information, see [learning curves](https://www.dataquest.io/blog/learning-curves-machine-learning/).
-We'll plot the error values for training and validation data while varying the size of the training set. Report a good size for training set for which there is a good balance between bias and variance.
-
-%% Cell type:markdown id: tags:
-
-### Rubric:
-* Correct training error calculation for different training set sizes +8, +8
-* Correct validation error calculation for different training set sizes +8, +8
-* Reasonable learning curve +4, +4
-
-%% Cell type:code id: tags:
-
-``` python
-# train using %10, %20, %30, ..., 100% of training data
-training_proportions = np.arange(0.10, 1.01, 0.10)
-train_size = len(train_indices)
-training_sizes = np.int(np.ceil(train_size*proportion))
-
-# TODO
-error_train = []
-error_val = []
-
-# For each size in training_sizes
-for size in training_sizes:
-    # fit the model using "size" data point
-    # Calculate error for training and validation sets
-    # populate error_train and error_val arrays.
-    # Each entry in these arrays
-    # should correspond to each entry in training_sizes.
-
-# plot the learning curve
-plt.plot(training_sizes, error_train, 'r', label = 'training_error')
-plt.plot(training_sizes, error_val, 'g', label = 'validation_error')
-plt.legend()
-plt.show()
-```
-
-%% Cell type:markdown id: tags:
-
-## TASK 5: Determining *k*
+## TASK 4: Determining *k*
 
 %% Cell type:markdown id: tags:
 
 ### Rubric:
 * Accuracies reported with various *k* values +5, +5
-* Confusion matrices shown for various *k* values +5, +5
+* Confusion matrix for new *k* +5, +5
 
 %% Cell type:markdown id: tags:
 
 We can use the validation set to come up with a *k* value that results in better performance in terms of accuracy.
 
 Below calculate the accuracies for different values of *k* using the validation set. Report a good *k* value and use it in the analyses that follow this section. Report confusion matrix for the new value of *k*.
 
 %% Cell type:code id: tags:
 
 ``` python
 # TODO
 
 # Change values of k.
 # Calculate accuracies for the validation set.
 # Report a good k value.
 # Calculate the confusion matrix for new k.
 ```
 
 %% Cell type:markdown id: tags:
 
-## TASK 6: ROC curve analysis
-* Correct implementation +20, +20
+## TASK 5: ROC curve analysis
+* Correct implementation +25, +25
 
 %% Cell type:markdown id: tags:
 
 ROC curves are a good way to visualize sensitivity vs. 1-specificity for varying cut off points. Now, implement, in **model.ipynb**, a "ROC" function. "ROC" takes a list containing different threshold values to try and returns two arrays; one where each entry is the sensitivity at a given threshold and the other where entries are 1-specificities.
 
 %% Cell type:markdown id: tags:
 
-Use the *k* value you found above, if you completed TASK 5, else use *k* = 10 to plot the ROC curve for values between 0.1 and 1.0.
+Use the *k* value you found above, if you completed TASK 4, else use *k* = 10 to plot the ROC curve for values between 0.1 and 1.0.
 
 %% Cell type:code id: tags:
 
 ``` python
 # TODO
 # ROC curve
 roc_sens, roc_spec_ = ROC(true_labels=..., preds=..., np.arange(0.1, 1.0, 0.1))
-plt.plot(roc_sens, roc_spec_)
+plt.plot(roc_spec_,  roc_sens)
 plt.show()
 ```
 
 %% Cell type:markdown id: tags:
 
-## TASK 7: Assess suitability of *k*-NN to your dataset
+## TASK 6: Assess suitability of *k*-NN to your dataset
+* +10, +10
 
 %% Cell type:markdown id: tags:
 
-Use this cell to write about your understanding of why *k*-NN performed well if it did or why not if it didn't. What properties of the dataset could have affected the performance of the algorithm?
+Insert a cell below to write about your understanding of why *k*-NN performed well if it did or why not if it didn't. What properties of the dataset could have affected the performance of the algorithm?

ProgrammingAssignment_1/model.ipynb

 %% Cell type:markdown id: tags:
 
 # JUPYTER NOTEBOOK TIPS
 
 Each rectangular box is called a cell.
 * Ctrl+ENTER evaluates the current cell; if it contains Python code, it runs the code, if it contains Markdown, it returns rendered text.
 * Alt+ENTER evaluates the current cell and adds a new cell below it.
 * If you click to the left of a cell, you'll notice the frame changes color to blue. You can erase a cell by hitting 'dd' (that's two "d"s in a row) when the frame is blue.
 
 %% Cell type:markdown id: tags:
 
 # Supervised Learning Model Skeleton
 
 We'll use this skeleton for implementing different supervised learning algorithms.
 
 %% Cell type:code id: tags:
 
 ``` python
 class Model:
 
     def fit(self):
 
         raise NotImplementedError
 
     def predict(self, test_points):
         raise NotImplementedError
 ```
 
 %% Cell type:code id: tags:
 
 ``` python
 def preprocess(feature_file, label_file):
     '''
     Args:
         feature_file: str
             file containing features
         label_file: str
             file containing labels
     Returns:
         features: ndarray
             nxd features
         labels: ndarray
             nx1 labels
     '''
 
     # read in features and labels
     return features, labels
 ```
 
 %% Cell type:code id: tags:
 
 ``` python
 def partition(size, t, v = 0):
     '''
     Args:
         size: int
             number of examples in the whole dataset
         t: float
             proportion kept for test
         v: float
             proportion kept for validation
     Returns:
         test_indices: ndarray
             1D array containing test set indices
         val_indices: ndarray
             1D array containing validation set indices
         train_indices: ndarray
             1D array containing training set indices
     '''
 
     # number of test and validation examples
 
     return test_indices, val_indices, train_indices
 ```
 
 %% Cell type:markdown id: tags:
 
 ## TASK 1: Implement `distance` function
 
 %% Cell type:markdown id: tags:
 
 "distance" function will be used in calculating cost of *k*-NN. It should take two data points and the name of the metric and return a scalar value.
 
 %% Cell type:code id: tags:
 
 ``` python
 #TODO: Programming Assignment 1
 def distance(x, y, metric):
     '''
     Args:
         x: ndarray
             1D array containing coordinates for a point
         y: ndarray
             1D array containing coordinates for a point
         metric: str
             Euclidean, Manhattan
     Returns:
         dist: float
     '''
    if metric == 'Euclidean':
        raise NotImplementedError
    elif metric == 'Manhattan':
        raise NotImplementedError
    else:
        raise ValueError('{} is not a valid metric.'.format(metric))
    return dist # scalar distance btw x and y
 ```
 
 %% Cell type:markdown id: tags:
 
 ## General supervised learning performance related functions
 
 %% Cell type:markdown id: tags:
 
 Implement the "conf_matrix" function that takes as input an array of true labels (*true*) and an array of predicted labels (*pred*). It should output a numpy.ndarray.
 
 %% Cell type:code id: tags:
 
 ``` python
 # TODO: Programming Assignment 1
 
 def conf_matrix(true, pred, n_classes):
     '''
     Args:
         true:  ndarray
             nx1 array of true labels for test set
         pred: ndarray
             nx1 array of predicted labels for test set
         n_classes: int
     Returns:
         result: ndarray
-            n_classes x n_classes array confusion matrix
+            n_classes x n_classes confusion matrix
     '''
     raise NotImplementedError
-    result = np.ndarray([n_classes, n_classes])
 
 
     # returns the confusion matrix as numpy.ndarray
     return result
 ```
 
 %% Cell type:markdown id: tags:
 
 ROC curves are a good way to visualize sensitivity vs. 1-specificity for varying cut off points. "ROC" takes a list containing different *threshold* parameter values to try and returns two arrays; one where each entry is the sensitivity at a given threshold and the other where entries are 1-specificities.
 
 %% Cell type:code id: tags:
 
 ``` python
 # TODO: Programming Assignment 1
 
 def ROC(true_labels, preds, value_list):
     '''
     Args:
         true_labels: ndarray
             1D array containing true labels
         preds: ndarray
             1D array containing thresholded value (e.g. proportion of neighbors in kNN)
         value_list: ndarray
             1D array containing different threshold values
     Returns:
         sens: ndarray
             1D array containing sensitivities
         spec_: ndarray
             1D array containing 1-specifities
     '''
 
     # calculate sensitivity, 1-specificity
     # return two arrays
 
     raise NotImplementedError
 
     return sens, spec_
 ```

ProgrammingAssignment_2/model.ipynb

+%% Cell type:markdown id: tags:
+
+# JUPYTER NOTEBOOK TIPS
+
+Each rectangular box is called a cell.
+* ctrl+ENTER evaluates the current cell; if it contains Python code, it runs the code, if it contains Markdown, it returns rendered text.
+* alt+ENTER evaluates the current cell and adds a new cell below it.
+* If you click to the left of a cell, you'll notice the frame changes color to blue. You can erase a cell by hitting 'dd' (that's two "d"s in a row) when the frame is blue.
+
+%% Cell type:markdown id: tags:
+
+# Supervised Learning Model Skeleton
+
+We'll use this skeleton for implementing different supervised learning algorithms.
+
+%% Cell type:code id: tags:
+
+``` python
+class Model:
+
+    def fit(self):
+
+        raise NotImplementedError
+
+    def predict(self, test_points):
+        raise NotImplementedError
+```
+
+%% Cell type:code id: tags:
+
+``` python
+def preprocess(data_f, feature_names_f):
+    '''
+    data_f: where to read the dataset from
+    feature_names_f: where to read the feature names from
+    Returns:
+        features: ndarray
+            nxd array containing `float` feature values
+        labels: ndarray
+            1D array containing `float` label
+    '''
+    # You might find np.genfromtxt useful for reading in the file. Be careful with the file delimiter,
+    # e.g. for comma-separated files use delimiter=',' argument.
+
+    data = np.genfromtxt(data_f)
+    features = data[:,:-1]
+    target = data[:,-1]
+    feature_names = np.genfromtxt(feature_names_f, dtype='unicode')
+
+    return features, feature_names, target
+```
+
+%% Cell type:markdown id: tags:
+
+In cases where data is not abundantly available, we resort to getting an error estimate from average of error on different splits of dataset. In this case, every fold of data is used for testing and for training in turns, i.e. assuming we split our data into 3 folds, we'd
+* train our model on fold-1+fold-2 and test on fold-3
+* train our model on fold-1+fold-3 and test on fold-2
+* train our model on fold-2+fold-3 and test on fold-1.
+
+We'd use the average of the error we obtained in three runs as our error estimate.
+
+Implement function "kfold" below.
+
+%% Cell type:code id: tags:
+
+``` python
+# TODO: Programming Assignment 2
+
+def kfold(size, k):
+
+    '''
+    Args:
+        size: int
+            number of examples in the dataset that you want to split into k
+        k: int
+            Number of desired splits in data.(Assume test set is already separated.)
+        Returns:
+        fold_dict: dict
+            A dictionary with integer keys corresponding to folds. Values are (training_indices, val_indices).
+
+        val_indices: ndarray
+            1/k of training indices randomly chosen and separates them as validation partition.
+        train_indices: ndarray
+            Remaining 1-(1/k) of the indices.
+
+            e.g. fold_dict = {0: (train_0_indices, val_0_indices),
+            1: (train_1_indices, val_1_indices), 2: (train_2_indices, val_2_indices)} for k = 3
+    '''
+
+    return fold_dict
+```
+
+%% Cell type:markdown id: tags:
+
+Implement "mse" and regularization functions. They will be used in the fit method of linear regression.
+
+%% Cell type:code id: tags:
+
+``` python
+#TODO: Programming Assignment 2
+def mse(y_pred, y_true):
+    '''
+    Args:
+        y_hat: ndarray
+            1D array containing data with `float` type. Values predicted by our method
+        y_true: ndarray
+            1D array containing data with `float` type. True y values
+    Returns:
+        cost: ndarray
+            1D array containing mean squared error between y_pred and y_true.
+
+    '''
+    raise NotImplementedError
+
+    return cost
+
+```
+
+%% Cell type:code id: tags:
+
+``` python
+#TODO: Programming Assignment 2
+def regularization(weights, method):
+    '''
+    Args:
+        weights: ndarray
+            1D array with `float` entries
+        method: str
+    Returns:
+        value: float
+            A single value. Regularization term that will be used in cost function in fit.
+    '''
+    if method == "l1":
+        value = None
+    elif method == "l2":
+        value = None
+    raise NotImplementedError
+    return value
+```

ProgrammingAssignment_3/model.ipynb

+%% Cell type:markdown id: tags:
+
+# JUPYTER NOTEBOOK TIPS
+
+Each rectangular box is called a cell.
+* ctrl+ENTER evaluates the current cell; if it contains Python code, it runs the code, if it contains Markdown, it returns rendered text.
+* alt+ENTER evaluates the current cell and adds a new cell below it.
+* If you click to the left of a cell, you'll notice the frame changes color to blue. You can erase a cell by hitting 'dd' (that's two "d"s in a row) when the frame is blue.
+
+%% Cell type:markdown id: tags:
+
+# Supervised Learning Model Skeleton
+
+We'll use this skeleton for implementing different supervised learning algorithms.
+
+%% Cell type:code id: tags:
+
+``` python
+class Model:
+
+    def fit(self):
+
+        raise NotImplementedError
+
+    def predict(self, test_points):
+        raise NotImplementedError
+```
+
+%% Cell type:markdown id: tags:
+
+## General supervised learning performance related functions
+
+%% Cell type:markdown id: tags:
+
+"conf_matrix" function that takes as input an array of true labels (*true*) and an array of predicted labels (*pred*).
+
+%% Cell type:code id: tags:
+
+``` python
+def conf_matrix(true, pred):
+    '''
+    Args:
+        true:  ndarray
+            nx1 array of true labels for test set
+        pred: ndarray
+            nx1 array of predicted labels for test set
+    Returns:
+        ndarray
+    '''
+
+    tp = tn = fp = fn = 0
+    # calculate true positives (tp), true negatives(tn)
+    # false positives (fp) and false negatives (fn)
+
+    size = len(true)
+    for i in range(size):
+        if true[i]==1:
+            if pred[i] == 1:
+                tp += 1
+            else:
+                fn += 1
+        else:
+            if pred[i] == 0:
+                tn += 1
+            else:
+                fp += 1
+
+    # returns the confusion matrix as numpy.ndarray
+    return np.array([[tp,fp], [fn, tn]])
+```

data/housing.names

+CRIM
+ZN
+INDUS
+CHAS
+NOX
+RM
+AGE
+DIS
+RAD
+TAX
+PTRATIO
+B
+LSTAT