diff --git a/ProgrammingAssignment1.ipynb b/ProgrammingAssignment1.ipynb
new file mode 100644
index 0000000000000000000000000000000000000000..0951185f36643a0d18b3b8e31866d574cd73b20e
--- /dev/null
+++ b/ProgrammingAssignment1.ipynb
@@ -0,0 +1,346 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# $k$-Nearest Neighbor\n",
+ "\n",
+ "We'll implement $k$-Nearest Neighbor ($k$-NN) algorithm for this assignment. A skeleton of a general supervised learning model is provided in \"model.ipynb\". Please look through it and complete the \"preprocess\" and \"partition\" methods.\n",
+ "\n",
+ "### Assignment Goals:\n",
+ "In this assignment, we will:\n",
+ "* learn to split a dataset into training/validation/test partitions \n",
+ "* use the validation dataset to find a good value for $k$\n",
+ "* Having found the \"best\" $k$, we'll obtain final performance measures:\n",
+ " * accuracy, generalization error and ROC curve\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "You can use numpy for array operations and matplotlib for plotting for this assignment. Please do not add other libraries."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import numpy as np\n",
+ "import matplotlib.pyplot as plt"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Following code makes the Model class and relevant functions available from model.ipynb."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%run 'model.ipynb'"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Choice of distance metric plays an important role in the performance of $k$-NN. Let's start by implementing a distance method in the \"distance\" function below. It should take two data points and the name of the metric and return a scalar value."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def distance(x, y, metric):\n",
+ " '''\n",
+ " x: a 1xd array\n",
+ " y: a 1xd array\n",
+ " metric: Euclidean, Hamming, etc.\n",
+ " '''\n",
+ " raise NotImplementedError\n",
+ " \n",
+ " return dist # scalar distance btw x and y"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### $k$-NN Class Methods"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We can start implementing our $k$-NN classifier. $k$-NN class inherits Model class. You'll need to implement \"fit\" and \"predict\" methods. Use the \"distance\" function you defined above. \"fit\" method takes $k$ as an argument. \"predict\" takes as input the feature vector for a single test point and outputs the predicted class and the proportion of predicted class labels in $k$ nearest neighbors."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "class kNN(Model):\n",
+ " '''\n",
+ " Inherits Model class. Implements the k-NN algorithm for classification.\n",
+ " '''\n",
+ " def __init__(self, preprocessor_f, partition_f, distance_f):\n",
+ " super().__init__(preprocessor_f, partition_f)\n",
+ " \n",
+ " # set self.distance_f and self.distance_metric\n",
+ " \n",
+ " \n",
+ " def fit(self, k):\n",
+ " '''\n",
+ " Fit the model. This is pretty straightforward for k-NN.\n",
+ " '''\n",
+ " \n",
+ " raise NotImplementedError\n",
+ " \n",
+ " return\n",
+ " \n",
+ " \n",
+ " def predict(self, test_point):\n",
+ " \n",
+ " raise NotImplementedError\n",
+ " \n",
+ " \n",
+ " # use self.distance_f(...,self.distance_metric)\n",
+ " \n",
+ " # return the predicted class label and the following ratio: \n",
+ " # number of points that have the same label as the test point / k\n",
+ " return predicted_label, ratio\n",
+ " "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Build and Evaluate the Model (Accuracy, Confidence Interval, Confusion Matrix)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "It's time to build and evaluate our model now. Remember you need to provide values to $p$, $v$ parameters for \"partition\" function and to $file\\_path$ for \"preprocess\" function."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# populate the keyword arguments dictionary kwargs\n",
+ "kwargs = {'p': 0.3, 'v': 0.1, 'file_path': 'mnist_test.csv', 'metric': 'Euclidean'}\n",
+ "# initialize the model\n",
+ "my_model = kNN(preprocessor_f=preprocess, partition_f=partition, distance=distance, **kwargs)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Assign a value to $k$ and fit the $k$-NN model."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "my_model.fit(k=10)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "You can use \"predict_batch\" function below to evaluate your model on the test data. You do not need to change the value of the threshold yet."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def predict_batch(model, indices, threshold=0.5):\n",
+ " '''\n",
+ " model: a fitted k-NN model\n",
+ " indices: for data points to predict\n",
+ " threshold: lower limit on the ratio for a point to be considered positive\n",
+ " '''\n",
+ " \n",
+ " predicted_labels = []\n",
+ " true_labels = []\n",
+ "\n",
+ " for index in indices:\n",
+ " # vary the threshold value for ROC analysis\n",
+ " predicted_classes.append(model.predict(model.features[index], threshold))\n",
+ " true_classes.append(model.labels[index])\n",
+ "\n",
+ " return predicted_labels, true_labels"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Use \"predict_batch\" function above to report your model's accuracy on the test set. Also, calculate and report the confidence interval on the generalization error estimate."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "predict_batch(my_model, my_model.test_indices)\n",
+ "# Calculate accuracy and generalization error with confidence interval here."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# TODO: leaa \n",
+ "\n",
+ "Now that we have the true labels and the predicted ones from our model, we can build a confusion matrix and see how accurate our model is. Implement the \"conf_matrix\" function that takes as input an array of true labels ($true$) and an array of predicted labels ($pred$). It should output a numpy.ndarray. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def conf_matrix(true, pred):\n",
+ " '''\n",
+ " true: nx1 array of true labels for test set\n",
+ " pred: nx1 array of predicted labels for test set\n",
+ " '''\n",
+ " raise NotImplementedError\n",
+ " # returns the confusion matrix as numpy.ndarray\n",
+ " return c_mat"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Finding a good value for $k$\n",
+ "\n",
+ "We can use the validation set to come up with a $k$ value that results in better performance in terms of accuracy. Additionally, in some cases, predicting examples from a certain class correctly is more critical than other classes. In those cases, we can use the confusion matrix to find a good trade off between correct and wrong predictions and allow more wrong predictions in some classes to predict more examples correctly in a that class.\n",
+ "\n",
+ "Below calculate the accuracies and confusion matrices for different values of $k$ using the validation set. Report a good $k$ value and use it in the analyses that follow this section."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Change values of $k. \n",
+ "# Calculate accuracies and confusion matrices for the validation set.\n",
+ "# Report a good k value that you'll use in the following analyses."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### ROC curve and confusion matrix for the final model\n",
+ "ROC curves are a good way to visualize sensitivity vs. 1-specificity for varying cut off points. Now, implement a \"ROC\" function that predicts the labels of the test set examples using different $threshold$ values in \"predict\" and plot the ROC curve. \"ROC\" takes a list containing different $threshold$ parameter values to try and returns two arrays; one where each entry is the sensitivity at a given threshold and the other where entries are 1-specificities."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def ROC(model, indices, value_list):\n",
+ " '''\n",
+ " model: a fitted k-NN model\n",
+ " indices: for data points to predict\n",
+ " value_list: array containing different threshold values\n",
+ " Calculate sensitivity and 1-specificity for each point in value_list\n",
+ " Return two nX1 arrays: sens (for sensitivities) and spec_ (for 1-specificities)\n",
+ " '''\n",
+ " \n",
+ " # use predict_batch to obtain predicted labels at different threshold values\n",
+ " raise NotImplementedError\n",
+ " \n",
+ " return sens, spec_"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We can finally create the confusion matrix and plot the ROC curve for our optimal $k$-NN classifier."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# confusion matrix\n",
+ "conf_matrix(true_classes, predicted_classes)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# ROC curve\n",
+ "roc_sens, roc_spec_ = ROC(my_model, my_model.test_indices, np.arange(0.1, 1.0, 0.1))\n",
+ "plt.plot(roc_sens, roc_spec_)\n",
+ "plt.show()"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.6.4"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/ProgrammingAssignment2.ipynb b/ProgrammingAssignment2.ipynb
new file mode 100644
index 0000000000000000000000000000000000000000..428855aa73245f97c4298e2c421df8e0737a1f45
--- /dev/null
+++ b/ProgrammingAssignment2.ipynb
@@ -0,0 +1,254 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Linear Regression & Naive Bayes\n",
+ "\n",
+ "We'll implement linear regression &