Skip to content
Snippets Groups Projects
Commit e9cccf6f authored by Zeynep Hakguder's avatar Zeynep Hakguder
Browse files

more code organization

parent e410cc67
No related branches found
No related tags found
No related merge requests found
Hide whitespace changes
Inline Side-by-side
ProgrammingAssignment1.ipynb
+ 33
− 64
View file @ e9cccf6f
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
# $k$-Nearest Neighbor # $k$-Nearest Neighbor
We'll implement $k$-Nearest Neighbor ($k$-NN) algorithm for this assignment. A skeleton of a general supervised learning model is provided in "model.ipynb". Please look through it and complete the "preprocess" and "partition" methods. We'll implement $k$-Nearest Neighbor ($k$-NN) algorithm for this assignment. We recommend using [Madelon](https://archive.ics.uci.edu/ml/datasets/Madelon) dataset, although it is not mandatory. If you choose to use a different dataset, it should meet the following criteria:
* dependent variable should be binary (suited for binary classification)
* number of features (attributes) should be at least 50
* number of examples (instances) should be at least 1,000
A skeleton of a general supervised learning model is provided in "model.ipynb". Please look through it and complete the "preprocess" and "partition" methods.
### Assignment Goals: ### Assignment Goals:
In this assignment, we will: In this assignment, we will:
* learn to split a dataset into training/validation/test partitions * learn to split a dataset into training/validation/test partitions
* use the validation dataset to find a good value for $k$ * use the validation dataset to find a good value for $k$
* Having found the "best" $k$, we'll obtain final performance measures: * Having found the "best" $k$, we'll obtain final performance measures:
* accuracy, generalization error and ROC curve * accuracy, generalization error and ROC curve
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
You can use numpy for array operations and matplotlib for plotting for this assignment. Please do not add other libraries. You can use numpy for array operations and matplotlib for plotting for this assignment. Please do not add other libraries.
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
import numpy as np import numpy as np
import matplotlib.pyplot as plt import matplotlib.pyplot as plt
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
Following code makes the Model class and relevant functions available from model.ipynb. Following code makes the Model class and relevant functions available from model.ipynb.
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
%run 'model.ipynb' %run 'model.ipynb'
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
Choice of distance metric plays an important role in the performance of $k$-NN. Let's start by implementing a distance method in the "distance" function below. It should take two data points and the name of the metric and return a scalar value. Choice of distance metric plays an important role in the performance of $k$-NN. Let's start with implementing a distance method in the "distance" function below. It should take two data points and the name of the metric and return a scalar value.
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
def distance(x, y, metric): def distance(x, y, metric):
''' '''
x: a 1xd array x: a 1xd array
y: a 1xd array y: a 1xd array
metric: Euclidean, Hamming, etc. metric: Euclidean, Hamming, etc.
''' '''
raise NotImplementedError raise NotImplementedError
return dist # scalar distance btw x and y return dist # scalar distance btw x and y
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
### $k$-NN Class Methods ### $k$-NN Class Methods
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
We can start implementing our $k$-NN classifier. $k$-NN class inherits Model class. You'll need to implement "fit" and "predict" methods. Use the "distance" function you defined above. "fit" method takes $k$ as an argument. "predict" takes as input the feature vector for a single test point and outputs the predicted class and the proportion of predicted class labels in $k$ nearest neighbors. We can start implementing our $k$-NN classifier. $k$-NN class inherits Model class. You'll need to implement "fit" and "predict" methods. Use the "distance" function you defined above. "fit" method takes $k$ as an argument. "predict" takes as input an $mxd$ array containing $d$-dimensional $m$ feature vectors for examples and outputs the predicted class and the proportion of predicted class labels in $k$ nearest neighbors.
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
class kNN(Model): class kNN(Model):
''' '''
Inherits Model class. Implements the k-NN algorithm for classification. Inherits Model class. Implements the k-NN algorithm for classification.
''' '''
def __init__(self, preprocessor_f, partition_f, distance_f):
super().__init__(preprocessor_f, partition_f)
# set self.distance_f and self.distance_metric
def fit(self, k): def fit(self, k, distance_f, **kwargs):
''' '''
Fit the model. This is pretty straightforward for k-NN. Fit the model. This is pretty straightforward for k-NN.
''' '''
# set self.k, self.distance_f, self.distance_metric
raise NotImplementedError raise NotImplementedError
return return
def predict(self, test_point): def predict(self, test_indices):
raise NotImplementedError raise NotImplementedError
pred = []
# for each point in test points
# use your implementation of distance function
# distance_f(..., distance_metric)
# to find the labels of k-nearest neighbors.
# Find the ratio of the positive labels
# and append to pred with pred.append(ratio).
# use self.distance_f(...,self.distance_metric)
# return the predicted class label and the following ratio: return np.array(pred)
# number of points that have the same label as the test point / k
return predicted_label, ratio
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
### Build and Evaluate the Model (Accuracy, Confidence Interval, Confusion Matrix) ### Build and Evaluate the Model (Accuracy, Confidence Interval, Confusion Matrix)
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
It's time to build and evaluate our model now. Remember you need to provide values to $p$, $v$ parameters for "partition" function and to $file\_path$ for "preprocess" function. It's time to build and evaluate our model now. Remember you need to provide values to $p$, $v$ parameters for "partition" function and to $file\_path$ for "preprocess" function.
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
# populate the keyword arguments dictionary kwargs # populate the keyword arguments dictionary kwargs
kwargs = {'p': 0.3, 'v': 0.1, 'file_path': 'mnist_test.csv', 'metric': 'Euclidean'} kwargs = {'p': 0.3, 'v': 0.1, seed: 123, 'file_path': 'madelon_train'}
# initialize the model # initialize the model
my_model = kNN(preprocessor_f=preprocess, partition_f=partition, distance=distance, **kwargs) my_model = kNN(preprocessor_f=preprocess, partition_f=partition, **kwargs)
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
Assign a value to $k$ and fit the $k$-NN model. Assign a value to $k$ and fit the kNN model.
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
my_model.fit(k=10) kwargs_f = {'metric': 'Euclidean'}
my_model.fit(k = 10, distance_f=distance, **kwargs_f)
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
You can use "predict_batch" function below to evaluate your model on the test data. You do not need to change the value of the threshold yet. Evaluate your model on the test data and report your accuracy. Also, calculate and report the confidence interval on the generalization error estimate.
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
def predict_batch(model, indices, threshold=0.5): final_labels = my_model.predict(my_model.test_indices)
'''
model: a fitted k-NN model
indices: for data points to predict
threshold: lower limit on the ratio for a point to be considered positive
'''
predicted_labels = []
true_labels = []
for index in indices:
# vary the threshold value for ROC analysis
predicted_classes.append(model.predict(model.features[index], threshold))
true_classes.append(model.labels[index])
return predicted_labels, true_labels
```
%% Cell type:markdown id: tags:
Use "predict_batch" function above to report your model's accuracy on the test set. Also, calculate and report the confidence interval on the generalization error estimate.
%% Cell type:code id: tags:
``` python
predict_batch(my_model, my_model.test_indices)
# Calculate accuracy and generalization error with confidence interval here. # Calculate accuracy and generalization error with confidence interval here.
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
# TODO: leaa # TODO: learning curve
Now that we have the true labels and the predicted ones from our model, we can build a confusion matrix and see how accurate our model is. Implement the "conf_matrix" function that takes as input an array of true labels ($true$) and an array of predicted labels ($pred$). It should output a numpy.ndarray. Now that we have the true labels and the predicted ones from our model, we can build a confusion matrix and see how accurate our model is. Implement the "conf_matrix" function (in model.ipynb) that takes as input an array of true labels ($true$) and an array of predicted labels ($pred$). It should output a numpy.ndarray. You do not need to change the value of the threshold parameter yet.
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
def conf_matrix(true, pred): # You should see array([ 196, 106, 193, 105]) with seed 123
''' conf_matrix(my_model.labels[my_model.test_indices], final_labels, threshold= 0.5)
true: nx1 array of true labels for test set
pred: nx1 array of predicted labels for test set
'''
raise NotImplementedError
# returns the confusion matrix as numpy.ndarray
return c_mat
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
### Finding a good value for $k$ ### Finding a good value for $k$
We can use the validation set to come up with a $k$ value that results in better performance in terms of accuracy. Additionally, in some cases, predicting examples from a certain class correctly is more critical than other classes. In those cases, we can use the confusion matrix to find a good trade off between correct and wrong predictions and allow more wrong predictions in some classes to predict more examples correctly in a that class. We can use the validation set to come up with a $k$ value that results in better performance in terms of accuracy. Additionally, in some cases, predicting examples from a certain class correctly is more critical than other classes. In those cases, we can use the confusion matrix to find a good trade off between correct and wrong predictions and allow more wrong predictions in some classes to predict more examples correctly in a that class.
Below calculate the accuracies and confusion matrices for different values of $k$ using the validation set. Report a good $k$ value and use it in the analyses that follow this section. Below calculate the accuracies and confusion matrices for different values of $k$ using the validation set. Report a good $k$ value and use it in the analyses that follow this section.
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
# Change values of $k. # Change values of $k.
# Calculate accuracies and confusion matrices for the validation set. # Calculate accuracies for the validation set.
# Report a good k value that you'll use in the following analyses. # Report a good k value that you'll use in the following analyses.
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
### ROC curve and confusion matrix for the final model ### ROC curve and confusion matrix for the final model
ROC curves are a good way to visualize sensitivity vs. 1-specificity for varying cut off points. Now, implement a "ROC" function that predicts the labels of the test set examples using different $threshold$ values in "predict" and plot the ROC curve. "ROC" takes a list containing different $threshold$ parameter values to try and returns two arrays; one where each entry is the sensitivity at a given threshold and the other where entries are 1-specificities. ROC curves are a good way to visualize sensitivity vs. 1-specificity for varying cut off points. Now, implement a "ROC" function that predicts the labels of the test set examples using different $threshold$ values in "predict" and plot the ROC curve. "ROC" takes a list containing different $threshold$ parameter values to try and returns two arrays; one where each entry is the sensitivity at a given threshold and the other where entries are 1-specificities.
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
def ROC(model, indices, value_list): def ROC(model, indices, value_list):
''' '''
model: a fitted k-NN model model: a fitted k-NN model
indices: for data points to predict indices: for data points to predict
value_list: array containing different threshold values value_list: array containing different threshold values
Calculate sensitivity and 1-specificity for each point in value_list Calculate sensitivity and 1-specificity for each point in value_list
Return two nX1 arrays: sens (for sensitivities) and spec_ (for 1-specificities) Return two nX1 arrays: sens (for sensitivities) and spec_ (for 1-specificities)
''' '''
# use predict_batch to obtain predicted labels at different threshold values # use predict_batch to obtain predicted labels at different threshold values
raise NotImplementedError raise NotImplementedError
return sens, spec_ return sens, spec_
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
We can finally create the confusion matrix and plot the ROC curve for our optimal $k$-NN classifier. We can finally create the confusion matrix and plot the ROC curve for our optimal $k$-NN classifier.
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
# confusion matrix # confusion matrix
conf_matrix(true_classes, predicted_classes) conf_matrix(true_classes, predicted_classes)
``` ```
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
# ROC curve # ROC curve
roc_sens, roc_spec_ = ROC(my_model, my_model.test_indices, np.arange(0.1, 1.0, 0.1)) roc_sens, roc_spec_ = ROC(my_model, my_model.test_indices, np.arange(0.1, 1.0, 0.1))
plt.plot(roc_sens, roc_spec_) plt.plot(roc_sens, roc_spec_)
plt.show() plt.show()
``` ```
......
ProgrammingAssignment2.ipynb
+ 5
− 5
View file @ e9cccf6f
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
# Linear Regression & Naive Bayes # Linear Regression & Naive Bayes
We'll implement linear regression & Naive Bayes algorithms for this assignment. Please modify the "preprocess" and "partition" methods in "model.ipynb" to suit your datasets for this assignment. In this assignment, we have a small dataset available to us. We won't have examples to spare for validation set, instead we'll use cross-validation to tune hyperparameters. We'll implement linear regression & Naive Bayes algorithms for this assignment. Please modify the "preprocess" and "partition" methods in "model.ipynb" to suit your datasets for this assignment. In this assignment, we have a small dataset available to us. We won't have examples to spare for validation set, instead we'll use cross-validation to tune hyperparameters.
### Assignment Goals: ### Assignment Goals:
In this assignment, we will: In this assignment, we will:
* implement linear regression * implement linear regression
* use gradient descent for optimization * use gradient descent for optimization
* use residuals to decide if we need a polynomial model * use residuals to decide if we need a polynomial model
* change our model to quadratic/cubic regression and use cross-validation to find the "best" polynomial degree * change our model to quadratic/cubic regression and use cross-validation to find the "best" polynomial degree
* implement regularization techniques * implement regularization techniques
* $l_1$/$l_2$ regularization * $l_1$/$l_2$ regularization
* use cross-validation to find a good regularization parameter $\lambda$ * use cross-validation to find a good regularization parameter $\lambda$
* implement Naive Bayes * implement Naive Bayes
* address sparse data problem with **pseudocounts** (**$m$-estimate**) * address sparse data problem with **pseudocounts** (**$m$-estimate**)
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
You can use numpy for array operations and matplotlib for plotting for this assignment. Please do not add other libraries. You can use numpy for array operations and matplotlib for plotting for this assignment. Please do not add other libraries.
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
import numpy as np import numpy as np
import matplotlib.pyplot as plt import matplotlib.pyplot as plt
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
Following code makes the Model class and relevant functions available from "model.ipynb". Following code makes the Model class and relevant functions available from "model.ipynb".
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
%run 'model.ipynb' %run 'model.ipynb'
``` ```
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
def mse(y_pred, y_true): def mse(y_pred, y_true):
''' '''
y_hat: values predicted by our method y_hat: values predicted by our method
y_true: true y values y_true: true y values
''' '''
raise NotImplementedError raise NotImplementedError
# returns mean squared error between y_pred and y_true # returns mean squared error between y_pred and y_true
return cost return cost
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
We'll start by implementing a partition function for $k$-fold cross-validation. $5$ and $10$ are commonly used values for $k$. You can use either one of them. We'll start by implementing a partition function for $k$-fold cross-validation. $5$ and $10$ are commonly used values for $k$. You can use either one of them.
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
def kfold(k): def kfold(k):
''' '''
k: number of desired splits in data. k: number of desired splits in data.
Assume test set is already separated. Assume test set is already separated.
This function chooses 1/k of training indices randomly and separates them as validation partition. This function chooses 1/k of training indices randomly and separates them as validation partition.
It returns the 1/k selected indices as val_indices and the remaining 1-(1/k) of the indices as train_indices It returns the 1/k selected indices as val_indices and the remaining 1-(1/k) of the indices as train_indices
''' '''
return train_indices, val_indices return train_indices, val_indices
``` ```
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
class linear_regression(Model): class linear_regression(Model):
def __init__(self, preprocessor_f, partition_f, **kwargs): def __init__(self, preprocessor_f, partition_f, **kwargs):
super().__init__(preprocessor_f, partition_f, **kwargs) super().__init__(preprocessor_f, partition_f, **kwargs)
theta = theta =
# You can disregard polynomial_degree and regularizer in your first pass # You can disregard polynomial_degree and regularizer in your first pass
def fit(self, learning_rate = 0.001, epochs = 1000, regularizer=None, polynomial_degree=1, **kwargs): def fit(self, learning_rate = 0.001, epochs = 1000, regularizer=None, polynomial_degree=1, **kwargs):
# for each epoch # for each epoch
# compute y_hat array which holds model predictions for training examples # compute y_hat array which holds model predictions for training examples
y_hat = None y_hat = None
# use mse function to find the cost # use mse function to find the cost
cost = None cost = None
# calculate gradients wrt theta # calculate gradients wrt theta
grad_theta = None grad_theta = None
# update theta # update theta
theta_curr = None theta_curr = None
raise NotImplementedError raise NotImplementedError
return theta return theta
def predict(self, indices): def predict(self, indices):
raise NotImplementedError raise NotImplementedError
return y_hat return y_hat
``` ```
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
# populate the keyword arguments dictionary kwargs # populate the keyword arguments dictionary kwargs
kwargs = {'p': 0.3, 'v': 0.0, 'file_path': 'mnist_test.csv', 'k': 5} kwargs = {'p': 0.3, 'v': 0.0, 'file_path': 'mnist_test.csv', 'k': 5}
# initialize the model # initialize the model
my_model = linear_regression(preprocessor_f=preprocess, partition_f=partition, k_fold=True, **kwargs) my_model = linear_regression(preprocessor_f=preprocess, partition_f=partition, k_fold=True, **kwargs)
``` ```
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
# use fit_kwargs to pass arguments to regularization function # use fit_kwargs to pass arguments to regularization function
fit_kwargs = {} fit_kwargs = {}
my_model.fit(**fit_kwargs) my_model.fit(**fit_kwargs)
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
Residuals are the differences between the predicted value $y_{hat}$ and the true value $y$ for each example. Predict $y_{hat}$ for the validation set. Calculate and plot residuals. Residuals are the differences between the predicted value $y_{hat}$ and the true value $y$ for each example. Predict $y_{hat}$ for the validation set. Calculate and plot residuals.
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
y_hat_val = my_model.predict(my_model.features[my_model.val_indices]) y_hat_val = my_model.predict(my_model.features[my_model.val_indices])
residuals = my_model.labels[my_model.val_indices] - y_hat_val residuals = my_model.labels[my_model.val_indices] - y_hat_val
plt.plot(residuals) plt.plot(residuals)
plt.show() plt.show()
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
If the data is better suited for quadratic/cubic regression, regions of positive and negative residuals will alternate in the plot. Regardless, modify the fit and predict in the class definition to raise the feature values to $polynomial\_degree$. You can directly make the modification in the above definition, do not repeat. Use the validation set to find among the degree of polynomial that results in lowest "mse". If the data is better suited for quadratic/cubic regression, regions of positive and negative residuals will alternate in the plot. Regardless, modify fit" and "predict" in the class definition to raise the feature values to $polynomial\_degree$. You can directly make the modification in the above definition, do not repeat. Use the validation set to find the degree of polynomial that results in lowest "mse".
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
# calculate mse for linear model # calculate mse for linear model
fit_kwargs = {} fit_kwargs = {}
my_model.fit(polynomial_degree = 1 ,**fit_kwargs) my_model.fit(polynomial_degree = 1 ,**fit_kwargs)
pred_3 = my_model.predict(my_model.features[my_model.val_indices]) pred_3 = my_model.predict(my_model.features[my_model.val_indices])
mse_1 = mse(pred_2, my_model.labels[my_model.val_indices]) mse_1 = mse(pred_2, my_model.labels[my_model.val_indices])
# calculate mse for quadratic model # calculate mse for quadratic model
my_model.fit(polynomial_degree = 2 ,**fit_kwargs) my_model.fit(polynomial_degree = 2 ,**fit_kwargs)
pred_2 = my_model.predict(my_model.features[my_model.val_indices]) pred_2 = my_model.predict(my_model.features[my_model.val_indices])
mse_2 = mse(pred_2, my_model.labels[my_model.val_indices]) mse_2 = mse(pred_2, my_model.labels[my_model.val_indices])
# calculate mse for cubic model # calculate mse for cubic model
my_model.fit(polynomial_degree = 3 ,**fit_kwargs) my_model.fit(polynomial_degree = 3 ,**fit_kwargs)
pred_3 = my_model.predict(my_model.features[my_model.val_indices]) pred_3 = my_model.predict(my_model.features[my_model.val_indices])
mse_3 = mse(pred_2, my_model.labels[my_model.val_indices]) mse_3 = mse(pred_2, my_model.labels[my_model.val_indices])
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
Define "regularization" function which implements $l_1$ and $l_2$ regularization. You'll use this function in "fit" method of "linear_regression" class. Define "regularization" function which implements $l_1$ and $l_2$ regularization. You'll use this function in "fit" method of "linear_regression" class.
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
def regularization(method): def regularization(method):
if method == "l1": if method == "l1":
raise NotImplementedError raise NotImplementedError
elif method == "l2": elif method == "l2":
raise NotImplementedError raise NotImplementedError
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
Using the validation set, and the value of $polynomial_{degree}$ you found above, try different values of $\lambda$ to find a a good value that results in low $mse$. You can Using the validation set, and the value of $polynomial_{degree}$ you found above, try different values of $\lambda$ to find a a good value that results in low $mse$. You can
......
model-Solution.ipynb
+ 34
− 1
View file @ e9cccf6f
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
You might need to preprocess your dataset depending on which dataset you are using. This step is for reading the dataset and for extracting features and labels. The "preprocess" function should return an $n \times d$ features array, and an $n \times 1$ labels array, where $n$ is the number of examples and $d$ is the number of features in the dataset. You might need to preprocess your dataset depending on which dataset you are using. This step is for reading the dataset and for extracting features and labels. The "preprocess" function should return an $n \times d$ features array, and an $n \times 1$ labels array, where $n$ is the number of examples and $d$ is the number of features in the dataset.
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
def preprocess(file_path): def preprocess(file_path):
''' '''
file_path: where to read the dataset from file_path: where to read the dataset from
returns nxd features, nx1 labels returns nxd features, nx1 labels
''' '''
# You might find np.genfromtxt useful for reading in the file. Be careful with the file delimiter, # You might find np.genfromtxt useful for reading in the file. Be careful with the file delimiter,
# e.g. for comma-separated files use delimiter=',' argument. # e.g. for comma-separated files use delimiter=',' argument.
#raise NotImplementedError #raise NotImplementedError
feature_path = file_path + '.data' feature_path = file_path + '.data'
label_path = file_path + '.labels' label_path = file_path + '.labels'
features = np.genfromtxt(feature_path) features = np.genfromtxt(feature_path)
labels = np.genfromtxt(label_path) labels = np.genfromtxt(label_path)
#features = data[:, 1:] #features = data[:, 1:]
#labels = data[:, 0] #labels = data[:, 0]
#################### ####################
return features, labels return features, labels
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
Next, you'll need to split your dataset into training and validation and test sets. The "split" function should take as input the size of the whole dataset and randomly sample a proportion $p$ of the dataset as test partition and a proportion of $v$ as validation partition. The remaining will be used as training data. For example, to keep 30% of the examples as test and %10 as validation, set $p=0.3$ and $v=0.1$. You should choose these values according to the size of the data available to you. The "split" function should return indices of the training, validation and test sets. These will be used to index into the whole training set. Next, you'll need to split your dataset into training and validation and test sets. The "split" function should take as input the size of the whole dataset and randomly sample a proportion $p$ of the dataset as test partition and a proportion of $v$ as validation partition. The remaining will be used as training data. For example, to keep 30% of the examples as test and %10 as validation, set $p=0.3$ and $v=0.1$. You should choose these values according to the size of the data available to you. The "split" function should return indices of the training, validation and test sets. These will be used to index into the whole training set.
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
def partition(size, p, v, seed): def partition(size, p, v, seed):
''' '''
size: number of examples in the whole dataset size: number of examples in the whole dataset
p: proportion kept for test p: proportion kept for test
v: proportion kept for validation v: proportion kept for validation
''' '''
# np.random.choice might come in handy. Do not sample with replacement! # np.random.choice might come in handy. Do not sample with replacement!
# Be sure to not use the same indices in test and validation sets! # Be sure to not use the same indices in test and validation sets!
#raise NotImplementedError #raise NotImplementedError
data_list = np.arange(size) data_list = np.arange(size)
p_size = np.int(np.ceil(size*p)) p_size = np.int(np.ceil(size*p))
v_size = np.int(np.ceil(size*v)) v_size = np.int(np.ceil(size*v))
np.random.seed(seed) np.random.seed(seed)
permuted = np.random.permutation(data_list) permuted = np.random.permutation(data_list)
test_indices = permuted[:p_size] test_indices = permuted[:p_size]
val_indices = permuted[p_size+1:p_size+v_size] val_indices = permuted[p_size+1:p_size+v_size]
########################## ##########################
# return two 1d arrays: one keeping validation set indices, the other keeping test set indices # return two 1d arrays: one keeping validation set indices, the other keeping test set indices
return val_indices, test_indices return val_indices, test_indices
``` ```
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
class Model: class Model:
# set the preprocessing function, partition_function # set the preprocessing function, partition_function
# use kwargs to pass arguments to preprocessor_f and partition_f # use kwargs to pass arguments to preprocessor_f and partition_f
# kwargs is a dictionary and should contain p, v and file_path # kwargs is a dictionary and should contain p, v and file_path
# e.g. {'p': 0.3, 'v': 0.1, 'file_path': some_path} # e.g. {'p': 0.3, 'v': 0.1, 'file_path': some_path}
def __init__(self, preprocessor_f, partition_f, distance_f=None, **kwargs): def __init__(self, preprocessor_f, partition_f, distance_f=None, **kwargs):
self.features, self.labels = preprocessor_f(kwargs['file_path']) self.features, self.labels = preprocessor_f(kwargs['file_path'])
self.size = len(self.labels) self.size = len(self.labels)
self.val_indices, self.test_indices = partition_f(self.size, kwargs['p'], kwargs['v'], kwargs['seed']) self.val_indices, self.test_indices = partition_f(self.size, kwargs['p'], kwargs['v'], kwargs['seed'])
self.training_indices = np.delete(np.arange(self.size), np.append(self.test_indices, self.val_indices), 0) self.training_indices = np.delete(np.arange(self.size), np.append(self.test_indices, self.val_indices), 0)
def fit(self): def fit(self):
raise NotImplementedError raise NotImplementedError
def predict(self): def predict(self):
raise NotImplementedError raise NotImplementedError
``` ```
%% Cell type:code id: tags:
``` python
def conf_matrix(true_l, pred, threshold):
tp = tn = fp = fn = 0
for i in range(len(true_l)):
tmp = -1
if pred[i] > threshold:
tmp = 1
if tmp == true_l[i]:
if true_l[i] == 1:
tp += 1
else:
tn += 1
else:
if true_l[i] == 1:
fn += 1
else:
fp += 1
return np.array([tp,tn, fp, fn])
# returns the confusion matrix as numpy.ndarray
#raise NotImplementedError
```
......
model.ipynb
+ 39
− 1
View file @ e9cccf6f
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
# JUPYTER NOTEBOOK TIPS # JUPYTER NOTEBOOK TIPS
Each rectangular box is called a cell. Each rectangular box is called a cell.
* ctrl+ENTER evaluates the current cell; if it contains Python code, it runs the code, if it contains Markdown, it returns rendered text. * ctrl+ENTER evaluates the current cell; if it contains Python code, it runs the code, if it contains Markdown, it returns rendered text.
* alt+ENTER evaluates the current cell and adds a new cell below it. * alt+ENTER evaluates the current cell and adds a new cell below it.
* If you click to the left of a cell, you'll notice the frame changes color to blue. You can erase a cell by hitting 'dd' (that's two "d"s in a row) when the frame is blue. * If you click to the left of a cell, you'll notice the frame changes color to blue. You can erase a cell by hitting 'dd' (that's two "d"s in a row) when the frame is blue.
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
# Supervised Learning Model Skeleton # Supervised Learning Model Skeleton
We'll use this skeleton for implementing different supervised learning algorithms. Please complete "preprocess" and "partition" methods below. We'll use this skeleton for implementing different supervised learning algorithms. Please complete "preprocess" and "partition" methods below.
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
You might need to preprocess your dataset depending on which dataset you are using. This step is for reading the dataset and for extracting features and labels. The "preprocess" function should return an $n \times d$ features array, and an $n \times 1$ labels array, where $n$ is the number of examples and $d$ is the number of features in the dataset. In cases where there is a big difference between the scales of feautures, we want to normalize the features to have values in the same range [0,1]. If that is the case with your dataset, output normalized features to get better prediction results. You might need to preprocess your dataset depending on which dataset you are using. This step is for reading the dataset and for extracting features and labels. The "preprocess" function should return an $n \times d$ features array, and an $n \times 1$ labels array, where $n$ is the number of examples and $d$ is the number of features in the dataset. In cases where there is a big difference between the scales of feautures, we want to normalize the features to have values in the same range [0,1]. If that is the case with your dataset, output normalized features to get better prediction results.
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
def preprocess(file_path): def preprocess(file_path):
''' '''
file_path: where to read the dataset from file_path: where to read the dataset from
returns nxd features, nx1 labels returns nxd features, nx1 labels
''' '''
# You might find np.genfromtxt useful for reading in the file. Be careful with the file delimiter, # You might find np.genfromtxt useful for reading in the file. Be careful with the file delimiter,
# e.g. for comma-separated files use delimiter=',' argument. # e.g. for comma-separated files use delimiter=',' argument.
raise NotImplementedError raise NotImplementedError
return features, labels return features, labels
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
Next, you'll need to split your dataset into training and validation and test sets. The "split" function should take as input the size of the whole dataset and randomly sample a proportion $p$ of the dataset as test partition and a proportion of $v$ as validation partition. The remaining will be used as training data. For example, to keep 30% of the examples as test and %10 as validation, set $p=0.3$ and $v=0.1$. You should choose these values according to the size of the data available to you. The "split" function should return indices of the training, validation and test sets. These will be used to index into the whole training set. Next, you'll need to split your dataset into training and validation and test sets. The "split" function should take as input the size of the whole dataset and randomly sample a proportion $p$ of the dataset as test partition and a proportion of $v$ as validation partition. The remaining will be used as training data. For example, to keep 30% of the examples as test and %10 as validation, set $p=0.3$ and $v=0.1$. You should choose these values according to the size of the data available to you. The "split" function should return indices of the training, validation and test sets. These will be used to index into the whole training set.
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
def partition(size, p, v): def partition(size, p, v):
''' '''
size: number of examples in the whole dataset size: number of examples in the whole dataset
p: proportion kept for test p: proportion kept for test
v: proportion kept for validation v: proportion kept for validation
''' '''
# np.random.choice might come in handy. Do not sample with replacement! # np.random.choice might come in handy. Do not sample with replacement!
# Be sure to not use the same indices in test and validation sets! # Be sure to not use the same indices in test and validation sets!
# use the first np.ceil(size*p) for test,
# the following np.ceil(size*v) for validation set.
raise NotImplementedError raise NotImplementedError
# return two 1d arrays: one keeping validation set indices, the other keeping test set indices # return two 1d arrays: one keeping validation set indices, the other keeping test set indices
return val_indices, test_indices return val_indices, test_indices
``` ```
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
class Model: class Model:
# set the preprocessing function, partition_function # set the preprocessing function, partition_function
# use kwargs to pass arguments to preprocessor_f and partition_f # use kwargs to pass arguments to preprocessor_f and partition_f
# kwargs is a dictionary and should contain p, v and file_path # kwargs is a dictionary and should contain p, v and file_path
# e.g. {'p': 0.3, 'v': 0.1, 'file_path': some_path} # e.g. {'p': 0.3, 'v': 0.1, 'file_path': some_path}
def __init__(self, preprocessor_f, partition_f, **kwargs): def __init__(self, preprocessor_f, partition_f, **kwargs):
self.features, self.labels = preprocessor_f(kwargs['file_path']) self.features, self.labels = preprocessor_f(kwargs['file_path'])
self.size = len(self.labels) # number of examples in dataset self.size = len(self.labels) # number of examples in dataset
self.feat_dim = self.features.shape[1] # number of features self.feat_dim = self.features.shape[1] # number of features
self.val_indices, self.test_indices = partition_f(self.size, kwargs['p'], kwargs['v']) self.val_indices, self.test_indices = partition_f(self.size, kwargs['p'], kwargs['v'])
self.train_indices = np.delete(np.arange(self.size), np.append(self.test_indices, self.val_indices), 0) self.train_indices = np.delete(np.arange(self.size), np.append(self.test_indices, self.val_indices), 0)
def fit(self): def fit(self):
raise NotImplementedError raise NotImplementedError
def predict(self, testpoint): def predict(self, testpoint):
raise NotImplementedError raise NotImplementedError
``` ```
%% Cell type:markdown id: tags:
## General supervised learning related functions
%% Cell type:markdown id: tags:
Implement the "conf_matrix" function that takes as input an array of true labels ($true$) and an array of predicted labels ($pred$). It should output a numpy.ndarray.
%% Cell type:code id: tags:
``` python
def conf_matrix(true, pred):
'''
true: nx1 array of true labels for test set
pred: nx1 array of predicted labels for test set
'''
raise NotImplementedError
tp = tn = fp = fn = 0
# calculate true positives (tp), true negatives(tn)
# false positives (fp) and false negatives (fn)
# returns the confusion matrix as numpy.ndarray
return np.array([tp,tn, fp, fn])
```
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment