" * address sparse data problem with **pseudocounts** (**$m$-estimate**)\n",
" \n",
"A skeleton of a general supervised learning model is provided in \"model.ipynb\". We won't make any implementations for this part of homework 2. A confusion matrix implementation is provided for you in \"model.ipynb\".\n",
"\n",
"### Note:\n",
"\n",
"You are not required to follow this exact template. You can change what parameters your functions take or partition the tasks across functions differently. However, make sure there are outputs and implementation for items listed in the rubric for each task. Also, indicate in code with comments which task you are attempting."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# GRADING\n",
"\n",
"You will be graded on parts that are marked with **\\#TODO** comments. Read the comments in the code to make sure you don't miss any.\n",
"Points are broken down further below in Rubric sections. The **first** score is for 478, the **second** is for 878 students. There are a total of 50 points in this part of assignment 2 for both 478 and 878 students."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"import json\n",
"%run 'model.ipynb'"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We will use the Enron spam/ham dataset for spam filtering. The emails are already tokenized. Below code reads in the processed data. There are 33,702 emails: 17157 spam and 16545 ham. \n",
"\n",
"**Please do not change the order of the test indices as you'll be graded on results for the first 5 test examples.**"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"dict_keys(['training', 'test'])"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# load tokenized email texts\n",
"with open('../data/enron_text.json') as f:\n",
" X = json.load(f)\n",
"# load email labels\n",
"with open('../data/enron_label.json') as f:\n",
" Y = json.load(f)\n",
"with open('../data/enron_split.json') as f:\n",
" indices = json.load(f)\n",
" \n",
"indices.keys()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"X is a list of lists. Each of its 33,702 entries correspond to a tokenized email. Y is a list. Each entry is a label for the tokenized email at the corresponding position in X. Below are 5 random emails and their corresponding labels."
"Implement the \"fit\" and \"predict\" methods for Naive Bayes. Use $m$-estimate to address missing attribute values (also called **Laplace smoothing** when $m$ = 1). In general, $m$ values should be small. We'll use $m$ = 1. We'll use log probabilities to avoid overflow."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"class Naive_Bayes(Model):\n",
" def __init__(self, m):\n",
" '''\n",
" Args:\n",
" m: int\n",
" Specifies the smoothing parameter\n",
" '''\n",
" self.m = m\n",
" \n",
" def fit(self, X, Y):\n",
" '''\n",
" Args:\n",
" X: list\n",
" list of lists where each entry is a tokenized email text\n",
" Y: ndarray\n",
" 1D array of true labels. 1: spam, 0: ham\n",
" '''\n",
" \n",
" #TODO\n",
" \n",
" # Replace Nones, empty lists and dictionaries below\n",
" \n",
" # List containing all distinct words in all emails\n",
" # A list might not be the best data structure for obtaining\n",
" # the vocabulary.\n",
" # Use a temporary more efficient data structure \n",
" # then populate self.vocabulary.\n",
" \n",
" self.vocabulary = []\n",
" \n",
" # find *log* class prior probabilities\n",
" self.prior = {'spam': None, 'ham': None}\n",
" \n",
" # find the number of words(counting repeats) summed across all emails in a class\n",
* address sparse data problem with **pseudocounts** (**$m$-estimate**)
A skeleton of a general supervised learning model is provided in "model.ipynb". We won't make any implementations for this part of homework 2. A confusion matrix implementation is provided for you in "model.ipynb".
### Note:
You are not required to follow this exact template. You can change what parameters your functions take or partition the tasks across functions differently. However, make sure there are outputs and implementation for items listed in the rubric for each task. Also, indicate in code with comments which task you are attempting.
%% Cell type:markdown id: tags:
# GRADING
You will be graded on parts that are marked with **\#TODO** comments. Read the comments in the code to make sure you don't miss any.
### Mandatory for 478 & 878:
| | Tasks | 478 | 878 |
|---|----------------------------|-----|-----|
| 1 | Implement `fit` method | 25 | 25 |
| 2 | Implement `predict` method | 25 | 25 |
Points are broken down further below in Rubric sections. The **first** score is for 478, the **second** is for 878 students. There are a total of 50 points in this part of assignment 2 for both 478 and 878 students.
%% Cell type:code id: tags:
``` python
importnumpyasnp
importjson
%run'model.ipynb'
```
%% Cell type:markdown id: tags:
We will use the Enron spam/ham dataset for spam filtering. The emails are already tokenized. Below code reads in the processed data. There are 33,702 emails: 17157 spam and 16545 ham.
**Please do not change the order of the test indices as you'll be graded on results for the first 5 test examples.**
%% Cell type:code id: tags:
``` python
# load tokenized email texts
withopen('../data/enron_text.json')asf:
X=json.load(f)
# load email labels
withopen('../data/enron_label.json')asf:
Y=json.load(f)
withopen('../data/enron_split.json')asf:
indices=json.load(f)
indices.keys()
```
%% Output
dict_keys(['training', 'test'])
%% Cell type:markdown id: tags:
X is a list of lists. Each of its 33,702 entries correspond to a tokenized email. Y is a list. Each entry is a label for the tokenized email at the corresponding position in X. Below are 5 random emails and their corresponding labels.
Implement the "fit" and "predict" methods for Naive Bayes. Use $m$-estimate to address missing attribute values (also called **Laplace smoothing** when $m$ = 1). In general, $m$ values should be small. We'll use $m$ = 1. We'll use log probabilities to avoid overflow.
%% Cell type:code id: tags:
``` python
classNaive_Bayes(Model):
def__init__(self,m):
'''
Args:
m: int
Specifies the smoothing parameter
'''
self.m=m
deffit(self,X,Y):
'''
Args:
X: list
list of lists where each entry is a tokenized email text
Y: ndarray
1D array of true labels. 1: spam, 0: ham
'''
#TODO
# Replace Nones, empty lists and dictionaries below
# List containing all distinct words in all emails
# A list might not be the best data structure for obtaining
# the vocabulary.
# Use a temporary more efficient data structure
# then populate self.vocabulary.
self.vocabulary=[]
# find *log* class prior probabilities
self.prior={'spam':None,'ham':None}
# find the number of words(counting repeats) summed across all emails in a class
self.total_count={'spam':None,'ham':None}
# find the number of each word summed across all emails in a class
# populate self.word_counts
# self.word_counts['spam'] is a dict with words as keys.
self.word_counts={'spam':{},'ham':{}}
defpredict(self,X):
'''
Args:
X: list
list of lists where each entry is a tokenized email text
Returns:
probs: ndarray
mx2 array containing unnormalized log posteriors for spam and ham (for grading purposes)
preds: ndarray
1D binary array containing predicted labels
'''
preds=[]
probs=[]
#TODO
# use the attributes calculated in fit to compute unnormalized class posterior probabilities
# and predicted labels
raiseNotImplementedError
returnnp.array(probs),np.array(preds)
```
%% Cell type:markdown id: tags:
### Rubric:
* correct vocabulary length +5,+5
* correct log class priors +10, +10
* correct word counts for the 5 most frequent words in each class +10, +10
%% Cell type:markdown id: tags:
### Test `fit`
%% Cell type:code id: tags:
``` python
# initialize the model
my_model=Naive_Bayes(...)
# pass the training emails and labels
my_model.fit(X=...,Y=...)
# display the most frequent 5 words in both classes
