Experiment 4 — Naive Bayes Classifier

Lab Manual

Aim

To write a program that implements the Naive Bayes classifier for a sample training dataset stored as a .CSV file, and to compute the accuracy of the classifier on a few test examples.

Objectives

Read a labelled training dataset from a CSV file.
Apply Bayes' theorem with the "naive" feature-independence assumption.
Estimate prior and conditional (likelihood) probabilities, with Laplace smoothing.
Predict the class of unseen samples and compute accuracy & a confusion matrix.

Software

Python 3.x · Jupyter / Colab · Pandas, scikit-learn (or pure Python).

Theory

Naive Bayes is a probabilistic classifier based on Bayes' theorem. It assumes all features are conditionally independent given the class ("naive"). For each class it multiplies the prior by the likelihood of every feature value, then picks the class with the highest result. Despite the unrealistic independence assumption, it is fast, needs little data, and works well — especially for text/spam classification.

Outcome

CO3 — building and evaluating a classification model.

The One Formula

P(class | data) ∝ P(class) × P(data | class)

and "naive" means: P(data|class) = P(f₁|class) × P(f₂|class) × P(f₃|class) × …

Posterior — what we wantHow likely this class is, given the new day's features.

Prior — the base rateHow common this class is overall (e.g. 9/14 days were "Play").

Likelihood — the evidenceHow typical these feature values are for that class, multiplied together.

Why "naive"? Multiplying the feature probabilities assumes they don't influence each other (e.g. humidity and outlook are independent). That's rarely true — yet the shortcut still classifies remarkably well.

Classify a New Day

Pick the weather for tomorrow. The classifier shows its full working — the priors, each feature's likelihood for both classes, the multiplied scores, and the final normalized decision. Everything updates live.

Naive Bayes — step by stepchange any button ↓

① Prior P(class)

Play (Yes)

No play

② Likelihood P(feature | class)

③ Score = prior × all likelihoods

Yes score

No score

④ Normalize → probability & decision

—

Tip: set Outlook = Overcast — every overcast day in the data ended in a game, so the classifier becomes very confident about "Play".

Test It & Measure Accuracy

The real task: run the trained classifier on a few test days whose true answers we already know, then count how many it got right. Press the button to classify them one by one.

Test set evaluationpredicted vs actual

Outlook	Temp	Humidity	Wind	Actual	Predicted	✓/✗

—

Accuracy

Confusion matrix

Pred Yes

Pred No

Act Yes

Act No

Accuracy = correct predictions ÷ total test samples. The confusion matrix breaks it down: the diagonal (teal) is correct, the off-diagonal (rose) are mistakes.

Watch the Concepts

Two clear explainers from StatQuest with Josh Starmer. The first covers Naive Bayes for categorical data (like our weather CSV); the second handles continuous numbers.

StatQuestNaive Bayes, Clearly ExplainedPriors, likelihoods and the multinomial classifier — exactly what this experiment uses.

StatQuestGaussian Naive BayesHow to handle continuous features using the normal distribution.

Videos are embedded from YouTube and belong to their creators. If a frame is blank, your network may block YouTube — search the titles on youtube.com.

The Program

Two ways to implement it: the quick scikit-learn version, and a from-scratch version that mirrors the live demo above.

1 — sklearn_naive_bayes.py

import pandas as pd
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import CategoricalNB
from sklearn.metrics import accuracy_score, confusion_matrix

df = pd.read_csv("play_tennis.csv")

# encode every text column to numbers
enc = {c: LabelEncoder() for c in df.columns}
for c in df.columns:
    df[c] = enc[c].fit_transform(df[c])

X = df[["Outlook","Temp","Humidity","Wind"]]
y = df["Play"]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=1)

model = CategoricalNB()           # Naive Bayes for categorical data
model.fit(X_tr, y_tr)
pred = model.predict(X_te)

print("Accuracy:", accuracy_score(y_te, pred))
print("Confusion matrix:\n", confusion_matrix(y_te, pred))

2 — naive_bayes_from_scratch.py

import pandas as pd

df = pd.read_csv("play_tennis.csv")
target, n = "Play", len(df)

def predict(query):
    best, best_p = None, -1
    for c in df[target].unique():
        sub = df[df[target] == c]
        p = len(sub) / n                      # prior
        for feat, val in query.items():
            k = df[feat].nunique()             # Laplace smoothing
            p *= (len(sub[sub[feat]==val]) + 1) / (len(sub) + k)
        if p > best_p: best, best_p = c, p
    return best

q = {"Outlook":"Sunny","Temp":"Cool",
     "Humidity":"High","Wind":"Strong"}
print("Prediction:", predict(q))    # -> No

Procedure & Result

Steps followed

1 · Read CSV
Load the labelled training data.

2 · Priors
Count each class → P(class).

3 · Likelihoods
P(feature value | class) with Laplace smoothing.

4 · Predict
Multiply, compare, pick the larger score.

5 · Test
Classify held-out test samples.

6 · Accuracy
Compare with true labels → accuracy & confusion matrix.

Result

The Naive Bayes classifier was implemented from a CSV file. Prior and conditional probabilities were estimated with Laplace smoothing, unseen samples were classified, and the accuracy was computed on a test set along with a confusion matrix, demonstrating CO3.

Conclusion: Naive Bayes turns counting into classification. By assuming features are independent it keeps the maths simple — just multiply probabilities — yet stays accurate and extremely fast. Laplace smoothing prevents a single unseen value from zeroing out a whole class. It is a strong, lightweight baseline, especially for categorical and text data.

Viva Questions

State Bayes' theorem.

P(A|B) = [ P(B|A) · P(A) ] / P(B). For classification: P(class|data) is proportional to P(data|class) · P(class), and we pick the class with the highest value.

Why is the classifier called "naive"?

Because it naively assumes all features are conditionally independent given the class, so the joint likelihood is just the product of individual feature likelihoods. This is rarely true in reality, but simplifies the maths greatly.

What are prior, likelihood and posterior?

Prior P(class) is the base rate of a class before seeing features. Likelihood P(data|class) is how probable the observed features are within that class. Posterior P(class|data) is the updated probability after combining both.

What is Laplace (add-one) smoothing and why is it needed?

It adds 1 to every count so no probability is ever zero. Without it, a single feature value never seen with a class would multiply the whole class score to 0, wrongly ruling it out.

Difference between Gaussian, Multinomial and Bernoulli Naive Bayes?

Gaussian assumes continuous features follow a normal distribution; Multinomial is for counts (e.g. word frequencies); Bernoulli is for binary present/absent features.

How do you compute accuracy and what is a confusion matrix?

Accuracy = correct predictions ÷ total predictions. A confusion matrix tabulates actual vs predicted classes; its diagonal counts correct predictions, off-diagonal counts errors (false positives/negatives).

Give real applications of Naive Bayes.

Spam/e-mail filtering, sentiment and text classification, medical diagnosis, document categorisation and recommendation — anywhere fast, low-data classification is useful.

What are the advantages and limitations?

Advantages: fast, simple, works with little data and many features, good for text. Limitations: the independence assumption is unrealistic, and it gives poorly calibrated probability estimates even when the class ranking is correct.

Guessing Smart
with Probability

Classifier at a glance