Naive Bayes

“After updating our initial beliefs about something with objective new information, we get a new, improved belief.” —- Mathematician Thomas Bayes (Thomas Bayes, 1702～1761)

When you cannot accurately know the essence of something, you can rely on how many events related to that specific essence appear to judge the probability of its essential attributes.

The more events supporting a certain attribute occur, the greater the possibility that attribute holds.

In 1774, French mathematician Pierre-Simon Laplace (Pierre-Simon Laplace, 1749-1827) independently rediscovered Bayes’ formula.

Alternative writing:

Let Computer Distinguish Fruits

We need to convert fruit characteristics into data that computers can understand. The most common method is to extract attributes of objects in the real world and convert these into numbers.

For example: shape, skin color, zebra texture, weight, grip feel, taste.

Convert these descriptions into numbers, convert weight from continuous value to discrete value, this is because Naive Bayes processes discrete values

Expand sample, only 3 fruits is not enough to constitute training samples needed for Naive Bayes classification

How Do We Use Bayes’ Formula

Use prior probability and conditional probability to estimate posterior probability.

Assume different attributes of data object are independent when affecting its classification. At this time, if attributes fi and fj appear simultaneously in data object o, then probability object o belongs to category c is like this

Naive Bayes algorithm assumes features are independent of each other, only then can both sides be equal, this is also the source of “naive” in Naive Bayes classification

Use 10 fruits’ data to build Naive Bayes model

Smoothing

Zero probability situations will occur, so we usually take a very small value smaller than minimum statistical probability in this dataset to replace “zero probability”. For example, we take 0.01 here. When filling attribute values that never appeared in training data, we use this technique, we call this technique Smoothing.

Example:

Assume we have a new fruit, its shape is round, taste is sweet, then according to Naive Bayes, what are probabilities it belongs to apple, sweet orange and watermelon respectively?

apple represents classification as apple, shape-2 represents shape attribute value is 2 (i.e., round), taste-2 represents taste attribute value is 2. Similarly, we can also calculate probability this fruit belongs to sweet orange and watermelon.

Comparing these three values, 0.00198<0.00798<0.26934, so computer can conclude, this fruit most likely belongs to sweet orange

Naive Bayes Classification Mainly Includes These Steps

• Prepare data: For fruit classification case, we collected several fruit instances, and starting from common fruit attributes, converted them into data computers can understand. This data is also called training samples.

• Build model: Through fruit instances at hand, we let computer count prior probability of each fruit and attribute occurrence, and conditional probability of certain attribute occurrence under certain fruit classification. This process is also called sample-based training.

• Classify new data: For a new fruit’s attribute data, computer derives and calculates according to already built model, gets probability this fruit belongs to each classification, achieving classification purpose. This process is also called prediction.

Advantages and Disadvantages of Naive Bayes Classification

• Advantages:

1. Simple algorithm logic, easy to implement

2. Small time and space overhead during classification

• Disadvantages:

Theoretically, Naive Bayes model has smallest error rate compared to other classification methods. But actually this is not always the case, this is because Naive Bayes model assumes attributes are independent of each other, this assumption is often not true in practical applications. When number of attributes is large or correlation between attributes is large, classification effect is poor.

Article Link: