In simple terms, Bayesian inference is simply updating your beliefs after considering new evidence.
In Bayesian world probability is measure of believability in an event, that is, how confident we are in an event occurring, whereas in, Frequentist world (classical version of statistics), assume that probability is the long-run frequency of events.
For example, the probability of plane accidents under a frequentist philosophy is interpreted as the long-term frequency of plane accidents. This makes logical sense for many probabilities of events, but becomes more difficult to understand when events have no long-term frequency of occurrences.
Bayesians, on the other hand, have a more intuitive approach. Bayesians interpret a probability as measure of belief, or confidence, of an event occurring. An individual who assigns a belief of 0 to an event has no confidence that the event will occur; conversely, assigning a belief of 1 implies that the individual is absolutely certain of an event occurring. Beliefs between 0 and 1 allow for weightings of other outcomes.
For eg:
- Your code either has a bug in it or not, but we do not know for certain which is true, though we have a belief about the presence or absence of a bug.
- A medical patient is exhibiting symptoms x, y and z. There are a number of diseases that could be causing all of them, but only a single disease is present. A doctor has beliefs about which disease, but a second doctor may have slightly different beliefs.
- We denote our belief about event A as P(A). We call this quantity the prior probability.
- After we see evidence, we update our beliefs. We denote our updated belief as P(A|X), interpreted as the probability of A given the evidence X.
- We call the updated belief the posterior probability so as to contrast it with the prior probability.