Introduction to Probability

A Beginner's Guide to Probability Theory

Level: Beginner → Intermediate
Prerequisites: Basic algebra and arithmetic
Applications: Mathematics competitions, statistics, computer science, and machine learning

---

1. What Is Probability?

Probability is a branch of mathematics that studies uncertainty and random events.

The probability of an event is a number between 0 and 1.

• : Event is impossible.
• : Event is certain.
• : Event has a 50% chance of occurring.

For example, when tossing a fair coin, there are two possible outcomes:

Here:

• : Heads
• : Tails
• : Sample space

Because the coin is fair:

1.1 Sample Space

The sample space is the set of all possible outcomes of a random experiment.

It is usually denoted by or .

Example 1: Rolling a Die

When rolling a standard six-sided die:

There are six equally likely outcomes.

The probability of rolling a 4 is:

Example 2: Two Coin Tosses

When tossing two coins:

There are four equally likely outcomes.

The probability of obtaining two heads is:

The probability of obtaining exactly one head is:

Important: Order matters when identifying outcomes.

and are distinct outcomes.

---

2. The Fundamental Probability Formula

When all elementary outcomes are equally likely:

Equivalently:

where:

• is the number of outcomes belonging to event .
• is the total number of outcomes.

This formula assumes that all elementary outcomes are equally likely.

Example 3: Even Numbers

A fair die is rolled.

What is the probability of obtaining an even number?

The sample space is:

The favorable outcomes are:

Therefore:

Example 4: Two Dice

Two fair six-sided dice are rolled.

What is the probability that their sum equals 7?

There are:

equally likely ordered outcomes.

The favorable outcomes are:

There are six favorable outcomes.

Therefore:

Notice that counting ordered pairs is essential.

---

3. Fundamental Rules of Probability

3.1 Complement Rule

The complement of event , denoted by , means that event does not occur.

The complement rule is:

Example 5: At Least One Head

A fair coin is tossed three times.

What is the probability of obtaining at least one head?

Instead of counting every case containing a head, calculate the probability of obtaining no heads.

The only outcome containing no heads is:

Its probability is:

Therefore:

Competition tip: For questions involving "at least one," the complement rule is often the simplest approach.

---

3.2 Addition Rule

Suppose we have two events, and .

The probability that at least one occurs is:

Here:

• : Event or event , or both.
• : Both and occur.

We subtract because it is counted twice when adding and .

Example 6: Even or Greater Than 4

A fair die is rolled.

Let:

represent rolling an even number.

Let:

represent rolling a number greater than 4.

Then:

We have:

Therefore:

Mutually Exclusive Events

Two events are mutually exclusive if they cannot occur simultaneously.

Thus:

and:

The addition rule becomes:

Example: A single die cannot simultaneously show 2 and 5.

Therefore:

---

4. Conditional Probability

Conditional probability describes the probability of an event occurring given that another event has already occurred.

It is written as:

Read this as:

The probability of A given B.

The formula is:

provided that .

Example 7: Conditional Probability with a Die

A fair die is rolled.

You are told that the result is greater than 3.

What is the probability that the result is even?

Initially:

However, we now know that:

Among these three possibilities, two are even:

Therefore:

Notice that the relevant sample space has changed.

Instead of considering six outcomes, we consider only three.

---

4.1 Multiplication Rule

Rearranging the conditional probability formula:

gives:

Equivalently:

These formulas are especially useful for sequential events.

Example 8: Drawing Balls Without Replacement

A bag contains:

• 3 red balls
• 2 blue balls

Two balls are randomly drawn without replacement.

What is the probability that both balls are red?

The probability that the first ball is red is:

After removing one red ball, the bag contains:

• 2 red balls
• 2 blue balls

Therefore:

By the multiplication rule:

Important: When sampling without replacement, the probabilities typically change after each selection.

---

5. Independent Events

Two events and are independent if knowing whether one occurred does not change the probability of the other.

Mathematically:

when .

Equivalently:

Example 9: Two Independent Coin Tosses

A fair coin is tossed twice.

What is the probability of obtaining two heads?

The first toss does not influence the second.

Therefore:

By independence:

For independent coin tosses:

Independent vs. Mutually Exclusive

These concepts are different.

For two events with positive probabilities, mutual exclusivity implies they are not independent.

For example, rolling a 2 and rolling a 5 on the same die are mutually exclusive, but not independent.

---

6. Counting Techniques

Many probability problems require counting the number of possible outcomes.

Three particularly important concepts are:

1. Multiplication principle
2. Permutations
3. Combinations

6.1 Multiplication Principle

Suppose a task consists of two steps.

• Step 1 has possibilities.
• Step 2 has possibilities for each choice in Step 1.

Then the total number of outcomes is:

Example 10: Passwords

A password consists of three decimal digits.

Each position can contain 0 through 9.

If repetition is allowed:

possible passwords exist.

If repetition is forbidden:

possible passwords exist.

---

6.2 Factorials

The factorial of a positive integer is:

Examples:

By definition:

Factorials are important for permutations and combinations.

---

6.3 Permutations

A permutation is an arrangement where order matters.

The number of ways to arrange distinct objects chosen from distinct objects is:

Example 11: Selecting a President and Vice President

There are five students.

We want to select:

• One president
• One vice president

Because the positions are different, order matters.

There are five choices for president and four remaining choices for vice president.

Therefore:

---

6.4 Combinations

A combination is a selection where order does not matter.

The number of ways to choose objects from distinct objects is:

This is also written as:

Example 12: Selecting a Committee

There are five students.

We want to select a committee containing two students.

The order of selection does not matter.

Therefore:

Key Difference

Remember:

• Order matters → Permutations
• Order does not matter → Combinations

---

7. Probability Using Combinations

Combinations become particularly useful when objects are randomly selected without replacement.

Example 13: Choosing Two Red Balls

A bag contains:

• 4 red balls
• 6 blue balls

Three balls are chosen simultaneously.

What is the probability that exactly two are red?

Step 1: Count All Possible Selections

There are 10 balls, and we choose 3.

Step 2: Count Favorable Selections

We must choose:

• 2 red balls from 4
• 1 blue ball from 6

The number of favorable selections is:

Step 3: Calculate Probability

This is an example of the hypergeometric probability model.

In general, if a population contains:

• objects
• objects of a particular type

and we sample objects without replacement, then:

where is the number of selected objects of the desired type.

---

8. Binomial Probability

The binomial distribution models the number of successes in repeated independent trials.

It applies when:

1. There is a fixed number of trials.
2. Every trial has two possible outcomes: success or failure.
3. The trials are independent.
4. The success probability stays constant.

If counts the number of successes, then:

The binomial probability formula is:

where:

• : Total number of trials
• : Number of successes
• : Probability of success in one trial

Example 14: Exactly Three Heads

A fair coin is tossed five times.

What is the probability of obtaining exactly three heads?

Here:

Using the formula:

We know:

Therefore:

Example 15: At Least Two Heads

A fair coin is tossed three times.

What is the probability of obtaining at least two heads?

Let count the number of heads.

We need:

First:

Second:

Therefore:

---

9. Bayes' Theorem

Bayes' theorem is used to update probabilities when new information becomes available.

It follows directly from the definition of conditional probability.

We have:

and:

Substituting:

This is Bayes' theorem.

Example 16: Choosing a Box

There are two boxes.

Box A:

• 2 red balls
• 1 blue ball

Box B:

• 1 red ball
• 2 blue balls

You randomly select one of the two boxes with equal probability.

Then you randomly draw one ball.

The ball is red.

What is the probability that it came from Box A?

Step 1: Identify Probabilities

The probability of drawing a red ball from Box A is:

The probability of drawing a red ball from Box B is:

Step 2: Calculate the Probability of Red

By the law of total probability:

Therefore:

Step 3: Apply Bayes' Theorem

Although Box A was initially selected with probability , observing a red ball increases the probability that Box A was selected to .

---

10. Expected Value

Expected value is the probability-weighted average of a random variable's possible values.

For a discrete random variable :

where:

• is a possible value of .
• is its probability.

Example 17: Expected Value of a Die

A fair six-sided die is rolled.

Let be the number shown.

Each outcome has probability .

Thus:

Notice that 3.5 is not a possible result of a single die roll.

Expected value represents the long-run average rather than a necessarily attainable individual outcome.

---

10.1 Linearity of Expectation

A powerful property is:

This holds even when and are not independent, provided their expectations exist.

More generally:

Example 18: Ten Coin Tosses

A fair coin is tossed ten times.

Let be the number of heads.

Define an indicator variable for each toss:

Then:

Since:

linearity of expectation gives:

We can calculate the expected number of heads without enumerating all possible sequences.

---

11. Variance and Standard Deviation

Expected value describes the center of a distribution.

Variance describes how far values typically deviate from that center, using squared deviations.

The variance of is:

An equivalent formula is:

The standard deviation is:

Example 19: A Fair Coin

Let:

Then:

Since for both possible values:

Therefore:

And:

For a binomial random variable :

---

12. A Systematic Strategy for Probability Problems

When solving a probability problem, follow these steps.

Step 1: Define the Random Experiment

Identify what is happening.

Examples:

• Tossing coins
• Rolling dice
• Selecting objects
• Arranging people
• Randomly generating numbers

Step 2: Identify the Sample Space

Determine all possible outcomes.

Ask:

Are the outcomes equally likely?

If they are, favorable-over-total counting may be used.

Step 3: Define the Desired Event

Translate the question into mathematical language.

Examples:

"At least one success":

"Exactly three successes":

"At most two successes":

"Both events occur":

"At least one event occurs":

Step 4: Choose the Appropriate Technique

---

13. Practice Problems

Try solving these problems before looking at the answers.

Problem 1 — Basic Probability

A fair six-sided die is rolled.

Find the probability that the result is either 2, 3, or 5.

---

Problem 2 — Two Dice

Two fair dice are rolled.

What is the probability that their sum is 9?

---

Problem 3 — Complement Rule

A fair coin is tossed five times.

What is the probability of obtaining at least one head?

---

Problem 4 — Without Replacement

A bag contains 5 red balls and 3 blue balls.

Two balls are drawn without replacement.

What is the probability that both balls are blue?

---

Problem 5 — Combinations

A group contains 6 boys and 4 girls.

Three students are selected uniformly at random.

What is the probability that exactly two are girls?

---

Problem 6 — Binomial Distribution

A fair coin is tossed six times.

What is the probability of obtaining exactly four heads?

---

Problem 7 — Conditional Probability

Two fair dice are rolled.

Given that their sum is 8, what is the probability that at least one die shows 3?

---

Problem 8 — Expected Value

A game has the following rules:

• You pay $2 to participate.
• A fair die is rolled.
• If the die shows 6, you receive $12.
• Otherwise, you receive nothing.

What is your expected net profit per game?

---

14. Practice Problem Solutions

Solution 1

There are three favorable outcomes among six:

Solution 2

The ordered pairs summing to 9 are:

There are four favorable outcomes among 36:

Solution 3

The probability of no heads is:

Therefore:

Solution 4

The probability of drawing two blue balls is:

Solution 5

Total selections:

Favorable selections:

Therefore:

Solution 6

Using the binomial probability formula:

Solution 7

Given that the sum is 8, the possible ordered pairs are:

There are five equally likely possibilities.

Two of them contain a 3:

Therefore:

Solution 8

If you roll a 6, your net profit is:

Otherwise, your net profit is:

Therefore:

The game is fair in the sense that its expected net profit is zero.

---

15. Formula Summary

Basic Probability

For equally likely outcomes.

Complement Rule

Addition Rule

Conditional Probability

Multiplication Rule

Independence

Permutations

Combinations

Binomial Probability

Bayes' Theorem

Expected Value

Variance

---

16. Further Reading

To develop a stronger foundation in probability, consider:

1. Introduction to Probability — Joseph K. Blitzstein and Jessica Hwang

   • Excellent for conceptual understanding and problem-solving.
   • Covers counting, conditional probability, random variables, and probability distributions.

2. Introduction to Probability — Dimitri P. Bertsekas and John N. Tsitsiklis

   • Mathematical introduction particularly useful for engineering and computer science.

3. Introduction to Counting & Probability — David Patrick (Art of Problem Solving)

   • Especially useful for AMC and AIME preparation.
   • Emphasizes combinatorial reasoning and competition problem-solving.

4. A First Course in Probability — Sheldon Ross

   • Standard undergraduate probability textbook.
   • Offers a more systematic mathematical treatment.

---

17. What to Learn Next

Once you understand these fundamentals, study the following topics:

Intermediate Probability

• Inclusion-exclusion principle
• Law of total probability
• Multinomial distributions
• Geometric probability
• Random variables and probability distributions
• Covariance and correlation
• Geometric and Poisson distributions

Advanced Competition Probability

• Indicator random variables
• Linearity of expectation
• Conditional expectation
• Recurrence relations in probability
• Symmetry arguments
• Random walks
• Probabilistic combinatorics

Probability for Computer Science and AI

• Joint and marginal distributions
• Conditional independence
• Markov chains
• Bayesian networks
• Monte Carlo methods
• Maximum likelihood estimation
• The law of large numbers
• The central limit theorem