Blog

MAT 5170 Probability Theory I – Lecture Notes
October 23, 2020
1 Probability Measures
A quick review for sets and events
denotes the empty set.
A B means A is a subset of B. That is, all elements in A also belong
to B. Typically, B is a bigger set. For example, if A = {1, 2, 3} and
B = {1, 2, 3, 4}, then A B. But if C = {3, 4, 5}, then C 6 B.
A ∪ B is the union of two sets. This is the set of all outcomes that
are either in A or in B, or in both. For example, if A = {1, 2, 3} and
B = {3, 4, 5}, then A ∪ B = {1, 2, 3, 4, 5}. Note that 3 is not counted
twice. If C = {1, 2, 3, 4}, then A ∪ C = {1, 2, 3, 4} = C.
A∩B is the intersection of two sets. This is the set of all outcomes that
belong to both A and B. For example, if A = {1, 2, 3} and B = {3, 4, 5},
then A ∪B = {3}. If C = {4, 5}, then A ∪ C = .
A B is the difference of two sets. We remove from B all elements
that are also in A. For example, if A = {1, 2, 3} and B = {3, 4, 5}, then
B A = {4, 5}.
A′ is the complement of A. That is, A′ is the set of all elements in S
that are not in A.
Commutative laws: A ∩B = B ∩A; A ∪B = B ∪A.
Associative laws:
(A ∪B) ∪ C = A ∪ (B ∪ C) = A ∪B ∪ C
(A ∩B) ∩ C = A ∩ (B ∩ C) = A ∩B ∩ C
Distributive laws:
A ∩ (B ∪ C) = (A ∩B) ∪ (A ∩ C)
A ∪ (B ∩ C) = (A ∪B) ∩ (A ∪ C)
1
De Morgan’s laws:
(A1 ∩ · · · ∩Ak)′ = A′1 ∪ · · · ∪A′k
(A1 ∪ · · · ∪Ak)′ = A′1 ∩ · · · ∩A′k
1.1 Spaces and Probabilities
In the probability theory denotes a probability space. It could be as simple
as {0, 1} in case of one coin tossing, or as complicated as a space of all contin-
uous functions defined on [0, 1].
A subset of is called an event and an element ω of is a sample point.
This is a definition that you learned in elementary probability class:
Definition 1. Probability is a real-valued set function P that assigns, to each
event A a number P(A), called the probability of the event A, such that the
following axioms of probability are satisfied:
1. 0 ≤ P(A) ≤ 1 for any event A
2. P( ) = 0, P( ) = 1
3. If A1, A2, ….Ak are mutually exclusive, then
P(A1 ∪A2 ∪ · · · ∪Ak) = P(A1) + P(A2) + · · ·+ P(Ak) .
You will learn in a moment that this definition is not 100% correct ….. Our
goal is to assign probabilities to as many events as possible.
For example, if is finite then we can assign a uniform probability measure:
P({ω}) = 1| | . But this is not only one choice. For example if you roll a fair dice
then = {1, 2, 3, 4, 5, 6} and then P({1}) = · · · = P({6}) = 1/6. But if the dice
is loaded as follows: 1 side with ”1”, 2 sides with ”2”, 3 sides with ”3”, then
P({2}) = 2/6, P({3}) = 3/6. In other words, for the same sample space we may
have different probabilities.
If = [0, 1] and A = (a, b), 0 < a < b < 1, then we can assign P(A) = b a . Using the third axiom of probability, we can extend it to all finite unions of intervals (see Theorem 11). What about more complicated events For example, what is P(Q), where Q is the set of all rationals in [0, 1] In fact, we cannot assign a probability to an arbitrary event, rather we need to consider some special classes of sets like fields or σ-fields. 1.2 Classes of sets A ”nice” class of sets should be closed under the formation of countable unions and intersections. 2 Definition 2. A class F of subsets of is called a field if: ∈ F ; If A ∈ F then Ac = A ∈ F ; If A,B ∈ F then A ∪B ∈ F . (”closed under union of two sets”) Note that De Morgan’s laws immediately imply that A ∩ B ∈ F . Indeed, A∩B = (Ac∪Bc)c. Also, the third property implies that a field is closed under any finite union. Definition 3. A class F of subsets of is called a σ-field if: ∈ F ; If A ∈ F then Ac = A ∈ F ; If A1, A2, . . . ∈ F then A1 ∪ A2 ∪ · · · ∈ F . (”closed under countable unions”) Example 4. Let F consist of the finite and the cofinite sets (A being cofinite if Ac is finite). For example, if = [0, 1] and A = [0, 1) then A is cofinite since Ac = {1}. Then F is a field. If is finite then F is also a σ-field. If is infinite then F may not be a σ-field. For example, let = {ω1, ω2, . . .}. Take A = {ω2, ω4, . . .}. Then A 6∈ F , but A is a count- able union of singletons {ω2i}, i = 1, 2, . . .. Let A be a (”small” and ”easy”) class of subsets of . For example, A can be a class of all subintervals of [0, 1]. Note that this class is not closed under finite unions since e.g. for a1 < a2 < a3 < a4, [a1, a2]∪ [a3, a4] is not an interval. So, it is not a field of a σ-field. Since we want to consider all sets in A and at the same time have a ”nice” structure of a σ-field, we will consider a σ-field generated by the class A, denoted by σ(A). This is the smallest σ-field that contains A. In general, σ(A) has the following properties: A σ(A); σ(A) is a σ-field; If A G and G is σ-field, then σ(A) G. Example 5. If = {1, 2, 3}, then σ({1}) = σ({2, 3}) = { , {1}, {2, 3}, }. Indeed, denote A = {1} and A′ = { , {1}, {2, 3}, }. Check (this is easy!) that A is not a field (since the complement of {1} does not belong to A), but A′ is a field. It is also a σ-field, since the space is finite. Of course, A A′. Now, is A′ the smallest field that contains A We list all the possible classes that contain A = {1}: 3 { , {1}, } - it is not a field, since it is not closed under taking comple- ments; {{1}, } - it is not a field, since it is not closed under taking complements; { , {1}} - it is not a field, since it is not closed under taking complements; { , {1}, {1, 2}, } - it is not a field, since it is not closed under taking complements; you can continue listing all the possibilities and you will see that most of them will not be fields. Now, I will list all the fields that include {1}: – A′ = { , {1}, {2, 3}, }; – { , {1}, {1, 2}, {3}, {2, 3}, } - it is bigger than A′; – { , {1}, {1, 3}, {2}, {2, 3}, } - it is bigger than A′; – { , {1}, {2}, {3}, {1, 2}, {1, 3}, {2, 3}, } - it is bigger than A′; Thus, we proved thatA′ is the smallest field that contains {1}, henceA′ = σ(A). Of course, this is not the most effective way of proving this. Example 6. If F is a σ-field, then σ(F) = F . Example 7. Let = [0, 1] and I be a class of all subintervals (a, b] of (0, 1]. Then B = σ(I) is called a Borel σ-field. It is a class of all countable unions and intersections of intervals. Its elements are called Borel sets. A set of rationals Q is a Borel set. Indeed, each rational q can be obtained as q = ∞ i=1(q 1/i, q]. The sets (q 1/i, q] ∈ I. Note that {q} 6∈ I, but {q} ∈ B. Since there are countably many rationals, Q ∈ B. 1.3 Probability Measures Definition 8. A set function P on a field F is called probability if 1. 0 ≤ P(A) ≤ 1 for any event A ∈ A 2. P( ) = 0, P( ) = 1 3. If A1, A2, . . . ,∈ F are mutually e

xclusive (disjoint) and
∞
i=1Ai ∈ F , then
P(A1 ∪A2 ∪ · · · ) = P(A1) + P(A2) + · · · .
From the axioms of probability we can derive further properties. Let A
B ∈ F . Then B = A ∪ (B A) is a union of disjoint sets. Thus, P(B) =
P(A) + P(B A) and hence we conclude that P is monotone:
4
P(A) ≤ P(B) , A B .
Another property is
P(A ∪B) = P(A) + P(B) P(A ∩B) .
Further, let B1 = A1 and B2 = A2∩Ac1. Then the events B1, B2 are disjoint
and B1 ∪B2 = A1 ∪A2. Thus,
P(A1 ∪A2) = P(B1 ∪B2) = P(B1) + P(B2)
= P(A1) + P(A2 ∩Ac1) ≤ P(A1) + P(A2) .
In the last step we sued the fact that A2 ∩Ac1 A2.
This can be generalized to obtain finite subadditivity:
P
(
n
i=1
Ai
)
≤
n∑
i=1
P(Ai) , Ai ∈ F .
Definition 9. Let F be a σ-field. A triple ( ,F ,P) is called a probability
space.
Theorem 10. Let F be a field.
(Continuity from below) If A,An ∈ F , n ≥ 1 and A1 A2 · · · A,
then
lim
n→∞P(An) = P(A) .
(Continuity from above) If A,An ∈ F , n ≥ 1 and A · · ·A2 A1, then
lim
n→∞P(An) = P(A) .
Proof. Set B1 = A1, Bk = Ak Ak 1. Then Bk are disjoint, A =
∞
k=1Bk
and An =
n
k=1Bk and An, n ≥ 1 is an increasing sequence. By countable
additivity:
P(A) = P
( ∞
k=1
Bk
)
=
∞∑
k=1
P(Bk)
= lim
n→∞
n∑
k=1
P(Bk) = lim
n→∞P
(
n
k=1
Bk
)
= lim
n→∞P(An) .
5
1.4 Lebesque measure
Let = [0, 1]. Let I be a class of subintervals (a, b] of (0, 1]. Let B0 be a class
that consists of all finite union of disjoint intervals. Note that B0 is a field. For
I = (a, b] its Lebesque measure is λ(I) = b a.
Theorem 11. The Lebesgue measure is a probability measure on B0.
Proof. See the textbook.
Example 12. Assume that we select a value from the interval (0, 1] at ”ran-
dom”. What is the probability that the selected number is rational
To each interval A = (a, b] we assigned P(A) = b a. Thus, P((q 1/i, q]) =
1/i. Then
P({q}) = P
( ∞
i=1
(q 1/i, q]
)
= lim
i→∞
P((q 1/i, q]) = lim
i→∞
(1/i) = 0 .
On the other hand, Q is a countable union thus P(Q) = 0. 1
2 Existence of Probability Measures
Some facts about sets and classes of sets. Let F and G be two classes of
sets. If G F then any set that belongs to G also belongs to F : A ∈ G A ∈ F .
On the other hand, if A ∈ G and B A (that is, B is a subset of set A) then
we cannot conclude that B ∈ G. Indeed, let G be a class of subintervals (a, b] of
(0, 10]. Then B = {1/2} is included in many of the intervals, but B 6∈ G.
2.1 Existence
The main result of today’s lecture is the following theorem:
Theorem 13. A probability measure on a field has a unique extension to the
generated σ-field.
The meaning of this theorem is the following. Let F0 be a field and let
F = σ(F0). Let P be a probability measure on F0. We are allowed to write
P(A) for A ∈ F0, but not for A ∈ F . Then there exists (only one) probability
measure P on F such that P(A) = P (A) for all A ∈ F0. Usually the extension
P of P is denoted just by P.
1Note: At this moment, thanks to Theorem 11, we can compute the Lebesgue measure
on finite unions and intersections of subintervals. We do not know if we can compute the
measure on countable unions and intersections. This will cam later. Thus, at this point, the
first displayed equation is not completely justified, since
∞
i=1(q 1/i, q] 6∈ B0.
6
Example 14. Let = [0, 1]. Let I be a class of subintervals I = (a, b] of (0, 1].
It is easy to define the Lebesgue measure on I by λ(I) = b a. This is easy. Let
B0 be a field that consists of all finite union of disjoint intervals. By Theorem
10 from Slides set A, the Lebesgue measure is also a probability measure on B0.
This is still easy. Our main theorem extends the Lebesque measure to all Borel
sets of (0, 1]. In particular, we are allowed to write λ(Q) (which happens to be
zero as we learned in Slides set A).
Let P be the probability measure on F0. For A ∈ define
P (A) = inf
∑
n
P(An) , (1)
where the infimum extends over all finite and infinite sequences A1, A2, . . . , such
that A nAn and An ∈ F0. Note that for A ∈ such that A 6∈ F0 we are
not allowed to write P(A). However, if A ∈ F0 then we will show that
P (A) = P(A) . (2)
The measure P is called the outer measure of P.
The measure P has the following properties:
1. P ( ) = 0;
2. P (A) ≥ 0 for all A;
3. P is monotone, that is, if A B, then P (A) ≤ P (B);
4. P is countably subadditive, P (∪nAn) ≤
∑
n P (An).
However, we do not know if P is a probability measure.
The inner measure P is defined as
P (A) = 1 P (A′) .
Both P and P are candidates for an extension of P.
If P and P agree then we have
P (A) + P (A′) = 1 .
We will ask for more:
P (A ∩ E) + P (A′ ∩ E) = P (E) (3)
for all sets E ∈ . Note that by subadditivity, (3) is equivalent to
P (A ∩ E) + P (A′ ∩ E) ≤ P (E) . (4)
We do not know at this point how many (if any at all!!!) sets A fulfill this. Let
M be the class of sets A that fulfill (3). We call such sets P -measurable.
7
Lemma 15. The class M is a field.
Proof. First, ∈M, since
P ( ∩ E) + P ( ′ ∩ E) = P (E) .
Next, M closed under complementation: if A ∈ M, then A′ ∈ M. This is
obvious. Furthermore, if A,B ∈ M, then A ∩ B ∈ M. For this one need to
show that if
P (A ∩ F ) + P (A′ ∩ F ) = P (F ) , P (B ∩ E) + P (B′ ∩ E) = P (E) ,
for all sets E,F , then
P ((A ∩B) ∩ E) + P ((A ∩B)′ ∩ E) ≤ P (E) .
We have
P (E) = P (B ∩ E︸ ︷︷ ︸
=F
) + P (B′ ∩ E︸ ︷︷ ︸
=F
)
= {P (A ∩B ∩ E) + P (A′ ∩B ∩ E)}
+ {P (A ∩B′ ∩ E) + P (A′ ∩B′ ∩ E)}
≥ P (A ∩B ∩ E) + P (((A′ ∩B) ∪ (A ∩B′) ∪ (A′ ∩B′)) ∩ E) .
In the first equation we used the property (3) applied to B, then in the next
equality the property of (3) applied to A twice. Then we used subadditivity of
P . Now,
((A′ ∩B) ∪ (A ∩B′) ∪ (A′ ∩B′)) = (A ∩B)′ ,
thus
P (E) ≥ P (A ∩B ∩ E) + P ((A ∩B)′ ∩ E) .
Hence, A ∩B ∈M.
Lemma 16. If A1, A2, . . . is a finite or infinite sequence of disjoint M-sets,
then for each E ∈ ,
P
(
E ∩

k
Ak
)
=
∑
k
P (E ∩Ak) . (5)
Proof. We will do a proof of A1, A2 only. If A1 ∪ A2 = , then A2 = A′1 and
hence (5) is just (3). If A1 ∪A2 , then write
E ∩ (A1 ∪A2)︸ ︷︷ ︸
=F
=
A1 ∩ (E ∩ (A1 ∪A2))︸ ︷︷ ︸
=F
∪
A′1 ∩ (E ∩ (A1 ∪A2))︸ ︷︷ ︸
=F

and use (20) with E replaced by F .
8
Lemma 17. The class M is a σ-field.
Proof. Note that we only need to prove thatM is closed under countable unions.
I will not prove it, you may consult the textbook.
Combining Lemma 17 with (5) (taking E = ) gives us the countable addi-
tivity on M:
P
(
k
Ak
)
=
∑
k
P (Ak) , (6)
where Ak are disjoint and belong to M.
Lemma 18. If P is defined by (3), then F0 M.
Proof. Let A ∈ F0 and E ∈ . Let > 0. Choose sets An ∈ F0 such that
E nAn and∑n P(An) ≤ P (E)+. The sets Bn = A∩An and Cn = A∩A′n
belong to F0, since it is field. Also, E ∩A