MAST30020
Probability for Inference
Lecture Slides
Written by K. Borovkov, last time modified on 1 January 2021 (or even later).
0-0
1. Probability Spaces
Random experiment (RE): You should be familiar with the concept, from
the 2nd year level probability course you have done. Anyway, that’s a
(somewhat vaguely defined) concept representing real-life phenomena that:
have a “mass character” (i.e. could be repeated many-many times, at least
in theory),
don’t display “deterministic regularity” (i.e. the outcome of any given trial
is uncertain, to the best of our prior knowledge), but
possess what’s called “statistical regularity”: the relative frequencies
(nA/n) of events one can observe in the experiment stabilize around some
values ∈ [0, 1] as the # of (“independent”) repetitions of the experiment
grows.
It is that last dot-point that makes Probability Theory possible.
Examples (Ex’s): coin tossing; dice rolling; gender of newborn babies etc.
1
Need to specify the outcome of our RE: how
In mathematics, we have sets = collections of objects.
Sample space is the set of all possible outcomes.
Ex. Coin tossing. H/T What if the coin doesn’t fall on its side What if
where it landed also matters for us Several different ’s are possible (and
can be used under different circumstances).
So outcomes are elements ω ∈ (a.k.a. elementary events, sample
points, realizations).
Events: these are subsets A for which probability is defined.
NB: in non-trivial cases, there are subsets A for which probability
CANNOT be defined. For example, one can partition = [0, 1] into
countably many “identical” sets Ai (they can be obtained from each other
by translations): =
∑
i≥1Ai. Assuming uniform probability distribution
on [0, 1], what would be the probability of A1
2
Sample spaces
Ex. Finitely many possible outcomes only need a finite . Just list all
the outcomes: = {H,T} ≡ {0, 1} for the experiment of tossing a coin
once; = {1, 2, . . . , 6} for rolling a die once.
Now consider an RE where we toss a coin
roll a die
n = 3 times in a row.
Product spaces come handy. For two sets A and B, their (Cartesian)
product is defined by
A×B := {(a, b) : a ∈ A, b ∈ B}
(the set of all pairs (a, b) s.t. a ∈ A and b ∈ B), and we put
An := A× · · · ×A︸ ︷︷ ︸
n copies of A
= {(a1, a2, . . . , an) : aj ∈ A, j = 1, 2, . . . , n}.
RE with a sample space 0 is replicated n times the sample space of the
composite experiment is = n0 , with ω = (ω1, ω2, . . . , ωn), ωj = outcome
of the jth replication of the basic (sub-)RE with the sample space 0.
3
Ex. Keep tossing a coin till H shows up. Will a finite suffice
There are countably many possible outcomes that can be represented by
points from the set N = {1, 2, . . . } (ω = j if we observed j 1 tails
followed by heads).a
Ex. You have a date with your girl/boyfriend. She/he shows up
ω ∈ R+ = [0,∞) minutes late. Here we need = R+.
One often uses the whole real line R or a subinterval thereof. For a
composite RE consisting of n dates, one can use
= Rn := {(x1, x2, . . . , xn) : xj ∈ R, j = 1, 2, . . . , n}.
aAny set A s.t. there exists a 1–1 mapping f : A→ N is said to be countable (denumerable).
Thus, Z = {. . . , 1, 0, 1, . . . } is countable. So is N2, but not [0, 1].
4
If there can be any number of dates (at least, in theory: imagine that if
your girl/boyfriend was less than 20′ late for a given date, you decide to
meet again), we’ll need
RN := {(x1, x2, . . . ) : xj ∈ R, j ∈ N},
the set of real sequences, i.e. all real-valued functions on N.
[Other functional spaces are used, too, e.g. C[0, 1] for the so-called
Brownian motion process.]
This models a situation where the basic RE can be repeated infinitely
many times. Likewise, in the coin tossing RE in which you are tossing a
coin till you get
H for the first time, one can use = {T,H}N.
BTW: Note that {T,H}N is uncountable (why ). In the last example, you
could actually use a countable space of outcomes — which one
5
Probabilities will be assigned to sets of outcomes, i.e. subsets of .
For a given RE, can one assign probabilities to all subsets of
If is countable, the answer is YES.
In the general case, the answer is NO.
It is impossible in the basic RE of choosing a point at random from [0, 1].
So we MUST restrain ourselves and consider only some of the subsets
of , chosen in such a way that there will be no problems with assigning
probabilities to them — and these subsets will be called “events”.
Now how to choose them One needs to be able to manipulate events,
and, quite naturally, such (admissible!) manipulations should be
producing events. Let’s look at the ways to manipulate events.
6
Ex. Choose a student from a large class. Want the events that the student:
1) is NOT smoking;
2) is a female AND more than 55 y.o.;
3) was born in Australia OR New Zealand;
4) was born in Australia AND is NOT smoking.
These can be expressed in terms of simpler events using set operations:
1) Let := population of all students in class, A := sub-population of
smokers. Take the complement Ac := {ω ∈ : ω 6∈ A}.
2) If B := sub-population of female students, C := those who are > 55 y.o.,
then take the intersection B ∩ C = {ω ∈ : ω ∈ B and ω ∈ C}.
3) If D := students born in Australia, E := students born in New Zealand,
then take the union D ∪ E = {ω ∈ : ω ∈ D or ω ∈ E}.
4) This will be the set difference D A := D ∩Ac.
7
Note that D∩E = , i.e. the events are disjoint (no common outcomes). Note
also that, in the case of disjoint sets, one often uses D + E to denote D ∪ E.
In fact, it is often more convenient to work with functions rather than sets.
How to “replace” sets with functions Use indicators (indicator functions):
1A(ω) :=
1, ω ∈ A,0, ω 6∈ A.
To the (main) set operations on events, there correspond the following
operations on indicator functions (draw diagrams & check!!):
1Ac = 1 1A, 1A∩B = 1A1B , 1A∪B = max{1A,1B}.
For the symmetric set difference A4B := A B +B A ≡ (A ∪B) (A ∩B),
1A4B = |1A 1B |.
8
Ex. Express the following events:
Neither A nor B occurred [i.e. 1A + 1B = 0].
I Ac ∩Bc = (A ∪B)c, de Morgan law.
A and B occurred, but C didn’t.
I A ∩B ∩ Cc ≡ (A ∩B) C.
Only one of A1, A2, A3 occurred [i.e.
∑3
j=1 1Aj = 1].
I A1 (A2 ∪A3) +A2 (A1 ∪A3) +A3 (A1 ∪A2).
Exactly two out of A1, . . . , A5 occurred [i.e.
∑5
j=1 1Aj = 2].
I
∑
i (Ai ∩Aj)
k 6=i,j
Ak
.
9
As we said earlier, want to manipulate events, need the resulting sets still be
events. Hence the requirement: the class of events on must be closed under
the main set operations, i.e. the complement of an event should still be an
event, and the union of events must still be an event (the same for
intersections, but this is automatic from de Morgan laws, see below).
To make things work, need a bit more: namely, that we are allowed to take
countable unions (and intersections!). In mathematics, when countable
infinity is allowed/involved, one often uses σ to indicate that.
Def. A family F of subsets of is said to be a σ-algebra on if
(A.1) ∈ F ,
(A.2) A ∈ F Ac ∈ F ,
(A.3) A1, A2, . . . ∈ F
∞
n=1An ∈ F .
In words: the family is closed under complementation and countable union and
intersection. Why the latter De Morgan + (A.2) + (A.3) + (A.2): ∞
n=1An =
[
(
∞
n=1An)
c]c
= [
∞
n=1A
c
n]
c
= [
∞
n=1Bn]
c
, Bn := A
c
n ∈ F .
10
Ex. (continued)
Infinitely many of the events A1, A2, . . . occurred [i.e.
∑∞
j=1 1Aj =∞].
Is this actually an event [Denoted: An, i.o.]
Why For instance: will a random walk S0, S1, S2, . . . on Z visit 0 infinitely
many times Let An := {Sn = 0}.
I