Misplaced Pages

Revision as of 22:56, 4 September 2010

This article may require cleanup to meet Misplaced Pages's quality standards. No cleanup reason has been specified. Please help improve this article if you can. (September 2009) (Learn how and when to remove this message)

In probability theory, there exist several different notions of convergence of random variables. The convergence (in one of the senses presented below) of sequences of random variables to some limit random variable is an important concept in probability theory, and its applications to statistics and stochastic processes. For example, if the average of n uncorrelated random variables Y_i, i = 1, ..., n, all having the same finite mean and variance, is given by

${\displaystyle X_{n}={\frac {1}{n}}\sum _{i=1}^{n}Y_{i}\,,}$

then as n tends to infinity, X_n converges in probability (see below) to the common mean, μ, of the random variables Y_i. This result is known as the weak law of large numbers. Other forms of convergence are important in other useful theorems, including the central limit theorem.

Throughout the following, we assume that (X_n) is a sequence of random variables, and X is a random variable, and all of them are defined on the same probability space (Ω, F, P).

In mathematics, stochastic convergence formalizes the idea that a sequence of essentially random or unpredictable events sometimes is expected to settle into a pattern. The pattern may for instance be

• Convergence in the classical sense to a fixed value, perhaps itself coming from a random event
• An increasing similarity of outcomes to what a purely deterministic function would produce
• An increasing preference towards a certain outcome
• An increasing "aversion" against straying far away from a certain outcome

Some less obvious, more theoretical patterns could be

• That the probability distribution describing the next outcome may grow increasingly similar to a certain distribution
• That the series formed by calculating the expected value of the outcome's distance from a particular value may converge to 0
• That the variance of the random variable describing the next event grows smaller and smaller.

Convergence in distribution

Examples of convergence in distribution

Dice factory
Suppose a new dice factory has just been built. The first few dice come out quite biased, due to imperfections in the production process. The outcome from tossing any of them will follow a distribution markedly different from the desired uniform distribution. As the factory is improved, the dice become less and less loaded, and the outcomes from tossing a newly produced die will follow the uniform distribution more and more closely.
Tossing coins
Let Xn be the fraction of heads after tossing up an unbiased coin n times. Then X1 has the Bernoulli distribution with expected value μ = 0.5 and variance σ = 0.25. The subsequent random variables X2, X3, … will all be distributed binomially. As n grows larger, this distribution will gradually start to take shape more and more similar to the bell curve of the normal distribution. If we shift and rescale Xn’s appropriately, then ${\displaystyle \scriptstyle Z_{n}={\sqrt {n}}(X_{n}-\mu )/\sigma }$ will be converging in distribution to the standard normal, the result that follows from the celebrated central limit theorem.
Graphic example
Suppose { Xi } is an iid sequence of uniform U(−1,1) random variables. Let ${\displaystyle \scriptstyle Z_{n}={\scriptscriptstyle {\frac {1}{\sqrt {n}}}}\sum _{i=1}^{n}X_{i}}$ be their (normalized) sums. Then according to the central limit theorem, the distribution of Zn approaches the normal N(0, ⅓) distribution. This convergence is shown in the picture: as n grows larger, the shape of the pdf function gets closer and closer to the Gaussian curve.

With this mode of convergence, we increasingly expect to see the next outcome in a sequence of random experiments becoming better and better modeled by a given probability distribution.

Convergence in distribution is the weakest form of convergence, since it is implied by all other types of convergence mentioned in this article. However convergence in distribution is very frequently used in practice, most often it arises from application of the central limit theorem.

Definition

A sequence {X₁, X₂, …} of random variables is said to converge in distribution, or converge weakly, or converge in law to a random variable X if

${\displaystyle \lim _{n\to \infty }F_{n}(x)=F(x),}$

for every number x ∈ R at which F is continuous. Here F_n and F are the cumulative distribution functions of random variables X_n and X correspondingly.

The requirement that only the continuity points of F should be considered is essential. For example if X_n are distributed uniformly on intervals (0, 1⁄n), then this sequence converges in distribution to a degenerate X = 0 random variable. For this random variable F(0) = 1, even though F_n(0) = 0 for all n. Thus the convergence of cdfs fails at the point x = 0 where F is discontinuous.

Convergence in distribution may be denoted as

${\displaystyle {\begin{aligned}&X_{n}\ {\xrightarrow {d}}\ X,\ \ X_{n}\ {\xrightarrow {\mathcal {D}}}\ X,\ \ X_{n}\ {\xrightarrow {\mathcal {L}}}\ X,\ \ X_{n}\ {\xrightarrow {d}}\ {\mathcal {L}}_{X},\\&X_{n}\rightsquigarrow X,\ \ X_{n}\Rightarrow X,\ \ {\mathcal {L}}(X_{n})\to {\mathcal {L}}(X),\\\end{aligned}}}$

where ${\displaystyle \scriptstyle {\mathcal {L}}_{X}}$ is the law (probability distribution) of X. For example if X is standard normal we can write ${\displaystyle X_{n}\,{\xrightarrow {d}}\,{\mathcal {N}}(0,\,1)}$.

For random vectors {X₁, X₂, …} ⊂ R the convergence in distribution is defined similarly. We say that this sequence converges in distribution to a random k-vector X if

${\displaystyle \lim _{n\to \infty }\operatorname {Pr} (X_{n}\in A)=\operatorname {Pr} (X\in A)}$

for every A ⊂ R which is a continuity set of the measure generated by X.

The definition of convergence in distribution may be extended from random vectors to more complex random elements in arbitrary metric spaces, and even to the “random variables” which are not measurable — a situation which occurs for example in the study of empirical processes. This is the “weak convergence of laws without laws being defined” — except asymptotically.

In this case the term weak convergence is preferable (see weak convergence of measures), and we say that a sequence of random elements {X_n} converges weakly to X (denoted as X_n ⇒ X) if

${\displaystyle \operatorname {E} ^{*}h(X_{n})\to \operatorname {E} \,h(X)}$

for all continuous bounded functions h(·). Here E* denotes the outer expectation, that is the expectation of a “smallest measurable function g above h(X_n)”.

Properties

• Since F(a) = Pr(X ≤ a), the convergence in distribution means that the probability for X_n to be in a given range is approximately equal to the probability that the value of X is in that range, provided n is sufficiently large.
• In general, convergence in distribution does not imply that the sequence of corresponding probability density functions will also converge. As an example one may consider random variables with densities ƒ_n(x) = (1 − cos(2πnx))1_{x∈(0,1)}. These random variables converge in distribution to a uniform U(0, 1), whereas their densities do not converge at all.
• Portmanteau lemma provides several equivalent definitions of convergence in distribution. Although these definitions are less intuitive, they are used to prove a number of statistical theorems. The lemma states that {X_n} converges in distribution to X if and only if any of the following statements are true:
  • Eƒ(X_n) → Eƒ(X) for all bounded, continuous functions ƒ;
  • Eƒ(X_n) → Eƒ(X) for all bounded, Lipschitz functions ƒ;
  • limsup{ Eƒ(X_n) } ≤ Eƒ(X) for every upper semi-continuous function ƒ bounded from above;
  • liminf{ Eƒ(X_n) } ≥ Eƒ(X) for every lower semi-continuous function ƒ bounded from below;
  • limsup{ Pr(X_n ∈ C) } ≤ Pr(X ∈ C) for all closed sets C;
  • liminf{ Pr(X_n ∈ U) } ≥ Pr(X ∈ U) for all open sets U;
  • lim{ Pr(X_n ∈ A) } = Pr(X ∈ A) for all continuity sets A of random variable X.
• Continuous mapping theorem states that for a continuous function g(·), if the sequence {X_n} converges in distribution to X, then so does {g(X_n)} converge in distribution to g(X).
• Lévy’s continuity theorem: the sequence {X_n} converges in distribution to X if and only if the sequence of corresponding characteristic functions {φ_n} converges pointwise to the characteristic function φ of X, and φ(t) is continuous at t = 0.
• Convergence in distribution is metrizable by the Lévy–Prokhorov metric.
• A natural link to convergence in distribution is the Skorokhod's representation theorem.

Convergence in probability

The basic idea behind this type of convergence is that the probability of an “unusual” outcome becomes smaller and smaller as the sequence progresses.

The concept of convergence in probability is used very often in statistics. For example, an estimator is called consistent if it converges in probability to the quantity being estimated. Convergence in probability is also the type of convergence established by the weak law of large numbers.

Definition

A sequence {X_n} of random variables converges in probability towards X if for all ε > 0

${\displaystyle \lim _{n\to \infty }\operatorname {Pr} {\big (}|{\overline {X}}_{n}-X|\geq \varepsilon {\big )}=0.}$

Formally, pick any ε > 0 and any δ > 0. Let P_n be the probability that ${\displaystyle {\overline {X}}_{n}}$ is outside the ball of radius ε centered at X. Then for ${\displaystyle {\overline {X}}_{n}}$ to converge in probability to X there should exist a number N_δ such that for all n ≥ N_δ the probability P_n is less than δ.

Convergence in probability is denoted by adding the letter p over an arrow indicating convergence, or using the “plim” probability limit operator:

${\displaystyle {\overline {X}}_{n}\ {\xrightarrow {p}}\ X,\ \ {\overline {X}}_{n}\ {\xrightarrow {P}}\ X,\ \ {\underset {n\to \infty }{\operatorname {plim} }}\,{\overline {X}}_{n}=X.}$

For random elements {X_n} on a separable metric space (S, d), convergence in probability is defined similarly by

${\displaystyle \operatorname {Pr} {\big (}d({\overline {X}}_{n},X)\geq \varepsilon {\big )}\to 0,\quad \forall \varepsilon >0.}$

Properties

• Convergence in probability implies convergence in distribution.
• In the opposite direction, convergence in distribution implies convergence in probability only when the limiting random variable X is a constant.
• The continuous mapping theorem states that for every continuous function g(·), if  ${\displaystyle \scriptstyle X_{n}{\xrightarrow {p}}X}$, then also  ${\displaystyle \scriptstyle g(X_{n}){\xrightarrow {p}}g(X)}$.
• Convergence in probability defines a topology on the space of random variables over a fixed probability space. This topology is metrizable by the Ky Fan metric:

  ${\displaystyle d(X,Y)=\inf \!{\big \{}\varepsilon >0:\ \operatorname {Pr} {\big (}|X-Y|>\varepsilon {\big )}\leq \varepsilon {\big \}}.}$

Almost sure convergence

This is the type of stochastic convergence that is most similar to pointwise convergence known from elementary real analysis.

Definition

To say that the sequence X_n converges almost surely or almost everywhere or with probability 1 or strongly towards X means that

${\displaystyle \operatorname {Pr} \!\left(\lim _{n\to \infty }\!X_{n}=X\right)=1.}$

This means that the values of X_n approach the value of X, in the sense (see almost surely) that events for which X_n does not converge to X have probability 0. Using the probability space ${\displaystyle \scriptstyle (\Omega ,{\mathcal {F}},P)}$ and the concept of the random variable as a function from Ω to R, this is equivalent to the statement

${\displaystyle \operatorname {Pr} {\Big (}\omega \in \Omega :\lim _{n\to \infty }X_{n}(\omega )=X(\omega ){\Big )}=1.}$

Another, equivalent, way of defining almost sure convergence is as follows:

${\displaystyle \operatorname {Pr} {\Big (}\liminf {\big \{}\omega \in \Omega :|X_{n}(\omega )-X(\omega )|<\varepsilon {\big \}}{\Big )}=1\quad {\text{for all}}\quad \varepsilon >0.}$

Almost sure convergence is often denoted by adding the letters a.s. over an arrow indicating convergence:

${\displaystyle X_{n}\,{\xrightarrow {\mathrm {a.s.} }}\,X.}$

For generic random elements {X_n} on a metric space (S, d), convergence almost surely is defined similarly:

${\displaystyle \operatorname {Pr} {\Big (}\omega \in \Omega :\,d{\big (}X_{n}(\omega ),X(\omega ){\big )}\,{\underset {n\to \infty }{\longrightarrow }}\,0{\Big )}=1}$

Properties

• Almost sure convergence implies convergence in probability, and hence implies convergence in distribution. It is the notion of convergence used in the strong law of large numbers.
• The concept of almost sure convergence does not come from a topology. This means there is no topology on the space of random variables such that the almost surely convergent sequences are exactly the converging sequences with respect to that topology. In particular, there is no metric of almost sure convergence.

Sure convergence

To say that the sequence or random variables (X_n) defined over the same probability space (i.e., a random process) converges surely or everywhere or pointwise towards X means

${\displaystyle \lim _{n\rightarrow \infty }X_{n}(\omega )=X(\omega ),\,\,\forall \omega \in \Omega .}$

where ${\displaystyle \Omega }$ is the sample space of the underlying probability space over which the random variables are defined.

This is the notion of pointwise convergence of sequence functions extended to sequence of random variables. (Note that random variables themselves are functions).

${\displaystyle {\big \{}\omega \in \Omega \,|\,\lim _{n\to \infty }X_{n}(\omega )=X(\omega ){\big \}}=\Omega .}$

Sure convergence of a random variable implies all the other kinds of convergence stated above, but there is no payoff in probability theory by using sure convergence compared to using almost sure convergence. The difference between the two only exists on sets with probability zero. This is why the concept of sure convergence of random variables is very rarely used.

Convergence in mean

We say that the sequence X_n converges in the r-th mean' or in the L 'norm towards X, if r ≥ 1, E|X_n| < ∞ for all n, and

${\displaystyle \lim _{n\rightarrow \infty }\mathrm {E} \left(\left|X_{n}-X\right|^{r}\right)=0}$

where the operator E denotes the expected value. Convergence in rth mean tells us that the expectation of the r-th power of the difference between X_n and X converges to zero.

This type of convergence is often denoted by adding the letter L over an arrow indicating convergence:

${\displaystyle X_{n}\,{\xrightarrow {L^{r}}}\,X.}$

The most important cases of convergence in r-th mean are:

• When X_n converges in r-th mean to X for r = 1, we say that X_n converges in mean to X.
• When X_n converges in r-th mean to X for r = 2, we say that X_n converges in mean square to X. This is also sometimes referred to as convergence in mean, and is sometimes denoted

${\displaystyle {\underset {n\to \infty }{\operatorname {l.i.m.} }}X_{n}=X.\,\!}$

Convergence in the r-th mean, for r > 0, implies convergence in probability (by Markov's inequality), while if r > s ≥ 1, convergence in r-th mean implies convergence in s-th mean. Hence, convergence in mean square implies convergence in mean.

Convergence in rth-order mean

This is a rather "technical" mode of convergence. We essentially compute a sequence of real numbers, one number for each random variable, and check if this sequence is convergent in the ordinary sense.

Formal definition

If ${\displaystyle \scriptstyle \lim _{n\to \infty }E(|X_{n}-a|^{r})=0}$ for some real number a, then {X_n} converges in rth-order mean to a.

Commonly used notation: ${\displaystyle X_{n}{\stackrel {L_{r}}{\rightarrow }}a.}$

Properties

The chain of implications between the various notions of convergence are noted in their respective sections. They are, using the arrow notation:

${\displaystyle {\begin{matrix}{\xrightarrow {L^{s}}}&{\underset {s>r\geq 1}{\Rightarrow }}&{\xrightarrow {L^{r}}}&&\\&&\Downarrow &&\\{\xrightarrow {a.s.}}&\Rightarrow &{\xrightarrow {\ p\ }}&\Rightarrow &{\xrightarrow {\ d\ }}\end{matrix}}}$

These properties, together with a number of other special cases, are summarized in the following list:

• Convergence almost surely implies convergence in probability:

  ${\displaystyle X_{n}\ {\xrightarrow {as}}\ X\quad \Rightarrow \quad X_{n}\ {\xrightarrow {p}}\ X}$

• Convergence in probability implies convergence in distribution:

  ${\displaystyle X_{n}\ {\xrightarrow {p}}\ X\quad \Rightarrow \quad X_{n}\ {\xrightarrow {d}}\ X}$

• Convergence in r-th order mean implies convergence in probability:

  ${\displaystyle X_{n}\ {\xrightarrow {L^{r}}}\ X\quad \Rightarrow \quad X_{n}\ {\xrightarrow {p}}\ X}$

• Convergence in r-th order mean implies convergence in lower order mean, assuming that both orders are greater than one:

  ${\displaystyle X_{n}\ {\xrightarrow {L^{r}}}\ X\quad \Rightarrow \quad X_{n}\ {\xrightarrow {L^{s}}}\ X,}$ provided r ≥ s ≥ 1.

• If X_n converges in distribution to a constant c, then X_n converges in probability to c:

  ${\displaystyle X_{n}\ {\xrightarrow {d}}\ c\quad \Rightarrow \quad X_{n}\ {\xrightarrow {p}}\ c,}$ provided c is a constant.

• If X_n converges in distribution to X and the difference between X_n and Y_n converges in probability to zero, then Y_n also converges in distribution to X:

  ${\displaystyle X_{n}\ {\xrightarrow {d}}\ X,\ \ |X_{n}-Y_{n}|\ {\xrightarrow {p}}\ 0\ \quad \Rightarrow \quad Y_{n}\ {\xrightarrow {d}}\ X}$

• If X_n converges in distribution to X and Y_n converges in distribution to a constant c, then the joint vector (X_n, Y_n) converges in distribution to (X, c):

  ${\displaystyle X_{n}\ {\xrightarrow {d}}\ X,\ \ Y_{n}\ {\xrightarrow {d}}\ c\ \quad \Rightarrow \quad (X_{n},Y_{n})\ {\xrightarrow {d}}\ (X,c)}$ provided c is a constant.

  Note that the condition that Y_n converges to a constant is important, if it were to converge to a random variable Y then we wouldn’t be able to conclude that (X_n, Y_n) converges to (X, Y).
• If X_n converges in probability to X and Y_n converges in probability to Y, then the joint vector (X_n, Y_n) converges in probability to (X, Y):

  ${\displaystyle X_{n}\ {\xrightarrow {p}}\ X,\ \ Y_{n}\ {\xrightarrow {p}}\ Y\ \quad \Rightarrow \quad (X_{n},Y_{n})\ {\xrightarrow {p}}\ (X,Y)}$

• If X_n converges in probability to X, and if ${\displaystyle \mathbb {P} (|X_{n}|\leq b)=1}$ for all n and some b, then X_n converges in rth mean to X for all r ≥ 1. In other words, if X_n converges in probability to X and all random variables X_n are almost surely bounded above and below, then X_n converges to X also in any rth mean.

• Almost sure representation. Usually, convergence in distribution does not imply convergence almost surely. However for a given sequence {X_n} which converges in distribution to X₀ it is always possible to find a new probability space (Ω, F, P) and random variables {Y_n, n = 0,1,…} defined on it such that Y_n is equal in distribution to X_n for each n ≥ 0, and Y_n converges to Y₀ almost surely.

• If for all ε > 0,

${\displaystyle \sum _{n}\mathbb {P} \left(|X_{n}-X|>\varepsilon \right)<\infty ,}$

then we say that X_n converges almost completely, or almost in probability towards X. When X_n converges almost completely towards X then it also converges almost surely to X. In other words, if X_n converges in probability to X sufficiently quickly (i.e. the above sequence of tail probabilities is summable for all ε > 0), then X_n also converges almost surely to X. This is a direct implication from the Borel-Cantelli lemma.

• If S_n is a sum of n real independent random variables:

${\displaystyle S_{n}=X_{1}+\cdots +X_{n}\,}$

then S_n converges almost surely if and only if S_n converges in probability.

• Lévy's convergence theorem gives sufficient conditions for almost sure convergence to imply L-convergence:

${\displaystyle \left.{\begin{array}{ccc}X_{n}{\xrightarrow {a.s.}}X\\\\|X_{n}|<Y\\\\\mathrm {E} (Y)<\infty \end{array}}\right\}\quad \Rightarrow \quad X_{n}{\xrightarrow {L^{1}}}X}$

• A necessary and sufficient condition for ${\displaystyle L^{1}}$ convergence is ${\displaystyle X_{n}{\xrightarrow {P}}X}$ and the sequence ${\displaystyle (X_{n})}$ is uniformly integrable.

See also

• Convergence of measures
• Continuous stochastic process: the question of continuity of a stochastic process is essentially a question of convergence, and many of the same concepts and relationships used above apply to the continuity question.
• Asymptotic distribution
• Big O in probability notation
• Skorokhod's representation theorem

Notes

1. Bickel et al. 1998, A.8, page 475
2. van der Vaart & Wellner 1996, p. 4
3. Romano & Siegel 1985, Example 5.26
4. Dudley 2002, Chapter 9.2, page 287
5. Dudley 2002, p. 289
6. Porat, B. (1994). Digital Processing of Random Signals: Theory & Methods. Prentice Hall. p. 19. ISBN 0130637513.
7. ^ van der Vaart 1998, Theorem 2.7
8. van der Vaart 1998, Th.2.19

References

• Bickel, Peter J. (1998). Efficient and adaptive estimation for semiparametric models. New York: Springer-Verlag. ISBN 0387984739. {{cite book}}: Unknown parameter |coauthors= ignored (|author= suggested) (help); Unknown parameter |lcc= ignored (help)
• Billingsley, Patrick (1986). Probability and Measure. Wiley Series in Probability and Mathematical Statistics (2nd ed.). Wiley.
• Billingsley, Patrick (1999). Convergence of probability measures (2nd ed.). John Wiley & Sons. pp. 1–28. ISBN 0471197459.
• Dudley, R.M. (2002). Real analysis and probability. Cambridge, UK: Cambridge University Press. ISBN 052180972X.
• Grimmett, G.R. (1992). Probability and random processes (2nd ed.). Clarendon Press, Oxford. pp. 271–285. ISBN 0-19-853665-8. {{cite book}}: Unknown parameter |coauthors= ignored (|author= suggested) (help)
• Jacobsen, M. (1992). Videregående Sandsynlighedsregning (Advanced Probability Theory) (3rd ed.). HCØ-tryk, Copenhagen. pp. 18–20. ISBN 87-91180-71-6.
• Ledoux, Michel (1991). Probability in Banach spaces. Berlin: Springer-Verlag. pp. xii+480. ISBN 3-540-52013-9. MR1102015. {{cite book}}: Unknown parameter |coauthor= ignored (|author= suggested) (help)
• Romano, Joseph P. (1985). Counterexamples in probability and statistics. Great Britain: Chapman & Hall. ISBN 0412989018. {{cite book}}: Unknown parameter |coauthor= ignored (|author= suggested) (help); Unknown parameter |lcc= ignored (help)
• van der Vaart, Aad W. (1996). Weak convergence and empirical processes. New York: Springer-Verlag. ISBN 0387946403. {{cite book}}: Unknown parameter |coauthor= ignored (|author= suggested) (help); Unknown parameter |lcc= ignored (help)
• van der Vaart, Aad W. (1998). Asymptotic statistics. New York: Cambridge University Press. ISBN 9780521496032. {{cite book}}: Unknown parameter |lcc= ignored (help)CS1 maint: ref duplicates default (link)
• Williams, D. (1991). Probability with Martingales. Cambridge University Press. ISBN 0521406056.
• Wong, E. (1985). Stochastic Processes in Engineering Systems. New York: Springer–Verlag. {{cite book}}: Unknown parameter |coauthors= ignored (|author= suggested) (help)

Template:Cz

• Articles needing cleanup from September 2009
• Cleanup tagged articles without a reason field from September 2009
• Misplaced Pages pages needing cleanup from September 2009
• Probability theory
• Statistical theory
• Stochastic processes