This article was published as a part of the Data Science Blogathon
Introduction
Chebyshev’s inequality and Weak law of large numbers are very important concepts in Probability and Statistics which are heavily used by Statisticians, Machine Learning Engineers, and Data Scientists when they are doing the predictive analysis.
So, In this article, we will be discussing these concepts with their applications in a detailed manner.
Table of Contents
1. Chebyshev’s Inequality
2. Applications of Chebyshev’s Inequality
3. Convergence in Probability
4. Chebyshev’s Theorem used in WLLN
5. Weak Law of Large Numbers(WLLN)
- WLLN for IID Random Variables
- Conditions for WLLN holds
- Some Important results related to WLLN
6. Applications of WLLN
Chebyshev’s Inequality
In probability theory, Chebyshev’s inequality, also known as “Bienayme-Chebyshev” inequality guarantees that, for a wide class of probability distributions, NO MORE than a certain fraction of values can be more than a certain distance from the mean.
Specifically, no more than 1/k2 of the distribution’s values can be more than k standard deviations away from the mean( or equivalently, at least 1-1/k2 of the distribution’s values are within k standard deviations of the mean).
Now, let’s formally define Chebyshev’s inequality:
Let X be a random variable with mean μ with a finite variance σ2, then for any real number k>0,
P(| X-μ | < kσ) ≥ 1-1/k2
OR
P(| X-μ | ≥ kσ) ≤ 1/k2
The rule is often known as Chebyshev’s theorem, tells about the range of standard deviations around the mean, in statistics.
This inequality has great utility because it can be applied to any probability distribution in which the mean and variance are defined.
For Example, it can be used to prove the weak law of large numbers, which we will be discussed later in this article.
Applications of Chebyshev’s Inequality
Numerical Example-1:
Suppose that it is known that the number of products formed in a factory during a week is a random variable with a mean of 50. If the variance of a week production is equal to 25, then what can be said about the productivity that it will be between 40 and 60?
Solution:
Step-1: Mean(μ) = 50, Variance(σ2) = 25 ⇒ σ= 5
Step-2: Required probability: P(40 < X < 60)
= P(40 < X < 60) = P(-10 < X-50 < 10) = P(| X-50 | < 10)
Step-3: Now, by using the Chebyshev’s theorem, we have P(| X-μ | < kσ) ≥ 1-1/k2
Find k by compare with general equation, therefore kσ = 10 ⇒ k(5) =10 ⇒ k=2
Step-4: Apply the Chebyshev’s Theorem to find the required probability:
≥ 1-1/k2 ≥ 1-(1/4) ≥ 3/4 ≥ 0.75
Step-5: Present the results
Therefore, the lower bound of the probability that the productivity lies between 40 and 60 is equal to 0.75.
Numerical Example-2:
A symmetric die is thrown 600 times. Find the lower bound for the probability of getting 80 to 120 sixes.
Solution:
Step-1: A symmetric die is thrown 600 times, so it follows Binomial Distribution and p=1/6.
Step-2: Now, by using the binomial distribution, we have to calculate the mean and variance of the random variables using the given below formula:
Mean = np = 600*(1/6) = 100
Variance = npq = 600*(1/6)*(5/6) = 500/6
Step-3: Required probability: P(80 < X < 120)
P(80 < X < 120) = P(-20 < X-100 < 20) = P(| X-100 | < 20)
Step-4: Now, by using the Chebyshev’s theorem, we have P(| X-μ | < kσ) ≥1-1/k2
Find k by compare with general equation, therefore kσ = 20 ⇒ k(√(500/6)) = 10 ⇒ k = 20√(6/500)
Step-5: Apply Chebyshev’s Theorem to find the required probability:
≥ 1-1/k2 ≥ 1-500/2400 ≥ 19/24 ≥ 0.79
Step-6: Present the results
Therefore, the lower bound of the probability of getting sixes between 80 and 120 is equal to 0.79.
Convergence in Probability
A sequence of random variables X1, X2, ——, Xn is said to convergence in probability to α if for any ε>0, the
lim n→∞ P(| Xn– α | < ε) = 1
OR
lim n→∞ P(| Xn– α | ≥ ε) = 0
We can write Xn-> α as n→∞ in probability.
Chebyshev’s theorem used in WLLN
Statement:
If X1, X2, —–, Xn is a sequence of random variables and if mean μn and standard deviation σn of Xn exists for all n and if σn->0 as n→∞, then Xn-μn -> 0 as n→∞ in probability.
Proof:
By Chebyshev’s inequality for ε >0,
P(| Xn-un | ≥ ε) ≤ σn2/ε2 -> 0 as n-> ∞.
Hence, Xn-un->0 as n-> ∞ in probability, provided σn->0 as n→∞
Weak law of large numbers (WLLN)
Let X1, X2,———, Xn is a sequence of random variables and μ1, μ2, ———-, μn be their respective means and let Bn= Var(X1+X2+———-+Xn)<∞.Then,
P(| {(X1+X2+———-+Xn)/n} – {(μ1+ μ2 ———-+μn)/n}| < ε ) ≥ 1- j
For all n > n0, where ε, j are arbitrary small positive numbers, provided
lim n→∞ (Bn/n2) -> 0
Remarks:
For the existence of WLLN, the following conditions:
- E(Xi) exists for all i
- Bn= Var(X1+X2+———-+Xn) exists, and
- lim n→∞ (Bn/n2) -> 0
Condition-1 is necessary(i.e, without it, the law itself cannot be stated).
Condition-2 and 3 are not necessary.
Condition-3 is a sufficient condition.
WLLN for IID Random Variables
If the variables, X1, X2,———, Xn are independent and identically distributed (IID), i.e, E(Xi)=μ and Var(Xi) = σ2 for all i, then
Bn = Var(X1+X2+———-+Xn) = Var(X1)+Var(X2)+————+Var(Xn) = nσ2
Hence, lim n→∞ (Bn/n2) = nσ2/n2= σ2/n -> 0
Thus, WLLN holds for the sequence of IID and we get
x̄n -> μ in probability i.e, x̄n converges in probability to μ.
Conditions for WLLN holds
Case-1: When the sequence of random variables { Xn } is independent, then
- E(Xi) exists for all i
- Bn= Var(X1+X2+———-+Xn) exists, and
- lim n→∞ (Bn/n2) -> 0
If at least one of the conditions is not met, then we apply the further test named Markov’s Theorem.
Case-2: When the sequence of random variables { Xn } is IID, then
E(Xi) exists for all i is enough for the existence of WLLN.
This result is known as Khinchin’s theorem.
Now, let’s see what the Markov Theorem states:
Markov’s theorem: The WLLN holds if for some δ >0,
E(|Xi|1+δ) exists and bounded
NOTE:
Markov’s theorem provides only a necessary condition for the WLLN to hold good. This means that if for some δ>0, E(|Xi|1+δ) unbounded then WLLN cannot hold for the sequence of random variables, { Xn }
Some Important Results
Result-1: If the variables are uniformly bounded then the condition,
lim n→∞ (Bn/n2) -> 0
is necessary as well as sufficient for WLLN to hold.
Result-2: The necessary and sufficient condition for sequence { Xn } to satisfy the WLLN is :
lim n→∞ E( Yn2/ 1+Yn2) -> 0
Where Yn = Sn– E(Sn)/n and Sn= X1+X2+——+Xn
Applications of WLLN
Numerical Example:
Let Xi assume that values i and -i with equal probabilities. Show that the law of large numbers cannot be applied to the independent variables X1, X2, ———-.
Solution:
Step-1: Calculate the mean and variance of the random variables
E(Xi) = Σ XiP(Xi) = i/2 – i/2 = 0
Var(Xi) = E(Xi2) – (E(Xi))2 = [i2/2+(-i)2/2] – (0)2 = i2
Step-2: Calculate the value of the limit of Bn/n2 as n tends to infinity
Since X1, X2, ———- are independent variables
Bn = Var(X1+X2+———-+Xn) = Var(X1) + Var(X2)+—————-+ Var(Xn) = 12+22+32+——-+n2
= n(n+1)(2n+1)/6
Therefore, Bn/n2 = (n+1)(2n+1)/6 tends to ∞ as n-> ∞
Step-3: Interpret the results using the results of WLLN
Hence, we cannot draw any conclusion on whether WLLN holds or not.
Step-4: Here we apply the further tests, such as Markov’s Test.
E(|Xi|1+δ) = i1+δ/2 + |-i|1+δ/2 = i1+δ
which is unbounded for i1+δ>0
Step-5: Present the final results
Hence, by Markov’s theorem, the WLLN cannot be applied to the sequence { Xi } of independent random variables.
This completes today’s discussion!
Endnotes
Thanks for reading!
I hope you enjoyed the article and increased your knowledge about Chebyshev’s Inequality and Weak Law of Large Numbers in Probability and Statistics.
Please feel free to contact me on Email
Something not mentioned or want to share your thoughts? Feel free to comment below And I’ll get back to you.
For the remaining articles, refer to the link.
About the Author
Aashi Goyal
Currently, I am pursuing my Bachelor of Technology (B.Tech) in Electronics and Communication Engineering from Guru Jambheshwar University(GJU), Hisar. I am very enthusiastic about Statistics, Machine Learning and Deep Learning.
The media shown in this article are not owned by Analytics Vidhya and are used at the Author’s discretion.
Currently, I am pursuing my Bachelor of Technology (B.Tech) in Electronics and Communication Engineering from Guru Jambheshwar University(GJU), Hisar. I am very enthusiastic about Statistics, Machine Learning and Deep Learning.
