Properties of MLE and MAP

NOTE

This is part of the 3rd homework, titled Probabilistic Inference, for the course Machine Learning (IN2064) in the Winter Semester 2024/25 at TUM.

Problem 8:

Consider a Bernoulli random variable $X$ and suppose we have observed $m$ occurrences of $X=1$ and $l$ occurrences of $X=0$ in a sequence of $N=m+l$ Bernoulli experiments. We are only interested in the number of occurrences of $X=1$—we will model this with a Binomial distribution with parameter $\theta$. A prior distribution for $\theta$ is given by the Beta distribution with parameters $a,b$. Show that the posterior mean value $\mathbb{E}[\theta \mid \mathcal{D}]$ (not the MAP estimate) of $\theta$ lies between the prior mean of $\theta$ and the maximum likelihood estimate for $\theta$.

To do this, show that the posterior mean can be written as $\lambda$ times the prior mean plus $(1-\lambda )$ times the maximum likelihood estimate, with $0\le \lambda \le 1$. This illustrates the concept of the posterior mean being a compromise between the prior distribution and the maximum likelihood solution.

The probability mass function of the Binomial distribution for some $m\in \{0,1,\dots ,N\}$ is:

$$p(x=m\mid N,\theta )=\left(\genfrac{}{}{0ex}{}{N}{m}\right){\theta }^m(1-\theta {)}^{N-m}.$$

Hint: Identify the posterior distribution. You may then look up the mean rather than computing it.

Solution

Our goal is to show that the posterior mean $\mathbb{E}[\theta \mid \mathcal{D}]$ lies between the prior mean $\mathbb{E}[\theta ]$ and the maximum likelihood estimate (MLE) ${\hat{\theta }}_{\text{MLE}}$. Specifically, we aim to show that:

$$\mathbb{E}[\theta \mid \mathcal{D}]=\lambda \cdot \mathbb{E}[\theta ]+(1-\lambda )\cdot {\hat{\theta }}_{\text{MLE}},$$

for some $\lambda \in [0,1]$. This illustrates the concept that the posterior mean is a compromise between the prior distribution and the maximum likelihood solution.

We will proceed in the following steps:

1. Calculating the posterior distribution.
2. Calculating the mean of the posterior distribution.
3. Calculating the maximum likelihood estimate (MLE) of $\theta$.
4. Expressing the posterior mean as a convex combination of the prior mean and the MLE.

---

1. Calculating the posterior distribution

First, we recall the forms of the prior distribution and the likelihood function:

• Prior distribution: $$P(\theta )=\text{Beta}(a,b)=\frac{\Gamma (a+b)}{\Gamma (a)\Gamma (b)}{\theta }^{a-1}(1-\theta {)}^{b-1},\theta \in [0,1].$$
• Likelihood function:

$$P(D\mid \theta )=\left(\genfrac{}{}{0ex}{}{N}{m}\right){\theta }^m(1-\theta {)}^{N-m},$$

where $D$ represents the observed data (with $m$ successes and $l=N-m$ failures).

Using Bayes’ theorem, the posterior distribution is proportional to the product of the likelihood and the prior:

$$P(\theta \mid D)\propto P(D\mid \theta )P(\theta ).$$

Substituting the expressions for the likelihood and the prior, we get:

$$\begin{array}{rl}P(\theta \mid D)& \propto \left[{\theta }^m(1-\theta {)}^{N-m}\right]\left[{\theta }^{a-1}(1-\theta {)}^{b-1}\right]\\ & ={\theta }^{(a+m-1)}(1-\theta {)}^{(b+N-m-1)}\\ & ={\theta }^{(a+m-1)}(1-\theta {)}^{(b+l-1)}.\end{array}$$

It’s not hard to notice that we arrived at unnormalized form of a Beta distribution with updated parameters. Thanks to our perceptiveness we don’t have to go through the hoops of calculating normalizing constant, as it will be equal to the gamma term in the PDF of Beta distribution. As such we have calculated a posterior distribution:

$$P(\theta \mid D)=\text{Beta}(a^{\prime },b^{\prime }),$$

where:

$$a^{\prime }=a+m,b^{\prime }=b+l.$$

2. Calculating the mean of the posterior distribution

The mean of a Beta distribution $\text{Beta}(\alpha ,\beta )$ is given by:

$$\mathbb{E}[\theta ]=\frac{\alpha }{\alpha +\beta }.$$

Therefore, the posterior mean as given as follows:

$$\begin{array}{rl}\mathbb{E}[\theta \mid D]& =\mathbb{E}[Beta(\alpha +m,\beta +l)]=\frac{a+m}{a+b+N}.\end{array}$$

3. Calculating the maximum likelihood estimate (MLE)

The maximum likelihood estimate of $\theta$ for the Binomial distribution is the sample proportion:

$${\hat{\theta }}_{\text{MLE}}=\frac{m}{N}.$$

---

4. Expressing the posterior mean as a convex combination

The prior mean is:

$$\mathbb{E}[\theta ]=\frac{a}{a+b}.$$

Now, we aim to express $\mathbb{E}[\theta \mid D]$ as a convex combination of $\mathbb{E}[\theta ]$ and ${\hat{\theta }}_{\text{MLE}}$. Consider:

$$\begin{array}{rl}\mathbb{E}[\theta \mid D]& =\frac{a+m}{a+b+N}\\ & =\frac{a+b}{a+b+N}\cdot \frac{a}{a+b}+\frac{N}{a+b+N}\cdot \frac{m}{N}\\ & =\left(\frac{a+b}{a+b+N}\right)\mathbb{E}[\theta ]+\left(\frac{N}{a+b+N}\right){\hat{\theta }}_{\text{MLE}}.\end{array}$$

Defining:

$$\lambda =\frac{a+b}{a+b+N},1-\lambda =\frac{N}{a+b+N},$$

we have:

$$\mathbb{E}[\theta \mid D]=\lambda \cdot \mathbb{E}[\theta ]+(1-\lambda )\cdot {\hat{\theta }}_{\text{MLE}}.$$

Since $a,b,N\ge 0$, it follows that $\lambda \in [0,1]$.

Conclusion

We have shown that the posterior mean $\mathbb{E}[\theta \mid D]$ is a weighted average of the prior mean $\mathbb{E}[\theta ]$ and the MLE ${\hat{\theta }}_{\text{MLE}}$, with weights $\lambda$ and $1-\lambda$, respectively, where $\lambda =\frac{a+b}{a+b+N}$.

Therefore, the posterior mean lies between the prior mean and the maximum likelihood estimate:

$$\mathbb{E}[\theta \mid D]=\lambda \cdot \mathbb{E}[\theta ]+(1-\lambda )\cdot {\hat{\theta }}_{\text{MLE}},\text{with }0\le \lambda \le 1.$$

This result illustrates how the posterior mean balances the prior belief and the observed data in Bayesian inference.

---

Problem 9

Consider the following probabilistic model:

$$\begin{array}{rl}p(\lambda |a,b)& =\text{Gamma}(\lambda |a,b)=\frac{b^a}{\Gamma (a)}{\lambda }^{a-1}\exp (-b\lambda )\\ p(x|\lambda )& =\text{Poisson}(x|\lambda )=\frac{{\lambda }^x\exp (-\lambda )}{x!}\end{array}$$

where $a\in (1,\infty )$ and $b\in (0,\infty )$. We have observed a single data point $x\in \mathbb{N}$. Task: Derive the maximum a posteriori (MAP) estimate of the parameter $\lambda$ for the above probabilistic model. Show your work.

Solution

To calculate the maximum a posteriori, we need to compute the following formula:

$$\arg \underset{\lambda }{\max }p(\lambda |a,b,x)\text{\hspace{1em}(1)}$$

From Bayes’ theorem we can derive:

$$\begin{array}{rl}p(\lambda |a,b,x)& =\frac{p(a,b,x|\lambda )\cdot p(\lambda )}{p(a,b,x)}\\ & =\frac{p(a,b|\lambda )\cdot p(x|\lambda )\cdot p(\lambda )}{p(a,b)\cdot p(x)}\\ & =\frac{p(x|\lambda )\cdot p(\lambda |a,b)}{p(x)}\\ & \propto p(x|\lambda )\cdot p(\lambda |a,b)\end{array}\text{\hspace{1em}(2)}$$

Applying Equation (2) to Equation (1), we get:

$$\begin{array}{r}\arg \underset{\lambda }{\max }\left[\log p(x|\lambda )+\log p(\lambda |a,b)\right]=\\ =\arg \underset{\lambda }{\max }\left[\log \left(\frac{{\lambda }^x\exp (-\lambda )}{x!}\right)+\log \left(\frac{b^a}{\Gamma (a)}{\lambda }^{a-1}\exp (-b\lambda )\right)\right]=\\ =\arg \underset{\lambda }{\max }\left[x\log \lambda -\lambda +(a-1)\log \lambda -b\lambda \right]=\\ =\arg \underset{\lambda }{\max }\left[x\log \lambda -\lambda +(a-1)\log \lambda -b\lambda \right]=\\ =\arg \underset{\lambda }{\max }\left[(x+a-1)\log \lambda -(1+b)\lambda \right]\end{array}\text{\hspace{1em}(3)}$$

From the assumptions, we know that:

• $a\in (1,\infty )$,
• $b\in (0,\infty )$,
• $x\in \mathbb{N}$.

Because of these and the concavity of the logarithm function, we know that Equation (3) is concave.

To find the maximum, we set the derivative to zero:

$$\frac{(x+a-1)}{\lambda }-(1+b)=0$$

Solving for $\lambda$:

$$\lambda =\frac{x+a-1}{1+b}$$

Answer:

$${\lambda }_{\text{MAP}}=\frac{x+a-1}{1+b}$$