Optimising Likelihoods - Monotonic Transforms

NOTE

This is part of the 3rd homework, titled Probabilistic Inference, for the course Machine Learning (IN2064) in the Winter Semester 2024/25 at TUM.

Intro

Usually, we maximise the log-likelihood, $\log p(x_1,\dots ,x_n\mid \theta )$ instead of the likelihood. The next two problems provide a justification for this. In the lecture, we encountered the likelihood maximisation problem:

$$\arg \underset{\theta \in [0,1]}{\max }{\theta }^t(1-\theta {)}^h,$$

where $t$ and $h$ denoted the number of tails and heads in a sequence of coin tosses, respectively.

Problem 6

Compute the first and second derivative of this likelihood with respect to $\theta$. Then compute the first and second derivative of the log-likelihood $\log {\theta }^t(1-\theta {)}^h$.

Solution

Likelihood:

• First derivative:

$$\frac{d}{d\theta }{\theta }^t(1-\theta {)}^h=t{\theta }^{t-1}(1-\theta {)}^h-h{\theta }^t(1-\theta {)}^{h-1}$$

• Second derivative:

$$\begin{array}{rl}\frac{d^2}{d{\theta }^2}[{\theta }^t(1-\theta {)}^h]& =\frac{d}{d\theta }[t{\theta }^{t-1}(1-\theta {)}^h-h{\theta }^t(1-\theta {)}^{h-1}]\\ & =\underset{\text{Derivative of first term}}{\underbrace{[t(t-1){\theta }^{t-2}(1-\theta {)}^h-ht{\theta }^{t-1}(1-\theta {)}^{h-1}]}}\\ & \underset{\text{Derivative of second term}}{\underbrace{-[ht{\theta }^{t-1}(1-\theta {)}^{h-1}-h(h-1){\theta }^t(1-\theta {)}^{h-2}]}}\\ & =t(t-1){\theta }^{t-2}(1-\theta {)}^h-2ht{\theta }^{t-1}(1-\theta {)}^{h-1}+h(h-1){\theta }^t(1-\theta {)}^{h-2}\end{array}$$

Log-likelihood:

1. First derivative:

$$\begin{array}{rl}\frac{d}{d\theta }[\log ({\theta }^t(1-\theta {)}^h)]& =\frac{d}{d\theta }[t\log \theta +h\log (1-\theta ]\\ & =\frac{t}{\theta }-\frac{h}{1-\theta }\end{array}$$

2. Second derivative:

$$\begin{array}{rl}\frac{d^2}{d{\theta }^2}[\log ({\theta }^t(1-\theta {)}^h)]& =\frac{d}{d\theta }[\frac{t}{\theta }-\frac{h}{1-\theta }]\\ & =t{\theta }^{-2}-h(1-\theta {)}^{-2}\end{array}$$

Problem 7

Show that for any differentiable, positive function $f(\theta )$, every local maximum of $\log f(\theta )$ is also a local maximum of $f(\theta )$. Considering this and the previous exercise, what is your conclusion?

Solution

Let $f(\theta )$ be a differentiable and positive function on its domain. Denote $g(\theta )=\log f(\theta )$. We will show that: *if ${\theta }_0$ is a local maximum of $g(\theta )$, then ${\theta }_0$ is also a local maximum of $f(\theta )$.

Since $f(\theta )>0$, $\log f(\theta )$ is well-defined and differentiable. The derivative of $g(\theta )$ is given by:

$$g^{\prime }(\theta )=\frac{f^{\prime }(\theta )}{f(\theta )}.$$

Now, if ${\theta }_0$ is a local maximum of $g(\theta )$, then $g^{\prime }({\theta }_0)=0$. Substituting $g^{\prime }({\theta }_0)$:

$$\frac{f^{\prime }({\theta }_0)}{f({\theta }_0)}=0.$$

Since $f({\theta }_0)>0$, this implies:

$$f^{\prime }({\theta }_0)=0.$$

Thus, ${\theta }_0$ is a critical point of $f(\theta )$. As ${\theta }_0$ is a local maximum of $g(\theta )$, there exists an interval $({\theta }_0-ϵ,{\theta }_0+ϵ)$ such that for all $\theta$ in this interval:

$$g(\theta )\le g({\theta }_0).$$

Equivalently:

$$\log f(\theta )\le \log f({\theta }_0).$$

Exponentiating both sides (which preserves the inequality since the exponential function is monotonic):

$$f(\theta )\le f({\theta }_0).$$

Thus, ${\theta }_0$ is a local maximum of $f(\theta )$.

Conclusion

After this and the previous exercise, we can conclude that maximising the likelihood function is the same as maximising the log-likelihood function. However, the log-likelihood is much simpler to derive, making it easier to use in practice.