Bayesian Hierarchical Model: Gamma and Inverse Gamma Prior

I have been taking a graduate course on computational statistics this semester. Albeit to the fact the choices for priors in Bayesian model is subjective and prone to human error, the paradigm of hierarchy comes pretty handy when we deal with sequential dependent models such as some RNNs.

It is more and more becoming why Bayesian statistics is computational intensive: mind you, what else you can resort to when you wish to integrate out a parameter over products of mixtures of exponential family densities?

Indeed, consider the following simple hierarchical model:

$$z \vert s \mathbin{\sim} N(0, s),\quad s \mathbin{\sim} ?.$$

We consider two cases: either $s \mathbin{\sim} \operatorname{\Gamma}(a, b)$ or $s \mathbin{\sim} \operatorname{Inv\operatorname{\Gamma}}(a, b)$. The integration for the first case is not very straightforward; nevertheless, it is given as a one-liner fact in the original paper [West 1987]. We proceed using this technique introduced on Cross Validated.

Gamma Prior

$$\begin{align*} z &= \int_{\mathbb{R}} \operatorname{\mathcal{N}}(0, s)\operatorname{\Gamma}(a, b) \mathop{}\!\operatorname{d} s\\ &= \frac{b^a}{\sqrt{2\pi}\operatorname{\Gamma}(a)} \int_{\mathbb{R}} s^ {-\frac{1}{2}} s ^ {a-1} \exp\left\{-\frac{1}{2}\left( \frac{\lvert z \rvert ^2}{s} + 2bs \right)\right\} \mathop{}\!\operatorname{d} s \\ &= \frac{b^a \mathop{}\!\mathrm{e}^{-\sqrt{2b}\lvert z \rvert}}{\sqrt{2\pi}\operatorname{\Gamma}(a)} \int_{\mathbb{R}} s^ {-\frac{1}{2}} s ^ {a-1} \exp\left\{-\frac{1}{2}\left( \frac{\lvert z \rvert}{\sqrt{s}} + \sqrt{2b}\sqrt{s}\right)^2\right\} \mathop{}\!\operatorname{d} s. \end{align*}$$

This suggests a change of variable as $v = \sqrt{s} > 0$. It follows that:

$$\begin{align*} z &= \frac{2 b^a \mathop{}\!\mathrm{e}^{-\sqrt{2b}\lvert z \rvert}}{\sqrt{2\pi}\operatorname{\Gamma}(a)} \int_{\mathbb{R}^+} v^{2(a-1)} \exp\left\{ -\frac{1}{2} \left( \frac{\lvert z \rvert}{v} - \sqrt{2b} v \right)^2 \right\} \mathop{}\!\operatorname{d} v. \end{align*}$$

Write $t = \sqrt{2b} v - \frac{\lvert z \rvert}{v} \in \mathbb{R}$. Noted $t(v)$ is monotone and $t \rightarrow -\infty$ as $v \rightarrow 0$; $t \rightarrow \infty$ as $v \rightarrow \infty$.

Moreover, noted

$$v = \frac{1}{2\sqrt{2b}} \left(t - \sqrt{t^2 + 4\sqrt{2b}\lvert z \rvert}\right),$$

where the other root is rejected since $v\ge 0$. Hence

$$\mathop{}\!\operatorname{d} v =\frac{1}{2\sqrt{2b}}\left(1 - \frac{t}{\sqrt{t^2 + 4\sqrt{2b} \lvert z \rvert}}\right) \mathop{}\!\operatorname{d} t.$$

It follows that

$$\begin{align*} z &= \frac{2 b^a \mathop{}\!\mathrm{e}^{-\sqrt{2b}\lvert z \rvert}}{\sqrt{2\pi}\operatorname{\Gamma}(a)} \int_{\mathbb{R}^+} v^{2(a-1)} \exp\left\{ -\frac{1}{2} t ^2 \right\} \mathop{}\!\operatorname{d} v \\ &\overset{a=1}{=} \frac{2 b \mathop{}\!\mathrm{e}^{-\sqrt{2b}\lvert z \rvert}}{\sqrt{2\pi}} \int_{\mathbb{R}} \exp\left\{ -\frac{1}{2} t ^ 2 \right\} \frac{1}{2\sqrt{2b}} \left( 1 - \frac{t}{ \sqrt{t^2+ 4\sqrt{2b}\lvert z \rvert} } \right) \mathop{}\!\operatorname{d} v \\ &= \frac{\sqrt{2b}}{2} \mathop{}\!\mathrm{e}^{-\sqrt{2b} \lvert z \rvert}, \end{align*}$$

where the last equality follows since the second term in the Jacobian is odd.

We therefore conclude that when $a=1$, $z\mathbin{\sim} \mathrm{DE}(\sqrt{2b})$.

Inverse Gamma Prior

$$\begin{align*} z &= \int_{\mathbb{R}} \operatorname{Inv\operatorname{\Gamma}}(a,b) \operatorname{\mathcal{N}}(0, s) \mathop{}\!\operatorname{d} s \\ &= \int_{\mathbb{R}} \frac{1}{\sqrt{2 \pi s}} \mathop{}\!\mathrm{e}^{-\frac{1}{2s} z^2} \frac{b^a}{\operatorname{\Gamma}(a)} s^{-a-1} \mathop{}\!\mathrm{e}^{-\frac{b}{s}} \mathop{}\!\operatorname{d} s \\ &= \frac{b^a}{\sqrt{2\pi}\operatorname{\Gamma}(a)} \int_{\mathbb{R}} s^{-(a+\frac{1}{2})-1} \mathop{}\!\mathrm{e}^{-\left(\frac{z^2}{2} + b\right) \frac{1}{s}} \mathop{}\!\operatorname{d} s \\ &= \frac{\operatorname{\Gamma}\left(a + \frac{1}{2}\right)} {\sqrt{2b\pi}\operatorname{\Gamma}(a)} \left(1 + \frac{z^2}{2b} \right) ^ {-\left(a + \frac{1}{2}\right)}. \end{align*}$$

I.e., when $a = b = v/2$, $z\mathbin{\sim} t_{\mathrm{df}=v}$.

Remark.

There may exist other simpler and more elegant method of tacking the first integral. For example, the use of moment generating functions or convert it into a contour integral. I may make a further update after I grasp them.

Moreover, the setting of $a=1$ in the first case is solely for the sake of integration. I am not sure whether the general integral is analytic otherwise.

References

1. West, M. 1987. On scale mixtures of normal distributions. Biometrika 74, 3, 646–648.