STA 702 Fall 2023 - Lecture 11: Conjugate Priors and Bayesian Regression

Semi-Conjugate Priors in Linear Regression

Regression Model (Sampling model) $Y ∣ β, ϕ \sim N (X β, ϕ^{- 1} I_{n})$
Semi-Conjugate Prior Independent Normal Gamma $\begin{aligned} β & \sim N (b_{0}, Φ_{0}^{- 1}) \\ ϕ & \sim Gamma (ν_{0} / 2, {SS}_{0} / 2) \end{aligned}$
- Conditional Normal for $β ∣ ϕ, Y$ and
- Conditional Gamma $ϕ ∣ Y, β$
- requires Gibbs sampling or other Metropolis-Hastings algorithms

Conjugate Priors in Linear Regression

Regression Model (Sampling model) $Y ∣ β, ϕ \sim N (X β, ϕ^{- 1} I_{n})$
Conjugate Normal-Gamma Model: factor joint prior $p (β, ϕ) = p (β ∣ ϕ) p (ϕ)$ $\begin{aligned} β ∣ ϕ & \sim N (b_{0}, ϕ^{- 1} Φ_{0}^{- 1}) & p (β ∣ ϕ) & = \frac{| ϕ Φ_{0} |^{1 / 2}}{(2 π)^{p / 2}} e^{{- \frac{ϕ}{2} (β - b_{0})^{T} Φ_{0} (β - b_{0})}} \\ ϕ & \sim Gamma (v_{0} / 2, {SS}_{0} / 2) & p (ϕ) & = \frac{1}{Γ (ν_{0} / 2)} {(\frac{{SS}_{0}}{2})}^{ν_{0} / 2} ϕ^{ν_{0} / 2 - 1} e^{- ϕ \frac{{SS}_{0}}{2}} \\ \Rightarrow (β, ϕ) & \sim NG (b_{0}, Φ_{0}, ν_{o}, {SS}_{0}) \end{aligned}$
Normal-Gamma distribution indexed by 4 hyperparameters
Note Prior Covariance for $β$ is scaled by $σ^{2} = 1 / ϕ$

Finding the Posterior Distribution

Likelihood: $L (β, ϕ) \propto ϕ^{n / 2} e^{- \frac{ϕ}{2} (Y - X β)^{T} (Y - X β)}$
$\begin{array}{rcl} p (β, ϕ ∣ Y) & \propto & ϕ^{\frac{n}{2}} e^{- \frac{ϕ}{2} (Y - X β)^{T} (Y - X β)} \times \\ ϕ^{\frac{ν_{0}}{2} - 1} e^{- ϕ \frac{{SS}_{0}}{2}} \times ϕ^{\frac{p}{2}} e^{- \frac{ϕ}{2} (β - b_{0})^{T} Φ_{0} (β - b_{0})} \end{array}$
Quadratic in Exponential $\exp {- \frac{ϕ}{2} (β - b)^{T} Φ (β - b)} = \exp {- \frac{ϕ}{2} (β^{T} Φ β - 2 β^{T} Φ b + b^{T} Φ b)}$
- Expand quadratics and regroup terms
- Read off posterior precision from Quadratic in $β$
- Read off posterior mean from Linear term in $β$
- will need to complete the quadratic in the posterior mean due to $ϕ$

Expand and Regroup

$\begin{array}{rcl} p (β, ϕ ∣ Y) & \propto & ϕ^{\frac{n}{2}} e^{- \frac{ϕ}{2} (Y^{T} Y - 2 β^{T} X^{T} Y + β^{T} X^{T} X β)} \times \\ ϕ^{\frac{ν_{0}}{2} - 1} e^{- ϕ \frac{{SS}_{0}}{2}} \times ϕ^{\frac{p}{2}} e^{- \frac{ϕ}{2} (β Φ_{0} β - 2 β^{T} Φ_{0} b + b_{0}^{T} Φ_{0} b_{0})} \end{array}$

$\begin{array}{rcl} p (β, ϕ ∣ Y) & \propto & ϕ^{\frac{n + p + ν_{0}}{2} - 1} e^{- \frac{ϕ}{2} ({SS}_{0} + Y^{T} Y + b_{0}^{T} Φ_{0} b_{0})} \times \\ e^{- \frac{ϕ}{2} (β^{T} (X^{T} X) β - 2 β^{T} X^{T} X (X^{T} X)^{- 1} X^{T} Y + β Φ_{0} β - 2 β^{T} Φ_{0} b)} \end{array}$

$\begin{array}{rcl} = & ϕ^{\frac{n + p + ν_{0}}{2} - 1} e^{- \frac{ϕ}{2} ({SS}_{0} + Y^{T} Y + b_{0}^{T} Φ_{0} b_{0})} \times \\ e^{- \frac{ϕ}{2} (β^{T} (X^{T} X + Φ_{0}) β)} \times \\ e^{- \frac{ϕ}{2} (- 2 β^{T} (X^{T} X \hat{β} + Φ_{0} b_{0}))} \end{array}$

Complete the Quadratic

$\begin{array}{rcl} p (β, ϕ ∣ Y) & \propto & ϕ^{\frac{n + p + ν_{0}}{2} - 1} e^{- \frac{ϕ}{2} ({SS}_{0} + Y^{T} Y + b_{0}^{T} Φ_{0} b_{0})} \times \\ e^{- \frac{ϕ}{2} (β^{T} (X^{T} X + Φ_{0}) β)} \times Φ_{n} \equiv X^{T} X + Φ_{0} \\ e^{- \frac{ϕ}{2} (- 2 β^{T} Φ_{n} Φ_{n}^{- 1} (X^{T} X \hat{β} + Φ_{0} b_{0}))} \times b_{n} \equiv Φ_{n}^{- 1} (X^{T} X \hat{β} + Φ_{0} b_{0}) \\ e^{- \frac{ϕ}{2} (b_{n}^{T} Φ_{n} b_{n} - b_{n}^{T} Φ_{n} b_{n})} \end{array}$

$\begin{array}{rcl} = & ϕ^{\frac{n + ν_{0}}{2} - 1} e^{- \frac{ϕ}{2} ({SS}_{0} + Y^{T} Y + b_{0}^{T} Φ_{0} b_{0} - b_{n}^{T} Φ_{n} b_{n})} \times \\ ϕ^{\frac{p}{2}} e^{- \frac{ϕ}{2} ((β^{T} - b_{n})^{T} Φ_{n} (β - b_{n}))} \end{array}$

$\begin{array}{rcl} \propto & ϕ^{\frac{n + ν_{0}}{2} - 1} e^{- \frac{ϕ}{2} ({SS}_{0} + Y^{T} Y + b_{0}^{T} Φ_{0} b_{0} - b_{n}^{T} Φ_{n} b_{n})} \times \\ | ϕ Φ_{n} |^{\frac{1}{2}} e^{- \frac{ϕ}{2} ((β^{T} - b_{n})^{T} Φ_{n} (β - b_{n}))} \end{array}$

Posterior Distributions

Posterior density (up to normalizing contants) $p (β, ϕ ∣ Y) = p (ϕ ∣ Y) p (β ∣ ϕ Y)$ $\begin{array}{rcl} p (ϕ ∣ Y) p (β ∣ ϕ Y) & \propto & ϕ^{\frac{n + ν_{0}}{2} - 1} e^{- \frac{ϕ}{2} ({SS}_{0} + Y^{T} Y + b_{0}^{T} Φ_{0} b_{0} - b_{n}^{T} Φ_{n} b_{n})} \times \\ (2 π)^{- \frac{p}{2}} | ϕ Φ_{n} |^{\frac{1}{2}} e^{- \frac{ϕ}{2} (β - b_{n})^{T} Φ_{n} (β - b_{n})} \end{array}$

Marginal $\begin{array}{rcl} p (ϕ ∣ Y) & \propto & ϕ^{\frac{n + ν_{0}}{2} - 1} e^{- \frac{ϕ}{2} ({SS}_{0} + Y^{T} Y + b_{0}^{T} Φ_{0} b_{0} - b_{n}^{T} Φ_{n} b_{n})} \times \\ \int_{R^{p}} (2 π)^{- \frac{p}{2}} | ϕ Φ_{n} |^{\frac{1}{2}} e^{- \frac{ϕ}{2} (β - b_{n})^{T} Φ_{n} (β - b_{n}) d β} \\ = & ϕ^{\frac{n + ν_{0}}{2} - 1} e^{- \frac{ϕ}{2} ({SS}_{0} + Y^{T} Y + b_{0}^{T} Φ_{0} b_{0} - b_{n}^{T} Φ_{n} b_{n})} \end{array}$

Conditional Normal for $β ∣ ϕ, Y$ and marginal Gamma for $ϕ ∣ Y$
No need for Gibbs sampling!

$NG$ Posterior Distribution

$\begin{array}{rcl} β ∣ ϕ, Y & \sim & N (b_{n}, (ϕ Φ_{n})^{- 1}) \\ ϕ ∣ Y & \sim & Gamma (\frac{ν_{n}}{2}, \frac{{SS}_{n}}{2}) \\ (β, ϕ) ∣ Y & \sim & NG (b_{n}, Φ_{n}, ν_{n}, {SS}_{n}) \end{array}$

Hyperparameters: $\begin{aligned} Φ_{n} & = X^{T} X + Φ_{0} & b_{n} & = Φ_{n}^{- 1} (X^{T} X \hat{β} + Φ_{0} b_{0}) \\ ν_{n} & = n + ν_{0} & {SS}_{n} & = {SS}_{0} + Y^{T} Y + b_{0}^{T} Φ_{0} b_{0} - b_{n}^{T} Φ_{n} b_{n} \end{aligned}$

$\begin{aligned} {SS}_{n} & = {SS}_{0} + ‖ Y - X b_{n} ‖^{2} + (b_{0} - b_{n})^{T} Φ_{0} (b_{0} - b_{n}) \\ = {SS}_{0} + ‖ Y - X b_{n} ‖^{2} + ‖ b_{0} - b_{n} ‖_{Φ_{0}}^{2} \end{aligned}$

Inner product induced by prior precision $⟨ u, v ⟩_{A} \equiv u^{T} A v$
$‖ b_{0} - b_{n} ‖_{Φ_{0}}^{2}$ mismatch of prior and posterior mean under prior

Marginal Distribution

Theorem: Student-t

Let

θ ∣ ϕ \sim N (m, \frac{1}{ϕ} Σ)

and

ϕ \sim Gamma (ν / 2, ν {\hat{σ}}^{2} / 2)

.

Then $θ$ $(p \times 1)$ has a $p$ dimensional multivariate $t$ distribution $θ \sim t_{ν} (m, {\hat{σ}}^{2} Σ)$ with location $m$ , scale matrix ${\hat{σ}}^{2} Σ$ and density

$p (θ) \propto {[1 + \frac{1}{ν} \frac{(θ - m)^{T} Σ^{- 1} (θ - m)}{{\hat{σ}}^{2}}]}^{- \frac{p + ν}{2}}$

Note - true for prior or posterior given $Y$

Derivation

Marginal density $p (θ) = \int_{0}^{\infty} p (θ ∣ ϕ) p (ϕ) d ϕ$

$\begin{array}{rcl} p (θ) & \propto & \int | Σ / ϕ |^{- 1 / 2} e^{- \frac{ϕ}{2} (θ - m)^{T} Σ^{- 1} (θ - m)} ϕ^{ν / 2 - 1} e^{- ϕ \frac{ν {\hat{σ}}^{2}}{2}} d ϕ \\ \propto & \int ϕ^{p / 2} ϕ^{ν / 2 - 1} e^{- ϕ \frac{(θ - m)^{T} Σ^{- 1} (θ - m) + ν {\hat{σ}}^{2}}{2}} d ϕ \\ \propto & \int ϕ^{\frac{p + ν}{2} - 1} e^{- ϕ \frac{(θ - m)^{T} Σ^{- 1} (θ - m) + ν {\hat{σ}}^{2}}{2}} d ϕ \\ = & Γ ((p + ν) / 2) {(\frac{(θ - m)^{T} Σ^{- 1} (θ - m) + ν {\hat{σ}}^{2}}{2})}^{- \frac{p + ν}{2}} \\ \propto & {((θ - m)^{T} Σ^{- 1} (θ - m) + ν {\hat{σ}}^{2})}^{- \frac{p + ν}{2}} \\ \propto & {(1 + \frac{1}{ν} \frac{(θ - m)^{T} Σ^{- 1} (θ - m)}{{\hat{σ}}^{2}})}^{- \frac{p + ν}{2}} \end{array}$

Marginal Posterior Distribution of $β$

$\begin{array}{rcl} β ∣ ϕ, Y & \sim & N (b_{n}, ϕ^{- 1} Φ_{n}^{- 1}) \\ ϕ ∣ Y & \sim & Gamma (\frac{ν_{n}}{2}, \frac{{SS}_{n}}{2}) \end{array}$

Let ${\hat{σ}}^{2} = {SS}_{n} / ν_{n}$ (Bayesian MSE)
The marginal posterior distribution of $β$ is multivariate Student-t $β ∣ Y \sim t_{ν_{n}} (b_{n}, {\hat{σ}}^{2} Φ_{n}^{- 1})$
Any linear combination $λ^{T} β$ has a univariate $t$ distribution with $v_{n}$ degrees of freedom $λ^{T} β ∣ Y \sim t_{ν_{n}} (λ^{T} b_{n}, {\hat{σ}}^{2} λ^{T} Φ_{n}^{- 1} λ)$
use for individual $β_{j}$ , the mean of $Y$ , $x^{T} β$ , at $x$ , or predictions $Y^{*} = {x^{*}}^{T} β + ϵ_{i}^{*}$

Predictive Distributions

Suppose $Y^{*} ∣ β, ϕ \sim N_{s} (X^{*} β, I_{s} / ϕ)$ and is conditionally independent of $Y$ given $β$ and $ϕ$

What is the predictive distribution of $Y^{*} ∣ Y$ ?
Use the representation that $Y^{*} \overset{D}{=} X^{*} β + ϵ^{*}$ and $ϵ^{*}$ is independent of $Y$ given $ϕ$

$\begin{array}{rcl} X^{*} β + ϵ^{*} ∣ ϕ, Y & \sim & N (X^{*} b_{n}, (X^{*} Φ_{n}^{- 1} X^{* T} + I_{s}) / ϕ) \\ Y^{*} ∣ ϕ, Y & \sim & N (X^{*} b_{n}, (X^{*} Φ_{n}^{- 1} X^{* T} + I_{s}) / ϕ) \\ ϕ ∣ Y & \sim & Gamma (\frac{ν_{n}}{2}, \frac{{\hat{σ}}^{2} ν_{n}}{2}) \end{array}$

Use the Theorem to conclude that $Y^{*} ∣ Y \sim t_{ν_{n}} (X^{*} b_{n}, {\hat{σ}}^{2} (I + X^{*} Φ_{n}^{- 1} X^{T}))$

Choice of Conjugate (or Semi-Conjugate) Prior

need to specify Normal prior mean $b_{0}$ and precision $Φ_{0}$
need to specify Gamma shape ( $ν_{o}$ prior df) and rate (estimate of $σ^{2}$ )
hard in higher dimensions!
default choices?
- Jeffreys’ prior
- unit-information prior
- Zellner’s g-prior
- ridge priors
- mixtures of conjugate priors

Jeffreys’ Prior

Jeffreys prior is invariant to model parameterization of $θ = (β, ϕ)$ $p (θ) \propto | I (θ) |^{1 / 2}$
$I (θ)$ is the Expected Fisher Information matrix $I (θ) = - E [[\frac{\partial^{2} \log (L (θ))}{\partial θ_{i} \partial θ_{j}}]]$
log likelihood expressed as function of sufficient statistics

$\log (L (β, ϕ)) = \frac{n}{2} \log (ϕ) - \frac{ϕ}{2} ‖ (I_{n} - P_{x}) Y ‖^{2} - \frac{ϕ}{2} (β - \hat{β})^{T} (X^{T} X) (β - \hat{β})$

projection matrix $P_{X} = X (X^{T} X)^{- 1} X^{T}$

Information matrix

$\begin{array}{rcl} \frac{\partial^{2} \log L}{\partial θ \partial θ^{T}} & = & [\begin{array}{cc} - ϕ (X^{T} X) & - (X^{T} X) (β - \hat{β}) \\ - (β - \hat{β})^{T} (X^{T} X) & - \frac{n}{2} \frac{1}{ϕ^{2}} \end{array}] \\ E [\frac{\partial^{2} \log L}{\partial θ \partial θ^{T}}] & = & [\begin{array}{cc} - ϕ (X^{T} X) & 0_{p} \\ 0_{p}^{T} & - \frac{n}{2} \frac{1}{ϕ^{2}} \end{array}] \\ I ((β, ϕ)^{T}) & = & [\begin{array}{cc} ϕ (X^{T} X) & 0_{p} \\ 0_{p}^{T} & \frac{n}{2} \frac{1}{ϕ^{2}} \end{array}] \end{array}$

Jeffreys’ Prior (don’t use!) $\begin{array}{rcl} p_{J} (β, ϕ) & \propto & | I ((β, ϕ)^{T}) |^{1 / 2} = | ϕ X^{T} X |^{1 / 2} {(\frac{n}{2} \frac{1}{ϕ^{2}})}^{1 / 2} \propto ϕ^{p / 2 - 1} | X^{T} X |^{1 / 2} \\ \propto & ϕ^{p / 2 - 1} \end{array}$

Recommended Independent Jeffreys Prior

Treat $β$ and $ϕ$ separately (orthogonal parameterization)
$p_{I J} (β) \propto | I (β) |^{1 / 2}$ and $p_{I J} (ϕ) \propto | I (ϕ) |^{1 / 2}$

$I ((β, ϕ)^{T}) = [\begin{array}{cc} ϕ (X^{T} X) & 0_{p} \\ 0_{p}^{T} & \frac{n}{2} \frac{1}{ϕ^{2}} \end{array}]$

$\begin{aligned} p_{I J} (β) & \propto | ϕ X^{T} X |^{1 / 2} \propto 1 \\ p_{I J} (ϕ) & \propto ϕ^{- 1} \\ p_{I J} (β, ϕ) & \propto p_{I J} (β) p_{I J} (ϕ) = ϕ^{- 1} \end{aligned}$

Formal Posterior Distribution

Use Independent Jeffreys Prior $p_{I J} (β, ϕ) \propto p_{I J} (β) p_{I J} (ϕ) = ϕ^{- 1}$
Formal Posterior Distribution $\begin{array}{rcl} β ∣ ϕ, Y & \sim & N (\hat{β}, (X^{T} X)^{- 1} ϕ^{- 1}) \\ ϕ ∣ Y & \sim & Gamma ((n - p) / 2, ‖ Y - X \hat{β} ‖^{2} / 2) \\ β ∣ Y & \sim & t_{n - p} (\hat{β}, {\hat{σ}}^{2} (X^{T} X)^{- 1}) \end{array}$
Bayesian Credible Sets $p (β \in C_{α}) ∣ Y) = 1 - α$ correspond to frequentist Confidence Regions $\frac{x^{T} β - x^{T} \hat{β}}{\sqrt{{\hat{σ}}^{2} x^{T} (X^{T} X)^{- 1} x}} \sim t_{n - p}$
conditional on $Y$ for Bayes and conditional on $β$ for frequentist

Other priors next

Invariance and Choice of Mean/Precision

the model in vector form $Y ∣ β, ϕ \sim N_{n} (X β, ϕ^{- 1} I_{n})$
What if we transform the mean $X β = X H H^{- 1} β$ with new $X$ matrix $\tilde{X} = X H$ where $H$ is $p \times p$ and invertible and coefficients $\tilde{β} = H^{- 1} β$ .
obtain the posterior for $\tilde{β}$ using $Y$ and $\tilde{X}$
$Y ∣ \tilde{β}, ϕ \sim N_{n} (\tilde{X} \tilde{β}, ϕ^{- 1} I_{n})$
since $\tilde{X} \tilde{β} = X H \tilde{β} = X β$ invariance suggests that the posterior for $β$ and $H \tilde{β}$ should be the same
plus the posterior of $H^{- 1} β$ and $\tilde{β}$ should be the same

Exercise for the Energetic Student

With some linear algebra, show that this is true for a normal prior if $b_{0} = 0$ and $Φ_{0}$ is $k X^{T} X$ for some $k$

Zellner’s g-prior

Popular choice is to take $k = ϕ / g$ which is a special case of Zellner’s g-prior $β ∣ ϕ, g \sim N (0, \frac{g}{ϕ} (X^{T} X)^{- 1})$
Full conditional $β ∣ ϕ, g \sim N (\frac{g}{1 + g} \hat{β}, \frac{1}{ϕ} \frac{g}{1 + g} (X^{T} X)^{- 1})$
one parameter $g$ controls shrinkage
if $ϕ \sim Gamma (v_{0} / 2, s_{0} / 2)$ then posterior is $ϕ ∣ y_{1}, \dots, y_{n} \sim Gamma (v_{n} / 2, s_{n} / 2)$
Conjugate so we could skip Gibbs sampling and sample directly from gamma and then conditional normal!

Ridge Regression

If $X^{T} X$ is nearly singular, certain elements of $β$ or (linear combinations of $β$ ) may have huge variances under the $g$ -prior (or flat prior) as the MLEs are highly unstable!
Ridge regression protects against the explosion of variances and ill-conditioning with the conjugate priors: $β ∣ ϕ \sim N (0, \frac{1}{ϕ λ} I_{p})$
Posterior for $β$ (conjugate case) $β ∣ ϕ, λ, y_{1}, \dots, y_{n} \sim N ((λ I_{p} + X^{T} X)^{- 1} X^{T} Y, \frac{1}{ϕ} (λ I_{p} + X^{T} X)^{- 1})$

Bayes Regression

Posterior mean (or mode) given $λ$ is biased, but can show that there always is a value of $λ$ where the frequentist’s expected squared error loss is smaller for the Ridge estimator than MLE!
related to penalized maximum likelihood estimation
Choice of $λ$
Bayes Regression and choice of $Φ_{0}$ in general is a very important problem and provides the foundation for many variations on shrinkage estimators, variable selection, hierarchical models, nonparameteric regression and more!
Be sure that you can derive the full conditional posteriors for $β$ and $ϕ$ as well as the joint posterior in the conjugate case!