PhD Course on Bayesian Inference, May-Sept 2011

Organized by Tomas Olofsson.

Bayesian inference or Bayesian probability theory deviates from classical probability theory in the way probabilities are interpreted. In Bayesian inference the probabilities are interpreted as degrees of belief rather than as an expected frequency of occurence of a repeated expriment, which is the interpretation in classical probability theory. (The classical frequency interpretation occurs merely as a special case of the more general Bayesian interpretation.) One consequence of this is that we by using the Bayesian definition of probability will get a unified framwork for probability theory and statistical inference.

Course Book: E.T. Jaynes: Probability Theory - The Logic of Science. Cambridge University Press, 2003.

Errata

Course details (pdf)

Exercises:

• Exercises session 2
• Exercises session 3
• Exercises session 4
• Exercises session 5
• Exercises session 6
• Exercises session 7
• Exercises session 8

Schedule:

• May 20, 15-17.
• May 27, 15-17. Responsible Annea. Presentation chapter 1.
• June 8, 15-17. Rikke. Presentation chapter 2.
• June 14, 15-17. Christofer. Presentation chapter 3.
• June 22, 15-17. Pierre
• Aug 19, 15-17. Simon (chapter 6.0-6.10).
• Aug 26, 13-15. cancelled
• Sept 2, 15-17. Adrian (remaining chapter 6). Note the time!
• Sept 9, 13-15. Frederik. Presentation chapter 11.
• Sept 16, 13-15. Tomas (MCMC, see Additional Material below.)

Additional Material:

• Bayesianska Numeriska Metoder I. Material on MCMC, part I. (In swedish)
• Bayesianska Numeriska Metoder II. Material on MCMC, part II. (In swedish)
• Introduction to Monte Carlo methods by D. MacKay. More details on MCMC. (In english)
• Table of formulas. Some sums and relations that might be useful.
• bincoeff.m. A matlab function for computing binomial coefficients. (A straightforward implementation using factorials typically yield overflow even for fairly small number.)
• multinomcoeff.m. A matlab function for computing multinomial coefficients. Same comment as for the binomial coefficient. Slightly different approach.

Links to further reading:

• Probability Theory As Extended Logic. A site with many of Jaynes' articles and links to other Bayesian work. The site is maintained by Larry Bretthorst, one of Jaynes' former PhD students.
• D.S Sivia, "Data Analysis -- A Bayesian Tutorial". This small book gives an easy-to-read introduction to probability theory as an extension to logic, and can serve as a companion to Jaynes' book.
• A. Zellner, "Introduction to Bayesian Inference in Economics". A very good complement to Jaynes, contains many useful results in time series analysis and control.
• H. Jeffreys, "Theory of Probability". The book that inspired Jaynes. Is more or less superseded by Jaynes' book today, although it still contains several useful derivations and comments.
• L. Bretthorst, Bayesian Spectrum Analysis and Parameter Estimation". Shows that the periodogram of a data series is a sufficient statistic for estimating the location of a single sinusoid in additive noise when the first and second moments of the disturbances are fixed. The resulting Bayesian spectrum analysis method is shown to outperform the periodogram by an order of magnitude in terms of estimation accuracy. Bretthorst then extends the simple frequency estimation problem to parameter estimation for a general linear model.