Upload
matt-moores
View
136
Download
5
Embed Size (px)
Citation preview
R packages Potts model Bayesian computation Conclusion
bayesImageS: a case study in Bayesiancomputation using Rcpp and OpenMP
Matt Moores
OxWaSP mini-symposiumDecember 2, 2016
R packages Potts model Bayesian computation Conclusion
Outline
1 R packagesPerformance of BLAS
2 Potts model
3 Bayesian computationChequerboard Gibbs samplerPseudolikelihoodThermodynamic integrationExchange algorithmApproximate Bayesian Computation (ABC)
R packages Potts model Bayesian computation Conclusion
Why write an R package?
PortabilityTest bed for new statistical methodsBuild on existing codeResearch impactKudos
Hadley Wickham (2015) R packages
R packages Potts model Bayesian computation Conclusion
Why C++?
Most statistical algorithms are iterativeMarkov chain Monte CarloScalability for large datasets
RcppOpenMPEigen or Armadillo
Dirk Eddelbuettel (2013) Seamless R and C++ integration with Rcpp
R packages Potts model Bayesian computation Conclusion
Inline
One function at a time:�l i b r a r y ( i n l i n e )
sum_ logs ← cxx func t i on ( s igna tu re ( log_vec = "numeric" ) , p l ug in = "RcppArmadillo" , body=’arma::vec log_prob = Rcpp::as<arma::vec>(log_vec);double suml = 0.0;double maxl = log_prob.max();for (unsigned i=0; i < log_prob.n_elem; i++){
if (arma::is_finite(log_prob(i)))suml += exp(log_prob(i) - maxl);
}return Rcpp::wrap(log(suml) + maxl);
’ )
R packages Potts model Bayesian computation Conclusion
Package Skeleton
Create a new R package:package.skeleton("myPackage", path=".")
Specific skeletons for each C++ library:
Rcpp.package.skeleton("myRcppPackage")
RcppArmadillo.package.skeleton("MyArmadilloPackage")
RcppEigen.package.skeleton("MyEigenPackage")
R packages Potts model Bayesian computation Conclusion
Annotations
Rcpp wrappers generated automatically:compileAttributes("myRcppPackage")R package documentation generated automatically:roxygenize("myRcppPackage")�
/ /’ Compute the effective sample size (ESS) of the particles./ /’/ /’ The ESS is a ‘‘rule of thumb’’ for assessing the degeneracy of/ /’’ the importance distribution:/ /’ \ deqn {ESS = \ f r a c { ( \ sum_ { q=1}^Q w_q ) ^ 2 } { \ sum_ { q=1}^Q w_q ^ 2 } }/ /’/ /’’ @param log_weights logarithms of the importance weights of each particle./ /’’ @return the effective sample size, a scalar between 0 and Q/ /’’ @references/ /’’ Liu, JS (2001) "Monte Carlo Strategies in Scientific Computing." Springer’/ / [ [ Rcpp : : export ] ]double ef fec t iveSampleSize ( NumericVector log_weights ){
double sum_wt = sum_ logs ( log_weights ) ;double sum_sq = sum_ logs ( log_weights + log_weights ) ;double res = exp (sum_wt + sum_wt − sum_sq ) ;i f ( s td : : i s f i n i t e ( res ) ) return res ;else return 0;
}
R packages Potts model Bayesian computation Conclusion
Common Problems
Rcpp parameters are passed by reference (not copied):Can rely on R for garbage collectionMemory allocation is slowerCan crash R (and Rstudio (and your OS))R is not thread safe
Cannot call any R functions (even indirectly)within parallel code!
Drew Schmidt (@wrathematics, 2015) Parallelism, R, and OpenMP
R packages Potts model Bayesian computation Conclusion
Performance
A simple performance comparison:�l i b r a r y ( RcppEigen )l i b r a r y ( RcppArmadil lo )l i b r a r y (RcppGSL)l i b r a r y ( rbenchmark )
source ( system . f i l e ("examples" , "lmBenchmark.R" ,package ="RcppEigen" ) )
Fitting a linear regression model(by default, n=100000 rows × p=40 columns,with 20 repeat measurements)
R packages Potts model Bayesian computation Conclusion
Results with default R BLAS
Table: lm benchmark for Mac OS X with reference R BLAS
test relative elapsed user.self sys.self
3 LDLt 1.000 3.611 3.448 0.1647 QR 1.356 4.896 4.801 0.0778 LLt 1.421 5.130 5.007 0.123
11 gpuLm.fit 2.634 9.510 8.332 0.3951 lm.fit 3.760 13.576 13.550 0.0256 SymmEig 5.475 19.770 19.634 0.1262 PivQR 5.517 19.923 19.760 0.1629 arma 20.416 73.723 73.459 0.2554 GESDD 24.815 89.607 89.429 0.175
10 GSL 198.975 718.500 718.157 0.2725 SVD 201.717 728.401 727.958 0.423
matrix dimensions:1650 × 875
R packages Potts model Bayesian computation Conclusion
Accelerate Umbrella Framework
Table: lm benchmark for Mac OS X with vecLib
test relative elapsed user.self sys.self
3 LDLt 1.000 3.682 3.517 0.1657 QR 1.330 4.898 4.809 0.0888 LLt 1.413 5.204 5.081 0.1211 lm.fit 1.832 6.745 6.702 0.0249 arma 2.396 8.821 20.572 1.229
11 gpuLm.fit 2.655 9.776 8.757 0.3444 GESDD 3.313 12.199 22.120 0.8846 SymmEig 5.405 19.902 19.775 0.1272 PivQR 5.506 20.273 20.095 0.159
10 GSL 195.244 718.889 718.565 0.2735 SVD 198.514 730.930 730.403 0.495
Iacus, Urbanek, Goedman & Ripley (2016) R for Mac OS X FAQ, §10.5
R packages Potts model Bayesian computation Conclusion
Nested Models
β ← 0
Hidden MRF
θ : scaleD > 1 : dimension∂i : neighbourhoodψ : noise param.
p (yi | zi, ψ)p(zi | z\i, θ
)p (θ | z)p(ψ) p(θ)
y, z, θ, ψ Hidden Ising/Potts
z ∈ {1 . . . k}θ = β : inversetemperature
p(zi | z\i, β
)p (β | z) p(β)
Mixture of Gaussians
λ = {λ1, . . . , λk} :∑k
j=1 λj = 1ψ =
{µ,σ2} : noise param.
p(yi | zi, µj, σ
2j
)p (λ | z) p(λ)
R packages Potts model Bayesian computation Conclusion
Hidden Markov Random Field
Joint distribution of observed pixel intensities y = {yi}ni=1
and latent labels z = {zi}ni=1:
p(y, z|µ,σ2, β) = p(y|µ,σ2, z)p(z|β) (1)
Additive Gaussian noise:
yi|zi= j iid∼ N(µj, σ
2j)
(2)
Potts model:
π(zi|z\i, β) =exp {β
∑i∼` δ(zi, z`)}∑k
j=1 exp {β∑
i∼` δ(j, z`)}(3)
Potts (1952) Proceedings of the Cambridge Philosophical Society 48(1)
R packages Potts model Bayesian computation Conclusion
Inverse Temperature
R packages Potts model Bayesian computation Conclusion
Doubly-intractable likelihood
p(β|z) ∝ C(β)−1π(β) exp {β S(z)} (4)
The normalising constant has computational complexity O(nkn):
C(β) =∑z∈Z
exp {β S(z)} (5)
S(z) is the sufficient statistic of the Potts model:
S(z) =∑
i∼`∈Lδ(zi, z`) (6)
where L is the set of all unique neighbour pairs.
R packages Potts model Bayesian computation Conclusion
Chequerboard Gibbs
A 2D or 3D regular lattice with first-order neighbourhood ∂i:
◦ ◦ ◦ ◦ ◦◦ ◦ • ◦ ◦◦ • × • ◦◦ ◦ • ◦ ◦◦ ◦ ◦ ◦ ◦
can be partitioned into 2 blocks:
• ◦ • ◦ • ◦ • ◦ • ◦◦ • ◦ • ◦ • ◦ • ◦ •• ◦ • ◦ • ◦ • ◦ • ◦◦ • ◦ • ◦ • ◦ • ◦ •• ◦ • ◦ • ◦ • ◦ • ◦◦ • ◦ • ◦ • ◦ • ◦ •• ◦ • ◦ • ◦ • ◦ • ◦◦ • ◦ • ◦ • ◦ • ◦ •• ◦ • ◦ • ◦ • ◦ • ◦◦ • ◦ • ◦ • ◦ • ◦ •
so that z◦ are conditionally independent, given z•Roberts & Sahu (1997) JRSS B 59(2): 291–317Winkler (2nd ed., 2003) Image analysis, random fields and MCMC methods
R packages Potts model Bayesian computation Conclusion
Chequerboard Gibbs II
Algorithm 1 Chequerboard sampling for z1: for all blocks b do2: for all pixels i ∈ b do3: for all labels j ∈ 1 . . . k do4: Compute λj ← p(yi | zi = j)π(zi = j | zi∼`, β)5: end for6: Draw zi ∼ Multinomial(λ1, . . . , λk)7: end for8: end for
R packages Potts model Bayesian computation Conclusion
Gibbs sampler in C++�void gibbsLabels ( const arma : : umat & neigh , const std : : vector <arma : : uvec> & blocks ,
arma : : umat & z , arma : : umat & a l l oc , const double beta ,const arma : : mat & l og_ x f i e l d )
{const Rcpp : : NumericVector randU = Rcpp : : r u n i f ( neigh . n_rows ) ;for ( unsigned b=0; b < blocks . s ize ( ) ; b++){
const arma : : uvec block = blocks [ b ] ;arma : : vec log_prob ( z . n_ co ls ) ;
#pragma omp p a r a l l e l for pr ivate ( log_prob )for ( unsigned i =0; i < b lock . s ize ( ) ; i ++){
for ( unsigned j =0; j < z . n_ co ls ; j ++){
unsigned sum_neigh = 0;for ( unsigned k =0; k < neigh . n_ co ls ; k++){
sum_neigh += z ( neigh ( b lock [ i ] , k ) , j ) ;}log_prob [ j ] = log_ x f i e l d ( b lock [ i ] , j ) + beta∗sum_neigh ;
}double t o t a l _ l l i k e = sum_ logs ( log_prob ) ;double cumProb = 0 . 0 ;z . row ( b lock [ i ] ) . zeros ( ) ;for ( unsigned j =0; j < log_prob . n_elem ; j ++){
cumProb += exp ( log_prob [ j ] − t o t a l _ l l i k e ) ;i f ( randU [ b lock [ i ] ] < cumProb ){
z ( b lock [ i ] , j ) = 1 ;a l l o c ( b lock [ i ] , j ) += 1 ;break ;
}}
}}
}
R packages Potts model Bayesian computation Conclusion
Pseudolikelihood (PL)
Algorithm 2 Metropolis-Hastings with PL1: Draw proposal β′ ∼ q(β′|β◦)2: Approximate p(β′|z) and p(β◦|z) using equation (7):
p̂PL(β|z) ≈n∏
i=1
exp{β∑
i∼` δ(zi, z`)}∑kj=1 exp{β
∑i∼` δ(j, z`)}
(7)
3: Calculate the M-H ratio ρ = p̂PL(β′|z)π(β′)q(β◦|β′)p̂PL(β◦|z)π(β◦)q(β′|β◦)
4: Draw u ∼ Uniform[0, 1]5: if u < min(1, ρ) then6: β ← β′
7: else8: β ← β◦
9: end ifRydén & Titterington (1998) JCGS 7(2): 194–211
R packages Potts model Bayesian computation Conclusion
Pseudolikelihood in C++
�double pseudol ike ( const arma : : mat & ne , const arma : : uvec & e , const double b ,
const unsigned n , const unsigned k ){
double num = 0 . 0 ;double denom = 0 . 0 ;
#pragma omp p a r a l l e l for reduc t ion ( + :num, denom)for ( unsigned i =0; i < n ; i ++){
num=num+ne ( e [ i ] , i ) ;double tdenom =0.0 ;for ( unsigned j =0; j < k ; j ++){
tdenom=tdenom+exp ( b∗ne ( j , i ) ) ;}denom=denom+log ( tdenom ) ;
}return b∗num−denom ;
}
R packages Potts model Bayesian computation Conclusion
Approximation Error
PL for n = 12, k = 3 in comparison to the exact likelihoodcalculated using a brute force method:
0 1 2 3 4
68
10
12
14
16
β
µ
exact
pseudolikelihood
(a) Expectation
0 1 2 3 4
0.0
0.5
1.0
1.5
2.0
2.5
β
σ
exact
pseudolikelihood
(b) Standard deviation
R packages Potts model Bayesian computation Conclusion
Thermodynamic Integration (TI)
Path sampling identity:
log{C(β◦)C(β′)
}=
∫ β◦
β′E z|β [S(z)] dβ (8)
0.0 0.5 1.0 1.5 2.0
50
00
00
10
00
00
01
50
00
00
20
00
00
0
ϕ
S(x
)
Gelman & Meng (1998) Stat. Sci. 13(2): 163–185.
R packages Potts model Bayesian computation Conclusion
TI algorithm
Algorithm 3 Random walk Metropolis with TI1: Draw random walk proposal β′ ∼ q(β′|β◦)2: Estimate S(z|β◦) and S(z|β′) by interpolation3: Evaluate the definite integral in equation (8)4: Calculate the log M-H acceptance ratio:
log{ρ} = log{C(β◦)C(β′)
}+ (β′ − β◦)S(z) (9)
5: Draw u ∼ Uniform[0, 1]6: if u < min(1, ρ) then7: β ← β′
8: else9: β ← β◦
10: end if
R packages Potts model Bayesian computation Conclusion
TI in C++
�unsigned pathBeta ( const arma : : umat & neigh , const s td : : vector <arma : : uvec> & blocks ,
const arma : : mat & path , const arma : : umat & z , double & beta ,const double p r i o r _beta [ 2 ] , const double bw)
{double bprime = rwmh( beta , bw, p r i o r _beta ) ; / / t runca ted Gaussian
/ / approximate log (Z ( bprime ) / Z( beta ) )double l og_ r a t i o = quadrature ( bprime , beta , path )
+ ( bprime−beta ) ∗ sum_ i den t ( z , neigh , b locks ) ;
/ / accept / r e j e c ti f ( u n i f _ rand ( ) < exp ( log_ r a t i o ) ){
beta = bprime ;return 1;
}return 0;
}
R packages Potts model Bayesian computation Conclusion
Approximate Exchange Algorithm (AEA)
Algorithm 4 AEA1: Draw random walk proposal β′ ∼ q(β′|β◦)2: Generate w|β′ by sampling from eq. (3)3: Calculate the M-H acceptance ratio according to eq. (4):
ρ =π(β′) exp {β′S(z)} C(β◦)π(β◦) exp {β◦S(z)} C(β′)
exp {β◦S(w)} C(β′)exp {β′S(w)} C(β◦)
(10)
4: Draw u ∼ Uniform[0, 1]5: if u < min(1, ρ) then6: β ← β′
7: else8: β ← β◦
9: end if
Murray, Ghahramani & MacKay (2006) Proc. 22nd Conf. UAI, 359–366
R packages Potts model Bayesian computation Conclusion
AEA in C++�unsigned exchangeBeta ( const arma : : umat & neigh , const std : : vector <arma : : uvec> & blocks ,
const arma : : uvec & s l i c e , const arma : : umat & z , double & beta ,const double p r i o r _beta [ 2 ] , const unsigned aux , const bool useSW,const bool swapAux , const double bw)
{double bprime = rwmh( beta , bw, p r i o r _beta ) ;arma : : umat a l l o c = arma : : zeros <arma : : umat >( z . n_rows−1, z . n_ co ls ) ;arma : : umat w;i f ( swapAux ) w = z ;else w = randomIndices ( z . n_rows−1, z . n_ co ls ) ;for ( unsigned i =0; i <aux ; i ++){
i f (useSW){
swLabelsNoData ( neigh , blocks , bprime , w. n_cols , w, a l l o c ) ;}else{
gibbsLabelsNoData ( neigh , blocks , w, a l l oc , bprime ) ;}
}double sum_z = sum_ i den t ( z , neigh , b locks ) ;double sum_w = sum_ i den t (w, neigh , b locks ) ;double l og_ r a t i o = ( bprime−beta )∗sum_z + ( beta−bprime )∗sum_w;/ / accept / r e j e c ti f ( u n i f _ rand ( ) < exp ( log_ r a t i o ) ){
beta = bprime ;return 1;
}return 0;
}
R packages Potts model Bayesian computation Conclusion
Approximate Bayesian Computation
Algorithm 5 ABC rejection sampler1: Draw independent proposal β′ ∼ π(β)2: Generate w|β′ by sampling from eq. (3)3: if ‖S(w)− S(z)‖ < ε then4: β ← β′
5: else6: β ← β◦
7: end if
Grelaud, Robert, Marin, Rodolphe & Taly (2009) Bayesian Analysis 4(2)Marin & Robert (2014) Bayesian Essentials with R §8.3
R packages Potts model Bayesian computation Conclusion
ABC with Metropolis-Hastings
Algorithm 6 ABC-MCMC1: Draw proposal β′ ∼ q(β′|β◦)2: Generate w|β′ by sampling from eq. (3)3: Draw u ∼ Uniform[0, 1]4: if u < π(β′)q(β◦|β′)
π(β◦)q(β′|β◦) and ‖S(w)− S(z)‖ < ε then5: β ← β′
6: else7: β ← β◦
8: end if
Marjoram, Molitor, Plagnol & Tavaré (2003) PNAS 100(26): 15324–28
R packages Potts model Bayesian computation Conclusion
ABC-MCMC in C++�unsigned abcBeta ( const arma : : umat & neigh , const std : : vector <arma : : uvec> & blocks ,
const arma : : umat & z , double & beta , const double p r i o r _beta [ 2 ] ,const unsigned aux , const bool useSW, const bool swapAux ,const double bw, const double eps i l on )
{double bprime = rwmh( beta , bw, p r i o r _beta ) ;arma : : umat a l l o c = arma : : zeros <arma : : umat >( z . n_rows−1, z . n_ co ls ) ;arma : : umat w;i f ( swapAux ) w = z ;else w = randomIndices ( z . n_rows−1, z . n_ co ls ) ;for ( unsigned i =0; i <aux ; i ++){
i f (useSW){
swLabelsNoData ( neigh , blocks , bprime , w. n_cols , w, a l l o c ) ;}else{
gibbsLabelsNoData ( neigh , blocks , w, a l l oc , bprime ) ;}
}double sum_z = sum_ i den t ( z , neigh , b locks ) ;double sum_w = sum_ i den t (w, neigh , b locks ) ;double de l t a = fabs (sum_w − sum_z ) ;i f ( de l t a < eps i l on ){
beta = bprime ;return 1;
}return 0;
}
R packages Potts model Bayesian computation Conclusion
Summary
bayesImageS supports methods for updating the latent labels:Chequerboard Gibbs sampling (Winkler 2003)Swendsen-Wang (1987)
and also methods for updating the inverse temperature β:Pseudolikelihood (Rydén & Titterington 1998)Thermodynamic integration (Gelman & Meng 1998)Exchange algorithm (Murray, Ghahramani & MacKay 2006)Approximate Bayesian computation (Grelaud et al. 2009)Sequential Monte Carlo (ABC-SMC) with pre-computation(Del Moral, Doucet & Jasra 2012; Moores et al. 2015)
Appendix
Acknowledgements
Principal supervisor: Kerrie MengersenAssociate supervisor: Fiona Harden
Radiation Oncology Mater Centre (ROMC), Queensland Health:
Cathy Hargrave
Mike Poulsen
Tim Deegan
Other co-authors:
Christopher Drovandi (QUT)
Anthony N. Pettitt (QUT)
Christian Robert(University of Warwick & Université Paris Dauphine)
PyMCMC team at QUT:
Clair Alston
Christopher Strickland
Appendix
For Further Reading I
D. EddelbuettelSeamless R and C++ integration with RcppSpringer-Verlag, 2013.
H. WickhamR packagesO’Reilly, 2015.
D. Bates & D. EddelbuettelFast and elegant numerical linear algebra using the RcppEigenpackage.J. Stat. Soft. 52(5): 1–24, 2013.
D. Eddelbuettel & C. SandersonRcppArmadillo: Accelerating R with high-performance C++ linearalgebra.Comput. Stat. Data Anal. 71: 1054–63, 2014.
Appendix
For Further Reading II
M. Moores & K. MengersenbayesImageS: Bayesian methods for image segmentation using ahidden Potts model.R package version 0.3-3https://CRAN.R-project.org/package=bayesImageS
M. Moores, A. N. Pettitt & K. MengersenScalable Bayesian inference for the inverse temperature of a hiddenPotts model.arXiv:1503.08066 [stat.CO], 2015.
M. Moores, C. C. Drovandi, K. Mengersen & C. P. RobertPre-processing for approximate Bayesian computation in imageanalysis.Statistics & Computing 25(1): 23–33, 2015.
M. Moores & K. MengersenBayesian approaches to spatial inference: modelling and computationalchallenges and solutions.In Proc. 33rd Int. Wkshp MaxEnt, AIP Conf. Proc. 1636: 112–117, 2014.
Appendix
For Further Reading III
G. WinklerImage analysis, random fields and Markov chain Monte Carlo methods2nd ed., Springer-Verlag, 2003.
J.-M. Marin & C. P. RobertBayesian Essentials with RSpringer-Verlag, 2014.
G. O. Roberts & S. K. SahuUpdating Schemes, Correlation Structure, Blocking andParameterization for the Gibbs SamplerJ. R. Stat. Soc. Ser. B 59(2): 291–317, 1997.
T. Rydén & D. M. TitteringtonComputational Bayesian Analysis of Hidden Markov Models.J. Comput. Graph. Stat., 7(2): 194–211, 1998.
Appendix
For Further Reading IV
A. Gelman & X.-L. MengSimulating normalizing constants: from importance sampling to bridgesampling to path sampling.Statist. Sci., 13(2): 163–185, 1998.
I. Murray, Z. Ghahramani & D. J. C. MacKayMCMC for Doubly-intractable Distributions.Proc. 22nd Conf. UAI, 359–366, 2006.
A. Grelaud, C. P. Robert, J.-M. Marin, F. Rodolphe & J.-F. TalyABC likelihood-free methods for model choice in Gibbs random fields.Bayesian Analysis 4(2): 317–336, 2009.
R. H. Swendsen & J.-S. WangNonuniversal critical dynamics in Monte Carlo simulations.Physical Review Letters, 58: 86–8, 1987.