NN

T:2

018S

AC

LA04

2

Uncertainty quantification and calibrationof a photovoltaic plant model: warranty ofperformance and robust estimation of the

long-term productionThese de doctorat de l’Universite Paris-Saclay

preparee a AgroParisTech (l’Institut des sciences et industries du vivant et del’environnement)

Ecole doctorale n581 Agriculture, Alimentation, Biologie, Environnement,Sante (ABIES)

Specialite de doctorat : Mathematiques appliquees

These presentee et soutenue a Paris, le 21 Decembre 2018, par

MATHIEU CARMASSI

Composition du Jury :

Liliane BelProfesseur, AgroParisTech (MIA Paris) PresidentAmandine MarrelIngenieur-Chercheur, CEA Cadarache (DER/SESI) RapporteurOlivier RoustantProfesseur, Ecole des Mines de Saint-Etienne (LIMOS) RapporteurLuc PronzatoDirecteur de Recherche, CNRS (I3S) ExaminateurEric ParentIGPEF, AgroParisTech (MIA Paris) Directeur de thesePierre BarbillonMaıtre de Conference, AgroParisTech (MIA Paris) Co-directeur de theseMatthieu ChiodettiIngenieur, EDF (TREE) Co-encadrantMerlin KellerIngenieur-Chercheur, EDF (PRISME) Invite

REMERCIEMENTS

Je voudrais avant tout remercier mon directeur de thèse Éric Parent pour m’avoir guidé durant ces trois années dethèse. Ses qualités scientifiques et humaines ont contribué au très bon déroulement de cette thèse. Éric a su trouverun bon équilibre dans l’encadrement, en étant très présent au début lorsque le besoin s’en ressentait et en me laissantune certaine marge de manœuvre par la suite. Je tiens à souligner sa constante bonne humeur et son côté paternelqui m’a soulagé de la pression lorsque celle-ci se faisait trop importante.

Ensuite, je tiens à remercier mon co-directeur de thèse Pierre Barbillon pour m’avoir aiguillé scientifiquementdurant ces trois années. J’ai pu bénéficier de ses grandes qualités scientifiques à n’importe quel moment et leséchanges que nous avons eus se sont toujours révélés très productifs. Son perfectionnisme m’a aussi amené à nerien négliger que ce soit au niveau rédactionnel ou scientifique. J’ai pu apprécier sa personnalité au quotidien, quece soit pendant les soirées à Rochebrune, pendant les réunions de travail ou pendant les pauses-café dans le bureaudes doctorants. Une chose est sûre c’est que les jeux de mots quasi quotidiens vont me manquer.

Je tiens aussi à adresser mes remerciements à Merlin Keller. Il est indéniablement la raison pour laquellej’ai pu en arriver là. Sa constante bonne humeur, sa modestie et ses qualités scientifiques ont été la source d’unecollaboration qui dure depuis mon stage de fin d’année d’école d’ingénieurs. J’ai pu le solliciter autant de fois queje le souhaitais pendant la thèse et même si je ne travaillais pas sur le même site, il a toujours été là pour m’aider. Àchaque réunion que nous avons eue, j’ai pu retrouver chez lui un juste recul sur les problèmes industriels avec unegrande connaissance mathématique.

Je remercie également Amy Lindsay qui m’a encadré avec une grande qualité jusqu’à sa mutation qui estintervenue au bout d’un an de thèse. C’est aussi pour cela que je remercie grandement Matthieu Chiodetti qui s’estretrouvé avec ma thèse sur les bras. Son encadrement a été d’une très grande qualité et il a su m’orienter sur lesbesoins industriels précis d’EDF et son expertise dans le domaine m’a permis de prendre du recul sur les résultatsque l’on pouvait avoir. De manière générale, j’ai pu tisser avec chacun des membres de mon encadrement des liensd’amitié qui se sont retranscrit dans des conférences ou écoles d’été (Rochebrune, ETICS,...) où nous étions parfoisamenés à sortir du cadre professionnel.

J’adresse également mes remerciements à Amandine Marrel et Olivier Roustant pour avoir accepté de rapporterma thèse. Le travail sur les rapports a grandement été apprécié. Je remercie aussi Liliane Bel d’avoir présidé masoutenance de thèse et Luc Pronzato d’avoir accepté de faire partie de mon jury en tant qu’examinateur.

Mes prochains remerciements seront pour mes collègues d’EDF. Même si le site des Renardières se trouvait trèsloin de mon domicile, c’était toujours avec un grand plaisir que je retrouvais tout le groupe. J’ai toujours appréciéles échanges, à table ou pendant les pauses, que nous avons eu. Je remercie tout particulièrement mon laboratoireMIA d’AgroParisTech qui m’a apporté énormément durant cette thèse. Au-delà des compétences mathématiquesdont j’ai pu bénéficier, j’ai pu tisser des liens d’amitiés forts avec d’autres doctorants. Merci Timothée, Marie P,Pierre G (déjà vieux docteur), Félix, Rana, Paul, Anna, Marie C, Loïc d’avoir animé, à un moment ou un autre, lebureau des doctorants. Les afterworks à la Montagne, au Vieux Chêne, ou dans la rue Mouffetard nous ont permisde décompresser avec des journées de travail bien chargées.

3

Je tiens bien entendu à remercier toute ma famille : ma mère Marie-Christine, mon père Patrick, mon frèreGuillaume et ma grand-mère Jeanine. Je pense sincèrement que le cadre familial qu’ils ont établi m’a conduit là oùje suis aujourd’hui et je n’aurais pas pu rêver d’un meilleur équilibre de vie.

Je tiens désormais à remercier tous mes amis qui m’ont épaulé de près ou de loin pendant la thèse. D’abord,je remercie Quentin Huchet qui a été dans la même galère que moi et avec qui j’ai partagé quelques écoles d’étéqui sont devenues mythiques (Porquerolles, Roscoff, ...). Merci de m’avoir permis de rigoler profondément à desmoments où j’en avais besoin, d’avoir tourné en dérision les situations compliquées, d’avoir toujours eu le motmarrant pour dédramatiser le contexte. Merci à Lambert pour les pauses printemps de Bourges ou pour les quelquesweek-ends pétanque, hors du temps, à Paris. Merci à Albin pour les pauses rugby au bar ou les brainstormingsnon fructueux qui se terminaient quasi systématiquement en bières/rugby. Je remercie toute la clique de l’IFMA: Tareck, Pilou, Vely, Alexia, ViVi, MiMi, Norman, Naf, Matthieu, Ludo pour les week-ends de retrouvailles oules réveillons du nouvel an passés ensemble. C’était aussi agréable de se retrouver des soirs aux Lombards, MacBride, Halls Beer, et autres tavernes. Les moments pendant la coupe du monde sur cette petite place pleine à craquerresteront aussi des moments marquants.

La conclusion de ces remerciements ne peut concerner que la personne qui m’a épaulé durant ces trois dernièresannées. Je remercie Camille pour m’avoir soutenu et supporté comme elle l’a fait même dans les moments où jedevenais insupportable. Merci de m’avoir poussé à faire des breaks qui m’ont permis de mieux repartir après. Mercipour ces voyages qui n’appartiennent qu’à nous et qui m’ont permis de m’évader le temps de quelques semaines.Merci de ne m’avoir jamais laissé tomber au fond du trou et de m’avoir poussé à repartir rapidement après. Mercipour ta joie de vivre et ta constante foi en moi qui me redonnait confiance dans les mauvais moments. Merci d’avoirrendu cette thèse plus facile.

4

CONTENTS

Remerciements 3

List of figures 9

Acronyms 11

Résumé 13

1 Introduction 191.1 Economic issue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

1.2 Physical phenomenon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

1.3 Several modeling approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

1.3.1 A first simple model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

1.3.2 Advanced electrical models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

1.4 Numerical codes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

1.4.1 General framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

1.4.2 Sources of uncertainties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

1.4.3 Python code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

1.4.4 Dymola code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

1.5 Thesis organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

2 Statistical tools for numerical code calibration 312.1 Sensitivity analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

2.1.1 Morris method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

2.1.2 Sobol indices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

2.2 Kriging / Gaussian processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

2.2.1 General framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

2.2.2 Parameter estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

2.2.3 Covariance functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

2.2.4 Gaussian process-based optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

2.3 Design of experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

2.3.1 Sampling criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

2.3.2 Distance between the points criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

2.4 Principal component analysis (PCA) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

2.4.1 Distance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

2.4.2 Moments of inertia . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

2.4.3 Axis of minimum inertia . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

2.4.4 Contribution to the total inertia . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

2.4.5 Graphical representations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

2.5 Monte Carlo Markov Chains techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

2.5.1 Gibbs sampler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

5

CONTENTS

2.5.2 Metropolis Hastings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 632.5.3 Metropolis within Gibbs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 652.5.4 Improvements of the Metropolis Hastings . . . . . . . . . . . . . . . . . . . . . . . . . . 65

3 Review of the main calibration methods 693.1 Numerical code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

3.1.1 Sensitivity analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 703.1.2 Prior propagation of uncertainty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

3.2 Calibration through statistical models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 733.2.1 Presentation of the models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 743.2.2 Likelihood . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 763.2.3 Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80

3.3 Application to the prediction of power from a photovoltaic (PV) plant . . . . . . . . . . . . . . . 823.3.1 Inference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 823.3.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 853.3.3 Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

3.4 Conclusion and discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88

4 CaliCo: a R package for Bayesian calibration 914.1 Guidelines for users . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 924.2 Multidimensional example with CaliCo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97

4.2.1 The models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 984.2.2 Priors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1024.2.3 Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1034.2.4 Additionnal tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108

4.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111

5 Performance monitoring on a large PV plant 1135.1 Sensitivity analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1135.2 Prior densities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1175.3 Propagation of uncertainties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1185.4 Bayesian calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118

5.4.1 Statistical models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1185.4.2 Modular estimation and likelihoods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1205.4.3 Application to the PV plant . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121

6 Conclusion and perspectives 129

Bibliography 136

6

LIST OF FIGURES

1.1 p-n junction and the equivalent electrical component (source: Raffamaiden – CC By SA). . . . . . 21

1.2 n side doping. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

1.3 p side doping. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

1.4 Displacement of an electron in the silicon (source: Freshman404 – CC By SA). . . . . . . . . . . 22

1.5 On the left panel, the electrical equivalence with 1 diode where IPV stands for a photo-current thatdepends on the incident sun rays, ID for the saturation current of the ideal diode, RP the shuntresistance, RS the series resistance representing losses proportional to I, I the current and V thevoltage generated by the cell. On the right panel the electrical equivalence with 2 diodes where ID1

and ID2 stands for the saturation current of both ideal diodes. . . . . . . . . . . . . . . . . . . . . 23

1.6 I/V curve of a toy example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

1.7 The power production by PVzen for August 2014 (on the left) and the power production averagedby hour for August 25th 2014 (on the right). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

1.8 On the top left, the original scaled power production gathered on the the PV plant during the year2015. On the top, right the same data but only on the first week. On the bottom left, the originaldata but averaged by hour. On the bottom right, only the positive power is kept among the origin data. 29

2.1 Major steps in uncertainty treatment for industrial matters (source: Bertrand Iooss – ENBIS-EMSE

2009 Conference). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

2.2 Sampling grid on the scaled space. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

2.3 On the left panel a result of Morris method on the Morris function and on the right panel 10repetitions of the method. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

2.4 Scatter plot of the Morris indices given by the 1500 iterations bootstrap. . . . . . . . . . . . . . . 39

2.5 Scatter plots of the Ishigami function where the output is given function of the each parameter. . . 41

2.6 Sobol’s index computed for the Ishigami function and the boxplots representing the variability of1000 bootstrap iterations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

2.7 Different Gaussian process emulations for the toy function defined Equation (2.30). On the firstpanel (on the left) the Gaussian process is estimated with σ2 = 1 and ψ = 0.1. On the second panel(on the middled left), σ2 = 5 and ψ = 0.1. On the third panel (on the middle right), σ2 = 1 andψ = 0.2. And, on the fourth panel (on the right), σ2 = 5 and ψ = 0.2. . . . . . . . . . . . . . . . 44

2.8 Different Gaussian process estimation for the toy function Equation (2.30) with σ2 = 5 and ψ = 0.1but with different covariance functions. On the first panel (on the left) the Gaussian process isestimated with Gaussian covariance function. On the second panel (on the middled left), with aMatérn 5/2. On the third panel (on the middle right), with a Matérn 3/2. And, on the fourth panel(on the right), with an exponential. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

2.9 Expected improvement computed for the Gaussian process established on the function definedEquation (2.30) with 5 points in the original design of experiments. . . . . . . . . . . . . . . . . . 49

2.10 2 EGO iterations with on top the GP updated based on the previous point found with the EI criterionand on the bottom the EI values corresponding to the GP on top. The point in orange is the EImaximum used to establish the following GP. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

7

LIST OF FIGURES

2.11 2 last of the 6 iterations of the EGO algorithm. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

2.12 6 points sampled with a LHS for Q = [0,1]2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

2.13 6 points sampled with a LHS for Q = [0,1]2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

2.14 2 6-sized maximin LHS performed with the algorithm of Morris and Mitchell (1995) for 2 parameters. 53

2.15 Graphical representation of the individuals (on the left) and the variables (on the right) of thedecathlon data set. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

2.16 Gibbs sampler completed for 10000 iteration with a 1000 burn-in sample. . . . . . . . . . . . . . 63

3.1 On the left panel Morris method at noon the 24th of September 2014 and all the EEs computed ateach time step over the two months of data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

3.2 Results of the PCA done on the trajectories of the Morris DOE. On the right panel the correlationcircle and on the right panel the eigenvalues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

3.3 Projection on the PCA axis of the Morris indices. . . . . . . . . . . . . . . . . . . . . . . . . . . 71

3.4 Sobol method completed for each time steps. On the left panel the first order indices and on theright panel the total effects indices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

3.5 π(η), π(µt) and π(ar) prior densities (represented on the left panel) and induced credibility intervalof the instantaneous power (right panel). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

3.6 Directed Acyclic Graph (DAG) representation of the different models. . . . . . . . . . . . . . . . 76

3.7 Prior (in blue) and posterior (in red) densities of η , µt , ar and σ2err for each model. On the two first

column the two first models (without and with surrogate) which have only these four parameters toestimate. The two other columns represent the third and the fourth models which have two moreparameters to estimate (see Figure 3.9). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

3.8 Correlation representation between the parameters. . . . . . . . . . . . . . . . . . . . . . . . . . 86

3.9 Prior (in blue) and posterior (in red) densities of σ2δ

and ψδ for M3 and M4. . . . . . . . . . . . 86

3.10 Calibration results for M2 and M4 that are using Gaussian processes build on a DOE extended bythe sequential design. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88

4.1 Displacement of the oscillator simulated. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97

4.2 Experimental data displayed when no parameter values are set in the model. . . . . . . . . . . . . 99

4.3 First and second model output for prior belief on parameter values. The left panel illustrates thefirst model and the right panel the second model with the Gaussian process estimated. . . . . . . . 101

4.4 Third and fourth model output for prior belief on parameter values. The left panel illustrates thethird model and the right one, the fourth model with the Gaussian process estimated. Both areencompassing the discrepancy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102

4.5 M4 displayed for some guessed values with the CI relative to the measurement error on the leftpanel, with the CI relative to the Gaussian process only on the middle panel and both credibilityintervals on the right panel. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102

4.6 Prior distributions for each parameter to calibrate in the application case. . . . . . . . . . . . . . . 103

4.7 Series of plot generated by the function plot for calibration on M1. . . . . . . . . . . . . . . . . 105

4.8 prior and posterior distributions for each parameter for calibration on M4. . . . . . . . . . . . . . 107

4.9 Result of calibration on M4 for the quantity of interest with the credibility interval at 95% a posteriori.107

4.10 Series of plot generated by the function plot for the sequential design on M2. . . . . . . . . . . . 111

5.1 The PCA performed on the results given by the Morris method. On the left the correlation circleand on the right the eigenvalues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114

5.2 Morris indices in the new space given by the PCA. On the left all the parameters names appear andon the right only the ones that are not overlapping are displayed. . . . . . . . . . . . . . . . . . . 114

8

LIST OF FIGURES

5.3 On the top left, the impact of different values of shunt resistances on the I/V curve. On the top rightthe impact of different values of shunt resistances on the evolution of the efficiency function ofthe irradiance. On the bottom left, the impact of different values of series resistances on the I/Vcurve. On the bottom right, the impact of different values of series resistances on the evolution ofthe efficiency function of the irradiance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116

5.4 Illustration of the inverter performance model and the factors describing the relationship of the ac-output to both dc-power and dc-voltage (source: https://energy.sandia.gov/wp-content/gallery/uploads/Performance-

Model-for-Grid-Connected-Photovoltaic-Inverters.pdf). . . . . . . . . . . . . . . . . . . . . . . . 1165.5 Prior densities for each parameter considered for further calibration. . . . . . . . . . . . . . . . . 1175.6 Propagation of uncertainties based on prior elicitation. On the top experimental data over 10 days

in 2015 are displayed with the credibility interval a priori and on the bottom, to zoom on thephenomenon, only one day has been plotted (28th of January 2015). . . . . . . . . . . . . . . . . 118

5.7 The PCA performed on the outputs gotten from the DOE of 300 points. On the left, the correlationcircle and on the right, the eigenvalues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121

5.8 The irradiation and the correlation of the power recorded with the three first PCA axes for the 27first days. On the top the scaled irradiance, on the middle top the correlation of power recorded withthe first axis given by the PCA. On the middle bottom, the correlation between the recorded powerand the second PCA axis, and on the bottom, the correlation with the third PCA axis. . . . . . . . 123

5.9 Correlation lengths for each component of the parameter vector θ and for the variance of the 5Gaussian processes on each PCA axis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124

5.10 Prior and posterior densities for each parameter in a M ′2 calibration. . . . . . . . . . . . . . . . . 125

5.11 Prior and posterior densities for each parameter in a M ′4 calibration. . . . . . . . . . . . . . . . . 126

9

ACRONYMS

PVEDFLCOECSPCREOPEXCAPEXUTCUQV & VSAHSICOATEETSIiid

LHSLHDEGOGPBLUPEBLUPEIDOEMSTSFDESEPCAMCMCMLEDAGSMLEMAPCVRMSECRAN

PhotovoltaicÉlectricté De FranceLevalized Cost Of EnergyConcentrate Solar PowerEnergy Regulation CommissionOperational ExpendituresCapital ExpendituresUniversal Time CoordinatedUncertainty QuantificationVerification and ValidationSensitivity AnalysisHilbert-Schmidt Independence CriterionOne At a TimeElementary EffectTotal Sensitivity Indexindependent and identically distributedLatin Hyperspace SamplingLatin Hyperspace DesignEfficient Global OptimizationGaussian processBest Linear Unbiased PredictorEmpirical Best Linear Unbiased PredictorExpected ImprovementDesign Of ExperimentsMinimum Spanning TreeSpace Filling DesignEnhanced Stochastic EvolutionaryPrincipal Component AnalysisMonte Carlo Markov ChainMaximum Likelihood EstimatesDirected Acyclic GraphSeparated Maximum of Likelihood EstimationMaximum A Posteriori

Cross ValidationRoot Mean Square ErrorComprehensive R Archive Network

11

RÉSUMÉ

Dans la plupart des industries, l’accès aux expériences de terrain peut s’avérer coûteux économiquement et trèschronophage dans certains cas. En effet, lorsque des tests sur des structures très volumineuses sont à réaliser oulorsque les phénomènes à observer dépendent du temps, ces essais deviennent alors des enjeux majeurs pour lessociétés qui les conçoivent. Des codes numériques, représentant les phénomènes physiques en jeux, sont alorsconçus pour diminuer les coûts. Cependant, un code numérique n’est qu’une représentation de ce qu’est la réalité.Il doit être en accord avec les résultats expérimentaux. C’est pour cela que l’on ne peut pas dissocier le codenumérique des expériences de terrain.

Les codes numériques présentent dès lors deux avantages. Ils sont, en effet, plus rapides pour obtenir des résultatsque l’expérimentation réelle et ils représentent, notamment, un moindre coût pour les industriels. Cependant, pourque les codes soient les plus représentatifs possible de la réalité, les ingénieurs les ont développés et perfectionnésà un tel point que leurs exécutions prend un temps non négligeable. Ce temps est, certes, diminué par rapport àl’expérimentation réelle, mais il assez pour remettre en question l’utilisation de méthodes qui nous permettraientd’obtenir des résultats sur la fiabilité du système. De plus, le code peut aussi transporter une erreur (appelée erreurde code ou discrépance) qui représente les difficultés du code à reproduire le système physique réel.

À EDF (Électricité De France), des codes numériques sont utilisés dans tous domaines (le nucléaire, l’éolien,l’hydraulique, le photovoltaïque, etc...). Ces domaines font appel à des codes numériques basés sur la simulationde phénomène physique qui comprennent, notamment, la thermohydraulique, l’hydraulique, la neutronique, lamécanique des fluides, la mécanique continue, la mécanique vibratoire, l’écotoxicologie, la thermique, etc... Danscertains domaines, des codes dits d’“échelle” peuvent être utilisés comme les codes éléments ou volumes finis, oudes codes système dits 0D/1D. Les enjeux de l’utilisation de tels codes numériques se concentrent sur la sûreté des in-stallations, l’environnement, la distribution ou la production. Dans le cas du photovoltaïque (PV), le code numériquepeut être utilisé à bien des égards (les smarts grids, smart cities, les offres de service liés à l’autoconsommation, lesétudes de dégradations, la physique du panneaux photovoltaïque, etc...). Dans la thèse, nous nous intéressons àdeux cas particuliers dans l’utilisation des codes numériques. Le premier concerne l’établissement du business pland’une centrale PV avant sa construction. En France, la CRE (Commission de Régulation de l’Énergie) lance unappel à projet pour la construction d’une centrale PV et les énergéticiens comme EDF doivent calculer les coûtsd’un tel projet. Pour ce faire, il faut connaître les coûts de construction et de maintenance de la centrale puis laproduction sur sa durée de vie ce qui permet ensuite de fixer un prix de facturation pour l’électricité produite parcette centrale. Pour calculer la production totale d’électricité de la centrale, EDF utilise un code numérique, qu’ilsait imprécis. Pour chiffrer le projet, EDF applique un coefficient qui tend à sous estimer la sortie du code pourdiminuer les risques financiers. Pour augmenter la rentabilité du projet et diminuer ces risques financiers, il convientdonc de connaître avec plus de précision quelles sont les incertitudes introduites dans le code numérique. Dans undeuxième temps, le code PV peut être utilisé à des fins de suivi de performances. En effet, lorsqu’une centrale estd’ores et déjà construite et que des données de production sont disponibles, le code numérique basés sur les donnéesde production est utilisé pour rectifier les prédictions sur les années suivantes. Dans les deux cas, le calage de codenumérique représente un enjeu majeur pour l’obtention de résultats plus complets qui permettent de prendre une

13

décision financière basée sur plus d’informations.

Le calage de code

Lorsque les données expérimentales de terrains sont prélevées, une erreur de mesure est à prendre en compte. Eneffet, les tolérances des capteurs et l’imprécision des outils de mesure créent un bruit additionnel. De plus, si l’onconsidère le phénomène physique comme une fonction déterministe ξ (Sacks et al., 1989) dépendant uniquementde variables dites de contrôle, l’équation suivante peut être écrite :

∀i ∈ J1, . . . ,nK Yexpi = ξ (Xi)+ εi, (1)

où Yexpi est le ime point de mesure parmi les n, εi est un bruit blanc Gaussien tel que εiiid∼ N (0,σ2

err) (oùtous les n εi sont choisis indépendamment et identiquement distribués), ξ représente le phénomène physique réelcorrespondant à la ime mesure, etXi le vecteur des variables de contrôle qui correspondent aux variables observéeset non modifiables (comme les données environnementales par exemple).

Le calage de code permet de mieux quantifier et d’estimer les valeurs des paramètres en entrée de code parrapport à des données de terrain. Considérons un code numérique fc qui possède deux types d’entrées: les variablesde contrôle (X , définies précédemment) et les paramètres (θ). Les paramètres sont généralement des constantesphysiques implémentées dans les équations sous-jacentes au code numérique. L’hypothèse que le code représenteparfaitement le phénomène physique réel à condition de connaître la “vraie” valeur de θ est faite dans un premiertemps. Ainsi, une nouvelle représentation des expériences de terrain peut être écrite comme il suit :

M1 : ∀i ∈ J1, . . . ,nK Yexpi = fc(Xi,θ)+ εi. (2)

Cependant lorsque le code fc est long à être exécuté, l’utilisation de méthodes statistiques sur le modèleprécédent n’est pas envisageable. Afin de palier ce problème, Sacks et al. (1989) proposent de mettre en place unprocessus Gaussien en remplacement au code. Dans le cadre du calage Cox et al. (2001), ont eu l’idée d’introduireun processus Gaussien dans le modèle statistique :

M2 : ∀i ∈ J1, . . . ,nK Yexpi = Fc(Xi,θ)+ εi, (3)

où Fc est un processus Gaussien définit tel que Fc(•,•)∼PG(

mS(•,•,•,•),cS(•,•,•,•))

(avec mS

la moyenne du processus Gaussien souvent considérées comme une forme linéaire avec un vecteur de coefficient βà estimer et cS la fonction de covariance du processus qui dépend d’une variance σ2

S , d’un noyau de corrélationet d’un vecteur ψ qui représente les longueurs de corrélation dans le noyau). Cependant, comme il a été intro-duit précédemment une discrépance peut apparaître avec l’introduction du code numérique en remplacement duphénomène physique. Des articles comme Higdon et al. (2004), Kennedy and O’Hagan (2001) et Bayarri et al.(2007) suggèrent d’introduire cette discrépance et d’effectuer le calage en considérant celle-ci comme étant uneréalisation d’un processus Gaussien. Le fait de considérer la discrépance comme un processus Gaussien permet dedétecter et de quantifier les erreurs structurelles qui seraient présentes dans les données expérimentales. Si le coden’est pas coûteux et que l’on ajoute une discrépance le modèle devient alors :

M3 : ∀i ∈ J1, . . . ,nK Yexpi = fc(Xi,θ)+δ (Xi)+ εi, (4)

où δ représente la discrépance telle que δ (•)∼PG(

mδ (•,•),cδ (•,•))

. Dans un cadre où le code numériqueest considéré comme coûteux, le modèle qui généralise les deux précédents peut s’écrire :

14

M4 : ∀i ∈ J1, . . . ,nK Yexpi = Fc(Xi,θ)+δ (Xi)+ εi. (5)

Un des enjeux majeur du calage est l’estimation des paramètres. Plus le modèle est complexe et plus l’estimationdes paramètres est compliquée. En effet, dans chaque modèle statistique vient s’ajouter, aux paramètres θ ducode, les paramètres dit de nuisances qui sont les variances σ2

err, σ2S , σ2

δet les vecteurs ψS, ψδ . Cela complexifie

l’estimation qui peut s’effectuer de plusieurs manières (par la méthode des moindres carrés, l’inversion directe, larégression quantile, etc...), mais qui est souvent réalisée de deux sortes : par maximum de vraisemblance (qui permetune estimation simple) ou par estimation bayésienne (si un besoin de régularisation est nécessaire). Le maximumde vraisemblance est utilisé notamment par Cox et al. (2001) pour effectuer l’estimation des paramètres de M2 et(Wong et al., 2017) ont étendu ces résultats au cas M4. En calage bayésien, deux méthodes s’opposent. Higdonet al. (2004) suggèrent d’effectuer une estimation a posteriori directement sur la vraisemblance complète (pourl’écriture des vraisemblance se référer à la section 3.2.2) alors que Kennedy and O’Hagan (2001) et Bayarri et al.(2007) propose une estimation en deux temps appelée “approche modulaire” par Liu et al. (2009). Cette méthodepermet de séparer en deux la méthode d’estimation classique (utilisée par Higdon et al. (2004)), afin de réduire lestemps de calcul. Il s’agit de trouver des estimateurs des paramètres de nuisances pour le processus Gaussien Fc etd’utiliser ces estimateurs dans la vraisemblance conditionnelle (plus de précision à la section 3.2.3).

A des fins de comparaisons, nous possédons un code numérique rapide qui reproduit la puissance instantanée dustand de test expérimental de la R&D d’EDF composé de 12 panneaux nommé “PVzen”. Grâce à sa flexibilité et àsa rapidité, ce code nous permet de reproduire les différents cas évoqués précédemment. Suite à cette comparaison,plusieurs conclusions émergent. La première concerne l’importance de la discrépance. En effet le calage du codedans le cas M1 indique que la valeur la plus probable de la variance de l’erreur de mesure doit être bien plus élevéeque ce que l’on pensait a priori. Cependant, cette valeur indiquée par le calage n’a aucun sens physique puisqu’elleest trop élevée pour être plausible. En effectuant le calage avec M3, on remarque que la valeur de la variance del’erreur de mesure diminue pour être cohérente avec l’a priori. La sur-estimation de σ2

err était due à la présenced’une erreur de code qui n’était pas prise en compte. Dans le cas où l’on considère le code comme coûteux, le pland’expériences pour établir le processus Gaussien doit être limité (nous avons fait le choix d’un plan de 50 points).Le calage pour le modèle M2 donne alors une incohérence dans les valeurs estimées par rapport aux densités a

priori. En effet le fait de prendre un processus Gaussien pas très performant dégrade la qualité du calage. Il estdonc important de ne pas négliger la qualité du processus Gaussien précédent le calage. Dans cette perspective,l’application d’une méthode d’établissement d’un plan d’expériences basé sur le critère EI (Expected Improvment)(Damblin et al., 2018)), permet d’améliorer les résultats.

CaliCo

Le codage d’un package, appelé CaliCo, en R a été effectué pour le calage bayésien. Ce qui diffère avec lesprécédents packages mis en ligne sur le site du CRAN (Comprehensive R Archive Network), c’est que CaliCo sebase sur les quatre modèles introduits précédemment et offre la possibilité à l’utilisateur d’utiliser au même titrechacun des modèles pour son code numérique et non pas uniquement M4. L’établissement du processus Gaussienpeut aussi être automatiquement géré dans le package avec la possibilité d’effectuer un calage séquentiel (Damblinet al., 2018). La majeure partie des algorithmes MCMC (Monte Carlo par Chaînes de Markov) sont implémentésen C++ ce qui rend leurs utilisations plus rapides. De plus, beaucoup d’outils de visualisation en ggplot2 ont étéajoutés pour donner à l’utilisateur un rapide accès à des graphiques qu’il pourra lui même modifier à sa guise.

15

Suivi de performances

L’application du calage bayésien dans un cadre industriel est mis en application sur une centrale PV de grandetaille. Nous possédons pour cela un autre code numérique qui est plus performant que celui que nous avions utiliséprécédemment. Il est ainsi plus coûteux en temps de calcul mais plus précis pour estimer la puissance PV dansdes conditions particulières (ombrages, effets de missmatch, etc...). Ce code produit en sortie une série temporellesur un an de puissances instantanées. Le calage s’appliquait jusqu’à présent à des sorties scalaires. Higdon et al.(2008) a introduit le calage de code sur une sortie multidimensionnelle en réalisant notamment une projection surd axes qui portent plus de 99% de l’information donné par une ACP (Analyse en Composante principales). Laqualité d’une telle projection est étudiée ainsi que l’erreur faite en projetant les données sur les axes de l’ACP.L’ajout d’une discrépance a aussi été faite dans l’espace de l’ACP. Ce travail a abouti à l’écriture de deux modèlessupplémentaires, dont celui avec discrépance, s’écrit:

M ′4 : PYexp = PF

fc1(θ)

...fcd (θ)

+Pδ δ +E, (6)

où P est la matrice de passage entre l’espace de l’ACP et l’espace physique, Yexp sont les puissances relevées, fci

sont les projections émulées par des processus Gaussien sur les d axes de l’ACP, Pδ la matrice qui comporte lesT −d derniers vecteurs propres contenus dans P , la matrice PF représente celle composée des d premier vecteurspropres contenus dans P et E le vecteur aléatoire des bruits de mesure. Ce modèle mis en application nous permetd’obtenir des résultats de calage du code numérique coûteux afin d’actualiser les prédictions de puissance sur lesannées suivantes.

Conclusions et perspectives

En conclusion, cette thèse se focalise sur les méthodes de calage bayésien. L’objectif était d’améliorer les con-naissances en les paramètres afin de rendre l’estimation de la sortie du code plus robuste. Cela présente un intérêtéconomique fort puisque le fait de mieux estimer la puissance générée par une centrale photovoltaïque permet demoins prendre de risques financiers que ce soit lors de l’établissement d’un business plan ou lorsque l’on veut mettreà jour des prévisions de productions. Le calage bayésien permet, à partir de données de terrain, de mieux connaîtrela loi de probabilité des paramètres pour ainsi mieux prendre des décisions.

Dans cette thèse nous avons effectué une revue des principales méthodes présentes dans la littératures. À l’aided’un code numérique peu coûteux en temps de calcul, nous avons pu mettre en place une comparaison des différentsmodèles introduits. Les conclusions que nous avons pu en tirer sont que l’introduction de la discrépance danscertains cas peut s’avérer importante. En effet, lors de l’estimation des densités a posteriori des paramètres, undécalage par rapport à la densité a priori de la variance du bruit de mesure peut s’effectuer. Cela pourrait avoir dusens si l’a priori n’était pas bon, cependant si l’augmentation de cette variance n’est pas justifiée d’un point de vuephysique, il est possible que la non prise en compte de la discrépance fausse le résultats. De plus lorsque le codeest coûteux et afin de réaliser les méthodes statistiques présentées, des émulateurs ou méta-modèles peuvent êtreutilisés. Là aussi, la précaution doit être d’usage lorsque l’on tente de reproduire le code numérique. En effet, nousavons constaté que si le méta-modèle n’était pas d’une qualité suffisante, cela crée un décalage dans les modes a

posteriori.

Un travail de développement informatique a également été réalisé durant cette thèse. Le package CaliCo permetde réaliser un calage bayésien avec une multitude de code numérique ou partir de plan d’expériences. Il offre une

16

flexibilité du choix du modèle pour l’utilisateur. De plus, un codage des MCMC en C++ permet d’accélérer lesparties d’estimation qui sont chronophages. Des outils de visualisation basés sur ggplot2 permettent aussi de tirerprofit des réalisations du package sans difficultés.

Un dernier cas d’étude, basé sur des données de centrale photovoltaïque de grande capacité de production,a enfin été partiellement traité. Le code numérique utilisé dans ce cas est chronophage car il est basé sur desoptimisations informatiques qui alourdissent le temps de calcul mais qui améliorent ses performances. La sortie dece code est une série temporelle ce qui ne permet pas d’appliquer les différents modèles introduits précédemment.Ce problème a abouti à la formalisation de deux nouveaux modèles qui permettent, à partir d’une ACP, de trouverune sous espace vectoriel orthonormé dans lequel le calage peut être effectué. Cette formalisation a été appliquée aucas d’étude et nous a permis d’estimer les densités a posteriori des paramètres du code mais aussi de la variance deserreurs de mesures ainsi que la variance de la discrépance. La valeur de la variance de l’erreur de mesure se retrouveplus élevée que ce que l’on attendait a piori lorsque l’on utilise le modèle sans discrépance. Ce décalage est rattrapépar l’ajout de la discrépance et nous permet de conclure que l’apport de la discrépance a permis d’expliquer uneerreur qui s’était retrouvée dans la variance de l’erreur de mesure et qui représentait une erreur de code.

Cependant, les aspects prédictifs du modèle utilisant l’ACP reste à être démontré. Une validation croisée auraitpu être effectué sur un mois de données. La nécessité d’ajouter une discrépance est très discutée dans beaucoup depapiers (Kennedy and O’Hagan, 2001; Bayarri et al., 2007; Higdon et al., 2004) et fait l’objet de la validation demodèle statistique basée sur le facteur de Bayes dans Damblin et al. (2016). Une validation statistique à l’aide d’unmodèle de mélange peut aussi être envisagé comme le propose Kamary (2016). La remise en question sur la qualitédu processus Gaussien en tant qu’émulateur de code interroge sur la nécessité de prendre un plan d’expériencesbien fourni. Des travaux comme Damblin et al. (2018) permettent dans ce cas d’améliorer le plan d’expériences envue du calage bayésien. Les méthodes permettant d’utiliser des codes à sorties multidimensionnelles dans le cadredu calage découle d’un article fondateur (Higdon et al., 2008) mais ne restent pas très développés en pratique. Unprocessus Gaussien multi-fidélité pourrait aussi être envisagé en remplacement du code à sortie multidimensionnelleet ainsi être intégré dans le calage de code.

17

Intr

oduc

tion

INTRODUCTION

1.1 Economic issue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

1.2 Physical phenomenon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

1.3 Several modeling approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

1.3.1 A first simple model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

1.3.2 Advanced electrical models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

1.4 Numerical codes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

1.4.1 General framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

1.4.2 Sources of uncertainties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

1.4.3 Python code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

1.4.4 Dymola code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

1.5 Thesis organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

In many industrial fields, numerical experiments have become more and more popular over the last few years.Field experiments are often really expensive and getting results from the real phenomenon is quite long. To limitthis investment, numerical simulations are run as a substitute for field experiments (Santner et al., 2013; Fanget al., 2005). As numerical simulations intend to be as close as possible to the physical system, they have beencontinuously improved. However, the development of computer processors did not catch up with this evolutionand some numerical simulations are still greedy in computational time (Sacks et al., 1989). Moreover, a differencebetween the numerical code and experiments is often observed. If there exists a bias between the code and thereality, what is the uncertainty in using it as a proxy of the physical system? Bayarri et al. (2007) introduce thenotion of validation which consists in comparing the code outputs to the field experiments. Such a task can bedifficult to achieve since it needs expensive field experiments and outputs from a code, often long to run. All alongthis thesis, we will use the word “code” as a proxy for numerical code, sometimes also called numerical model,simulator or computational code and field experiment for real world experiment.

In that respect, EDF (Électricité De France) uses numerical simulations in many fields, in particular to estimatethe power produced by a photovoltaic (PV) plant. This thesis is motivated by the uncertainty quantification andcalibration of a numerical code that intends to estimate the power produced by a PV plant. In this introduction, wewill present the economic context that explains the needs of EDF for such a study. Then, a brief explanation of howa PV panel works is given that is followed by details on the different physical models developed by EDF. Finally,the numerical codes, used as application cases in this thesis, are detailed and presented in the last section.

1.1 Economic issue

Due to global warming, new “fuel-free” technologies are increasingly being developed. EDF focuses its research onsome of them which are, without being exhaustive, the photovolatic, wind turbines or concentrated solar power(CSP). In each field the same economic problems appear especially in the photovoltaic (PV) where more andmore PV plants are built in France and all around the would. The goal for energy suppliers such as EDF is to becompetitive in this new market. In France, the apparition of new plants is regulated by an entity called CRE (EnergyRegulation Commission). The right to build and manage a new plant usually comes by winning a bidding call for

19

Intr

oduc

tion

Chapter 1 – Introduction

project. The business model of such a project is particularly based on a factor called the levelized cost of energyalso named LCOE.

The average minimum cost at which electricity must be sold in order to break-evenover the lifetime of the project.

LCOE

It can be written as:

LCOE =sum o f actualized costs over the li f etime

sum o f electrical energy produced over li f etime=

∑nt=1

It+Mt(1+r)t

∑nt=1

Et(1+r)t

, (1.1)

where It stands for the investment expenditures in the year t, Mt for the operations and maintenance expenditures inthe year t, Et for the electrical energy generated in the year t, r for the discount rate and n for the expected lifetimeof the system or power station.

The costs are relatively well estimated. Based on previous experience, the operational expenditures (also namedOPEX, which encompass the operations and maintenance expenditures) and the capital expenditures (also calledCAPEX, which cover investment expenditures It ) are approximated with a narrow credibility interval. The CAPEXrepresents the invested money for building a photovoltaic plant and are fixed costs, when the OPEX is the moneyspent to build and maintain the photovolatic plant and stands for the variable costs. However, to predict the totalpower generated over the lifetime of the plant, energy suppliers use a homemade or commercial numerical code.For the uncertainty quantification of the code, the actual methodologies used give credibility intervals around ±8%of the output of the code. A compromise is found between the risk and the price based on the estimated uncertainty.This compromise tends to have a LCOE secure for investors but it increases the price of the project. Other energysupplier companies, which could have more accurately predicted the power or have a more aggressive policy on theestablishment of business plans, could win the project. The main stake is to better assess the credibility interval onthe power produced so that even the estimation of the less power produced is better than the final power estimationof the competition.

Once EDF has won the call for project, the PV plant is built in the specific location. When its activationis effective, data of power production are recorded. After some time, EDF engineers can compare the estima-tion made in the business plan to the real power produced. If, as expected, the prediction is lower than realpower gathered on the field, the electricity price based on the business plan is no longer adequate. A new sim-ulation can be run, based on these new data, which better estimates the power produced for the next few years.The business plan can then be updated based on the new estimation. This operation is called performance monitoring.

In France the economical stakes are important because it is the fifth photovoltaic field in Europe. The estimationof the total production capacity is more than 400 GWp. Wp stands for Watt peak which represents the powerdelivered by a photovolatic panel or plant under nominal conditions (1000 W/m2 of enlightening and a temperatureof 25°C). The development deadlines are also shorter in photovolatic (3-4 years) than onshore wind turbine (7-9years). The costs of the electricity produced by the photovoltaic is decreasing (it was more than 200 e/MWh in2012 and it was equal to 55 e/MWh in 2018 for large scaled PV plants). So far, in France only 8,159 MW ofPV capacity is installed. In 2018 EDF has announced a Solar Plan which aims to install 30 GW of photovoltaicpower between 2020 and 2035. A major part of this 30 GW will be constituted of by large PV plants. The resourcesmobilized by the EDF Group are the identification of the land to be mobilized, the mobilization of the subcontractingchain and EDF’s partners, the development of self-consumption offers, the cooperation with public authorities to

20

Intr

oduc

tion

1.2. Physical phenomenon

make large areas available, the development of an industrial model adapted to the challenge, etc... In that contextEDF is looking to better control the uncertainties made by the production estimations to limit financial risks whenestablishing the project.

The aim of this thesis is to quantify the uncertainty of the numerical codes used by EDF. Based on Bayesiancalibration, the main framework will be the performance monitoring, where recorded power data are available. Then,the new predicted power can then be compared to the estimated one in the business plan and quantify plausibleearnings.

1.2 Physical phenomenon

The photovoltaic principle mainly lies in characteristics of the semiconductor material used for the cell which is,for most of the technologies developed up to now, the silicon. The energy contribution of the sun to the PV cell isvisible at a quantum level. Indeed, the energy present in the light spectrum changes locally the energy levels of thesilicon until the emission of an electron. A panel encompasses a high number of cells and a cell is composed of twosemiconductor materials. One is called the p-type and the other one the n-type. They are linked by the p-n junction.The anode corresponds to the p-type and the cathode to the n-type. The anode includes an excess of holes (it ispositively charged, because a hole is a lack of electron) and the cathode contains an excess of electron (negativelycharged). Figure 1.1 illustrates this dipole which can be “electrically” modeled by a diode.

Figure 1.1: p-n junction and the equivalent electrical component (source: Raffamaiden – CC By SA).

The contact between both parts can lead to a displacement, by diffusion, of an electron toward a hole. Thecreation of an electron/hole pair can occur with a sufficient energy supply. When an electron/hole pair is created,the displacement of the electron is generating current. The energy needed to create such a displacement, is calledthe gap energy. It corresponds to the difference of the energy levels between conduction and valence bands. Figure1.4 illustrates the different bands and the creation of the current. To facilitate this creation, a doping of the poles canbe done. The p-side doping creates an electron deficiency to establish a new pseudo level higher than the valenceband (cf Figure 1.3). Similarly, the n-side doping is an excess of electron production to set up a pseudo level lowerthan the conduction band (cf Figure 1.2).

Figure 1.2: n side doping. Figure 1.3: p side doping.

Once the photon (elementary particle particle brought here by the light) with enough energy has arrived on thecell, the creation of an electron/hole pair is done. Figure 1.4 illustrates the displacement, after the photon arrival,

21

Intr

oduc

tion

Chapter 1 – Introduction

from the n side to the p side of the electron.

Figure 1.4: Displacement of an electron in the silicon (source: Freshman404 – CC By SA).

The silicon is a fragile material and needs to be protected from the environment. Anti-reflective coatings,transparent adhesive and glass cover are responsible of losses due to reflection of sun rays. The metal contacts, thegap energy, the probability to create an electron hole pair (also called Quantum efficiency) generate other losses.The efficiency of a panel, which is the total energy used to create electricity over the total energy arriving on thepanel, stands between 15 and 20% according to manufacturers. As the global interest increases, technologies of PVcells and experiments are evolving. It is now possible to make a characterization in laboratory, very quickly, for asingle panel. However, to get the total energy produced by a PV plant is a much longer and expensive experimentaloperation. That is why numerical codes based on a physical model have been developed.

1.3 Several modeling approaches

Several numerical codes, based on physical models have been developed to mimic the PV cell behavior. Thissection presents only models developed at EDF. What differentiates them from one another is their level of accuracy,especially when it comes to predict the power of a PV plant in constraining conditions (when shades or mismatcheffects appear). If one is interested in modeling a PV system very quickly with a correct approximation, one canlook into the first model described below. However, for a more accurate prediction in constraining conditions, oneshould refer to the second model detailed beneath.

1.3.1 A first simple model

First, for a good approximation of the power produced by a PV plant, a model based on physical equations is estab-lished. As the power produced by a panel is mainly proportional to the nominal power of the panel and the incidentdirect irradiation, the equations can be written explicitly without the use of complex computer optimization to solve

22

Intr

oduc

tion

1.3. Several modeling approaches

eventually challenging equations. To consider the losses described in Section 1.2, parameters are implemented inthe model and can encompass the module photo-conversion efficiency or the module temperature coefficient forexample (more details are given in Section 1.4.3). This model, that does not require any optimization to get theresolution of eventual differential equations, is quick to implement but presents some limitations.

1.3.2 Advanced electrical models

To model, more accurately, the PV system, a refined representation of the cell is achieved. The p-n junction can be"electrically represented" by a diode (Figure 1.1), so that an electrical representation can be set up to reproducethe power generated by a PV cell, panel or plant. Two different electrical schemes have been developed. The firstdrawn, on the left panel of Figure 1.5, has only one diode when the second one (on the right panel of Figure 1.5)takes two. The second diode is added because it takes into account the cases where the functioning conditions aremore difficult (shades or low irradiations for example).

Figure 1.5: On the left panel, the electrical equivalence with 1 diode where IPV stands for a photo-current thatdepends on the incident sun rays, ID for the saturation current of the ideal diode, RP the shunt resistance, RS theseries resistance representing losses proportional to I, I the current and V the voltage generated by the cell. On theright panel the electrical equivalence with 2 diodes where ID1 and ID2 stands for the saturation current of both idealdiodes.

To establish the performances of the PV cell, the power is the quantity looked for. Then, from both electricalschemes the voltage V needs to be expressed as a function of the current I. Kirchhoff’s equations allow to solve theelectrical model with one diode (Tian et al., 2012) and:

I = IPV − ID

[expV + IRs

nVt−1]− V + IRs

Rp, (1.2)

where Vt and n are the thermodynamic potential and the quality factor of the diode. The solution of the electricalmodel with two diodes is given by:

I = IPV − ID1

[expV + IRs

n1Vt1−1]− ID2

[expV + IRs

n2Vt2−1]− V + IRs

Rp, (1.3)

where Vti and ni are the thermodynamic potential and the quality factor of the diode i (Ishaque et al., 2011). APV panel is generally made of around 60 to 72 PV cells and the voltage at the output of the panel is continuous.To be integrated in the network grid, it has to be transformed into an alternating voltage. To do so, an inverter isadded at the output of a group of panel in a PV plant. The inverter finds the “best” functioning point (which is themaximum of IV , the available power) and generates the alternating power corresponding. To get this maximumpower possible, the relation between I and V is studied. Figure 1.6 illustrates the typical behavior of the current as afunction of the voltage for a module or a PV system.

The working conditions of the inverter are such that it physically performs an “optimization” to get to the

23

Intr

oduc

tion

Chapter 1 – Introduction

Figure 1.6: I/V curve of a toy example.

point Pmax (Figure 1.6) and generates the alternating voltage afterward. Having an explicit expression of I as afunction of V is intractable regarding Equation (1.2) and Equation (1.3). Both equations are implicit and to solvethem, computers usually run optimization operations which are time consuming. To bypass this burden, someapproximation can be proposed to render explicit the equations. The use of Lambert’s function W is required inmany cases (Petrone et al., 2007; Ding and Radhakrishnan, 2008; Picault et al., 2010). It is defined as:

∀x≥−e−1 W (x)exp

W (x)= x , (1.4)

and if x is near +∞ or 0:

W (x) = logx− log(log(x))+∞

∑k=0

∞

∑m=1

ckm(log(log(x)))m

(log(x))k+m , (1.5)

with

ckm =(−1)k

m!S[k+m,k+1] , (1.6)

and S[k+m,k+1] is the number of Stirling’s cycle. Using the Lambert’s function, I and V can be separated andEquation (1.2) now written:

I =Rp(IPV + ID)−V

Rp +Rs− nVt

RsW (α(V )) , (1.7)

with

α(V ) =RpRs

Rp +RsID

nVtexp Rp

Rp +Rs

V +Rs(IPV + ID)

nVt

.

Newton or Halley’s method allows to find the couple (I,V ) maximizing the available power IV . However,themore elaborated Equation (1.3) cannot be changed to be rendered explicit. The question is to know whether onewants a numerical code fast to run or an accurate code. Is the approximation worth the saved computation time?

24

Intr

oduc

tion

1.4. Numerical codes

1.4 Numerical codes

In this section, we detail the general framework and notations of the numerical codes used in this thesis and weintroduce the main sources of uncertainties that can be found in this context. Then we present, in details, the twonumerical codes further used in this work.

1.4.1 General framework

A computer code generally depends on two kinds of inputs: variables and parameters. The variables representthe input variables (also called controllable variables in Higdon et al. (2008) or general inputs in Plumlee (2017))which are set during a field experiment and can encompass environmental variables which can be measured in fieldexperiments. In contrast, the parameters are generally interpreted as physical constants defining the mathematicalmodel of the system of interest, but can also contain the so-called tuning parameters, which have no physicalinterpretation. They have to be set by the user to run the code and chosen carefully to make the code mimic the realphysical phenomenon. The code can be mathematically represented by a function fc. Let us note in what followsthat, θ ∈Q ⊂ Rp to represent the parameter vector and x ∈H ⊂ Rd which is the variable vector. The space Q iscalled the input parameter space and H the input variable space. The physical quantity of interest (QOI) is denotedby ζ and only depends on variables in vector x ∈H because the parameter vector θ has no counterpart in fieldexperiments.

A code output is then written as fc(x,θ) (considered as a deterministic code all along the thesis) whereas ζ (x)

denotes the physical phenomenon for the same variable x. This is of course an idealized formalization, in whichwe assume that the code variables x are exhaustive to describe the phenomenon of interest, in the sense that thequantity to be predicted can take a single deterministic value ζ (x) for a given x.

1.4.2 Sources of uncertainties

In general, there are two main kinds of uncertainties considered: the epistemic and the statistical. The statisticaluncertainty represents the random fluctuations of the input variables, and the associated measurement errors andthe epistemic uncertainty comes from the uncertainty on parameters, that one could in principle know but doesnot in practice. The latter can be estimated but can also be reduced as the number of experiments increases. Theuncertainties relative to the numerical code are then epistemic. The code is deterministic so no variability is visiblebetween two launches. In this thesis, we focus only on the epistemic uncertainties that are detailed below.

The numerical code takes two inputs that are uncertain: θ and X . In the calibration framework, only theuncertainty on θ is considered because we do not know the true value of θ and we need to adjust it.

In the PV plant context, θ represents physical constants or manufacturer values that are carrying uncertainty.Indeed, the building process of a PV panel encompasses tolerances at each step of the fabrication. At the end of thechain, the parameter, that characterizes the nominal power of the panel for example, might be altered. The inputvariablesX represent mainly meteorological data. These are also carrying uncertainty because in both, predictionor performance monitoring, contexts, X is averaged from previous data where modification due to global warmingis added.

This thesis focuses only on parametric uncertainties, because the main aim is to calibrate the parameter vector θgiven a data set, the uncertainty of the input variable being out of the scope of this study. EDF experts judge that

25

Intr

oduc

tion

Chapter 1 – Introduction

parameter uncertainty and input variable uncertainty are each responsible for about 4% of the output variability.

1.4.3 Python code

Code

The Python code has been developed based on the first physical model described in Section 1.3.1. It implements thephysical equations that encompass production approximation estimation but also the losses relative to the panel. Asno optimization are needed in this code, it runs very fast (about 36µs each run). The code, that does not take intoaccount the inverter, produces an estimation of the instantaneous power. That means:

fc : H ×Q→ R

(x,θ) 7→ y .(1.8)

The code depends on some parameter vector θ and input variables x detailed as follows: θ =

η

µt

nt

al

ar

ninc

and

x=

t

L

l

Ig

Id

Te

.

The physical meaning of the parameters θ is explained below (Duffie and Beckman, 2013):

• η : module photo-conversion efficiency in nominal test conditions (1000W/m2, 25°C),

• µt : module temperature coefficient (the efficiency decreases when the temperature rises) in %/°C,

• nt : reference temperature for the normal operating conditions of the module in °C,

• al : reflection power of the ground (albedo),

• ar: describes the transmission of the radiation as a function of the incidence angle of solar rays, whichdepends on optical properties and the cleanliness,

• ninc: transmission factor for normal incidence.

The input variables x contain all measurable data:

• t: the UTC time since the beginning of the year in s,

• L: the latitude in °,

• l: the longitude in °,

• Ig: global irradiation (normal incidence of the sun rays to the panel) in W/m2,

• Id : diffuse irradiation (horizontal incidence of the sun rays to the panel) in W/m2,

26

Intr

oduc

tion

1.4. Numerical codes

• Te: ambient temperature in °C.

Note that temporal aspects are taken into account through the input variables. We do not consider any delay inthe PV reaction to the forcing conditions. Time t indicates here a snap shot corresponding to the instant when thepower has to be computed. This code only focuses on a specific time and if the evolution of the power over a day iswhat we look for, a repetition over the specific durations has to be made. This operation has to consider the numberof time steps available. For example, if 300 configurations of x are accessible for one day, the code will have to beexecuted 300 times to have the power evolution over a day. For the rest of this thesis, we will denote the code outputreferring to the ith time step by fc(xi,θ) and by fc(X,θ) the code outputs corresponding to the whole time framecontained in matrix X.

Experimental data

The Python code has been created to reproduce the instantaneous power of a test stand at EDF called “PVzen”. Thestand is a group of 12 panels connected together. Data over two months are available at a time step of 10s: Augustand September 2014. Figure 1.7 is an example of data gathered on the stand. Note that on the left panel of Figure1.7, several days have no production. It is mainly because of the sensor malfunctions and data need to be cleanedbefore being used. The power collected on the stand is the one before the inverter and physically matches with theone simulated by the numerical code introduced in Section 1.4.3.

Pow

erin

W

0

500

1000

1500

1 4 7 10 13 16 19 22 25 28 31

0

200

400

600

7 10 13 16 20

Days Hours

Figure 1.7: The power production by PVzen for August 2014 (on the left) and the power production averaged byhour for August 25th 2014 (on the right).

1.4.4 Dymola code

Code

The code Dymola is based on the “electrical” modeling introduced in Section 1.3.2, especially the physicalmodel with one diode. Dymola is a modeling and simulation environment based on the language Modelica. Thiscode has been developed for a specific PV plant that EDF maintains. It implements the shades that appear ina big plant configuration but also takes into account the mismatch effects. The mismatch effects are when thepanels from a PV plant do not possess the same nominal power value. The inverter has to find the minimal

27

Intr

oduc

tion

Chapter 1 – Introduction

one and not the averaged. Mismatch effects is also a concern when one or several cells are shaded but not thewhole panel. Shunt resistances are then activated so this part of the panel does not affect the panel overall production.

This code is then much longer to run than the previous one (about 20s for each call) but the output is a temporaltrajectory over one year of the instantaneous power with the time step of 900s. This means that, for n points in thetrajectory:

fc : Q→ Rn

θ 7→ y .(1.9)

This numerical code does not take X as input variables because they are implicitly implemented in Dymola.As a matter of fact, X represent the meteorological, the mismatch data and the projected rays files for one yearcorresponding to the n points produced by fc. The mismatch and the projected rays files are input data that give theinformation of the mismatch effects and the shades on the PV plant panels we are focusing on. In these conditions,the output power is more complex to determine. The parameter vector θ takes 26 components that we will not detailhere. The parameter vector encompasses those which have an electrical meaning such as Ipv, Rp or Rs of Figure 1.5(on the left panel) but also those which characterize the inverter. That means the output given by the Dymola codecorresponds to the power after the optimization performed by the inverter.

Experimental data

The data available for the Dymola code is the power gathered during one year 2015 (sometimes data are partiallycollected). Data are scaled in Figure (1.8) for confidentiality matters. Identically as in Figure 1.7, there is somedays where the production is null. These issues are common and also correspond to recording errors. Figure 1.8represents the temporal series given by the PV plant for the year 2015.

28

Intr

oduc

tion

1.4. Numerical codes

0

1

2

3

0e+00 1e+07 2e+07 3e+07time in s

Sca

led

pow

er

0.0

0.5

1.0

1.5

2.0

0e+00 2e+05 4e+05time in s

Sca

led

pow

er

0.0

0.5

1.0

1.5

2.0

0e+00 1e+07 2e+07 3e+07time in s

Sca

led

pow

er

0.0

0.5

1.0

1.5

2.0

0e+00 1e+07 2e+07 3e+07time in s

Sca

led

pow

er

Figure 1.8: On the top left, the original scaled power production gathered on the the PV plant during the year 2015.On the top, right the same data but only on the first week. On the bottom left, the original data but averaged by hour.On the bottom right, only the positive power is kept among the origin data.

29

Intr

oduc

tion

Chapter 1 – Introduction

1.5 Thesis organization

This thesis presents the work done on Bayesian calibration especially conditioned by the two different applicationcases detailed above. First, Chapter 2 recalls the main tools in sensitivity analysis, design of experiments, principlecomponent analysis, Monte Carlo Markov chains and Gaussian processes for a good understanding of BayesianCalibration. Chapter 3 gives a state of the art of Bayesian calibration methods. This chapter uses the applicationcase of the Python code to illustrate and compare the different statistical models that are existing. Then, Chapter 4presents a package, called CaliCo, that completes Bayesian calibration in R which has been developed in the frameof this thesis. Chapter 5 illustrates a comprehensive industrial study of calibration using the Dymola code and datafrom a real PV plant. The document then concludes with a discussion and perspectives to be explored.

30

Cha

pter

2

CHAPTER2STATISTICAL TOOLS FOR NUMERICAL

CODE CALIBRATION

2.1 Sensitivity analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

2.1.1 Morris method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

2.1.2 Sobol indices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

2.2 Kriging / Gaussian processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

2.2.1 General framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

2.2.2 Parameter estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

2.2.3 Covariance functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

2.2.4 Gaussian process-based optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

2.3 Design of experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

2.3.1 Sampling criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

2.3.2 Distance between the points criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

2.4 Principal component analysis (PCA) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

2.4.1 Distance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

2.4.2 Moments of inertia . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

2.4.3 Axis of minimum inertia . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

2.4.4 Contribution to the total inertia . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

2.4.5 Graphical representations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

2.5 Monte Carlo Markov Chains techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

2.5.1 Gibbs sampler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

2.5.2 Metropolis Hastings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

2.5.3 Metropolis within Gibbs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

2.5.4 Improvements of the Metropolis Hastings . . . . . . . . . . . . . . . . . . . . . . . . . . 65

Uncertainty Quantification (UQ) in an industrial context has become important over the last few years. All alongthe process in an industrial cycle, from the research to the in-service and maintenance, UQ has an equivalent impacton business and risk reliability. However, the procedure for establishing the impact of the variability of severalquantities on the output of interest has to be performed in multiple steps. From the identification of which parameteris responsible for the most of the output variation to the propagation of uncertainty, there is several steps that aredetailed in Rocquigny (2009). Figure 2.1 is a graphical representation of the main steps in the UQ in an industrialcontext:

• Step A is the problem specification. An identification of the quantities of interest, input variables, inputparameters, and of the numerical codes have to be done.

31

Cha

pter

2

Chapter 2 – Statistical tools for numerical code calibration

• Step B is the quantification of the uncertainty sources. Some components of the input variables or of the inputparameters might be randomly distributed as mentioned in Section 1.4.2. Their variabilities are identified andthe distribution densities determined in this step.

• Step B’ is the V& V step (Verification and Validation) which can be followed by calibration of the numericalcode.

• Step C is the propagation of the uncertainty achieved through the code. The distribution of the quantity ofinterest is looked for and particularly its moments (generally the expectancy or the variance), quantiles ormodes.

• Step C’ is the sensitivity analysis. The variables or parameters can be sorted in order of importance to identifywhich variable or parameter has the most responsibility for the variability of the quantity of interest.

Figure 2.1: Major steps in uncertainty treatment for industrial matters (source: Bertrand Iooss – ENBIS-EMSE 2009Conference).

This thesis focuses essentially on step B’ and code calibration. However, before running any calibration on anumerical code, preliminary studies have to be conducted. This chapter aims to introduce some of them in orderto help the reader in understanding the main steps of code calibration in an industrial context. First, two differentmethods of sensitivity analysis are presented. The second section provides theoretical developments on Gaussianprocesses that can be used as much in machine learning as in modeling some structural error. The third sectionintroduces the aspects of design of experiments and recalls some of the major tools used in the thesis. Then, the twolast sections present, theoretically, the principal component analysis and Monte Carlo Markov Chains techniquesthat are useful for a complete understanding of the extensions used in the further work.

32

Cha

pter

2

2.1. Sensitivity analysis

2.1 Sensitivity analysis

The quantification of the influence of each input parameter on the output is the task performed by sensitivity analysis.It is interesting to access such an information because if the numerical code is time consuming, it can be helpful tofocus on a reduced number of parameters. In that sense, sensitivity analysis (SA) can be considered as a prerequisitefor model building in any setting (Saltelli et al., 2000). SA has a different meaning in different contexts. For anengineer, it could mean to move each component of the input parameter vector at the same range and comparethe impacts on the quantity of interest. For a statistician, SA is the study of the variation of the distribution of thedensity of the quantity of interest according to the change of specific parameters.

There are two major categories in SA: global and local methods. Local SA tends to quantify the impact ofa parameter around reference values. It is mainly done with partial derivatives. This kind of analysis is usefulwhen one is interested in understanding the behavior of the physical model nearby these values. However, thesemethods do not allow to study the effect of the input parameters on the output when they have an important area ofuncertainty. Contrary to local SA, global SA aims to study the variability of the quantity of interest driven by inputparameters variation on all their area of uncertainty. In this section, we present only global SA methods mainlybecause in this thesis we are interested on the overall impact of the parameters on the variability of the quantity ofinterest.

Global SA can be performed in several different ways. First, screening methods allow to explore quickly thevariability of the output, given by a numerical code, induced by a variation of the, potentially large, input parametervector. The importance criterion in such methods is the amplitude of the variations of the code output obtained fordifferent input parameter values. Screening methods can also be categorized but we only focus on Morris methodSection 2.1.1 (Morris, 1991) because it requires less assumptions about the model compared to the other screeningmethods. These methods only allow to identify the non-influent parameters. If one is interested in classifying byorder of importance the parameter influence, sensibility indices can be used. Sensibility indices are defined as ameasure of the influence of an input parameter on the variability of the output. If the chosen measure of importanceis the variance, then the indices are called Sobol indices (Sobol’, 1990) and are presented in Section 2.1.2. However,these methods require a lot of code calls and if the code is time consuming they become quickly intractable. Toestimate, with a limited computational impact, sensitivity indices, smoothing methods or surrogate of the codefunction have been developed. For example, Sudret (2008) demonstrates that Sobol indices straightforwardly arisefrom the chaos polynomial decomposition. Kriging surrogate also allows to obtain analytically the sensitivityindices formulation as it is shown in Oakley and O’Hagan (2004); Marrel et al. (2009). Da Veiga (2015) alsointroduced global SA methods which use dependence measures as the mutual information, the distance correlationor the Hilbert-Schmidt Independence Criterion (HSIC). Then, De Lozzo and Marrel (2016) have extended this workfor a screening purpose and allow to decrease the computational time burden.

2.1.1 Morris method

Morris method aims to evaluate the influence of each input parameter by considering the impact of its variation onthe output considering the other ones as constant. This way to operate consists in moving each parameter one at atime ("One At a Time" or OAT method). Then, the input parameter space is discretized on a grid.

Sampling space

Since the parameter range values are not all of the same order, the sampling design of the Morris method isstandardized over the interval [0,1]. Thus, for p parameters, the sampling plan will be contained in a hypercubeω = [0,1]p. To generate a comparable variation for each parameters, a step δ is defined. For each parameter, the

33

Cha

pter

2

Chapter 2 – Statistical tools for numerical code calibration

sampling plan is divided in Q levels such as D = [0, 1Q−1 ,

2Q−1 , ...,1]

p (where D stands for the sampling grid). Eachaxis is split into Q−1 equals sections. Usually, the step δ is chosen equally as the sampling step 1

Q−1 but one canalso pick a k ∈ N+∗ such as δ = k

Q−1 . Figure 2.2 illustrates these variations. To define them, a point is randomlyselected on the grid D. Then, a sign is also randomly selected (practically a Bernoulli random variable) to indicatethe direction of displacement. The new point being found, a new random test on the sign of δ is run to move inthe second direction. This procedure is repeated p−1 times and is called a trajectory. In its paper Morris (1991)advocates to select δ = Q

2(Q−1) and Q even such as not to favor any area of the space. However, the step of suchselection might imply some lost of information on the parameter. For the rest of the thesis, the choice of δ = 1

Q−1

will always be made.

Let us define a toy function ft such as:

[0,1]2 7→ R

(θ1,θ2)→ ft(θ1,θ2)

θ → ft(θ).

(2.1)

The parameter vector θ can be extended to a vector with p component as θ =

θ1...

θp

(if the function takes p

input parameters as it is the case for fc). If we choose a sampling level of Q = 9, Figure 2.2 illustrates the gridobtained and 4 different trajectories for ft .

P1 P2

P3

P4 P5

P6

P7P8

P9

P10 P11

P12

sampling step

0.00

0.25

0.50

0.75

1.00

0.00 0.25 0.50 0.75 1.00

θ1

θ 2

Figure 2.2: Sampling grid on the scaled space.

Let us call r the number of trajectories accomplished (Figure 2.2, r = 4). In his paper, Morris (1991) advicesnot to take a too high r. The aim is to screen the input parameter space with some displacements, not to generate adesign of experiments representative of the original space (some of them are developed in Section 2.3). Note that ifone of the chosen point is taken on the boundary of the grid, then the sign of δ is not randomly chosen. Indeed, tostay in the input parameter space, the sign is set such as the displacement stays in the grid.

34

Cha

pter

2

2.1. Sensitivity analysis

Elementary effects

These variations of each parameter are quantified by elementary effects (EE) (Morris, 1991). For the function fc

defined on [0,1]p, the EEs can be written as:

∆(i)k =

fc(θ i + ε(i)k δ )− fc(θ i + ε

(i)k−1δ )

δ ε(i)k (k)

, (2.2)

Elementary effects

where:

ε(i) =

s1 . . . s1 . . . s1...

. . ....

0 sk sk...

. . . . . ....

0 . . . 0 . . . sp

(i)

,

with ε(i)0 = (0, . . . ,0)(i)T and where ε

(i)k stands for the kth column vector of the matrix ε(i),∀i ∈ J1,rK ε

(i)k (k) = s(i)k ,

ε(i) possesses p lines with s(i)k =±1 ∀k ∈ J1, pK, θ i = (θ1, . . . ,θp)T(i) is the ith point in the design, and ∆

(i)k fc is the

elementary effect of the kth parameter at the iteration i and δ stands for the step.

In the 2 dimensional example (Figure 2.2), P2 coordinates are given by ft

(θ1 +δ

θ2

)and P1 coordinates by

ft

(θ1

θ2

). The direction is positive so: ε1 =

(10

). From this example, the elementary effect for the first trajectory

and for θ1 is defined by:

∆11 =

ft

((θ1

θ2

)+

(10

)δ

)− ft

((θ1

θ2

)+

(00

)δ

)δ

=

ft

(θ1 +δ

θ2

)− ft

(θ1

θ2

)δ

.

This matches with the finite difference on θ1 and for a step δ . Visually in Figure 2.2, it coincides with thedifference of the values of ft at the unnormalized points P2 and P1 and divided it by δ .

For r Morris trajectories, there will be N = r(p+1) calls to the function fc. Two indices computed with the EEsallow to compare the effect of the variation of each parameter on the output. The first is the expectancy of the EEsfor each parameter (Faivre et al., 2013; Saltelli et al., 2000, 2004; Morris, 1991):

µk =1r

r

∑i=1

∆(i)k fc. (2.3)

If fc is periodic, the mean of the EEs is near zero. This does not mean that the parameter has no impact, that iswhy it is better to use the mean of the absolute value of the EEs µ∗ (Faivre et al., 2013; Saltelli et al., 2000, 2004):

35

Cha

pter

2

Chapter 2 – Statistical tools for numerical code calibration

µ∗k =1r

r

∑i=1| ∆(i)

k fc | . (2.4)

the estimated expectancy µ∗

The second index is the standard deviation of the EEs (Morris, 1991; Saltelli et al., 2004):

σk =

√1

r−1

r

∑i=1

(∆(i)k fc−µk)2. (2.5)

the estimated standard deviation σk

Indices µk, µ∗k and σ2k are Monte-Carlo estimators of respectively µk = EΘ[∆k fc(θ)], µ∗k = EΘ[

∣∣∆k fc(Θ)∣∣] and

σ2k = Var(∆k fc(Θ)) with Θ ∼Uni f [0, ..., k

Q−1 , ...,1p]. Thus, under regularity conditions of fc and its deriva-

tives, when δ → 0 these estimators converge toward∼µk =

∫[0,1]k

∂ fc∂θk

(θ)dθ ,∼µ∗k =

∫[0,1]k

∣∣ ∂ fc∂θk

∣∣(θ)dθ =∣∣∣∣ ∂ fc

∂θk

∣∣∣∣1 and

∼σ2

k =∫[0,1]k

∂ fc∂θk

2(θ)dθ −

∼µ2

k =∣∣∣∣ ∂ fc

∂θk

∣∣∣∣22−

∼µ2

k .

If the relation between the kth input parameter and the output is linear then the mean of the elementary effect isproportional to the intensity of the relation. The index σk, on the other hand, allows to detect interactions betweeninput parameters and is sensitive to the non-linearity of the output function of the kth input parameter (Faivre et al.,2013). A graph of the elementary effects can be display considering the x-axis as µ∗ and the y-axis as σ . Then,three areas appear:

• for low values of µ∗k and σk, the kth parameter is considered as having a negligible impact on the output,

• for hight values of µ∗k compared to σk, the kth parameter has a strong linear effect on the output,

• for hight values of σk the kth parameter has either a strong non-linear effect on the output and/or includes oneor several interactions with other ones.

One of the limits of this method is that we cannot differentiate parameters that have interaction with other onesfrom those which have a non-linear effect on the output (Faivre et al., 2013). To access this information anotherstudy has to be performed afterward. Computing the Sobol indices on the parameters left can be a solution (seeSection 2.1.2 for further details).

Morris function

In its paper, Morris (1991) introduced a test function of the method that allows to visualize the three areas. Thefunction takes 20 input parameters and is defined as:

Y = β0 +20

∑i=1

βiwi +20

∑i< j

βi, jwiw j +20

∑i< j<l

βi, j,lwiw jwl +20

∑i< j<l<s

βi, j,l,swiw jwlws, (2.6)

where,

wi =

2(1.1 Xi

Xi+0.1 −0.5) for i= 3, 5, 7,

2(Xi−0.5) otherwise.

36

Cha

pter

2

2.1. Sensitivity analysis

The β are defined as:

βi = 20 for i=1,...,10,βi, j =−15 for i,j=1,...,6,βi, j,l =−10 for i,j,l=1,...,5,βi, j,l,s = 5 for i,j,l,s=1,...,4.

All the βi, βi, j left and β0 are sampled according to a standard normal distribution. The number of parameter p

is equal to 20 and r = 5 trajectories are achieved on a sampled space with Q = 9. Figure 2.3 illustrates on the leftthe results of the method with the, previously defined, settings for the Morris function and on the right 10 repetitionsof the method on the same function.

X1

X2

X3

X4

X5

X6

X7

X8

X9

X10X11X12

X13

X14X15

X16

X17X18

X19X20

0

20

40

60

0 10 20 30 40

μ*

σ

X1X2

X3

X4

X5

X6

X7

X8X9X10X11X12X13

X14X15X16

X17X18X19X20

X1X2

X3

X4

X5

X6

X7X8

X9X10X11

X12X13X14X15X16X17X18X19X20

X1

X2

X3

X4

X5

X6

X7

X8X9X10X11X12X13X14X15X16X17X18X19X20

X1

X2

X3

X4

X5

X6

X7X8X9X10

X11X12X13X14X15X16X17X18X19X20

X1

X2

X3

X4

X5

X6

X7

X8X9

X10X11X12X13

X14X15X16X17X18

X19X20

X1

X2X3

X4

X5

X6X7

X8X9X10X11X12

X13X14

X15X16X17X18X19X20

X1X2

X3

X4

X5X6

X7

X8X9X10X11X12X13X14X15X16X17X18X19X20

X1

X2

X3X4

X5

X6

X7

X8X9X10X11X12X13X14X15X16X17

X18X19X20

X1

X2

X3

X4

X5

X6

X7

X8X9X10X11X12X13X14X15X16X17X18X19X20

X1

X2

X3 X4

X5

X6X7

X8X9X10X11X12X13

X14X15X16X17X18X19X20

0

50

100

150

0 50 100 150

μ*

σ

Figure 2.3: On the left panel a result of Morris method on the Morris function and on the right panel 10 repetitionsof the method.

The three areas presented above are visible in Figure 2.3. The parameters X8, X9 and X10 are in the area wherethe mean of the EEs are high compared to the standard deviation of the EEs. That means these parameters have alinear impact on the output. Then, parameters X1, X2, X3, X4, X5, X6 and X7 are in the area where σ is high. Allother parameters (from X11 to X20) are negligible since they possess low values of µ∗ and σ . This method onlyallows to identify the parameters to neglect for further study. However, some variability is visible on the right panelof Figure 2.3 because 10 repetitions of the method gives different results.

Non-parametric bootstrap

To quantify the variability of the method, a non-parametric bootstrap is run. This has the advantage to run only oncethe Morris method. A N re-sample is done on the matrix of the EEs called ∆. In this case the aim is to estimate thefollowing: bias, variance and credibility interval at 95%:

37

Cha

pter

2

Chapter 2 – Statistical tools for numerical code calibration

b(θ) = E(θ −θ) (2.7)

σ(θ)2 =Var(θ) =Var(θ −θ) (2.8)

CI(θ) = θ +[q97.5%(θ − θ);q2.5%(θ − θ)] (2.9)

These values straightforwardly depend on the distribution L(θ −θ). Bootstrapping allows, here, to simulateanother distribution L(θ ∗− θ) to estimate these coefficients.

θ = (µ∗,σ)

↓

∆ =

∆1

1 ... ∆1p

......

∆r1 ... ∆r

p

→ θ =

µ∗1 σ1...

...µ∗p σp

↓

∆∗ =

∆1∗

1 ... ∆1∗p

......

∆r∗1 ... ∆r∗

p

→ θ ∗ =

µ∗1∗

σ∗1...

...µ∗p∗

σ∗p

Because of the low number of simulations used in Morris method, the uncertainty on the estimated matrix θ

can be high. The lines ∆∗ (∆i∗k , . . . ,∆

r∗k ) are each of them sampled uniformly and re-injected among the lines ∆

(∆ik, . . . ,∆

rk). This operation is completed N times to get N estimations of θ ∗ (θ ∗1 , . . . , θ

∗N). Then for each sample,

the coefficients introduced above can be estimated as:

b(θ) =1N

N

∑i=1

θ ∗i − θi, (2.10)

σ(θ)2=

1N

N

∑i=1

θ ∗i2−( 1

N

N

∑i=1

θ ∗i

)2, (2.11)

IC(θ) = θ −[θ − θ ∗[97.5%∗N]; θ − θ ∗[2.5%∗N]

]= 2θ −

[θ ∗[97.5%∗N]; θ ∗[2.5%∗N]

],

(2.12)

where θ ∗[97.5%∗N] corresponds to θ ∗[1] ≤ ...≤ θ ∗[x] and [x] = E(x).

To visualize graphically the results given by the bootstrap, scatter plots can be displayed. Figure 2.4 givesthe bootstrap results for N = 1500 and for r = 100 trajectories of Morris. This shows the trustworthiness of thehypothesis about the variability of the method. The biggest variability concerns only the parameters from X1 to X7.However even with the variability, the three areas are distinct and the decision about the parameter to neglect canstill be achieved.

38

Cha

pter

2

2.1. Sensitivity analysis

X1X2

X3

X4

X5

X6

X7

X8X9X10X11X12X13X14X15X16X17X18X19X20

X1X2

X3

X4

X5

X6

X7

X8X9X10X11X12X13X14X15X16X17X18X19X20

0

50

100

0 20 40 60 80

μ*

σ

Figure 2.4: Scatter plot of the Morris indices given by the 1500 iterations bootstrap.

2.1.2 Sobol indices

The Morris method only allows to detect parameters that have no impact on the output and hence can be negligible.However to get more information on which parameter has the more influence on the output, another method needs tobe applied. Sensitivity index based on a measure of importance can be used and in this section, the variance-basedsensitivity analysis is presented. This method is also called Sobol method and uses the variance as the measure ofinfluence of the output to get sensitivity index.

First order sensitivity index

Sobol method has been introduced by Sobol’ (1990) and, as it is the case for the Morris method, the parameter inputspace is normalized between 0 and 1. It means that the space Ω is the k sized hypercube. From Fourier Haar’s series,Sobol, in a previous work, had proposed a decomposition of the function fc such as:

fc(θ1, . . . ,θk) = fc0 +k

∑i=1

fci(θi)+ ∑1≤i< j≤k

fci j(θi,θ j)+ · · ·+ fc1,2,...,k(θ1, . . . ,θk). (2.13)

For Equation (2.13) to make sense, fc0 must be constant and the integral of each part of the sum must be null:

∫ 1

0fci1 ,...,is

(θi1 , . . . ,θis)dθik = 0 if 1≤ k ≤ s. (2.14)

The consequence of Equation (2.13) and Equation (2.14) is that all the parts of the sum are orthogonal. So, if(i1, . . . , is) 6= ( j1, . . . , jl) then: ∫

Ω

fci1 ,...,isfc j1 ,..., js

dθ = 0. (2.15)

Sobol shows, in its article Sobol (1993), that the decomposition given in Equation (2.13) is unique and all theterms can be evaluated thanks to multidimensional integrals:

fci(θi) =− fc0 +∫ 1

0· · ·∫ 1

0fc(θ)dθ∼i,

fci j(θi,θ j) =− fc0 − fci(θi)− fc j(θ j)+∫ 1

0· · ·∫ 1

0fc(θ)dθ∼(i j).

where dθ∼i and dθ∼(i j) stands for the integrations over the whole domain Ω without (respectively) θi and, θi

39

Cha

pter

2

Chapter 2 – Statistical tools for numerical code calibration

and θ j. Total variance D of fc(θ) can be expressed by:

D =∫

Ωkf 2c (θ)dθ− f 2

c0. (2.16)

In a similar way and thank to the decomposition introduced in Equation (2.13), partial variances can be writtenas:

Di1,...,is =∫ 1

0· · ·∫ 1

0f 2ci1 ,...,is

(θi1 , . . . ,θis)dθi1 . . .dθis , (2.17)

where 1≤ i1 < · · ·< is ≤ k and s = 1, . . . ,k. When the square of Equation (2.13) is taken and then integratedover Ωk, the total variance is written:

D =k

∑i=1

Di + ∑1≤i< j≤k

Di j + · · ·+D1,2,...,k. (2.18)

The first index in the Sobol method is called the first order Sobol index and is defined as:

Si =Di

D. (2.19)

First order Sobol’s index

The first order Sobol’s index is Si for the parameter θi. This index allows to quantify the principal effects of thisparameter on the variance of the output. The formula Si =

Var[Y |θi]Var[Y ] is, here, found back. The Si j for i 6= j are called

second order index and allow to quantify the interactions between θi and θ j that are not taken into account in Si. IfEquation (2.18) is divided by D and replaced by Sobol’s index defined in Equation (2.19), then:

k

∑i=1

Si + ∑1≤i< j≤k

Si j + · · ·+S1,2,...,k = 1. (2.20)

Total effect index

Total effect index or total sensitivity index (TSI) are defined as the sum of the all the indices encompassing thestudied parameter. This definition can be rewritten as:

T S(i) = ∑I⊆i

Si. (2.21)

Total sensitivity index

In a 3-dimensional toy example, the TSI of the parameter θ1 is given by T S(1) = S1 +S12 +S13 +S123 where S1

is the first-order index of the parameter θ1 on the output yc, S12 and S13 are the second order index which arequantifying the interactions of θ1 with θ2 and θ3, and S123 the third order index. If T S(i) ≈ Si, that means onlythe first-order index has a significant impact on the output and the superior order indices are then negligible. It iscommon to compute the first order index and the total index to check if interactions are present.

Compared to the Morris method, Sobol indices allow to visualize interactions between parameters. However,the computation of the first order and the total effect Sobol’s indices are performed with a design D. To have aglobal study on the overall impact, for the range ofD, the analysis has to be run for each θ inD which is really

40

Cha

pter

2

2.1. Sensitivity analysis

time consuming. Some optimal design of experiments can be used to compute the Sobol indices such as optimalLatin Hypercube Sampling (LHS detailed in Section 2.3) described in Saltelli (2002) for example.

Test on Ishigami functions

The Ishigami function is a non-monotonic test function which is defined by:

Y = sinθ1 +Asin2θ2 +Bθ

43 sinθ1, (2.22)

with θi ∼U(−π,π), A = 7 and B = 0.1. From a first prospect, no parameter seems having a bigger impact onthe output than another. As a matter of fact trigonometric functions and different coefficients lead to a difficultinterpretation of the function. The scatter plots in Figure 2.5, represent the output function of each parameter andallow to visualize the behavior of the function according to a specific parameter.

−10

0

10

−2 0 2θ1

y

−10

0

10

−2 0 2θ2

y

−10

0

10

−2 0 2θ3

y

Figure 2.5: Scatter plots of the Ishigami function where the output is given function of the each parameter.

The results of the first order and the total effect Sobol’s indices with the Ishigami function are represented inFigure 2.6. The can point out a first remark that the total index is close to the first index for the parameter θ2 butthey are far away for θ1 and θ3. It means that for θ2 there is likely no interaction with other parameters. However, astrong interaction is denoted for θ1 and θ3 by the Sobol’s indices. As there is only three parameters, the interactionis only between θ1 and θ3. Figure 2.6 also represents the box-plot of a bootstrap of 1000 iterations. The variabilityof this method is lower than the one obtained by Morris.

0.0

0.2

0.4

0.6

θ1 θ2 θ3

First−order sensitivity index

Total sensitivity index

Figure 2.6: Sobol’s index computed for the Ishigami function and the boxplots representing the variability of 1000bootstrap iterations.

41

Cha

pter

2

Chapter 2 – Statistical tools for numerical code calibration

2.2 Kriging / Gaussian processes

In mining engineering, predicting the content at site, knowing the content of neighbor sites by a Gaussian processcomes from Krige (1951) and is called Kriging. Then Matheron, in Matheron (1963), has proposed the Krigingmethod to model spatial data in geostatistics (Cressie and Noel, 1993; Stein, 2012). This is Sacks et al. (1989) thathave used this formal modeling as surrogate in a numerical experiment framework. More than a surrogate, this kindof modeling brings also an indicator of the uncertainty according to a prediction of the surrogate at a given point.This section will focus, first, on the general framework of the Gaussian process and in a second part on the mainparameter estimation methods. Then, on a third part some general covariance functions, used to depict correlationbetween nearby sites, are developed. To optimize the design of experiments regarding the quality of the Gaussianprocess, some Gaussian process-based optimization method have been developed and one of them, named EfficientGlobal Optimization (EGO), is presented in the fourth part.

2.2.1 General framework

Let us consider a probability space (Ω,F ,π) where Ω stands for a sample space, F a σ -algebra on Ω and π aprobability on F . A stochastic process X is a family as Xt ; t ∈ T where T ⊂ Rd . It is said that the aleatoryprocess is indexed by t. At t fixed, the application Xt : Ω→R is a random variable. At ω ∈Ω fixed, the applicationt→ Xt(ω) is a trajectory of the stochastic process.

For t1 ∈T , . . . , tn ∈T , the probability distribution of the random vector (Xt1 , . . . ,Xtn) is called finite-dimensionaldistributions of the stochastic process Xtt∈T . Hence, the probability distribution of an aleatory process is deter-mined by its finite-dimensional distributions. Kolmogorov’s theorem guaranties the existence of such a stochasticprocess if a suitably collection of coherent finite-dimensional distributions is provided.

A random vector Z such as Z = (Z1, . . . ,Zn) is Gaussian if ∀λ1, . . . ,λn ∈ R the random variable ∑ni=1 λiZi

is Gaussian. The distribution of Z is straightforwardly determined by its two first moments: the mean µ =

(E[Z1], . . . ,E[Zn]) and the variance covariance matrix Σ = cov(Zi,Z j)1≤i, j≤n. When Σ is positive definite, Z has aprobability density defined by Equation (2.23).

π(z) =|Σ|−1/2

(2π)n/2 exp− 1

2(z−µ)T Σ−1(z−µ)

. (2.23)

Let us consider two Gaussian vectors called U1 and U2 such as:(U1

U2

)∼N

((µ1

µ2

),

(Σ1,1 Σ1,2

Σ2,1 Σ2,2

)).

The conditional distribution U2|U1 is also Gaussian (Equation (2.24)). This property is the base of Krigingwhen a Gaussian process is used as a surrogate of a function.

U2|U1 ∼N(µ2 +Σ2,1Σ−1

1,1(U1−µ1),Σ2,2−Σ2,1Σ−11,1Σ1,2

). (2.24)

Conditional distribution

A stochastic process Xtt∈T is a Gaussian process if each of its finite-dimensional distributions are Gaussian.Let us introduce the mean function such as m : t ∈ T → m(t) = E[Xt ] and the correlation function such asK : (t, t ′) ∈T ×T → K(t, t ′) = corr(Xt ,Xt ′) (as well noted r(t, t ′) in the thesis). A Gaussian process with a certainvariance noted σ2 will be defined as Equation (2.25).

42

Cha

pter

2

2.2. Kriging / Gaussian processes

X(.)∼PG (m(.),σ2K(., .)). (2.25)

Gaussian processes are used in this thesis in two cases. In the fist one, fc is a code function long to run and theGaussian process emulates its behaviour. Then the Gaussian process is the surrogate of the code. The second case iswhen we want to model the error made by the code (called code error or discrepancy). For the former, we want tocreate a surrogate fc of a deterministic function fc. Let us define this function to emulate fc such as:

fc : Q ⊂ Rp→ R

θ 7→ fc(θ).(2.26)

Following previous statement, fc is a realization of a Gaussian process. In a Bayesian framework, the Gaussianprocess is a “functional” a priori on fc (Currin et al., 1991). The natural idea of the Gaussian process emulation isto use known evaluations of fc at some selected points from a design of experiments D, and access information ofthe evaluation of fc at some point θ0 where θ0 6∈ D. The modeling with a Gaussian process of the function fc canbe written as:

fc(•)∼PG (H(•)Tβ,σ2Kψ(•,•)), (2.27)

where β is a parameter vector such as β = (β1, . . . ,βp), σ2 and ψ respectively stands for the variance and thecorrelation length of the covariance function cσ ,ψ(•,•) = c(•,•) = σ2Kψ(•,•) andH(•) the matrix of regressors.The covariance function is defined as:

(θ,θ′) ∈Q×Q, c(θ,θ′) = σ2Kψ(θ,θ′). (2.28)

Covariance function

Equation (2.27) can also be written as:

θ ∈Q, fc(θ) =H(θ)Tβ+Z(θ), (2.29)

Gaussian process

where Z(θ) represents a centered Gaussian process characterized by its covariance function c(Z(θ),Z(θ′) =

σ2Kψ(θ,θ′). Let us consider a toy function ft we try to emulated by a Gaussian process with 5 points in the designof experiments:

ft : [0,1]→ R

θ 7→ y = (6θ −2)2sin(11θ −4) .(2.30)

To visualize the impact of parameter values, some Gaussian process realizations are performed with differentvalues of σ2 and ψ. The results are displayed in Figure 2.7 and allows to note that these parameters values impactsthe Gaussian process quality.

Usually β, σ2 and ψ are unknown. Some modeling consider them as determined values but, practically, theyare estimated upstream with methods described in Section 2.2.2. They also can be considered as unknown duringthe Gaussian process modeling with their associated uncertainty encompassed. However, this could lead to complex

43

Cha

pter

2

Chapter 2 – Statistical tools for numerical code calibration

−5

0

5

10

0.00 0.25 0.50 0.75 1.00x

y

−5

0

5

10

0.00 0.25 0.50 0.75 1.00x

y

−5

0

5

10

0.00 0.25 0.50 0.75 1.00x

y

−5

0

5

10

0.00 0.25 0.50 0.75 1.00x

y

Figure 2.7: Different Gaussian process emulations for the toy function defined Equation (2.30). On the first panel(on the left) the Gaussian process is estimated with σ2 = 1 and ψ = 0.1. On the second panel (on the middled left),σ2 = 5 and ψ = 0.1. On the third panel (on the middle right), σ2 = 1 and ψ = 0.2. And, on the fourth panel (onthe right), σ2 = 5 and ψ = 0.2.

modelings. In what follows, we consider β, σ2 and ψ as determined values.

Let us consider that N evaluations of fc are available (where fc is the general function of Equation (2.27)) at N

different points of a design of experiments D. We note the design of experiments (DOE) D = (θ1, . . . ,θN)T and the

corresponding outputs of D by fc, yc = (y1, . . . ,yN)T . To get the evaluation of θ0 ∈Q−D by fc, we are interested

in knowing the distribution of fc(θ0) conditionally to Yc where Yc = ( fc(θ1), . . . , fc(θN)). This can be written as:

(fc(θ0)

Yc

)∼N

((H(θ0)

Tβ

H(D)Tβ

),σ2

(Σψ(θ0) Σψ(θ0,D)

Σψ(θ0,D)T Σψ(D)

)), (2.31)

whereH(D) = (H(θ1), . . . ,H(θN )),H(θ0) = (H(θ 10 ), . . . ,H(θ p

0 )), Σψ(θ0) = Kψ(θ0,θ0) = Ip (becauseof the kernel regularity), (Σψ(θ0,D))1≤i≤N = Kψ(θ0,θi) and (Σψ(D))1≤i, j≤N = Kψ(θi,θj). Using Equation(2.24), it comes straightforwardly that:

fc(θ0)|Yc = yc ∼N (µθ0|D,σ2θ0|D),

with,µθ0|D =H(θ0)

Tβ+Σψ(θ0,D)Σψ(D)−1(yc−H(D)Tβ), (2.32)

σ2θ0|D = σ

2(

1−Σψ(θ0,D)T Σψ(D)−1Σψ(θ0,D)). (2.33)

In prediction, the mean can be used to approach the value of fc(θ0) and fc : θ0 7→ µθ0|D. The variance σ2θ0|D

represents the uncertainty associated to the prediction of fc(θ0) by fc(θ0). A confidence interval can be establishedfor the surrogate because:

fc(θ0)−µθ0|D√σ2θ0|D

∼N (0,1). (2.34)

Jones et al. (1998) proposes to validate the surrogate with a “leave-one-out” cross validation method whichconsists in testifying that 99.7% of the i = 1, ...,n:

fc(θi)− fc−i(θi)√σ2θ0|D

∈ [−3,3], (2.35)

where fc−i(θi) and σ2θ0|D−i

respectively stands for the posterior mean and the posterior variance obtained byEquation (2.32) and Equation (2.33) for a design of experiments D = θ1, . . . ,θi−1,θi+1, . . . ,θN.

44

Cha

pter

2

2.2. Kriging / Gaussian processes

The predictor fc(θ0) is the Best Linear Predictor (BLP) if it minimizes the following Mean Square Error (MSE):

MSE(θ0) = E[( fc(θ0)− fc(θ0))2]. (2.36)

With β, σ2 and ψ fixed, and under hypotheses of Equation (2.27), the best linear predictor of fc(θ0) is:

θ0 7→H(θ0)Tβ+Σψ(θ0,D)Σψ(D)−1(yc−H(D)Tβ). (2.37)

Best linear predictor

This predictor is unbiased and its mean square error is equal to MSE(θ0) = σ2θ0|D. If the evaluated point is

taken in the original design D (i.e. θ0 = θi ∈Q), then µ2θ0|D = yi and σ2

θ0|D = 0.

2.2.2 Parameter estimation

Maximum likelihood estimates

Practically, β is unknown, and, to estimate it, a generalized least square method is commonly applied whichcorresponds to the maximum likelihood of Equation (2.23). This gives:

β = (H(D)T Σψ(D)−1H(D))−1H(D)T Σψ(D)−1yc. (2.38)

β prediction

The predictor given Equation (2.37) can be rewritten as:

fc(θ0) =H(θ0)T β+Σψ(θ0,D)Σψ(D)−1(yc−H(D)T β), (2.39)

and its mean square error is:

Var( fc(θ0)) = E( fc(θ0)− fc(θ0))2

= σ2(1+u(θ0)

T (H(D)TΣψ(D)−1H(D))−1u(θ0)−Σψ(θ0,D)T

Σψ(D)−1Σψ(θ0,D),

(2.40)

where u(θ0) =H(θ0)T Σψ(D)−1Σψ(θ0,D)−H(θ0). This prediction mean square error is applied when β is not

fixed. If it is fixed the prediction mean square error is given Equation (2.33) but in both cases the predictor meansquare error corresponds to the predictor variance since it is unbiased. If we consider σ2 and β as unknown, thecovariance matrix known, then fc(θ0) is the Best Linear Unbiased Predictor (BLUP) of fc(θ0).

To estimate the parameter σ2, that only intervenes in the calculation of the root mean square error, a maximumlikelihood can be applied and:

σ2 =

1N(Yc−H(D)β)T

Σψ(D)−1(Yc−H(D)β). (2.41)

σ2 prediction

45

Cha

pter

2

Chapter 2 – Statistical tools for numerical code calibration

Then, the parameter ψ is estimated after having plugged β and σ2 in the likelihood and getting the maximum.After parameter plugging, the opposite log-likelihood can be written as:

L (Yc;ψ) ∝ Nlog(σ2)+ log(|Σψ(D)|), (2.42)

and, ψ = argminψ

L (Yc;ψ).

The resulting predictor

fc(θ0) =H(θ0)T β+Σψ(θ0,D)T

Σψ(D)(Yc−H(D)β) (2.43)

is called EBLUP (Empirical Best Linear Unbiased Predictor) although it is neither linear nor unbiased. Maximumlikelihood estimation, and “plug-in” techniques under estimate the prediction variance (Handcock and Stein, 1993;Helbert et al., 2009) and does not take into account any uncertainty affecting the parameters β, σ2 and ψ.

Bayesian estimation

In this section we focus on Bayesian estimation of σ2 and β. We will consider a three level hierarchical model,in the sense of Robert (2007), in where the lowest level only deals with the parameter β. At the second level, theparameter σ2 controls the distribution of β and at the top level, the parameter ψ controls the distribution of σ2 andβ. Note that the estimation of σ2 causes the loss of normality a posteriori. The prior density of (β,σ2) can bewritten:

π(β,σ2) = π(β|σ2)π(σ2). (2.44)

In this section we will work with the four configurations of priors present in Santner et al. (2013); Le Gratiet(2013):

1. σ2 ∼I G (α,γ) and β|σ2 ∼N (b0,σ2V0)

2. π(σ2) ∝1

σ2 and β|σ2 ∼N (b0,σ2V0)

3. σ2 ∼I G (α,γ) and π(β|σ2) ∝ 1

4. π(σ2) ∝1

σ2 and π(β|σ2) ∝ 1

The first case is the more common in Bayesian Kriging because both priors are conjugate to a normal likelihoodand the posterior density can, explicitly, be written. In the presented configurations, I G (α,γ) stands for the inverseGamma distribution

π(x) =γα

Γ(α)

e−γ/x

xα+1 1x>0.

The computation of the posterior density can be done for each cases and for ψ fixed. In each cases, the posterior

can be expressed as π(β|yc,σ2)∼N (νΣ,Σ) with:

Σ−1 =

(HT Σψ(D)H+V0)/σ2 for the cases (1) and (2),

(HT Σψ(D)H)/σ2 for the cases (3) and (4),(2.45)

and,

ν =

(HT Σψ(D)yc+V0b0)/σ2 for the cases (1) and (2),

(HT Σψ(D)yc)/σ2 for the cases (3) and (4).(2.46)

46

Cha

pter

2

2.2. Kriging / Gaussian processes

In the second level of the hierarchical model, we are now considering σ2. We can write:

π(σ2|yc) =L (yc;β,σ2)π(β|σ2)π(σ2)

π(yc), (2.47)

and this equation leads the posterior distribution σ2|yc ∼I G (ασ ,γσ ) where

γσ =

2γ +(b0−β∗)(V0 +(HT Σ−1ψ H)−1(b0−β∗+ γ∗) for (1),

(b0−β∗)(V0 +(HT Σ−1ψ H)−1(b0−β∗+ γ∗) for (2) ,

2γ + γ∗ for (3) ,

γ∗ for (4) ,

(2.48)

with β∗ = (HT Σ−1ψ H)−1(HT Σ

−1ψ yc), γ∗ = yT

c (Σ−1ψ −Σ

−1ψ H(HT Σ

−1ψ H)−1HT Σ

−1ψ )yc and:

ασ =

N2 +α for (1),N2 for (2) ,

N− p2 +α for (3) ,

N− p2 for (4) .

(2.49)

A posterior predictive distribution can also be available by integrating, over the domain of β, π(β|yc,σ2) andthen integrating the result over the domain of σ2. Details of this method are pursued in Le Gratiet (2013). Anothermethod, based on cross validation, is used by Bachoc et al. (2014) to estimate the parameters.

2.2.3 Covariance functions

The choice of the covariance function is crucial in establishing a Gaussian process. Figure 2.8 illustrates the differentresults that one can have with different covariance functions.

−5

0

5

10

0.00 0.25 0.50 0.75 1.00x

y

−5

0

5

10

0.00 0.25 0.50 0.75 1.00x

y

−5

0

5

10

0.00 0.25 0.50 0.75 1.00x

y

−5

0

5

10

0.00 0.25 0.50 0.75 1.00x

y

Figure 2.8: Different Gaussian process estimation for the toy function Equation (2.30) with σ2 = 5 and ψ = 0.1 butwith different covariance functions. On the first panel (on the left) the Gaussian process is estimated with Gaussiancovariance function. On the second panel (on the middled left), with a Matérn 5/2. On the third panel (on themiddle right), with a Matérn 3/2. And, on the fourth panel (on the right), with an exponential.

This section recalls the main correlation functions, that will be used for further purpose in the thesis, but thereader is invited to refer to Rasmussen (2004) or Santner et al. (2013) to have more details. In this section and in allthe thesis, when it not specified, the Gaussian processes will be considered as stationary (the covariance function isa function of the only argument θ−θ′ and depends on the distance and the direction of two points in the space)and isotropic (the covariance function is a function of the only argument |θ−θ′| the euclidean distance betweenθ and θ′). When the Gaussian process is used to emulate the function fc, the choice of the correlation functiondepends on the regularity of the function. Just as a recall, cσ2,ψ = σ2Kψ where c represents the covariance and Kψ

47

Cha

pter

2

Chapter 2 – Statistical tools for numerical code calibration

the correlation functions.

The exponential functions

The exponential family can be written for a euclidean distance d such as d= |θ−θ′| with θ = (θ1, . . . ,θp)T and

θ′ = (θ ′1, . . . ,θ′p)

T :

for 0 < ν ≤ 2 Kψ,ν(d) =p

∏j=1

exp− d

ν

ψ

. (2.50)

Exponential function family

This function is a product of unidimensional correlation functions. The parameter ψ is a vector that specifiesthe correlation length in each direction. If ψ is a scalar the function is isotropic. This parameter influence thecorrelation scale of the process. If ν = 1 the function is called exponential, but if ν = 2 the function is calledGaussian (otherwise it is a generalized exponential function). For 0 < ν ≤ 2, the covariance function is differentiablein root mean square and the trajectories are almost likely continuous. However, the Gaussian covariance function isinfinitely differentiable which generates a smooth process.

The Matérn functions

The Matérn family (Matérn, 1960) can be written for a distance d such as d= |θ−θ′| with θ = (θ1, . . . ,θp)T and

θ′ = (θ ′1, . . . ,θ′p)

T :

Kψ,ν(d) =p

∏j=1

ψ j|d|ν

Γ(ν)2ν−1Jν(ψ j|d|), (2.51)

Matérn function family

where Jν is a modified ν order Bessel function. Identically as for the exponential family, the parameter ν controlsthe regularity of the process. For the Matérn family, if ν > m then the process is m differentiable in root mean square.The parameter ψ still influences the correlation scale. Usually, the Matérn 5/2 and Matérn 3/2 are considered andcorrespond to ν = 5/2 and ν = 3/2 (Rasmussen, 2004).

2.2.4 Gaussian process-based optimization

When the code to emulate is time consuming, the number of code calls is limited. Once the Gaussian process isestablished, if its variance is too high, it is possible to identify sequentially the inputs locations where the code fc

has to be run to improve the estimation. These new points are generally found around global minimum of fc overthe initial design of experiments D. This method is called Expected Improvements (EI) strategy (Jones et al., 1998)and EI is defined as:

EI(θ) = E(m− fc(θ))1 fc(θ)>m, (2.52)

Expected improvement (EI)

48

Cha

pter

2

2.2. Kriging / Gaussian processes

where θ is the parameter vector, m the global minimum of fc over the initial design of experiments D and theexpectancy is taken on the distribution of fc. Figure 2.9 illustrates the values of the EI for the emulated Gaussianprocess on the function defined Equation (2.30).

−10

−5

0

5

10

0.00 0.25 0.50 0.75 1.00

y

0.0

0.1

0.2

0.3

0.00 0.25 0.50 0.75 1.00x

EI

Figure 2.9: Expected improvement computed for the Gaussian process established on the function defined Equation(2.30) with 5 points in the original design of experiments.

Let us consider that k simulations are run and the outputs are expressed by fc(Dk) where Dk is the design ofexperiment used to establish the kth Gaussian process. The EI criterion expresses the expected improvement broughtby a new point close to the global minimum of fc. The new point θk+1 will then be the one that maximizes theexpected improvement:

θk+1 = argmaxθ

EIk(θ),

= argmaxθ

E[((mk)−Fk(θk))1Fk(θk)>m

],

(2.53)

where mk = min f (Dk) and Fk is the current Gaussian process (established with Dk). If the Gaussian process(GP) used to emulate fc was deterministic, the EI criterion would be mk−µF(θ) if µF(θ)< mk (with µF(θ) themean of the Gaussian process) and 0 otherwise. In the case where F is stochastic, the expectation of its truncateddifference with respect to the distribution of Fk represents the EI criterion. The algorithm that consists in improvingthe Gaussian process by updating each time the design of experiments based on the EI criterion is called EfficientGlobal Optimization (EGO) (Jones et al., 1998). Figure 2.10 illustrates two iterations of the algorithm and shows ateach time the new point to add in the new DOE.

The convergence of the EGO has been proven by Vazquez and Bect (2010) under some assumptions suchas non-degeneracy of the covariance function of the GP (called the Non-Empty-Ball property) and π-almost allcontinuous functions, where π is the probability distribution of the GP. Then, Bull (2011) has shown that expectedimprovement can converge near-optimally, but a naive implementation may not converge at all. Practically the algo-rithm is stopped after a limited (and authorized) number of calls or when the improvement mk becomes negligible.In Ginsbourger (2009) it is shown that with comparable levels of performance and based the second stop criterion,EGO algorithm does not need as much code call as for other regular optimization methods. Figure 2.11 illustratesthe end of the EGO algorithm for the toy function of Equation (2.30).

49

Cha

pter

2

Chapter 2 – Statistical tools for numerical code calibration

−5

0

5

10

0.00 0.25 0.50 0.75 1.00

y

0.000

0.025

0.050

0.075

0.100

0.00 0.25 0.50 0.75 1.00x

EI

−5

0

5

10

0.00 0.25 0.50 0.75 1.00

y

0.0

0.1

0.2

0.3

0.4

0.00 0.25 0.50 0.75 1.00x

EI

Figure 2.10: 2 EGO iterations with on top the GP updated based on the previous point found with the EI criterionand on the bottom the EI values corresponding to the GP on top. The point in orange is the EI maximum used toestablish the following GP.

−5

0

5

10

0.00 0.25 0.50 0.75 1.00

y

0.000

0.003

0.006

0.009

0.00 0.25 0.50 0.75 1.00x

EI

−5

0

5

10

0.00 0.25 0.50 0.75 1.00

y

−0.50

−0.25

0.00

0.25

0.50

0.00 0.25 0.50 0.75 1.00x

EI

Figure 2.11: 2 last of the 6 iterations of the EGO algorithm.

50

Cha

pter

2

2.3. Design of experiments

2.3 Design of experiments

So far, the Design Of Experiments (DOE) to build a Gaussian process as a surrogate or for the Sobol sensitivityanalysis was considered known. In both cases, the number of code calls matches with the number of points in theDOE. So it is important, especially when the code is time consuming, to find the design that gives the maximumof information but with a limited number of points. Let us keep the same notation introduced in Subsection 2.1.2where the DOE was represented by D = θ1, . . . ,θN with D⊂QN and Q ⊂ Rp. A mathematical criterion, thatspecifies that the points are well spread in the parameter input space, needs to be defined. Several criteria existsuch as the sampling property of the points in D or the distances between the points. The DOE introduced forthe Morris method, in Subsection 2.1.1, is particular to the method because it focuses on evaluating the impact ofa displacement in the parameter input space. This design, called factorial design, is particular to this sensitivityanalysis method, and is not relevant here. In this section the space Q will be consider hypercubic, which means thateach dimension is bounded. First the sampling criterion of the points in D is presented and will be followed bydetails on the distance criterion between the points.

2.3.1 Sampling criteria

In the particular cases of the sensitivity analysis or the establishment of the surrogate of a function fc, considered asa deterministic function, it is interesting to sample points in Q such as to have a fairly good representation of fc(D).The θ1, . . . ,θN are generated such as the expected value of E( fc(θ)) (for the sake of simplicity, fc(D) = y(D)) canbe estimated by:

¯y(D) =1N

N

∑i=1

fc(θi). (2.54)

For a time consuming code function fc, it is important to chose a sampling scheme that allows to estimateE( fc(θ)) well but with N as small as possible. The first idea for sampling D is to generate N iid random vectors withan identical probability distribution (usually uniform) on D. This is called the simple random sampling. A secondsampling method, called stratified sampling, considers that a population is represented by multiple subpopulations(or strata) and sampled in each stratum. However, McKay et al. (1979) introduced an alternative method calledLatin Hypercube Design (LHD), also named Latin Hypercube Sampling (LHS). Considering that Q = [0,1]p, itconsists in placing the point of D as for i = 1, . . . ,N and j = 1, . . . , p:

θi j =pi j−ui j

N, (2.55)

Coordinates in the LHD

where P = (pi j)1≤i≤N,1≤ j≤p is a matrix which takes in the jth column random permutations of the integers1, . . . ,N and where U(ui j)1≤i≤N,1≤ j≤p is a matrix which takes in the jth column a N sampling in a Uniformdistribution between 0 and 1. The LHS is based on the stratified sampling and allows to separate the points in N

subdivisions of the space Q. Let us take, as an example, Q such as Q = [0,1]2 with θ = (θ1,θ2)T . The LHS of

N = 6 points in Q obtained is displayed in Figure 2.12.

McKay et al. (1979) shows that if D1 is a simple random sample and if D2 is a LHS then:

Var( ¯Y (D1))≥Var( ¯Y (D2)), (2.56)

where ¯Y (D) is a random variable which takes as a realization ¯y(D) defined in Equation (2.54). It shows that thesampling error is lower when the LHS is performed. Without any hypotheses on the monotony of fc, Stein (1987)has demonstrated that Var( ¯Y (D2)) is asymptotically lower than Var( ¯Y (D1)) if the second order moment exists.

51

Cha

pter

2

Chapter 2 – Statistical tools for numerical code calibration

0.00

0.25

0.50

0.75

1.00

0.00 0.25 0.50 0.75 1.00θ1

θ2

Figure 2.12: 6 points sampled with a LHS for Q = [0,1]2.

Moreover, the LHS performs well when there are worthless dimensions. However, the LHD is not necessarily agood design for an exploratory DOE. So far the LHD has been build under the hypothesis that the input parametervector θ was following an uniform distribution on [0,1]p with independence between each components. The LHScan be performed with a different distribution for θ on [0,1]p but also with dependence between the components ofθ (Stein, 1987). A generalization of the LHD design called randomized orthogonal arrays will not be developedhere. For details the reader is referred to the articles of Owen (1992); Tang (1993).

2.3.2 Distance between the points criteria

As it is said above, the LHD can be not optimal regarding space exploration. For example in Figure 2.13 shows thatsome configuration of LHD does not cover properly the entire input parameter space.

0.00

0.25

0.50

0.75

1.00

0.00 0.25 0.50 0.75 1.00θ1

θ2

Figure 2.13: 6 points sampled with a LHS for Q = [0,1]2.

Another criterion can be added to the original design of the LHD to optimize the exploration of the space.Johnson et al. (1990) has introduced two criteria based on the distance between two points. These criteria are usedto testify the exploration quality of a DOE. First, a DOE D (defined above) is called minimax if it minimizes:

52

Cha

pter

2

2.3. Design of experiments

hD = supθ∈Q

min1≤i≤N

||θ−θi||2. (2.57)

minimax

The distance distance by || • || refers in this section and along the thesis for the euclidean distance. Similarly, aDOE D is called maximin if it maximizes:

δD = min1≤i, j≤N

||θi−θ j||2. (2.58)

maximin

As the parameter input space is a compact and as the functions D 7→ hD and D 7→ δD are continuous , the existenceof minimax and maximin design is guarantied. A minimax design will ensure that all points in Q will not be farfrom a point in D. A maximin design tries to create a maximum space between the points to avoid replications.

Morris and Mitchell (1995) proposed to look for a maximin design in the latin hypercube classes. It is thenpossible to use the maximin criterion on a LHS to add dispersion property on the original DOE. Figure 2.14illustrates two different maximin LHS designs obtained with the algorithm introduced in Morris and Mitchell (1995).Several optimal designs exists and they can be compared to Figure 2.12. In the rest of the thesis, maximin LHSrefers to this algorithm of Morris and Mitchell (1995).

0.00

0.25

0.50

0.75

1.00

0.00 0.25 0.50 0.75 1.00θ1

θ2

0.00

0.25

0.50

0.75

1.00

0.00 0.25 0.50 0.75 1.00θ1

θ2

Figure 2.14: 2 6-sized maximin LHS performed with the algorithm of Morris and Mitchell (1995) for 2 parameters.

When the number of input parameter increase, the question of the optimization according distance criterion canbe highlighted. Indeed, it exists several other distance criteria than maximin distance such as the L2-discrepancy, theminimum spanning tree (MST) (Dussert et al., 1986; Franco et al., 2009) etc... Some optimizations have also beendeveloped as the Enhanced Stochastic Evolutionary algorithm (ESE) (Jin et al., 2003) or the Simulated Annealing(SA). A review of the optimization methods is available in Viana et al. (2010). Morris and Mitchell (1995) haveproposed a version called MM SA (for Morris and Mitchell SA, that is detailed above) and Marrel (2008) hasproposed a version called the Boussouf SA. In their paper, Damblin et al. (2013) aim to compare several optimization

53

Cha

pter

2

Chapter 2 – Statistical tools for numerical code calibration

methods in the high dimensional case. They analyzed that the ESE algorithm behaves more efficiently than the MMSA and seems to be a good choice as an optimization. They also state that the MST criterion, which analyzes thegeometrical profile design according to the distance between the points, is preferable to the the maximin design.

The maximin design is nonetheless good. Indeed, in its paper Schaback (2007) had provided convergence proofsfor a generalized non-square version of Kansa’s collocation method (Kansa, 1985), showing that the convergencerates are determined by approximation results for non-stationary meshless-kernel-based trial spaces. Furthermore,Auffray et al. (2010) had shown that the maximin design which ensures that any point of Q is not far from thepoints of the design that leads to the best performances.

2.4 Principal component analysis (PCA)

The Principal Component Analysis (PCA) (introduced in Pearson (1901)) belongs to the group of multivariateanalysis descriptive methods called factorial methods (Husson et al., 2017). In the sense where these methods aredescriptive, they are not based on a probabilistic model, but depend on a geometric model. The PCA proposes,from a rectangular table of data composed of n observations (also called individuals) with p variables, graphicalrepresentations of these observations and variables. These data can be derived from a sampling procedure or fromthe observation of an entire population. Graphical representations allow to visualize if it exists a structure, a priori

unknown, on these data. Similarly, graphical representations can help to identify the linear link structures onvariables considered.

Let us consider that the measures are performed on n units u1, . . . ,un and the p quantitative variables thatrepresent these measures are v1, . . . ,vp. The table of data can be written as:

X =

x11 . . . x1p...

...xn1 . . . xnp

, (2.59)

where the jth column represents the data on the n units for the quantitative variable v j and the ith line the data of p

variables for the unit ui. Let us call Ui the vector of the data for the ith unit such as Ui = (xi1, . . . ,xip)T and Vj the

vector of data for the variable j such as Vj = (x1 j, . . . ,xn j)T .

For the graphical representation of the units, we choose an affine space with , as an origin, a particular point ofRp (for example the point with the null coordinates). Similarly, in Rn each variable can be represented by a point.The set of points that represent the variables is called a “scatter plot”. However, the dimension of these spaces isin general larger than 2 and even at 3 and we cannot visualize these representations. The general idea of factorialmethods is to find a new plan or axis system such as the projections of these scatter plots on this axis or plan allowto reconstitute points positions relative to each other (i.e. having the least distorted images as possible).

2.4.1 Distance

To make a geometric representation, it is necessary to choose a distance between two points of the space. Thedistance used by the PCA is the classical Euclidean distance:

d2(ui,ui′) =p

∑j=1

(xi j− xi′ j)2. (2.60)

54

Cha

pter

2

2.4. Principal component analysis (PCA)

With this distance, all the variables are playing the same role and the axes defined by the variables are constitutingan orthogonal basis. A dot product can be associated to this distance:

〈 →oui,→

oui′〉=p

∑j=1

xi jxi′ j =UTi Ui′ , (2.61)

as well as a norm:

|| →oui||2 =p

∑j=1

x2i j =U

Ti Ui. (2.62)

We can then define the angle α between two vectors by its cosine:

cos(α) =〈 →oui,

→oui′〉

|| →oui|| ||→

oui′ ||=

UTi Ui′√

(UTi Ui)(U

Ti′ Ui′)

. (2.63)

The point with null coordinates is not always a satisfactory origin because if the coordinates of the points in thescatter plot are high, the plotted points are far away from the origin. It seems wiser to chose an origin linked to thescatter plot itself: the center of gravity. We will consider for the following that we center the matrixX such as:

Xc =

x11− x•1 . . . x1p− x•p

......

xn1− x•1 . . . xnp− x•p

. (2.64)

where x•p is the mean of the corresponding column inX . The coordinates of the vectors Ui and Vj are alsocentered and can be written as:

Uci =

xi1− x•1

...xip− x•p

,and Vc j =

xi j− x• j

...xn j− x• j

. (2.65)

2.4.2 Moments of inertia

Les us note IG the moment of inertia of the individuals scatter plot with respect to the center of gravity G:

IG =1n

n

∑i=1

d2(G,ui) =1n

n

∑i=1

p

∑j=1

(xi j− x• j)2 =

1n

n

∑i=1UT

ciUci. (2.66)

This moment of total inertia is of interest because it measures the dispersion of the individual scatter plot fromthe center of gravity. If this moment of inertia is large, it means the scatter plot is dispersed. However, if it is small,the scatter plot is highly concentrated on its center of gravity. Another way to write Equation (2.66) can be obtainedby inverting the sums:

IG =p

∑j=1

[1n

n

∑i=1

(xi j− x• j)2]=

p

∑j=1

Var(v j), (2.67)

where Var(v j) is the empirical variance of the variable v j. From this form, it can be seen that total inertia isequal to the trace of the covariance matrix Σ of the p variables v j:

IG = trace(Σ). (2.68)

55

Cha

pter

2

Chapter 2 – Statistical tools for numerical code calibration

The inertia of the scatter plot from an axis ∆ passing by G is by definition:

I∆ =1n

n

∑i=1

d2(h∆i,ui), (2.69)

where h∆i is the orthogonal projection of ui on ∆. This inertia measures the proximity of the scatter plot to the axis∆.

The inertia of the scatter plot from a linear subspace V passing by G, is similarly defined, and is equal to:

IV =1n

n

∑i=1

d2(hVi,ui), (2.70)

where hVi is the orthogonal projection of ui on the linear subspace V . Let us not V ∗ the orthogonal complementof V in Rp and hV ∗i the orthogonal projection of ui on V ∗. By the Pythagorean theorem:

d2(hVi,ui)+d2(hV ∗i,ui) = d2(G,ui) = d2(G,hVi)+d2(G,hV ∗i). (2.71)

From Equation (2.71), it can be concluded that:

IV + IV ∗ = IG. (2.72)

Huygens theorem

In the particular case where the linear subspace V has a dimension of 1, IV ∗ is a measure of the scatter plotelongation along the axis of the linear subspace. By projecting the scatter plot on a linear subspace V , the inertiameasured by IV is lost (only IV ∗ is kept). If Rp is decomposed in a sum of orthogonal linear subspace such as:

∆1⊕∆2⊕·· ·⊕∆p, (2.73)

then (with a Pythagorean theorem):IG = I∆∗1

+ · · ·+ I∆∗p . (2.74)

2.4.3 Axis of minimum inertia

The axis ∆1 which is the closest to all the individuals in the scatter plot is an axis passing by G and has the minimuminertia I∆1 . This axis is interesting because, when the projection will be done, this will give the least distorted imageof the scatter plot. Looking for ∆1 such as I∆1 is minimum is equivalent to look for ∆1 such as I∆∗1

is maximum

(Equation (2.74)). Let us note the unitary vector→

Ga1 of the axis ∆1. Then, the vector→

Ga1 is looked for such as I∆∗1

is maximum under the constraint of ||→

Ga1||2 = 1.

We can write:d2(G,h∆1i) = 〈

→Gui,

→Ga1〉2 = aT

1UciUTci a1. (2.75)

Using the property of symmetry of the dot product, it can be deduced that:

I∆∗1=

1n

n

∑i=1

aT1UciU

Tci a1 = aT

1

[1n

n

∑i=1UciU

Tci

]a1. (2.76)

56

Cha

pter

2

2.4. Principal component analysis (PCA)

The empirical variance Σ of the p variables is identified under the bracket of Equation (2.76). Then:

I∆∗1= aT

1 Σa1, (2.77)

with,

||→

Ga1||2 = aT1 a1. (2.78)

The problem is summarized at finding a1 that maximize aT1 Σa1 under the constraint aT

1 a1 = 1. The componentsof a1 are unknown and are linked by a constraint. To solve such a problem, a Lagrange multiplier method canbe used. This method consists in finding the optimums of a function f (t1, . . . , tp) where the p variables arelinked by a relation l(t1, . . . , tp) = cte. The p partial derivatives with respect to each variable of the functiong(t1, . . . , tp) = f (t1, . . . , tp)−λ (l(t1, . . . , tp)− cte) are calculated and annulled to get a system of p+1 equationswith p+1 variables (after adding the constraint term). The term λ is called the Lagrange multiplier. In the previouscase, partial derivatives of

g(a1) = g(a11, . . . ,a1p) = aT1 Σa1−λ1(aT

1 a1−1) (2.79)

has to be found. Using the derivative of a quadratic form with respect of a vector, it can be proven that:

∂g(a1)

∂a1= 2Σa1−2λ1a1 = 0. (2.80)

The system to solver is then: Σ−λ1a1 = 0 (1)

aT1 a1−1 = 0 (2)

(2.81)

From Equation (2.81), matrix Equation (1) allows to show that a1 is an eigenvector of Σ with eigenvalue λ1.Multiplying matrix Equation (1) by aT

1 on the left

aT1 Σa1−λ1aT

1 a1 = 0 (2.82)

is obtained and using the Equation (2) in Equation 2.81, it is shown that:

aT1 Σa1 = λ1. (2.83)

From Equation (2.77), the first term of Equation (2.83) is replaced and it can be stated that λ1 = I∆∗1. That means

the value λ1 is the highest eigenvalue of Σ and this eigenvalue is the inertia carried by ∆1. Then, the axis ∆1 has forunit vector the first eigenvector associated at the highest eigenvalue of the covariance matrix Σ.

However, only one axis might not be sufficient to represent all the data. Then, a second axis ∆2 is looked forwhich is orthogonal to ∆1 and has a minimal inertia. Identically as before, let us define the axis ∆2 passing by G anddefined by its unit vector u2. The inertia of the individuals scatter plot with respect to it orthogonal complement is:

I∆∗2= aT

2 Σa2. (2.84)

The inertia defined Equation (2.84) has to be maximum under the constraints:

aT2 a2 = 1 and aT

2 a1 = 0. (2.85)

57

Cha

pter

2

Chapter 2 – Statistical tools for numerical code calibration

The second constraint Equation (2.85) represents the orthogonality of u1 with u2. With the Lagrange multipliersmethod, but with two constraints, it can be shown that a2 is the eigenvector of the covariance matrix Σ whichcorresponds to the second highest eigenvalue. It can be also demonstrated that the inertia is maximum in the linearsubspace defined with the unit vectors u1 and u2.

The same procedure can be extended but can only be done p times. The covariance matrix Σ is real andsymmetrical, thus it has only p eigenvectors (building an orthogonal basis of Rp).

2.4.4 Contribution to the total inertia

All the axes does not have the same contribution on the total inertia. Using the Huygens theorem Equation 2.72, thetotal inertia can be decomposed as:

IG = I∆∗1+ · · ·+ I∆∗p = λ1 + · · ·+λp. (2.86)

The absolute contribution to the total inertia of the axis ∆k is equal to:

ca(∆k/IG) = λk, (2.87)

but its relative contribution is defined as:

cr(∆k/IG) =λk

IG. (2.88)

For the rest of the thesis, the term “percent of inertia explained by the axis ∆k” will be used as a reference of therelative contribution. It is possible to extend these definitions at all the generated linear subspaces. For example, thepercent of inertia explained by the linear subspace generated by the two eigenvector associated at the two highesteigenvalues is defined as:

cr(∆1⊕∆2/IG) =λ1 +λ2

IG. (2.89)

These inertia percentages are indicators of the part of variability of the scatter plot explained by these subspaces.Practically, only the d (d < p) first axis are considered because they explain a percentage of inertia close to 1. Theother axes are then neglected.

2.4.5 Graphical representations

Individuals representation quality

To have a graphical representation of the individuals in the plans defined by the new axis, it is necessary to calculatethe coordinates of each individuals in the new axis. To get yik the coordinate of the unit ui on the axis ∆ j, an

orthogonal projection of the vector→

Gui is achieved:

yik = 〈→

Gui,→ak〉= aT

kUci, (2.90)

and,Yi =A

TUci, (2.91)

where Yi is the vector of coordinates of the unit ui andA is the change of basis matrix (A is the matrix composedof the orthogonal eigenvectors, which have a norm equal to 1, is an orthogonal matrix and its inverse is equal to its

58

Cha

pter

2

2.4. Principal component analysis (PCA)

transposed).

The square of the cosine of the angle αik between→

Gui and an axis ∆k is equal to:

cos2(αik) =〈→

Gui,→

Gak〉2

||→

Gui||2=

aTkUciU

Tci ak

UTciUci

=

[∑

pj=1(xi j− x• j)ak j

]2

∑pj=1(xi j− x• j)2 . (2.92)

As mentioned above, by using the Pythagorean theorem, the square of the cosine of the angle αikk′ between→

Gui

and the plan generated by the two axes ∆k⊕∆k′ can be calculated and:

cos2(αikk′) = cos2(αik)+ cos2(αik′). (2.93)

Contribution of an individual to an axis

When I∆∗k(the inertia carried by the axes ∆k) is calculated, it is possible to see the part of the inertia attributable to

an individual ui in particular. From Equation (2.70), I∆∗k= 1

n ∑ni=1 d2(h∆ki ,G) and the absolute contribution of ui of

this inertia is:ca(ui/∆k) =

1n

d2(h∆ki ,G), (2.94)

because all the individuals have the same weight. Practically, an individual will have a greater contribution to theaxis when its projection will be far from the center of gravity. The other way around, an individual which has aprojection close to the center of gravity will have a weak contribution to the axis. It is also possible, for the sameparticular individual ui, to give its relative contribution to the inertia carried by ∆k:

cr(ui/∆k) =1n d2(h∆ki ,G)

I∆∗k

=1n 〈→

Gui,→

Gak〉2

λk=

1n aT

kUciUTci ak

λk. (2.95)

Note that ∑ni=1 cr(ui/∆k) = 1.

Graphical representation of the variables

The individuals have been represented in the space of the original variables and the change of basis has been madein this original space. New axes are then linear combination of original axes and can be considered as new variableswhich are a linear combination of original variables. These new variables are called “principal components”. Let usnote Z1, . . . ,Zp the principal components, Zk being the new variable corresponding to the axes ∆k and:

Zk =p

∑j=1

ak jVc j =Xcak, (2.96)

where Vc j is the centered variable vector defined in Equation (2.65) and Xc the matrix of centering dataintroduced Equation (2.64). In general, the principal components can be written as:

Z = [Z1, . . . ,Zp] =XcA. (2.97)

It is also interesting to visualize how original variables are linked to the principal components by calculating thecorrelations. Original variables representation will be done by taking as coordinates the correlation coefficients withthe new variables. This representation is called “correlation circle” because of the fact that a correlation coefficientis always between −1 and 1 which leads to have all the points in a circle of radius 1. The variances of the principalcomponents are:

59

Cha

pter

2

Chapter 2 – Statistical tools for numerical code calibration

Var(Zk) =1n

aTkX

TcXxak = aT

k Σak = λk, (2.98)

their covariances are:

Cov(Zk,Vc j) =1n

aTkX

TcVc j =

1n

aTkX

TcXc

0...1...0

= aT

k Σ

0...1...0

= λkaT

k

0...1...0

= λkak j, (2.99)

and their correlations with the original variables are:

Cor(Zk,Vc j) =√

λkak j√

Var(V j), (2.100)

where ak j is the jth coordinates of the unit vector ak of ∆k. Generally, the covariance matrix of the principalcomponents ΣZ can be written as:

ΣZ =1nATXT

cXcA=AT ΣA= Λ, (2.101)

where Λ is the diagonal matrix of the eigenvalues of Σ:

Λ =

λ1 (0)

. . .

(0) λp

, (2.102)

and the covariance matrix between the principal components and original variables is:

Cov(Z,V ) =1nXTcXcA= ΣA=AΛ. (2.103)

Example on the decathlon data set

As an example, let us consider the decathlon data set in the FactoMineR (Husson et al., 2018) package. The data setis an example of decathlon data which refers to athletes’ performance during two athletic meetings (2004 OlympicGames or 2004 Decastar). The data set is made of 41 rows and 13 columns where the ten first column only are kept.These columns correspond to the performances of the athletes of the ten events of the decathlon.

Figure 2.15: Graphical representation of the individuals (on the left) and the variables (on the right) of the decathlondata set.

60

Cha

pter

2

2.5. Monte Carlo Markov Chains techniques

Figure 2.15 represents the individuals and the variables of the decathlon data set. For each axis, the percentageof the inertia carried is indicated and both axis (the ones used for the representation) contain 50% of the totalinertia. The right panel of Figure 2.15 illustrates the correlation between the variables and for example the variable“100m” is negatively correlated with the variable “long.jump”. This means that when an athlete jumps far, hedoes not achieve so good performances in the 100m. The left panel of Figure 2.15 illustrates in two axis a globalrepresentation of the performances of the athletes. The first axis strongly represents the speed events and long jumpevent (given from the right panel of Figure 2.15). This means for an individual that has a high value on this axis, heis more likely better on these events than an individual that has a weak value on the axis. For example Karpov seemsto have better performed than Bourguignon in these events. The second axis represents the variables “Discuss” and“shot put” and similarly a high value of the individual on this axis means that he performed well on these events.The winner of the 2004 Olympic Games is Sebrie who has reasonable value on both axis. The variable “Discus”,“Shot.put” and “High.jump” are not correlated to “100m”, “400m”, “110m.hurdle” and “Long.jump” which meansthat strength and speed are not correlated.

2.5 Monte Carlo Markov Chains techniques

Sampling from a posterior distribution is the only mean to get information about the posterior distribution if thereis no conjugacy in the model. Several Monte Carlo Markov Chains (MCMC) have been developed in that sense(Robert and Casella, 2013; Andrieu et al., 2003). Let us consider a general example, in a Bayesian framework,where the posterior density of a p dimensional parameter vector θ is looked for. Let us define a probability space(Ω,F ,π) where Ω stands for a sample space, F a σ -algebra on Ω and π a probability on F . The parameter vectoris defined as θ ∈Q ⊂ Rp and the n dimensional experimental vector as y ∈ E ⊂ Rn. From Bayes formula, it ispossible to write straightforwardly that:

π(θ|y) = L (y;θ)π(θ)m(y)

, (2.104)

where m(y) =∫θ∈Q L (y;θ)π(θ), but as the term m(y) is independent from θ, Equation (2.104) can be

rewritten as:π(θ|y) ∝ L (y;θ)π(θ). (2.105)

When the prior distribution is not conjugate to the the likelihood, π(θ|y) cannot be expressed analytically. Thechoice of the sampling method is partly conditioned by the knowledge of full conditional distributions (i.e. theknowledge of π(θi|y,θ(−i)) ∀i ∈ [1, . . . ,n], θ(−i) representing the vector θ without the ith component). In casewhere these full conditional distributions are known, the Gibbs sampler can be used. Otherwise, Metropolis Hastingsalgorithms could be applied to sample in the posterior distribution. The full conditional distribution can be writtenproportionally to the joint distribution such as:

π(θi|y,θ−i) =π(θ,y)

π(θ1, . . . ,θi−1,θi+1, . . . ,θp,y)∝ π(θ,y). (2.106)

Then from Equation (2.105):

π(θi|θ−i,y) ∝ L (y;θ)π(θ)

∝ L (y;θ)π(θ1|θ2, . . . ,θp)π(θ2|θ3, . . . ,θp) . . .π(θp−1|θp)π(θp).(2.107)

61

Cha

pter

2

Chapter 2 – Statistical tools for numerical code calibration

And, if all the components of θ are independent then:

π(θi|θi,y) ∝ L (y;θ)p

∏i=1

π(θi). (2.108)

The aim of the MCMC is then to produce a Markov chain whose invariant distributionis the posterior distribution

MCMC goal

2.5.1 Gibbs sampler

If the full conditional distributions are known, then it is possible to sample in each of them. This way, if an initialstate θ(s), such as θ(s) = (θ

(s)1 , . . . ,θ

(s)p ), is considered, a new state can be sampled from:

1. sample θ(s+1)1 ∼ π(θ

(s)1 |θ

(s)2 , . . . ,θ

(s)p ,y)

2. sample θ(s+1)2 ∼ π(θ

(s)2 |θ

(s)1 ,θ

(s)3 , . . . ,θ

(s)p ,y)

...

(p). sample θ(s+1)p ∼ π(θ

(s)p |θ(s)1 , . . . ,θ

(s)p−1,y)

(p+1). θ(s+1) = (θ(s+1)1 , . . . ,θ

(s+1)p )

The algorithm that repeats this sequence S times is called Gibbs sampler and generates a dependent sequence ofparameters: θ = θ(1), . . . ,θ(S) (Robert and Casella, 2013; Hoff, 2009). The algorithm can be written as:

Algorithm 1 Gibbs sampler

θ(1) = θinitfor i in 2 : S do

for j in 1 : p doθ(i)j ∼ π(θ

(i−1)j |θ(i)1... j−1,θ

(i)j+1...p,y)

end forθ(i) = (θ

(i)1 , . . . ,θ

(i)p )

end for

For example, let us consider a sample of data y such as each component are a realization of the random variableY which is following Y ∼N (µ, 1

τ) where µ is the population mean and τ the population precision (inverse of the

variance). Let us consider n the sample size, y the sample mean and s2 the sample variance. A priori, there is nocorrelation between the population mean and the population variance:

π(µ,τ) = π(µ)π(τ). (2.109)

Considering that π(µ) ∝ 1 and π(τ) ∝ τ−1 as prior densities (Casella and George, 1992), the conditionalposterior densities are explicit and can be expressed as:

62

Cha

pter

2

2.5. Monte Carlo Markov Chains techniques

µ|τ,y∼N(

y,1

nτ

), (2.110)

τ|µ,y∼ Γ

(n2,

2(n−1)s2 +n(µ− y)2

). (2.111)

As the conditional posterior densities can be written, the Gibbs sampler can be run and then have access to thejoint posterior density. Figure 2.16 is the result of the Gibbs sampler in this example.

0.2

0.4

0.6

0.8

13.5 14.0 14.5 15.0 15.5 16.0μ

τ

Figure 2.16: Gibbs sampler completed for 10000 iteration with a 1000 burn-in sample.

When conjugate prior densities are not accessible, the Gibbs sampler cannot be used. To sample in the posterior

joint density, another algorithm can be performed.

2.5.2 Metropolis Hastings

The Metropolis Hastings algorithm has been introduced in Metropolis et al. (1953) by Metropolis but with aspecific Boltzmann distribution. In Hastings (1970), Hastings has generalized the algorithm for every specificcases. To approximate the posterior density, it is not possible to generate independent and identically distributed(i.i.d.) samples from π(θ|y). To do so, a large collection of θ values θ(1), . . . ,θ(s), whose empirical distributionapproximates π(θ|y), are generated. Let us suppose that we have a working collection such as θ(1), . . . ,θ(s)to which we would like to add a new value θ(s+1). Let us consider a new point θ(∗) which is nearby θ(s). Ifπ(θ(∗)|y)> π(θ(s)|y), it means that θ(∗) has a probability to be in the set higher than θ(s) which is already in theset. In that particular case θ(s+1) can take the value of θ(∗). However, if π(θ(∗)|y) < π(θ(s)|y), θ (∗) should notbe necessarily rejected. A decision, based on the comparison of π(θ(∗)|y) and π(θ(s)|y) has to be established.Fortunately, this comparison can be made even if π(θ|y) cannot be computed (from Equation (2.104)):

r =π(θ(∗)|y)π(θ(s)|y)

=L (θ(∗);y)π(θ(∗))

m(y)

m(y)

L (θ(s);y)π(θ(s))=

L (θ(∗);y)π(θ(∗))L (θ(s);y)π(θ(s))

. (2.112)

The intuition is when r > 1, π(θ(∗)|y)> π(θ(s)|y) and θ(∗) should be accepted in the set. On the other hand ifr < 1, the probability to add θ(∗) in the set is equal to the min(1,r). It is similarly as comparing r to a value that hasbeen sampled from an uniform distribution between 0 and 1 (let us call this value u). Then if r > u, θ(∗) is acceptedin the set, if r < u, then θ(∗) is rejected.

63

Cha

pter

2

Chapter 2 – Statistical tools for numerical code calibration

The new point θ(∗), which is nearby θ(s), is generated from a proposal distribution (also called jumping

distribution in Gelman et al. (1995)). This distribution q(θ(∗)|θ(s)) can be distinguished in two cases. The first, thedistribution is symmetric such as q(θ(∗)|θ(s)) = q(θ(s)|θ(∗)) which is the case for:

• q(θ(∗)|θ(s)) = uniform(θ(s)−δ ,θ(s)+δ ),

• q(θ(∗)|θ(s)) = normal(θ(s),kΣ2).

In that particular case the algorithm is called Metropolis algorithm and the computation of the ratio is straight-forwardly obtained from Equation (2.112). Algorithm 2 recalls the steps of this particular MCMC.

Algorithm 2 Metropolis algorithm

θ(1) = θinitτaccept = 0for i in 1 : S doθ(∗) ∼N (θ(i),kΣ)

r = π(θ(∗)|y)π(θ(i)|y)

= π(θ(∗))L (θ(∗);y)π(θ(i))L (θ(i);y)

if r > u, with u∼U (0,1) thenθ(i+1) = θ(∗)

τaccept = τaccept +1elseθ(i+1) = θ(i)

end ifend forθM = θ[Nburn−in : S]

On the other hand if q(θ(∗)|θ(s)) is not symmetric, the ratio is now given by:

r =L (θ(∗);y)π(θ(∗))L (θ(s);y)π(θ(s))

q(θ(s)|θ(∗))q(θ(∗)|θ(s))

. (2.113)

The ratio is almost surely defined because a jump can only occur if both q(θ(∗)|θ(s)) and q(θ(s)|θ(∗)) are nonzero. Such a proposal can be useful for increasing the speed of the random walk. Algorithm 3 details the steps ofthe, so-called, Metropolis Hastings algorithm.

Algorithm 3 Metropolis Hastings algorithm

θ(1) = θinitτaccept = 0for i in 1 : S doθ(∗) ∼ q(θ(i), .)

r = π(θ(∗)|y)q(θ(∗),θ(i))π(θ(i)|y)q(θ(i),θ(∗)) =

π(θ(∗))L (θ(∗);y)q(θ(∗),θ(i))π(θ(i))L (θ(i);y)q(θ(i),θ(∗))

if r > u, with u∼U (0,1) thenθ(i+1) = θ(∗)

τaccept = τaccept +1elseθ(i+1) = θ(i)

end ifend forθMH = θ[Nburn−in : S]

The choice of the nature of q is problem relevant. To simplify the programing, it is common to choose aMetropolis algorithm. It is difficult to tune all the parameters of a Metropolis algorithm, so it simplifies this problem

64

Cha

pter

2

2.5. Monte Carlo Markov Chains techniques

if a symmetric proposal distribution is taken. Between the normal or the uniform random walk, the major difficultyis to tune the parameter k or δ . The performance of such an algorithm is evaluated by its faculty to explore the inputparameter space Q. So, if Q ⊂ Rp with p high, then it is hard to find the right convergence direction.

In the case where the Metropolis is chosen with a normal proposal distribution, some improvements can bebrought especially when p is high.

2.5.3 Metropolis within Gibbs

When the number of parameter of parameter is high, the input parameter space becomes a space where it iscomplicated to move in the right direction. If the access of conditional distributions are available, one solution couldbe to compute the ratio according to the conditional distribution relative to the active parameter. Then, the algorithmwould move parameter by parameter, that increases the efficiency of the displacement. The ratio introduced inEquation (2.112) becomes:

r =π(θ

(∗)j |θ

(∗)− j ,y)

π(θ(s)j |θ

(s)− j,y)

=π(θ

(∗)j )L (θ

(∗)j ;θ(∗)1... j−1,θ

(∗)j+1...p,y)

π(θ(S)j )L (θ

(s)j ;θ(s)1... j−1,θ

(s)j+1...p,y)

. (2.114)

Algorithm 4 Metropolis within Gibbs algorithm

θ(1) = θinitτaccept = (0, . . . ,0)T

for i in 1 : S dofor j in 1 : p doθ(∗)j ∼N (θ

(i)j ,kΣ[ j, j])

r =π(θ

(∗)j )L (θ

(∗)j ;θ(∗)1... j−1,θ

(∗)j+1...p,y)

π(θ(i)j )L (θ

(i)j ;θ(i)1... j−1,θ

(i)j+1...p,y)

if r > u, with u∼U (0,1) thenθ(i+1)j = θ

(∗)

τaccept j = τaccept j +1else

θ(i+1)j = θ

(i)j

end ifend for

end forθMWG = θ[Nburn−in : S]

The strength of Algorithm 4 is to sample well in a multidimensional space. However, as it is moving componentby component, it is more time consuming than a regular Metropolis or Metropolis Hastings algorithm.

2.5.4 Improvements of the Metropolis Hastings

The major difficulty in applying a Metropolis Hastings algorithm is to use a right proposal distribution such as thenew point point is neither too far from the previous one or too close. The acceptation ratio controls the quality ofthe chain. If the acceptation ratio is too high, it means that the number of the new accepted points is too high andthat the proposal distribution is not adequate to the problem because it generates points too close to the previousone. Similarly, a too low acceptation ratio is synonym of a proposal distribution that generates points that are toofar from the previous one. Let us consider for the rest of this section (and for the major part of the thesis) that theconsidered proposal distribution is normal centered on the previous point and with variance covariance kΣ. Theproposal distribution is in that case symmetric and the major difficulty is to find the right k and Σ that make a good

65

Cha

pter

2

Chapter 2 – Statistical tools for numerical code calibration

proposal distribution.

To tune the matrix Σ is tricky because before running the Metropolis algorithm, the structure of the covariancematrix that is representative of the space is unknown. Especially, if correlation exists between parameters. A Laplaceapproximation can be performed upstream to get an initial point (which is the estimate Maximum A Posteriori) anda covariance structure which comes from the Hessian of the likelihood. This can be burdensome if the number ofparameter is high. An algorithm, called Adaptive Metropolis, has been developed by Haario et al. (2001) that tries toimprove the original Metropolis algorithm. It mainly proposes to run a certain amount, say 1000, of samples, withat first Σ = Ip and k = 1, and evaluates the covariance matrix on the 1000 samples generated. Then, by changingΣ to this new calculated covariance matrix, it improves the proposal distribution based on the samples alreadysimulated. The direction of the proposal distribution has better features than the previous one because it takes nowinto account, possible, correlation between parameters and variances corresponding to the right intensities to movein the posterior density. The Adaptive Metropolis (Algorithm 5) also continues this adaptation each 500 iterations toconverge toward the right covariance matrix and to finally have a proper proposal distribution.

Algorithm 5 Adaptive Metropolis

θ (1) = θinitτaccept = 0Σ = Ipfor i in 1 : S do

θ (∗) ∼N (θ (i),kΣ)

r = π(θ (∗)|y)π(θ (i)|y)

= π(θ (∗))L (θ (∗);y)π(θ (i))L (Θi;y)

if r > u, with u∼U (0,1) thenθ (i+1) = θ (∗)

τaccept = τaccept +1else

θ (i+1) = θ (i)

if i mod A = 0 & i > B thenΣ = cov(θ [1 : i])

end ifend if

end forΘAM = Θ[Nburn−in : S]

The condition added in Algorithm 5 compared to Algorithm 2 introduces the adaptive behavior of the algorithmwhere B stands for the threshold to compute the covariance (in the previous example 1000) and A stands for thevalue that specifies at which steps the covariance matrix is computed after the threshold (in the previous example500). Ergodicity, reversibility and convergence criteria of MCMC chains of such an algorithm are well explained inHaario et al. (2001) and will not be developed here.

It is also common to focus only on the tuning parameter k. As the mixing quality is related to the acceptanceratio, it is possible to establish an algorithm that, function of the actual acceptation rate, modify the parameter k.In Roberts et al. (1997), it is demonstrated that the “optimal efficiency” of the MCMC chain is obtained for anacceptation rate at 0.234. Algorithm 6 introduces such an algorithm that adapts the parameter k, each C iterations,function of the actual acceptation rate.

66

Cha

pter

2

2.5. Monte Carlo Markov Chains techniques

Algorithm 6 k tuned Metropolis

θ (1) = θinitτaccept = 0for i in 1 : S do

θ (∗) ∼N (θ (i),kΣ)

r = π(θ (∗)|y)π(θ (i)|y)

= π(θ (∗))L (θ (∗);y)π(θ (i))L (Θi;y)

if r > u, with u∼U (0,1) thenθ (i+1) = θ (∗)

τaccept = τaccept +1else

θ (i+1) = θ (i)

if i mod C = 0 thenif τaccept/S < 0.2 then

k = k(1− r)else

if τaccept/S > 0.5 thenk = k(1+ r)

end ifend if

end ifend if

end forΘKM = Θ[Nburn−in : S]

To improve drastically the performances the MCMC chain, the Adaptive Metropolis can be adapted by runningupstream a Metropolis within Gibbs algorithm on a limiter number of iterations. The aim is to move in the rightdirection (but slowly) with the Metropolis within Gibbs, to estimate a covariance structure. Once this covarianceis known, a regulation on the parameter k can still be useful to better tune the displacement in the space Q. Theregulation is stopped after an amount of iterations to keep convergence properties. Indeed, the convergence propertiesare preserved when changes in the chain decrease. This hybrid algorithm (detailed Algorithm 7) is the one used allalong the thesis to perform MCMC sampling estimation.

67

Cha

pter

2

Chapter 2 – Statistical tools for numerical code calibration

Algorithm 7 Hybrid Metropolis

θ(1) = θinitτaccept = (0, . . . ,0)T

for i in 1 : SMWG dofor j in 1 : p doθ(∗)j ∼N (θ

(i)j ,kΣ[ j, j])

r =π(θ

(∗)j )L (θ

(∗)j ;θ(∗)1... j−1,θ

(∗)j+1...p,y)

π(θ(i)j )L (θ

(i)j ;θ(i)1... j−1,θ

(i)j+1...p,y)

if r > u, with u∼U (0,1) thenθ(i+1)j = θ

(∗)

τaccept j = τaccept j +1else

θ(i+1)j = θ

(i)j

end ifend for

end forθMWG = θ[Nburn−in : SMWG]Σ = cov(θMWG)θ (1) = θinitτaccept = 0for i in 1 : SM do

θ (∗) ∼N (θ (i),kΣ)

r = π(θ (∗)|y)π(θ (i)|y)

= π(θ (∗))L (θ (∗);y)π(θ (i))L (Θi;y)

if r > u, with u∼U (0,1) thenθ (i+1) = θ (∗)

τaccept = τaccept +1else

θ (i+1) = θ (i)

if i mod C = 0 thenif τaccept/SM < 0.2 then

k = k(1− r)else

if τaccept/SM > 0.5 thenk = k(1+ r)

end ifend if

end ifend if

end forΘHM = Θ[Nburn−in : SM]

Some other developments have been completed for improving the Metropolis algorithm as the Delayed rejectionintroduced in Mira et al. (2001). It consists in, in case of rejection, proposing another point, based on the sameproposal distribution, with a probability of 0.5 (sampled from a binomial distribution). If the realization of thebinomial is 1, then a new point is proposed and the ratio is computed as if it was the first point. If another rejectionoccurs, the same procedure is repeated but if it is accepted, the current iteration in the algorithm can end. If therealization if the binomial is 0, the rejection is accepted and the iteration of the algorithm can also stop. Thisalgorithm allows to postpone the rejection and gives another chance to accept a specific point. The aim of thismethod is not to stay on the same point which tends to increase autocorrelation.

68

Cha

pter

3

CHAPTER3REVIEW OF THE MAIN CALIBRATION

METHODS

3.1 Numerical code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

3.1.1 Sensitivity analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

3.1.2 Prior propagation of uncertainty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

3.2 Calibration through statistical models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

3.2.1 Presentation of the models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

3.2.2 Likelihood . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

3.2.3 Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80

3.3 Application to the prediction of power from a photovoltaic (PV) plant . . . . . . . . . . . . . . . 82

3.3.1 Inference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

3.3.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

3.3.3 Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

3.4 Conclusion and discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88

The general framework of this chapter is the one introduce in Section 1.4.1 where we considered, the outputs of ζ

and fc lie in R. A vector of experimental data (yexp) which are noisy measurements of ζ is observed as a realizationof the statistical model:

M0 : ∀i ∈ J1, . . . ,nK yexpi = ζ (xi)+ εi

where ∀i ∈ J1, . . . ,nK εiiid∼N (0,σ2

err). The corresponding values of the variables xi are also observed.

Calibrating the code consists in setting the vector of parameters θ consistent in somesense with these n field data.

Code calibration

Several statistical modeling strategies have been proposed in the literature. When only measurement errorsare considered, Cox et al. (2001) use a rather simple model, considering that the code does not differ from thephenomenon under study. As for Higdon et al. (2004), Kennedy and O’Hagan (2001) and Bayarri et al. (2007),they advocate some extensions, which additionally encompass a model bias or a model error term, also dubbeddiscrepancy in the following. All of these models are reviewed and discussed in Section 3.2. The identifiabilityissues between the parameter θ and the discrepancy were already discussed in the written discussions of Kennedyand O’Hagan (2001). Tuo et al. (2015) consider the calibration task as a minimization of a loss function between

69

Cha

pter

3

Chapter 3 – Review of the main calibration methods

the code and the physical reality. In Tuo and Wu (2016), they show that this loss function leads to an estimation ofθ depending on the the chosen prior distribution of the discrepancy. Then, Plumlee (2017) advices an orthogonalityspecification for the discrepancy i.e. the discrepancy should be orthogonal to the gradient of the computer codewith respect to a loss function. Finally, Konomi et al. (2017) also propose a methodology to model the discrep-ancy as a non-stationary Gaussian process and apply it on the carbon capture with AX sorbent mathematical function.

This chapter presents the main calibration statistical model that one can found in literature which are illustratedby an application case introduced in Section 1.4.3. Section 3.1 focuses on sensitivity analysis performed for thenumerical code and the prior propagation of uncertainty, then Section 3.2 recalls the main statistical models forcode calibration with the associated likelihoods and the main estimation methods. Then Section 3.3 applies andcompares the four statistical models through the application case.

3.1 Numerical code

3.1.1 Sensitivity analysis

Based on the code presented in Section 1.4.3, a sensitivity analysis is run upstream calibration to identify theimportant parameter to keep for further study. A Morris method is first applied to screen, upstream, the non-influentparameters that can be set to their nominal value for the rest of the study. Figure 3.1 illustrates the Morris studyperformed on the numerical code.

0

1

2

3

0.0 2.5 5.0 7.5

μ*

σ

n_t

mu_t

eta

a_l

a_r

n_int

0

10

20

30

40

50

0 50 100 150 200

μ*

σ

n_t

mu_t

eta

a_l

a_r

n_int

Figure 3.1: On the left panel Morris method at noon the 24th of September 2014 and all the EEs computed at eachtime step over the two months of data.

However, as described in Section 2.1.1, Morris method only concerns functions that create a scalar output. Inthis particular case, it is necessary to have an indication of the parameter impacts over the whole time frame. On theleft panel of Figure 3.1, the Morris method is applied at noon the 24th of September 2014. The impact of severalparameters seems negligible in the graph but they could not be negligible on another moment of the day. Whenthe Morris method is completed for each time step, a “dynamic” version is visible on the right panel of Figure 3.1.Thus, it is not possible to conclude on which parameter has no influence on all the time frame of the data. Thatis why, a PCA (Section 2.4) on all the trajectories of the Morris DOE is performed. In the new subspace, “new”indices µ∗ and σ2 can be computed and then summarize the information contained in the whole time frame. Itmeans that the PCA allows to visualize eventual temporal correlation and to find new representation axis for the

70

Cha

pter

3

3.1. Numerical code

Morris indices. The left panel of Figure 3.2 represents the circle of correlation obtained by the PCA and on the rightpanel the eigenvalues for the ten first axis found.

−1.0 −0.5 0.0 0.5 1.0

−1.

0−

0.5

0.0

0.5

1.0

Variables factor map (PCA)

Dim 1 (84.26%)

Dim

2 (

15.3

9%)

V1V2V3

V4

V5

V6V7V8

V9V10V11

V12

V13

V14V15V16V17V18V19V20V21

V22

V23

V24

V25V26V27

V28

V29

V30V31

V32V33V34

V35

V36V37V38

V39

V40

V41

V42V43

V44

V45V46V47V48V49V50

V51

V52

V53

V54V55

V56

V57V58

V59

V60V61V62

V63V64V65

V66

V67

V68V69

V70

V71V72

V73

V74

V75

V76V77V78V79V80V81V82V83V84V85V86V87V88V89V90

V91

V92

V93V94V95V96V97V98V99

V100

V101V102

V103

V104V105

V106

V107V108V109V110V111V112V113V114V115V116

V117V118

V119

V120

V121V122V123V124V125V126V127V128V129V130V131V132V133V134V135V136

V137V138

V139V140V141

V142V143V144

V145

V146

V147

V148

V149V150V151

V152

V153

V154

V155V156V157V158V159V160V161V162V163

V164

V165

V166

V167V168

V169V170V171V172V173V174

V175V176

V177

V178V179

V180

V181

V182V183

V184V185V186V187V188V189V190

V191V192V193

V194

V195

V196

V197

V198V199V200

V201V202

V203

V204

V205

V206

V207

V208

V209

V210

V211

V212

V213V214

V215V216V217

V218

V219

V220

V221V222

V223

V224

V225

V226V227V228V229V230V231V232V233V234

V235

V236V237

V238V239

V240

V241V242

V243V244

V245

V246V247

V248V249

V250

V251V252

V253

V254

V255V256

V257V258

V259

V260V261

V262

V263

V264V265

V266

V267

V268

V269V270

V271V272

V273

V274

V275

V276V277

V278

V279V280V281V282

V283V284V285V286V287

V288

V289

V290

V291V292V293

V294

V295V296V297

V298V299V300V301V302

V303V304

V305

V306

V307

V308

V309

V310

V311V312V313

V314V315

V316

V317

V318

V319

V320V321

V322

V323

V324V325V326V327V328V329V330

V331V332V333

V334V335V336V337V338V339V340V341V342V343V344V345V346V347V348V349

V350

V351

V352

V353

V354V355V356V357V358V359V360

V361

V362V363

V364V365V366V367V368V369V370V371V372V373V374V375

V376

V377

V378

V379

V380V381

V382V383V384V385

V386

V387

V388

V389

V390V391

V392

V393

V394

V395V396V397

V398V399

V400

V401

V402

V403

V404

V405

V406V407V408V409V410V411

V412V413

V414

V415V416

V417

V418

V419V420

V421V422

V423

V424V425

0

100

200

300

1 2 3 4 5 6 7 8 9 10

eigenvalue

Figure 3.2: Results of the PCA done on the trajectories of the Morris DOE. On the right panel the correlation circleand on the right panel the eigenvalues.

From Figure 3.2, three axis of representation can be chosen because they are covering more than 99% of theinformation. The different Morris indices in the PCA subspace composed of these axes are visible in Figure 3.3.

n_tmu_tetaa_l

a_r

n_intn_t

mu_teta

a_l

a_r

n_intn_tmu_teta

a_l

a_r

n_int0

5

10

15

0 10 20 30 40 50μ*

σ

a

aa

PCA3

PCA2

PCA1

Figure 3.3: Projection on the PCA axis of the Morris indices.

Figure 3.3 is more visual than the one presented on the right panel of Figure 3.1. It allows to conclude onthe non-influent parameter and to keep as random η , µt and ar. As this numerical code is quick to be executed,these results are double checked with a Sobol method. It is also a way to better acknowledge the input parameters,especially if interaction between them and/or non linear effects are quantifiable. The results of the first order indicesand the total effects indices are visible in Figure 3.4. The Sobol indices has been computed for each time step,which allows to state on the evolution of the influence of a parameter over the time frame.

Figure 3.4 confirms the results obtained by the Morris method after the PCA visible in Figure 3.3. The three

71

Cha

pter

3

Chapter 3 – Review of the main calibration methods

0.00

0.25

0.50

0.75

1.00

0 10 20 30 40time step

Firs

t Ord

er o

f Sob

ol in

dice

s

n_t

mu_t

eta

a_l

a_r

n_int

0.00

0.25

0.50

0.75

1.00

0 10 20 30 40time step

Tota

l effe

ct o

f Sob

ol in

dice

s

n_t

mu_t

eta

a_l

a_r

n_int

Figure 3.4: Sobol method completed for each time steps. On the left panel the first order indices and on the rightpanel the total effects indices.

parameters to consider as random are η , µt and ar. With the Sobol method, it is also possible to sort the order ofimportance of the parameters and in this case ar is the most influent all along the time frame. One can also notethat the differences between the first order indices and the total effects are practically equal which means that nonon-linear effects affect the input parameters.

3.1.2 Prior propagation of uncertainty

So far, experts are using the code with some parameter values with the knowledge that these parameters are uncertain(the so called reference values). They can also provide more expertise on the nature of the parameter. For example forar, the nominal value is 0.17 and experts state that the parameter lies within the 95% confidence interval [0.05,0.29].We chose to consider ar as Gaussian with ar ∼N (µ = 0.17,σ2 = 3.6.10−3). The standard deviation is chosenequal to 0.06 because we have considered the upper bound and the lower bound of the given interval as respectivelythe quantiles ar0.975 and ar0.025 . Similarly, η and µt are taken as Gaussian such as η ∼N (µ = 0.143,σ2 = 2.5.10−3)

and µt ∼N (µ =−0.4,σ2 = 10−2). If 100 realizations are drawn from the joint distribution of η , µt and ar, theproduction curve and the prior credibility interval can be simultaneously plotted on a same graph to see howuncertain the predicted power is over a day. Figure 3.5 illustrates on the left the distribution of η , µt and ar and onthe right the production curve obtained for reference values and the prior credibility interval at 90%. On the rightside, experiments collected that same day are also displayed. One can check that the prior credibility interval, buildgiven by the experts, looks coherent regarding the experimental data.

If one is interested in the energy produced rather than the power (the energy in kWh is the power in kW multipliedby a duration), one can easily compute the maximum and the minimum energy for say 100 realizations. The energyfor collected power is Wexp = 3.44kWh, the maximum energy computed Wmax = 3.65kWh and the minimum energyWmin = 2.93kWh. Straightforwardly Wmin < Wexp < Wmax which means that the experts interval seems correctfor that day. With the considered uncertainty on η , µt and ar, the error made is about 20% over only one day.Considering this error over a day, cumulative error over a lifetime plant could be too prejudicial. The aim of thecalibration is to quantify this error and, at the same time, increase the knowledge on the parameter distribution. Theresults of calibration for this application case are detailed in Section 3.3.

72

Cha

pter

3

3.2. Calibration through statistical models

0

2

4

6

0.0 0.1 0.2 0.3

prior

dens

ity

η

dens

ity

0

1

2

3

−0.6 −0.5 −0.4 −0.3 −0.2 −0.1

prior

µt

dens

ity

0

2

4

0.0 0.1 0.2 0.3

prior

ar

Pow

erin

W

0

200

400

600

7 10 13 16 20

ExperimentsSimulated

90% credibility interval a priori

Hours

Figure 3.5: π(η), π(µt) and π(ar) prior densities (represented on the left panel) and induced credibility interval ofthe instantaneous power (right panel).

3.2 Calibration through statistical models

Calibration intends to find the “best fitting” parameters of a computational code, in order to minimize the differencebetween the output and the experiments. It can be used in two cases. In a forecasting context (Craig et al., 2001),calibrated code on data collected on site can be used to compute the behavior of the power plant over the next timeperiod. In a prediction context, data from an experimental stand are used to predict the behavior of a non-existingstand (assuming they have the same features).

A simple way to express calibration is to write down a first, straightforward model. The computational code isset up to entirely replace the physical system. Intuitively, we can assume that ∀x ∈H ,ζ (x) = fc(x,θ) for somewell-chosen θ which leads to the following equation:

M1 : ∀i ∈ J1, . . . ,nK yexpi = fc(xi,θ)+ εi, (3.1)

Model1

with ∀i ∈ J1, . . . ,nK εiiid∼N (0,σ2

err).

The likelihood of such a model is a function of fc. In methods such as Maximum Likelihood Estimation (MLE)or as in Bayesian estimation (making recourse to many MCMC iterations), it becomes intractable to work with atime consuming fc.

For the sake of simplicity we will consider, in what follows, the code as deterministic. It means that for thesame inputs, the output of the code is identical, which is generally the case. Even in a deterministic context, a gapbetween the code and the physical system is often unavoidable. This gap is called code error or discrepancy. Somepapers advocate for adding this discrepancy in the statistical models (Kennedy and O’Hagan, 2001; Higdon et al.,2004; Bayarri et al., 2007; Bachoc et al., 2014). In the following, we present three models which take into account a

73

Cha

pter

3

Chapter 3 – Review of the main calibration methods

time consuming code and/or an additional discrepancy.

3.2.1 Presentation of the models

A time consuming code

Let us consider a time consuming code. As said above, in this particular case, the computational burden becometoo huge to perform calibration. That is why Sacks et al. (1989) introduced an emulation of the not yet computedoutputs from the code by a random function, i.e. a stochastic process. The common choice is oriented toward theGaussian process because the conditional Gaussian process is still a Gaussian process (see Section 2.2 for moredetails). It is, parsimoniously, defined by its mean and covariance functions. The first “simple” and straightforwardmodel was introduced by Cox et al. (2001) which uses this emulation of fc.

M2 : ∀i ∈ J1, . . . ,nK yexpi = F(xi,θ)+ εi (3.2)

F(•,•) ∼ G P(

mS(•,•),cS(•,•),(•,•))

Model 2

where ∀i ∈ J1, . . . ,nK εiiid∼N (0,σ2

err) and the random function F(xi,θ) stands for a Gaussian process (GP) overthe joint domain of xi and θ . Its mean function mS(xi,θ) is generally a linear form of simple functions of xi and θ

and its covariance function cS(x∗i ,θ ∗),(xi,θ)= σ2S rψS(x

∗i ,θ∗),(xi,θ) is such as the function rψS(•,•),(•,•)

is the correlation function with a vector parameter ψS which is the scale and the regularity of the kernel and whereσ2

S represents the variance. The mean mS(xi,θ) can be written:

mS(xi,θ) = mβ S(xi,θ) = E[F(xi,θ)] = βS0 +

M

∑j=1

βS j hS j(xi,θ) = hS(xi,θ)β S (3.3)

where βTS = (βS0 , . . . ,βSM ) is the coefficient vector to be estimated and hS(•,•) = (hS0(•,•), . . . ,hSM (•,•)) the

row vector of regression functions where hS0 = 1. Similarly we define the n× (M+1) matrix HS(X ,θ) such as itsith row is hS(xi,θ). The correlation function can take multiple forms as Gaussian or Matérn for instance (see Santneret al., 2013, for more examples). We will consider, for now and for all theoretical developments, the general form ofcS(•,•),(•,•)= σ2

S rψS(•,•),(•,•) where σ2S is the variance and r is the correlation function with a parameter

vector ψS. The advantage of using a surrogate model, for fc(X ,θ) is to alleviate the computational burden, at thecost of adding an additional source of uncertainty, and of increasing the number of uncertain parameters. Specifichypotheses, for instance a known smoothness of the random field, may help to choose the size of the parametricfamily in which the correlation shape is to be assessed.

When the code is time consuming, a fixed number N of simulations is set up. The ensuing simulated data (wewill call them yc) are usually the image of a design of experiments (DOE) representative of the input space. Someinteresting developments have been made on using the least possible points in the input space with some wiserepartitions (the Latin Hilbert Space sampling is one example, some good insights are available in Pronzato andMüller (2012)).

Let us call D a DOE, a set of N points sampled in the input space defined as the product of H and Q. We canwrite D = (xD

1 ,τD1 ), . . .(x

DN ,τ

DN) where ∀i ∈ J1, . . . ,NK (xD

i ,τDi ) are chosen in H ×Q. The establishment of the

74

Cha

pter

3

3.2. Calibration through statistical models

DOE will lead to simulated data which are defined as yc = fc(D). The error made by the surrogate strongly dependson the numerical design of experiments used to fit the emulator. Adaptive numerical designs introduced in Damblinet al. (2018) is a way to enhance the emulator when the goal is to calibrate the code.

With a code error

Considering the computational code as a perfect representation of the physical system may be a too strong hypothesisand it is legitimate to wonder whether the code might differ from the phenomenon. This error (called discrepancyand introduced above) can be defined as:

δ (xi) = ζ (xi)− fc(xi,θ).

In all the works cited above, this unknown discrepancy is modeled as a realization of a Gaussian process, thistime yielding a random function over the domain of X variables only. For the sake of simplicity we will denote byms, cS (c

σ2S ,ψS

) and rS (rψS ) the mean, covariance (with σ2S as the variance) and correlation function relative to the

surrogate and by mδ , cδ (cσ2

δ,ψδ

) and rδ (rψδ) the same functions relative to the discrepancy (respectively σ2

δfor the

variance in the covariance function). Note that mδ and cδ are functions of x only and not θ . The aim of adding thediscrepancy lies in the fact that correlation is sometimes visible in the residuals and/or that no value of θ makes thecomputer close to experiments. However, the discrepancy could lead to an identifiability issue. For example, it couldeasily exist two couples (θ ,δ (xi)) and (θ ∗,δ ∗(xi)) that verify the two equalities: δ (xi) = ζ (xi)− fc(xi,θ) andδ ∗(xi) = ζ (xi)− fc(xi,θ

∗). Some papers (Higdon et al., 2004; Bachoc et al., 2014; Bayarri et al., 2007) advocate toset the mean of the discrepancy to 0 to solve this identifiability issue. The contribution of the discrepancy is widelydiscussed in literature and make the object of comparative studies in validation (Damblin, 2015).

When the code is not time consuming, the real code fc is used:

M3 : ∀i ∈ J1, . . . ,nK yexpi = fc(xi,θ)+δ (xi)+ εi (3.4)

δ (•) ∼ G P(

mδ (•),cδ (•,•))

Model 3

where ∀i ∈ J1, . . . ,nK εiiid∼N (0,σ2

err), and δ (•) stands for a Gaussian process which mimics the discrepancy andwill only depends on the input variables x. We can write δ (•)∼ G P(mδ (•),cδ (•,•)) with ∀x, mδ (x) = hδ (x)β δ

(where hδ is a row vector and β δ is a column vector if we choose a parametric representation of the mean) andcδ the covariance function of the discrepancy. We also denote Hδ (X) the n row matrix, the ith row of which is hδ (xi).

When the code is time consuming, the systematic use of fc is not computationally acceptable. Then, as forModel M2, the code is replaced with a Gaussian process. This leads to the more generic model introduced inKennedy and O’Hagan (2001).

M4 : ∀i ∈ J1, . . . ,nK yexpi = F(xi,θ)+δ (xi)+ εi (3.5)

Model 4

75

Cha

pter

3

Chapter 3 – Review of the main calibration methods

where ∀i ∈ J1, . . . ,nK εiiid∼N (0,σ2

err), F(xi,θ) and δ (xi) are two Gaussian processes defined as before. In theirmodel, Kennedy and O’Hagan (2001) also used a scale parameter ρ in front of F . This parameter is usually set to 1in many works in order to achieve the best estimate on θ . Thus, we omit this parameter in the model definition.

A quantification of the bias form is the aim of both models. If we are interested in improving the computationalcode or its surrogate, it is usually fair to set the mean of the discrepancy to zero and find the best tuning parametervector which compensates a potential bias (Higdon et al., 2004; Bachoc et al., 2014).

yexp

yc(X ,θ)

σ 2err

θ

β δ , σ 2δ

, ψδβ S , σS , ψS

δ (x)

Figure 3.6: Directed Acyclic Graph (DAG) representation of the different models.

Figure 3.6 is a summary of all the models introduced above. The directed acyclic graph (DAG) allows usto compare the structures of all the previously introduced models. Specifically: if one considers only the greynodes, the obtained DAG corresponds to Model M1. Adding the green node, the resulting DAG represents M2.Considering the grey and red nodes, yields a DAG for model M3 and the whole DAG represents the general modelM4. Note that two categories of parameters are considered. The tuning parameters are only related to the code andother parameters (also called nuisance parameter) concern the measurement error, the surrogate or the discrepancyintroduced in the models. In calibration, we only focus on the value of θ but the other parameters introduced needto be estimated as well. We will dig into these estimation issues in Section 3.2.3.

All these models introduce new parameters and need to be estimated as well as tuning parameters. Estimationneeds to dive into technical aspects such as writing the likelihood for each model. The following section providesall the elements required to go one step further and carry out estimation.

3.2.2 Likelihood

For estimating parameters (whatever framework used, Bayesian or Maximization Likelihood Estimation (MLE)),expressing the likelihood comes as the first requirement. Two major categories stand out. When the code is nottime consuming, the main issue in code calibration (i.e. the computational time burden) is avoided. When the codeis time consuming, new parameters have to be taken into account and to be estimated. That is why, in the cases 1and 3 data are only field data yexp and in the cases 2 and 4, numerical data (outputs of the code) are added to formthe whole data vector yT = (yT

exp,yTc ). In what follows, we will denote by θ

∗ the true parameter vector. Note thatit is well-defined only in Models M1 and M2, as the value of θ which satisfies : ζ (x) = fc(x,θ ∗) for all possiblex, (assuming such a θ exists and is unique). On the other hand, the models M3 and M4 are both defined by therelation ζ (x) = fc(x,θ)+δ (x), which holds for infinitely many couples (θ ,δ (•)), as discussed earlier. Kennedyand O’Hagan (2001) avoid this issue by defining θ

∗ as a “best-fitting” value, but it is unclear what this means

76

Cha

pter

3

3.2. Calibration through statistical models

exactly (see the discussion section of their paper for further details).

In order to simplify the notation, we will use for the rest of the paper Φ = σ2S ,σ

2δ,ψS,ψδ and ΦS = σ2

S ,ψSand Φδ = σ2

δ,ψδ, where σ2

S and σ2δ

are the variances of the two Gaussian processes respectively relative to thesurrogate and the discrepancy. The two parameter vectors ψS and ψδ are relative to the correlation functions. Letus call β

T = (β TS ,β

Tδ ) the vector of collected coefficient vectors.

Both cases of time consuming or not time consuming code will be dealt with. The likelihood equations willbe written for the generic forms of M3 and M4. The likelihoods for the simpler models M1 and M2 will be thenderived since M1 ⊂M3 and M2 ⊂M4.

A fast code

The generic model which deals with calibration with a code quick to run is detailed in Equation (3.4). Experimentaldata are the only one needed because simulation data are free but will not bring additional information for theparameters of Model M3. Experimental data follow a Gaussian distribution, the expectation of which is:

E[yexp|θ ,β δ ;X ] = mβ δexp(X ,θ) = mexp(X ,θ) = fc(X ,θ)+Hδ (X)β δ .

Then, the expression of the variance is given by:

Var[yexp|Φδ ;X ] =V Φδ ,σ2err

exp (X) =V exp(X) = Σδ (X)+σ2errIn

with ∀(i, j) ∈ J1, . . . ,nK2 : (Σδ (X))i, j = (ΣΦδ

δ(X))i, j = cδ (xi,x j). The likelihood in this particular case can be

written as

L F(θ ,β δ ,Φδ ;yexp,X) =1

(2π)n/2|V exp(X)|1/2 exp

− 1

2

(yexp−mexp(X ,θ)

)TV exp(X)−1

(yexp−mexp(X ,θ)

).

(3.6)

This likelihood is relative to Model M3 (Equation (3.4)). For the specific case, where no discrepancy isconsidered (corresponding to M1 Equation (3.1)) the likelihood can be written in a similar way but with mexp(X ,θ)=

fc(X ,θ) and V exp(X) = σ2errIn. Note that the covariance matrix depends only on σerr

2. It implies that if we seek toestimate the posterior density on θ (in a Bayesian framework), this covariance term is superfluous.

Then the likelihood can be rewritten in an simpler way:

L F(θ ,σ2err;yexp,X) =

1(2π)n/2σn

errexp

− 1

2σ2err||yexp− fc(X ,θ)||22

. (3.7)

The models using the code with or without the discrepancy do look quite similar. For theoretical development,it might be easier to work with the one without discrepancy. From an experimental point of view, it could beinteresting to study the role of the code error.

77

Cha

pter

3

Chapter 3 – Review of the main calibration methods

A time consuming code

When a code is time consuming and replaced by a surrogate, additional parameters are to be estimated. As introducedabove, a DOE is set up and intends to be a representative sample of the input space (variable and parameter inputspace). Simulated data from this DOE (called yc) will constitute additional data for the estimation of the nuisanceparameters. Depending on how we consider that two sources of data are linked, multiple likelihoods can be set up.For the theoretical development, we will consider the general model M4 and we will detail the particular case M2

hereafter.

The first likelihood useful in estimation is the full likelihood. This one concerns the distribution of allcollected data (yT = (yT

exp,yTc )). That means, we are interested in estimating the parameters of the distribu-

tion π(y|θ ,β ,Φ,σ2err;X ,D) which is Gaussian. The expectations can be written from both expectancies of

π(yexp|θ ,β ,Φ,σ2err;X) and π(yc|θ ,β S,ΦS;X).E[yc|β S;D] = mβ S

c (D) = mc(D) = HS(D)β S

E[yexp|θ ,β ;X ] = mβexp(X ,θ) = mexp(X ,θ) = HS(X ,θ)β S +Hδ (X)β δ

(3.8)

This can be summed up for two component vectors yT = (yTexp,y

Tc ):

E[y|θ ,β ;X ,D] = mβy ((X ,θ),D) = my((X ,θ),D) = H((X ,θ),D)β

=

(HS(X ,θ) Hδ (X)

HS(D) 0

)β .

(3.9)

The variance matrix now includes the covariance functions of the discrepancy and the surrogate.

Var[y|θ ,Φ,σ2err;X ,D] =V Φ,σ2

err((X ,θ),D) =V ((X ,θ),D)

=

(Σexp,exp(X ,θ)+Σδ (X)+σ2

errIn Σexp,c((X ,θ),D)

Σexp,c((X ,θ),D)T Σc,c(D)

)(3.10)

where

• ∀(i, j) ∈ J1, . . . ,nK2 : (Σexp,exp(X ,θ))i, j = cS(xi,θ),(x j,θ),

• ∀(i, j) ∈ J1, . . . ,nK× J1, . . . ,NK : (Σexp,c((X ,θ),D))i, j = cS(xi,θ i),(xDj ,τ

Dj ),

• ∀(i, j) ∈ J1, . . . ,nK2 : (Σδ (X))i, j = cδ(xi,x j),

• ∀(i, j) ∈ J1, . . . ,NK2 : (Σc,c(D))i, j = cS(xDi ,τ

Di ),(x

Dj ,τ

Dj ).

As a reminder D is the DOE set up to build the surrogate and is defined as D = (xD1 ,τ

D1 ), . . .(x

DN ,τ

DN). The

general expression of the full likelihood can then be expressed:

L F(θ ,β ,Φ,σ2err;y,X ,D) =

1(2π)(n+N)/2|V ((X ,θ),D)|1/2 exp

− 1

2

(y−my((X ,θ),D)

)T

V ((X ,θ),D)−1(

y−my((X ,θ),D))

.

(3.11)

78

Cha

pter

3

3.2. Calibration through statistical models

The particular cases of Bayarri et al. (2007) and Higdon et al. (2004) a zero mean for the discrepancy is

considered. Under this condition, we have my((X ,θ),D) =

(HS(X ,θ)

HS(D)

)β S and the other terms remain the same.

For the model M2 where a surrogate is used without any discrepancy (Cox et al., 2001), the expectation becomes:

E[y|θ ,β S;X ,D] = my((X ,θ),D) = H((X ,θ),D)β S =

(HS(X ,θ)

HS(D)

)β S (3.12)

and the covariance:

Var[y|θ ,Φ,σ2err;X ,D] =V ((X ,θ),D) =

(Σexp,exp(X ,θ)+σ2

errIn Σexp,c((X ,θ),D)

Σexp,c((X ,θ),D)T Σc,c(D)

)(3.13)

where covariances matrices are the same as defined before.

The estimation can be separate into different steps where the partial likelihood (Equation (3.14)) could be useful.This one only concerns simulated data and the corresponding surrogate. The partial likelihoods of the model M2

and M4 are then the same. That means we are only interesting in estimating the distribution π(β S,ΦS|yc) whereΦS = σ2

S ,ψS. The expectation can be obtained by considering only the mean function of the surrogate (Equation(3.8)) and the variance is straightforwardly linked to the variance of the surrogate.

Var[yc|ΦS;D] =V ΦSc (D) =V c(D) = Σc,c(D),

where ∀(i, j) ∈ [1, . . . ,N]2 : (Σc,c(D))i, j = cS(xDi ,θ

Di ),(x

Dj ,θ

Dj ). Let us recall that Equation (3.8) established

that mc(D) = HS(D)β S. It implies that the partial likelihood relative to M4 and M2 is:

L M(β S,ΦS;yc,D) =1

(2π)N/2|V c(D)|1/2 exp

− 1

2

(yc−mc(D)

)TV c(D)−1

(yc−mc(D)

). (3.14)

As the other model introduced by Higdon et al. (2004) and Bayarri et al. (2007) only deals with changes on thediscrepancy, the partial likelihood remains identical as the one for the model in Bayarri et al. (2007).

From what has been introduced before, one can write the conditional distribution π(yexp|yc) (see Section 2.2 formore details) from the joint distribution π(yexp,yc) and for M4:(

yexp

yc

)∼N

((mexp(X ,θ)

mc(D)

),

(Σexp,exp(X ,θ)+Σδ +σ2

errIn Σexp,c((X ,θ),D)

Σexp,c((X ,θ),D)T Σc,c(D)

))where mc and mexp are defined Equation (3.8) and covariance matrices defined above before Equation (3.11). Then,

yexp|yc ∼N (µexp|c((X ,θ),D),Σexp|c((X ,θ),D))

with:

µexp|c((X ,θ),D) = mexp(X ,θ)+Σexp,c((X ,θ),D)Σc,c(D)−1(yc−mc(D)), (3.15)

Σexp|c((X ,θ),D) = Σexp,exp(X ,θ)+Σδ +σ2errIn−Σexp,c((X ,θ),D)Σc,c(D)−1

Σexp,c((X ,θ),D)T . (3.16)

For M2, the variance term Σexp,exp(X ,θ)+Σδ +σ2errIn in Equation (3.2.2) becomes Σexp,exp(X ,θ)+σ2

errIn

because there is no discrepancy. It means that Σexp|c((X ,θ),D) for M2 can be rewritten as:

79

Cha

pter

3

Chapter 3 – Review of the main calibration methods

Σexp|c((X ,θ),D) = Σexp,exp(X ,θ)+σ2errIn−Σexp,c((X ,θ),D)Σc,c(D)−1

Σexp,c((X ,θ),D)T . (3.17)

The conditional likelihood can then be expressed as:

L C(θ ,β δ ,Φδ ;β S,ΦS,yexp|yc,X ,D) ∝|Σexp|c((X ,θ),D)|−1/2

exp− 1

2(yexp−µexp|c((X ,θ),D))T

Σexp|c((X ,θ),D)−1

(yexp−µexp|c((X ,θ),D)).

(3.18)

Usually in a Bayesian framework, β is distributed according to a Jeffreys prior. In this case, π(β ) = π(β S,β δ )∝

1 and we can integrate out β from the full likelihood expressed by Equation (3.11).

3.2.3 Estimation

Maximum likelihood estimator

In this section, we comment remarkable insights developed in Cox et al. (2001). For estimating the parameters θ ,β and Φ, a first approach (for M1 and M2) would be to maximize the full likelihood introduced in the previoussection. This method is called Full Maximization of Likelihood Estimator. The major drawback of this method is todeal with a high number of parameters and in certain cases this leads to a very heavy computational operation.

A second method, introduced in Cox et al. (2001) only for M2 to overcome this issue, is called the SeparatedMaximization of Likelihood Estimation (SMLE). The estimation is made in two steps. The first step is to maximizethe partial likelihood (Equation (3.14)) to get estimators of the parameters of the Gaussian Process. Then theseestimators (Φ and β ) are plugged into µexp|c((X ,θ),D) and Σexp|c((X ,θ),D) which are the mean and the varianceof the conditional distribution. A likelihood is set up from those quantities and maximized to get θ . The SMLEmethod can also be seen as an approximation of the generalized non linear least squares.

These methods are applied in Cox et al. (2001) for M2. For models M3 and M4, (Wong et al., 2017) havedeveloped a new approach which deals with the identifiability problem when the discrepancy is added in thisframework. Then, the estimation part is performed in two times. The first step consists in estimating θ in

θ = argminθ∈Q

Mn(θ) with Mn(θ) =1n

n

∑i=1yexpi −F(xi,θ)2. (3.19)

Then the estimation of the discrepancy is done by applying a nonparametric regression to the data xi,yexpi −F(xi, θ)i=1,...,n. Any nonparametric regressions are subject to offer working alternatives with this method whichshows an interesting flexibility of the approach.

Bayesian estimation

Under the Bayesian framework, there are several ad hoc short cuts to find estimators without evaluating and samplingfrom the entire joint posterior distribution of the unknowns. The idea behind is to consider a prior distribution oneach parameters which we will separate into two different categories. The first category represents the nuisanceparameters which are typically σ2

S ,σ2δ,ψS,ψδ, σ2

err and β . Those parameters are added because of the modeling.The second category regroups the other parameters to estimate such as θ . We will work on the two generic modelsM3 and M4 with the corresponding sets of parameters to estimate.

80

Cha

pter

3

3.2. Calibration through statistical models

The difference between both models lies in the fact that for M3 the code can be used as such and for M4 asurrogate is used to avoid running the code. In the further developments, the parameters to estimate will be rela-tive to M4 and for going back to M3 it will be just necessary to omit the nuisance parameters relative to the surrogate.

As introduced before, it is common to take a weakly informative prior on β such as π(β S,β δ ) ∝ 1. It is alsoreasonable to suppose that prior information about θ is independent from the prior information about Φ and β . Theprior density can then be expressed as

π(θ ,β ,Φ) = π(θ)×1×π(Φ). (3.20)

Once the full likelihood integrated L F on the prior distribution of β , the posterior distribution can be expressed(all details are pursued in Kennedy and O’Hagan (2001)).

For a full Bayesian analysis, integrating Φ out is needed to finally get π(θ |y). However this integration canbe quite difficult because of the high number of nuisance parameters. It would also demand a full and carefulconsideration of the prior π(Φ). Two methods are mainly used for estimating θ and Φ. In Higdon et al. (2004), thechoice made is to jointly estimate all parameters from Equation (3.11). The strength of this method is to stand withinthe pure Bayesian tracks: recourse is made to all collected data (the simulated with the DOE and experimental data)to estimate all parameters and nuisance parameters at the same time.

However, Kennedy and O’Hagan (2001) and Bayarri et al. (2007) have chosen an estimation in separate steps.This method called modularization by Liu et al. (2009) makes inference simpler but gives only a convenient approx-imation of the exact posterior (that separates the components of parameter Φ for each Gaussian Process involved).The first step consists in maximizing the likelihood L M(β S,ΦS|yc;D) (Equation (3.14)) to get the maximumlikelihood estimates (MLE) β S and ΦS of β S and ΦS. In the second stage, these estimators are plugged into theconditional likelihood L C(θ ,β δ ,Φδ ;β S,ΦS,yexp|yc,X ,D) (Equation 3.18) from which the posterior density issampled with MCMC methods. Note that this last step is the only one that differs from SMLE method from Coxet al. (2001).

Kennedy and O’Hagan (2001) arrange to first estimate the nuisance parameters of the surrogate of the codewith the partial likelihood (Equation (3.8)), then they make an integration on the prior density of θ to estimatethe nuisance parameters of the discrepancy and the variance of the measurement error. They finally estimatethe parameter vector θ by sampling in the posterior density with a MCMC. Bayarri et al. (2007) are doing thesame thing in estimating first the nuisance parameters of the surrogate with the partial likelihood. However, theyuse “virtual” residuals (defined as yexp− fc(X ,θ prior) where θ prior is a prior value on θ ) to compute a maximumlikelihood estimates for estimating the nuisance parameters relative to the discrepancy and to the measurement error.Then the posterior densities of θ , σ2

δand σ2

err are sampled with a Gibbs algorithm based on conditional completedistribution. Practically, this estimation is very time consuming. Indeed, the Gibbs sampler will compute at eachiteration the full likelihood which contains a (n+N)× (n+N) matrix to invert.

We made the choice to use the modularization method for estimating first the nuisance parameters of thesurrogate with the partial likelihood. Then, to avoid the computational time burden of the method introduced byBayarri et al. (2007) and the integration on the prior density of θ of Kennedy and O’Hagan (2001), we chose tosample, with the Algorithm 7, in the posterior densities of the parameter vector θ , of the nuisance parameters of thediscrepancy and of the variance of the measurement error. The first part of the algorithm is a Metropolis Hastingswithin Gibbs that is run for a limited number of times. This algorithm allows to estimate the covariance structure onthe samples generated and to use it in a Metropolis algorithm. These way to proceed help the Metropolis algorithm

81

Cha

pter

3

Chapter 3 – Review of the main calibration methods

to better perform.

3.3 Application to the prediction of power from a photovoltaic (PV) plant

In this section, the PV plant code is a toy example to try out all the models. First, we test the model M1 (Equation(3.1)), in which only the initial code and the measurement error are considered. The code is supposed, in this case,quick to run although, in most industrial case studies, numerical codes are time consuming. This is the first issue offeasibility met by engineers. In a second part, we apply Model M2 on our example to mimic the case when the codecannot be run at will. This model introduces a surrogate of the code and its characteristics will be detailed below.M3 is motivated by the gap between the reality and the code observed, most of the time, by engineers. In this case,we will add to M1 an error term for the discrepancy between the code and the phenomenon. This code error will berepresented by a Gaussian process also detailed below. The final case is when both issues are occurring. That willlead to the consideration of M4 for the application case.

The Bayesian framework starts with the elicitation of priors densities (that will not be discussed here (Albertet al., 2012)). According to the experts we choose:

• η ∼N (0.143,2.5.10−3),

• µt ∼N (−0.4,10−2),

• ar ∼N (0.17,3.6.10−3),

• σ2err ∼ Γ(2,169),

• σ2δ∼ Γ(3,1),

• ψδ ∼U (0,1).

This application section is developed in two subsections. The first subsection details the practical implementationprocedure of the inference for each model. In the second subsection, we discuss all the results obtained for themodels that we tried out.

3.3.1 Inference

The sensitivity analysis run Section 3.1.1 on the parameter vector θ allowed to conclude that only η , µt and ar

are relevant considering the power output. The inference only concerns these three parameters and the additionalnuisance parameters depending on the model. For the sake of simplicity, data, recorded every 10s, is averagedper hour and only data corresponding to a strictly positive power are kept. The Bayesian framework is chosen forthe following study. It is motivated by the availability of strongly informative priors, elicited from experts, on theparameters we want to estimate. To perform the inference, a Markov Chain Monte Carlo algorithm is used (Robert,1996) (especially the algorithm 7 in Section 2.5). Here, the Metropolis within Gibbs is launched for 3000 iterations.The values of this first sampling phase are kept to improve the covariance structure of the auxiliary distribution usedto make proposals by the algorithm. This will lead to better mixing properties for the following Metropolis Hastings(10000 iterations including a burn in phase of 3000).

82

Cha

pter

3

3.3. Application to the prediction of power from a photovoltaic (PV) plant

Two months of data are studied. The PV production over August and September 2014 are available. We usedthose two months of data averaged per hour which makes 1019 points. For the cross validation, three days ofinstantaneous power (51 points) are taken off the learning set and used to evaluate the predictive power of the modelconsidering the rest of the available data.

The Gaussian process

As said in Section 1.4.3, 6 input variables are needed to run the code. These are t the UTC time, L the latitude, l

the longitude, Ig the global irradiation, Id the diffuse irradiation and Te the ambient temperature. The test stand isprecisely located. Therefore, the latitude and the longitude are not to be considered since they do not change byrecords.

The major issue in emulating the behavior of the code is to deal with correlated variables. Actually, the globalirradiation, diffuse irradiation and ambient temperature depend on the time which defines the sun position. If aspace filling DOE, is taken into [0,1]4 and then unnormalized between the upper and lower bounds of the 4 inputvariables and the parameter, some configurations tested would not make any sense. We could obtain for example, atime which indicates the morning and a global irradiation value which corresponds at noon. To solve this problem,we choose to run a PCA (Principle Component Analysis) on the matrix containing all the xi’s (over the durationused for the calibration). The aim is to access an uncorrelated space in which we could sample a DOE which wouldkeep a physical sense and then go back to the original space with the transformation matrix.

The main steps of this method are:

1. the PCA is performed on the matrix X where the ith line contains xi =

tiIgi

Idi

Tei

where xi ∈ R4,

2. the maximin LHS is sampled in the uncorrelated space given by the PCA,

3. the transformation matrix T allows us to go back in the original space,

4. the code is run for those points and gives the computed data yc.

The Gaussian processes emulated from this method reveal to work much better. We also could have developedthe method with an adaptive numerical design (see (Damblin et al., 2018)) to the correlated input variables.

To position the application case to a time consuming context, a limited number of experiments is allowed for theDOE which establishes the surrogate. We will limit the number of code calls to 50 to investigate the time consumingsituation and compare it to other situations more favorables. This number of experiments is taken when computercodes are extremely time consuming. To compare the quality of M2 with a DOE of 50 points, we chose to compareit with a model M2 established with a DOE of 100 points.

Table 3.1: Results in cross validation for M2 for 2 different DOE sizes

50 points 100 pointscoverage rate at 90% (in %) 32 65

RMSE of the instantaneous power (W ) 21.61 19.7

83

Cha

pter

3

Chapter 3 – Review of the main calibration methods

The degradation in prediction with the decrease of the number of points in the DOE (Table 3.1) is in line withour desire to place ourselves in the most unfavorable case possible.

The first model M1

Model M1 described by Equation (3.1) only deals with the measurement error. The code used in its simplest formonly makes recourse to the parameters η , µt and ar. In this case the parameters to infer on are η , µt , ar and σ2

err

(where εiiid∼N (0,σ2

err)).

The second model M2

As defined in Section 3.2, when the code is time consuming, the solution is to mimic it with a Gaussian process(GP). For the GP emulator, we made the choice to consider the mean function mS(•,•) as a linear combination oflinear functions. That means HS is a matrix of linear functions. The correlation function rS (cS = σ2

S rS) chosen isdefined by the following equation that corresponds to a Gaussian kernel (Equation (2.50)):

rS(x,θ),(x∗,θ ∗)= exp− 1

2||(x,θ)− (x∗,θ ∗)||22

ψ2S

(3.21)

where || • ||2 stands for the Euclidean norm.In this case, five parameters have to be estimated: η , µt , ar, σ2

err, σ2S and ψS.

The third model M3

The third model introduces another GP for the discrepancy. We choose a different covariance kernel which isa Matérn 5/2 (Equation (3.22) from Equation (2.51) with ν = 5/2). Note that compared to Equation (3.4), thediscrepancy mean has been set to 0 (i.e. mδ (.) = 0). These choices are motivated by the fact that the purpose ofcalibration is to estimate the "best-fitting" vector parameter θ . We do not want any compensation into any additionalbias. This decision is consistent with Bachoc et al. (2014) where the same hypothesis has been made.

rδ (x,x∗) =

(1+

√5||x− x∗||2

ψδ

+5||x− x∗||22

3ψ2δ

)exp−√

5||x− x∗||2ψδ

. (3.22)

In this case, there are also five parameters to estimate that are η , µt , ar σ2δ

, ψδ and σ2err.

The fourth model M4

This part focuses on a time consuming code with discrepancy. This model uses the same surrogate and discrepancydefined above. The two correlation function for the surrogate and the discrepancy were chosen with differentregularity in order to distinguish the two Gaussian processes. It seems relevant to assume that the discrepancy issmoother than the code. That is why a Matérn correlation function is chosen for the code and a Gaussian correlationfunction for the discrepancy. In this case seven parameters need to be estimated which are η , µt , ar, σ2

err, σ2S , ψS,

σ2δ

and ψδ .

Estimation of the nuisance parameters

In our Bayesian framework, the choice of an estimation by modularization is made. It concerns only the second andthe fourth model. As it is the case in Kennedy and O’Hagan (2001), a maximization of the probability π(ΦS|yc)

is done to estimate β S, σ2S and ψS where yc are the outputs of the code for all the points given by the DOE. This

maximization is included in the R function km from the package DiceKriging (Roustant et al., 2012).

84

Cha

pter

3

3.3. Application to the prediction of power from a photovoltaic (PV) plant

3.3.2 Results

M1 M2 M3 M4

dens

ity

0

200

400

600

0.0 0.1 0.2 0.3

posteriorprior

0

50

100

150

200

0.0 0.1 0.2 0.3

posteriorprior

0

300

600

900

1200

0.0 0.1 0.2 0.3

posteriorprior

0

50

100

150

200

250

0.0 0.1 0.2 0.3

posteriorprior

η

dens

ity

0

10

20

30

40

50

−0.6 −0.4 −0.2

posteriorprior

0

5

10

15

20

−0.6 −0.4 −0.2 0.0

posteriorprior

0

50

100

150

200

−0.8 −0.6 −0.4 −0.2

posteriorprior

0

5

10

15

20

25

−0.6 −0.4 −0.2

posteriorprior

µt

dens

ity

0

100

200

300

0.0 0.1 0.2 0.3

posteriorprior

0

20

40

60

0.0 0.1 0.2 0.3

posteriorprior

0

50

100

150

0.0 0.1 0.2 0.3

posteriorprior

0

20

40

60

0.0 0.1 0.2 0.3

posteriorprior

ar

dens

ity

0.00

0.02

0.04

0.06

0 500 1000

posteriorprior

0.00

0.01

0.02

0.03

0 200 400 600 800

posteriorprior

0.00

0.05

0.10

0.15

0 250 500 750

posteriorprior

0.00

0.02

0.04

0.06

0.08

0 500 1000

posteriorprior

σ2err

Figure 3.7: Prior (in blue) and posterior (in red) densities of η , µt , ar and σ2err for each model. On the two first

column the two first models (without and with surrogate) which have only these four parameters to estimate. Thetwo other columns represent the third and the fourth models which have two more parameters to estimate (seeFigure 3.9).

Figure 3.7 compares the results obtained (with the help of the R package CaliCo (Carmassi, 2018)) for η , µt , ar

and σ2err for each model. In each case, the MCMC chains converge. For the first model, a strong disagreement has

appeared between the prior and posterior densities for σ2err. The Maximum A Posteriori (MAP) of σ2

err’s density,for M1, is 1283 W 2. That makes a standard deviation of 36W which is too high and has no physical trustworthiness.For M2, M3 and M4 the MAP estimations of σ2

err’s densities seem to be coherent with physics. The addition of thediscrepancy between M1 and M3 then between M2 and M4 depicts a correlation (an error structure) in ye. When,no code error is applied, the variance from this covariance matrix is added to the variance of the measurement error.

85

Cha

pter

3

Chapter 3 – Review of the main calibration methods

From M1 to M2, a surrogate emulates the numerical code. Figure 3.7 illustrates a bigger variance a posteriori

for the parameters densities of M2 than M1. Replacing the code by a surrogate had brought more uncertainty.Moreover, the densities for M2 appear to be out of step with M1. Calibration behaves as if a bias has appeared withthe surrogate. This is worth to note that using a surrogate not only increase the variance of the posterior distributionof the parameter θ but may also change the mode. This is also observed when moving from M3 to M4. Addingthe discrepancy (from M1 to M3 and from M2 to M4) has almost always reduced the variances of the posteriordistributions.

We also depict correlation between the parameters. As a matter of fact, a strong positive and linear correlationlinks every parameters (η , µt and ar) with each other as illustrated in Figure 3.8. A strong correlation appearsbetween µt and ar. A lower, but still meaningful, correlation is also visible between η and µr, and ar and η .

µt

−0.510

−0.505

−0.500

−0.495

0.10 0.11 0.12 0.13

a r

0.210

0.215

0.220

0.225

0.230

−0.510 −0.505 −0.500 −0.495

η

0.10

0.11

0.12

0.13

0.210 0.215 0.220 0.225 0.230

η µt ar

Figure 3.8: Correlation representation between the parameters.

M3 M4

dens

ity

0.00

0.05

0.10

0.15

0 200 400 600 800

posteriorprior

0.0

0.1

0.2

0.3

0 200 400 600

posteriorprior

σ2δ

dens

ity

0

20

40

60

0.00 0.25 0.50 0.75 1.00

posteriorprior

0

20

40

60

0.00 0.25 0.50 0.75 1.00

posteriorprior

ψδ

Figure 3.9: Prior (in blue) and posterior (in red) densities of σ2δ

and ψδ for M3 and M4.

Figure 3.9 illustrates the estimation of the parameters from the discrepancy term. As expected, learning fromdata has improved our prior belief by decreasing the prior uncertainty of the parameters. It shows that in both cases(with and without surrogate) that the convergence seems to be reached at some point.

86

Cha

pter

3

3.3. Application to the prediction of power from a photovoltaic (PV) plant

3.3.3 Comparison

To compare the prediction ability of the four models, a cross validation (CV) is performed. Three days of data(chosen randomly) are taken off the calibration dataset for each of the 100 repetitions of the CV. The power densities,generated from the MCMC samples, allow us to compute, for each model, the 90% predictive credibility intervals.The coverage rate at 90% represents the quantity of validation experiments contained in these credibility intervals.The Root Mean Square Error (RMSE) is also computed for the instantaneous power. The results are displayed Table3.2.

Table 3.2: Comparison of the RMSEs and coverage rates in prediction of 100 test-sets on three randomly selecteddays.

M1 M2 M3 M4

coverage rate at 90% (in %) 91 32 87 23RMSE of the instantaneous power (W ) 12.69 21.61 5.91 18.7

The coverage rates for M1 and M3 corresponds to the chosen credibility level. However for M2 and M4 thecoverage rates are below this level. This was expected since the coverage rates for the code emulation displayedin Table 3.1 were below the fixed credibility level especially when the DOE had only 50 points. We recall thatthese coverage rates only account for the surrogate error and not for the uncertainty on θ . We also notice that thepredictive power increases when the discrepancy is added.

Overall, the model M3 has better results than the others. This conclusion can be explained twofold. First, thecode achieves a better prediction than the surrogate. Second, a correlation structure remains in the error. Adding thediscrepancy in the model allows to catch up the real results. The fact that the models, encompassing a surrogate,produces worse results is expected. The Gaussian process used for the surrogate had trouble to fill every variation ofpower. To compensate this lack of information, the posterior credibility interval becomes wide and less informative.

Figure 3.7 illustrates that the use of a Gaussian process with a low number of points creates a bad estima-tion of the parameters. Indeed the modes a posteriori found with M2 or M4 are shifted compared to the a

priori modes. A particular attention has to be given to the Gaussian process quality regarding further calibra-tion. A better DOE could have been proposed as for example the sequential design introduced in Damblin et al.(2018). If we start from an initial maximin LHS DOE with a higher number of points, say 150 points, and ifwe use the sequential design to add say 30 points, calibration is modified and the results are illustrated in Figure 3.10.

87

Cha

pter

3

Chapter 3 – Review of the main calibration methods

M ′2 M ′

4

dens

ity0

100

200

300

0.0 0.1 0.2 0.3

posterior

prior

0

100

200

0.0 0.1 0.2 0.3

posterior

prior

η

dens

ity

0

10

20

30

40

−0.75 −0.50 −0.25 0.00

posterior

prior

0

20

40

60

80

−0.75 −0.50 −0.25 0.00

posterior

prior

µt

dens

ity

0

30

60

90

0.0 0.1 0.2 0.3

posterior

prior

0

50

100

150

0.0 0.1 0.2 0.3

posterior

prior

ar

dens

ity

0.00

0.02

0.04

0 500 1000 1500 2000

posterior

prior

0.0

0.1

0.2

0 500 1000 1500 2000

posterior

prior

σ2err

Figure 3.10: Calibration results for M2 and M4 that are using Gaussian processes build on a DOE extended by thesequential design.

With the new DOE, the estimation of the nuisance parameters is better than the previous one and allow to beconsistent with the prior densities and the physical meaning. The question highlighted here is to wonder how manypoints to add in the design to have a proper Gaussian process which leads to a coherent calibration.

3.4 Conclusion and discussion

This chapter focuses on code calibration which can be a very interesting way to deal with uncertainties in numericalexperiments. The code used in this chapter, to allow comparisons with time consuming codes, is a quick codepredicting power from small PV plant. This work can be extended to bigger computational codes in application atlarger PV plants. As we are working with a physical code, it is important in this case to keep in mind the reality ofthe physical boundaries. This aspect had allowed us to confirm the presence of the discrepancy.

88

Cha

pter

3

3.4. Conclusion and discussion

In a case where input variables are correlated, additional issues of DOE appear when the surrogate is fitted. Tocope with these issues we made recourse to a PCA. The design of numerical experiments could have been enhancedby using adaptive designs proposed by Damblin et al. (2018).

The hypotheses made for the application case can also be discussed. For example setting ρ and mδ (.) to 1 and 0is a preliminary decision which goes along with calibration. We do not want to quantify the bias because the aim ofcalibration is to find the parameter value to compensate that bias. If one’s goal is to check where the uncertaintygoes, other hypotheses could have been made. For example, a non zero discrepancy expectation would quantify themean of the gap between the code and the experiments. In calibration we want this gap to be taken into account inthe code through adjusted θ .

The quality of the Gaussian process is also mentioned. Indeed, in code calibration if the estimation of theparameters of the Gaussian process is based on a not satisfactory DOE, calibration quality is then affected. Sometechniques as the sequential design, based of the expected improvement criterion, allow to enhance the initial DOEby finding new points regarding further calibration.

One can wonder which models to use in a given particular case. There is no obvious answer to this questionbut it depends first on the numerical code. If it is time consuming, the first model to try on is M2 and, if it is not,one can use M1. One may then wonder whether it is worth adding a discrepancy term, going from M1 to M3 orfrom M2 to M4. In-depth work on statistical validation had been developed in Damblin et al. (2016) in which thecomparison between two models (with and without discrepancy) is studied in a simplified context. The Bayes factorhelps to decide whether the discrepancy is relevant or not. However, this Bayes factor is burdensome to compute ina general context.

89

Cha

pter

4

CHAPTER4CALICO: A R PACKAGE FOR BAYESIAN

CALIBRATION

4.1 Guidelines for users . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92

4.2 Multidimensional example with CaliCo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97

4.2.1 The models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98

4.2.2 Priors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102

4.2.3 Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103

4.2.4 Additionnal tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108

4.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111

This chapter presents and illustrates the package CaliCo that has been published on CRAN (Comprehensive RArchive Network). It is based on all theoretical developments given Chapter 3. Three packages have been developedfor Bayesian calibration. The package BACCO (Hankin, 2013b) is a bundle of several other packages. It importsemulator (Hankin, 2014), mvtnorm (Genz et al., 2018), calibrator (Hankin, 2013c) and approximator (Hankin,2013a). These packages contain functions that perform Bayesian calibration and also prediction. The statisticalmodel implemented concerns only the case where the numerical code is time consuming and when a code error isadded (the model introduced by Kennedy and O’Hagan (2001)). Moreover, BACCO explores the prior distribution,of the parameters from the statistical model, using analytic or numerical integration. Similarly, another packagecalled SAVE (Palomo et al., 2017) deals with Bayesian calibration through four main functions: SAVE, bayesfit,predictcode and validate. The function SAVE creates the statistical model when bayesfit, predictcode andvalidate perform respectively Bayesian calibration, prediction and validation of a model. Calibration is donein SAVE in a similar way to BACCO because it is based on the same statistical model. Both packages are notflexible with the numerical code and the statistical model. A design of experiments has to be run upstream on thecode before running calibration. The package RobustCalibration based on Gu and Wang (2017) (Gu, 2018b) is apackage that achieves calibration of inexact mathematical models and implements the discrepancy with a “scaledGaussian process”.

CaliCo offers more flexibility on the statistical model choice. If one is interested in calibrating a numericalcode inexpensive in computation time, CaliCo allows the user to upload the numerical code in the model andrun Bayesian calibration with it. A very intuitive perspective is given to the user by using four functions calledmodel, prior, calibrate and forecast, that are detailed in Section 4.1. CaliCo also allows the user to accessseveral ggplot2 (Wickham and Chang, 2016) graphs very easily and to load each of them to change the layoutat one’s convenience. At some point, if the code is time consuming, calibration needs a surrogate to emulate it.Usually, a Gaussian process is chosen (Sacks et al., 1989; Cox et al., 2001). Three packages are related to theestablishment of a Gaussian process as a surrogate: GPfit (MacDoanld et al., 2015), DiceKriging (Roustant et al.,

91

Cha

pter

4

Chapter 4 – CaliCo: a R package for Bayesian calibration

2015) and RobustGaSP based on Gu (2018a) (Mengyang Gu and Berger, 2018). We use in CaliCo, the packageDiceDesign (Franco et al., 2015) to establish design of experiments (DOE). For compatibility matters, we havechosen DiceKriging to generate surrogates. Moreover, the time consuming steps of the Bayesian calibration arecoded in C++ and linked to R via the package Rcpp (Eddelbuettel et al., 2018).

In this section, the first part (Section 4.1) presents the main functions and functionalities of the package. Thesecond section provides a Bayesian calibration illustration on a toy example. It is based on a physical model of adamped harmonic oscillator with five parameters to calibrate.

4.1 Guidelines for users

CaliCo performs a Bayesian calibration through 3 different steps:

1. creation of the statistical model,

2. selection of the prior distributions,

3. running calibration with some simulation options.

CaliCo allows the user to easily take advantage of the calibration performed. Indeed, a prediction can beperformed on a new data set using the calibrated code in the statistical model. The main functions of the packageCaliCo are detailed Table 4.1.

Function Descriptionmodel generates a statistical modelprior creates one or a list of prior distributionscalibrate performs calibration for the model and prior specifiedforecast predicts the output over a new data set

Table 4.1: Main functions necessary for calibration in CaliCo.

CaliCo is coded in R6 (Chang, 2017) which is an oriented object language. Each function generates an R6object that can be used by other functions (in this case methods) that are proper to the object. The R6 layer is notvisible to the user. The main classes implemented with the associated functions are detailed Table 4.2.

Function R6 class calledmodel model.classprior prior.classcalibrate calibrate.classforecast forecast.class

Table 4.2: R6 classes called by the main functions in CaliCo.

To define the statistical model, which is the first step in calibration, several elements are necessary. The codefunction must be defined in R and takes two arguments X and θ respecting this order. For example:

code <- function(X,theta)

return((6*X-2)^2*sin(theta*X-4))

92

Cha

pter

4

4.1. Guidelines for users

If the numerical code is called from another language, one can implement a wrapper that calls from R thenumerical code according to the above writing. It is also possible to build a design of experiments and run the codeoutside R to get the outputs. Then, the DOE and the relative outputs are used instead of the numerical code in thestatistical model (more details below). The function model takes several other arguments (Table 4.3) as for examplethe vector or the matrix of the input variables (described in Section 1.4.1), the vector of experimental data or thestatistical model chosen for calibration.

model description Arguments to be specifiedfc(x,θ) the function to calibrate code (defined as code(x,theta))X the matrix of the input variables Xye the vector of experimental data YexpM the statistical model selected (Default value model1) model1, model2,

model3, model4Gaussian process options (optional only for (Optional) opt.gp (is a list)M2 and M4)Emulation options (optional only for (Optional) opt.emul (is a list)M2 and M4)Simulation options (optional only for (Optional) opt.sim (is a list)M2 and M4)Discrepancy options (necessary only for (Optional) opt.disc (is a list)M3 and M4)

Table 4.3: description of the arguments of the function model.

If the chosen model is M2 or M4, then a Gaussian process will be created as a surrogate of the function code.In each case the Gaussian process option (opt.gp, see Table 4.3) is needed. It is a list which encompasses:

• type: type of covariance function chosen for the surrogate established by the package DiceKriging (Roustantet al., 2015),

• DOE: design of experiments for the surrogate (default value NULL).

Three cases can occur. First, the numerical code is available and the user does not possess any Design OfExperiments (DOE). In this case, only the Gaussian process option opt.gp and the emulation option opt.emul

(Table 4.3) are needed. The emulation option controls the establishment of the DOE. It is a list which contains:

• p: the number of parameters in the model,

• n.emul: the number of points for constituting the DOE,

• binf: the lower bound of the parameter vector,

• bsup: the upper bound of the parameter vector.

The second possible case is when the user want to enforce a specific DOE. Note that in opt.gp, the DOE option isNULL. One can upload a specific DOE in this option. As the new DOE will be used, the emulation option opt.emul

is not needed anymore.

The third case is when no numerical code is available. The user is only in possession of a DOE and thecorresponding code evaluations. Then, the simulation option opt.sim is added. This option encompasses:

• Ysim: the code evaluations of DOEsim,

• DOEsim: the specific DOE used to get simulated data.

93

Cha

pter

4

Chapter 4 – CaliCo: a R package for Bayesian calibration

When this option is added, the emulation option is not necessary anymore. The code argument in the functionmodel can then be set to code=NULL. Table 4.4 presents a summary of these three cases and the options to add inthe function model.

cases options needed in the function modelnumerical code without DOE opt.gp and opt.emulnumerical code with DOE opt.gpno numerical code opt.gp and opt.sim

Table 4.4: Summary of the options needed depending on the case.

If M3 or M4 is chosen, a discrepancy term is added in the statistical model. This discrepancy is createdin CaliCo with the option opt.disc in the function model. It is a list composed of one component calledtype.kernel which corresponds to the correlation function of the discrepancy chosen. The list of the correlationfunctions implemented are detailed in Table 4.5.

kernel.type covariance implemented

gauss g(d) = σ2exp(− 1

2 (dψ)2)

exp g(d) = σ2exp(− 1

2dψ

)matern3_2 g(d) = σ2

(1+√

3 d2

ψ

)exp(−√

3 d2

ψ

)matern5_2 g(d) = σ2

(1+√

5 d2

ψ+5 d2

3ψ2

)exp(−√

5 d2

ψ

)Table 4.5: Kernel implemented for the discrepancy covariance.

The model function creates an R6 object in which two methods have been coded and are able to be used asregular functions. These function are plot and print. The function print gives an access to a short summaryof the statistical model created. The function plot allows to get a visual representation. However, to get a visualrepresentation, parameter values have to be specified in the model. A pipe %<% is available in CaliCo to parametrizea model. Let us consider a created random model called myModel. The code line myModel %<% param is the wayto give the model parameter values. The param variable is a list containing values of θ (named theta in thelist), θ δ for M3 and M4 (variance and correlation length of the discrepancy, named thetaD in the list) and σ2

e

(named var). Section 4.2 gives an overview of how the pipe works for each models. The plot function takes twoarguments that are the model generated by model and the x-axis to draw the results. An additional option CI (bydefault CI="all") allows to select which credibility interval at 95% one wants to display:

• CI="err" only the credibility interval of the measurement error with (or without) the discrepancy is given,

• CI="GP", only the credibility interval of the Gaussian process is plotted,

• CI="all" all credibility intervals are displayed.

The second step for calibration is to define the prior distributions of the parameters we seek to estimate. At least,two prior distributions have to be set and it is in the case where the code function takes only one input parameter θ .That means, only the posterior distributions of this parameter and the variance σ2

e (for the model M1) are what weseek to sample in calibration.

Table 4.6 describes the options needed into the function prior. Three choices of type.prior are available sofar (gaussian, gamma and unif which respectively stands for Gaussian, Gamma and Uniform distributions, seeTable 4.7 for details). For calibration with 2 parameters (which is the lower dimensional case), type.prior is avector (type.prior=c("gaussian","gamma") for example). Then, opt.prior is a list containing characteris-tics of each distribution. For the Gaussian distribution, it will be a vector of the mean and the variance, for the

94

Cha

pter

4

4.1. Guidelines for users

prior arguments descriptiontype.prior vector or scalar of string among ("gaussian", "gamma" and "unif")opt.prior list of vector corresponding to the distribution parameters

Table 4.6: description of the arguments of the function prior.

Gamma distribution it will be the shape and the scale and for the Uniform distribution the lower bound and theupper bound (opt.prior=list(c(1,0.1),c(0.01,1)) for example).

type.prior distribution arguments in opt.prior vector

gaussian f (x) = 1√2πV

exp(− 1

2 ((x−m)2

V ))

c(m,V)

gamma f (x) = 1(ka∗Γ(a))x(a−1)exp(− x

k ) c(a,k)unif f (x) = 1

b−a c(a,b)

Table 4.7: description of the arguments of the function prior.

When the prior distributions and the model are defined, calibration can be run. The function calibrate

implements a Markov Chain Monte Carlo (MCMC) according to specific conditions all controlled by the user.

calibrate arguments Descriptionmd the model generated with the function modelpr the list of prior generated by the function prioropt.estim estimation options for calibrationopt.valid (optional) cross validation options (default value NULL)

Table 4.8: description of the arguments of the function calibrate.

The MCMC implemented in CaliCo is composed of two algorithms as described in Section 2.5. The imple-mented algorithm (Algorithm 7) is coded in C++ thanks to Rcpp package (Eddelbuettel et al., 2018) in order tolimit the time consuming aspect of these non-parallelizable loops. Note that there is an adaptability present to reg-ulate the parameter k according to the acceptation rate. The user is free to set that regulation at the wanted percentage.

Then, the opt.estim option is a list composed of:

• Ngibbs: the number of iterations of the Metropolis within Gibbs algorithm,

• Nmh: the number of iteration of the Metropolis Hastings algorithm,

• thetaInit: the starting point,

• r: the vector of regulation of the covariance in the Metropolis within Gibbs algorithm (in the propositiondistribution the variance is kΣ),

• sig: the variance of the proposition distribution Σ,

• Nchains: (default value 1) the number of MCMC chains to run,

• burnIn: the number of iteration to withdraw from the Metropolis Hastings algorithm.

In the function calibrate, one optional argument is available to run a cross validation. This option calledopt.valid, is a list composed of two options which have to be filled:

• type.CV: the type of cross validation wanted (leave one out is the only cross validation implemented so fartype.CV="loo"),

95

Cha

pter

4

Chapter 4 – CaliCo: a R package for Bayesian calibration

• nCV: the number of iteration to run in the cross validation.

After calibration is complete, an R6 object is created and two methods (print and plot) are available and arealso able to be used as regular functions. The print function is a summary that recalls the selected model, thecode used for calibration, the acceptation rate of the Metropolis within Gibbs algorithm, the acceptation rate of theMetropolis Hastings algorithm, the maximum a posteriori and the mean a posteriori. It allows to quickly check theacceptation ratios and see if the chains have properly mixed. The plot function generates automatically, a series ofgraphs that displays, notably, the output of the calibrated code. Two arguments are necessary to run this function:the calibrated model and the x-axis to draw the results. An additional option graph (by default graph="all")allows to control which graphic layout one wants to plot:

• if graph="chains" a layout containing the autocorrelation graphs, the MCMC chains and the prior andposterior distributions for each parameter is given,

• if graph="corr" a layout containing in the diagonal the prior and posterior distributions for the parametervector θ and the scatterplot between each pair of parameters is plotted,

• if graph="results" the result of calibration is displayed,

• if graph="all all of them are printed.

Note that all these graphs (made in ggplot2) are proposed in a particular layout but one can easily load all ofthem into a variable and extract the particular graph one wants. Indeed, if a variable p is used to store all the graphs,then p is a list containing "ACF" (the autocorrelation graphs), "MCMC" (the MCMC chains), "corr" (the scatterplot between each pair of parameters), "dens" (prior and posterior distributions) and "out" (the calibration resultgraph) variables.

Two external functions can be run on an object generated by the function calibrate:

• chain: function that allows to extract the chains sampled in the posterior distribution. If the variableNchains, in opt.estim option, is higher than 1 then the function chain return a coda (Plummer et al.,2016) object with the sampled chains,

• estimators: function that accesses the maximum a posteriori(MAP) and the mean a posteriori.

Sequential design introduced in Damblin et al. (2018) allows to improve the Gaussian process estimation forM2 and M4. Based on the expected improvement (EI), introduced in Jones et al. (1998), new points are added to aninitial DOE in order to improve the quality of calibration. The arguments of the function are given Table 4.9.

sequentialDesign arguments Descriptionmd the model generated with the function model

(for M2 or M4)pr the list of prior generated by the function prioropt.estim estimation options for calibrationk number of points to add in the design

Table 4.9: description of the arguments of the function sequentialDesign.

The last main function in CaliCo is forecast which produces a prediction of a selected model on a new dataand based on previous calibration.

The object generated by forecast.class possesses the two similar methods print and plot. The print

function gives a summary identical to the one in model.class except that it adds the MAP estimator. The plot

96

Cha

pter

4

4.2. Multidimensional example with CaliCo

forecast arguments Descriptionmodelfit calibrated model (run by calibrate function)x.new new data for prediction

Table 4.10: description of the arguments of the function calibrate.

function displays the calibration results and also adds the predicted results. The arguments of plot are the predictedmodel and the x-axis which is the axis corresponding to calibration extended with the axis corresponding to theforecast.

4.2 Multidimensional example with CaliCo

An illustration is provided in this section to help the user to easily handle the functionalities of CaliCo. Thisexample, represents a damped harmonic oscillator and experimental data are simulated for specific values of theparameter vector θ . These parameters to calibrate are A the constant amplitude, the damping ratio ξ , the springconstant k, the mass of the spring m and φ the phase. The recorded displacement of the damped oscillator isrepresented in Figure 4.1 and the equation of the displacement of a damped harmonic oscillator is:

x(t,θ) : R×R5 7→ R (4.1)

(t,θ = (A,ξ ,k,m,φ)T )→ Ae−ξ

√km tsin(

√1−ξ 2

√km

t +φ) (4.2)

−0.5

0.0

0.5

1.0

0.0 0.5 1.0 1.5 2.0t

x

Figure 4.1: Displacement of the oscillator simulated.

There is five parameters to calibrate. Let us consider that experiments are available for the 2 first seconds of themovement (these experiments have been simulated for the interval time [0,2] with the time step of 40ms and forspecific parameter values). Visually, at time t = 0 the position of the mass seems to be at the position x = 1. So thevalues a priori of A and φ are A = 1 and φ = π

2 . The company states that the spring has a constant of k = 6N/m andthe mass is weighing at m = 50g. The major uncertainty lies in the knowledge of ξ . It is indeed a difficult parameterto estimate. However, the value of the damping ratio ξ determines the behavior of the system. A damped harmonicoscillator can be:

97

Cha

pter

4

Chapter 4 – CaliCo: a R package for Bayesian calibration

• over-damped (ξ > 1): the system exponentially decays to steady state without oscillating,

• critically damped (ξ = 1): the system returns to steady state as quickly as possible without oscillating,

• under-damped (ξ < 1): The system oscillates with the amplitude gradually decreasing to zero.

Physical experts provide us a value of ξ = 0.3 but says that the parameter can oscillate between the value[0.15,0.45] at 95%.

4.2.1 The models

To define the first statistical model, the function code has to be defined such as:

n <- 50

t <- seq(0,2,length.out=n)

code <- function(t,theta)

w0 <- sqrt(theta[3]/theta[4])

return(theta[1]*exp(-theta[2]*w0*t)*sin(sqrt(1-theta[2]^2)*w0*t+theta[5]))

In CaliCo, one function allows to define the statistical model. This function model takes as inputs the code

function, the input variables X, experimental data and the model choice. If a numerical code has no input variables,it is just enough to put X=0.

model1 <- model(code,X=t,Yexp,"model1")

In this particular case where, the input variables are unidimensional, it is easy to choose a graphical representationof the model. As mentioned in Section 4.1, when the function model is called, a model.class object is generated.This object owns several methods as plot or print that behave as regular functions.

print(model1)

## Call:

## [1] "model1"

##

## With the function:

## function(t,theta)

##

## w0 <- sqrt(theta[3]/theta[4])

## return(theta[1]*exp(-theta[2]*w0*t)*sin(sqrt(1-theta[2]^2)*w0*t+theta[5]))

##

##

## No surrogate is selected

##

## No discrepancy is added

To get a visual representation, parameter values need to be added to the model. To achieve such an operationin CaliCo, one can use the defined pipe %<%. Following the pipe, a list containing all parameter values allows toselect these values for the visual representation. The parameter vector (theta) and the value of the variance of the

98

Cha

pter

4

4.2. Multidimensional example with CaliCo

measurement error (var), here, are needed in the list to set a proper parametrization of the model:

model1 %<% list(theta=c(1,0.3,6,50e-3,pi/2),var=1e-4)

## Warning: Please be careful to the size of the parameter vector

The Warning is present at each use of the pipe. It appears as a reminder for the user to be careful with the sizeof the parameter vector. When the model is defined nothing indicates the number of parameter within. To get avisual representation of the model with such parameters values, the plot function can be straightforwardly appliedon the model object created by model and completed by the pipe %<%. The x-axis needs to be filled in plot to getan x-axis for display. The left panel of Figure 4.3 is the result of:

plot(model1,t)

If no parameter value is added to the model and the visual representation is required, a Warning appears andremind the user that no parameter value has been defined and only experiments are plotted (Figure 4.2):

model1bis <- model(code,X=t,Yexp,"model1")plot(model1bis,t)

## Warning: no theta and var has been given to the model, experiments only areplotted

−0.5

0.0

0.5

1.0

0.0 0.5 1.0 1.5 2.0t

x

exp

Figure 4.2: Experimental data displayed when no parameter values are set in the model.

If no x-axis is defined, then no visual representation is possible and the function plot breaks:

model1bis <- model(code,X=t,Yexp,"model1")

plot(model1bis)

## Error: No x-axis selected, no graph is displayed

For M2 several cases may occur (see Table 4.4 for more details). First the user only has the time consuming codewithout any Design Of Experiments (DOE). Then, the definition of the model is done by delimiting the boundariesof the parameters. The option opt.gp allows the user to set the kernel type of the Gaussian process and to specifyif the user has a particular DOE. In this first case the DOE is not available, so DOE=NULL in the opt.gp option. Toparametrize the DOE created in the function model, the option called opt.emul needs to be filled by p, n.emul,

99

Cha

pter

4

Chapter 4 – CaliCo: a R package for Bayesian calibration

binf, bsup. Where p stands for the number of parameter to calibrate, n.emul for the number of experiments in theDOE, binf and bsup for the lower and upper bounds of the parameter vector.

binf <- c(0.9,0.15,5.8,48e-3,1.49)

bsup <- c(1.1,0.45,6.2,52e-3,1.6)

model2 <- model(code,t,Yexp,"model2",

opt.gp = list(type="matern5_2",DOE=NULL),

opt.emul = list(p=5,n.emul=60,binf=binf,bsup=bsup))

The second case is when the users has a numerical code and a specific DOE. In CaliCo, the option DOE inopt.gp allows to consider a particular DOE wanted by the user. As no DOE is build with the function model, theoption opt.emul is not necessary anymore:

library(DiceDesign)

DOE <- maximinSA_LHS(lhsDesign(60,6)$design)$design

DOE <- unscale(DOE,c(0,binf),c(2,bsup))

model2doe <- model(code,t,Yexp,"model2",

opt.gp=list(type="matern5_2",DOE=DOE))

When one does not possess any numerical code, but only the DOE and the corresponding output, another option,called opt.sim, needs to be filled. The opt.gp option is still needed to specify the chosen kernel but the opt.emul

option is no longer necessary (for the same reasons as in the second case). The opt.sim option is the list containingthe DOE and the output of the code. As the user does not possess the numerical code, the code option in thefunction model can be set to code=NULL.

Ysim <- code(DOE[1,1],DOE[1,2:6])

for (i in 2:60)Ysim <- c(Ysim,code(DOE[i,1],DOE[i,2:6]))

model2code <- model(code=NULL,t,Yexp,"model2",

opt.gp = list(type="matern5_2", DOE=NULL),

opt.sim = list(Ysim=Ysim,DOEsim=DOE))

The package CaliCo is comfortable with these three situations and bring flexibility according to the differentproblems of the users. Similarly as before, parameter values need to be added to each models and the functionprint and plot can be directly used:

ParamList <- list(theta=c(1,0.3,6,50e-3,pi/2),var=1e-4)

model2 %<% ParamList

model2doe %<% ParamList

model2code %<% ParamList

plot(model2,t)

plot(model2doe,t)

plot(model2code,t)

These three lines of code produce the same graphs because CaliCo uses a maximin Latin Hypercube Sample(LHS) to establish the DOE. Several credibility interval are displayed. For the first model only the 95% credibility

100

Cha

pter

4

4.2. Multidimensional example with CaliCo

interval of the measurement error is available. For the second model, the 95% credibility interval of the Gaussianprocess can also be shown. Figure 4.3 illustrates M1 and M2.

M1 M2

−0.5

0.0

0.5

1.0

0.0 0.5 1.0 1.5 2.0t

x

CI 95% noise

expmodel output

−0.5

0.0

0.5

1.0

0.0 0.5 1.0 1.5 2.0t

x

expmodel output

CI 95% GPCI 95% noise

Figure 4.3: First and second model output for prior belief on parameter values. The left panel illustrates the firstmodel and the right panel the second model with the Gaussian process estimated.

One can be interested in modeling a code discrepancy. When the code is not time consuming, the model tochoose is M3. The opt.disc option allows to specify the kernel type of the discrepancy. Note that the visualrepresentation requires initial values for the discrepancy and are needed in the pipe %<%. This vector thetaD iscomposed of σ2 and ψ according to Table 4.5.

model3 <- model(code,t,Yexp,"model3",

opt.disc = list(kernel.type="gauss"))

model3 %<% list(theta=c(1,0.3,6,50e-3,pi/2),thetaD=c(1e-4,0.2),var=1e-4)

When the code is time consuming, then M4 is selected. The same cases can occur as for M2 but only the casewhere the code is not available will be considered, here, for M4.

model4 <- model(code=NULL,t,Yexp,"model4",

opt.gp = list(type="matern5_2", DOE=NULL),

opt.sim = list(Ysim=Ysim,DOEsim=DOE),

opt.disc = list(kernel.type="gauss"))

model4 %<% list(theta=c(1,0.3,6,50e-3,pi/2),thetaD=c(1e-4,0.2),var=1e-4)

To get a visual representation of M3 and M4, the function plot is defined identically as before. The results aredisplayed in Figure 4.4.

plot(model3,t)

plot(model4,t)

Note that several credibility intervals can be displayed. By default all of them are shown (for example see rightpanels of Figure 4.3 and Figure 4.4). For M1 and M3 only one credibility interval is given. It represents the 95%credibility interval of the measurement error, in the case of M1, and the 95% credibility interval of the measurementerror plus the discrepancy, in the case of M3. For M2 and M4, two credibility intervals are available. Compared toM1 and M3 the credibility interval at 95% of the Gaussian process, that emulates the code, is added. It allows to

101

Cha

pter

4

Chapter 4 – CaliCo: a R package for Bayesian calibration

M3 M4

−0.5

0.0

0.5

1.0

0.0 0.5 1.0 1.5 2.0t

x

expmodel output

CI 95% discrepancy + noise

−0.5

0.0

0.5

1.0

0.0 0.5 1.0 1.5 2.0t

x

CI 95% discrepancy + noiseCI 95% GP

expmodel ouput

Figure 4.4: Third and fourth model output for prior belief on parameter values. The left panel illustrates thethird model and the right one, the fourth model with the Gaussian process estimated. Both are encompassing thediscrepancy.

quickly visualize from where the variability of the model comes before calibration.

With the option CI, one can deactivate or select which credibility interval (CI) one wants to display. By defaultCI="all", but if CI="err" only the 95% CI of the measurement error with, or without, the discrepancy is given.Similarly, for M2 and M4, if CI="GP", only the 95% CI of the Gaussian process is shown. For example, for M4

the three possibilities are obtained with the following code and are displayed in Figure 4.5.

plot(model4,t,CI="err")

plot(model4,t,CI="GP")

plot(model4,t,CI="all")

−0.5

0.0

0.5

1.0

0.0 0.5 1.0 1.5 2.0t

x

expmodel output

CI 95% discrepancy + noise

−0.5

0.0

0.5

1.0

0.0 0.5 1.0 1.5 2.0t

x

CI 95% GP

expmodel ouput

−0.5

0.0

0.5

1.0

0.0 0.5 1.0 1.5 2.0t

x

CI 95% discrepancy + noiseCI 95% GP

expmodel ouput

Figure 4.5: M4 displayed for some guessed values with the CI relative to the measurement error on the left panel,with the CI relative to the Gaussian process only on the middle panel and both credibility intervals on the rightpanel.

4.2.2 Priors

To a proper Bayesian calibration, prior distributions have to be defined on every parameters we want to estimate(parameters of interest as θ or nuisance parameter such as θ δ or σ2

err). It means that the number of parameters toestimate differs according to the model. In CaliCo, the possible distributions are detailed in Table 4.7.

102

Cha

pter

4

4.2. Multidimensional example with CaliCo

To define a proper prior distribution in CaliCo, a prior.class object with the function prior is generated.One or several prior distributions can be produced with this function. Two arguments have to be completed:type.prior and opt.prior. The argument type.prior can be a string (if only one prior distribution is lookedfor) or a vector of strings (if several prior distributions are wanted). Similarly, the argument opt.prior can be avector of the distribution parameters or a list of vectors.

In this example, 5 parameters have to be calibrated. For M3 and M4, the discrepancy is added and the varianceσ2

δwith the correlation length ψ have to be estimated as much as the other parameters. It means that for these

models, two more prior distributions have to be added compared at M1 and M2. The order to define them are, firstthe parameters θ , then θ δ and σ2

err. In the following code lines, pr1 stands for the prior distributions for M1 andM2 where pr2 for M3 and M4. In the first prior definition, only the 5 parameters and σ2

err prior distributions aredefined. In the second definition, the θ δ prior distributions are added between θ and σ2

err ones.

type.prior <- c(rep("gaussian",5),"gamma")

opt.prior <- list(c(1,1e-3),c(0.3,1e-3),c(6,1e-3),c(50e-3,1e-5),

c(pi/2,1e-2),c(1,1e-3))

pr1 <- prior(type.prior,opt.prior)

type.prior <- c(rep("gaussian",5),"gamma","unif","gamma")

opt.prior <- list(c(1,1e-3),c(0.3,1e-3),c(6,1e-3),c(50e-3,1e-5),

c(pi/2,1e-2),c(1,1e-3),c(0,1),c(1,1e-3))

pr2 <- prior(type.prior,opt.prior)

0.0

2.5

5.0

7.5

10.0

0.9 1.0 1.1A

dens

ity

prior

0.0

2.5

5.0

7.5

10.0

0.2 0.3 0.4ξ

prior

0.0

2.5

5.0

7.5

10.0

5.9 6.0 6.1k

prior

0

30

60

90

0.04 0.05 0.06m

prior

0

1

2

3

1.25 1.50 1.75 2.00

φ2

dens

ity

prior

0

50

100

150

200

250

0.0000 0.0025 0.0050 0.0075

σδ2

prior

0.00

0.25

0.50

0.75

1.00

0.00 0.25 0.50 0.75 1.00ψδ

prior

0

100

200

300

0.0000 0.0025 0.0050 0.0075

σerr2

prior

Figure 4.6: Prior distributions for each parameter to calibrate in the application case.

Figure 4.6 illustrates the prior distributions considered for the parameters of the damped harmonic oscillatorcode. For M1 and M2, only A, ξ , k, m, φ 2 and σ2

err distributions are useful. Calibration with M3 or M4 the twolast distributions (σ2

δand ψδ ) are then used.

4.2.3 Calibration

Calibration is run thanks to the function calibrate in CaliCo. Estimation option (opt.estim) has to be filledto run the algorithm properly. As it is described in Section 4.1, two MCMC algorithms are run by the functioncalibrate.

103

Cha

pter

4

Chapter 4 – CaliCo: a R package for Bayesian calibration

opt.estim = list(Ngibbs=1000,Nmh=5000,thetaInit=c(1,0.25,6,50e-3,pi/2,1e-3),

r=c(0.05,0.05),sig=diag(6),Nchains=1,burnIn=2000)

mdfit1 <- calibrate(model1,pr1,opt.estim)

In the terminal, a loading bar represents the execution time of the inference algorithm. Then, the method print

can be used to quickly access some information (see Section 4.1).

print(mdfit1)

## Call:

##

## With the function:

## function(t,theta)

##

## w0 <- sqrt(theta[3]/theta[4])

## return(theta[1]*exp(-theta[2]*w0*t)*sin(sqrt(1-theta[2]^2)*w0*t+theta[5]))

##

## <bytecode: 0x561a96e4f068>

##

## Selected model : model1

##

## Acceptation rate of the Metropolis within Gibbs algorithm:

## [1] "46.6%" "27%" "27.9%" "8.9%" "5.1%" "4.4%"

##

## Acceptation rate of the Metropolis Hastings algorithm:

## [1] "44.52%"

##

## Maximum a posteriori:

## [1] 1.0138252830 0.2032794864 5.9924680899 0.0505938887 1.5199179525

## [6] 0.0002565248

##

## Mean a posteriori:

## [1] 1.0107256606 0.2010363776 5.9755232004 0.0497000370 1.5042868046

## [6] 0.0002493378

To visualize the results, the plot method allows to generate ggplot2 objects and if the option graph is notdeactivated, the function plot will create a layout of graphs displayed in Figure 4.7 which contains:

1. the auto-correlation graphs,

2. the chains trajectories,

3. the prior and posterior distributions,

4. the correlation between parameters,

5. the results on the quantity of interest.

To generate all the graphs of Figure 4.7, the plot function is used similarly as for a model.class object withthe following code line.

104

Cha

pter

4

4.2. Multidimensional example with CaliCo

plot(mdfit1,t)

0.00

0.25

0.50

0.75

1.00

0 10 20 30

Lag

AC

F

0.00

0.25

0.50

0.75

1.00

0 10 20 30

LagA

CF

0.00

0.25

0.50

0.75

1.00

0 10 20 30

Lag

AC

F

0.00

0.25

0.50

0.75

1.00

0 10 20 30

Lag

AC

F

0.00

0.25

0.50

0.75

1.00

0 10 20 30

Lag

AC

F

0.00

0.25

0.50

0.75

1.00

0 10 20 30

Lag

AC

F

1.010

1.012

1.014

0 1000 2000 3000

iterations

θ 1

0.199

0.200

0.201

0.202

0.203

0 1000 2000 3000

iterations

θ 2

5.96

5.97

5.98

5.99

0 1000 2000 3000

iterations

θ 3

0.0490

0.0495

0.0500

0.0505

0 1000 2000 3000

iterations

θ 4

1.49

1.50

1.51

1.52

0 1000 2000 3000

iterations

θ 5

0.000244

0.000248

0.000252

0.000256

0 1000 2000 3000

iterations

σ err

2

0

100

200

300

400

0.9 1.0 1.1θ1

dens

ity

posterior

prior

0

100

200

300

400

0.2 0.3 0.4θ2

dens

ity

posterior

prior

0

20

40

60

5.9 6.0 6.1θ3

dens

ityposterior

prior

0

250

500

750

1000

1250

0.04 0.05 0.06θ4

dens

ity

posterior

prior

0

20

40

60

80

1.25 1.50 1.75 2.00θ5

dens

ity

posterior

prior

0e+00

1e+05

2e+05

3e+05

0.0000 0.0025 0.0050 0.0075σerr

2

dens

ity

posterior

prior

0

100

200

300

400

0.9 1.0 1.1

θ1

dens

ity

posterior

prior

1.010

1.012

1.014

0.199 0.200 0.201 0.202 0.203

θ2

θ 1

1.010

1.012

1.014

5.96 5.97 5.98 5.99

θ3

θ 1

1.010

1.012

1.014

0.0490 0.0495 0.0500 0.0505

θ4

θ 1

1.010

1.012

1.014

1.49 1.50 1.51 1.52

θ5

θ 1

0.199

0.200

0.201

0.202

0.203

1.010 1.012 1.014

θ1

θ 2

0

100

200

300

400

0.2 0.3 0.4

θ2

dens

ity

posterior

prior

0.199

0.200

0.201

0.202

0.203

5.96 5.97 5.98 5.99

θ3

θ 2

0.199

0.200

0.201

0.202

0.203

0.0490 0.0495 0.0500 0.0505

θ4

θ 2

0.199

0.200

0.201

0.202

0.203

1.49 1.50 1.51 1.52

θ5

θ 2

5.96

5.97

5.98

5.99

1.010 1.012 1.014

θ1

θ 3

5.96

5.97

5.98

5.99

0.199 0.200 0.201 0.202 0.203

θ2

θ 3

0

20

40

60

5.9 6.0 6.1

θ3

dens

ity

posterior

prior

5.96

5.97

5.98

5.99

0.0490 0.0495 0.0500 0.0505

θ4

θ 3

5.96

5.97

5.98

5.99

1.49 1.50 1.51 1.52

θ5

θ 3

0.0490

0.0495

0.0500

0.0505

1.010 1.012 1.014

θ1

θ 4

0.0490

0.0495

0.0500

0.0505

0.199 0.200 0.201 0.202 0.203

θ2

θ 4

0.0490

0.0495

0.0500

0.0505

5.96 5.97 5.98 5.99

θ3

θ 4

0

250

500

750

1000

1250

0.04 0.05 0.06

θ4

dens

ity

posterior

prior

0.0490

0.0495

0.0500

0.0505

1.49 1.50 1.51 1.52

θ5

θ 4

1.49

1.50

1.51

1.52

1.010 1.012 1.014

θ1

θ 5

1.49

1.50

1.51

1.52

0.199 0.200 0.201 0.202 0.203

θ2

θ 5

1.49

1.50

1.51

1.52

5.96 5.97 5.98 5.99

θ3

θ 5

1.49

1.50

1.51

1.52

0.0490 0.0495 0.0500 0.0505

θ4

θ 5

0

20

40

60

80

1.25 1.50 1.75 2.00

θ5

dens

ity

posterior

prior

−0.5

0.0

0.5

1.0

0.0 0.5 1.0 1.5 2.0

x

y

calibrated

experiments

95% credibility interval a posteriori

Figure 4.7: Series of plot generated by the function plot for calibration on M1.

Same procedures are available for M2.

mdfit2 <- calibrate(model2,pr1,opt.estim)

For M3 and M4, the estimation options are slightly different because the number of parameter to estimate hasincreased. The prior object also has changed.

opt.estim2=list(Ngibbs=1000,Nmh=5000,thetaInit=c(1,0.3,6,50e-3,pi/2,1e-3,0.5,1e-3),

r=c(0.05,0.05),sig=diag(8),Nchains=1,burnIn=2000)

mdfit3 <- calibrate(model3,pr2,opt.estim2)

mdfit4 <- calibrate(model4,pr2,opt.estim2)

105

Cha

pter

4

Chapter 4 – CaliCo: a R package for Bayesian calibration

print(mdfit4)

## Call:

##

## With the function:

## NULL

##

## Selected model : model4

##

## Acceptation rate of the Metropolis within Gibbs algorithm:

## [1] "97.8%" "94.7%" "96.8%" "90.3%" "87.5%" "94.3%" "97.5%" "94.1%"

##

## Acceptation rate of the Metropolis Hastings algorithm:

## [1] "59.2%"

##

## Maximum a posteriori:

## [1] 1.0145661817 0.3052534056 6.0274228120 0.0521952278 1.6229728079

## [6] 0.0009970529 0.4769160005 0.0007652975

##

## Mean a posteriori:

## [1] 0.9968290203 0.2835127409 6.0032790767 0.0506295380 1.5845041957

## [6] 0.0009225639 0.4235883680 0.0006717480

Figure 4.7 illustrates the several graphs layout one can obtain with the use of the function plot. To select whichspecific graph one wants to display, the option graph can be added to the function plot:

• graph="chains": only the table of the autocorrelation, chains points and distributions a priori and a

posteriori is produced . It represents only the top part of Figure 4.7,

• graph="corr": only the table of the correlation graph between each parameter is displayed. It representsonly the bottom left part of Figure 4.7,

• graph="result": only the result on the quantity of interest is given. It represents only the bottom right partof Figure 4.7,

• graph=NULL: no graphs are produced automatically.

If one does not want to produce these graphs automatically, one can set the graph option to NULL. As the plot

function generates ggplot2 objects, it is possible to load all the generated graphs apart.

p <- plot(mdfit4,t,graph=NULL)

The variable p is a list of all the graphs displayed in Figure 4.7. The elements in p are:

• ACF a list of all autocorrelation graphs in the chains for each variable,

• MCMC a list of all the MCMC chains for each variable,

• corrplot a list of all correlation graphs between each parameter,

• dens a list of all distribution a priori and a posteriori graphs for each variable,

• out the ggplot2 object of the result on the quantity of interest.

106

Cha

pter

4

4.2. Multidimensional example with CaliCo

Figure 4.8 illustrates the prior and posterior distributions resulted from calibration on M4.

0

25

50

75

100

0.9 1.0 1.1A

dens

ity

posteriorprior

0

20

40

60

80

0.2 0.3 0.4ξ

posteriorprior

0

20

40

60

5.9 6.0 6.1k

posteriorprior

0

250

500

750

0.04 0.05 0.06m

posteriorprior

0

10

20

30

1.25 1.50 1.75 2.00

φ2

dens

ity

posteriorprior

0

5000

10000

15000

20000

0.0000 0.0025 0.0050 0.0075

σδ2

posteriorprior

0

5

10

15

20

0.00 0.25 0.50 0.75 1.00ψδ

posteriorprior

0

5000

10000

15000

0.0000 0.0025 0.0050 0.0075

σerr2

posteriorprior

Figure 4.8: prior and posterior distributions for each parameter for calibration on M4.

Similarly, if one desires to access the graph of the result on the quantity of interest one only needs to run p$out.

p$out

−0.5

0.0

0.5

1.0

0.0 0.5 1.0 1.5 2.0

t

x

calibrated

experiments

95% credibility interval a posteriori

95% credibility interval for the discrepancy

Figure 4.9: Result of calibration on M4 for the quantity of interest with the credibility interval at 95% a posteriori.

107

Cha

pter

4

Chapter 4 – CaliCo: a R package for Bayesian calibration

4.2.4 Additionnal tools

A function in CaliCo called estimators allows to access estimators as the MAP and the mean a posteriori.

estimators(mdfit1)

## $MAP

## [1] 1.0105967765 0.2011318973 5.9741254796 0.0496559638 1.5042583733

## [6] 0.0002488385

##

## $MEAN

## [1] 1.0107256606 0.2010363776 5.9755232004 0.0497000370 1.5042868046

## [6] 0.0002493378

If one is interested in running convergence diagnostics on the MCMC chains run by the function calibrate,one is free to increase the number of chains in the opt.estim options. This operation is accomplished in parallelwith an automatically detected number of cores.

opt.estim=list(Ngibbs=1000,Nmh=5000,thetaInit=c(1,0.25,6,50e-3,pi/2,1e-3),

r=c(0.05,0.05),sig=diag(6),Nchains=3,burnIn=2000)

mdfitMCMC <- calibrate(model1,pr1,opt.estim)

By setting Nchains=3, calibration is run 3 times. The function chain allows to load the coda object generatedand then to use coda tools as Gelman-Rubin diagnostics (Gelman and Rubin, 1992) for example.

mcmc <- chain(mdfitMCMC)

library(coda)

gelman.diag(mcmc)

## Potential scale reduction factors:

##

## Point est. Upper C.I.

## [1,] 4.00 7.81

## [2,] 1.55 2.39

## [3,] 8.58 16.65

## [4,] 1.25 1.68

## [5,] 3.52 7.23

## [6,] 38.66 75.48

##

## Multivariate psrf

##

## 31.3

The user can also run very easily a cross validation (a leave one out) to estimate how accurately the modelprediction will perform in practice. An additional option, called opt.valid, is then necessary to run this crossvalidation. This option is a list containing the number of iteration (nCV) and the type cross validation method(type.valid).

108

Cha

pter

4

4.2. Multidimensional example with CaliCo

mdfitCV <- calibrate(model1,pr1,

opt.estim = list(Ngibbs=1000,

Nmh=5000,

thetaInit=c(1,0.25,6,50e-3,pi/2,1e-3),

r=c(0.05,0.05),

sig=diag(6),

Nchains=1,

burnIn=2000),

opt.valid = list(type.valid="loo",

nCV=50))

The activation of the cross validation will run the regular calibration and then the nCV iterations requested by theuser. To decrease the computational burden of such operation, a parallel operation is performed by to the packageparallel present in R core.

print(mdfitCV)

## Call:

##

## With the function:

## function(t,theta)

##

## w0 <- sqrt(theta[3]/theta[4])

## return(theta[1]*exp(-theta[2]*w0*t)*sin(sqrt(1-theta[2]^2)*w0*t+theta[5]))

##

## <bytecode: 0x561a93b33348>

##

## Selected model : model1

##

## Acceptation rate of the Metropolis within Gibbs algorithm:

## [1] "43.8%" "27.5%" "27.3%" "9.1%" "4.3%" "4.4%"

##

## Acceptation rate of the Metropolis Hastings algorithm:

## [1] "48.56%"

##

## Maximum a posteriori:

## [1] 1.0144067620 0.2032408379 6.0077799738 0.0508023686 1.5169972084

## [6] 0.0002536493

##

## Mean a posteriori:

## [1] 1.0121094549 0.2016018588 5.9975259667 0.0499179223 1.5060238122

## [6] 0.0002488799

##

##

## Cross validation:

## Method: loo

## Predicted Real Error

## 1 0.581834516 0.58394961 0.0021150930

109

Cha

pter

4

Chapter 4 – CaliCo: a R package for Bayesian calibration

## 2 0.251110861 0.25245428 0.0013434211

## 3 0.860725971 0.86208615 0.0013601740

## 4 0.860627965 0.86208615 0.0014581796

## 5 0.009987004 0.00981318 0.0001738244

## 6 0.251052820 0.25245428 0.0014014613

##

## RMSE: [1] 0.03737526

##

## Cover rate:

## [1] "94%"

The print method displays the head of the first iterations of the cross validation and the root mean square error(RMSE) associated. The coverage rate is also printed to check the accuracy of the posterior credibility interval.

The implemented function sequentialDesign is available only for M2 and M4. This function allows to run asequential design as described in Damblin et al. (2018). Based on the expected improvement (Jones et al., 1998),it improves the estimation of the Gaussian process that emulates the code by adding new points in the design.Calibration quality is, as expected, increased.

binf <- c(0.9,0.05,5.8,40e-3,1.49)

bsup <- c(1.1,0.55,6.2,60e-3,1.6)

model2 <- model(code,t,Yexp,"model2",

opt.gp = list(type="matern5_2",DOE=NULL),

opt.emul = list(p=5,n.emul=200,binf=binf,bsup=bsup))

type.prior <- c(rep("gaussian",5),"gamma")

opt.prior <- list(c(1,1e-3),c(0.3,1e-3),c(6,1e-3),c(50e-3,1e-5),

c(pi/2,1e-2),c(1,1e-3))

pr1 <- prior(type.prior,opt.prior)

newModel2 <- sequentialDesign(model2,pr1,

opt.estim = list(Ngibbs=100,

Nmh=600,

thetaInit=c(1,0.25,6,50e-3,pi/2,1e-3),

r=c(0.05,0.05),

sig=diag(6),

Nchains=1,

burnIn=200),

k=20)

110

Cha

pter

4

4.3. Conclusion

0.1

0.2

0.3

0.4

0.5

0.90 0.95 1.00 1.05 1.10θ1

θ 2

Initial DOE

Additional points

5.8

5.9

6.0

6.1

6.2

0.90 0.95 1.00 1.05 1.10θ1

θ 3

Initial DOE

Additional points

0.040

0.045

0.050

0.055

0.060

0.90 0.95 1.00 1.05 1.10θ1

θ 4

Initial DOE

Additional points

1.52

1.56

1.60

0.90 0.95 1.00 1.05 1.10θ1

θ 5

Initial DOE

Additional points

5.8

5.9

6.0

6.1

6.2

0.1 0.2 0.3 0.4 0.5θ2

θ 3

Initial DOE

Additional points

0.040

0.045

0.050

0.055

0.060

0.1 0.2 0.3 0.4 0.5θ2

θ 4

Initial DOE

Additional points

1.52

1.56

1.60

0.1 0.2 0.3 0.4 0.5θ2

θ 5

Initial DOE

Additional points

0.040

0.045

0.050

0.055

0.060

5.8 5.9 6.0 6.1 6.2θ3

θ 4

Initial DOE

Additional points

1.52

1.56

1.60

5.8 5.9 6.0 6.1 6.2θ3

θ 5

Initial DOE

Additional points

1.52

1.56

1.60

0.040 0.045 0.050 0.055 0.060θ4

θ 5

Initial DOE

Additional points

Before sequential design

0

50

100

150

200

0.9 1.0 1.1θ1

dens

ity

posterior

prior

0

100

200

300

0.2 0.3 0.4θ2

dens

ity

posterior

prior

0

20

40

60

80

5.9 6.0 6.1θ3

dens

ityposterior

prior

0

200

400

600

800

0.04 0.05 0.06θ4

dens

ity

posterior

prior

0

20

40

60

80

1.25 1.50 1.75 2.00θ5

dens

ity

posterior

prior

0

20000

40000

60000

0.00000.00250.00500.0075σerr

2

dens

ity

posterior

prior

After sequential design

0

100

200

0.9 1.0 1.1θ1

dens

ity

posterior

prior

0

100

200

0.2 0.3 0.4θ2

dens

ity

posterior

prior

0

20

40

60

80

5.9 6.0 6.1θ3

dens

ity

posterior

prior

0

400

800

1200

0.04 0.05 0.06θ4

dens

ityposterior

prior

0

20

40

60

80

1.25 1.50 1.75 2.00θ5

dens

ity

posterior

prior

0

20000

40000

60000

80000

0.00000.00250.00500.0075σerr

2

dens

ity

posterior

prior

Figure 4.10: Series of plot generated by the function plot for the sequential design on M2.

4.3 Conclusion

In conclusion, CaliCo is a package that deals with Bayesian calibration through four main functions. For anindustrial numerical code, every specific cases is covered by CaliCo (if the user has a DOE or not, with a numericalcode or not). The R6 classes used in the implementation makes the package more robust. Even if the class layeris not visible to the user, the standardized formulation allows a rigorous treatment. The multiple ggplot2 graphsavailable for each class allow the user to take advantage of the graphical display without any knowledge of ggplot2.The flexibility of ggplot2 enables also the user to modify the frame, scale, title, labels of the graphs really quickly.All the MCMC calls are implemented in C++, which reduces the time of these time-consuming algorithms. TheMetropolis within Gibbs algorithm provides a better learning of the covariance matrix that the Metropolis Hastingswill use in its proposition distribution. That improves the performance of the algorithms.

Many developments can be brought to the package. For example, statistical validation can be added to thepackage and permit the user to elect the best model according to the data. Based on Damblin et al. (2016) avalidation using the Bayes factor or mixture models can be implemented. The dependences on DiceKriging orDiceDesign can also be a weakness of the package. When too many dependencies are implemented, the chances tohave bad configuration also increase.

111

Cha

pter

5

CHAPTER5PERFORMANCE MONITORING ON A

LARGE PV PLANT

5.1 Sensitivity analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1135.2 Prior densities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1175.3 Propagation of uncertainties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1185.4 Bayesian calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118

5.4.1 Statistical models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1185.4.2 Modular estimation and likelihoods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1205.4.3 Application to the PV plant . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121

This chapter concerns the study of a PV plant built and maintained by EDF. The precise location is not given anddata are normalized for a confidentiality matter. Mainly, the numerical code, based on the one diode modeling(among the advanced physical models presented in Section 1.3.2) is considered in this chapter. Particularly, thenumerical code introduced in Section 1.4.4 is used to complete the study in a performance monitoring context.Based on actual production and meteorological data over one year, code calibration better assesses the uncertaintieson the parameters input of the Dymola code to this specific case. Once calibration has been run, the code with thenew parameter distributions, could predict better the power for a next period of time. The business plan initiallyconceived on the prior parameter distributions, can then be reviewed and the price of electricity adapted to thenew estimated power. This chapter presents first the numerical code and the sensitivity analysis performed onthe 26 parameters. The prior distributions of the parameters are also presented which leads to a propagation ofuncertainties a priori. Then calibration, in this particular case, is performed but the following issues differ fromprevious chapters. Indeed, the Dymola code outputs are time series and dealing with multidimensional output incalibration requires additional work.

5.1 Sensitivity analysis

The Dymola code, as presented in Section 1.4.4, is more accurate in the prediction of a large PV plant productionthan the code used in Chapter 3. Indeed, one of the limitations of the code used to compare calibration models,is to over estimate the power when shades appear. This case scenario occurs every days in a large PV plantconfiguration because when the sun comes up, the panel in the first raw automatically creates a shade on theone behind. However, because it adds implicit equations that need to be numerically solved, such a developmenthas the effect to slow down the code speed. Moreover, the code estimates the power of the PV plant over ayear and produces a time series of 15,858 points. Applying regular sensitivity analysis (as it was the case forthe methods presented in Section 2.1) is more complicated because it has to take into account the behavior ofthe code on the whole time frame and not only in the instantaneous time. To overcome these issues, the same

113

Cha

pter

5

Chapter 5 – Performance monitoring on a large PV plant

method as the one developed in Section 3.1.1 is used. A Morris method is applied and a PCA is performed onthe Morris trajectories to give a graphical representation of the indices. The results of the PCA are given in Figure 5.1.

−1.0 −0.5 0.0 0.5 1.0

−1.

0−

0.5

0.0

0.5

1.0

Variables factor map (PCA)

Dim 1 (90.50%)

Dim

2 (

6.06

%)

V1V2

V3V4

V5

V6V7V8V9V10V11V12V13V14

V15V16V17V18V19V20V21V22V23V24V25V26V27

V28V29V30V31V32V33V34V35

V36

V37

V38

V39

V40

V41

V42V43V44V45V46V47V48V49V50V51V52V53V54V55V56V57V58V59V60V61V62V63

V64V65

V66V67V68V69V70V71V72V73V74

V75V76V77

V78

V79

V80

V81V82

V83

V84V85V86V87V88V89V90V91

V92

V93V94V95V96V97V98V99

V100

V101

V102V103

V104

V105V106V107V108V109V110V111

V112V113V114V115V116V117

V118V119

V120

V121V122V123

V124V125

V126

V127V128V129V130

V131V132V133V134

V135V136V137

V138

V139V140V141

V142

V143V144

V145V146V147V148

V149V150V151V152V153V154V155V156

V157

V158

V159

V160V161V162V163V164V165V166V167V168V169V170V171V172V173V174V175V176V177V178V179V180V181V182V183V184V185V186V187V188V189V190V191V192V193V194V195

V196V197

V198

V199

V200

V201

V202

V203V204V205

V206

V207V208

V209

V210V211V212

V213V214

V215V216V217

V218V219V220V221V222V223V224V225V226V227V228V229V230

V231

V232

V233V234V235V236

V237

V238V239

V240

V241V242V243V244V245V246V247

V248V249V250V251

V252V253V254V255V256

V257V258V259V260

V261V262

V263

V264V265V266

V267V268V269V270V271

V272

V273V274

V275V276

V277

V278V279V280V281V282

V283

V284

V285V286V287V288

V289

V290

V291V292V293V294V295

V296

V297

V298V299

V300

V301

V302V303V304

V305V306V307V308

V309V310V311

V312V313V314

V315

V316V317V318

V319

V320

V321

V322

V323

V324

V325V326

V327

V328

V329

V330V331V332V333V334V335

V336

V337V338

V339V340V341

V342

V343V344V345

V346V347V348

V349

V350

V351V352V353V354V355V356V357

V358V359V360V361V362V363

V364V365V366

V367

V368

V369V370V371V372V373

V374

V375V376V377

V378V379

V380V381V382V383

V384

V385V386V387

V388V389

V390

V391

V392V393V394

V395

V396

V397V398

V399

V400V401

V402V403V404

V405

V406V407

V408

V409V410

V411V412V413

V414

V415

V416

V417V418V419V420V421

V422

V423

V424

V425

V426V427V428V429V430V431V432V433V434V435V436V437V438V439V440V441V442V443V444V445V446V447V448V449V450V451V452V453V454V455V456V457V458V459V460V461

V462

V463V464

V465

V466

V467V468V469V470V471V472V473V474V475V476V477V478V479V480V481V482V483V484V485V486V487V488V489V490V491V492V493V494V495V496V497V498V499V500V501V502

V503

V504

V505

V506

V507V508V509V510V511V512V513V514V515V516V517V518V519V520

V521V522

V523V524V525V526V527V528V529V530V531V532V533V534V535V536V537V538

V539

V540V541V542V543V544

V545

V546

V547

V548V549V550V551V552V553V554

V555

V556V557V558

V559V560V561

V562V563

V564

V565

V566

V567V568

V569

V570V571

V572

V573V574

V575V576V577

V578

V579

V580

V581V582

V583

V584V585V586V587

V588V589V590V591V592V593V594V595V596V597V598V599V600

V601V602V603V604V605

V606

V607

V608

V609V610

V611

V612V613

V614

V615

V616

V617

V618V619

V620V621

V622

V623

V624

V625

V626

V627V628

V629V630V631

V632V633V634

V635V636V637V638

V639

V640V641

V642V643V644V645V646V647V648V649V650V651V652V653V654V655V656V657V658

V659

V660

V661

V662

V663

V664V665V666V667V668V669V670V671V672V673V674V675V676V677V678V679V680V681V682V683V684V685V686V687V688V689V690V691V692V693V694V695V696V697V698V699

V700

V701

V702

V703V704

V705V706V707V708V709V710V711

V712V713

V714

V715V716

V717

V718V719V720V721V722

V723V724

V725

V726

V727

V728

V729V730V731V732

V733V734V735V736V737

V738

V739

V740V741

V742

V743

V744

V745V746V747V748V749V750V751V752V753V754V755V756V757V758V759V760V761V762V763V764V765V766V767V768V769V770V771V772V773V774V775V776V777V778V779V780V781

V782

V783

V784V785V786V787V788V789V790V791V792V793V794V795V796V797V798V799V800V801V802V803V804V805V806V807V808V809V810V811V812V813V814V815V816V817V818V819V820V821V822V823

V824V825V826V827V828V829V830V831V832V833V834V835V836V837V838V839V840V841V842V843V844V845V846V847V848V849V850V851V852V853V854V855V856V857V858V859V860V861V862V863

V864V865V866V867V868V869V870V871V872V873V874V875V876V877V878V879V880V881V882V883V884V885V886V887V888V889V890V891V892V893V894V895V896V897V898V899V900V901

V902V903

V904

V905V906

V907

V908

V909

V910V911V912V913V914V915V916V917V918V919V920V921V922V923V924V925V926V927V928V929V930V931V932V933V934V935V936V937V938V939V940V941V942V943

V944

V945

V946

V947

V948

V949V950

V951

V952V953V954V955V956V957V958V959V960V961V962V963V964V965V966

V967V968

V969V970

V971V972V973V974V975V976V977V978V979V980V981V982V983V984

V985

V986V987V988

V989V990V991V992

V993

V994V995V996V997V998V999V1000V1001

V1002V1003V1004V1005V1006V1007V1008V1009V1010V1011V1012V1013V1014

V1015V1016V1017V1018V1019V1020V1021V1022V1023

V1024

V1025V1026V1027

V1028V1029V1030V1031

V1032

V1033

V1034V1035V1036V1037V1038V1039V1040V1041V1042V1043V1044V1045

V1046V1047V1048V1049V1050V1051V1052V1053V1054V1055V1056V1057V1058V1059V1060V1061

V1062V1063

V1064V1065V1066V1067

V1068

V1069

V1070V1071V1072V1073V1074V1075V1076V1077V1078V1079

V1080V1081

V1082V1083

V1084V1085

V1086V1087

V1088V1089V1090

V1091

V1092V1093V1094V1095V1096V1097

V1098V1099V1100

V1101

V1102V1103

V1104V1105V1106V1107

V1108

V1109

V1110V1111

V1112V1113V1114

V1115V1116V1117V1118V1119V1120V1121V1122V1123V1124V1125V1126V1127V1128V1129V1130V1131V1132V1133V1134V1135V1136V1137V1138V1139V1140V1141V1142V1143V1144V1145V1146V1147

V1148

V1149

V1150V1151

V1152

V1153V1154V1155V1156V1157V1158V1159V1160V1161V1162V1163V1164V1165V1166V1167V1168V1169V1170V1171V1172V1173V1174V1175V1176V1177V1178V1179V1180V1181V1182V1183V1184V1185V1186V1187V1188V1189

V1190

V1191

V1192V1193

V1194

V1195V1196V1197V1198V1199V1200V1201V1202V1203V1204V1205V1206V1207V1208V1209V1210V1211V1212V1213V1214V1215V1216V1217V1218V1219V1220V1221V1222V1223V1224V1225V1226V1227V1228V1229V1230V1231V1232

V1233

V1234V1235

V1236

V1237V1238V1239V1240V1241V1242V1243V1244V1245V1246V1247V1248V1249V1250V1251V1252V1253V1254V1255V1256V1257V1258V1259V1260V1261V1262V1263V1264V1265V1266V1267V1268V1269V1270V1271V1272V1273V1274

V1275

V1276V1277V1278

V1279

V1280V1281V1282V1283V1284V1285V1286V1287V1288V1289

V1290V1291V1292V1293V1294V1295V1296V1297V1298V1299V1300V1301V1302V1303V1304V1305V1306V1307V1308V1309V1310V1311V1312V1313V1314V1315V1316V1317

V1318

V1319V1320

V1321

V1322V1323V1324V1325V1326V1327V1328V1329V1330V1331V1332V1333V1334V1335V1336V1337V1338V1339V1340V1341V1342V1343V1344V1345V1346V1347V1348

V1349V1350V1351V1352V1353V1354V1355V1356V1357V1358

V1359

V1360

V1361

V1362V1363

V1364

V1365V1366V1367V1368V1369V1370V1371V1372V1373V1374V1375V1376V1377V1378V1379V1380V1381V1382

V1383

V1384

V1385V1386V1387V1388

V1389V1390V1391V1392V1393V1394V1395V1396V1397V1398V1399V1400V1401

V1402

V1403

V1404

V1405V1406

V1407

V1408V1409V1410V1411V1412V1413V1414V1415V1416V1417V1418V1419V1420V1421V1422V1423V1424V1425V1426V1427V1428V1429V1430V1431V1432V1433V1434V1435V1436V1437V1438V1439V1440V1441V1442V1443V1444V1445

V1446

V1447

V1448V1449

V1450

V1451

V1452V1453V1454V1455

V1456V1457V1458V1459V1460V1461

V1462V1463V1464V1465

V1466V1467V1468

V1469

V1470V1471

V1472V1473V1474V1475V1476V1477

V1478V1479

V1480

V1481

V1482V1483

V1484

V1485

V1486V1487

V1488

V1489

V1490

V1491

V1492

V1493V1494V1495V1496V1497V1498

V1499V1500

V1501V1502V1503V1504V1505V1506V1507V1508V1509V1510V1511

V1512

V1513V1514V1515

V1516V1517V1518V1519V1520V1521V1522V1523V1524V1525V1526V1527V1528

V1529

V1530V1531V1532

V1533V1534V1535

V1536

V1537V1538V1539V1540

V1541V1542

V1543

V1544V1545V1546V1547V1548V1549

V1550V1551

V1552V1553V1554V1555V1556

V1557V1558V1559V1560V1561V1562V1563

V1564

V1565

V1566

V1567V1568

V1569V1570V1571V1572

V1573V1574V1575V1576

V1577

V1578V1579V1580V1581V1582V1583V1584V1585V1586V1587V1588V1589V1590V1591V1592V1593V1594V1595V1596V1597V1598V1599V1600V1601V1602V1603V1604V1605V1606V1607V1608V1609V1610V1611V1612V1613V1614V1615

V1616

V1617

V1618

V1619

V1620

V1621V1622V1623V1624V1625V1626V1627V1628V1629V1630V1631V1632V1633V1634V1635V1636V1637V1638V1639V1640V1641V1642V1643V1644V1645V1646V1647V1648V1649V1650V1651V1652V1653V1654

V1655

V1656V1657V1658V1659

V1660V1661

V1662V1663

V1664

V1665V1666V1667V1668V1669V1670V1671V1672V1673V1674V1675V1676V1677V1678V1679V1680V1681V1682V1683V1684V1685V1686V1687V1688

V1689

V1690

V1691

V1692

V1693

V1694V1695

V1696

V1697

V1698V1699V1700V1701V1702 V1703

V1704

V1705V1706V1707V1708V1709V1710V1711V1712V1713V1714V1715V1716V1717V1718V1719V1720V1721V1722V1723V1724V1725V1726V1727

V1728V1729V1730V1731V1732V1733V1734V1735V1736V1737V1738V1739V1740V1741

V1742

V1743V1744

V1745

V1746

V1747

V1748V1749V1750V1751V1752V1753V1754V1755V1756V1757V1758V1759V1760V1761

V1762V1763V1764V1765V1766V1767V1768V1769V1770V1771V1772V1773

V1774

V1775

V1776

V1777V1778V1779V1780

V1781

V1782

V1783

V1784

V1785

V1786

V1787

V1788

V1789

V1790V1791V1792V1793V1794V1795V1796V1797V1798V1799V1800V1801V1802V1803V1804V1805V1806V1807

V1808V1809V1810

V1811V1812

V1813V1814

V1815

V1816V1817

V1818V1819V1820V1821V1822V1823V1824V1825V1826V1827V1828V1829

V1830

V1831V1832 V1833V1834V1835V1836V1837V1838V1839V1840V1841V1842V1843V1844V1845V1846V1847V1848V1849

V1850V1851V1852V1853V1854V1855V1856V1857V1858

V1859

V1860V1861

V1862V1863

V1864V1865V1866V1867V1868V1869V1870V1871V1872

V1873

V1874V1875

V1876

V1877

V1878V1879

V1880V1881

V1882

V1883V1884

V1885

V1886V1887V1888

V1889V1890V1891V1892

V1893V1894

V1895

V1896V1897

V1898V1899

V1900

V1901

V1902

V1903V1904V1905V1906

V1907V1908V1909V1910V1911V1912V1913V1914

V1915

V1916V1917

V1918V1919

V1920

V1921V1922V1923V1924V1925V1926V1927V1928V1929V1930V1931V1932

V1933V1934

V1935

V1936V1937

V1938

V1939V1940

V1941V1942

V1943V1944

V1945

V1946V1947V1948

V1949V1950V1951V1952V1953V1954

V1955V1956V1957

V1958

V1959

V1960

V1961

V1962V1963

V1964V1965V1966

V1967V1968V1969V1970V1971V1972V1973V1974V1975V1976

V1977

V1978

V1979

V1980V1981

V1982

V1983V1984V1985V1986V1987V1988

V1989

V1990V1991V1992V1993V1994V1995V1996V1997

V1998

V1999V2000

V2001V2002

V2003

V2004V2005

V2006V2007V2008V2009V2010

V2011

V2012V2013

V2014V2015

V2016V2017

V2018

V2019V2020V2021V2022V2023V2024V2025V2026V2027

V2028

V2029V2030V2031

V2032V2033

V2034

V2035V2036V2037

V2038

V2039

V2040V2041V2042V2043

V2044V2045V2046

V2047

V2048

V2049

V2050V2051V2052V2053V2054V2055V2056V2057

V2058

V2059

V2060

V2061

V2062 V2063

V2064

V2065

V2066

V2067

V2068V2069

V2070V2071

V2072V2073V2074V2075V2076V2077

V2078

V2079

V2080

V2081

V2082V2083

V2084V2085

V2086V2087

V2088V2089V2090

V2091

V2092V2093V2094

V2095

V2096

V2097V2098

V2099V2100

V2101

V2102V2103

V2104

V2105

V2106V2107

V2108

V2109

V2110

V2111

V2112

V2113

V2114V2115V2116V2117V2118

V2119V2120V2121V2122V2123V2124V2125

V2126

V2127

V2128V2129

V2130

V2131V2132V2133V2134V2135V2136V2137V2138V2139V2140V2141V2142V2143

V2144V2145V2146V2147V2148V2149V2150V2151V2152V2153V2154V2155

V2156V2157V2158V2159V2160V2161V2162

V2163V2164V2165V2166V2167

V2168

V2169V2170V2171

V2172

V2173

V2174

V2175V2176V2177

V2178V2179

V2180V2181V2182V2183V2184V2185V2186V2187V2188V2189V2190V2191V2192V2193V2194V2195V2196V2197V2198V2199V2200V2201V2202

V2203V2204V2205V2206V2207V2208V2209

V2210

V2211

V2212V2213

V2214V2215V2216V2217

V2218V2219V2220

V2221

V2222

V2223V2224V2225V2226V2227V2228

V2229

V2230

V2231

V2232V2233

V2234

V2235

V2236V2237V2238V2239V2240

V2241

V2242V2243

V2244

V2245V2246V2247V2248V2249V2250V2251

V2252

V2253

V2254

V2255

V2256

V2257

V2258V2259V2260

V2261

V2262

V2263

V2264

V2265V2266V2267V2268V2269V2270V2271V2272V2273V2274V2275V2276V2277V2278V2279V2280V2281

V2282V2283

V2284

V2285

V2286

V2287

V2288V2289V2290V2291V2292V2293V2294V2295V2296V2297V2298V2299V2300V2301V2302V2303

V2304

V2305

V2306

V2307

V2308

V2309V2310V2311V2312V2313V2314V2315V2316V2317V2318V2319V2320V2321V2322

V2323V2324V2325V2326V2327V2328V2329V2330V2331V2332V2333V2334V2335V2336V2337V2338V2339V2340V2341V2342V2343V2344V2345V2346V2347V2348V2349

V2350

V2351

V2352

V2353

V2354V2355V2356V2357V2358V2359V2360V2361V2362V2363V2364V2365V2366V2367V2368V2369V2370V2371V2372V2373V2374V2375V2376V2377V2378V2379V2380V2381

V2382

V2383V2384V2385V2386V2387V2388V2389V2390V2391V2392V2393

V2394

V2395

V2396

V2397

V2398

V2399

V2400V2401V2402V2403V2404V2405V2406V2407V2408

V2409V2410V2411V2412V2413

V2414V2415

V2416V2417V2418

V2419

V2420V2421

V2422V2423V2424V2425V2426V2427V2428V2429V2430V2431V2432V2433V2434V2435V2436V2437V2438

V2439

V2440

V2441V2442

V2443

V2444V2445V2446V2447V2448V2449V2450V2451V2452V2453V2454V2455V2456V2457V2458V2459V2460V2461V2462V2463V2464V2465V2466V2467V2468V2469V2470V2471V2472V2473V2474V2475V2476V2477V2478V2479V2480V2481V2482

V2483

V2484V2485

V2486

V2487

V2488

V2489V2490V2491V2492V2493V2494V2495V2496V2497V2498V2499

V2500

V2501

V2502

V2503

V2504V2505

V2506V2507V2508

V2509

V2510

V2511

V2512V2513V2514V2515V2516V2517V2518V2519V2520V2521V2522V2523

V2524V2525V2526

V2527V2528

V2529

V2530

V2531

V2532

V2533

V2534V2535V2536V2537V2538V2539V2540V2541V2542V2543V2544V2545V2546V2547V2548V2549V2550V2551V2552V2553V2554V2555V2556V2557V2558V2559V2560V2561V2562V2563V2564V2565V2566V2567V2568V2569V2570V2571V2572V2573

V2574

V2575V2576V2577

V2578

V2579

V2580

V2581

V2582

V2583V2584

V2585

V2586V2587V2588

V2589V2590V2591

V2592

V2593

V2594

V2595V2596

V2597V2598V2599V2600V2601V2602V2603

V2604V2605V2606V2607V2608V2609V2610V2611V2612V2613V2614V2615V2616

V2617

V2618

V2619V2620V2621

V2622

V2623V2624V2625V2626V2627V2628V2629V2630V2631V2632V2633V2634V2635V2636V2637V2638V2639V2640V2641V2642V2643V2644V2645V2646V2647V2648V2649V2650V2651V2652V2653V2654V2655V2656V2657V2658V2659V2660V2661V2662V2663

V2664

V2665

V2666V2667V2668

V2669

V2670

V2671V2672V2673V2674V2675V2676V2677V2678V2679V2680V2681V2682V2683V2684V2685V2686V2687V2688V2689V2690V2691V2692V2693V2694V2695V2696V2697V2698V2699V2700V2701V2702V2703V2704V2705V2706V2707V2708V2709V2710

V2711

V2712

V2713V2714

V2715

V2716

V2717V2718V2719V2720V2721V2722V2723V2724V2725V2726V2727V2728V2729V2730V2731V2732V2733V2734V2735V2736V2737V2738V2739V2740V2741V2742V2743V2744V2745V2746V2747V2748V2749V2750V2751V2752V2753V2754V2755V2756

V2757

V2758

V2759

V2760V2761

V2762

V2763V2764V2765V2766V2767V2768V2769V2770V2771V2772V2773V2774V2775V2776V2777V2778V2779V2780

V2781V2782V2783V2784V2785V2786V2787V2788V2789

V2790V2791V2792V2793V2794

V2795V2796V2797V2798V2799V2800V2801V2802

V2803

V2804

V2805

V2806

V2807

V2808V2809V2810V2811V2812

V2813

V2814V2815V2816V2817

V2818V2819V2820V2821V2822V2823V2824V2825V2826V2827V2828V2829

V2830

V2831V2832V2833

V2834V2835V2836V2837V2838

V2839V2840V2841V2842V2843V2844V2845

V2846

V2847

V2848

V2849V2850V2851

V2852

V2853

V2854

V2855V2856V2857V2858V2859V2860V2861V2862V2863V2864

V2865V2866

V2867

V2868

V2869

V2870V2871

V2872

V2873

V2874V2875

V2876

V2877V2878V2879V2880V2881V2882

V2883V2884V2885V2886V2887V2888V2889V2890V2891V2892V2893V2894V2895V2896

V2897V2898

V2899

V2900

V2901

V2902V2903V2904V2905V2906V2907V2908V2909V2910V2911

V2912V2913V2914

V2915

V2916V2917V2918

V2919

V2920

V2921

V2922V2923V2924

V2925V2926V2927V2928V2929V2930V2931V2932V2933V2934V2935V2936V2937V2938V2939V2940V2941V2942V2943

V2944

V2945

V2946V2947

V2948

V2949

V2950V2951V2952V2953V2954V2955V2956V2957V2958V2959V2960V2961V2962V2963V2964V2965V2966V2967V2968V2969V2970V2971V2972V2973V2974V2975V2976V2977V2978V2979V2980V2981V2982V2983V2984V2985V2986V2987V2988V2989V2990

V2991

V2992

V2993

V2994

V2995

V2996

V2997V2998V2999V3000V3001V3002V3003

V3004V3005V3006V3007

V3008

V3009V3010V3011V3012V3013V3014

V3015V3016V3017V3018V3019V3020V3021V3022V3023V3024

V3025V3026V3027V3028V3029V3030V3031V3032V3033V3034V3035V3036

V3037V3038V3039

V3040

V3041

V3042V3043V3044

V3045V3046V3047V3048V3049V3050V3051V3052V3053

V3054

V3055

V3056

V3057V3058V3059

V3060V3061

V3062

V3063V3064V3065

V3066

V3067

V3068V3069

V3070

V3071V3072V3073

V3074

V3075V3076V3077V3078V3079V3080V3081V3082V3083V3084

V3085V3086V3087

V3088V3089

V3090

V3091

V3092V3093V3094V3095V3096V3097V3098V3099V3100V3101V3102V3103V3104V3105V3106V3107

V3108V3109V3110V3111V3112V3113V3114V3115V3116V3117V3118V3119V3120V3121V3122

V3123

V3124V3125V3126V3127V3128V3129V3130

V3131

V3132V3133V3134V3135V3136

V3137V3138

V3139

V3140V3141V3142V3143

V3144

V3145

V3146V3147

V3148V3149V3150V3151V3152V3153V3154V3155V3156V3157V3158

V3159

V3160V3161V3162V3163V3164

V3165V3166

V3167

V3168

V3169V3170V3171

V3172

V3173V3174V3175V3176V3177V3178V3179

V3180V3181

V3182

V3183

V3184

V3185V3186V3187V3188

V3189

V3190V3191V3192V3193V3194V3195V3196V3197V3198V3199V3200V3201V3202V3203V3204V3205V3206V3207V3208V3209V3210V3211V3212V3213V3214V3215V3216V3217V3218V3219V3220V3221V3222V3223V3224V3225V3226V3227

V3228

V3229V3230

V3231

V3232

V3233V3234

V3235

V3236V3237V3238V3239

V3240V3241

V3242V3243

V3244

V3245V3246V3247V3248V3249

V3250V3251V3252V3253V3254V3255

V3256V3257V3258V3259V3260

V3261V3262V3263

V3264V3265V3266V3267V3268V3269V3270V3271V3272V3273V3274

V3275

V3276V3277V3278

V3279

V3280

V3281

V3282

V3283V3284V3285V3286V3287V3288V3289V3290V3291V3292

V3293

V3294

V3295V3296

V3297V3298V3299V3300

V3301

V3302V3303V3304

V3305

V3306V3307V3308V3309V3310

V3311V3312V3313V3314V3315V3316

V3317V3318V3319

V3320

V3321

V3322

V3323V3324

V3325

V3326

V3327

V3328

V3329V3330V3331V3332V3333V3334V3335V3336V3337V3338

V3339V3340V3341V3342V3343V3344

V3345V3346V3347V3348

V3349V3350V3351

V3352V3353V3354V3355V3356

V3357

V3358

V3359V3360

V3361

V3362

V3363

V3364

V3365

V3366

V3367V3368V3369V3370

V3371

V3372V3373

V3374

V3375V3376V3377V3378V3379V3380V3381V3382V3383V3384

V3385

V3386V3387V3388V3389V3390V3391

V3392

V3393V3394V3395V3396V3397V3398V3399V3400V3401V3402V3403V3404V3405V3406V3407V3408V3409V3410V3411V3412V3413V3414V3415

V3416V3417V3418

V3419

V3420

V3421

V3422V3423V3424V3425V3426V3427V3428V3429V3430V3431V3432V3433V3434V3435V3436V3437V3438V3439V3440V3441V3442V3443V3444V3445V3446V3447V3448V3449V3450V3451V3452V3453V3454V3455V3456V3457V3458V3459V3460V3461V3462

V3463

V3464V3465

V3466V3467V3468

V3469V3470

V3471V3472V3473V3474V3475V3476V3477V3478V3479V3480

V3481V3482

V3483

V3484V3485

V3486V3487

V3488V3489V3490

V3491V3492

V3493

V3494V3495V3496V3497V3498V3499V3500V3501V3502V3503V3504V3505V3506V3507V3508V3509V3510V3511V3512

V3513

V3514V3515V3516V3517

V3518

V3519V3520

V3521

V3522V3523V3524V3525

V3526V3527V3528V3529

V3530V3531

V3532V3533V3534

V3535

V3536V3537V3538V3539

V3540

V3541V3542V3543

V3544V3545V3546V3547V3548V3549V3550V3551V3552V3553V3554V3555V3556

V3557V3558V3559

V3560

V3561

V3562

V3563V3564

V3565

V3566

V3567V3568V3569V3570V3571V3572V3573V3574

V3575

V3576

V3577

V3578V3579V3580V3581V3582V3583V3584V3585V3586

V3587V3588V3589V3590

V3591V3592V3593V3594V3595V3596V3597V3598V3599

V3600V3601V3602V3603V3604V3605V3606V3607

V3608

V3609

V3610

V3611

V3612V3613

V3614V3615V3616

V3617V3618V3619V3620V3621V3622

V3623V3624

V3625

V3626V3627

V3628V3629V3630

V3631

V3632V3633

V3634

V3635V3636

V3637V3638V3639V3640V3641V3642V3643V3644V3645V3646V3647V3648

V3649V3650V3651V3652V3653V3654V3655V3656V3657V3658

V3659

V3660

V3661

V3662

V3663

V3664

V3665

V3666V3667V3668V3669V3670V3671

V3672

V3673

V3674

V3675V3676

V3677

V3678V3679V3680

V3681V3682V3683

V3684V3685

V3686V3687

V3688V3689V3690V3691V3692V3693V3694

V3695V3696V3697V3698V3699V3700V3701V3702V3703V3704V3705V3706V3707V3708

V3709

V3710

V3711V3712

V3713

V3714

V3715

V3716V3717V3718V3719V3720V3721V3722V3723V3724V3725V3726V3727V3728V3729V3730V3731V3732V3733V3734V3735V3736V3737V3738V3739V3740V3741V3742V3743V3744V3745V3746V3747V3748V3749V3750V3751V3752V3753V3754V3755V3756V3757V3758

V3759

V3760

V3761

V3762

V3763

V3764

V3765

V3766V3767V3768V3769V3770V3771V3772V3773V3774V3775V3776V3777V3778V3779V3780V3781V3782V3783V3784V3785V3786V3787V3788V3789V3790V3791V3792V3793V3794V3795V3796V3797V3798V3799V3800V3801V3802V3803V3804V3805V3806V3807V3808

V3809

V3810

V3811

V3812

V3813

V3814

V3815

V3816V3817V3818V3819V3820V3821V3822V3823V3824V3825V3826V3827V3828V3829V3830V3831V3832V3833V3834V3835V3836V3837V3838V3839V3840V3841V3842V3843V3844V3845V3846V3847V3848V3849V3850V3851V3852V3853V3854V3855V3856V3857V3858

V3859

V3860

V3861V3862

V3863

V3864

V3865

V3866V3867V3868V3869V3870V3871V3872V3873

V3874

V3875

V3876

V3877V3878V3879V3880

V3881V3882V3883V3884V3885V3886V3887V3888V3889V3890V3891V3892V3893

V3894V3895

V3896V3897

V3898

V3899V3900V3901V3902V3903V3904V3905V3906V3907V3908

V3909

V3910

V3911

V3912V3913V3914V3915

V3916

V3917V3918V3919V3920V3921V3922V3923V3924V3925V3926

V3927V3928

V3929V3930V3931V3932

V3933V3934

V3935V3936V3937

V3938

V3939V3940

V3941

V3942V3943V3944V3945V3946V3947V3948V3949V3950V3951V3952V3953V3954V3955V3956

V3957

V3958

V3959 V3960V3961

V3962

V3963V3964V3965V3966

V3967

V3968

V3969

V3970

V3971V3972V3973V3974

V3975

V3976

V3977V3978

V3979

V3980

V3981

V3982

V3983

V3984

V3985

V3986

V3987

V3988

V3989

V3990V3991V3992

V3993V3994V3995V3996V3997V3998V3999V4000V4001V4002V4003V4004

V4005V4006

V4007

V4008V4009

V4010V4011

V4012

V4013

V4014V4015V4016V4017

V4018V4019

V4020V4021

V4022V4023V4024V4025

V4026

V4027V4028

V4029

V4030V4031V4032

V4033V4034

V4035

V4036V4037V4038V4039V4040V4041V4042V4043V4044V4045V4046V4047V4048V4049V4050V4051

V4052

V4053V4054

V4055V4056

V4057V4058

V4059

V4060V4061V4062V4063V4064V4065

V4066V4067V4068V4069V4070V4071V4072V4073V4074

V4075V4076

V4077V4078V4079V4080V4081V4082V4083V4084V4085V4086V4087V4088V4089V4090V4091V4092V4093V4094V4095V4096V4097V4098V4099V4100V4101V4102

V4103

V4104V4105

V4106

V4107

V4108V4109

V4110

V4111V4112V4113V4114V4115V4116V4117V4118V4119V4120V4121V4122V4123V4124V4125V4126V4127V4128V4129V4130V4131V4132V4133V4134V4135V4136V4137V4138V4139V4140V4141V4142V4143V4144V4145V4146V4147V4148V4149V4150V4151V4152V4153

V4154

V4155V4156

V4157

V4158

V4159V4160

V4161V4162V4163V4164V4165V4166V4167V4168V4169V4170V4171V4172V4173V4174V4175V4176V4177V4178V4179V4180V4181V4182V4183V4184V4185V4186V4187V4188V4189V4190V4191V4192V4193V4194V4195V4196V4197V4198V4199V4200V4201V4202V4203

V4204

V4205V4206V4207

V4208

V4209V4210

V4211

V4212

V4213V4214V4215V4216V4217V4218V4219V4220V4221V4222V4223V4224V4225V4226V4227V4228V4229V4230V4231V4232V4233V4234V4235V4236V4237V4238V4239V4240V4241V4242V4243V4244V4245V4246V4247V4248V4249V4250V4251V4252V4253V4254V4255

V4256

V4257V4258

V4259V4260

V4261

V4262

V4263

V4264

V4265V4266V4267V4268V4269V4270V4271V4272V4273

V4274

V4275V4276

V4277V4278

V4279V4280

V4281V4282V4283V4284V4285V4286V4287V4288V4289V4290V4291V4292V4293V4294V4295V4296V4297V4298V4299

V4300V4301V4302V4303V4304V4305V4306

V4307

V4308V4309V4310

V4311V4312

V4313V4314

V4315

V4316V4317V4318V4319V4320V4321V4322V4323V4324V4325V4326V4327V4328

V4329V4330

V4331V4332V4333

V4334V4335

V4336V4337V4338V4339V4340V4341

V4342

V4343

V4344V4345V4346V4347V4348

V4349

V4350V4351

V4352V4353V4354V4355V4356V4357V4358

V4359

V4360V4361

V4362

V4363V4364

V4365

V4366V4367V4368V4369V4370V4371V4372V4373V4374

V4375V4376V4377V4378

V4379V4380V4381V4382

V4383V4384

V4385V4386V4387

V4388V4389V4390V4391

V4392V4393

V4394

V4395

V4396

V4397

V4398

V4399

V4400V4401V4402V4403V4404V4405V4406V4407V4408V4409

V4410

V4411

V4412

V4413V4414

V4415V4416

V4417

V4418V4419V4420V4421V4422V4423V4424V4425V4426V4427V4428V4429V4430V4431V4432V4433V4434V4435V4436V4437V4438V4439V4440V4441V4442V4443V4444V4445V4446V4447V4448V4449V4450V4451V4452V4453V4454V4455V4456V4457V4458V4459V4460V4461

V4462

V4463V4464

V4465V4466

V4467V4468

V4469

V4470

V4471V4472V4473V4474V4475V4476V4477V4478V4479V4480V4481V4482V4483V4484V4485V4486V4487V4488V4489V4490V4491V4492V4493V4494V4495V4496V4497V4498V4499V4500V4501V4502V4503V4504V4505V4506V4507V4508V4509V4510V4511V4512V4513

V4514

V4515V4516

V4517

V4518V4519

V4520V4521

V4522

V4523V4524V4525V4526V4527V4528V4529V4530V4531V4532V4533V4534V4535V4536V4537V4538V4539V4540V4541V4542V4543V4544V4545V4546V4547V4548V4549V4550V4551V4552V4553V4554V4555V4556V4557V4558V4559V4560V4561V4562V4563V4564V4565V4566

V4567

V4568V4569V4570

V4571

V4572

V4573V4574

V4575

V4576V4577V4578V4579V4580V4581V4582V4583V4584V4585V4586V4587V4588V4589V4590V4591V4592V4593V4594V4595V4596V4597V4598V4599V4600V4601V4602V4603V4604V4605V4606V4607V4608V4609V4610V4611V4612V4613V4614V4615V4616V4617V4618V4619

V4620

V4621V4622V4623

V4624

V4625

V4626V4627

V4628

V4629V4630V4631V4632V4633V4634V4635V4636V4637V4638V4639V4640V4641V4642V4643V4644V4645V4646V4647V4648V4649V4650V4651V4652V4653V4654V4655V4656V4657V4658V4659V4660V4661V4662V4663V4664V4665V4666V4667V4668V4669V4670V4671V4672

V4673

V4674V4675V4676V4677

V4678V4679V4680

V4681

V4682

V4683

V4684V4685V4686

V4687V4688V4689

V4690V4691V4692V4693V4694V4695V4696V4697V4698V4699V4700V4701V4702V4703V4704V4705V4706V4707V4708V4709V4710V4711V4712V4713V4714V4715V4716V4717V4718V4719V4720V4721V4722V4723V4724

V4725

V4726V4727

V4728

V4729

V4730

V4731V4732

V4733

V4734V4735V4736V4737V4738V4739

V4740V4741

V4742

V4743

V4744V4745V4746V4747V4748V4749V4750V4751V4752V4753V4754V4755V4756V4757V4758V4759V4760V4761V4762V4763V4764V4765V4766V4767V4768V4769V4770V4771V4772V4773V4774V4775V4776V4777

V4778

V4779V4780

V4781V4782

V4783V4784V4785

V4786

V4787V4788V4789V4790V4791V4792V4793V4794

V4795V4796

V4797

V4798V4799

V4800V4801V4802

V4803V4804V4805V4806V4807V4808V4809V4810V4811V4812V4813V4814V4815V4816V4817V4818V4819V4820V4821V4822V4823V4824V4825V4826

V4827V4828V4829V4830

V4831

V4832

V4833V4834

V4835

V4836V4837V4838

V4839

V4840V4841V4842V4843V4844V4845V4846

V4847V4848V4849V4850V4851V4852V4853V4854V4855V4856V4857V4858V4859V4860V4861V4862V4863V4864V4865V4866V4867V4868V4869V4870V4871V4872V4873V4874V4875V4876V4877V4878V4879V4880V4881V4882V4883

V4884

V4885V4886

V4887

V4888

V4889V4890

V4891

V4892

V4893V4894V4895V4896V4897

V4898V4899V4900

V4901V4902V4903

V4904V4905V4906V4907V4908

V4909V4910

V4911V4912

V4913V4914V4915V4916V4917V4918V4919V4920V4921V4922V4923V4924V4925V4926V4927V4928V4929V4930V4931V4932V4933V4934V4935V4936

V4937

V4938

V4939V4940

V4941

V4942

V4943V4944

V4945

V4946V4947V4948V4949V4950V4951V4952V4953V4954V4955V4956V4957V4958V4959V4960V4961V4962V4963V4964V4965V4966

V4967V4968V4969V4970V4971V4972V4973V4974V4975V4976V4977V4978V4979V4980V4981

V4982V4983V4984V4985V4986V4987V4988V4989

V4990

V4991

V4992V4993

V4994

V4995V4996

V4997

V4998

V4999V5000V5001V5002V5003V5004V5005V5006V5007V5008V5009V5010V5011V5012V5013V5014V5015V5016V5017V5018V5019V5020V5021V5022V5023V5024V5025V5026V5027V5028V5029

V5030V5031V5032V5033V5034V5035V5036V5037V5038V5039V5040V5041V5042

V5043V5044

V5045V5046V5047

V5048

V5049V5050V5051V5052V5053V5054V5055V5056V5057V5058V5059V5060V5061V5062V5063V5064V5065V5066V5067V5068V5069V5070V5071V5072V5073V5074V5075V5076V5077V5078V5079V5080V5081V5082V5083V5084V5085V5086V5087V5088V5089V5090V5091V5092

V5093

V5094V5095

V5096

V5097V5098

V5099

V5100

V5101V5102V5103V5104V5105V5106V5107V5108V5109V5110V5111V5112V5113V5114V5115V5116V5117V5118V5119V5120V5121V5122V5123V5124V5125V5126V5127V5128V5129V5130V5131V5132V5133V5134V5135V5136V5137V5138V5139V5140V5141V5142V5143V5144

V5145

V5146V5147

V5148

V5149

V5150V5151

V5152

V5153

V5154V5155V5156V5157V5158V5159V5160V5161V5162V5163V5164V5165V5166V5167V5168V5169V5170V5171V5172V5173V5174V5175V5176V5177V5178V5179V5180V5181V5182V5183V5184V5185V5186V5187V5188V5189V5190V5191V5192V5193V5194V5195V5196V5197

V5198

V5199V5200V5201V5202V5203V5204

V5205

V5206

V5207

V5208V5209V5210V5211V5212V5213V5214

V5215V5216V5217V5218

V5219V5220V5221V5222V5223V5224

V5225V5226V5227V5228

V5229V5230V5231

V5232V5233

V5234V5235V5236V5237V5238V5239V5240V5241V5242V5243V5244V5245V5246V5247V5248V5249

V5250

V5251V5252V5253

V5254

V5255V5256

V5257

V5258

V5259V5260V5261V5262V5263V5264V5265

V5266V5267

V5268

V5269

V5270V5271V5272V5273

V5274

V5275

V5276V5277V5278V5279

V5280V5281V5282

V5283

V5284

V5285

V5286V5287V5288V5289V5290V5291

V5292V5293V5294V5295V5296V5297V5298

V5299

V5300

V5301

V5302

V5303V5304V5305

V5306

V5307

V5308V5309V5310

V5311

V5312V5313V5314V5315V5316V5317V5318V5319V5320V5321V5322V5323V5324V5325V5326V5327V5328V5329V5330V5331V5332V5333V5334V5335V5336V5337V5338V5339V5340V5341V5342V5343V5344V5345V5346V5347V5348V5349

V5350V5351V5352V5353V5354V5355

V5356

V5357V5358

V5359

V5360

V5361V5362

V5363

V5364

V5365V5366V5367V5368V5369V5370V5371V5372V5373V5374V5375V5376V5377V5378V5379V5380V5381V5382V5383V5384V5385V5386V5387V5388V5389V5390V5391V5392V5393V5394V5395V5396V5397V5398V5399V5400V5401V5402V5403V5404V5405V5406V5407V5408

V5409

V5410V5411

V5412

V5413

V5414V5415

V5416V5417

V5418V5419V5420V5421V5422V5423V5424

V5425V5426V5427V5428V5429V5430V5431V5432V5433V5434V5435V5436

V5437V5438V5439V5440V5441V5442V5443V5444V5445V5446V5447V5448V5449V5450V5451V5452V5453V5454V5455V5456V5457V5458V5459V5460V5461

V5462

V5463V5464V5465

V5466

V5467

V5468V5469V5470

V5471

V5472V5473V5474V5475V5476V5477V5478V5479V5480V5481V5482V5483V5484V5485V5486V5487V5488V5489V5490V5491V5492V5493V5494V5495V5496V5497V5498V5499V5500V5501V5502V5503V5504V5505V5506V5507V5508V5509V5510V5511V5512V5513V5514V5515

V5516

V5517V5518V5519

V5520V5521

V5522V5523

V5524V5525V5526V5527V5528V5529V5530V5531V5532V5533V5534V5535V5536V5537V5538V5539V5540V5541V5542V5543V5544V5545V5546V5547V5548V5549V5550V5551V5552V5553V5554V5555V5556V5557V5558V5559V5560V5561V5562V5563V5564V5565V5566V5567V5568V5569

V5570

V5571V5572V5573

V5574

V5575

V5576V5577V5578

V5579

V5580V5581V5582V5583V5584V5585V5586V5587V5588V5589V5590V5591V5592V5593V5594V5595V5596V5597V5598V5599V5600V5601V5602V5603V5604V5605V5606V5607V5608V5609V5610V5611V5612V5613V5614V5615V5616V5617V5618V5619V5620V5621V5622V5623

V5624

V5625V5626V5627

V5628

V5629V5630V5631

V5632

V5633

V5634V5635V5636V5637V5638V5639V5640V5641V5642V5643V5644V5645V5646V5647V5648V5649V5650V5651V5652V5653V5654V5655V5656V5657V5658V5659V5660V5661V5662V5663V5664V5665V5666V5667V5668V5669V5670V5671V5672V5673V5674V5675V5676V5677

V5678

V5679V5680V5681V5682

V5683

V5684V5685V5686

V5687V5688V5689V5690V5691V5692V5693V5694V5695V5696V5697V5698V5699V5700V5701V5702V5703V5704V5705V5706V5707V5708V5709V5710V5711V5712V5713V5714V5715V5716V5717V5718V5719V5720V5721V5722V5723V5724V5725V5726V5727V5728V5729V5730V5731

V5732

V5733

V5734V5735

V5736V5737

V5738

V5739V5740V5741

V5742V5743V5744V5745V5746V5747V5748V5749

V5750V5751V5752V5753V5754V5755

V5756V5757

V5758V5759V5760V5761V5762V5763

V5764

V5765V5766V5767V5768

V5769

V5770

V5771V5772V5773V5774V5775V5776V5777V5778V5779V5780V5781V5782V5783V5784V5785V5786

V5787

V5788V5789V5790V5791

V5792V5793

V5794

V5795V5796

V5797V5798V5799V5800V5801V5802V5803V5804

V5805V5806V5807V5808V5809

V5810

V5811V5812

V5813V5814

V5815V5816V5817V5818

V5819

V5820

V5821V5822V5823V5824V5825V5826

V5827V5828V5829V5830V5831

V5832V5833

V5834V5835V5836V5837

V5838V5839V5840V5841

V5842

V5843V5844V5845

V5846

V5847

V5848V5849V5850

V5851

V5852V5853V5854V5855V5856V5857V5858V5859V5860

V5861V5862V5863

V5864V5865V5866V5867V5868V5869V5870V5871

V5872V5873V5874V5875V5876V5877V5878V5879V5880V5881V5882V5883V5884V5885V5886V5887V5888V5889V5890V5891V5892V5893V5894V5895V5896V5897

V5898

V5899V5900V5901

V5902

V5903V5904V5905

V5906

V5907V5908V5909V5910V5911V5912V5913

V5914

V5915V5916V5917V5918V5919V5920V5921V5922

V5923V5924

V5925V5926V5927

V5928

V5929V5930V5931V5932V5933V5934

V5935V5936V5937V5938V5939V5940V5941V5942V5943V5944V5945V5946V5947V5948V5949V5950V5951V5952

V5953

V5954V5955V5956

V5957

V5958V5959V5960

V5961

V5962V5963V5964V5965V5966V5967V5968V5969V5970V5971V5972V5973V5974V5975V5976V5977V5978V5979V5980V5981V5982V5983V5984V5985V5986V5987V5988V5989V5990V5991V5992V5993V5994V5995V5996V5997V5998V5999V6000V6001V6002V6003V6004V6005V6006

V6007

V6008V6009V6010V6011

V6012

V6013V6014

V6015

V6016

V6017

V6018V6019V6020V6021V6022V6023V6024V6025V6026V6027V6028V6029V6030V6031V6032V6033V6034V6035V6036V6037V6038V6039V6040

V6041V6042V6043

V6044V6045V6046V6047V6048

V6049V6050V6051

V6052V6053V6054

V6055V6056V6057V6058V6059V6060V6061V6062

V6063V6064V6065

V6066

V6067

V6068V6069

V6070V6071

V6072V6073V6074V6075V6076V6077V6078V6079V6080V6081V6082V6083V6084

V6085V6086

V6087V6088V6089V6090V6091V6092V6093V6094V6095V6096V6097V6098V6099V6100V6101V6102V6103V6104

V6105V6106V6107V6108V6109V6110V6111V6112V6113V6114V6115V6116

V6117

V6118V6119V6120

V6121V6122V6123V6124

V6125

V6126V6127V6128V6129V6130V6131V6132V6133V6134V6135V6136V6137V6138

V6139V6140V6141V6142V6143V6144V6145V6146V6147V6148V6149V6150V6151V6152V6153V6154V6155V6156V6157V6158

V6159V6160

V6161V6162V6163V6164V6165V6166V6167V6168V6169

V6170

V6171V6172V6173

V6174

V6175

V6176V6177

V6178V6179

V6180V6181V6182V6183V6184V6185V6186V6187V6188V6189V6190V6191V6192

V6193V6194V6195V6196V6197V6198V6199

V6200V6201

V6202V6203V6204V6205V6206V6207V6208V6209V6210V6211V6212V6213V6214V6215V6216V6217V6218V6219V6220V6221V6222V6223V6224V6225

V6226V6227

V6228V6229

V6230

V6231V6232

V6233V6234

V6235V6236V6237V6238V6239V6240V6241V6242V6243V6244V6245V6246V6247V6248V6249

V6250V6251V6252

V6253V6254V6255V6256V6257V6258V6259V6260V6261V6262V6263V6264V6265V6266V6267V6268V6269V6270V6271V6272V6273V6274V6275V6276V6277V6278V6279V6280

V6281V6282

V6283V6284

V6285

V6286V6287

V6288V6289

V6290V6291V6292V6293V6294V6295V6296V6297

V6298V6299V6300V6301V6302V6303V6304

V6305V6306V6307

V6308

V6309

V6310

V6311V6312V6313V6314V6315V6316V6317V6318V6319V6320V6321V6322V6323V6324V6325V6326V6327V6328V6329V6330V6331V6332V6333V6334V6335

V6336

V6337V6338V6339

V6340

V6341V6342V6343

V6344

V6345

V6346V6347V6348V6349V6350V6351V6352

V6353V6354V6355V6356V6357V6358V6359V6360V6361V6362V6363V6364V6365V6366V6367V6368V6369V6370V6371V6372V6373V6374V6375V6376V6377V6378V6379

V6380V6381

V6382V6383V6384V6385V6386

V6387

V6388

V6389V6390V6391V6392V6393V6394V6395V6396V6397

V6398

V6399V6400V6401V6402V6403

V6404V6405V6406V6407V6408V6409V6410

V6411V6412V6413V6414V6415V6416V6417V6418V6419V6420V6421V6422V6423V6424V6425V6426V6427V6428V6429V6430V6431V6432V6433V6434V6435V6436V6437V6438V6439V6440V6441V6442V6443V6444

V6445

V6446

V6447V6448

V6449

V6450V6451V6452

V6453

V6454V6455V6456V6457V6458V6459V6460V6461V6462V6463V6464V6465V6466V6467V6468V6469V6470V6471V6472V6473V6474V6475V6476V6477V6478V6479V6480V6481V6482V6483V6484V6485V6486V6487V6488V6489V6490V6491V6492V6493V6494V6495V6496V6497V6498V6499

V6500

V6501

V6502V6503

V6504

V6505V6506

V6507V6508

V6509V6510V6511V6512V6513V6514V6515V6516V6517V6518V6519V6520V6521V6522V6523V6524V6525V6526V6527V6528V6529V6530V6531V6532V6533V6534V6535V6536V6537V6538V6539V6540V6541V6542V6543V6544V6545V6546V6547V6548V6549V6550V6551V6552V6553V6554

V6555

V6556V6557V6558

V6559

V6560V6561V6562V6563

V6564

V6565V6566V6567V6568V6569V6570V6571

V6572V6573

V6574V6575

V6576

V6577

V6578V6579V6580

V6581V6582

V6583

V6584

V6585

V6586V6587V6588V6589V6590V6591V6592V6593V6594V6595V6596

V6597V6598V6599V6600V6601V6602V6603V6604V6605V6606

V6607V6608V6609

V6610

V6611

V6612V6613

V6614

V6615V6616V6617V6618V6619

V6620V6621V6622V6623V6624V6625V6626V6627V6628V6629V6630V6631V6632

V6633V6634V6635V6636V6637V6638V6639V6640V6641V6642V6643V6644V6645V6646V6647V6648V6649V6650

V6651V6652V6653V6654V6655

V6656V6657V6658V6659V6660V6661V6662V6663V6664V6665

V6666V6667V6668

V6669

V6670V6671V6672

V6673

V6674V6675V6676V6677V6678V6679V6680V6681V6682V6683V6684V6685V6686V6687V6688V6689V6690V6691V6692V6693V6694V6695V6696V6697V6698V6699V6700V6701V6702V6703V6704V6705V6706V6707V6708V6709V6710V6711V6712V6713V6714V6715V6716V6717V6718V6719

V6720

V6721V6722V6723

V6724

V6725V6726V6727

V6728

V6729V6730V6731V6732V6733V6734V6735V6736V6737V6738V6739V6740V6741V6742V6743V6744V6745V6746V6747V6748V6749V6750V6751V6752V6753V6754V6755

V6756V6757V6758V6759V6760V6761V6762V6763V6764V6765V6766V6767V6768V6769V6770V6771V6772V6773V6774

V6775

V6776V6777V6778

V6779V6780V6781

V6782

V6783

V6784V6785V6786V6787V6788V6789V6790V6791V6792V6793V6794V6795

V6796

V6797V6798V6799V6800V6801V6802V6803V6804V6805V6806V6807V6808V6809V6810V6811V6812V6813V6814V6815V6816V6817V6818V6819V6820V6821V6822V6823V6824V6825V6826V6827V6828

V6829

V6830V6831V6832

V6833

V6834 V6835

V6836V6837

V6838V6839V6840V6841V6842V6843V6844

V6845V6846

V6847

V6848V6849V6850V6851V6852V6853V6854

V6855V6856V6857V6858V6859V6860V6861V6862V6863V6864V6865V6866V6867V6868V6869V6870V6871V6872V6873

V6874V6875V6876V6877V6878V6879V6880V6881V6882V6883

V6884

V6885V6886V6887

V6888

V6889V6890V6891

V6892

V6893V6894V6895V6896V6897V6898V6899V6900

V6901

V6902

V6903V6904

V6905V6906

V6907

V6908

V6909V6910

V6911

V6912V6913

V6914V6915V6916

V6917V6918V6919V6920V6921V6922V6923V6924V6925V6926V6927V6928V6929V6930V6931V6932V6933V6934V6935V6936V6937V6938

V6939

V6940

V6941V6942

V6943

V6944

V6945V6946

V6947

V6948V6949V6950V6951V6952V6953V6954V6955

V6956V6957V6958

V6959V6960

V6961

V6962V6963V6964V6965V6966V6967

V6968V6969V6970V6971V6972V6973V6974V6975V6976V6977V6978V6979V6980V6981V6982V6983V6984V6985V6986V6987V6988V6989V6990V6991V6992V6993

V6994

V6995V6996V6997

V6998

V6999V7000V7001

V7002

V7003V7004V7005V7006V7007V7008V7009V7010

V7011V7012V7013

V7014

V7015V7016

V7017

V7018V7019V7020

V7021V7022V7023V7024V7025V7026V7027V7028V7029V7030V7031V7032V7033V7034V7035V7036V7037V7038V7039

V7040V7041V7042V7043V7044V7045V7046V7047V7048

V7049

V7050V7051V7052

V7053

V7054 V7055V7056

V7057

V7058

V7059V7060V7061V7062V7063V7064V7065V7066

V7067

V7068V7069

V7070

V7071V7072V7073

V7074

V7075V7076V7077

V7078V7079V7080V7081V7082

V7083V7084

V7085V7086

V7087

V7088V7089V7090V7091

V7092V7093V7094V7095V7096V7097V7098V7099V7100V7101V7102V7103

V7104

V7105V7106V7107

V7108

V7109 V7110V7111

V7112

V7113

V7114V7115V7116V7117V7118V7119V7120V7121V7122

V7123

V7124

V7125V7126V7127V7128V7129

V7130V7131V7132V7133V7134V7135V7136V7137V7138V7139V7140V7141V7142V7143V7144V7145V7146V7147V7148V7149V7150V7151V7152V7153V7154V7155V7156V7157V7158

V7159

V7160V7161

V7162

V7163

V7164V7165V7166

V7167

V7168V7169V7170V7171V7172V7173V7174

V7175V7176V7177

V7178V7179V7180V7181V7182V7183V7184V7185V7186V7187V7188V7189V7190V7191V7192V7193V7194V7195V7196V7197V7198V7199V7200V7201V7202V7203V7204V7205V7206V7207V7208V7209V7210V7211V7212V7213

V7214

V7215V7216V7217

V7218V7219

V7220V7221V7222

V7223

V7224V7225V7226V7227V7228V7229V7230V7231V7232V7233V7234V7235V7236V7237V7238V7239V7240V7241V7242V7243V7244V7245V7246V7247V7248V7249V7250V7251V7252V7253V7254V7255V7256V7257V7258V7259V7260V7261V7262V7263V7264V7265V7266V7267V7268V7269

V7270

V7271V7272V7273

V7274V7275

V7276V7277V7278

V7279

V7280V7281V7282V7283V7284V7285V7286V7287V7288V7289V7290V7291V7292V7293V7294V7295V7296V7297V7298V7299V7300V7301V7302V7303V7304V7305V7306V7307V7308V7309V7310V7311V7312V7313V7314V7315V7316V7317V7318V7319V7320V7321V7322V7323V7324V7325

V7326

V7327V7328V7329

V7330V7331

V7332V7333V7334

V7335

V7336

V7337V7338V7339V7340V7341V7342V7343V7344V7345V7346V7347V7348V7349V7350V7351V7352V7353V7354V7355V7356V7357V7358V7359V7360V7361V7362V7363V7364V7365V7366V7367V7368V7369V7370V7371V7372V7373V7374V7375V7376V7377V7378V7379V7380V7381

V7382

V7383V7384V7385V7386

V7387

V7388 V7389V7390

V7391

V7392

V7393V7394V7395V7396V7397V7398V7399V7400V7401

V7402V7403

V7404

V7405V7406V7407V7408

V7409

V7410V7411V7412V7413V7414V7415V7416V7417V7418V7419V7420V7421V7422V7423V7424V7425V7426V7427V7428V7429V7430V7431V7432V7433V7434V7435V7436V7437

V7438

V7439V7440V7441

V7442

V7443V7444V7445V7446

V7447V7448V7449V7450V7451V7452V7453V7454V7455

V7456

V7457V7458

V7459V7460

V7461

V7462V7463V7464V7465V7466V7467V7468V7469V7470V7471V7472V7473V7474V7475V7476V7477V7478V7479V7480V7481V7482V7483V7484V7485V7486V7487V7488V7489V7490

V7491V7492

V7493

V7494V7495V7496

V7497

V7498 V7499V7500

V7501

V7502V7503V7504V7505V7506V7507V7508V7509V7510V7511V7512V7513V7514V7515V7516V7517V7518

V7519V7520V7521V7522V7523V7524

V7525

V7526

V7527V7528

V7529V7530V7531V7532V7533V7534

V7535V7536V7537V7538V7539V7540V7541V7542V7543V7544V7545V7546V7547

V7548V7549V7550V7551V7552

V7553

V7554V7555V7556

V7557

V7558

V7559V7560V7561V7562V7563

V7564V7565V7566V7567

V7568V7569V7570

V7571

V7572

V7573V7574V7575

V7576

V7577

V7578V7579V7580

V7581V7582V7583V7584

V7585V7586V7587V7588V7589

V7590V7591V7592V7593V7594V7595V7596V7597V7598V7599V7600V7601V7602V7603

V7604

V7605V7606V7607V7608V7609V7610

V7611

V7612

V7613V7614V7615V7616V7617V7618V7619V7620V7621V7622V7623V7624V7625

V7626V7627V7628V7629V7630V7631V7632V7633V7634V7635

V7636V7637V7638V7639V7640V7641V7642V7643V7644V7645V7646V7647V7648V7649V7650V7651V7652V7653V7654V7655V7656V7657

V7658

V7659V7660V7661

V7662

V7663

V7664V7665V7666V7667

V7668

V7669V7670V7671V7672V7673V7674V7675V7676V7677V7678V7679V7680V7681V7682V7683V7684V7685V7686V7687V7688V7689V7690V7691V7692V7693V7694V7695V7696V7697V7698V7699V7700V7701V7702V7703V7704V7705V7706V7707V7708V7709V7710V7711V7712V7713

V7714

V7715V7716V7717V7718

V7719V7720V7721

V7722

V7723

V7724

V7725V7726V7727V7728V7729V7730V7731V7732

V7733V7734V7735V7736V7737

V7738V7739V7740V7741V7742V7743V7744V7745V7746V7747V7748V7749V7750V7751V7752V7753V7754V7755V7756V7757V7758V7759V7760V7761V7762V7763V7764V7765V7766

V7767

V7768V7769V7770

V7771

V7772 V7773 V7774

V7775

V7776

V7777V7778V7779V7780V7781V7782V7783V7784

V7785V7786V7787V7788

V7789

V7790V7791V7792V7793V7794V7795V7796V7797V7798V7799V7800V7801V7802V7803V7804V7805V7806V7807V7808V7809V7810V7811V7812V7813V7814V7815V7816V7817V7818V7819V7820V7821

V7822

V7823V7824

V7825

V7826V7827

V7828V7829V7830V7831

V7832

V7833V7834V7835V7836V7837V7838V7839V7840

V7841V7842V7843V7844V7845

V7846

V7847V7848V7849V7850V7851V7852V7853V7854V7855V7856V7857V7858V7859V7860V7861V7862V7863V7864V7865

V7866V7867V7868V7869V7870V7871V7872V7873V7874V7875V7876V7877

V7878

V7879V7880V7881V7882

V7883

V7884V7885V7886V7887

V7888

V7889V7890V7891V7892V7893V7894V7895V7896V7897V7898V7899V7900V7901V7902V7903V7904V7905V7906V7907V7908V7909V7910V7911V7912V7913V7914V7915V7916V7917V7918V7919V7920V7921V7922V7923V7924V7925V7926V7927V7928V7929V7930V7931V7932V7933

V7934

V7935V7936V7937V7938

V7939V7940V7941V7942

V7943

V7944V7945V7946V7947V7948V7949V7950V7951

V7952V7953V7954V7955V7956V7957V7958V7959V7960V7961V7962V7963V7964V7965V7966V7967V7968V7969V7970V7971V7972V7973V7974V7975V7976V7977V7978V7979V7980V7981V7982V7983V7984V7985V7986V7987V7988

V7989

V7990V7991V7992

V7993

V7994 V7995V7996

V7997

V7998

V7999V8000V8001V8002V8003V8004V8005V8006V8007V8008

V8009V8010V8011V8012V8013V8014V8015V8016V8017V8018V8019V8020V8021V8022V8023V8024V8025V8026V8027V8028V8029V8030V8031V8032V8033V8034V8035V8036V8037V8038V8039V8040V8041V8042V8043

V8044

V8045V8046V8047

V8048V8049

V8050V8051

V8052V8053

V8054

V8055V8056V8057V8058V8059V8060V8061V8062V8063V8064V8065V8066V8067V8068V8069V8070V8071V8072V8073V8074V8075V8076V8077V8078V8079V8080V8081V8082V8083V8084V8085V8086V8087V8088V8089V8090V8091V8092V8093V8094V8095V8096V8097V8098

V8099

V8100V8101V8102V8103

V8104

V8105V8106V8107

V8108

V8109

V8110V8111V8112V8113V8114V8115V8116V8117V8118V8119V8120V8121V8122V8123V8124V8125V8126V8127V8128V8129V8130V8131V8132V8133V8134V8135V8136V8137V8138V8139V8140V8141V8142V8143V8144V8145V8146V8147V8148V8149V8150V8151V8152V8153V8154

V8155

V8156V8157V8158

V8159

V8160

V8161V8162

V8163

V8164

V8165V8166V8167V8168V8169V8170V8171V8172V8173V8174V8175V8176V8177V8178V8179V8180V8181V8182V8183V8184V8185V8186V8187V8188V8189V8190V8191V8192V8193V8194V8195V8196V8197V8198V8199V8200V8201V8202V8203V8204V8205V8206V8207V8208V8209

V8210

V8211V8212V8213

V8214

V8215

V8216V8217V8218

V8219

V8220V8221V8222V8223V8224V8225V8226V8227

V8228V8229V8230V8231

V8232V8233V8234V8235V8236V8237V8238V8239V8240V8241V8242V8243V8244V8245V8246V8247V8248V8249V8250V8251V8252V8253V8254V8255V8256V8257V8258V8259V8260V8261V8262V8263V8264

V8265

V8266V8267V8268

V8269V8270

V8271V8272V8273

V8274

V8275V8276V8277V8278V8279

V8280V8281

V8282

V8283V8284V8285V8286V8287V8288V8289V8290V8291V8292V8293V8294V8295V8296V8297V8298V8299V8300V8301V8302V8303V8304V8305V8306V8307V8308V8309V8310V8311V8312V8313V8314V8315V8316V8317V8318V8319

V8320

V8321V8322V8323V8324V8325

V8326V8327V8328

V8329

V8330V8331V8332V8333V8334V8335V8336V8337V8338V8339V8340V8341V8342V8343V8344V8345V8346V8347V8348V8349V8350V8351V8352V8353V8354V8355V8356V8357V8358V8359V8360V8361V8362V8363V8364V8365V8366

V8367V8368V8369V8370V8371V8372V8373V8374

V8375

V8376V8377V8378V8379

V8380

V8381V8382

V8383

V8384

V8385V8386V8387V8388V8389V8390V8391V8392V8393V8394V8395V8396V8397V8398V8399V8400V8401V8402V8403V8404V8405V8406V8407V8408V8409V8410V8411V8412V8413V8414V8415V8416V8417V8418V8419V8420V8421V8422V8423V8424V8425V8426V8427V8428V8429

V8430

V8431V8432V8433V8434

V8435V8436V8437V8438

V8439

V8440

V8441V8442V8443V8444V8445V8446V8447V8448V8449

V8450

V8451

V8452V8453V8454V8455V8456V8457V8458V8459

V8460V8461V8462V8463V8464V8465

V8466V8467V8468V8469V8470V8471V8472V8473V8474V8475V8476

V8477V8478V8479V8480V8481V8482V8483V8484V8485

V8486V8487V8488V8489

V8490

V8491V8492V8493

V8494

V8495V8496V8497V8498V8499V8500V8501

V8502

V8503

V8504V8505V8506V8507

V8508

V8509

V8510

V8511V8512V8513V8514V8515V8516V8517V8518V8519V8520V8521V8522V8523V8524V8525V8526V8527V8528V8529V8530

V8531V8532V8533V8534V8535V8536V8537V8538V8539

V8540

V8541V8542V8543

V8544

V8545V8546V8547

V8548

V8549V8550V8551V8552V8553V8554V8555

V8556V8557

V8558V8559V8560V8561V8562V8563V8564V8565V8566V8567V8568V8569V8570V8571V8572V8573V8574V8575V8576V8577V8578V8579V8580V8581V8582V8583V8584V8585V8586V8587V8588V8589V8590V8591V8592V8593

V8594

V8595V8596V8597

V8598

V8599V8600

V8601

V8602

V8603V8604V8605V8606V8607V8608V8609V8610

V8611

V8612V8613V8614V8615V8616V8617V8618V8619V8620V8621V8622V8623V8624V8625V8626V8627V8628V8629V8630V8631V8632V8633V8634V8635V8636V8637V8638V8639V8640V8641V8642V8643V8644V8645V8646V8647

V8648

V8649V8650V8651

V8652V8653

V8654V8655V8656

V8657

V8658V8659V8660V8661V8662V8663V8664V8665

V8666V8667V8668V8669V8670V8671V8672V8673V8674V8675V8676V8677V8678V8679V8680V8681V8682V8683V8684V8685V8686V8687V8688V8689V8690V8691V8692V8693V8694V8695V8696V8697V8698V8699V8700V8701V8702

V8703

V8704V8705V8706

V8707

V8708V8709

V8710

V8711

V8712V8713V8714V8715V8716V8717V8718V8719V8720V8721V8722V8723V8724V8725V8726V8727V8728V8729V8730V8731V8732V8733V8734V8735V8736V8737V8738V8739V8740V8741V8742V8743V8744V8745V8746V8747V8748V8749V8750V8751V8752V8753V8754V8755V8756

V8757

V8758V8759

V8760

V8761V8762

V8763V8764

V8765

V8766

V8767V8768V8769V8770V8771V8772V8773V8774V8775V8776V8777V8778V8779V8780V8781V8782V8783V8784V8785V8786V8787V8788V8789V8790V8791V8792V8793V8794V8795V8796V8797V8798V8799V8800V8801V8802V8803V8804V8805V8806V8807V8808V8809V8810V8811

V8812

V8813V8814V8815

V8816V8817

V8818V8819

V8820

V8821V8822V8823V8824V8825

V8826V8827V8828

V8829V8830V8831V8832V8833V8834V8835V8836V8837V8838V8839V8840V8841V8842V8843V8844V8845V8846V8847V8848V8849V8850V8851V8852V8853V8854V8855V8856V8857V8858V8859V8860V8861V8862V8863V8864

V8865

V8866V8867

V8868

V8869

V8870V8871

V8872

V8873

V8874V8875V8876V8877V8878V8879V8880

V8881V8882V8883V8884V8885V8886V8887V8888V8889V8890V8891V8892V8893V8894V8895V8896V8897V8898V8899V8900V8901V8902V8903V8904V8905V8906V8907V8908V8909V8910V8911V8912V8913V8914V8915V8916V8917V8918

V8919

V8920V8921V8922

V8923

V8924V8925V8926

V8927

V8928V8929V8930V8931V8932V8933V8934V8935V8936V8937V8938V8939V8940V8941V8942V8943V8944V8945V8946V8947V8948V8949V8950V8951V8952V8953V8954V8955V8956V8957V8958V8959V8960V8961V8962V8963V8964V8965V8966V8967V8968V8969V8970V8971V8972

V8973

V8974V8975

V8976

V8977

V8978V8979V8980

V8981

V8982V8983V8984V8985V8986V8987V8988V8989V8990V8991V8992V8993V8994V8995V8996V8997V8998V8999V9000V9001V9002V9003V9004V9005V9006V9007V9008V9009V9010V9011V9012V9013V9014V9015V9016V9017V9018V9019V9020V9021V9022V9023V9024V9025V9026

V9027

V9028V9029V9030

V9031

V9032V9033V9034

V9035

V9036V9037V9038V9039V9040V9041V9042V9043V9044V9045V9046V9047V9048V9049V9050V9051V9052V9053V9054V9055V9056V9057V9058V9059V9060V9061V9062V9063V9064V9065V9066V9067V9068V9069V9070V9071V9072V9073V9074V9075V9076V9077V9078V9079V9080

V9081

V9082V9083V9084

V9085

V9086V9087V9088

V9089

V9090V9091V9092V9093V9094V9095V9096V9097V9098V9099V9100V9101V9102V9103V9104V9105V9106V9107V9108V9109V9110V9111V9112V9113V9114V9115V9116V9117V9118V9119V9120V9121V9122V9123V9124V9125V9126V9127V9128V9129V9130V9131V9132V9133V9134

V9135

V9136V9137

V9138

V9139

V9140V9141V9142

V9143

V9144V9145V9146V9147V9148V9149V9150V9151V9152V9153V9154V9155V9156V9157V9158V9159V9160V9161V9162V9163V9164V9165V9166V9167V9168V9169V9170V9171V9172V9173V9174V9175V9176V9177V9178V9179V9180V9181V9182V9183V9184V9185V9186V9187V9188

V9189

V9190V9191

V9192

V9193

V9194V9195V9196

V9197

V9198V9199V9200V9201V9202V9203V9204

V9205V9206V9207V9208V9209V9210V9211V9212V9213V9214V9215V9216V9217V9218V9219V9220V9221V9222V9223V9224V9225V9226V9227V9228V9229V9230V9231V9232V9233V9234V9235V9236V9237V9238V9239V9240V9241V9242

V9243

V9244V9245

V9246

V9247V9248V9249

V9250

V9251V9252V9253V9254V9255V9256V9257

V9258V9259V9260V9261V9262V9263V9264V9265V9266V9267V9268V9269V9270V9271V9272V9273V9274V9275V9276V9277V9278V9279V9280V9281V9282V9283V9284V9285V9286V9287V9288V9289V9290V9291V9292V9293V9294V9295

V9296

V9297V9298

V9299

V9300

V9301

V9302V9303

V9304

V9305V9306V9307V9308V9309V9310V9311V9312V9313V9314V9315V9316V9317V9318V9319V9320V9321V9322V9323V9324V9325V9326V9327V9328V9329V9330V9331V9332V9333V9334V9335V9336V9337V9338V9339V9340V9341V9342V9343V9344V9345V9346V9347V9348V9349

V9350

V9351V9352V9353

V9354

V9355V9356V9357

V9358

V9359V9360V9361V9362V9363V9364V9365V9366V9367V9368V9369V9370V9371V9372V9373V9374V9375V9376V9377V9378V9379V9380V9381V9382V9383V9384V9385V9386V9387V9388V9389V9390V9391V9392V9393V9394V9395V9396V9397V9398V9399V9400V9401V9402V9403

V9404

V9405V9406V9407

V9408V9409V9410

V9411

V9412V9413V9414V9415V9416V9417V9418V9419V9420V9421V9422V9423V9424V9425V9426V9427V9428V9429V9430V9431V9432V9433V9434V9435V9436V9437V9438V9439V9440V9441V9442V9443V9444V9445V9446V9447V9448V9449V9450V9451V9452V9453V9454V9455V9456

V9457

V9458V9459V9460

V9461V9462V9463

V9464

V9465V9466V9467V9468V9469V9470V9471V9472V9473V9474V9475V9476V9477V9478V9479V9480V9481V9482V9483V9484V9485V9486V9487V9488V9489V9490V9491V9492V9493V9494V9495V9496V9497V9498V9499V9500V9501V9502V9503V9504V9505V9506V9507V9508V9509

V9510

V9511V9512V9513V9514V9515V9516

V9517

V9518V9519V9520V9521V9522V9523V9524V9525V9526V9527V9528V9529V9530V9531V9532V9533V9534V9535V9536V9537V9538V9539V9540V9541V9542V9543V9544V9545V9546V9547V9548V9549V9550V9551V9552V9553V9554V9555V9556V9557V9558V9559V9560V9561

V9562

V9563V9564V9565V9566V9567V9568V9569V9570

V9571

V9572V9573V9574V9575V9576

V9577V9578V9579V9580V9581V9582V9583V9584V9585

V9586V9587V9588V9589V9590V9591V9592V9593V9594V9595V9596V9597V9598V9599V9600V9601V9602V9603V9604V9605V9606V9607V9608V9609V9610V9611V9612V9613V9614

V9615

V9616V9617V9618V9619V9620V9621

V9622

V9623

V9624V9625V9626V9627V9628V9629V9630V9631V9632V9633V9634V9635V9636V9637V9638V9639V9640V9641V9642V9643V9644V9645V9646V9647V9648V9649V9650V9651V9652V9653V9654V9655V9656V9657V9658V9659V9660V9661V9662V9663V9664V9665V9666V9667

V9668

V9669V9670V9671

V9672 V9673V9674

V9675

V9676V9677V9678V9679V9680V9681V9682

V9683V9684V9685V9686V9687V9688V9689V9690V9691V9692V9693V9694V9695V9696V9697V9698V9699V9700V9701V9702V9703V9704V9705V9706V9707V9708V9709V9710V9711V9712V9713V9714V9715V9716V9717V9718V9719V9720

V9721

V9722V9723

V9724V9725V9726

V9727

V9728V9729V9730V9731V9732V9733V9734V9735V9736V9737V9738V9739V9740V9741V9742V9743V9744V9745V9746V9747V9748V9749V9750V9751V9752V9753V9754V9755V9756V9757V9758V9759V9760V9761V9762V9763V9764V9765V9766V9767V9768V9769V9770V9771V9772

V9773

V9774V9775V9776

V9777 V9778V9779V9780

V9781V9782V9783V9784V9785V9786V9787V9788V9789V9790V9791V9792V9793V9794V9795V9796V9797V9798V9799V9800V9801V9802V9803V9804V9805V9806V9807V9808V9809V9810V9811V9812V9813V9814V9815V9816V9817V9818V9819V9820V9821V9822V9823V9824V9825

V9826

V9827V9828V9829V9830 V9831

V9832V9833

V9834V9835V9836V9837V9838V9839V9840V9841V9842V9843V9844V9845V9846V9847V9848V9849V9850V9851V9852V9853V9854V9855V9856V9857V9858V9859V9860V9861V9862V9863V9864V9865V9866V9867V9868V9869V9870V9871V9872V9873V9874V9875V9876V9877V9878

V9879

V9880V9881V9882V9883 V9884

V9885V9886

V9887V9888V9889V9890V9891V9892V9893V9894V9895V9896V9897V9898V9899V9900V9901V9902V9903V9904V9905V9906V9907V9908V9909V9910V9911V9912V9913V9914V9915V9916V9917V9918V9919V9920V9921V9922V9923V9924V9925V9926V9927V9928V9929V9930V9931

V9932

V9933V9934

V9935

V9936V9937V9938

V9939

V9940V9941V9942V9943V9944V9945V9946V9947V9948

V9949V9950V9951V9952V9953V9954V9955V9956V9957V9958V9959V9960V9961V9962V9963V9964V9965V9966V9967V9968V9969V9970V9971V9972V9973V9974V9975V9976V9977V9978V9979V9980V9981V9982V9983

V9984

V9985V9986

V9987

V9988V9989V9990

V9991

V9992V9993V9994V9995V9996V9997V9998V9999

V10000V10001V10002V10003

V10004

V10005V10006V10007

V10008

V10009V10010V10011V10012V10013V10014V10015V10016V10017V10018V10019V10020V10021V10022V10023V10024V10025

V10026V10027V10028V10029V10030V10031V10032V10033V10034V10035

V10036

V10037V10038

V10039

V10040V10041

V10042

V10043V10044V10045V10046V10047V10048V10049V10050V10051V10052

V10053V10054V10055V10056V10057V10058

V10059V10060V10061V10062V10063V10064V10065V10066V10067V10068V10069V10070V10071V10072V10073V10074V10075V10076V10077

V10078V10079V10080V10081V10082V10083V10084V10085V10086V10087

V10088

V10089V10090

V10091

V10092V10093V10094

V10095V10096V10097V10098V10099V10100V10101V10102V10103V10104V10105V10106V10107V10108V10109V10110V10111V10112V10113V10114V10115V10116V10117V10118V10119V10120V10121V10122V10123V10124V10125V10126V10127V10128V10129V10130V10131V10132V10133V10134V10135V10136V10137V10138V10139

V10140

V10141V10142

V10143

V10144V10145

V10146

V10147V10148V10149V10150V10151V10152V10153V10154V10155V10156V10157V10158V10159V10160V10161V10162V10163V10164V10165V10166V10167V10168V10169V10170V10171V10172V10173V10174V10175V10176V10177V10178V10179V10180V10181V10182V10183V10184V10185V10186V10187V10188V10189V10190V10191

V10192V10193V10194

V10195

V10196V10197V10198

V10199V10200V10201V10202V10203V10204V10205V10206V10207V10208V10209V10210V10211V10212V10213V10214V10215V10216V10217V10218V10219V10220V10221V10222V10223V10224V10225V10226V10227V10228V10229V10230V10231V10232V10233V10234V10235V10236V10237V10238V10239V10240V10241V10242

V10243

V10244V10245V10246

V10247

V10248V10249V10250

V10251

V10252V10253V10254V10255V10256V10257V10258V10259V10260V10261V10262V10263V10264V10265V10266V10267V10268V10269V10270V10271V10272V10273V10274V10275V10276V10277V10278V10279V10280V10281V10282V10283V10284V10285V10286V10287V10288V10289V10290V10291V10292V10293V10294

V10295

V10296V10297V10298

V10299

V10300V10301V10302

V10303

V10304V10305V10306V10307V10308V10309V10310

V10311V10312

V10313V10314V10315V10316V10317V10318V10319V10320V10321V10322V10323V10324V10325V10326V10327V10328V10329V10330V10331V10332V10333V10334V10335V10336V10337V10338V10339V10340V10341V10342V10343V10344V10345V10346

V10347

V10348V10349V10350

V10351

V10352V10353V10354

V10355

V10356V10357V10358V10359V10360V10361V10362V10363V10364V10365V10366V10367V10368V10369V10370V10371V10372V10373V10374V10375V10376V10377V10378V10379V10380V10381V10382V10383V10384V10385V10386V10387V10388V10389V10390V10391V10392V10393V10394V10395V10396V10397V10398

V10399

V10400V10401

V10402V10403

V10404V10405V10406

V10407

V10408V10409V10410V10411V10412V10413V10414

V10415V10416

V10417V10418V10419V10420V10421V10422

V10423V10424

V10425V10426V10427V10428V10429V10430V10431V10432V10433V10434V10435V10436V10437V10438V10439V10440V10441V10442V10443V10444V10445V10446V10447V10448V10449V10450

V10451

V10452V10453V10454

V10455

V10456V10457V10458

V10459

V10460V10461V10462V10463V10464V10465V10466V10467V10468V10469V10470

V10471V10472V10473

V10474

V10475V10476V10477V10478V10479V10480V10481V10482V10483V10484V10485V10486V10487V10488V10489V10490V10491V10492V10493V10494V10495V10496V10497V10498V10499V10500V10501V10502

V10503

V10504V10505V10506

V10507V10508

V10509V10510V10511

V10512

V10513V10514V10515V10516V10517V10518V10519V10520V10521

V10522V10523

V10524V10525V10526V10527

V10528V10529V10530V10531V10532V10533V10534V10535V10536V10537V10538V10539

V10540

V10541V10542

V10543

V10544

V10545V10546V10547V10548V10549V10550V10551V10552V10553V10554V10555

V10556

V10557V10558

V10559

V10560V10561V10562

V10563

V10564V10565V10566V10567V10568V10569V10570V10571V10572V10573V10574V10575

V10576

V10577V10578

V10579

V10580V10581

V10582V10583

V10584V10585

V10586V10587

V10588V10589V10590V10591V10592V10593V10594V10595V10596V10597V10598V10599V10600V10601V10602V10603V10604V10605V10606

V10607

V10608V10609

V10610

V10611V10612

V10613

V10614V10615V10616V10617V10618V10619V10620V10621V10622V10623

V10624

V10625

V10626V10627

V10628V10629

V10630V10631V10632

V10633

V10634V10635V10636V10637V10638V10639V10640V10641V10642V10643V10644V10645V10646

V10647V10648V10649V10650V10651V10652V10653V10654V10655V10656V10657V10658V10659V10660V10661V10662V10663V10664V10665V10666V10667V10668V10669V10670V10671V10672V10673V10674V10675V10676

V10677

V10678V10679

V10680V10681V10682

V10683

V10684V10685V10686V10687V10688V10689V10690V10691V10692V10693V10694V10695V10696V10697V10698V10699V10700V10701V10702V10703V10704V10705V10706V10707V10708V10709V10710V10711V10712V10713V10714V10715V10716V10717V10718V10719V10720V10721V10722V10723V10724V10725V10726

V10727

V10728V10729V10730

V10731V10732

V10733

V10734V10735V10736V10737V10738V10739

V10740

V10741V10742

V10743V10744V10745V10746

V10747

V10748V10749V10750V10751V10752V10753V10754V10755V10756V10757V10758V10759V10760V10761V10762V10763V10764V10765V10766V10767V10768V10769V10770V10771V10772V10773V10774V10775

V10776

V10777V10778V10779V10780 V10781V10782

V10783

V10784V10785V10786V10787V10788V10789V10790V10791V10792V10793V10794

V10795

V10796V10797V10798V10799V10800V10801V10802V10803V10804V10805V10806V10807V10808V10809V10810V10811V10812V10813V10814V10815V10816V10817V10818V10819V10820V10821V10822V10823V10824V10825V10826

V10827

V10828V10829V10830

V10831V10832

V10833

V10834

V10835V10836V10837V10838V10839V10840V10841V10842

V10843V10844V10845V10846V10847V10848

V10849

V10850

V10851V10852

V10853V10854

V10855V10856V10857V10858V10859V10860V10861V10862V10863V10864V10865

V10866V10867V10868V10869V10870V10871V10872V10873V10874V10875V10876

V10877

V10878

V10879

V10880V10881

V10882

V10883V10884V10885V10886V10887V10888

V10889

V10890

V10891V10892V10893V10894V10895V10896V10897V10898V10899V10900V10901

V10902V10903V10904V10905V10906V10907V10908V10909V10910V10911V10912V10913V10914V10915V10916V10917V10918V10919V10920V10921V10922V10923V10924V10925

V10926

V10927

V10928

V10929V10930

V10931

V10932V10933V10934V10935V10936V10937V10938

V10939V10940V10941V10942

V10943V10944V10945V10946V10947

V10948V10949V10950V10951V10952V10953V10954V10955V10956V10957V10958V10959V10960V10961V10962V10963V10964V10965V10966V10967V10968V10969V10970V10971V10972V10973

V10974

V10975V10976V10977

V10978

V10979V10980

V10981

V10982V10983V10984V10985V10986V10987V10988V10989V10990V10991

V10992V10993V10994

V10995V10996V10997V10998

V10999V11000V11001V11002V11003V11004V11005V11006V11007V11008V11009V11010V11011V11012

V11013V11014V11015

V11016V11017V11018V11019V11020V11021V11022V11023

V11024

V11025V11026

V11027

V11028V11029

V11030

V11031V11032V11033V11034V11035V11036V11037V11038V11039V11040V11041V11042V11043V11044

V11045V11046V11047V11048V11049V11050V11051V11052V11053V11054V11055V11056V11057V11058V11059V11060V11061V11062V11063V11064V11065V11066V11067V11068V11069V11070V11071V11072

V11073

V11074V11075

V11076

V11077V11078

V11079

V11080V11081V11082V11083V11084V11085V11086V11087V11088V11089V11090V11091V11092V11093V11094V11095V11096V11097V11098V11099V11100V11101V11102V11103V11104V11105V11106V11107V11108V11109V11110V11111V11112V11113V11114V11115V11116V11117V11118V11119V11120V11121

V11122

V11123V11124

V11125

V11126V11127

V11128

V11129V11130V11131V11132V11133V11134V11135V11136V11137V11138V11139V11140V11141V11142V11143V11144

V11145V11146V11147V11148V11149V11150V11151V11152V11153V11154V11155V11156V11157V11158V11159V11160V11161V11162V11163V11164V11165V11166V11167V11168V11169V11170

V11171

V11172V11173V11174V11175V11176V11177V11178

V11179

V11180

V11181V11182V11183V11184V11185V11186V11187V11188V11189V11190V11191V11192V11193V11194V11195V11196V11197V11198V11199V11200V11201V11202V11203V11204V11205V11206V11207V11208V11209V11210

V11211

V11212V11213V11214V11215V11216V11217V11218V11219V11220V11221

V11222

V11223V11224V11225V11226V11227V11228V11229V11230V11231V11232V11233V11234V11235V11236V11237V11238V11239V11240V11241V11242V11243V11244V11245V11246V11247V11248V11249V11250V11251V11252V11253V11254V11255V11256V11257V11258V11259

V11260

V11261V11262V11263V11264V11265V11266V11267

V11268

V11269V11270V11271V11272V11273V11274V11275V11276V11277V11278V11279V11280V11281V11282V11283V11284V11285V11286V11287V11288V11289V11290V11291V11292V11293V11294V11295V11296V11297V11298V11299V11300V11301V11302V11303V11304V11305V11306V11307

V11308

V11309V11310V11311V11312V11313

V11314

V11315V11316V11317V11318V11319V11320V11321V11322V11323V11324V11325V11326V11327V11328V11329V11330V11331V11332V11333V11334V11335V11336V11337V11338V11339V11340V11341V11342V11343V11344V11345V11346V11347V11348V11349V11350V11351V11352V11353V11354

V11355

V11356V11357V11358

V11359V11360

V11361

V11362V11363V11364V11365V11366V11367V11368V11369V11370V11371V11372V11373V11374V11375V11376V11377V11378

V11379

V11380V11381V11382V11383V11384V11385V11386V11387V11388V11389V11390V11391V11392V11393V11394V11395V11396V11397V11398V11399V11400V11401

V11402

V11403V11404

V11405

V11406V11407V11408

V11409V11410V11411V11412V11413V11414V11415

V11416V11417V11418V11419V11420V11421V11422V11423V11424V11425V11426V11427V11428V11429V11430V11431V11432V11433V11434V11435V11436V11437V11438V11439

V11440V11441V11442V11443V11444V11445V11446V11447V11448V11449

V11450

V11451V11452

V11453V11454

V11455

V11456V11457V11458V11459V11460V11461V11462V11463

V11464

V11465V11466V11467V11468V11469V11470V11471V11472V11473V11474V11475V11476

V11477

V11478V11479V11480

V11481

V11482

V11483V11484V11485V11486V11487V11488V11489V11490V11491V11492V11493V11494V11495V11496

V11497

V11498V11499

V11500V11501

V11502

V11503

V11504V11505V11506V11507V11508V11509V11510V11511V11512V11513V11514V11515

V11516V11517V11518V11519V11520V11521V11522V11523V11524V11525V11526V11527V11528V11529V11530V11531V11532V11533V11534V11535V11536V11537V11538V11539V11540V11541V11542V11543

V11544

V11545V11546V11547

V11548

V11549

V11550V11551V11552V11553V11554V11555V11556V11557V11558V11559V11560V11561

V11562

V11563V11564V11565V11566

V11567V11568

V11569

V11570

V11571V11572V11573V11574

V11575V11576

V11577V11578

V11579V11580V11581V11582V11583V11584V11585V11586V11587V11588V11589V11590V11591

V11592

V11593

V11594

V11595

V11596

V11597V11598V11599V11600V11601V11602V11603V11604V11605V11606V11607V11608V11609V11610V11611V11612V11613V11614V11615V11616V11617V11618V11619V11620V11621V11622V11623V11624V11625V11626V11627V11628V11629V11630V11631V11632V11633V11634V11635V11636V11637V11638

V11639

V11640

V11641

V11642V11643V11644

V11645

V11646V11647V11648V11649V11650

V11651V11652

V11653V11654V11655

V11656

V11657V11658V11659V11660V11661

V11662V11663V11664V11665V11666V11667V11668V11669V11670V11671V11672V11673V11674V11675V11676V11677V11678V11679V11680V11681V11682V11683V11684V11685

V11686

V11687V11688

V11689

V11690

V11691

V11692

V11693V11694V11695V11696V11697V11698V11699V11700V11701V11702V11703V11704V11705V11706V11707V11708V11709V11710V11711V11712V11713V11714V11715V11716V11717V11718V11719V11720V11721V11722V11723V11724V11725V11726V11727V11728V11729V11730V11731V11732V11733V11734

V11735

V11736

V11737

V11738

V11739

V11740

V11741V11742V11743V11744V11745V11746V11747V11748V11749V11750V11751V11752V11753V11754V11755V11756V11757V11758V11759V11760V11761V11762V11763V11764V11765V11766V11767V11768V11769V11770V11771V11772V11773V11774V11775V11776V11777V11778V11779V11780V11781

V11782

V11783V11784V11785

V11786

V11787

V11788V11789V11790V11791V11792V11793V11794V11795V11796V11797V11798V11799V11800V11801V11802V11803V11804V11805V11806V11807V11808V11809V11810V11811V11812V11813V11814V11815

V11816

V11817

V11818V11819V11820

V11821V11822V11823V11824V11825V11826V11827

V11828

V11829V11830V11831

V11832

V11833

V11834

V11835

V11836V11837V11838V11839V11840V11841V11842V11843V11844V11845V11846V11847V11848V11849V11850V11851V11852V11853V11854V11855V11856V11857V11858V11859V11860V11861V11862V11863V11864V11865V11866V11867V11868V11869V11870V11871V11872V11873V11874V11875V11876

V11877

V11878

V11879

V11880

V11881

V11882

V11883V11884V11885V11886V11887V11888V11889V11890V11891V11892V11893V11894V11895V11896V11897V11898V11899V11900V11901V11902V11903V11904V11905V11906V11907V11908V11909V11910V11911V11912V11913V11914V11915V11916V11917V11918V11919V11920V11921V11922V11923

V11924

V11925V11926

V11927V11928

V11929

V11930

V11931V11932V11933V11934V11935V11936V11937V11938V11939V11940V11941V11942

V11943V11944V11945V11946V11947V11948V11949V11950V11951V11952V11953V11954V11955V11956V11957V11958V11959V11960V11961

V11962V11963V11964V11965V11966V11967V11968V11969V11970V11971

V11972

V11973V11974

V11975V11976

V11977

V11978

V11979V11980V11981V11982V11983V11984V11985V11986V11987V11988V11989V11990V11991V11992V11993V11994V11995V11996V11997V11998V11999V12000V12001V12002V12003V12004V12005V12006V12007V12008V12009V12010V12011V12012V12013V12014V12015V12016V12017V12018V12019

V12020

V12021

V12022

V12023

V12024

V12025V12026V12027V12028V12029V12030V12031V12032V12033V12034

V12035

V12036V12037

V12038

V12039V12040

V12041

V12042V12043V12044V12045V12046V12047

V12048

V12049

V12050

V12051V12052

V12053V12054V12055V12056V12057V12058V12059V12060V12061V12062

V12063

V12064V12065V12066V12067

V12068V12069V12070V12071

V12072V12073V12074V12075V12076V12077V12078V12079

V12080

V12081V12082V12083

V12084

V12085

V12086

V12087V12088V12089V12090V12091V12092V12093

V12094

V12095

V12096V12097

V12098V12099

V12100

V12101V12102V12103V12104V12105V12106V12107V12108

V12109V12110V12111V12112V12113V12114V12115V12116V12117V12118V12119V12120V12121V12122V12123V12124V12125V12126V12127

V12128

V12129

V12130

V12131

V12132

V12133V12134V12135V12136V12137V12138V12139V12140V12141V12142V12143V12144V12145V12146V12147V12148V12149V12150V12151V12152V12153V12154V12155V12156V12157V12158V12159V12160V12161V12162V12163V12164V12165V12166V12167V12168V12169V12170V12171V12172

V12173

V12174

V12175

V12176

V12177V12178

V12179

V12180V12181V12182V12183V12184V12185V12186V12187V12188V12189V12190

V12191V12192

V12193V12194V12195

V12196

V12197V12198

V12199V12200V12201V12202

V12203

V12204V12205V12206V12207V12208V12209V12210V12211V12212V12213V12214V12215V12216V12217V12218

V12219

V12220V12221

V12222

V12223V12224

V12225V12226V12227V12228V12229V12230V12231V12232

V12233V12234

V12235V12236

V12237

V12238

V12239

V12240V12241V12242V12243

V12244V12245

V12246

V12247V12248V12249

V12250

V12251

V12252

V12253V12254

V12255V12256V12257

V12258V12259V12260V12261V12262V12263V12264V12265

V12266

V12267V12268V12269

V12270V12271V12272V12273V12274V12275V12276V12277V12278V12279V12280V12281V12282V12283

V12284V12285V12286

V12287

V12288

V12289V12290V12291V12292V12293

V12294V12295V12296V12297V12298V12299V12300V12301V12302V12303V12304V12305V12306V12307V12308

V12309

V12310V12311V12312

V12313V12314

V12315V12316V12317V12318

V12319

V12320V12321

V12322

V12323

V12324V12325V12326V12327V12328V12329V12330

V12331

V12332

V12333

V12334V12335V12336

V12337

V12338

V12339

V12340V12341V12342V12343

V12344V12345

V12346

V12347

V12348V12349V12350

V12351V12352V12353V12354

V12355

V12356

V12357V12358V12359V12360

V12361V12362V12363

V12364

V12365

V12366

V12367

V12368

V12369

V12370V12371

V12372V12373V12374

V12375

V12376

V12377V12378

V12379

V12380V12381

V12382V12383V12384

V12385V12386V12387V12388V12389V12390V12391V12392V12393V12394V12395V12396V12397V12398

V12399

V12400V12401

V12402

V12403

V12404

V12405V12406V12407V12408V12409V12410V12411V12412

V12413V12414V12415V12416V12417

V12418V12419

V12420V12421

V12422

V12423

V12424

V12425

V12426V12427

V12428V12429

V12430V12431V12432

V12433V12434

V12435

V12436V12437V12438V12439V12440V12441V12442V12443V12444

V12445

V12446V12447

V12448

V12449

V12450V12451V12452V12453V12454V12455V12456V12457V12458

V12459V12460V12461V12462V12463

V12464

V12465

V12466

V12467V12468V12469V12470V12471

V12472V12473V12474V12475

V12476V12477V12478V12479V12480V12481V12482V12483V12484V12485V12486V12487V12488V12489V12490

V12491

V12492V12493

V12494

V12495V12496

V12497V12498V12499V12500V12501V12502V12503V12504V12505V12506V12507V12508V12509V12510

V12511V12512V12513V12514V12515V12516V12517V12518V12519V12520V12521V12522V12523V12524V12525V12526V12527V12528V12529V12530V12531V12532V12533V12534V12535V12536

V12537

V12538

V12539

V12540V12541

V12542

V12543V12544V12545V12546V12547V12548V12549V12550V12551V12552V12553V12554V12555V12556V12557V12558V12559V12560

V12561V12562V12563V12564V12565V12566V12567V12568V12569V12570V12571V12572V12573V12574V12575V12576V12577V12578V12579V12580

V12581

V12582V12583V12584

V12585

V12586V12587

V12588

V12589V12590V12591V12592V12593V12594V12595V12596V12597

V12598V12599

V12600V12601V12602V12603V12604V12605

V12606

V12607V12608V12609V12610V12611V12612V12613V12614V12615V12616

V12617

V12618V12619

V12620V12621V12622V12623V12624V12625

V12626V12627

V12628

V12629

V12630V12631

V12632V12633

V12634V12635V12636V12637V12638V12639V12640V12641V12642V12643V12644V12645V12646V12647

V12648V12649V12650

V12651V12652V12653V12654V12655

V12656V12657V12658V12659V12660V12661V12662V12663V12664V12665V12666V12667V12668V12669V12670V12671V12672V12673

V12674

V12675

V12676

V12677V12678

V12679V12680V12681V12682V12683V12684V12685V12686V12687V12688V12689V12690V12691V12692V12693V12694V12695V12696V12697V12698V12699V12700V12701V12702V12703V12704V12705V12706V12707V12708V12709V12710V12711V12712V12713V12714V12715V12716V12717V12718

V12719

V12720

V12721

V12722

V12723

V12724V12725V12726V12727V12728V12729V12730V12731V12732V12733V12734V12735V12736V12737V12738V12739V12740V12741V12742V12743V12744V12745V12746V12747V12748V12749V12750V12751V12752V12753V12754V12755V12756V12757V12758V12759V12760V12761V12762V12763

V12764

V12765

V12766

V12767

V12768

V12769

V12770V12771V12772V12773V12774V12775V12776V12777V12778V12779V12780V12781V12782V12783V12784V12785V12786V12787V12788V12789V12790V12791V12792V12793V12794V12795V12796V12797V12798V12799V12800V12801V12802V12803V12804V12805V12806V12807V12808

V12809

V12810

V12811

V12812

V12813

V12814V12815V12816V12817V12818V12819V12820V12821V12822V12823V12824V12825

V12826V12827V12828V12829V12830V12831V12832V12833V12834V12835V12836

V12837V12838V12839

V12840V12841V12842

V12843V12844V12845

V12846V12847V12848V12849V12850

V12851V12852

V12853

V12854V12855V12856

V12857

V12858

V12859V12860V12861V12862V12863V12864V12865

V12866V12867V12868V12869V12870V12871V12872V12873V12874V12875V12876V12877V12878V12879V12880

V12881

V12882V12883V12884V12885V12886V12887

V12888

V12889V12890

V12891

V12892

V12893V12894V12895

V12896V12897V12898

V12899V12900V12901

V12902

V12903V12904V12905V12906V12907V12908V12909

V12910

V12911V12912

V12913

V12914V12915V12916V12917V12918V12919V12920V12921V12922V12923

V12924

V12925V12926V12927V12928V12929V12930V12931

V12932V12933V12934V12935V12936V12937V12938V12939V12940

V12941

V12942V12943

V12944

V12945V12946V12947V12948V12949V12950V12951V12952V12953V12954V12955V12956V12957V12958V12959V12960V12961V12962V12963V12964V12965V12966V12967V12968V12969V12970V12971V12972V12973V12974V12975V12976V12977V12978V12979V12980V12981V12982V12983V12984

V12985

V12986

V12987

V12988

V12989V12990

V12991

V12992

V12993V12994V12995V12996V12997V12998

V12999V13000V13001V13002V13003V13004V13005

V13006V13007V13008V13009

V13010V13011V13012V13013V13014V13015V13016V13017V13018V13019V13020V13021

V13022V13023V13024V13025V13026V13027V13028V13029

V13030

V13031

V13032V13033

V13034

V13035V13036V13037V13038V13039V13040V13041V13042V13043V13044V13045

V13046V13047V13048V13049

V13050

V13051V13052

V13053V13054V13055V13056V13057V13058V13059V13060V13061V13062V13063V13064V13065V13066V13067V13068V13069V13070V13071

V13072

V13073V13074V13075

V13076

V13077V13078V13079V13080V13081V13082V13083V13084V13085V13086V13087V13088V13089V13090

V13091

V13092

V13093V13094V13095V13096V13097V13098V13099V13100

V13101V13102V13103V13104V13105V13106V13107V13108V13109V13110V13111V13112V13113V13114

V13115

V13116V13117

V13118

V13119V13120V13121

V13122

V13123V13124V13125V13126V13127V13128

V13129V13130V13131

V13132V13133

V13134

V13135V13136V13137V13138V13139

V13140V13141V13142

V13143

V13144V13145

V13146V13147V13148V13149V13150V13151 V13152V13153V13154 V13155V13156

V13157

V13158V13159

V13160

V13161V13162V13163V13164V13165

V13166V13167

V13168V13169V13170V13171V13172

V13173

V13174V13175V13176

V13177

V13178

V13179

V13180

V13181V13182V13183

V13184

V13185V13186

V13187

V13188

V13189V13190V13191V13192V13193V13194V13195V13196

V13197

V13198

V13199V13200

V13201

V13202V13203

V13204

V13205V13206V13207V13208V13209V13210V13211

V13212V13213V13214V13215V13216

V13217

V13218V13219

V13220V13221V13222V13223

V13224

V13225

V13226V13227V13228

V13229V13230

V13231

V13232V13233

V13234V13235V13236V13237

V13238V13239V13240V13241V13242V13243

V13244

V13245

V13246V13247V13248V13249V13250V13251V13252V13253V13254V13255V13256V13257

V13258V13259

V13260

V13261V13262V13263V13264

V13265V13266V13267

V13268

V13269V13270V13271V13272V13273V13274

V13275V13276V13277V13278V13279V13280V13281V13282V13283V13284

V13285V13286

V13287

V13288V13289V13290V13291V13292V13293V13294V13295V13296V13297V13298V13299V13300

V13301V13302

V13303

V13304

V13305

V13306V13307

V13308V13309V13310V13311V13312V13313

V13314V13315V13316V13317

V13318V13319V13320V13321V13322V13323V13324V13325V13326V13327

V13328

V13329

V13330

V13331V13332V13333V13334V13335V13336V13337V13338V13339V13340V13341V13342V13343

V13344V13345

V13346

V13347V13348

V13349V13350

V13351V13352V13353V13354V13355V13356V13357V13358V13359V13360V13361V13362V13363V13364V13365V13366V13367V13368V13369

V13370

V13371

V13372V13373V13374V13375V13376V13377V13378V13379V13380V13381V13382V13383V13384V13385V13386V13387V13388V13389V13390V13391V13392V13393V13394V13395V13396V13397V13398V13399V13400V13401V13402V13403V13404V13405V13406V13407V13408V13409V13410V13411V13412V13413

V13414

V13415

V13416

V13417

V13418V13419V13420V13421V13422V13423V13424V13425V13426V13427V13428V13429V13430V13431V13432V13433V13434V13435V13436V13437V13438V13439V13440V13441V13442V13443V13444V13445V13446V13447V13448V13449V13450V13451V13452V13453V13454V13455

V13456

V13457

V13458

V13459

V13460

V13461V13462V13463V13464V13465V13466V13467V13468V13469V13470V13471V13472V13473V13474V13475V13476V13477V13478V13479V13480V13481V13482V13483V13484V13485V13486V13487V13488V13489V13490V13491V13492V13493V13494V13495V13496V13497V13498

V13499

V13500

V13501

V13502

V13503

V13504

V13505V13506V13507V13508V13509V13510V13511V13512V13513V13514V13515V13516V13517V13518V13519

V13520

V13521

V13522

V13523

V13524V13525V13526

V13527

V13528V13529V13530V13531V13532V13533

V13534V13535

V13536

V13537V13538V13539V13540

V13541

V13542V13543V13544

V13545V13546

V13547

V13548V13549V13550V13551V13552V13553V13554V13555V13556V13557V13558V13559

V13560V13561

V13562V13563V13564

V13565

V13566

V13567V13568V13569V13570V13571V13572V13573V13574V13575V13576V13577V13578V13579V13580V13581

V13582

V13583V13584

V13585V13586

V13587V13588

V13589

V13590V13591V13592V13593V13594V13595V13596V13597V13598V13599V13600V13601V13602V13603V13604V13605V13606V13607V13608V13609V13610V13611V13612V13613V13614V13615V13616V13617V13618V13619V13620V13621V13622V13623

V13624

V13625

V13626V13627

V13628V13629

V13630V13631V13632V13633V13634

V13635V13636V13637V13638V13639

V13640V13641

V13642V13643V13644

V13645V13646V13647V13648

V13649V13650

V13651V13652V13653

V13654V13655V13656

V13657V13658V13659

V13660

V13661

V13662V13663

V13664V13665V13666

V13667

V13668V13669

V13670

V13671V13672V13673V13674V13675V13676V13677V13678

V13679

V13680V13681V13682V13683

V13684

V13685V13686

V13687

V13688

V13689V13690V13691V13692V13693V13694V13695

V13696V13697V13698

V13699V13700V13701V13702V13703V13704V13705

V13706

V13707V13708

V13709V13710

V13711

V13712

V13713V13714V13715V13716V13717V13718V13719V13720V13721V13722V13723V13724V13725V13726

V13727

V13728V13729

V13730V13731

V13732V13733V13734V13735V13736V13737V13738

V13739V13740

V13741V13742

V13743V13744V13745V13746V13747V13748V13749

V13750V13751

V13752

V13753V13754

V13755

V13756V13757V13758

V13759

V13760

V13761

V13762V13763

V13764

V13765V13766V13767V13768

V13769

V13770V13771

V13772

V13773V13774

V13775V13776V13777V13778V13779V13780

V13781

V13782

V13783V13784

V13785V13786

V13787V13788

V13789V13790V13791

V13792

V13793V13794

V13795

V13796V13797

V13798

V13799

V13800V13801V13802

V13803V13804V13805V13806

V13807

V13808

V13809V13810

V13811V13812V13813V13814V13815

V13816V13817V13818V13819

V13820V13821V13822V13823V13824V13825V13826V13827V13828V13829V13830V13831

V13832

V13833

V13834

V13835V13836

V13837

V13838V13839V13840V13841V13842V13843V13844V13845V13846V13847V13848V13849V13850V13851

V13852V13853V13854

V13855V13856V13857V13858V13859V13860

V13861V13862V13863V13864V13865V13866V13867V13868V13869V13870V13871

V13872

V13873V13874

V13875

V13876

V13877

V13878V13879V13880V13881V13882V13883V13884V13885V13886V13887V13888V13889V13890V13891V13892V13893V13894V13895V13896V13897V13898V13899V13900V13901V13902V13903V13904V13905V13906V13907V13908V13909V13910V13911V13912V13913V13914V13915

V13916

V13917

V13918

V13919

V13920V13921V13922V13923V13924V13925V13926V13927V13928V13929V13930V13931V13932V13933V13934V13935V13936V13937V13938V13939V13940V13941V13942V13943V13944V13945V13946V13947V13948V13949V13950V13951V13952V13953V13954V13955V13956V13957

V13958

V13959V13960

V13961

V13962V13963V13964V13965V13966V13967V13968V13969V13970V13971V13972V13973V13974V13975V13976V13977V13978V13979V13980V13981V13982V13983V13984V13985V13986V13987V13988V13989V13990V13991V13992V13993V13994V13995V13996V13997

V13998

V13999V14000

V14001

V14002V14003

V14004

V14005

V14006V14007V14008V14009V14010V14011V14012V14013

V14014

V14015V14016V14017V14018V14019V14020V14021V14022V14023V14024V14025V14026V14027

V14028V14029V14030V14031V14032V14033V14034V14035V14036

V14037

V14038V14039V14040V14041V14042V14043V14044

V14045V14046

V14047

V14048

V14049

V14050

V14051

V14052V14053V14054V14055V14056

V14057V14058

V14059V14060

V14061 V14062V14063V14064V14065V14066

V14067

V14068

V14069V14070V14071V14072V14073

V14074V14075V14076

V14077V14078V14079V14080V14081V14082V14083

V14084

V14085V14086V14087V14088V14089V14090V14091

V14092

V14093

V14094V14095

V14096

V14097V14098V14099

V14100

V14101

V14102

V14103

V14104V14105V14106V14107V14108V14109V14110V14111V14112V14113V14114

V14115V14116

V14117

V14118

V14119V14120V14121

V14122

V14123V14124V14125V14126V14127

V14128V14129

V14130V14131V14132V14133V14134V14135V14136V14137V14138V14139

V14140

V14141

V14142

V14143

V14144

V14145

V14146V14147V14148V14149V14150V14151V14152V14153V14154V14155V14156V14157V14158V14159V14160V14161V14162V14163V14164V14165V14166V14167V14168

V14169

V14170

V14171

V14172V14173V14174V14175V14176V14177V14178V14179V14180

V14181

V14182

V14183

V14184

V14185

V14186V14187V14188V14189V14190V14191V14192V14193V14194V14195V14196V14197V14198V14199V14200V14201V14202V14203V14204V14205V14206V14207V14208V14209V14210V14211V14212

V14213V14214V14215V14216V14217V14218

V14219

V14220V14221

V14222V14223

V14224

V14225

V14226

V14227

V14228V14229V14230V14231V14232V14233V14234V14235V14236V14237V14238V14239V14240

V14241V14242

V14243V14244

V14245

V14246

V14247V14248

V14249V14250V14251

V14252V14253V14254V14255V14256V14257V14258V14259

V14260V14261V14262V14263

V14264

V14265

V14266

V14267V14268V14269V14270V14271V14272V14273V14274V14275V14276V14277V14278V14279V14280V14281V14282V14283V14284V14285V14286V14287V14288V14289V14290V14291V14292V14293V14294V14295V14296V14297V14298V14299V14300V14301V14302V14303

V14304

V14305V14306

V14307

V14308

V14309V14310V14311V14312V14313V14314V14315V14316V14317V14318V14319V14320V14321V14322V14323V14324V14325V14326V14327V14328V14329V14330V14331V14332V14333V14334V14335V14336V14337V14338V14339V14340V14341V14342V14343V14344

V14345

V14346V14347

V14348

V14349V14350V14351V14352V14353V14354V14355V14356V14357V14358V14359V14360V14361V14362V14363V14364

V14365

V14366

V14367V14368V14369V14370V14371V14372V14373V14374V14375V14376V14377V14378V14379V14380V14381V14382V14383V14384V14385

V14386

V14387V14388V14389

V14390V14391

V14392

V14393V14394V14395V14396V14397V14398V14399V14400V14401V14402V14403V14404V14405V14406V14407V14408V14409V14410V14411V14412V14413V14414V14415V14416V14417V14418V14419V14420V14421V14422V14423

V14424

V14425V14426V14427V14428

V14429V14430

V14431V14432

V14433

V14434

V14435V14436V14437V14438V14439V14440V14441V14442V14443V14444V14445V14446V14447

V14448V14449V14450V14451V14452V14453V14454V14455V14456V14457V14458V14459V14460V14461V14462V14463V14464V14465V14466

V14467

V14468

V14469

V14470

V14471

V14472V14473V14474V14475V14476V14477V14478V14479V14480V14481V14482V14483V14484V14485V14486V14487V14488V14489V14490V14491V14492V14493V14494V14495V14496V14497V14498V14499V14500V14501V14502V14503V14504V14505V14506V14507

V14508

V14509

V14510V14511

V14512

V14513V14514V14515V14516V14517V14518V14519V14520V14521V14522V14523V14524V14525V14526V14527V14528

V14529

V14530V14531V14532V14533V14534

V14535V14536V14537V14538V14539V14540V14541

V14542V14543V14544V14545V14546

V14547

V14548V14549V14550

V14551

V14552

V14553V14554V14555V14556V14557V14558V14559V14560V14561V14562V14563V14564V14565V14566V14567V14568V14569V14570V14571V14572V14573V14574V14575V14576V14577V14578V14579V14580V14581V14582V14583V14584V14585V14586V14587V14588V14589

V14590

V14591

V14592

V14593V14594V14595V14596V14597V14598V14599V14600V14601V14602V14603V14604V14605V14606

V14607

V14608V14609V14610

V14611

V14612V14613V14614

V14615V14616V14617V14618V14619V14620V14621

V14622V14623V14624V14625V14626

V14627V14628V14629V14630

V14631V14632

V14633V14634

V14635

V14636V14637

V14638

V14639

V14640V14641V14642V14643V14644V14645

V14646

V14647

V14648V14649

V14650

V14651

V14652V14653V14654V14655V14656

V14657V14658

V14659

V14660

V14661V14662V14663V14664

V14665V14666V14667

V14668

V14669V14670V14671V14672V14673V14674V14675V14676V14677V14678

V14679V14680V14681V14682V14683V14684V14685V14686V14687V14688V14689V14690V14691V14692

V14693V14694V14695V14696

V14697V14698V14699V14700

V14701V14702V14703V14704V14705V14706V14707V14708

V14709

V14710

V14711V14712

V14713V14714

V14715

V14716

V14717V14718

V14719V14720

V14721V14722

V14723V14724V14725

V14726

V14727

V14728

V14729V14730V14731

V14732V14733V14734V14735V14736V14737V14738V14739V14740

V14741V14742

V14743

V14744

V14745

V14746V14747

V14748

V14749V14750V14751V14752V14753V14754V14755V14756V14757V14758V14759V14760V14761V14762V14763V14764V14765V14766V14767V14768V14769V14770V14771V14772V14773V14774V14775V14776V14777V14778V14779V14780V14781V14782V14783V14784

V14785

V14786V14787

V14788

V14789V14790V14791V14792V14793V14794V14795V14796V14797V14798V14799V14800V14801V14802V14803V14804V14805V14806V14807V14808V14809V14810V14811V14812V14813V14814V14815V14816V14817V14818V14819V14820V14821V14822V14823V14824

V14825

V14826

V14827

V14828

V14829V14830V14831V14832V14833V14834V14835V14836V14837V14838V14839V14840V14841V14842V14843V14844V14845V14846V14847V14848V14849V14850V14851V14852V14853V14854V14855V14856V14857V14858V14859V14860V14861V14862V14863V14864

V14865

V14866V14867

V14868

V14869

V14870V14871V14872V14873V14874V14875V14876V14877V14878V14879V14880V14881V14882V14883V14884V14885V14886V14887V14888V14889V14890V14891V14892V14893V14894V14895V14896V14897V14898V14899V14900V14901V14902V14903V14904V14905

V14906

V14907V14908V14909

V14910

V14911V14912V14913V14914V14915V14916V14917V14918V14919V14920V14921V14922V14923V14924V14925V14926V14927V14928V14929V14930V14931V14932V14933V14934V14935V14936V14937V14938V14939V14940V14941V14942V14943V14944V14945V14946

V14947

V14948V14949

V14950

V14951V14952V14953V14954V14955V14956V14957V14958V14959V14960V14961V14962V14963V14964V14965V14966V14967V14968V14969V14970V14971V14972V14973V14974V14975V14976V14977V14978V14979V14980V14981V14982V14983V14984V14985V14986

V14987

V14988V14989V14990

V14991

V14992V14993V14994V14995V14996V14997V14998V14999V15000V15001V15002V15003V15004V15005V15006V15007V15008V15009V15010V15011V15012V15013V15014V15015V15016V15017V15018V15019V15020V15021V15022V15023V15024V15025V15026V15027

V15028

V15029V15030

V15031

V15032V15033V15034V15035V15036V15037V15038V15039V15040V15041V15042V15043V15044V15045V15046V15047V15048V15049V15050V15051V15052

V15053

V15054V15055V15056

V15057V15058

V15059V15060V15061V15062V15063V15064V15065V15066

V15067

V15068

V15069

V15070

V15071V15072V15073V15074V15075V15076V15077V15078V15079V15080V15081V15082V15083V15084V15085V15086V15087V15088V15089V15090V15091V15092V15093V15094V15095

V15096V15097V15098V15099V15100V15101V15102V15103V15104V15105

V15106

V15107V15108V15109V15110

V15111

V15112V15113V15114V15115V15116V15117V15118V15119V15120V15121V15122V15123V15124V15125V15126V15127V15128

V15129V15130V15131V15132V15133

V15134V15135V15136

V15137

V15138V15139

V15140V15141V15142

V15143

V15144

V15145

V15146V15147V15148V15149V15150V15151

V15152

V15153V15154

V15155V15156

V15157

V15158V15159

V15160V15161V15162V15163

V15164V15165V15166V15167V15168V15169V15170V15171

V15172V15173

V15174V15175V15176V15177V15178V15179V15180

V15181

V15182

V15183V15184

V15185V15186

V15187

V15188V15189

V15190V15191V15192V15193V15194V15195V15196V15197V15198

V15199V15200V15201

V15202V15203

V15204

V15205

V15206

V15207V15208V15209V15210V15211V15212

V15213V15214V15215V15216V15217V15218V15219V15220

V15221

V15222

V15223V15224

V15225

V15226

V15227V15228V15229V15230V15231V15232V15233V15234V15235V15236V15237V15238V15239

V15240V15241V15242V15243V15244V15245V15246V15247

V15248

V15249V15250V15251

V15252V15253

V15254V15255

V15256V15257V15258

V15259V15260V15261

V15262

V15263

V15264

V15265

V15266V15267V15268V15269V15270V15271V15272V15273V15274

V15275V15276V15277V15278V15279V15280

V15281V15282

V15283V15284V15285

V15286

V15287V15288

V15289

V15290

V15291V15292V15293V15294V15295V15296V15297V15298V15299

V15300

V15301

V15302

V15303

V15304V15305V15306V15307V15308V15309V15310V15311V15312V15313V15314V15315V15316V15317V15318V15319V15320V15321V15322V15323V15324V15325V15326V15327V15328V15329

V15330V15331V15332V15333V15334V15335V15336V15337V15338V15339

V15340

V15341

V15342

V15343V15344V15345V15346V15347V15348V15349V15350V15351

V15352V15353V15354V15355V15356

V15357V15358V15359V15360V15361V15362V15363V15364V15365V15366V15367V15368V15369V15370V15371

V15372V15373V15374V15375V15376V15377V15378

V15379

V15380V15381

V15382

V15383V15384V15385V15386V15387V15388V15389V15390V15391V15392V15393V15394V15395V15396V15397V15398V15399V15400V15401V15402V15403V15404V15405V15406V15407V15408V15409V15410V15411V15412V15413V15414V15415V15416V15417

V15418

V15419

V15420V15421

V15422

V15423V15424V15425V15426V15427V15428V15429V15430V15431V15432V15433V15434V15435V15436V15437V15438V15439V15440V15441V15442V15443V15444V15445V15446V15447V15448V15449V15450V15451V15452V15453V15454V15455V15456V15457

V15458

V15459

V15460V15461

V15462

V15463V15464V15465V15466V15467V15468V15469V15470V15471V15472V15473V15474V15475V15476V15477V15478V15479V15480V15481V15482V15483V15484V15485V15486V15487V15488V15489V15490V15491V15492V15493V15494V15495V15496V15497V15498

V15499

V15500V15501V15502V15503V15504V15505V15506V15507V15508V15509V15510V15511V15512V15513V15514V15515V15516V15517V15518V15519V15520V15521V15522V15523V15524V15525V15526V15527V15528V15529V15530V15531V15532V15533V15534V15535V15536V15537V15538

V15539

V15540V15541

V15542V15543

V15544

V15545

V15546

V15547

V15548

V15549

V15550V15551V15552V15553V15554V15555V15556V15557

V15558V15559V15560V15561

V15562V15563V15564V15565V15566V15567V15568V15569V15570V15571V15572V15573V15574V15575V15576V15577V15578

V15579

V15580V15581V15582V15583V15584V15585V15586V15587V15588V15589V15590V15591V15592V15593V15594V15595V15596V15597V15598V15599V15600V15601V15602V15603V15604V15605V15606V15607V15608V15609V15610V15611V15612V15613V15614V15615V15616V15617V15618

V15619

V15620V15621V15622V15623V15624V15625V15626V15627V15628V15629V15630V15631V15632V15633V15634V15635V15636V15637V15638V15639V15640V15641V15642V15643V15644V15645V15646V15647V15648V15649V15650V15651

V15652V15653V15654V15655V15656V15657V15658

V15659

V15660V15661V15662V15663V15664V15665V15666V15667V15668V15669V15670V15671V15672V15673V15674V15675V15676V15677V15678V15679V15680V15681V15682V15683V15684V15685V15686V15687V15688V15689V15690V15691V15692V15693V15694V15695V15696V15697V15698

V15699

V15700

V15701V15702

V15703

V15704V15705V15706V15707V15708V15709V15710V15711V15712V15713V15714V15715V15716V15717V15718V15719V15720V15721V15722V15723V15724V15725V15726V15727V15728V15729V15730V15731V15732V15733

V15734

V15735

V15736

V15737

V15738V15739

V15740V15741

V15742

V15743V15744V15745V15746V15747V15748V15749V15750V15751V15752

V15753V15754V15755V15756V15757

V15758

V15759

V15760

V15761V15762V15763V15764V15765V15766V15767V15768V15769V15770V15771V15772V15773V15774V15775V15776V15777

V15778

V15779

V15780

V15781V15782V15783

V15784V15785

V15786V15787

V15788

V15789

V15790V15791

V15792

V15793V15794

V15795V15796V15797V15798

V15799

V15800

V15801

V15802

V15803V15804

V15805V15806V15807

V15808V15809V15810V15811

V15812

V15813V15814V15815

V15816

V15817

V15818

V15819V15820

V15821

V15822V15823

V15824V15825

V15826V15827V15828V15829V15830

V15831V15832

V15833

V15834

V15835

V15836

V15837V15838V15839

V15840V15841V15842V15843

V15844

V15845V15846

V15847V15848

V15849V15850V15851V15852V15853V15854

V15855

V15856

V15857

V15858

0

5000

10000

1 2 3 4 5 6 7 8 9 10

eigenvalue

Figure 5.1: The PCA performed on the results given by the Morris method. On the left the correlation circle and onthe right the eigenvalues.

Figure 5.1 shows that the time series of the outputs, given by the Morris method, can be projected on a 5 axisbasis with keeping more that 99% of the information. Then, this projection in the new space is used to compute thenew indices of the Morris method. Figure 5.2 illustrates them all for each of the 5 axes of the dimensional subspacegiven by the PCA.

IscAlphaEg

nD

nDbc_bc_aGms_aT

Rshunt_stc_1D

Rserie_b_1DRserie_aT_1D

Rshunt_0_1D

k_1D

l_elecl_modcharl_modquall_ohmicPacoPdcoPsoVdcoCoC1C2C3PdcmaxIscAlphaEgnD

nDbc_bc_aGms_aT

Rshunt_stc_1D

Rserie_b_1DRserie_aT_1D

Rshunt_0_1D

k_1D

l_elecl_modcharl_modquall_ohmicPacoPdcoPsoVdcoCoC1C2C3PdcmaxIscAlphaEg

nD

nDbc_bc_aGms_aTRshunt_stc_1D

Rserie_b_1DRserie_aT_1D

Rshunt_0_1D

k_1D

l_elecl_modcharl_modquall_ohmicPacoPdcoPsoVdcoCoC1C2C3PdcmaxIscAlphaEg

nD

nDbc_bc_aGms_aT

Rshunt_stc_1DRserie_b_1D

Rserie_aT_1D

Rshunt_0_1D

k_1D

l_elecl_modcharl_modquall_ohmicPacoPdcoPsoVdcoCoC1C2C3Pdcmax IscAlphaEg

nD

nDbc_bc_aGms_aTRshunt_stc_1DRserie_b_1D

Rserie_aT_1D

Rshunt_0_1D

k_1D

l_elecl_modcharl_modquall_ohmicPacoPdcoPsoVdcoCoC1C2C3Pdcmax0

100

200

300

0 100 200 300 400 500μ*

σ

a

a

aaa

PCA5

PCA4

PCA3

PCA2

PCA1

Isc

nD

Rshunt_0_1D

k_1D

Alpha

Rshunt_stc_1D

Rshunt_0_1D

Alpha

nD

Rserie_b_1D

Rshunt_0_1D

k_1D

Alpha

Rshunt_0_1D

k_1D

nD

k_1D

l_ohmic0

100

200

300

0 100 200 300 400 500μ*

σ

a

a

aaa

PCA5

PCA4

PCA3

PCA2

PCA1

Figure 5.2: Morris indices in the new space given by the PCA. On the left all the parameters names appear and onthe right only the ones that are not overlapping are displayed.

Figure 5.2 can also be seen as a representation of the Morris analysis but in a smaller dimension to allow aproper graphical interpretation. On the left panel in Figure 5.2, the new indices on the 5 axes that are carrying 99%of the total inertia is given and allow to identify the parameter that have no impact on the time series output over thetime frame. Graphically, 19 parameters are identified as having no impact on the output. It means that calibration

114

Cha

pter

5

5.1. Sensitivity analysis

will need to focus on only 7 parameters that are:

• Isc is the short circuit current. It is the current delivered by a PV panel when the voltage at its terminals iszero.

• nD is the ideal factor of the diode in the equivalent electric circuit. It would be close to 1 for a perfect diode.The diode corresponds to the diode of the one in the left electrical scheme in Figure 1.5.

• Rshunt_stc_1D is the shunt resistance in standard test conditions. Shunt resistances are typically due tomanufacturing defects. Low shunt resistance causes power losses in solar cells by providing an alternatecurrent path for the light-generated current. Its impact on the I/V curve and on the efficiency function of theirradiance is given in Figure 5.3.

• Rserie_b_1D is the series resistance in standard test conditions. Series resistance in a solar cell has threecauses: first, the movement of current through the emitter and base of the solar cell; second, the contactresistance between the metal contact and the silicon; and finally the resistance of the top and rear metalcontacts. Its impact on the I/V curve and on the efficiency function of the irradiance is also given in Figure5.3.

• Rshunt_0_1D and K_1D coefficients that allow the shunt resistance to vary with the irradiation.

• Paco is the maximum ac-power “rating” for inverter at reference or nominal operating condition, assumed tobe an upper limit value (see Figure 5.4 for more details).

115

Cha

pter

5

Chapter 5 – Performance monitoring on a large PV plant

Figure 5.3: On the top left, the impact of different values of shunt resistances on the I/V curve. On the top right theimpact of different values of shunt resistances on the evolution of the efficiency function of the irradiance. On thebottom left, the impact of different values of series resistances on the I/V curve. On the bottom right, the impact ofdifferent values of series resistances on the evolution of the efficiency function of the irradiance.

Figure 5.4: Illustration of the inverter performance model and the factors describing the relationship of the ac-outputto both dc-power and dc-voltage (source: https://energy.sandia.gov/wp-content/gallery/uploads/Performance-Model-for-Grid-Connected-Photovoltaic-Inverters.pdf).

116

Cha

pter

5

5.2. Prior densities

5.2 Prior densities

Prior densities are generally established from the expert’s knowledge. For all of them a normal density is chosen,except for the variance of the measurement error which is a Gamma. The means of the normal densities are chosenaccordingly to the data sheet of the PV panel or of the inverter. The variances are chosen thanks to the credibilityintervals given by the manufacturer. As the sensitivity analysis has elected only 7 parameters, the prior densitieswill concern only these ones, the others will be fixed to nominal/reference values. Figure 5.5 illustrates all of themaccording the credibility intervals given by the experts.

0

1

2

3

4

8.2 8.4 8.6 8.8Isc

Den

sity

prior

0.00

0.25

0.50

0.75

1.00

0 1 2nD

Den

sity

prior

0

1

2

3

0.00 0.25 0.50 0.75 1.00Rshunt_stc_1D

Den

sity

prior

0

10000

20000

30000

0.000075 0.000100 0.000125 0.000150 0.000175Rserie_b_1D

Den

sity

prior

0

1

2

3

1.00 1.25 1.50 1.75 2.00Rshunt_0_1D

Den

sity

prior

0.0

0.1

0.2

0.3

2.5 5.0 7.5 10.0K_1D

Den

sity

prior

0e+00

2e−05

4e−05

820000 840000 860000Paco

Den

sity

prior

0e+00

2e−10

4e−10

0e+00 2e+09 4e+09 6e+09

σerr2

Den

sity

prior

Figure 5.5: Prior densities for each parameter considered for further calibration.

117

Cha

pter

5

Chapter 5 – Performance monitoring on a large PV plant

5.3 Propagation of uncertainties

The prior credibility interval is obtained from sampling in prior densities values of θ and getting the output of thecode for each of them. The 95% credibility interval can be obtained. Figure 5.6 illustrates the prior credibilityinterval for the time series confronted with experimental data.

0

0.5

1

1.5

2

29 Jan 30 Jan 31 Jan 1 Fev 2 Fev 3 Fev 4 Fev 5 Fev 6 Fev 7 Fev

Days in 2015

Sca

led

pow

er in

W

CI 95% a priori

0

0.5

1

1.5

2

0 6 12 18 24

Time in hour

Sca

led

pow

er in

W

CI 95% a priori

Figure 5.6: Propagation of uncertainties based on prior elicitation. On the top experimental data over 10 days in2015 are displayed with the credibility interval a priori and on the bottom, to zoom on the phenomenon, only oneday has been plotted (28th of January 2015).

In Figure 5.6, the data are not centered but the scale in the y-axis has been changed. If the centered power hasbeen considered the credibility interval a priori would not have been seen smaller than the reality. One can notice, inFigure 5.6 that the credibility interval is not constant during the time series. It increases when the power productiongrows. It is damaging when one wants to estimate the production for a next period of time. The poor knowledgecarried by the prior parameter densities does not allow to access to a narrow prediction.

5.4 Bayesian calibration

So far, calibration model and estimation methods have been presented when the numerical code output lies in R. Inthe non scalar case where the code output is a time series or multidimensional, one can wonder how to write thestatistical models and the associated likelihoods to perform calibration. First, we present the models useful to dealwith calibration that implies time series outputs. The likelihoods corresponding to each models are also presentedand then, the results based on the Dymola code will be presented and commented.

5.4.1 Statistical models

Let us consider the framework introduced in Section 1.4.1. The numerical code with the time series output can bewritten as:

118

Cha

pter

5

5.4. Bayesian calibration

fc : Q→ RT

θ 7→ Y ,(5.1)

where T stands for the number of instantaneous power observed and Q ⊂ Rp. Note that no input variablesX aretaken into account in this formalization since they are implicitly implemented in the Dymola code as mentioned inSection 1.4.4 which is significantly different from the Higdon et al. (2008).

The case of a time consuming code is assumed in this section. To have a surrogate that reproduces a time series,the first idea would be to generate a surrogate for each time step. When T is too high a reduction of dimensionmust be considered. In that respect, a LHS maximin DOE (see Section 2.3) of n points is generated and the timesseries are computed for each configurations of θ. The results are stored in a matrix Y of size n×T . A PCA isperformed on the reduced and centered results matrix Y . It is, then, possible to identify d principal components thatare carrying more than 99% of the total inertia. It means that the sub-space with the d dimensional PCA basis issufficient to represent well enough the simulations. Let us note P the transition matrix from the physical basis tothe PCA basis such as:

P : RT → RT . (5.2)

The matrix P represents the T ×T matrix that allows to get the coordinates of Y on the T PCA axes.

In the space of the PCA, the model M4 (Equation (3.5)) can be projected as:

PYexp = P fc(θ)+P δ +PE. (5.3)

The PCA on simulated data gives the information that all the generated trajectories by the code can be representedat 99% in a d sized orthogonal subspace. It is then possible to identify two subspaces. The first is built with the d

first eigenvectors given by the PCA and the second in the orthogonal subspace generated with the T −d eigenvectorsleft. The PCA tells that P fc(θ) mainly lies in a d-dimension orthogonal subspace. It is then possible to representaccurately the numerical code by using d independent Gaussian processes that emulates the projections of the codeon the d axes of the PCA. The hypothesis made here is to consider that the discrepancy represents everything that isleft over in the T −d dimensional subspace. The statistical model would become:

M ′4 : PYexp = PF

fc1(θ)

...fcd (θ)

+Pδ δ +E = PFF +Pδ δ +E, (5.4)

Model with PCA

where fc1 , . . . , fcd (the components of F ) are equal to P1 fc, . . . ,Pd fc with P1, . . . ,Pd the projections on the d principalcomponents, and E is equal to PE in probability distribution because the matrix P is orthonormal. The PCA isperformed on centered data, so it is possible to formulate the hypothesis that the Gaussian processes on each axiswould be defined as fci(•)∼ G P(0,σ2

pirpi(•,•)). This is a prior belief that is expressed by Higdon et al. (2008)and as code parameters might not have the same impact on the different axis, we rely on anisotropic Gaussianprocesses. The discrepancy represents the rest of the projection of the data on the T − d axes of the PCA andcan be modeled as an independent and identically distributed Gaussian noise which is a random vector definedas δ ∼N (0T−d ,σ

2δ

IT−d). The matrix PF and Pδ are two matrix that are composed of, respectively, the d first

119

Cha

pter

5

Chapter 5 – Performance monitoring on a large PV plant

eigenvectors and the T −d remaining eigenvectors.

If no discrepancy is considered, the statistical model can be simplified as:

M ′2 : PYexp = PFF +E, (5.5)

Model with PCA without discrepancy

5.4.2 Modular estimation and likelihoods

To fulfill a proper modular estimation in the sense of modularization (Liu et al., 2009), the likelihoods have tobe written. The estimation is performed in two steps. On each of the d axis that are supporting the Gaussianprocesses, the partial likelihood can be written from the coordinates of the n points obtained from the DOED usedto perform the PCA. The coordinates on the d axis of the PCA are called Yc1 , . . . ,Ycd . The ith partial likelihood canbe expressed as:

L Mi (Φi;Yci ,D) =

1(2π)n/2|σ2

pirpi(D)|1/2 exp

− 1

2

(Yci

)T(σ2

pirpi(D))−1(Yci

), (5.6)

where σ2pi and rpi are the prior variance of the covariance function and the prior anisotropic correlation function

relative to the ith Gaussian process. The estimation of all the nuisance parameters of each Gaussian process can bedone in a Bayesian framework where prior densities of all the parameters are given and the posterior densities aresampled with MCMC algorithms. It could also be performed using maximum likelihood estimates to get estimatorsof the nuisance parameters.

In a Bayesian framework, the prior densities can be chosen according to Higdon et al. (2008) where they considerσ2

pi as a Gamma distribution, and each ψi j (the correlation lengths in rpi) as a Beta distribution where i would be theith axis and j would be the jth parameter component. Once estimators of the parameters a posteriori, such as the Max-imum A Posteriori (MAP), are found they are plugged into the conditional likelihood. For the following, the methodof using the maximum likelihood estimates to find estimators of each Gaussian processes is picked. Let us, then,consider that the ith Gaussian process conditioned on the DOE is defined as fci(•)|Pi fc(D)∼G P(mi(•),σ2

i ci(•,•)).

In the case where the discrepancy is a white Gaussian noise and independent Gaussian processes are chosen foreach as surrogates on each axis, the full likelihood can be written straightforwardly. If we consider Φ as the set ofthe nuisance parameters for the d surrogates estimators, then the full likelihood of the model M ′

4 is:

L f (PYexp;θ ,σ2err,σ

2δ,Φ) =

1

(2π)d/2Πdi=1

√σ2

err +σ2i (θ)

exp

(−1

2

d

∑i=1

(P TFiYexp−mi(θ))

2

σ2err +σ2

i (θ)

)

1(2π(σ2 +σ2

δ))(T−d)/2 exp

(− 1

2(σ2err +σ2

δ)‖P T

δYexp‖2

).

(5.7)

120

Cha

pter

5

5.4. Bayesian calibration

And for the model without discrepancy M ′2:

L f (PYexp;θ ,σ2err,Φ) =

1

(2π)d/2Πdi=1

√σ2

err +σ2i (θ)

exp

(−1

2

d

∑i=1

(P TFiYexp−mi(θ))

2

σ2err +σ2

i (θ)

)1

(2π(σ2err))

(T−d)/2 exp(− 1

2(σ2err)‖P T

δYexp‖2

).

(5.8)

5.4.3 Application to the PV plant

The Dymola code is time consuming and produces a time series output. In this section, both models M ′2 and M ′

4

will be applied to the code and the results will be commented. First, the necessary PCA is performed before carryingout calibration. Then, after getting the estimators for the maximization of the partial likelihood, calibration isperformed.

The PCA

As introduced earlier, the Dymola code possesses a time series outputs of 15,858 points. A PCA is performed onthe output of the code of a maximin LHS DOE (see Section 2.3) of n = 300 points. The results of the PCA obtainedare displayed Figure 5.7.

−1.0 −0.5 0.0 0.5 1.0

−1.

0−

0.5

0.0

0.5

1.0

Variables factor map (PCA)

Dim 1 (97.20%)

Dim

2 (

2.28

%)

V1V2V3

V4V5

V6V7V8V9V10V11V12V13V14V15V16V17V18V19V20V21V22V23V24V25V26V27V28V29V30V31V32V33V34V35V36V37V38V39V40V41V42

V43

V44 V45V46

V47V48

V49V50V51

V52V53

V54V55V56V57V58

V59V60V61

V62

V63

V64V65V66V67V68V69V70

V71

V72V73

V74

V75V76

V77V78V79

V80V81V82

V83V84V85V86

V87V88V89 V90

V91V92V93

V94

V95

V96V97 V98

V99V100

V101V102V103V104V105V106V107V108V109V110V111V112V113V114V115V116V117V118V119V120V121V122V123V124V125V126V127V128V129V130V131V132V133V134V135V136V137V138

V139

V140

V141V142

V143V144

V145V146V147

V148

V149V150

V151V152

V153

V154V155 V156

V157V158V159V160

V161V162V163V164V165

V166

V167V168V169

V170V171V172 V173V174

V175V176

V177V178

V179V180V181 V182

V183V184V185V186

V187

V188V189V190 V191

V192

V193V194

V195V196

V197V198V199V200V201V202V203V204V205V206V207V208V209

V210V211V212V213V214V215V216

V217V218V219V220V221V222V223V224V225V226V227V228V229V230V231V232V233V234V235

V236

V237V238V239V240V241

V242V243V244V245V246

V247V248

V249

V250

V251V252V253

V254

V255V256

V257

V258V259

V260V261V262 V263

V264V265V266V267V268

V269

V270

V271V272

V273

V274

V275V276V277V278

V279

V280V281

V282

V283V284V285V286V287V288

V289

V290V291

V292

V293V294V295V296V297V298V299V300V301V302V303V304V305V306V307V308

V309V310V311V312

V313V314V315

V316V317V318

V319V320V321V322V323V324V325V326V327V328V329V330

V331

V332V333

V334V335

V336V337

V338 V339V340

V341V342

V343V344V345V346V347

V348

V349

V350V351V352

V353

V354

V355V356

V357V358V359V360V361

V362V363

V364V365V366

V367V368V369

V370V371 V372V373

V374

V375V376V377

V378V379

V380V381

V382

V383V384V385

V386V387

V388

V389V390V391V392V393V394V395V396V397V398V399V400V401V402V403V404V405V406V407V408V409V410V411V412V413V414V415V416V417V418V419V420V421V422V423V424V425V426

V427V428V429V430V431V432V433V434V435V436V437V438V439V440V441V442V443V444V445V446V447V448V449V450V451V452V453V454V455V456V457V458V459V460V461V462V463V464V465V466V467V468V469V470V471V472V473V474V475V476V477V478V479V480V481V482V483V484

V485V486V487V488V489V490V491V492

V493V494V495V496V497V498V499V500V501V502V503V504V505V506V507V508V509V510V511V512V513V514V515V516V517V518V519V520V521V522V523

V524V525V526V527V528V529V530V531V532V533V534V535V536V537V538V539V540V541V542V543V544V545V546V547V548V549V550V551V552V553V554V555V556V557V558V559V560V561V562V563V564V565V566V567V568V569V570V571V572V573V574V575V576V577V578V579V580V581V582V583

V584

V585V586V587V588V589V590V591V592V593V594

V595V596V597V598

V599V600V601V602V603

V604V605V606V607

V608V609V610V611V612

V613V614V615V616V617V618

V619

V620V621V622V623V624V625V626V627V628V629V630V631V632V633V634V635V636V637V638V639V640V641V642V643V644V645V646V647V648V649V650V651V652V653V654V655V656V657V658V659V660V661V662V663V664V665V666V667V668V669V670V671V672V673V674V675V676V677

V678V679V680V681V682V683V684V685V686V687V688V689V690V691V692V693

V694V695V696V697V698V699V700V701V702V703V704V705V706V707V708V709V710V711

V712V713

V714V715V716V717V718V719V720V721V722V723V724V725V726V727V728V729V730V731V732V733V734V735V736V737V738V739V740V741V742V743V744V745V746V747V748V749V750V751V752V753V754V755V756V757V758V759V760V761V762V763V764V765V766V767V768V769V770V771V772V773

V774V775V776

V777V778V779V780V781V782V783V784V785V786V787V788V789V790V791V792V793V794V795V796V797V798V799V800V801V802V803V804V805V806V807

V808V809

V810V811V812V813V814V815V816V817V818V819V820V821V822V823V824V825V826V827V828V829V830V831V832V833V834V835V836V837V838V839V840V841V842V843V844V845V846V847V848V849V850V851V852V853V854V855V856V857V858V859V860V861V862V863V864V865V866V867V868

V869V870V871V872V873V874V875V876V877V878

V879V880V881V882V883V884V885V886V887V888V889V890V891

V892V893V894V895V896V897V898V899V900V901V902

V903V904

V905V906V907V908V909V910V911V912V913V914V915V916V917V918V919V920V921V922V923V924V925V926V927V928V929V930V931V932V933V934V935V936V937V938V939V940V941V942V943V944V945V946V947V948V949V950V951V952V953V954V955V956V957V958V959V960V961V962V963V964

V965V966V967V968V969V970

V971V972

V973V974V975V976V977

V978

V979

V980V981V982V983V984V985

V986

V987V988

V989

V990V991

V992V993V994

V995V996V997V998V999V1000V1001V1002

V1003

V1004V1005V1006V1007V1008V1009V1010V1011V1012V1013V1014V1015V1016V1017V1018V1019V1020V1021V1022V1023V1024V1025V1026V1027V1028V1029V1030V1031V1032V1033V1034V1035V1036V1037V1038V1039V1040V1041V1042V1043V1044V1045V1046V1047V1048V1049V1050V1051V1052V1053V1054V1055V1056V1057V1058V1059V1060

V1061V1062V1063V1064V1065V1066V1067V1068V1069V1070V1071V1072V1073V1074V1075V1076V1077V1078V1079V1080V1081V1082V1083V1084V1085V1086V1087V1088V1089V1090V1091V1092V1093V1094V1095V1096V1097V1098

V1099

V1100V1101V1102V1103V1104V1105V1106V1107V1108V1109V1110V1111V1112V1113V1114V1115V1116V1117V1118V1119V1120V1121V1122V1123V1124V1125V1126V1127V1128V1129V1130V1131V1132V1133V1134V1135V1136V1137V1138V1139V1140V1141V1142V1143V1144V1145V1146V1147V1148V1149V1150V1151V1152V1153V1154V1155V1156

V1157V1158V1159V1160V1161V1162V1163V1164V1165V1166V1167V1168V1169V1170V1171V1172V1173V1174V1175V1176V1177V1178V1179V1180V1181V1182V1183V1184V1185V1186V1187V1188V1189V1190V1191V1192V1193V1194

V1195

V1196V1197V1198V1199V1200V1201V1202V1203V1204V1205V1206V1207V1208V1209V1210V1211V1212V1213V1214V1215V1216V1217V1218V1219V1220V1221V1222V1223V1224V1225V1226V1227V1228V1229V1230V1231V1232V1233V1234V1235V1236V1237V1238V1239V1240V1241V1242V1243V1244V1245V1246V1247V1248V1249V1250V1251V1252

V1253V1254V1255V1256V1257V1258V1259V1260V1261V1262V1263V1264V1265V1266V1267V1268

V1269V1270

V1271V1272V1273V1274V1275V1276V1277V1278V1279V1280V1281V1282V1283V1284V1285V1286V1287

V1288V1289V1290V1291

V1292V1293V1294V1295V1296V1297V1298V1299V1300V1301V1302V1303V1304V1305V1306V1307V1308V1309V1310V1311V1312V1313V1314V1315V1316V1317V1318V1319V1320V1321V1322V1323V1324V1325V1326V1327V1328V1329V1330V1331V1332V1333V1334V1335V1336V1337V1338V1339V1340V1341V1342V1343V1344V1345V1346V1347V1348

V1349V1350V1351V1352V1353V1354V1355V1356V1357V1358V1359

V1360

V1361V1362V1363

V1364V1365V1366

V1367V1368V1369V1370V1371V1372V1373

V1374V1375V1376V1377V1378V1379V1380V1381

V1382V1383V1384

V1385V1386V1387V1388V1389V1390V1391V1392V1393V1394V1395V1396V1397V1398V1399V1400V1401V1402V1403V1404V1405V1406V1407V1408V1409V1410V1411V1412V1413V1414V1415V1416V1417V1418V1419V1420V1421V1422V1423V1424V1425V1426V1427V1428V1429V1430V1431V1432V1433V1434V1435V1436V1437V1438V1439V1440V1441V1442V1443V1444

V1445V1446V1447V1448V1449V1450V1451V1452V1453V1454V1455V1456V1457V1458V1459V1460V1461V1462V1463V1464

V1465V1466V1467V1468V1469

V1470

V1471V1472

V1473V1474

V1475V1476V1477V1478V1479V1480V1481V1482V1483

V1484

V1485V1486V1487V1488V1489V1490V1491V1492V1493V1494V1495V1496V1497V1498V1499V1500V1501V1502V1503V1504V1505V1506V1507V1508V1509V1510V1511V1512V1513V1514V1515V1516V1517V1518V1519V1520V1521V1522V1523V1524V1525V1526V1527V1528V1529V1530V1531V1532V1533V1534V1535V1536V1537V1538V1539V1540

V1541V1542V1543V1544V1545V1546V1547V1548V1549V1550V1551V1552

V1553V1554

V1555V1556V1557V1558

V1559

V1560V1561V1562V1563V1564V1565V1566V1567V1568V1569V1570V1571V1572V1573V1574V1575V1576V1577V1578V1579

V1580

V1581V1582V1583V1584V1585V1586V1587V1588V1589V1590V1591V1592V1593V1594V1595V1596V1597V1598V1599V1600V1601V1602V1603V1604V1605V1606V1607V1608V1609V1610V1611V1612V1613V1614V1615V1616V1617V1618V1619V1620V1621V1622V1623V1624V1625V1626V1627V1628V1629V1630V1631V1632V1633V1634V1635V1636

V1637V1638V1639V1640V1641V1642V1643V1644V1645V1646V1647V1648V1649V1650V1651V1652V1653V1654V1655V1656V1657V1658V1659V1660V1661V1662V1663V1664V1665V1666V1667V1668V1669V1670V1671V1672V1673V1674

V1675V1676

V1677V1678V1679V1680V1681V1682V1683V1684V1685V1686V1687V1688V1689V1690V1691V1692V1693V1694V1695V1696V1697V1698V1699V1700V1701V1702V1703V1704V1705V1706V1707V1708V1709V1710V1711V1712V1713V1714V1715V1716V1717V1718V1719V1720V1721V1722V1723V1724V1725V1726V1727V1728V1729V1730V1731V1732

V1733V1734V1735V1736V1737V1738V1739V1740V1741V1742V1743V1744

V1745V1746V1747V1748V1749V1750V1751V1752V1753V1754

V1755V1756V1757

V1758

V1759V1760V1761V1762V1763V1764V1765V1766V1767V1768V1769V1770V1771

V1772

V1773V1774V1775V1776V1777V1778V1779V1780V1781V1782V1783V1784V1785V1786V1787V1788V1789V1790V1791V1792V1793V1794V1795V1796V1797V1798V1799V1800V1801V1802V1803V1804V1805V1806V1807V1808V1809V1810V1811V1812V1813V1814V1815V1816V1817V1818V1819V1820V1821V1822V1823V1824V1825V1826V1827V1828

V1829

V1830V1831V1832V1833V1834V1835V1836V1837V1838V1839V1840V1841V1842V1843V1844V1845V1846V1847V1848V1849V1850V1851V1852V1853V1854V1855V1856V1857V1858V1859V1860V1861V1862V1863V1864V1865V1866V1867

V1868

V1869V1870V1871V1872V1873V1874V1875V1876V1877V1878V1879V1880V1881V1882V1883V1884V1885V1886V1887V1888V1889V1890V1891V1892V1893V1894V1895V1896V1897V1898V1899V1900V1901V1902V1903V1904V1905V1906V1907V1908V1909V1910V1911V1912V1913V1914V1915V1916V1917V1918V1919V1920V1921V1922V1923V1924V1925V1926V1927V1928V1929V1930V1931V1932V1933V1934V1935V1936V1937V1938V1939V1940V1941V1942V1943V1944V1945V1946V1947V1948V1949V1950V1951V1952V1953V1954V1955V1956V1957V1958V1959V1960V1961V1962V1963

V1964V1965V1966V1967V1968V1969V1970V1971V1972V1973V1974V1975V1976V1977V1978V1979V1980V1981V1982V1983V1984V1985V1986V1987V1988V1989V1990V1991V1992V1993V1994V1995V1996V1997V1998V1999V2000V2001V2002V2003V2004V2005V2006V2007V2008V2009V2010V2011V2012V2013V2014V2015V2016V2017V2018V2019V2020V2021

V2022V2023V2024V2025V2026V2027V2028V2029V2030V2031V2032V2033V2034V2035V2036V2037V2038V2039V2040V2041V2042V2043V2044V2045V2046V2047V2048V2049V2050V2051V2052V2053V2054V2055V2056V2057V2058V2059

V2060V2061V2062V2063V2064V2065V2066V2067V2068V2069V2070V2071V2072V2073V2074V2075V2076V2077V2078V2079V2080V2081V2082V2083V2084V2085V2086V2087V2088V2089V2090V2091V2092V2093V2094V2095V2096V2097V2098V2099V2100V2101V2102V2103V2104V2105V2106V2107V2108V2109V2110V2111V2112V2113V2114V2115V2116V2117V2118V2119V2120V2121V2122V2123V2124V2125V2126V2127V2128V2129V2130V2131V2132V2133V2134V2135V2136V2137V2138V2139V2140V2141V2142V2143V2144V2145V2146V2147V2148V2149V2150V2151V2152V2153V2154V2155V2156

V2157V2158V2159V2160V2161V2162V2163V2164V2165V2166V2167V2168V2169V2170V2171V2172V2173V2174V2175V2176V2177V2178V2179V2180V2181V2182V2183V2184V2185V2186V2187V2188V2189V2190V2191V2192V2193V2194V2195V2196V2197V2198V2199V2200V2201V2202V2203V2204V2205V2206V2207V2208V2209V2210V2211V2212

V2213

V2214V2215V2216V2217V2218V2219V2220V2221V2222V2223V2224V2225V2226V2227V2228V2229V2230V2231V2232V2233V2234V2235V2236V2237V2238V2239V2240V2241V2242V2243V2244V2245V2246V2247V2248V2249V2250V2251

V2252

V2253V2254V2255V2256V2257V2258V2259V2260V2261V2262V2263V2264V2265V2266V2267V2268V2269V2270V2271V2272V2273V2274V2275V2276V2277V2278V2279V2280V2281V2282V2283V2284V2285V2286V2287V2288V2289V2290V2291V2292V2293V2294V2295V2296V2297V2298V2299V2300V2301V2302V2303V2304V2305V2306V2307V2308

V2309V2310V2311V2312V2313V2314V2315V2316V2317V2318V2319V2320V2321V2322V2323V2324V2325V2326V2327V2328V2329V2330V2331V2332V2333V2334V2335V2336V2337V2338V2339V2340V2341V2342V2343V2344V2345V2346V2347V2348

V2349V2350V2351V2352V2353V2354V2355V2356V2357V2358V2359V2360V2361V2362V2363V2364V2365V2366V2367V2368V2369V2370V2371V2372V2373V2374V2375V2376V2377V2378V2379V2380V2381V2382V2383V2384V2385V2386V2387V2388V2389V2390V2391V2392V2393V2394V2395V2396V2397V2398V2399V2400V2401V2402V2403V2404

V2405V2406V2407V2408V2409V2410V2411V2412V2413V2414V2415V2416V2417

V2418V2419V2420V2421V2422V2423V2424V2425V2426V2427V2428V2429V2430V2431V2432V2433V2434V2435V2436V2437V2438V2439V2440V2441V2442V2443V2444

V2445V2446V2447V2448V2449V2450V2451V2452V2453V2454V2455V2456V2457V2458V2459V2460V2461V2462V2463V2464V2465V2466V2467V2468V2469V2470V2471V2472V2473V2474V2475V2476V2477V2478V2479V2480V2481V2482V2483V2484V2485V2486V2487V2488V2489V2490V2491V2492V2493V2494V2495V2496V2497V2498V2499V2500

V2501V2502V2503V2504V2505V2506V2507V2508V2509V2510V2511V2512V2513V2514V2515V2516V2517V2518V2519V2520V2521V2522V2523V2524V2525V2526V2527V2528V2529V2530V2531V2532V2533V2534V2535V2536V2537V2538V2539V2540

V2541V2542V2543V2544V2545V2546V2547V2548V2549V2550V2551V2552V2553V2554V2555V2556V2557V2558V2559V2560V2561V2562V2563V2564V2565V2566V2567V2568V2569V2570V2571V2572V2573V2574V2575V2576V2577V2578V2579V2580V2581V2582V2583V2584V2585V2586V2587V2588V2589V2590V2591V2592V2593V2594V2595V2596

V2597

V2598V2599V2600V2601V2602V2603V2604V2605V2606V2607V2608V2609V2610V2611

V2612V2613

V2614V2615

V2616V2617V2618

V2619V2620V2621V2622V2623V2624V2625

V2626V2627V2628V2629V2630V2631V2632V2633V2634V2635

V2636

V2637V2638V2639V2640V2641V2642V2643V2644V2645V2646V2647V2648V2649V2650V2651V2652V2653V2654V2655V2656V2657V2658V2659V2660V2661V2662V2663V2664V2665V2666V2667V2668V2669V2670V2671V2672V2673V2674V2675V2676V2677V2678V2679V2680V2681V2682V2683V2684V2685V2686V2687V2688V2689V2690V2691V2692

V2693V2694V2695V2696V2697V2698V2699V2700V2701V2702V2703V2704V2705V2706V2707V2708V2709V2710V2711V2712V2713V2714V2715V2716V2717V2718V2719V2720V2721V2722V2723V2724V2725V2726V2727V2728V2729V2730V2731

V2732

V2733V2734V2735V2736V2737V2738V2739V2740V2741V2742V2743V2744V2745V2746V2747V2748V2749V2750V2751V2752V2753V2754V2755V2756V2757V2758V2759V2760V2761V2762V2763V2764V2765V2766V2767V2768V2769V2770V2771V2772V2773V2774V2775V2776V2777V2778V2779V2780V2781V2782V2783V2784V2785V2786V2787V2788

V2789

V2790V2791V2792V2793V2794V2795V2796V2797V2798V2799V2800V2801V2802V2803V2804V2805V2806V2807V2808V2809V2810V2811V2812V2813V2814V2815V2816V2817V2818V2819V2820V2821V2822V2823V2824V2825V2826V2827V2828

V2829V2830V2831V2832V2833V2834V2835V2836V2837V2838V2839V2840V2841V2842V2843V2844V2845V2846V2847V2848V2849V2850V2851V2852V2853V2854V2855V2856V2857V2858V2859V2860V2861V2862V2863V2864V2865V2866V2867V2868V2869V2870V2871V2872V2873V2874V2875V2876V2877V2878V2879V2880V2881V2882V2883V2884

V2885

V2886V2887V2888V2889V2890V2891V2892V2893V2894V2895V2896V2897V2898V2899V2900V2901V2902V2903V2904V2905V2906V2907V2908V2909V2910V2911V2912V2913V2914V2915V2916V2917V2918V2919V2920V2921V2922V2923V2924

V2925V2926V2927V2928V2929V2930V2931V2932V2933V2934V2935V2936V2937V2938V2939V2940V2941V2942V2943V2944V2945V2946V2947V2948V2949V2950V2951V2952V2953V2954V2955V2956V2957V2958V2959V2960V2961V2962V2963V2964V2965V2966V2967V2968V2969V2970V2971V2972V2973V2974V2975V2976V2977V2978V2979V2980

V2981

V2982V2983V2984V2985V2986V2987V2988V2989V2990V2991V2992V2993V2994V2995V2996V2997V2998V2999V3000V3001V3002V3003V3004V3005V3006V3007V3008V3009V3010V3011V3012V3013V3014V3015V3016V3017V3018V3019V3020

V3021V3022V3023V3024V3025V3026V3027V3028V3029V3030V3031V3032V3033V3034V3035V3036V3037V3038V3039V3040V3041V3042V3043V3044V3045V3046V3047V3048V3049V3050V3051V3052V3053V3054V3055V3056V3057V3058V3059V3060V3061V3062V3063V3064V3065V3066V3067V3068V3069V3070V3071V3072V3073V3074V3075V3076

V3077

V3078V3079V3080V3081V3082V3083V3084V3085V3086V3087V3088V3089V3090V3091V3092V3093V3094V3095V3096V3097V3098V3099V3100V3101V3102V3103V3104V3105V3106V3107V3108V3109V3110V3111V3112V3113V3114V3115

V3116

V3117V3118V3119V3120V3121V3122V3123V3124V3125V3126V3127V3128V3129V3130V3131V3132V3133V3134V3135V3136V3137V3138V3139V3140V3141V3142V3143V3144V3145V3146V3147V3148V3149V3150V3151V3152V3153V3154V3155V3156V3157V3158V3159V3160V3161V3162V3163V3164V3165V3166V3167V3168V3169V3170V3171V3172

V3173

V3174V3175V3176V3177V3178V3179V3180V3181V3182V3183V3184V3185V3186V3187V3188V3189V3190V3191V3192V3193V3194V3195V3196V3197V3198V3199V3200

V3201V3202V3203V3204V3205V3206V3207V3208V3209V3210V3211V3212

V3213

V3214V3215V3216V3217V3218V3219V3220V3221V3222V3223V3224V3225V3226V3227V3228V3229V3230V3231V3232V3233V3234V3235V3236V3237V3238V3239V3240V3241V3242V3243V3244V3245V3246V3247V3248V3249V3250V3251V3252V3253V3254V3255V3256V3257V3258V3259V3260V3261V3262V3263V3264V3265V3266V3267V3268

V3269V3270V3271V3272V3273V3274V3275V3276V3277V3278V3279V3280V3281V3282V3283V3284V3285V3286V3287V3288V3289

V3290V3291V3292V3293V3294V3295V3296V3297V3298V3299V3300V3301V3302V3303V3304V3305V3306V3307V3308

V3309

V3310V3311V3312V3313V3314V3315V3316V3317V3318V3319V3320V3321V3322V3323V3324V3325V3326V3327V3328V3329V3330V3331V3332V3333V3334V3335V3336V3337V3338V3339V3340V3341V3342V3343V3344V3345V3346V3347V3348V3349V3350V3351V3352V3353V3354V3355V3356V3357V3358V3359V3360V3361V3362V3363V3364

V3365

V3366V3367V3368V3369V3370V3371V3372V3373V3374V3375V3376V3377V3378V3379V3380V3381V3382V3383V3384V3385V3386V3387V3388V3389V3390V3391V3392V3393V3394V3395V3396V3397V3398V3399V3400V3401V3402V3403V3404

V3405

V3406V3407V3408V3409V3410V3411V3412V3413V3414V3415V3416V3417V3418V3419V3420V3421V3422V3423V3424V3425V3426V3427V3428V3429V3430V3431V3432V3433V3434V3435V3436V3437V3438V3439V3440V3441V3442V3443V3444V3445V3446V3447V3448V3449V3450V3451V3452V3453V3454V3455V3456V3457V3458V3459V3460

V3461V3462V3463V3464V3465V3466V3467V3468V3469V3470V3471V3472

V3473V3474V3475V3476

V3477V3478V3479V3480V3481V3482

V3483V3484V3485V3486V3487V3488V3489V3490

V3491

V3492

V3493V3494

V3495V3496V3497V3498V3499V3500

V3501

V3502V3503V3504V3505V3506V3507V3508V3509V3510V3511V3512V3513V3514V3515V3516V3517V3518V3519V3520V3521V3522V3523V3524V3525V3526V3527V3528V3529V3530V3531V3532V3533V3534V3535V3536V3537V3538V3539V3540V3541V3542V3543V3544V3545V3546V3547V3548V3549V3550V3551V3552V3553V3554V3555

V3556V3557V3558V3559V3560V3561V3562V3563V3564

V3565V3566

V3567V3568V3569V3570V3571V3572V3573V3574V3575V3576V3577

V3578V3579V3580V3581V3582V3583V3584V3585V3586V3587V3588V3589V3590V3591V3592V3593V3594V3595V3596

V3597

V3598V3599V3600V3601V3602V3603V3604V3605V3606V3607V3608V3609V3610V3611V3612V3613V3614V3615V3616V3617V3618V3619V3620V3621V3622V3623V3624V3625V3626V3627V3628V3629V3630V3631V3632V3633V3634V3635V3636V3637V3638V3639V3640V3641V3642V3643V3644V3645V3646V3647V3648V3649V3650V3651

V3652V3653V3654V3655V3656V3657V3658V3659V3660

V3661V3662

V3663

V3664V3665V3666V3667V3668V3669

V3670V3671V3672V3673V3674V3675V3676V3677V3678V3679V3680V3681V3682V3683V3684V3685V3686V3687V3688V3689V3690V3691V3692

V3693V3694V3695V3696V3697V3698V3699V3700V3701V3702V3703V3704V3705V3706V3707V3708V3709V3710V3711V3712V3713V3714V3715V3716V3717V3718V3719V3720V3721V3722V3723V3724V3725V3726V3727V3728V3729V3730V3731V3732V3733V3734V3735V3736V3737V3738V3739V3740V3741V3742V3743V3744V3745V3746V3747V3748

V3749V3750V3751V3752V3753V3754V3755V3756V3757V3758V3759V3760V3761V3762V3763V3764V3765V3766V3767V3768V3769V3770V3771V3772V3773V3774V3775V3776V3777V3778V3779V3780V3781V3782V3783V3784V3785V3786V3787V3788

V3789

V3790V3791V3792V3793V3794V3795V3796V3797V3798V3799V3800V3801V3802V3803V3804V3805V3806V3807V3808V3809V3810V3811V3812V3813V3814V3815V3816V3817V3818V3819V3820V3821V3822V3823V3824V3825V3826V3827V3828V3829V3830V3831V3832V3833V3834V3835V3836V3837V3838V3839V3840V3841V3842V3843

V3844V3845V3846V3847V3848V3849V3850V3851V3852V3853V3854V3855V3856V3857V3858V3859V3860V3861V3862V3863V3864V3865V3866V3867V3868V3869V3870V3871V3872V3873V3874V3875V3876V3877V3878V3879V3880V3881V3882V3883V3884V3885

V3886V3887V3888V3889V3890V3891V3892V3893V3894V3895V3896V3897V3898V3899V3900V3901V3902V3903V3904V3905V3906V3907V3908V3909V3910V3911V3912V3913V3914V3915V3916V3917V3918V3919V3920V3921V3922V3923V3924V3925V3926V3927V3928V3929V3930V3931V3932V3933V3934V3935V3936V3937V3938V3939

V3940V3941V3942V3943V3944V3945V3946V3947V3948V3949V3950V3951V3952V3953V3954V3955V3956V3957V3958V3959V3960V3961V3962V3963V3964V3965V3966V3967V3968V3969V3970V3971V3972V3973V3974V3975V3976V3977V3978V3979V3980V3981

V3982V3983V3984V3985V3986V3987V3988V3989V3990V3991V3992V3993V3994V3995V3996V3997V3998V3999V4000V4001V4002V4003V4004V4005V4006V4007V4008V4009V4010V4011V4012V4013V4014V4015V4016V4017V4018V4019V4020V4021V4022V4023V4024V4025V4026V4027V4028V4029V4030V4031V4032V4033V4034V4035

V4036V4037V4038V4039V4040V4041V4042V4043V4044V4045V4046V4047V4048V4049V4050V4051V4052V4053V4054V4055V4056V4057V4058V4059V4060

V4061V4062V4063V4064V4065V4066V4067V4068V4069V4070V4071V4072V4073V4074V4075V4076

V4077

V4078V4079V4080V4081V4082V4083V4084V4085V4086V4087V4088V4089V4090V4091V4092V4093V4094V4095V4096V4097V4098V4099V4100V4101V4102V4103V4104V4105V4106V4107V4108V4109V4110V4111V4112V4113V4114V4115V4116V4117V4118V4119V4120V4121V4122V4123V4124V4125V4126V4127V4128V4129V4130V4131

V4132

V4133V4134V4135V4136V4137V4138V4139V4140V4141V4142V4143V4144V4145V4146V4147

V4148V4149V4150V4151V4152V4153V4154V4155V4156V4157V4158V4159

V4160V4161V4162

V4163V4164V4165V4166

V4167V4168V4169V4170V4171V4172V4173

V4174V4175V4176V4177V4178V4179V4180V4181V4182V4183V4184V4185V4186V4187V4188V4189V4190V4191V4192V4193V4194V4195V4196V4197V4198V4199V4200V4201V4202V4203V4204V4205V4206V4207V4208V4209V4210V4211V4212V4213V4214V4215V4216V4217V4218V4219V4220V4221V4222V4223V4224V4225V4226V4227

V4228

V4229V4230V4231V4232V4233V4234V4235V4236V4237V4238V4239V4240V4241V4242V4243V4244V4245V4246V4247V4248V4249V4250V4251

V4252V4253

V4254V4255V4256V4257V4258V4259V4260V4261V4262V4263V4264V4265V4266V4267V4268

V4269

V4270V4271V4272V4273V4274V4275V4276V4277V4278V4279V4280V4281V4282V4283V4284V4285V4286V4287V4288V4289V4290V4291V4292V4293V4294V4295V4296V4297V4298V4299V4300V4301V4302V4303V4304V4305V4306V4307V4308V4309V4310V4311V4312V4313V4314V4315V4316V4317V4318V4319V4320V4321V4322V4323V4324

V4325V4326V4327V4328V4329V4330V4331V4332V4333V4334V4335V4336V4337V4338V4339V4340V4341V4342V4343V4344V4345V4346V4347V4348V4349V4350V4351V4352V4353V4354V4355V4356V4357V4358V4359V4360V4361V4362V4363V4364

V4365V4366V4367V4368V4369V4370V4371V4372V4373V4374V4375V4376V4377V4378V4379V4380V4381V4382V4383V4384V4385V4386V4387V4388V4389V4390V4391V4392V4393V4394V4395V4396V4397V4398V4399V4400V4401V4402V4403V4404V4405V4406V4407V4408V4409V4410V4411V4412V4413V4414V4415V4416V4417V4418V4419

V4420V4421V4422V4423V4424V4425V4426V4427V4428V4429V4430V4431V4432V4433V4434

V4435V4436V4437V4438

V4439V4440

V4441

V4442V4443V4444V4445

V4446

V4447V4448V4449V4450V4451V4452V4453V4454V4455V4456V4457V4458V4459V4460

V4461

V4462V4463V4464V4465V4466V4467V4468V4469V4470V4471V4472V4473V4474V4475V4476V4477V4478V4479V4480V4481V4482V4483V4484V4485V4486V4487V4488V4489V4490V4491V4492V4493V4494V4495V4496V4497V4498V4499V4500V4501V4502V4503V4504V4505V4506V4507V4508V4509V4510V4511V4512V4513V4514V4515

V4516V4517V4518V4519V4520V4521V4522V4523V4524V4525V4526V4527V4528V4529V4530V4531V4532

V4533

V4534V4535V4536V4537V4538V4539V4540

V4541V4542

V4543

V4544V4545V4546

V4547V4548V4549V4550V4551V4552V4553V4554V4555V4556V4557

V4558

V4559V4560V4561V4562V4563V4564V4565V4566V4567V4568V4569V4570V4571V4572V4573V4574V4575V4576V4577V4578V4579V4580V4581V4582V4583V4584V4585V4586V4587V4588V4589V4590V4591V4592V4593V4594V4595V4596V4597V4598V4599V4600V4601V4602V4603V4604V4605V4606V4607V4608V4609V4610V4611

V4612V4613V4614V4615V4616V4617V4618V4619

V4620V4621V4622V4623V4624V4625V4626V4627

V4628V4629

V4630V4631V4632V4633V4634V4635V4636V4637V4638

V4639V4640V4641V4642V4643V4644V4645V4646V4647V4648V4649V4650V4651V4652V4653

V4654

V4655V4656V4657V4658V4659V4660V4661V4662V4663V4664V4665V4666V4667V4668V4669V4670V4671V4672V4673V4674V4675V4676V4677V4678V4679V4680V4681V4682V4683V4684V4685V4686V4687V4688V4689V4690V4691V4692V4693V4694V4695V4696V4697V4698V4699V4700V4701V4702V4703V4704V4705V4706V4707

V4708V4709V4710V4711V4712V4713V4714V4715V4716V4717V4718V4719V4720V4721V4722V4723V4724V4725V4726V4727V4728V4729V4730V4731

V4732V4733V4734V4735

V4736V4737

V4738

V4739V4740V4741V4742V4743

V4744

V4745V4746

V4747V4748

V4749V4750V4751V4752V4753V4754V4755V4756V4757V4758V4759V4760V4761V4762V4763V4764V4765V4766V4767V4768V4769V4770V4771V4772V4773V4774V4775V4776V4777V4778V4779V4780V4781V4782V4783V4784V4785V4786V4787V4788V4789V4790V4791V4792V4793V4794V4795V4796V4797V4798V4799V4800V4801V4802V4803

V4804V4805V4806V4807

V4808

V4809V4810V4811V4812V4813V4814V4815V4816V4817V4818V4819V4820V4821V4822V4823

V4824V4825

V4826V4827

V4828V4829V4830

V4831V4832

V4833V4834V4835V4836V4837V4838

V4839

V4840V4841V4842V4843V4844V4845

V4846

V4847V4848V4849V4850V4851V4852V4853V4854V4855V4856V4857V4858V4859V4860V4861V4862V4863V4864V4865V4866V4867V4868V4869V4870V4871V4872V4873V4874V4875V4876V4877V4878V4879V4880V4881V4882V4883V4884V4885V4886V4887V4888V4889V4890V4891V4892V4893V4894V4895V4896V4897V4898

V4899V4900V4901V4902V4903V4904V4905V4906V4907V4908V4909

V4910

V4911

V4912V4913V4914V4915

V4916

V4917V4918

V4919V4920

V4921V4922V4923V4924

V4925

V4926

V4927V4928V4929V4930V4931V4932V4933

V4934V4935V4936V4937V4938V4939V4940V4941

V4942

V4943V4944V4945V4946V4947V4948V4949V4950V4951V4952V4953V4954V4955V4956V4957V4958V4959V4960V4961V4962V4963V4964V4965V4966V4967V4968V4969V4970V4971V4972V4973V4974V4975V4976V4977V4978V4979V4980V4981V4982V4983V4984V4985V4986V4987V4988V4989V4990V4991V4992V4993V4994V4995

V4996V4997V4998V4999V5000V5001V5002V5003V5004V5005V5006V5007V5008V5009V5010V5011V5012V5013V5014V5015V5016V5017V5018V5019V5020V5021

V5022V5023V5024V5025V5026V5027V5028

V5029V5030V5031V5032V5033V5034V5035V5036V5037

V5038

V5039V5040V5041V5042V5043V5044V5045V5046V5047V5048V5049V5050V5051V5052V5053V5054V5055V5056V5057V5058V5059V5060V5061V5062V5063V5064V5065V5066V5067V5068V5069V5070V5071V5072V5073V5074V5075V5076V5077V5078V5079V5080V5081V5082V5083V5084V5085V5086V5087V5088V5089V5090

V5091V5092V5093V5094V5095V5096V5097V5098V5099V5100V5101V5102V5103V5104V5105V5106V5107V5108V5109V5110V5111V5112V5113V5114V5115V5116V5117V5118V5119V5120V5121V5122V5123V5124V5125V5126V5127V5128V5129V5130V5131V5132V5133V5134

V5135V5136V5137V5138V5139V5140V5141V5142V5143V5144V5145V5146V5147V5148V5149V5150V5151V5152V5153V5154V5155V5156V5157V5158V5159V5160V5161V5162V5163V5164V5165V5166V5167V5168V5169V5170V5171V5172V5173V5174V5175V5176V5177V5178V5179V5180V5181V5182V5183V5184V5185V5186

V5187V5188V5189V5190V5191V5192V5193V5194V5195V5196V5197V5198V5199V5200V5201

V5202V5203

V5204

V5205

V5206V5207V5208V5209V5210

V5211

V5212V5213V5214V5215V5216V5217V5218V5219V5220V5221V5222V5223V5224V5225V5226V5227V5228V5229V5230

V5231V5232V5233V5234V5235V5236V5237V5238V5239V5240V5241V5242V5243V5244V5245V5246V5247V5248V5249V5250V5251V5252V5253V5254V5255V5256V5257V5258V5259V5260V5261V5262V5263V5264V5265V5266V5267V5268V5269V5270V5271V5272V5273V5274V5275V5276V5277V5278V5279V5280V5281V5282

V5283V5284V5285V5286V5287V5288V5289V5290V5291V5292V5293V5294V5295V5296V5297V5298V5299V5300V5301V5302

V5303V5304

V5305

V5306V5307V5308V5309V5310V5311V5312V5313V5314V5315V5316V5317V5318V5319V5320V5321V5322V5323V5324V5325

V5326

V5327V5328V5329V5330V5331V5332V5333V5334V5335V5336V5337V5338V5339V5340V5341V5342V5343V5344V5345V5346V5347V5348V5349V5350V5351V5352V5353V5354V5355V5356V5357V5358V5359V5360V5361V5362V5363V5364V5365V5366V5367V5368V5369V5370V5371V5372V5373V5374V5375V5376V5377V5378

V5379V5380V5381V5382V5383V5384V5385V5386V5387V5388V5389V5390V5391V5392V5393V5394V5395V5396V5397V5398V5399V5400V5401V5402V5403V5404V5405V5406V5407V5408V5409V5410V5411V5412V5413V5414V5415V5416V5417V5418V5419V5420V5421

V5422

V5423V5424V5425V5426V5427V5428V5429V5430V5431V5432V5433V5434V5435V5436V5437V5438V5439V5440V5441V5442V5443V5444V5445V5446V5447V5448V5449V5450V5451V5452V5453V5454V5455V5456V5457V5458V5459V5460V5461V5462V5463V5464V5465V5466V5467V5468V5469V5470V5471V5472V5473V5474

V5475

V5476V5477V5478V5479V5480V5481V5482V5483V5484V5485V5486V5487V5488V5489V5490V5491V5492V5493V5494V5495V5496V5497V5498V5499V5500V5501V5502V5503V5504V5505V5506V5507V5508V5509V5510V5511V5512V5513V5514V5515V5516V5517

V5518

V5519V5520V5521V5522V5523V5524V5525V5526V5527V5528V5529V5530V5531V5532V5533V5534V5535V5536V5537V5538V5539V5540V5541V5542V5543V5544V5545V5546V5547V5548V5549V5550V5551V5552V5553V5554V5555V5556V5557V5558V5559V5560V5561V5562V5563V5564V5565V5566V5567V5568V5569V5570

V5571V5572V5573V5574V5575V5576V5577V5578V5579V5580V5581V5582V5583V5584V5585V5586V5587V5588V5589

V5590V5591V5592

V5593

V5594V5595V5596V5597V5598V5599V5600V5601V5602V5603V5604V5605V5606V5607V5608V5609V5610V5611V5612V5613

V5614

V5615V5616V5617V5618V5619V5620V5621V5622V5623V5624V5625V5626V5627V5628V5629V5630V5631V5632V5633V5634V5635V5636V5637V5638V5639V5640V5641V5642V5643V5644V5645V5646V5647V5648V5649V5650V5651V5652V5653V5654V5655V5656V5657V5658V5659V5660V5661V5662V5663V5664V5665V5666V5667

V5668V5669V5670V5671V5672V5673V5674V5675V5676V5677V5678V5679V5680V5681V5682V5683V5684V5685V5686V5687V5688V5689V5690V5691V5692V5693V5694V5695V5696V5697V5698V5699V5700V5701V5702V5703V5704V5705V5706V5707V5708V5709V5710

V5711V5712V5713V5714V5715V5716V5717V5718V5719V5720V5721V5722V5723V5724V5725V5726V5727V5728V5729V5730V5731V5732V5733V5734V5735V5736V5737V5738V5739V5740V5741V5742V5743V5744V5745V5746V5747V5748V5749V5750V5751V5752V5753V5754V5755V5756V5757V5758V5759V5760V5761V5762

V5763

V5764V5765V5766V5767V5768V5769V5770V5771V5772V5773V5774V5775V5776V5777V5778V5779V5780V5781

V5782V5783V5784

V5785

V5786V5787V5788V5789V5790V5791V5792V5793V5794V5795V5796V5797V5798V5799V5800V5801V5802V5803V5804V5805V5806

V5807V5808V5809V5810V5811V5812V5813V5814V5815V5816V5817V5818V5819V5820V5821V5822V5823V5824V5825V5826V5827V5828V5829V5830V5831V5832V5833V5834V5835V5836V5837V5838V5839V5840V5841V5842V5843V5844V5845V5846V5847V5848V5849V5850V5851V5852V5853V5854V5855V5856V5857V5858

V5859

V5860V5861V5862V5863V5864V5865V5866V5867V5868V5869V5870V5871V5872V5873V5874V5875V5876V5877V5878V5879V5880V5881V5882V5883V5884V5885V5886V5887V5888V5889V5890V5891V5892V5893V5894V5895V5896V5897V5898V5899V5900V5901

V5902

V5903V5904V5905V5906V5907V5908V5909V5910V5911V5912V5913V5914V5915V5916V5917V5918V5919V5920V5921V5922V5923V5924V5925V5926V5927V5928V5929V5930V5931V5932V5933V5934V5935V5936V5937V5938V5939V5940V5941V5942V5943V5944V5945V5946V5947V5948V5949V5950V5951V5952V5953V5954

V5955V5956V5957V5958V5959V5960V5961V5962V5963V5964

V5965V5966V5967V5968

V5969V5970V5971

V5972

V5973

V5974

V5975V5976

V5977V5978V5979V5980V5981V5982V5983V5984V5985V5986V5987V5988V5989V5990V5991V5992V5993V5994V5995V5996V5997

V5998

V5999V6000V6001V6002V6003V6004V6005V6006V6007V6008V6009V6010V6011V6012V6013V6014V6015V6016V6017V6018V6019V6020V6021V6022V6023V6024V6025V6026V6027V6028V6029V6030V6031V6032V6033V6034V6035V6036V6037V6038V6039V6040V6041V6042V6043V6044V6045V6046V6047V6048V6049V6050

V6051

V6052V6053V6054V6055V6056V6057V6058V6059V6060V6061V6062V6063V6064V6065V6066V6067V6068V6069V6070V6071V6072V6073

V6074V6075V6076V6077V6078V6079V6080V6081V6082V6083V6084V6085V6086V6087V6088V6089V6090V6091V6092V6093V6094

V6095V6096V6097V6098V6099V6100V6101V6102V6103V6104V6105V6106V6107V6108V6109V6110V6111V6112V6113V6114V6115V6116V6117V6118V6119V6120V6121V6122V6123V6124V6125V6126V6127V6128V6129V6130V6131V6132V6133V6134V6135V6136V6137V6138V6139V6140V6141V6142V6143V6144V6145V6146

V6147

V6148V6149V6150V6151V6152V6153V6154V6155V6156V6157V6158V6159V6160V6161V6162V6163V6164V6165V6166V6167V6168V6169V6170V6171V6172V6173V6174V6175V6176V6177V6178V6179V6180V6181V6182V6183V6184V6185V6186V6187V6188V6189V6190

V6191V6192V6193V6194V6195V6196V6197V6198V6199V6200V6201V6202V6203V6204V6205V6206V6207V6208V6209V6210V6211V6212V6213V6214V6215V6216V6217V6218V6219V6220V6221V6222V6223V6224V6225V6226V6227V6228V6229V6230V6231V6232V6233V6234V6235V6236V6237V6238V6239V6240V6241V6242

V6243V6244V6245V6246V6247V6248V6249V6250V6251V6252V6253V6254V6255V6256V6257V6258V6259V6260V6261V6262V6263V6264V6265V6266V6267V6268V6269V6270V6271V6272V6273V6274V6275V6276V6277V6278V6279V6280V6281V6282V6283V6284V6285V6286

V6287V6288V6289V6290V6291V6292V6293V6294V6295V6296V6297V6298V6299V6300V6301V6302V6303V6304V6305V6306V6307V6308V6309V6310V6311V6312V6313V6314V6315V6316V6317V6318V6319V6320V6321V6322V6323V6324V6325V6326V6327V6328V6329V6330V6331V6332V6333V6334V6335V6336V6337V6338

V6339V6340V6341V6342V6343V6344V6345V6346V6347V6348V6349V6350V6351V6352V6353V6354V6355V6356V6357V6358V6359V6360V6361V6362V6363V6364V6365V6366V6367V6368V6369V6370V6371V6372V6373V6374V6375V6376V6377V6378V6379V6380V6381

V6382V6383

V6384V6385V6386V6387V6388V6389V6390V6391V6392V6393V6394V6395V6396V6397V6398V6399V6400V6401V6402V6403V6404V6405V6406V6407V6408V6409V6410V6411V6412V6413V6414V6415V6416V6417V6418V6419V6420V6421V6422V6423V6424V6425V6426V6427V6428V6429V6430V6431V6432V6433V6434

V6435V6436V6437V6438V6439V6440V6441V6442V6443V6444V6445V6446V6447V6448V6449V6450V6451V6452V6453V6454V6455V6456V6457V6458V6459V6460V6461V6462V6463V6464V6465V6466V6467V6468V6469V6470V6471V6472V6473V6474V6475V6476V6477V6478

V6479

V6480V6481V6482V6483V6484V6485V6486V6487V6488V6489V6490V6491V6492V6493V6494V6495V6496V6497V6498V6499V6500V6501V6502V6503V6504V6505V6506V6507V6508V6509V6510V6511V6512V6513V6514V6515V6516V6517V6518V6519V6520V6521V6522V6523V6524V6525V6526V6527V6528V6529

V6530V6531V6532V6533V6534V6535V6536V6537V6538V6539V6540V6541V6542V6543V6544V6545V6546V6547V6548V6549V6550V6551V6552V6553V6554V6555V6556V6557V6558V6559V6560V6561V6562V6563V6564V6565V6566V6567V6568V6569V6570V6571V6572V6573V6574

V6575

V6576V6577V6578V6579V6580V6581V6582V6583V6584V6585V6586V6587V6588V6589V6590V6591V6592V6593V6594V6595V6596V6597V6598V6599V6600V6601V6602V6603V6604V6605V6606V6607V6608V6609V6610V6611V6612V6613V6614V6615V6616V6617V6618V6619V6620V6621V6622V6623V6624V6625V6626

V6627V6628V6629V6630V6631V6632V6633V6634V6635V6636V6637V6638V6639V6640V6641V6642V6643V6644V6645V6646V6647V6648V6649V6650V6651V6652V6653V6654V6655V6656V6657V6658V6659V6660V6661V6662V6663V6664V6665V6666V6667V6668V6669V6670

V6671

V6672V6673V6674V6675V6676V6677V6678V6679V6680V6681V6682V6683V6684V6685V6686V6687V6688V6689V6690V6691V6692V6693V6694V6695V6696V6697V6698V6699V6700V6701V6702V6703V6704V6705V6706V6707V6708V6709V6710V6711V6712V6713V6714V6715V6716V6717V6718V6719V6720V6721V6722

V6723V6724V6725V6726V6727V6728V6729V6730V6731V6732V6733V6734V6735

V6736V6737V6738V6739

V6740V6741V6742V6743V6744V6745

V6746

V6747V6748V6749V6750V6751V6752V6753V6754

V6755V6756V6757V6758V6759V6760V6761V6762V6763V6764V6765V6766

V6767

V6768V6769V6770V6771V6772V6773V6774V6775V6776V6777V6778V6779V6780V6781V6782V6783V6784V6785V6786V6787V6788V6789V6790V6791V6792V6793V6794V6795V6796V6797V6798V6799V6800V6801V6802V6803V6804V6805V6806V6807V6808V6809V6810V6811V6812V6813V6814V6815V6816V6817V6818V6819V6820V6821V6822V6823V6824V6825V6826V6827V6828V6829V6830V6831V6832V6833V6834V6835V6836V6837V6838V6839V6840V6841V6842V6843V6844V6845V6846V6847V6848V6849V6850V6851V6852V6853V6854V6855V6856V6857V6858V6859V6860V6861V6862V6863V6864V6865V6866V6867V6868V6869V6870V6871V6872V6873V6874V6875V6876V6877V6878V6879V6880V6881V6882V6883V6884V6885V6886V6887V6888V6889V6890V6891V6892V6893V6894V6895V6896V6897V6898V6899V6900V6901V6902V6903V6904V6905V6906V6907V6908V6909V6910V6911V6912V6913V6914

V6915V6916V6917V6918V6919V6920V6921V6922V6923V6924V6925V6926V6927V6928V6929V6930V6931V6932V6933V6934V6935V6936V6937V6938V6939

V6940V6941V6942V6943V6944V6945V6946V6947V6948V6949V6950V6951V6952V6953V6954V6955V6956V6957V6958

V6959

V6960V6961V6962V6963V6964V6965V6966V6967V6968V6969V6970V6971V6972V6973V6974V6975V6976V6977V6978V6979V6980V6981V6982V6983V6984V6985V6986V6987V6988V6989V6990V6991V6992V6993V6994V6995V6996V6997V6998V6999V7000V7001V7002V7003V7004V7005V7006V7007V7008V7009

V7010

V7011V7012V7013V7014V7015V7016V7017V7018V7019V7020V7021V7022V7023

V7024

V7025V7026

V7027V7028V7029

V7030

V7031V7032

V7033

V7034V7035V7036V7037V7038V7039V7040V7041V7042V7043V7044V7045V7046V7047V7048V7049V7050V7051V7052V7053V7054

V7055

V7056V7057V7058V7059V7060V7061V7062V7063V7064V7065V7066V7067V7068V7069V7070V7071V7072V7073V7074V7075V7076V7077V7078V7079V7080V7081V7082V7083V7084V7085V7086V7087V7088V7089V7090V7091V7092V7093V7094V7095V7096V7097V7098V7099V7100V7101V7102V7103V7104V7105

V7106

V7107V7108V7109V7110V7111V7112V7113V7114V7115V7116V7117

V7118V7119V7120

V7121

V7122V7123V7124

V7125

V7126

V7127V7128V7129V7130V7131V7132V7133V7134V7135V7136V7137V7138V7139V7140V7141V7142V7143V7144V7145V7146V7147V7148V7149V7150

V7151

V7152V7153V7154V7155V7156V7157V7158V7159V7160V7161V7162V7163V7164V7165V7166V7167V7168V7169V7170V7171V7172V7173V7174V7175V7176V7177V7178V7179V7180V7181V7182V7183V7184V7185V7186V7187V7188V7189V7190V7191V7192V7193V7194V7195V7196V7197V7198V7199V7200V7201

V7202V7203V7204V7205V7206V7207

V7208V7209V7210V7211V7212V7213V7214V7215V7216V7217

V7218

V7219

V7220V7221V7222V7223V7224

V7225

V7226V7227V7228V7229V7230V7231V7232V7233V7234V7235V7236V7237V7238V7239V7240V7241V7242V7243V7244V7245V7246

V7247

V7248V7249V7250V7251V7252V7253V7254V7255V7256V7257V7258V7259V7260V7261V7262V7263V7264V7265V7266V7267V7268V7269V7270V7271V7272V7273V7274V7275V7276V7277V7278V7279V7280V7281V7282V7283V7284V7285V7286V7287V7288V7289V7290V7291V7292V7293V7294V7295V7296V7297

V7298V7299V7300V7301V7302V7303V7304V7305V7306V7307V7308V7309V7310V7311V7312V7313V7314V7315V7316V7317V7318V7319V7320V7321V7322V7323V7324V7325V7326V7327V7328V7329V7330V7331V7332V7333V7334V7335V7336V7337V7338V7339V7340V7341V7342

V7343

V7344V7345V7346V7347V7348V7349V7350V7351V7352V7353V7354V7355V7356V7357V7358V7359V7360V7361V7362V7363V7364V7365V7366V7367V7368V7369V7370V7371V7372V7373V7374V7375V7376V7377V7378V7379V7380V7381V7382V7383V7384V7385V7386V7387V7388V7389V7390V7391V7392V7393

V7394V7395V7396V7397V7398V7399V7400V7401V7402

V7403V7404V7405V7406

V7407V7408V7409V7410V7411V7412V7413V7414V7415V7416V7417V7418V7419V7420V7421V7422V7423

V7424V7425V7426V7427V7428V7429V7430V7431V7432V7433V7434V7435V7436V7437V7438V7439

V7440V7441V7442V7443V7444V7445V7446V7447V7448V7449V7450V7451V7452V7453V7454V7455V7456V7457V7458V7459V7460V7461V7462V7463V7464V7465V7466V7467V7468V7469V7470V7471V7472V7473V7474V7475V7476V7477V7478V7479V7480V7481V7482V7483V7484V7485V7486V7487V7488

V7489V7490V7491V7492V7493V7494V7495V7496V7497V7498V7499V7500V7501

V7502

V7503

V7504V7505V7506V7507V7508V7509

V7510V7511V7512V7513

V7514

V7515V7516V7517V7518V7519V7520V7521V7522V7523V7524V7525V7526V7527V7528V7529V7530V7531V7532V7533V7534V7535

V7536V7537V7538V7539V7540V7541V7542V7543V7544V7545V7546V7547V7548V7549V7550V7551V7552V7553V7554V7555V7556V7557V7558V7559V7560V7561V7562V7563V7564V7565V7566V7567V7568V7569V7570V7571V7572V7573V7574V7575V7576V7577V7578V7579V7580V7581V7582V7583V7584V7585

V7586V7587V7588V7589V7590V7591V7592V7593V7594V7595V7596V7597V7598V7599V7600V7601V7602V7603V7604V7605V7606V7607V7608V7609V7610V7611V7612V7613V7614V7615V7616V7617V7618V7619V7620V7621V7622V7623V7624V7625V7626V7627V7628V7629V7630V7631

V7632V7633V7634V7635V7636V7637V7638V7639V7640V7641V7642V7643V7644V7645V7646V7647V7648V7649V7650V7651V7652V7653V7654V7655V7656V7657V7658V7659V7660V7661V7662V7663V7664V7665V7666V7667V7668V7669V7670V7671V7672V7673V7674V7675V7676V7677V7678V7679V7680

V7681V7682V7683V7684V7685V7686V7687V7688V7689V7690V7691

V7692

V7693V7694V7695V7696V7697V7698V7699V7700V7701V7702V7703V7704V7705V7706V7707V7708V7709V7710V7711V7712

V7713

V7714V7715V7716V7717V7718V7719V7720V7721V7722V7723V7724V7725V7726V7727

V7728V7729V7730V7731V7732V7733V7734V7735V7736V7737V7738V7739V7740V7741V7742V7743V7744V7745V7746V7747V7748V7749V7750V7751V7752V7753V7754V7755V7756V7757V7758V7759V7760V7761V7762V7763V7764V7765V7766V7767V7768V7769V7770V7771V7772V7773V7774V7775V7776

V7777V7778V7779V7780V7781V7782V7783V7784V7785V7786V7787V7788V7789V7790V7791V7792V7793V7794V7795V7796V7797V7798V7799V7800V7801V7802V7803V7804V7805V7806V7807V7808V7809V7810V7811V7812V7813V7814V7815V7816V7817V7818V7819V7820V7821V7822V7823

V7824V7825V7826V7827V7828V7829V7830V7831V7832V7833V7834V7835V7836V7837V7838V7839V7840V7841V7842V7843V7844V7845V7846V7847V7848V7849V7850V7851V7852V7853V7854V7855V7856V7857V7858V7859V7860V7861V7862V7863V7864V7865V7866V7867V7868V7869V7870V7871V7872

V7873V7874V7875V7876V7877V7878V7879V7880V7881V7882V7883V7884V7885V7886V7887V7888V7889V7890V7891V7892V7893V7894V7895V7896V7897V7898V7899V7900V7901V7902V7903V7904V7905V7906V7907V7908V7909V7910V7911V7912V7913V7914V7915V7916V7917V7918

V7919

V7920V7921V7922V7923V7924V7925V7926V7927V7928V7929V7930V7931V7932V7933V7934V7935V7936V7937V7938V7939V7940V7941V7942V7943V7944V7945V7946V7947V7948V7949V7950V7951V7952V7953V7954V7955V7956V7957V7958V7959V7960V7961V7962V7963V7964V7965V7966V7967V7968

V7969V7970V7971V7972V7973V7974V7975V7976V7977V7978

V7979V7980V7981V7982V7983V7984V7985V7986V7987V7988V7989V7990V7991V7992V7993V7994V7995V7996V7997V7998V7999V8000V8001V8002V8003V8004V8005V8006V8007V8008V8009V8010V8011V8012V8013V8014V8015

V8016V8017V8018V8019V8020V8021V8022V8023V8024V8025V8026V8027V8028V8029V8030V8031V8032V8033V8034V8035V8036V8037V8038V8039V8040V8041V8042V8043V8044V8045V8046V8047V8048V8049V8050V8051V8052V8053V8054V8055V8056V8057V8058V8059V8060V8061V8062V8063V8064

V8065

V8066V8067V8068V8069V8070V8071V8072V8073V8074V8075V8076V8077V8078V8079V8080V8081V8082V8083V8084V8085V8086V8087V8088V8089V8090V8091V8092V8093V8094V8095V8096V8097V8098V8099V8100V8101V8102V8103V8104V8105V8106V8107V8108V8109V8110

V8111

V8112V8113V8114V8115V8116V8117V8118V8119V8120V8121V8122V8123V8124V8125V8126V8127V8128V8129V8130V8131V8132V8133V8134V8135V8136V8137V8138V8139V8140V8141V8142V8143V8144V8145V8146V8147V8148V8149V8150V8151V8152V8153V8154V8155V8156V8157V8158V8159V8160

V8161V8162V8163V8164V8165V8166V8167V8168V8169V8170V8171V8172V8173V8174

V8175V8176V8177

V8178V8179V8180V8181V8182

V8183V8184V8185V8186V8187V8188V8189V8190V8191

V8192V8193V8194V8195V8196V8197V8198V8199V8200V8201V8202V8203V8204V8205V8206

V8207

V8208V8209V8210V8211V8212V8213V8214V8215V8216V8217V8218V8219V8220V8221V8222V8223V8224V8225V8226V8227V8228V8229V8230V8231V8232V8233V8234V8235V8236V8237V8238V8239V8240V8241V8242V8243V8244V8245V8246V8247V8248V8249V8250V8251V8252V8253V8254V8255V8256

V8257V8258V8259V8260V8261V8262V8263V8264V8265V8266V8267V8268V8269V8270V8271V8272V8273V8274V8275V8276V8277V8278V8279

V8280V8281V8282

V8283V8284V8285V8286V8287V8288

V8289

V8290V8291

V8292V8293V8294V8295V8296V8297V8298V8299V8300V8301

V8302V8303V8304V8305V8306V8307V8308V8309V8310V8311V8312V8313V8314V8315V8316V8317V8318V8319V8320V8321V8322V8323V8324V8325V8326V8327V8328V8329V8330V8331V8332V8333V8334V8335V8336V8337V8338V8339V8340V8341V8342V8343V8344V8345V8346V8347V8348V8349V8350V8351V8352

V8353V8354V8355V8356V8357V8358V8359V8360V8361V8362V8363V8364V8365

V8366V8367V8368V8369V8370V8371V8372V8373

V8374V8375V8376V8377V8378V8379V8380V8381V8382V8383V8384V8385V8386V8387V8388V8389V8390V8391V8392V8393V8394V8395V8396V8397V8398V8399

V8400V8401V8402V8403V8404V8405V8406V8407V8408V8409V8410V8411V8412V8413V8414V8415V8416V8417V8418V8419V8420V8421V8422V8423V8424V8425V8426V8427V8428V8429V8430V8431V8432V8433V8434V8435V8436V8437V8438V8439V8440V8441V8442V8443V8444V8445V8446V8447V8448

V8449V8450V8451V8452V8453V8454V8455V8456V8457V8458V8459V8460V8461V8462V8463V8464V8465V8466V8467V8468V8469V8470V8471V8472V8473V8474V8475V8476V8477V8478V8479V8480V8481V8482V8483V8484V8485V8486V8487V8488V8489V8490V8491V8492V8493V8494

V8495

V8496V8497V8498V8499V8500V8501V8502V8503V8504V8505V8506V8507V8508V8509V8510V8511V8512V8513V8514V8515V8516V8517V8518V8519V8520V8521V8522V8523V8524V8525V8526V8527V8528V8529V8530V8531V8532V8533V8534V8535V8536V8537V8538V8539V8540V8541V8542V8543V8544

V8545V8546V8547V8548V8549V8550V8551V8552V8553V8554V8555V8556

V8557V8558V8559V8560V8561V8562V8563V8564V8565V8566

V8567V8568

V8569

V8570V8571V8572V8573V8574V8575V8576V8577V8578V8579V8580V8581V8582V8583V8584V8585V8586V8587V8588V8589V8590V8591

V8592

V8593V8594V8595V8596V8597V8598V8599V8600V8601V8602V8603V8604V8605V8606V8607V8608V8609V8610V8611V8612V8613V8614V8615V8616V8617V8618V8619V8620V8621V8622V8623V8624V8625V8626V8627V8628V8629V8630V8631V8632V8633V8634V8635V8636V8637V8638V8639

V8640V8641V8642V8643V8644V8645V8646V8647V8648V8649V8650V8651V8652

V8653V8654V8655

V8656V8657V8658V8659V8660V8661V8662V8663V8664V8665

V8666

V8667V8668V8669

V8670V8671V8672V8673V8674V8675V8676V8677V8678V8679V8680V8681V8682V8683V8684V8685V8686V8687

V8688

V8689V8690V8691V8692V8693V8694V8695V8696V8697V8698V8699V8700V8701V8702V8703V8704V8705V8706V8707V8708V8709V8710V8711V8712V8713V8714V8715V8716V8717V8718V8719V8720V8721V8722V8723V8724V8725V8726V8727V8728V8729V8730V8731V8732V8733V8734V8735V8736

V8737V8738V8739V8740V8741V8742V8743V8744V8745V8746V8747V8748

V8749

V8750V8751V8752V8753V8754V8755V8756V8757V8758

V8759V8760V8761V8762V8763V8764V8765V8766V8767V8768V8769V8770V8771V8772V8773V8774V8775V8776V8777V8778V8779V8780V8781V8782V8783

V8784

V8785V8786V8787V8788V8789V8790V8791V8792V8793V8794V8795V8796V8797V8798V8799V8800V8801V8802V8803V8804V8805V8806V8807V8808V8809V8810V8811V8812V8813V8814V8815V8816V8817V8818V8819V8820V8821V8822V8823V8824V8825V8826V8827V8828V8829V8830V8831

V8832V8833V8834V8835V8836V8837V8838V8839V8840V8841V8842V8843

V8844

V8845V8846

V8847V8848V8849

V8850

V8851V8852

V8853V8854V8855V8856V8857V8858V8859V8860V8861V8862V8863V8864V8865V8866V8867V8868V8869V8870V8871V8872V8873V8874V8875V8876V8877V8878V8879

V8880

V8881V8882V8883V8884V8885V8886V8887V8888V8889V8890V8891V8892V8893V8894V8895V8896V8897V8898V8899V8900V8901V8902V8903V8904V8905V8906V8907V8908V8909V8910V8911V8912V8913V8914V8915V8916V8917V8918V8919V8920V8921V8922V8923V8924V8925V8926V8927

V8928V8929V8930V8931V8932V8933V8934V8935V8936

V8937

V8938V8939

V8940V8941

V8942V8943V8944V8945

V8946V8947V8948

V8949V8950

V8951V8952

V8953V8954V8955V8956V8957V8958V8959V8960V8961V8962V8963V8964V8965V8966V8967V8968V8969V8970V8971V8972V8973V8974V8975

V8976V8977V8978V8979V8980V8981V8982V8983V8984V8985V8986V8987V8988V8989V8990V8991V8992V8993V8994V8995V8996V8997V8998V8999V9000V9001V9002V9003V9004V9005V9006V9007V9008V9009V9010V9011V9012V9013V9014V9015V9016V9017V9018V9019V9020V9021V9022V9023

V9024V9025V9026V9027V9028V9029V9030V9031V9032V9033V9034V9035V9036V9037V9038V9039V9040V9041V9042V9043V9044V9045V9046V9047V9048V9049V9050V9051V9052V9053V9054V9055V9056V9057V9058V9059V9060V9061V9062V9063V9064V9065V9066V9067V9068V9069V9070V9071

V9072

V9073V9074V9075V9076V9077V9078V9079V9080V9081V9082V9083V9084V9085V9086V9087V9088V9089V9090V9091V9092V9093V9094V9095V9096V9097V9098V9099V9100V9101V9102V9103V9104V9105V9106V9107V9108V9109V9110V9111V9112V9113V9114V9115V9116V9117V9118V9119

V9120V9121V9122V9123V9124V9125V9126V9127V9128V9129V9130V9131V9132V9133V9134V9135V9136V9137V9138V9139V9140V9141V9142V9143V9144V9145V9146V9147V9148V9149V9150V9151V9152V9153V9154V9155V9156V9157V9158V9159V9160V9161V9162V9163V9164V9165V9166V9167

V9168

V9169V9170V9171V9172V9173V9174V9175V9176V9177V9178V9179V9180V9181V9182V9183V9184V9185V9186V9187V9188V9189V9190V9191V9192V9193V9194V9195V9196V9197V9198V9199V9200V9201V9202V9203V9204V9205V9206V9207V9208V9209V9210V9211V9212V9213V9214V9215

V9216

V9217V9218V9219V9220V9221V9222V9223V9224V9225V9226V9227V9228V9229V9230V9231V9232V9233V9234V9235V9236V9237V9238V9239V9240V9241V9242V9243V9244V9245V9246V9247V9248V9249V9250V9251V9252V9253V9254V9255V9256V9257V9258V9259V9260V9261V9262V9263

V9264V9265V9266V9267V9268V9269V9270V9271V9272V9273V9274V9275V9276V9277V9278V9279V9280V9281V9282V9283V9284V9285V9286V9287V9288V9289V9290V9291V9292V9293V9294V9295V9296V9297V9298V9299V9300V9301V9302V9303V9304V9305V9306V9307V9308V9309V9310V9311

V9312V9313V9314V9315V9316V9317V9318V9319V9320V9321V9322

V9323V9324V9325V9326V9327V9328V9329V9330V9331V9332V9333V9334V9335V9336V9337V9338V9339V9340V9341V9342V9343V9344V9345V9346V9347V9348V9349V9350V9351V9352V9353V9354V9355V9356V9357V9358V9359

V9360V9361V9362V9363V9364V9365V9366V9367V9368V9369V9370V9371V9372V9373V9374V9375V9376V9377V9378V9379V9380V9381V9382V9383V9384V9385V9386V9387V9388V9389V9390V9391V9392V9393V9394V9395V9396V9397V9398V9399V9400V9401V9402V9403V9404V9405V9406V9407

V9408V9409V9410V9411V9412V9413V9414V9415V9416V9417V9418V9419

V9420V9421V9422V9423

V9424V9425V9426V9427V9428V9429

V9430V9431V9432V9433V9434

V9435

V9436V9437V9438V9439V9440V9441V9442V9443V9444V9445V9446V9447V9448V9449V9450V9451V9452V9453V9454V9455

V9456

V9457V9458V9459V9460V9461V9462V9463V9464V9465V9466V9467V9468V9469V9470V9471V9472V9473V9474V9475V9476V9477V9478V9479V9480V9481V9482V9483V9484V9485V9486V9487V9488V9489V9490V9491V9492V9493V9494V9495V9496V9497V9498V9499V9500V9501V9502V9503

V9504V9505V9506V9507V9508V9509V9510V9511V9512V9513V9514V9515V9516V9517V9518

V9519V9520V9521V9522V9523V9524V9525V9526V9527V9528V9529V9530V9531V9532V9533V9534V9535V9536V9537V9538V9539V9540V9541V9542V9543V9544

V9545V9546V9547V9548V9549V9550V9551

V9552

V9553V9554V9555V9556V9557V9558V9559V9560V9561V9562V9563V9564V9565V9566V9567V9568V9569V9570V9571V9572V9573V9574V9575V9576V9577V9578V9579V9580V9581V9582V9583V9584V9585V9586V9587V9588V9589V9590V9591V9592V9593V9594V9595V9596V9597V9598V9599V9600V9601V9602V9603V9604V9605V9606V9607V9608V9609V9610V9611V9612V9613V9614V9615V9616V9617V9618V9619V9620V9621V9622V9623V9624V9625V9626V9627V9628V9629V9630V9631V9632V9633V9634V9635V9636V9637V9638V9639V9640V9641V9642V9643V9644V9645V9646V9647V9648V9649V9650V9651V9652V9653V9654V9655V9656V9657V9658V9659V9660V9661V9662V9663V9664V9665V9666V9667V9668V9669V9670V9671V9672V9673V9674V9675V9676V9677V9678V9679V9680V9681V9682V9683V9684V9685V9686V9687V9688V9689V9690V9691V9692V9693V9694V9695

V9696V9697V9698V9699V9700V9701V9702V9703

V9704V9705

V9706

V9707V9708V9709V9710V9711V9712V9713V9714V9715V9716

V9717V9718V9719

V9720

V9721

V9722V9723V9724

V9725V9726

V9727V9728V9729

V9730V9731V9732V9733V9734V9735V9736V9737V9738V9739V9740V9741

V9742V9743V9744

V9745V9746V9747V9748V9749V9750V9751V9752V9753V9754V9755V9756V9757V9758V9759V9760V9761V9762V9763V9764V9765V9766V9767V9768V9769V9770V9771V9772V9773V9774V9775V9776V9777V9778V9779V9780V9781V9782V9783V9784V9785V9786V9787V9788V9789V9790V9791V9792V9793V9794V9795V9796V9797V9798V9799V9800V9801V9802V9803V9804V9805V9806V9807V9808V9809V9810V9811V9812V9813V9814V9815V9816V9817V9818V9819V9820V9821V9822V9823V9824V9825V9826V9827V9828V9829V9830V9831V9832V9833V9834V9835V9836V9837V9838V9839V9840V9841V9842V9843V9844V9845V9846V9847V9848V9849V9850V9851V9852V9853V9854V9855V9856V9857V9858V9859V9860V9861V9862V9863V9864V9865V9866V9867V9868V9869V9870V9871V9872V9873V9874V9875V9876V9877V9878V9879V9880V9881V9882V9883V9884V9885V9886V9887V9888V9889V9890V9891V9892V9893V9894V9895V9896V9897V9898V9899V9900V9901V9902V9903V9904V9905V9906V9907V9908V9909V9910V9911V9912V9913V9914V9915V9916V9917V9918V9919V9920V9921V9922V9923V9924V9925V9926V9927V9928V9929V9930V9931V9932V9933V9934V9935V9936V9937V9938V9939V9940V9941V9942V9943V9944V9945V9946V9947V9948V9949V9950V9951V9952V9953V9954V9955V9956V9957V9958V9959V9960V9961V9962V9963V9964V9965V9966V9967V9968V9969V9970V9971V9972V9973V9974V9975V9976V9977V9978V9979V9980V9981V9982

V9983V9984V9985V9986V9987V9988V9989V9990V9991V9992V9993V9994V9995V9996V9997V9998V9999V10000V10001V10002V10003V10004V10005V10006V10007V10008V10009V10010V10011V10012V10013V10014V10015V10016V10017V10018V10019V10020V10021V10022V10023V10024V10025V10026V10027V10028V10029V10030V10031V10032

V10033V10034V10035V10036V10037V10038V10039V10040V10041V10042V10043V10044V10045V10046V10047V10048V10049V10050V10051V10052V10053V10054V10055V10056V10057V10058V10059V10060V10061V10062V10063V10064V10065V10066V10067V10068V10069V10070V10071V10072V10073V10074V10075V10076V10077V10078V10079V10080

V10081V10082V10083V10084V10085V10086V10087V10088V10089V10090V10091V10092

V10093V10094

V10095V10096V10097V10098V10099V10100V10101V10102

V10103V10104V10105V10106V10107V10108V10109V10110V10111V10112V10113V10114V10115V10116V10117V10118V10119V10120V10121V10122V10123V10124V10125V10126V10127V10128

V10129V10130V10131V10132V10133V10134V10135V10136V10137V10138V10139V10140V10141V10142V10143V10144V10145V10146V10147V10148V10149V10150V10151V10152V10153V10154V10155V10156V10157V10158V10159V10160V10161V10162V10163V10164V10165V10166V10167V10168V10169V10170V10171V10172V10173V10174

V10175V10176V10177V10178V10179V10180V10181V10182V10183V10184V10185V10186V10187V10188V10189V10190V10191V10192V10193

V10194V10195

V10196V10197V10198V10199V10200V10201V10202V10203V10204V10205V10206V10207V10208V10209V10210V10211V10212V10213V10214V10215V10216V10217V10218V10219V10220V10221V10222V10223V10224

V10225V10226V10227V10228V10229V10230V10231V10232V10233V10234V10235V10236V10237V10238V10239V10240V10241V10242V10243V10244V10245V10246V10247V10248V10249V10250V10251V10252V10253V10254V10255V10256V10257V10258V10259V10260V10261V10262V10263V10264V10265V10266V10267V10268V10269V10270V10271V10272V10273V10274V10275V10276V10277V10278V10279V10280V10281V10282V10283V10284V10285V10286V10287V10288V10289V10290V10291V10292V10293V10294V10295V10296V10297V10298V10299V10300V10301V10302V10303V10304V10305V10306V10307V10308V10309V10310V10311V10312V10313V10314V10315V10316V10317V10318V10319V10320V10321V10322V10323V10324V10325V10326V10327V10328V10329V10330V10331V10332V10333V10334V10335V10336V10337V10338V10339V10340V10341V10342V10343V10344V10345V10346V10347V10348V10349V10350V10351V10352V10353V10354V10355V10356V10357V10358V10359V10360V10361V10362V10363V10364V10365V10366

V10367V10368V10369V10370V10371V10372V10373V10374V10375V10376V10377V10378V10379V10380V10381V10382V10383V10384V10385V10386V10387V10388V10389V10390V10391V10392V10393V10394V10395V10396V10397V10398V10399V10400V10401V10402V10403V10404V10405V10406V10407V10408V10409V10410V10411V10412V10413V10414V10415V10416

V10417V10418V10419V10420V10421V10422V10423V10424V10425V10426V10427V10428V10429V10430V10431V10432V10433V10434V10435V10436V10437V10438V10439V10440V10441V10442V10443V10444V10445V10446V10447V10448V10449V10450V10451V10452V10453V10454V10455V10456V10457V10458V10459V10460V10461V10462

V10463

V10464V10465V10466V10467V10468V10469V10470V10471V10472V10473V10474V10475V10476V10477V10478V10479V10480V10481V10482V10483V10484V10485V10486V10487V10488V10489V10490V10491V10492V10493V10494V10495V10496V10497V10498V10499V10500V10501V10502V10503V10504V10505V10506V10507V10508V10509V10510V10511V10512V10513

V10514V10515V10516V10517V10518V10519V10520V10521V10522V10523V10524V10525V10526V10527V10528V10529V10530V10531V10532V10533V10534V10535V10536V10537V10538V10539V10540V10541V10542V10543V10544V10545V10546V10547V10548V10549V10550V10551V10552V10553V10554V10555V10556V10557V10558

V10559V10560V10561V10562V10563V10564V10565V10566V10567V10568V10569V10570V10571V10572V10573V10574V10575V10576V10577V10578V10579V10580V10581V10582V10583V10584V10585V10586V10587V10588V10589V10590V10591V10592V10593V10594V10595V10596V10597V10598V10599V10600V10601V10602V10603V10604V10605V10606V10607V10608

V10609V10610V10611V10612V10613V10614V10615V10616V10617V10618V10619V10620V10621V10622V10623V10624V10625V10626V10627V10628V10629V10630V10631V10632V10633V10634V10635V10636V10637V10638V10639V10640V10641V10642V10643V10644V10645V10646V10647V10648V10649V10650V10651V10652V10653V10654

V10655V10656V10657V10658V10659V10660V10661V10662V10663V10664V10665V10666V10667V10668

V10669V10670

V10671V10672V10673V10674

V10675V10676V10677V10678V10679V10680V10681V10682V10683V10684V10685V10686V10687V10688V10689V10690V10691V10692V10693V10694V10695V10696V10697V10698V10699V10700V10701V10702V10703V10704

V10705

V10706V10707V10708V10709V10710V10711V10712V10713V10714V10715V10716V10717V10718V10719V10720V10721V10722V10723V10724V10725V10726V10727V10728V10729V10730V10731V10732V10733V10734V10735V10736V10737V10738V10739V10740V10741V10742V10743V10744V10745V10746V10747V10748V10749V10750V10751

V10752V10753V10754V10755V10756V10757V10758V10759V10760V10761V10762V10763V10764V10765V10766V10767V10768V10769

V10770V10771V10772

V10773V10774V10775V10776V10777V10778V10779V10780V10781V10782

V10783V10784V10785V10786V10787V10788V10789V10790

V10791V10792V10793V10794V10795V10796V10797V10798V10799V10800

V10801

V10802V10803V10804V10805V10806V10807V10808V10809V10810V10811V10812V10813V10814V10815V10816V10817V10818V10819V10820V10821V10822V10823V10824V10825V10826V10827V10828V10829V10830V10831V10832V10833V10834V10835V10836V10837V10838V10839V10840V10841V10842V10843V10844V10845V10846

V10847V10848V10849V10850V10851V10852V10853V10854V10855V10856V10857V10858V10859V10860V10861V10862

V10863V10864V10865V10866V10867V10868

V10869V10870V10871V10872V10873V10874V10875V10876V10877

V10878

V10879

V10880

V10881

V10882

V10883V10884V10885V10886V10887V10888V10889V10890V10891V10892V10893V10894V10895

V10896

V10897V10898V10899V10900V10901V10902V10903V10904V10905V10906V10907V10908V10909V10910V10911V10912V10913V10914V10915V10916V10917V10918V10919V10920V10921V10922V10923V10924V10925V10926V10927V10928V10929V10930V10931V10932V10933V10934V10935V10936V10937V10938V10939V10940V10941V10942

V10943V10944V10945V10946V10947V10948V10949V10950V10951V10952V10953V10954V10955V10956V10957V10958V10959V10960V10961V10962V10963V10964V10965V10966V10967V10968V10969V10970V10971V10972V10973V10974V10975V10976V10977V10978V10979V10980V10981V10982V10983V10984V10985V10986V10987V10988V10989V10990V10991V10992

V10993V10994V10995V10996V10997V10998V10999V11000V11001V11002V11003V11004V11005V11006V11007V11008V11009V11010V11011V11012V11013V11014V11015V11016V11017V11018V11019V11020V11021V11022V11023V11024V11025V11026V11027V11028V11029V11030V11031V11032V11033V11034V11035V11036V11037V11038

V11039V11040V11041V11042V11043V11044V11045V11046V11047V11048V11049V11050V11051V11052V11053V11054V11055V11056V11057V11058V11059V11060V11061V11062V11063V11064V11065V11066V11067V11068V11069V11070V11071V11072V11073V11074V11075V11076V11077V11078V11079V11080V11081V11082V11083V11084V11085V11086V11087V11088

V11089

V11090V11091V11092V11093V11094V11095V11096V11097V11098V11099V11100V11101V11102V11103V11104V11105V11106V11107V11108V11109V11110V11111V11112V11113V11114V11115V11116V11117V11118V11119V11120V11121V11122V11123V11124V11125V11126V11127V11128V11129V11130V11131V11132V11133V11134

V11135V11136V11137V11138V11139V11140V11141V11142V11143V11144V11145V11146V11147V11148V11149V11150V11151V11152V11153V11154V11155V11156V11157V11158V11159V11160V11161V11162V11163V11164V11165V11166V11167V11168V11169V11170V11171V11172V11173V11174V11175V11176V11177V11178V11179V11180V11181V11182V11183V11184V11185

V11186V11187V11188V11189V11190V11191V11192V11193V11194V11195V11196V11197V11198V11199V11200V11201V11202V11203V11204V11205V11206V11207V11208V11209V11210V11211V11212V11213V11214V11215V11216V11217V11218V11219V11220V11221V11222V11223V11224V11225V11226V11227V11228V11229

V11230V11231V11232V11233V11234V11235V11236V11237V11238V11239V11240V11241V11242V11243V11244V11245V11246V11247V11248V11249V11250V11251V11252V11253V11254V11255V11256V11257V11258V11259V11260V11261V11262V11263V11264V11265V11266V11267V11268V11269V11270V11271V11272V11273V11274V11275V11276V11277V11278V11279V11280

V11281

V11282V11283V11284V11285V11286V11287V11288V11289V11290V11291V11292V11293V11294V11295V11296V11297V11298V11299V11300V11301V11302V11303V11304V11305V11306V11307V11308V11309V11310V11311V11312V11313V11314V11315V11316V11317V11318V11319V11320V11321V11322V11323V11324V11325

V11326V11327V11328V11329V11330V11331V11332V11333V11334V11335V11336V11337V11338V11339V11340V11341V11342V11343V11344V11345V11346V11347V11348V11349V11350V11351V11352V11353V11354V11355V11356V11357V11358V11359V11360V11361V11362V11363V11364V11365V11366V11367V11368V11369V11370V11371V11372V11373V11374V11375V11376V11377

V11378V11379V11380V11381V11382V11383V11384V11385V11386V11387V11388V11389V11390V11391V11392V11393V11394V11395V11396V11397V11398V11399V11400V11401V11402V11403V11404V11405V11406V11407V11408V11409V11410V11411V11412V11413V11414V11415V11416V11417V11418V11419V11420V11421

V11422V11423V11424V11425V11426V11427V11428V11429V11430V11431

V11432V11433V11434V11435V11436V11437V11438V11439V11440V11441V11442V11443V11444V11445V11446V11447V11448V11449V11450V11451V11452V11453V11454V11455V11456V11457V11458V11459V11460V11461V11462V11463V11464V11465V11466V11467V11468V11469V11470V11471

V11472V11473

V11474V11475V11476V11477V11478V11479V11480V11481V11482V11483V11484V11485V11486V11487V11488V11489V11490V11491V11492V11493V11494V11495V11496V11497V11498V11499V11500V11501V11502V11503V11504V11505V11506V11507V11508V11509V11510V11511V11512V11513V11514V11515V11516V11517

V11518V11519V11520V11521V11522V11523V11524V11525V11526V11527

V11528V11529

V11530V11531V11532V11533V11534V11535V11536V11537V11538V11539V11540V11541V11542V11543V11544V11545V11546V11547V11548V11549V11550V11551V11552V11553V11554V11555V11556V11557V11558V11559V11560V11561V11562V11563V11564V11565V11566V11567

V11568

V11569V11570V11571V11572V11573V11574V11575V11576V11577V11578V11579V11580V11581V11582V11583V11584V11585V11586V11587V11588V11589V11590V11591V11592V11593V11594V11595V11596V11597V11598V11599V11600V11601V11602V11603V11604V11605V11606V11607V11608V11609V11610V11611V11612V11613

V11614V11615V11616V11617V11618V11619V11620V11621V11622V11623V11624V11625

V11626V11627V11628V11629V11630

V11631V11632V11633V11634V11635V11636V11637V11638V11639V11640V11641V11642V11643V11644V11645V11646V11647V11648V11649V11650V11651V11652V11653V11654V11655V11656V11657V11658V11659V11660V11661V11662V11663

V11664V11665

V11666V11667V11668V11669V11670V11671V11672V11673V11674V11675V11676V11677V11678V11679V11680V11681V11682V11683V11684V11685V11686V11687V11688V11689V11690V11691V11692V11693V11694V11695V11696V11697V11698V11699V11700V11701V11702V11703V11704V11705V11706V11707V11708V11709

V11710V11711V11712V11713V11714V11715V11716V11717V11718V11719V11720V11721V11722V11723V11724V11725V11726V11727V11728V11729V11730V11731V11732V11733V11734V11735V11736V11737V11738V11739V11740V11741V11742V11743V11744V11745V11746V11747V11748V11749V11750V11751V11752V11753V11754V11755V11756V11757V11758V11759V11760

V11761

V11762V11763V11764V11765V11766V11767V11768V11769V11770V11771V11772V11773V11774V11775V11776V11777V11778V11779V11780V11781V11782V11783V11784V11785V11786V11787V11788V11789V11790V11791V11792V11793V11794V11795V11796V11797V11798V11799V11800V11801V11802V11803V11804V11805

V11806V11807V11808V11809V11810V11811V11812V11813V11814V11815V11816V11817

V11818V11819V11820

V11821V11822V11823V11824V11825V11826V11827V11828V11829

V11830V11831V11832V11833V11834V11835V11836V11837V11838V11839V11840V11841V11842V11843V11844V11845V11846V11847V11848V11849V11850V11851V11852V11853V11854V11855V11856

V11857

V11858V11859V11860V11861V11862V11863V11864V11865V11866V11867V11868V11869V11870V11871V11872V11873V11874V11875V11876V11877V11878V11879V11880V11881V11882V11883V11884V11885V11886V11887V11888V11889V11890V11891V11892V11893V11894V11895V11896V11897V11898V11899V11900V11901

V11902V11903V11904V11905V11906V11907V11908V11909V11910V11911V11912V11913V11914V11915V11916V11917V11918V11919V11920V11921V11922V11923V11924V11925V11926V11927V11928V11929V11930V11931V11932V11933V11934V11935V11936V11937V11938V11939V11940V11941V11942V11943V11944V11945V11946V11947V11948V11949V11950V11951V11952

V11953

V11954V11955V11956V11957V11958V11959V11960V11961V11962V11963V11964V11965V11966V11967V11968V11969V11970V11971V11972V11973V11974V11975V11976V11977V11978V11979V11980V11981V11982V11983V11984V11985V11986V11987V11988V11989V11990V11991V11992V11993V11994V11995V11996V11997

V11998V11999V12000V12001V12002V12003V12004V12005V12006V12007V12008V12009V12010V12011V12012V12013V12014V12015V12016V12017V12018V12019V12020V12021V12022V12023V12024V12025V12026V12027V12028V12029V12030V12031V12032V12033V12034V12035V12036V12037V12038V12039V12040V12041V12042V12043V12044V12045V12046

V12047V12048V12049V12050V12051V12052V12053V12054V12055V12056V12057V12058V12059V12060V12061V12062V12063V12064V12065V12066V12067V12068V12069V12070V12071V12072V12073V12074V12075V12076V12077V12078V12079V12080V12081V12082V12083V12084V12085V12086V12087V12088V12089V12090V12091V12092V12093

V12094V12095V12096V12097V12098V12099V12100V12101V12102V12103V12104V12105V12106V12107V12108V12109V12110V12111V12112V12113V12114V12115V12116V12117V12118V12119V12120V12121V12122V12123V12124V12125V12126V12127V12128V12129V12130V12131V12132V12133V12134V12135V12136V12137V12138V12139V12140V12141V12142V12143V12144V12145

V12146V12147V12148V12149V12150V12151V12152V12153V12154V12155V12156V12157V12158V12159V12160V12161V12162V12163V12164V12165V12166V12167V12168V12169V12170V12171V12172V12173V12174V12175V12176V12177V12178V12179V12180V12181V12182V12183V12184V12185V12186V12187V12188V12189

V12190V12191V12192V12193V12194V12195V12196V12197V12198V12199V12200V12201V12202V12203V12204V12205V12206V12207V12208V12209V12210V12211V12212V12213V12214V12215V12216V12217V12218V12219V12220V12221V12222V12223V12224V12225V12226V12227V12228V12229V12230V12231V12232V12233V12234V12235V12236V12237V12238V12239V12240V12241

V12242V12243V12244V12245V12246V12247V12248V12249V12250V12251V12252V12253V12254V12255V12256V12257V12258V12259V12260V12261V12262V12263V12264V12265V12266V12267V12268V12269V12270V12271V12272V12273V12274V12275V12276V12277V12278V12279V12280V12281V12282V12283V12284V12285

V12286V12287V12288V12289V12290V12291V12292V12293V12294V12295V12296V12297V12298V12299V12300V12301V12302V12303V12304V12305V12306V12307V12308V12309V12310V12311V12312V12313V12314V12315V12316V12317V12318V12319V12320V12321V12322V12323V12324V12325V12326V12327V12328V12329V12330V12331V12332V12333V12334V12335V12336V12337

V12338V12339V12340V12341V12342V12343V12344V12345V12346V12347V12348V12349V12350V12351V12352V12353V12354V12355V12356V12357V12358V12359V12360V12361V12362V12363V12364V12365V12366V12367V12368V12369V12370V12371V12372V12373V12374V12375V12376V12377V12378V12379V12380V12381

V12382V12383V12384V12385V12386V12387V12388V12389V12390V12391V12392V12393V12394V12395V12396V12397V12398V12399V12400V12401V12402V12403V12404

V12405V12406V12407V12408V12409V12410V12411V12412V12413V12414V12415V12416V12417V12418V12419V12420V12421V12422V12423V12424V12425V12426V12427V12428V12429V12430V12431V12432V12433

V12434V12435V12436V12437V12438V12439V12440V12441V12442V12443V12444V12445V12446V12447V12448V12449V12450V12451V12452V12453V12454V12455V12456V12457V12458V12459V12460V12461V12462V12463V12464V12465V12466V12467V12468V12469V12470V12471V12472V12473V12474V12475V12476V12477

V12478V12479V12480V12481V12482V12483V12484V12485V12486V12487V12488V12489V12490V12491V12492

V12493V12494V12495V12496V12497

V12498

V12499V12500V12501V12502

V12503V12504V12505

V12506

V12507V12508

V12509V12510V12511V12512V12513V12514V12515V12516V12517V12518V12519V12520V12521V12522V12523V12524V12525V12526V12527V12528V12529

V12530V12531V12532V12533V12534V12535V12536V12537V12538V12539V12540V12541V12542V12543V12544V12545V12546V12547V12548V12549V12550V12551V12552V12553V12554V12555V12556V12557V12558V12559V12560V12561V12562V12563V12564V12565V12566V12567V12568V12569V12570V12571V12572V12573

V12574V12575V12576V12577V12578V12579V12580V12581V12582V12583V12584V12585V12586V12587V12588V12589V12590V12591V12592V12593V12594V12595V12596V12597V12598V12599V12600V12601V12602V12603V12604V12605V12606V12607V12608V12609V12610V12611V12612V12613V12614V12615V12616V12617V12618V12619V12620V12621V12622V12623V12624V12625

V12626V12627V12628V12629V12630V12631V12632V12633V12634V12635V12636V12637V12638V12639V12640V12641V12642V12643V12644V12645V12646V12647V12648V12649V12650V12651V12652V12653V12654V12655V12656V12657V12658V12659V12660V12661V12662V12663V12664V12665V12666V12667V12668V12669

V12670V12671V12672V12673V12674V12675V12676V12677V12678V12679V12680V12681V12682V12683V12684V12685V12686V12687V12688V12689V12690V12691V12692V12693V12694V12695V12696V12697V12698V12699V12700V12701V12702V12703V12704V12705V12706V12707V12708V12709V12710V12711V12712V12713V12714V12715V12716V12717V12718V12719V12720V12721

V12722

V12723V12724V12725V12726V12727V12728V12729V12730V12731V12732V12733V12734V12735V12736V12737V12738V12739V12740V12741V12742V12743V12744V12745V12746V12747V12748V12749V12750V12751V12752V12753V12754V12755V12756V12757V12758V12759V12760V12761V12762V12763V12764V12765V12766V12767V12768V12769V12770V12771V12772V12773V12774V12775V12776V12777V12778V12779V12780V12781V12782V12783V12784V12785V12786V12787V12788V12789V12790V12791V12792V12793V12794V12795V12796V12797V12798V12799V12800V12801V12802V12803V12804V12805V12806V12807V12808V12809V12810V12811V12812V12813V12814V12815V12816V12817V12818V12819V12820V12821V12822V12823V12824V12825V12826V12827V12828V12829V12830V12831V12832V12833V12834V12835V12836V12837V12838V12839V12840V12841V12842V12843V12844V12845V12846V12847V12848V12849V12850V12851V12852V12853V12854V12855V12856V12857V12858V12859V12860

V12861V12862V12863V12864V12865V12866V12867V12868V12869V12870V12871V12872

V12873V12874V12875V12876V12877V12878V12879V12880V12881V12882V12883V12884V12885V12886V12887V12888V12889V12890V12891V12892V12893V12894V12895V12896V12897V12898V12899V12900V12901V12902V12903V12904V12905V12906V12907V12908V12909V12910V12911V12912V12913

V12914V12915V12916V12917V12918V12919V12920V12921V12922V12923V12924V12925V12926V12927V12928V12929V12930V12931V12932V12933V12934V12935V12936V12937V12938V12939V12940V12941V12942V12943V12944V12945V12946V12947V12948V12949V12950V12951V12952V12953V12954V12955V12956

V12957V12958V12959V12960V12961V12962V12963V12964V12965V12966V12967V12968V12969V12970V12971V12972V12973V12974V12975V12976V12977V12978V12979V12980V12981V12982V12983V12984V12985V12986V12987V12988V12989V12990V12991V12992V12993V12994V12995V12996V12997V12998V12999V13000V13001V13002V13003V13004V13005V13006V13007V13008V13009

V13010V13011V13012V13013V13014V13015V13016V13017V13018V13019V13020V13021V13022V13023V13024V13025V13026V13027V13028V13029V13030V13031V13032V13033V13034V13035V13036V13037V13038V13039V13040V13041V13042V13043V13044V13045V13046V13047V13048V13049V13050V13051V13052V13053

V13054V13055V13056V13057V13058V13059V13060V13061V13062V13063V13064V13065V13066V13067V13068V13069V13070V13071V13072V13073V13074V13075V13076V13077V13078V13079V13080V13081V13082V13083V13084V13085V13086V13087V13088V13089V13090V13091V13092V13093V13094V13095V13096V13097V13098V13099V13100V13101V13102V13103V13104V13105

V13106V13107V13108V13109V13110V13111V13112V13113V13114V13115V13116V13117V13118V13119V13120V13121V13122V13123V13124V13125V13126V13127V13128V13129V13130V13131V13132V13133V13134V13135V13136V13137V13138V13139V13140V13141V13142V13143V13144V13145V13146V13147V13148

V13149V13150V13151V13152V13153V13154V13155V13156V13157V13158V13159V13160V13161V13162V13163V13164V13165V13166V13167V13168V13169V13170V13171V13172V13173V13174V13175V13176V13177V13178V13179V13180V13181V13182V13183V13184V13185V13186V13187V13188V13189V13190V13191V13192V13193V13194V13195V13196V13197V13198V13199V13200V13201V13202

V13203V13204V13205V13206V13207V13208V13209V13210V13211V13212V13213V13214V13215V13216V13217V13218V13219V13220V13221V13222V13223V13224V13225V13226V13227V13228V13229V13230V13231V13232V13233V13234V13235V13236V13237V13238V13239V13240V13241V13242V13243V13244

V13245V13246V13247V13248V13249V13250V13251V13252V13253V13254V13255V13256V13257V13258V13259V13260V13261V13262V13263V13264V13265V13266V13267V13268V13269V13270V13271V13272V13273V13274V13275V13276V13277V13278V13279V13280V13281V13282V13283V13284V13285V13286V13287V13288V13289V13290V13291V13292V13293V13294V13295V13296V13297V13298

V13299V13300V13301V13302V13303V13304V13305V13306V13307V13308V13309V13310V13311V13312V13313V13314V13315V13316V13317V13318V13319V13320V13321V13322V13323V13324V13325V13326V13327V13328V13329V13330V13331V13332V13333V13334V13335V13336V13337V13338V13339V13340

V13341V13342V13343V13344V13345V13346V13347V13348V13349V13350V13351V13352V13353V13354V13355V13356V13357V13358V13359V13360V13361V13362V13363V13364V13365V13366V13367V13368V13369V13370V13371V13372V13373V13374V13375V13376V13377V13378V13379V13380V13381V13382V13383V13384V13385V13386V13387V13388V13389V13390V13391V13392V13393

V13394V13395V13396V13397V13398V13399V13400V13401V13402V13403V13404V13405V13406V13407V13408V13409V13410V13411V13412V13413V13414V13415V13416V13417V13418V13419V13420V13421V13422V13423V13424V13425V13426V13427V13428V13429V13430V13431V13432V13433V13434V13435V13436

V13437V13438V13439V13440V13441V13442V13443V13444V13445V13446V13447V13448V13449V13450V13451V13452V13453V13454V13455V13456

V13457V13458V13459V13460V13461V13462V13463

V13464V13465V13466V13467V13468

V13469

V13470V13471V13472V13473V13474V13475V13476V13477V13478V13479V13480V13481V13482V13483V13484V13485V13486V13487V13488V13489

V13490

V13491V13492V13493V13494V13495V13496V13497V13498V13499V13500V13501V13502V13503V13504V13505V13506V13507V13508V13509V13510V13511V13512V13513V13514V13515V13516V13517V13518V13519V13520V13521V13522V13523V13524V13525V13526V13527V13528V13529V13530V13531V13532V13533

V13534V13535V13536V13537V13538V13539V13540V13541V13542V13543V13544V13545V13546V13547V13548V13549

V13550V13551V13552V13553V13554V13555V13556V13557V13558V13559

V13560

V13561V13562V13563V13564V13565V13566V13567V13568V13569V13570V13571V13572V13573V13574V13575V13576V13577V13578V13579V13580V13581V13582V13583V13584V13585

V13586

V13587V13588V13589V13590V13591V13592V13593V13594V13595V13596V13597V13598V13599V13600V13601V13602V13603V13604V13605V13606V13607V13608V13609V13610V13611V13612V13613V13614V13615V13616V13617V13618V13619V13620V13621V13622V13623V13624V13625V13626V13627V13628

V13629V13630V13631V13632V13633V13634V13635V13636V13637V13638V13639V13640V13641

V13642V13643V13644V13645V13646V13647V13648V13649V13650V13651V13652V13653V13654V13655V13656V13657V13658V13659V13660V13661V13662V13663V13664V13665V13666V13667V13668V13669V13670V13671V13672V13673V13674V13675V13676V13677V13678V13679V13680V13681

V13682

V13683V13684V13685V13686V13687V13688V13689V13690V13691V13692V13693V13694V13695V13696V13697V13698V13699V13700V13701V13702V13703V13704V13705V13706V13707V13708V13709V13710V13711V13712V13713V13714V13715V13716V13717V13718V13719V13720V13721V13722V13723V13724

V13725V13726V13727V13728V13729V13730V13731V13732V13733V13734V13735V13736

V13737V13738V13739V13740V13741V13742V13743V13744V13745V13746V13747V13748V13749V13750V13751V13752V13753V13754V13755V13756

V13757V13758V13759V13760V13761V13762V13763V13764V13765V13766V13767V13768V13769V13770V13771V13772V13773V13774V13775V13776V13777V13778

V13779V13780V13781V13782V13783V13784V13785V13786V13787V13788V13789V13790V13791V13792V13793V13794V13795V13796V13797V13798V13799V13800V13801V13802V13803V13804V13805V13806V13807V13808V13809V13810V13811V13812V13813V13814V13815V13816V13817V13818V13819V13820

V13821V13822V13823V13824V13825V13826V13827V13828V13829V13830V13831V13832V13833V13834V13835V13836V13837V13838V13839V13840V13841V13842V13843V13844V13845V13846V13847V13848V13849V13850V13851V13852V13853V13854V13855V13856V13857V13858V13859V13860V13861V13862V13863V13864V13865V13866V13867V13868V13869V13870V13871V13872V13873V13874

V13875V13876V13877V13878V13879V13880V13881V13882V13883V13884V13885V13886V13887V13888V13889V13890V13891V13892V13893V13894V13895V13896V13897V13898V13899V13900V13901V13902V13903V13904V13905V13906V13907V13908V13909V13910V13911V13912V13913V13914V13915V13916

V13917V13918V13919V13920V13921V13922V13923V13924V13925V13926V13927V13928V13929V13930V13931V13932V13933V13934V13935V13936V13937V13938V13939V13940V13941V13942V13943V13944

V13945V13946V13947

V13948V13949V13950V13951V13952V13953V13954V13955V13956V13957V13958

V13959V13960V13961V13962V13963V13964V13965V13966V13967V13968V13969V13970

V13971V13972V13973V13974V13975V13976V13977V13978V13979V13980V13981V13982V13983V13984V13985V13986V13987V13988V13989V13990V13991V13992V13993V13994V13995V13996V13997V13998V13999V14000V14001V14002V14003V14004V14005V14006V14007V14008V14009V14010V14011V14012

V14013V14014V14015V14016V14017V14018V14019V14020V14021V14022V14023V14024V14025V14026V14027V14028V14029V14030V14031V14032V14033V14034V14035V14036V14037V14038V14039V14040V14041V14042V14043V14044V14045V14046V14047V14048V14049V14050V14051V14052V14053V14054V14055V14056V14057V14058V14059V14060V14061V14062V14063V14064V14065V14066

V14067V14068V14069V14070V14071V14072V14073V14074V14075V14076V14077V14078V14079V14080V14081V14082V14083V14084V14085V14086V14087V14088V14089V14090V14091V14092V14093V14094V14095V14096V14097V14098V14099V14100V14101V14102V14103V14104V14105V14106V14107V14108

V14109V14110V14111V14112V14113V14114V14115V14116V14117V14118V14119V14120V14121V14122V14123V14124V14125V14126

V14127V14128V14129V14130V14131V14132V14133V14134V14135V14136V14137V14138V14139V14140V14141V14142V14143V14144V14145V14146V14147V14148V14149V14150V14151V14152V14153V14154V14155V14156V14157V14158V14159V14160V14161V14162

V14163V14164V14165V14166V14167V14168V14169V14170V14171V14172V14173V14174V14175V14176V14177V14178V14179V14180V14181V14182V14183V14184V14185V14186V14187V14188V14189V14190V14191V14192V14193V14194V14195V14196V14197V14198V14199V14200V14201V14202V14203V14204

V14205V14206V14207V14208V14209V14210V14211V14212V14213V14214V14215V14216V14217V14218V14219V14220V14221V14222V14223V14224V14225V14226V14227V14228

V14229V14230V14231V14232V14233V14234V14235V14236V14237V14238V14239V14240V14241V14242V14243V14244V14245V14246V14247V14248V14249V14250V14251V14252V14253V14254V14255V14256

V14257V14258

V14259V14260V14261V14262V14263V14264V14265V14266V14267V14268V14269V14270V14271V14272V14273V14274V14275V14276V14277V14278V14279V14280V14281V14282V14283V14284V14285V14286V14287V14288V14289V14290V14291V14292V14293V14294V14295V14296V14297V14298V14299V14300

V14301V14302V14303V14304V14305V14306V14307V14308V14309V14310V14311V14312V14313V14314V14315V14316V14317V14318V14319V14320V14321V14322V14323V14324V14325V14326V14327V14328V14329V14330V14331V14332V14333V14334V14335V14336V14337V14338V14339V14340V14341V14342V14343V14344V14345V14346V14347V14348V14349V14350V14351V14352

V14353V14354

V14355V14356V14357V14358V14359V14360V14361V14362V14363V14364V14365V14366V14367V14368V14369V14370V14371V14372V14373V14374V14375V14376V14377V14378V14379V14380V14381V14382V14383V14384V14385V14386V14387V14388V14389V14390V14391V14392V14393V14394V14395V14396

V14397V14398V14399V14400V14401V14402V14403V14404V14405V14406V14407V14408V14409V14410V14411V14412V14413V14414V14415

V14416V14417V14418V14419

V14420

V14421V14422V14423V14424V14425V14426V14427V14428V14429V14430V14431V14432V14433V14434V14435V14436V14437V14438V14439V14440V14441V14442V14443V14444V14445V14446V14447V14448V14449

V14450

V14451V14452V14453V14454V14455V14456V14457V14458V14459V14460V14461V14462V14463V14464V14465V14466V14467V14468V14469V14470V14471V14472V14473V14474V14475V14476V14477V14478V14479V14480V14481V14482V14483V14484V14485V14486V14487V14488V14489V14490V14491V14492

V14493V14494V14495V14496V14497V14498V14499V14500V14501V14502V14503V14504V14505V14506V14507V14508V14509V14510V14511V14512V14513V14514V14515V14516V14517V14518V14519V14520V14521V14522V14523V14524V14525V14526V14527V14528V14529V14530V14531V14532V14533V14534V14535V14536V14537V14538V14539V14540V14541V14542V14543V14544V14545V14546

V14547V14548V14549V14550V14551V14552V14553V14554V14555V14556V14557V14558V14559V14560V14561V14562V14563V14564V14565V14566V14567V14568V14569V14570V14571V14572V14573V14574V14575V14576V14577V14578V14579V14580V14581V14582V14583V14584V14585V14586V14587V14588

V14589V14590V14591V14592V14593V14594V14595V14596V14597V14598V14599V14600V14601V14602V14603V14604V14605V14606V14607V14608V14609V14610V14611V14612V14613V14614V14615V14616V14617V14618V14619V14620V14621V14622V14623V14624V14625V14626V14627V14628V14629V14630V14631V14632V14633V14634V14635V14636V14637V14638V14639V14640

V14641V14642

V14643V14644V14645V14646V14647V14648V14649V14650V14651V14652V14653V14654V14655V14656V14657V14658V14659V14660V14661V14662V14663V14664V14665V14666V14667V14668V14669V14670V14671V14672V14673V14674V14675V14676V14677V14678V14679V14680V14681V14682V14683V14684

V14685V14686V14687V14688V14689V14690V14691V14692V14693V14694V14695V14696V14697V14698V14699V14700V14701V14702V14703V14704V14705V14706V14707V14708V14709V14710V14711V14712V14713V14714V14715V14716V14717V14718V14719V14720V14721V14722V14723V14724V14725V14726V14727V14728V14729V14730V14731V14732V14733V14734V14735V14736

V14737V14738

V14739V14740V14741V14742V14743V14744V14745V14746V14747V14748V14749V14750V14751V14752V14753V14754V14755V14756V14757V14758V14759V14760V14761V14762V14763V14764V14765V14766V14767V14768V14769V14770V14771V14772V14773V14774V14775V14776V14777V14778V14779V14780

V14781V14782V14783V14784V14785V14786V14787V14788V14789V14790V14791V14792V14793V14794V14795V14796V14797V14798V14799V14800V14801V14802V14803V14804V14805V14806V14807V14808V14809V14810V14811V14812V14813V14814V14815V14816V14817V14818V14819V14820V14821V14822V14823V14824V14825V14826V14827V14828V14829V14830V14831V14832V14833V14834

V14835V14836V14837V14838V14839V14840V14841V14842V14843V14844V14845V14846V14847V14848V14849V14850V14851V14852V14853V14854V14855V14856V14857V14858V14859V14860V14861V14862V14863V14864V14865V14866V14867V14868V14869V14870V14871V14872V14873V14874V14875V14876

V14877V14878V14879V14880V14881V14882V14883V14884V14885V14886V14887V14888V14889V14890V14891V14892V14893

V14894

V14895V14896V14897

V14898V14899

V14900

V14901

V14902

V14903V14904V14905V14906V14907V14908V14909V14910V14911V14912V14913V14914V14915V14916V14917V14918V14919V14920V14921V14922V14923V14924V14925V14926V14927V14928

V14929V14930

V14931V14932V14933V14934V14935V14936V14937V14938V14939V14940V14941V14942V14943V14944V14945V14946V14947V14948V14949V14950V14951V14952V14953V14954V14955V14956V14957V14958V14959V14960V14961V14962V14963V14964V14965V14966V14967V14968V14969V14970V14971V14972

V14973V14974V14975V14976V14977V14978V14979V14980V14981V14982V14983V14984V14985V14986V14987V14988V14989V14990V14991V14992V14993V14994V14995V14996V14997V14998V14999V15000V15001V15002V15003V15004V15005V15006V15007V15008V15009V15010V15011V15012V15013V15014V15015V15016V15017V15018V15019V15020V15021V15022V15023V15024V15025V15026V15027

V15028V15029V15030V15031V15032V15033V15034V15035V15036V15037V15038V15039V15040V15041V15042V15043V15044V15045V15046V15047V15048V15049V15050V15051V15052V15053V15054V15055V15056V15057V15058V15059V15060V15061V15062V15063V15064V15065V15066V15067V15068

V15069V15070V15071V15072V15073V15074V15075V15076V15077V15078V15079V15080V15081V15082V15083V15084V15085V15086V15087V15088V15089V15090V15091V15092V15093V15094V15095V15096V15097V15098V15099V15100V15101V15102V15103V15104V15105V15106V15107V15108V15109V15110V15111V15112V15113V15114V15115V15116V15117V15118V15119V15120V15121V15122

V15123

V15124V15125V15126V15127V15128V15129V15130V15131V15132V15133V15134V15135V15136V15137V15138V15139V15140V15141V15142V15143V15144V15145V15146V15147V15148V15149V15150V15151V15152V15153V15154V15155V15156V15157V15158V15159V15160V15161V15162V15163V15164

V15165V15166V15167V15168V15169V15170V15171V15172V15173V15174V15175V15176V15177V15178V15179V15180V15181V15182V15183V15184V15185V15186V15187V15188V15189V15190V15191V15192V15193V15194V15195V15196V15197V15198V15199V15200V15201V15202V15203V15204V15205V15206V15207V15208V15209V15210V15211V15212V15213V15214V15215V15216V15217V15218

V15219V15220V15221V15222V15223V15224V15225V15226V15227V15228V15229V15230V15231V15232V15233V15234V15235V15236V15237V15238V15239V15240V15241V15242V15243V15244V15245V15246V15247V15248V15249V15250V15251V15252V15253V15254V15255V15256V15257V15258V15259V15260

V15261V15262V15263V15264V15265V15266V15267V15268V15269V15270V15271V15272V15273V15274V15275V15276V15277V15278V15279V15280V15281V15282V15283V15284V15285V15286V15287V15288V15289V15290V15291V15292V15293V15294V15295V15296V15297V15298V15299V15300V15301V15302V15303V15304V15305V15306V15307V15308V15309V15310V15311V15312

V15313V15314

V15315V15316V15317V15318V15319V15320V15321V15322V15323V15324V15325V15326V15327V15328V15329V15330V15331V15332V15333V15334V15335V15336V15337V15338V15339V15340V15341V15342V15343V15344V15345V15346V15347V15348V15349V15350V15351V15352V15353V15354V15355V15356

V15357V15358V15359V15360V15361V15362V15363V15364V15365V15366V15367V15368V15369

V15370V15371V15372V15373V15374V15375V15376V15377

V15378V15379V15380V15381V15382V15383V15384V15385V15386V15387V15388V15389V15390V15391V15392V15393V15394V15395V15396

V15397V15398V15399V15400V15401V15402V15403V15404V15405V15406V15407V15408

V15409V15410V15411

V15412V15413V15414V15415V15416V15417V15418V15419V15420V15421V15422V15423V15424V15425V15426V15427V15428V15429V15430V15431V15432V15433V15434V15435V15436V15437V15438V15439V15440V15441V15442V15443V15444V15445V15446V15447V15448V15449V15450V15451V15452

V15453V15454V15455V15456V15457V15458V15459V15460V15461V15462V15463V15464

V15465

V15466

V15467V15468

V15469V15470V15471

V15472

V15473V15474

V15475V15476V15477V15478V15479V15480V15481V15482V15483V15484V15485V15486V15487V15488V15489V15490V15491V15492V15493V15494V15495V15496V15497V15498V15499V15500V15501V15502V15503V15504

V15505V15506

V15507V15508V15509V15510V15511V15512V15513V15514V15515V15516V15517V15518V15519V15520V15521V15522V15523V15524V15525V15526V15527V15528V15529V15530V15531V15532V15533V15534V15535V15536V15537V15538V15539V15540V15541V15542V15543V15544V15545V15546V15547V15548

V15549V15550V15551V15552V15553V15554V15555V15556V15557V15558V15559V15560V15561V15562V15563V15564V15565

V15566V15567V15568V15569V15570V15571V15572

V15573V15574V15575V15576V15577V15578V15579V15580V15581V15582V15583V15584V15585V15586V15587V15588V15589V15590V15591V15592V15593V15594V15595V15596V15597V15598V15599V15600V15601V15602

V15603V15604V15605V15606V15607V15608V15609V15610V15611V15612V15613V15614V15615V15616V15617V15618V15619V15620V15621V15622V15623V15624V15625V15626V15627V15628V15629V15630V15631V15632V15633V15634V15635V15636V15637V15638V15639V15640V15641V15642V15643V15644

V15645V15646V15647V15648V15649V15650V15651V15652V15653V15654V15655V15656V15657V15658V15659

V15660

V15661V15662V15663V15664V15665V15666V15667V15668V15669V15670V15671V15672V15673V15674V15675V15676V15677V15678V15679V15680V15681V15682V15683V15684V15685V15686V15687V15688V15689V15690V15691V15692V15693V15694V15695V15696V15697V15698

V15699

V15700V15701V15702V15703V15704V15705V15706V15707V15708V15709V15710V15711V15712V15713V15714V15715V15716V15717V15718V15719V15720V15721V15722V15723V15724V15725V15726V15727V15728V15729V15730V15731V15732V15733V15734V15735V15736V15737V15738V15739V15740

V15741V15742V15743V15744V15745V15746V15747V15748V15749V15750V15751V15752V15753V15754

V15755V15756V15757V15758V15759V15760V15761V15762V15763V15764

V15765V15766V15767V15768V15769V15770V15771

V15772V15773

V15774

V15775V15776V15777V15778V15779V15780V15781V15782V15783V15784V15785V15786V15787V15788V15789V15790V15791V15792V15793V15794

V15795V15796V15797V15798V15799V15800V15801V15802V15803V15804V15805V15806V15807V15808V15809V15810V15811V15812V15813V15814V15815V15816V15817V15818V15819V15820V15821V15822V15823V15824V15825V15826V15827V15828V15829V15830V15831V15832V15833V15834V15835V15836

V15837V15838V15839V15840V15841V15842V15843V15844V15845V15846V15847V15848V15849V15850

V15851

V15852

V15853V15854V15855V15856V15857V15858

0

2000

4000

6000

1 2 3 4 5 6 7 8 9 10

eigenvalue

Figure 5.7: The PCA performed on the outputs gotten from the DOE of 300 points. On the left, the correlationcircle and on the right, the eigenvalues.

The PCA performed on the maximin LHS gives different results from Figure 5.1 because between both 7parameters has been selected by the sensitivity analysis. The DOE generated by the Morris method was implyingdisplacements on a 26 dimensional space and the maximin LHS only focuses on these 7 parameters. It means thatwe might have lost a little piece of information which is not relevant regarding the variations of power in the timeseries output. That is why in the PCA performed with the Morris DOE more dispersion is visible.

Figure 5.7 illustrates the correlation circle and the eigenvalues obtained after the PCA. It allows to considerthat only d = 5 axes are carrying more than 99% of the total inertia of the points. Regarding Figure 5.7, 3 axiscould have been considered instead of 5 but, as the time consuming part is focused on the establishment of theDOE, it does not cost a lot to use 5 axis and then limit the projection error. In this example, PF : RT → R5 is theprojection on the subspace of the first 5 principal components. The matrix that is representing the projection on the

121

Cha

pter

5

Chapter 5 – Performance monitoring on a large PV plant

orthogonal subspace to the 5 principal components is then noted Pδ : RT → RT−5. The correlation between thepower computed by the numerical code and the the PCA axis (computed from Equation (2.100)) is visible in Figure5.8. The irradiation is plotted on the figure as well because we want to link the physical phenomenon to each axis ofthe PCA. Each PCA axis should reproduce a different characteristic of the time series and Figure 5.8 shows that thefirst component of the PCA is representing the average value of the power, when the second axis seems to take careof power fluctuations. The first axis presents some peaks at low values of irradiation because when the sun is low inthe sky, some non-linear effects appear which affect the production and the averaged power. The third axis lookslike it re-creates the behavior of the PV plant when the irradiation is low. The values are near zero and the peaks areoccurring at the same time as the peaks represented on the first axis. This may represent the non linear behaviorof the panel when the sun rays angle is low. The moments when the sun rises and when it sets are when shadesare occurring. So the third axis can represent the parts of the code that are dealing with the shades and missmatcheffects. The fourth and the fifth axis are not given because they are almost entirely flat and does not carry a stronginterpretation.

122

Cha

pter

5

5.4. Bayesian calibration

−1

0

1

2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27

Sca

led

irrad

ianc

e in

W/m

2

−1.00

−0.95

−0.90

−0.85

−0.80

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27

PC

A 1

−0.3

−0.2

−0.1

0.0

0.1

0.2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27

PC

A 2

0.0

0.2

0.4

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27

Days

PC

A 3

Figure 5.8: The irradiation and the correlation of the power recorded with the three first PCA axes for the 27 firstdays. On the top the scaled irradiance, on the middle top the correlation of power recorded with the first axis givenby the PCA. On the middle bottom, the correlation between the recorded power and the second PCA axis, and onthe bottom, the correlation with the third PCA axis.

123

Cha

pter

5

Chapter 5 – Performance monitoring on a large PV plant

Calibration

Calibration is performed under some hypotheses. The first one is to consider anisotropic Matérn 5/2 kernels forthe Gaussian processes. A maximum of the partial likelihood is used to find the nuisance parameters relative toeach Gaussian process. Different values of correlation lengths according the parameters and different variances areexpected because the influence of each Gaussian process is diverse. The values for these parameters are given inFigure 5.9.

0.0

0.2

0.4

0.6

0.8

PCA1 PCA2 PCA3 PCA4 PCA5PCA axis

Val

ues

0.0

0.2

0.4

0.6

0.8

PCA1 PCA2 PCA3 PCA4 PCA5PCA axis

Val

ues

0.0

0.1

0.2

PCA1 PCA2 PCA3 PCA4 PCA5PCA axis

Val

ues

0

5

10

15

20

PCA1 PCA2 PCA3 PCA4 PCA5PCA axis

Val

ues

ψIsc ψnD ψRshunt_stc_1D ψRserie_b_1D

0e+00

1e−04

2e−04

3e−04

4e−04

5e−04

PCA1 PCA2 PCA3 PCA4 PCA5PCA axis

Val

ues

0

5

10

15

20

PCA1 PCA2 PCA3 PCA4 PCA5PCA axis

Val

ues

0.0

2.5

5.0

7.5

10.0

PCA1 PCA2 PCA3 PCA4 PCA5PCA axis

Val

ues

0e+00

2e+12

4e+12

6e+12

PCA1 PCA2 PCA3 PCA4 PCA5PCA axis

Log

valu

es

ψRshunt_0_1D ψK_1D ψPaco σ2

Figure 5.9: Correlation lengths for each component of the parameter vector θ and for the variance of the 5 Gaussianprocesses on each PCA axis.

Figure 5.9 illustrates the histograms of each correlation length value for each parameter and the variance of thekernel for the 5 Gaussian processes used. The value of the variance is noticeable because after the third principalcomponent the variance decrease drastically. It means that the variance of the phenomenon carried by the fourthand fifth axes is negligible regarding the three first ones. This result is in agreement with the right panel of Figure5.7 where the the fourth and the fifth eigenvalues were significantly lower than the three first ones. The correlationlengths are homogeneous except for the component R_shunt_1D where the correlation length is decreasing. Thecorrelation function used to establish the Gaussian process is a Matérn 5/2 (Equation (2.51)). In the packageDiceKriging (Roustant et al., 2015), the Matérn correlation function is coded such as the correlation factor is in thedenominator of an exponential. It means that if the value of ψ decreases, the weight of the correlation structure isdeclining.

Once the Gaussian processes parameters are estimated by MLE, the estimators are plugged into the conditionallikelihood that allows to sample in the full likelihood for a regular MCMC. This method might bring instabilityin calibration because of the computation of the full likelihood. Indeed, in Equation (5.7), the second term, thatintroduces the term ||PδYexp||2 can potentially be huge and counterparts the first one. A separation in the estimationcan be done. The way to do it would be to separate the likelihoods in two, and perform two Bayesian calibrations.

124

Cha

pter

5

5.4. Bayesian calibration

The first one, the estimation is performed on the first part of the likelihood which is:

1

(2π)d/2Πdi=1

√σ2

err +σ2i (θ)

exp

(−1

2

d

∑i=1

(P TFiYexp−mi(θ))

2

σ2err +σ2

i (θ)

). (5.9)

Estimators of θ, σ2err and σ2

δare then found and used as new starting points of the estimation based on the

second part of the likelihood:

1(2π(σ2 +σ2

δ))(T−d)/2 exp

(− 1

2(σ2err +σ2

δ)‖P T

δYexp‖2

). (5.10)

This method allows to perform the estimation even if the second term in the likelihood creates too much weight.This case can easily occur because it is highly dependent on the size of the vector PδYexp (which is T ). In timeseries outputs, the number of points is often high and having recourse to this split estimation solves the issue.Calibration is performed via the CaliCo package presented in Chapter 4. The chosen prior densities are the onepresented in Section 5.2. First the model without discrepancy is considered. This model M ′

2 encompasses the 5Gaussian processes estimated above. The results are displayed in Figure 5.10.

0

50

100

150

200

8.2 8.4 8.6 8.8Isc

dens

ity

posterior

prior

0

1

2

3

4

0 1 2nD

dens

ity

posterior

prior

0

5

10

15

0.00 0.25 0.50 0.75 1.00Rshunt_stc_1D

dens

ity

posterior

prior

0

50000

100000

150000

200000

250000

0.000075 0.000100 0.000125 0.000150 0.000175Rserie_b_1D

dens

ity

posterior

prior

0

10

20

30

1.00 1.25 1.50 1.75 2.00Rshunt_0_1_D

dens

ity

posterior

prior

0

1

2

3

2.5 5.0 7.5 10.0K_1D

dens

ity

posterior

prior

0e+00

1e−04

2e−04

3e−04

4e−04

5e−04

820000 840000 860000Paco

dens

ity

posterior

prior

0.0e+00

5.0e-10

1.0e-09

1.5e-09

0e+00 2e+09 4e+09 6e+09σerr

2

dens

ity

posteriorprior

Figure 5.10: Prior and posterior densities for each parameter in a M ′2 calibration.

The results given in Figure 5.10 are obtained after visual checking of the good mixing properties in the MCMCchains. The posterior density for each parameter has less variance than the prior density which indicates thatinformation in the PCA subspace has been useful to improve the knowledge on the parameter densities. The varianceof the measurement error looks coherent regarding the prior density but is a little over estimated in that case becausethe projection errors are not taken into account and might have introduced this increase of variance.

Let us now consider the model that encompasses the discrepancy modeled as a multidimensional Gaussian noisewith a variance σ2

δ. The results for that model M ′

4 are given in Figure 5.11.

125

Cha

pter

5

Chapter 5 – Performance monitoring on a large PV plant

0

25

50

75

100

8.2 8.4 8.6 8.8Isc

dens

ity

posterior

prior

0

2

4

6

0 1 2nD

dens

ity

posterior

prior

0

1

2

3

4

5

0.00 0.25 0.50 0.75 1.00Rshunt_stc_1D

dens

ity

posterior

prior

0

25000

50000

75000

0.000075 0.000100 0.000125 0.000150 0.000175Rserie_b_1D

dens

ity

posterior

prior

0

2

4

6

1.00 1.25 1.50 1.75 2.00Rshunt_0_1_D

dens

ity

posterior

prior

0.00

0.25

0.50

0.75

2.5 5.0 7.5 10.0K_1D

dens

ity

posterior

prior

0.0e+00

2.5e−05

5.0e−05

7.5e−05

820000 840000 860000Paco

dens

ity

posterior

prior

0.0e+00

5.0e-09

1.0e-08

1.5e-08

2.0e-08

2.5e-08

0e+00 1e+09 2e+09 3e+09σδ

2

dens

ity

posteriorprior

0e+00

1e-08

2e-08

3e-08

4e-08

0e+00 2e+09 4e+09 6e+09σerr

2

dens

ity

posteriorprior

Figure 5.11: Prior and posterior densities for each parameter in a M ′4 calibration.

Figure 5.11 shows that the maximum a posteriori are the same for the second model. When the projection erroris taken into account in the model, the variance of the measurement error has decreased and has an acceptablevalue. However, when the discrepancy is introduced the posterior parameter densities have larger variances thanthe ones estimated without. It can be explained by the introduction of a new parameter σ2

δthat tends to increase

the uncertainty when the estimation is looked for. The modes a posteriori look coherent for both estimation. Theinterest of adding the discrepancy here is low because the variances of the parameter densities have increased and theestimation of the variance of the measurement error was not so bad with M ′

2. When the numerical code is accurate,the introduction of the discrepancy is maybe not necessary. Some methods, in statistical validation, allow to pickthe most likely model given experimental data between the one with discrepancy and the one without (Damblinet al., 2016).

The split estimation may have underestimated the variance a posteriori of the σ2δ

and σ2err variances. Another

practical solution could have been to compute the Maximum Likelihood Estimates and start the algorithm at thisestimator. The MLE can be expressed straightforwardly by deriving Equation 5.7 regarding σ2

err or σ2δ

. Indeed, ifthe log-likelihood of Equation (5.7) is derived regarding σ2

δthen we obtain:

(T −d)2(σ2

err +σ2δ)−||PδYexp||2

2(σ2err +σ2

δ)2 = 0, (5.11)

which leads to:

(σ2δ+σ

2err) =

||PδYexp||2

(T −d). (5.12)

If the derivative is done regarding σ2err, the following relationship is found:

126

Cha

pter

5

5.4. Bayesian calibration

− d2σ2

err+||PFF −PFYexp||2

2σ4err

=(T −d)

2(σ2err +σ2

δ)−||PδYexp||2

2(σ2err +σ2

δ)2 .

Thanks to Equation (5.11) the right term is equal to 0 and:

σ2err =

||PFF −PFYexp||2

d, (5.13)

σ2δ=||PδYexp||2

(T −d)−||PFF −PFYexp||2

d. (5.14)

The use of the PCA had allowed us to summarize the information contained in the time series output of the code,and to emulate the projections in a restricted space. The hypotheses made on the discrepancy can be discussed.Indeed, representing the discrepancy as the projection on the orthogonal space might be a too subtle representation.One could have think of a representation of the discrepancy as an error term on several axis and not on all theorthogonal space. The results obtained after calibration state that a numerical code which reproduces well enoughthe physical system does not need the add of the discrepancy. Regarding the results, the discrepancy just adda parameter to estimate and increases the variance a posteriori of the parameter densities. It could have beeninteresting to apply statistical validation, with the Bayes factor for example (Damblin et al., 2016), and compare theresults with the ones obtained in Chapter 3 where the discrepancy had an positive impact. The predictive aspect canalso be developed. The cross validation has not been tested on these models but a month of data could have beentaken off, calibration performed on the remaining test data set and predictive tests on the power could have beenrun on the month that have been taken off. The characterization of the meteorological data could also have beenconsidered. To get back in the frame of the article of Higdon et al. (2008), the numerical code could have beenmodified such as aX matrix, which represents the input variables, would be introduced in the input of the code.

127

Con

clus

ion

CONCLUSION AND PERSPECTIVES

The general framework of this thesis is Bayesian calibration of numerical codes. The objective was to better assesthe credibility interval a posteriori of the quantity of interest when using an industrial code. This objective is reallyimportant for industrial companies such as EDF because numerical codes are used in prediction in many contextsand these companies have difficulties to estimate the predictive error they could make. In the economical frameworkat EDF, it becomes relevant especially in the forecasting context where power plants are already built and data arealready gathered. Then, the numerical code initially used to predict the power before the construction of the plant isused again and the results are confronted with data. Calibration is then performed to better asses the knowledge ofthe input parameters of the code corresponding to the specific plant.

In this thesis, we have introduced four concepts of calibration that are using different kinds of numerical code.We reviewed the main statistical methods that are available in the literature and we have detailed the inferenceassociated with these statistical models. The purpose of the first part is to confront the models with and withoutdiscrepancy. We have noticed that, in the case of the numerical code, we used the discrepancy appears to be reallyimportant in the modeling. Indeed, the estimation of the variance of the measurement errors was not concordantwith the physical reality. Then, when the discrepancy was added, the estimation became correct which implies thatthe code was carrying an error not taken into account in the model without discrepancy. The second conclusioncomes when the initial numerical code turns out to be long to run. In this scenario case, a Gaussian process is usedto emulate the behavior of the code. In a real industrial case, numerical codes can be so expensive in time, thatwe only have recourse to a few code calls. In that case the estimations after calibration are deteriorating becausethe emulator is not precise enough. It means that the physical interpretation of the parameter density does notstand anymore and new density parameters corresponding to the, bad, emulator are found. If one is interested infinding the right parameter densities even with an emulator, it is possible to have recourse to an adaptive sequentialdesign that finds points regarding further calibration. It drastically improves the emulation quality and allows to findestimators in correspondence with a physical meaning.

Based on the models developed in the first part, I developed a package. The CaliCo package is coded inobject-oriented language that allows to manipulate easily users requests. It performs Bayesian calibration withrecourse to MCMC that are coded in C++ to improve the speed of these time consuming operations. The packageprovides flexibility on the model choice which makes it different from other packages published so far. For eachstep, visual representations are available and the user can easily take advantage of each graph. A graphical interfacehas also been developed so that EDF will be able to use the package and take advantage of the coded functions forrunning test and illustrates the improvements they could make using calibration.

The third part focuses on a real test case which brings lots of issues. Indeed, instead of having a scalar output,the code is producing time series outputs. It is a common fact for numerical codes in industries, so calibration has tobe adapted to be performed on these codes. To deal with this issue, Higdon et al. (2008) have proposed to run aPCA on a DOE that encompass the simulations of the code. The aim is to reproduce the time series output witha limited amount of Gaussian processes. Indeed the PCA allows to specify a reasonable number of axis that arecarrying a majority of the total inertia of the simulations. Then, the projected simulations on these axes are usedto estimate the parameters of the Gaussian processes (when one Gaussian process is used to emulate the behaviorof the numerical code on one axis). In this particular case we have considered the discrepancy as the projection

129

Con

clus

ion

Chapter 6 – Conclusion and perspectives

on the orthogonal subspace. We have detailed the inference associated with these statistical models and proposedand applied them on a numerical code that predicts the power of a large PV power plant. The result show that theestimation of the parameters of the code are coherent in both cases. The variance a posteriori of the parameterdensities is higher when the estimation is performed with the model with discrepancy than with the model without.However, the variance of the measurement errors is a little higher when it is estimated with the model without dis-crepancy but when the discrepancy is introduced, the value stands to decrease to have more reasonable physical sense.

The predictive aspect of the models that encompasses the PCA has to be completed. It could be interesting tovisualize how the models works in prediction. Data already gathered on the field could also be split by keeping amonth of data as a validation data set. However, with such a complex model and code, for each test set, the DOEhas to be computed again and the PCA performed at each time. Such an operation could turn out to be intractablefor desktop computers. The impact of the discrepancy is also the subject of statistical validation. Some studieshighlight the fact that data gathered on the field also allow to elect the right model. Bayes factor (Damblin, 2015) ormixture models (Kamary, 2016) have been developed and it could be interesting to try out these methods on theapplication cases provided in this thesis. Working with a time consuming code also turned out to be difficult todeal with. To run proper calibration on a function that emulates the code, this function has to be close to the initialnumerical code, otherwise the prediction of the parameter could loose their physical sense. It could be interesting tostudy deeper the way to find adapted designs for calibration or other kind of emulators. Also calibration on nestednumerical codes could be a real challenge. Indeed, emulating a nested code properly with a Gaussian process mightbe a difficult task and integrating it in a Bayesian calibration framework would be interesting for companies that areusing this kind of codes.

130

BIBLIOGRAPHY

Albert, I., Donnet, S., Guihenneuc-Jouyaux, C., Low-Choy, S., Mengersen, K., Rousseau, J., et al. (2012). Combiningexpert opinions in prior elicitation. Bayesian Analysis, 7(3):503–532.

Andrieu, C., De Freitas, N., Doucet, A., and Jordan, M. I. (2003). An introduction to mcmc for machine learning.Machine learning, 50(1-2):5–43.

Auffray, Y., Barbillon, P., and Marin, J.-M. (2010). Maximin design on non hypercube domain and kernelinterpolation. arXiv preprint arXiv:1004.0784.

Bachoc, F., Bois, G., Garnier, J., and Martinez, J.-M. (2014). Calibration and improved prediction of computermodels by universal kriging. Nuclear Science and Engineering, 176(1):81–97.

Bayarri, M. J., Berger, J. O., Paulo, R., Sacks, J., Cafeo, J. A., Cavendish, J., Lin, C.-H., and Tu, J. (2007). Aframework for validation of computer models. Technometrics, 49(2):138–154.

Bull, A. D. (2011). Convergence rates of efficient global optimization algorithms. Journal of Machine Learning

Research, 12(Oct):2879–2904.

Carmassi, M. (2018). CaliCo: Code Calibration in a Bayesian Framework. R package version 0.1.0.

Casella, G. and George, E. I. (1992). Explaining the gibbs sampler. The American Statistician, 46(3):167–174.

Chang, W. (2017). R6: Classes with Reference Semantics. R package version 2.2.2.

Cox, D. D., Park, J.-S., and Singer, C. E. (2001). A statistical method for tuning a computer code to a data base.Computational statistics & data analysis, 37(1):77–92.

Craig, P. S., Goldstein, M., Rougier, J. C., and Seheult, A. H. (2001). Bayesian forecasting for complex systemsusing computer simulators. Journal of the American Statistical Association, 96(454):717–729.

Cressie, N. A. and Noel, A. (1993). Cassie (1993). statistics for spatial data. vol. 900.

Currin, C., Mitchell, T., Morris, M., and Ylvisaker, D. (1991). Bayesian prediction of deterministic functions, withapplications to the design and analysis of computer experiments. Journal of the American Statistical Association,86(416):953–963.

Da Veiga, S. (2015). Global sensitivity analysis with dependence measures. Journal of Statistical Computation and

Simulation, 85(7):1283–1305.

Damblin, G. (2015). Contributions statistiques au calage et à la validation des codes de calcul. PhD thesis, PhDthesis, Université Paris Saclay.

Damblin, G., Barbillon, P., Keller, M., Pasanisi, A., and Parent, É. (2018). Adaptive numerical designs for thecalibration of computer codes. SIAM/ASA Journal on Uncertainty Quantification, 6(1):151–179.

Damblin, G., Couplet, M., and Iooss, B. (2013). Numerical studies of space-filling designs: optimization of latinhypercube samples and subprojection properties. Journal of Simulation, 7(4):276–289.

131

Damblin, G., Keller, M., Barbillon, P., Pasanisi, A., and Parent, É. (2016). Bayesian model selection for thevalidation of computer codes. Quality and Reliability Engineering International, 32(6):2043–2054.

De Lozzo, M. and Marrel, A. (2016). New improvements in the use of dependence measures for sensitivity analysisand screening. Journal of Statistical Computation and Simulation, 86(15):3038–3058.

Ding, J. and Radhakrishnan, R. (2008). A new method to determine the optimum load of a real solar cell using thelambert w-function. Solar Energy Materials and Solar Cells, 92(12):1566–1569.

Duffie, J. A. and Beckman, W. A. (2013). Solar engineering of thermal processes. John Wiley & Sons.

Dussert, C., Rasigni, G., Rasigni, M., Palmari, J., and Llebaria, A. (1986). Minimal spanning tree: A new approachfor studying order and disorder. Physical Review B, 34(5):3528.

Eddelbuettel, D., Francois, R., Allaire, J., Ushey, K., Kou, Q., Russell, N., Bates, D., and Chambers, J. (2018).Rcpp: Seamless R and C++ Integration. R package version 0.12.16.

Faivre, R., Iooss, B., Mahévas, S., Makowski, D., and Monod, H. (2013). Analyse de sensibilité et exploration de

modèles: application aux sciences de la nature et de l’environnement. Editions Quae.

Fang, K.-T., Li, R., and Sudjianto, A. (2005). Design and modeling for computer experiments. CRC Press.

Franco, J., Dupuy, D., Roustant, O., Damblin, G., and Iooss, B. (2015). DiceDesign: Designs of Computer

Experiments. R package version 1.7.

Franco, J., Vasseur, O., Corre, B., and Sergent, M. (2009). Minimum spanning tree: A new approach to assess thequality of the design of computer experiments. Chemometrics and intelligent laboratory systems, 97(2):164–169.

Gelman, A., Carlin, J. B., Stern, H. S., and Rubin, D. B. (1995). Bayesian data analysis. Chapman and Hall/CRC.

Gelman, A. and Rubin, D. B. (1992). Inference from iterative simulation using multiple sequences. Statistical

science, pages 457–472.

Genz, A., Bretz, F., Miwa, T., Mi, X., Leisch, F., Scheipl, F., Bornkamp, B., Maechler, M., and Hothorn, T. (2018).mvtnorm: Multivariate Normal and t Distributions. R package version 1.0-7.

Ginsbourger, D. (2009). Multiples metamodeles pour l’approximation et l’optimisation de fonctions numeriques

multivariables. PhD thesis, Ecole Nationale Supérieure des Mines de Saint-Etienne.

Gu, M. (2018a). Jointly robust prior for gaussian stochastic process in emulation, calibration and variable selection.arXiv preprint arXiv:1804.09329.

Gu, M. (2018b). RobustCalibration: Robust Calibration of Imperfect Mathematical Models. R package version0.5.1.

Gu, M. and Wang, L. (2017). Scaled gaussian stochastic process for computer model calibration and prediction.

Haario, H., Saksman, E., Tamminen, J., et al. (2001). An adaptive metropolis algorithm. Bernoulli, 7(2):223–242.

Handcock, M. S. and Stein, M. L. (1993). A bayesian analysis of kriging. Technometrics, 35(4):403–410.

Hankin, R. K. S. (2013a). approximator: Bayesian prediction of complex computer codes. R package version 1.2-6.

Hankin, R. K. S. (2013b). BACCO: Bayesian Analysis of Computer Code Output (BACCO). R package version2.0-9.

Hankin, R. K. S. (2013c). calibrator: Bayesian calibration of complex computer codes. R package version 1.2-6.

132

Hankin, R. K. S. (2014). emulator: Bayesian emulation of computer programs. R package version 1.2-15.

Hastings, W. K. (1970). Monte carlo sampling methods using markov chains and their applications. Biometrika,57(1):97–109.

Helbert, C., Dupuy, D., and Carraro, L. (2009). Assessment of uncertainty in computer experiments from universalto bayesian kriging. Applied Stochastic Models in Business and Industry, 25(2):99–113.

Higdon, D., Gattiker, J., Williams, B., and Rightley, M. (2008). Computer model calibration using high-dimensionaloutput. Journal of the American Statistical Association, 103(482):570–583.

Higdon, D., Kennedy, M., Cavendish, J. C., Cafeo, J. A., and Ryne, R. D. (2004). Combining field data andcomputer simulations for calibration and prediction. SIAM Journal on Scientific Computing, 26(2):448–466.

Hoff, P. D. (2009). A first course in Bayesian statistical methods. Springer Science & Business Media.

Husson, F., Josse, J., Le, S., and Mazet, J. (2018). FactoMineR: Multivariate Exploratory Data Analysis and Data

Mining. R package version 1.41.

Husson, F., Lê, S., and Pagès, J. (2017). Exploratory multivariate analysis by example using R. Chapman andHall/CRC.

Ishaque, K., Salam, Z., Taheri, H., et al. (2011). Modeling and simulation of photovoltaic (pv) system during partialshading based on a two-diode model. Simulation Modelling Practice and Theory, 19(7):1613–1626.

Jin, R., Chen, W., and Sudjianto, A. (2003). An efficient algorithm for constructing optimal design of computerexperiments. In ASME 2003 International Design Engineering Technical Conferences and Computers and

Information in Engineering Conference, pages 545–554. American Society of Mechanical Engineers.

Johnson, M. E., Moore, L. M., and Ylvisaker, D. (1990). Minimax and maximin distance designs. Journal of

statistical planning and inference, 26(2):131–148.

Jones, D. R., Schonlau, M., and Welch, W. J. (1998). Efficient global optimization of expensive black-box functions.Journal of Global optimization, 13(4):455–492.

Kamary, K. (2016). Non-informative priors and modelization by mixtures. PhD thesis, PhD thesis, Université ParisDauphine.

Kansa, E. (1985). Application of hardy’s multiquadric interpolation to hydrodynamics. Technical report, LawrenceLivermore National Lab.

Kennedy, M. and O’Hagan, A. (2001). Supplementary details on bayesian calibration of computer. rap. tech.,university of nottingham. Statistics Section.

Kennedy, M. C. and O’Hagan, A. (2001). Bayesian calibration of computer models. Journal of the Royal Statistical

Society: Series B (Statistical Methodology), 63(3):425–464.

Konomi, B. A., Karagiannis, G., Lai, K., and Lin, G. (2017). Bayesian treed calibration: an application to carboncapture with ax sorbent. Journal of the American Statistical Association, 112(517):37–53.

Krige, D. G. (1951). A statistical approach to some basic mine valuation problems on the witwatersrand. Journal of

the Southern African Institute of Mining and Metallurgy, 52(6):119–139.

Le Gratiet, L. (2013). Multi-fidelity Gaussian process regression for computer experiments. PhD thesis, UniversitéParis-Diderot-Paris VII.

133

Liu, F., Bayarri, M., Berger, J., et al. (2009). Modularization in bayesian analysis, with emphasis on analysis ofcomputer models. Bayesian Analysis, 4(1):119–150.

MacDoanld, B., Chipman, H., and Ranjan, P. (2015). GPfit: Gaussian Processes Modeling. R package version1.0-0.

Marrel, A. (2008). Mise en oeuvre et exploitation du métamodèle processus gaussien pour l’analyse de modèlesnumériques-application à un code de transport hydrogéologique. These de l’INSA Toulouse.

Marrel, A., Iooss, B., Laurent, B., and Roustant, O. (2009). Calculations of sobol indices for the gaussian processmetamodel. Reliability Engineering & System Safety, 94(3):742–751.

Matérn, B. (1960). Spatial variation: Meddelanden fran statens skogsforskningsinstitut. Lecture Notes in Statistics,36:21.

Matheron, G. (1963). Principles of geostatistics. Economic geology, 58(8):1246–1266.

McKay, M. D., Beckman, R. J., and Conover, W. J. (1979). Comparison of three methods for selecting values ofinput variables in the analysis of output from a computer code. Technometrics, 21(2):239–245.

Mengyang Gu, J. P. and Berger, J. (2018). RobustGaSP: Robust Gaussian Stochastic Process Emulation. R packageversion 0.5.5.

Metropolis, N., Rosenbluth, A. W., Rosenbluth, M. N., Teller, A. H., and Teller, E. (1953). Equation of statecalculations by fast computing machines. The journal of chemical physics, 21(6):1087–1092.

Mira, A. et al. (2001). On metropolis-hastings algorithms with delayed rejection. Metron, 59(3-4):231–241.

Morris, M. D. (1991). Factorial sampling plans for preliminary computational experiments. Technometrics,33(2):161–174.

Morris, M. D. and Mitchell, T. J. (1995). Exploratory designs for computational experiments. Journal of statistical

planning and inference, 43(3):381–402.

Oakley, J. E. and O’Hagan, A. (2004). Probabilistic sensitivity analysis of complex models: a bayesian approach.Journal of the Royal Statistical Society: Series B (Statistical Methodology), 66(3):751–769.

Owen, A. B. (1992). Orthogonal arrays for computer experiments, integration and visualization. Statistica Sinica,pages 439–452.

Palomo, J., Garcia-Donato, G., Paulo, R., Berger, J., Bayarri, M. J., and Sacks, J. (2017). SAVE: Bayesian Emulation,

Calibration and Validation of Computer Models. R package version 1.0.

Pearson, K. (1901). Liii. on lines and planes of closest fit to systems of points in space. The London, Edinburgh,

and Dublin Philosophical Magazine and Journal of Science, 2(11):559–572.

Petrone, G., Spagnuolo, G., and Vitelli, M. (2007). Analytical model of mismatched photovoltaic fields by means oflambert w-function. Solar energy materials and solar cells, 91(18):1652–1657.

Picault, D., Raison, B., Bacha, S., De La Casa, J., and Aguilera, J. (2010). Forecasting photovoltaic array powerproduction subject to mismatch losses. Solar Energy, 84(7):1301–1309.

Plumlee, M. (2017). Bayesian calibration of inexact computer models. Journal of the American Statistical

Association, 112(519):1274–1285.

134

Plummer, M., Best, N., Cowles, K., Vines, K., Sarkar, D., Bates, D., Almond, R., and Magnusson, A. (2016). coda:

Output Analysis and Diagnostics for MCMC. R package version 0.19-1.

Pronzato, L. and Müller, W. G. (2012). Design of computer experiments: space filling and beyond. Statistics and

Computing, 22(3):681–701.

Rasmussen, C. E. (2004). Gaussian processes in machine learning. In Advanced lectures on machine learning,pages 63–71. Springer.

Robert, C. (1996). Méthodes de Monte Carlo par chaînes de Markov. Economica.

Robert, C. (2007). The Bayesian choice: from decision-theoretic foundations to computational implementation.Springer Science & Business Media.

Robert, C. and Casella, G. (2013). Monte Carlo statistical methods. Springer Science & Business Media.

Roberts, G. O., Gelman, A., Gilks, W. R., et al. (1997). Weak convergence and optimal scaling of random walkmetropolis algorithms. The annals of applied probability, 7(1):110–120.

Rocquigny, E. d. (2009). Quantifying uncertainty in an industrial approach: an emerging consensus in an oldepistemological debate. SAPI EN. S. Surveys and Perspectives Integrating Environment and Society, (2.1).

Roustant, O., Gainsbourger, D., and Deville, Y. (2015). DiceKriging: Kriging Methods for Computer Experiments.R package version 1.5.5.

Roustant, O., Ginsbourger, D., and Deville, Y. (2012). Dicekriging, diceoptim: Two r packages for the analysisof computer experiments by kriging-based metamodelling and optimization. Journal of Statistical Software,51(1):54p.

Sacks, J., Welch, W. J., Mitchell, T. J., and Wynn, H. P. (1989). Design and analysis of computer experiments.Statistical science, pages 409–423.

Saltelli, A. (2002). Making best use of model evaluations to compute sensitivity indices. Computer physics

communications, 145(2):280–297.

Saltelli, A., Chan, K., Scott, E. M., et al. (2000). Sensitivity analysis, volume 1. Wiley New York.

Saltelli, A., Tarantola, S., Campolongo, F., and Ratto, M. (2004). Sensitivity analysis in practice: a guide to

assessing scientific models. John Wiley & Sons.

Santner, T. J., Williams, B. J., and Notz, W. I. (2013). The design and analysis of computer experiments. SpringerScience & Business Media.

Schaback, R. (2007). Convergence of unsymmetric kernel-based meshless collocation methods. SIAM Journal on

Numerical Analysis, 45(1):333–351.

Sobol’, I. M. (1990). On sensitivity estimation for nonlinear mathematical models. Matematicheskoe modelirovanie,2(1):112–118.

Sobol, I. M. (1993). Sensitivity estimates for nonlinear mathematical models. Mathematical modelling and

computational experiments, 1(4):407–414.

Stein, M. (1987). Large sample properties of simulations using latin hypercube sampling. Technometrics, 29(2):143–151.

Stein, M. L. (2012). Interpolation of spatial data: some theory for kriging. Springer Science & Business Media.

135

Sudret, B. (2008). Global sensitivity analysis using polynomial chaos expansions. Reliability Engineering & System

Safety, 93(7):964–979.

Tang, B. (1993). Orthogonal array-based latin hypercubes. Journal of the American statistical association,88(424):1392–1397.

Tian, H., Mancilla-David, F., Ellis, K., Muljadi, E., and Jenkins, P. (2012). A cell-to-module-to-array detailed modelfor photovoltaic panels. Solar energy, 86(9):2695–2706.

Tuo, R., Wu, C. J., et al. (2015). Efficient calibration for imperfect computer models. The Annals of Statistics,43(6):2331–2352.

Tuo, R. and Wu, J. (2016). A theoretical framework for calibration in computer models: parametrization, estimationand convergence properties. SIAM/ASA Journal on Uncertainty Quantification, 4(1):767–795.

Vazquez, E. and Bect, J. (2010). Convergence properties of the expected improvement algorithm with fixed meanand covariance functions. Journal of Statistical Planning and inference, 140(11):3088–3095.

Viana, F. A., Venter, G., and Balabanov, V. (2010). An algorithm for fast optimal latin hypercube design ofexperiments. International journal for numerical methods in engineering, 82(2):135–156.

Wickham, H. and Chang, W. (2016). ggplot2: Create Elegant Data Visualisations Using the Grammar of Graphics.R package version 2.2.1.

Wong, R. K., Storlie, C. B., and Lee, T. (2017). A frequentist approach to computer model calibration. Journal of

the Royal Statistical Society: Series B (Statistical Methodology), 79(2):635–648.

136

Titre : Quantification des incertitudes et calage d’un modele de centrale photovoltaıque : garantie de perfor-mance et estimation robuste de la production long-terme.

Mots cles : Centrale photovoltaıque / Calage bayesien / Quantification d’incertitudes / Code numerique

Resume : Les difficultes de mise en œuvred’experiences de terrain ou de laboratoire, ainsi queles couts associes, conduisent les societes indus-trielles a se tourner vers des codes numeriques decalcul. Ces codes, censes etre representatifs desphenomenes physiques en jeu, entraınent neanmoinstout un cortege de problemes. Le premier de cesproblemes provient de la volonte de predire larealite a partir d’un modele informatique. En ef-fet, le code doit etre representatif du phenomeneet, par consequent, etre capable de simuler desdonnees proches de la realite. Or, malgre le constantdeveloppement du realisme de ces codes, des er-reurs de prediction subsistent. Elles sont de deux na-tures differentes. La premiere provient de la differenceentre le phenomene physique et les valeurs releveesexperimentalement. La deuxieme concerne l’ecartentre le code developpe et le phenomene physique.Pour diminuer cet ecart, souvent qualifie de biais oud’erreur de modele, les developpeurs complexifienten general les codes, les rendant tres chronophages

dans certains cas. De plus, le code depend de pa-rametres a fixer par l’utilisateur qui doivent etre choi-sis pour correspondre au mieux aux donnees de ter-rain. L’estimation de ces parametres propres au codes’appelle le calage. Cette these propose dans unpremier temps une revue des methodes statistiquesnecessaires a la comprehension du calage Bayesien.Ensuite, une revue des principales methodes de ca-lage est presentee accompagnee d’un exemple com-paratif base sur un un code de calcul servant a predirela puissance d’une centrale photovoltaıque. Le pa-ckage appele CaliCo qui permet de realiser un calagerapide de beaucoup de codes numeriques est alorspresente. Enfin, un cas d’etude reel d’une grandecentrale photovoltaıque sera introduit et le calagerealise pour effectuer un suivi de performance de lacentrale. Ce cas de code industriel particulier intro-duit des specificites de calage numeriques qui serontabordees et deux modeles statistiques y seront ex-poses.

Title : Uncertainty quantification and calibration of a photovoltaic plant model: warranty of performance androbust estimation of the long-term production.

Keywords : Photovoltaic power plant / Bayesian calibration / Uncertainty quantification / Numerical code

Abstract : Field experiments are often difficult andexpensive to make. To bypass these issues, indus-trial companies have developed computational codes.These codes intend to be representative of the phy-sical system, but come with a certain amount of pro-blems. The code intends to be as close as possibleto the physical system. It turns out that, despite conti-nuous code development, the difference between thecode outputs and experiments can remain significant.Two kinds of uncertainties are observed. The firstone comes from the difference between the physi-cal phenomenon and the values recorded experimen-tally. The second concerns the gap between the codeand the physical system. To reduce this difference, of-ten named model bias, discrepancy, or model error,computer codes are generally complexified in order tomake them more realistic. These improvements lead

to time consuming codes. Moreover, a code often de-pends on parameters to be set by the user to make thecode as close as possible to field data. This estimationtask is called calibration. This thesis first proposes areview of the statistical methods necessary to unders-tand Bayesian calibration. Then, a review of the maincalibration methods is presented with a comparativeexample based on a numerical code used to predictthe power of a photovoltaic plant. The package calledCaliCo which allows to quickly perform a Bayesian ca-libration on a lot of numerical codes is then presented.Finally, a real case study of a large photovoltaic powerplant will be introduced and the calibration carried outas part of a performance monitoring framework. Thisparticular case of industrial code introduces numeri-cal calibration specificities that will be discussed withtwo statistical models.

Universite Paris-SaclayEspace Technologique / Immeuble DiscoveryRoute de l’Orme aux Merisiers RD 128 / 91190 Saint-Aubin, France