Upload
others
View
5
Download
0
Embed Size (px)
Citation preview
Mathematical Modelling and Statistical
Analysis of School-Based Student
Performance Data
Jessica Y. C. Tan
Thesis submitted for the degree of
Master of Philosophy
in
Applied Mathematics and Statistics
at
The University of Adelaide
School of Mathematical Sciences
January 2013
Contents
Page
List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii
Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii
Signed Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix
Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxi
1 Introduction 1
1.1 The Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Background and Literature Review . . . . . . . . . . . . . . . . . . . 2
1.2.1 National Assessment Program - Literacy and Numeracy . . . . 2
1.2.2 My School . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2.3 Value-added Measurement . . . . . . . . . . . . . . . . . . . . 6
Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.2.4 Rasch Scaling . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.2.5 The Estimation of School E�ects . . . . . . . . . . . . . . . . 12
1.2.6 Hierarchical and Longitudinal Modelling . . . . . . . . . . . . 13
1.3 Underlying Research Question . . . . . . . . . . . . . . . . . . . . . . 15
1.4 Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2 Data Analysis 17
2.1 The Basic Skills Test Data . . . . . . . . . . . . . . . . . . . . . . . 17
2.2 The Variables: Background Information . . . . . . . . . . . . . . . . 18
2.3 Descriptive Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.3.1 Number of Participants in Schools . . . . . . . . . . . . . . . 21
2.3.2 Univariate Statistics . . . . . . . . . . . . . . . . . . . . . . . 21
Drop-o� in participants in 2004 . . . . . . . . . . . . . . . . . 22
iii
Test scores . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Cohort Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.3.3 Number of Tests: Longitudinal View . . . . . . . . . . . . . . 43
2.3.4 Test Aspects . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
2.3.5 Missing Test Aspects . . . . . . . . . . . . . . . . . . . . . . 44
Missing Literacy Writing Results . . . . . . . . . . . . . . . . 46
Goodness of Fit - Binomial Model . . . . . . . . . . . . . . . . 47
2.4 Cleaning the Data: Forensic Statistics . . . . . . . . . . . . . . . . . 49
2.4.1 Consistency of School Data . . . . . . . . . . . . . . . . . . . 49
2.4.2 Consistency of Student Data . . . . . . . . . . . . . . . . . . 50
2.4.3 Score Check . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
2.4.4 Inference of Data . . . . . . . . . . . . . . . . . . . . . . . . . 61
2.5 Bivariate Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
2.5.1 Categorical Variables . . . . . . . . . . . . . . . . . . . . . . . 61
2.5.2 Quantitative Variables . . . . . . . . . . . . . . . . . . . . . . 71
2.5.3 Principal Component Analysis (PCA) . . . . . . . . . . . . . . 74
Background Theory . . . . . . . . . . . . . . . . . . . . . . . . 74
Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
2.6 Final Data Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
3 Initial Model Selection 81
3.1 Manual Reduction of Data and Predictor Variables . . . . . . . . . . 81
Categorical Variables . . . . . . . . . . . . . . . . . . . . . . . 82
Quantitative Variables . . . . . . . . . . . . . . . . . . . . . . 82
Missing Data . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
3.2 Statistical Reduction of Predictor Variables . . . . . . . . . . . . . . . 83
3.2.1 Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
Linear Regression . . . . . . . . . . . . . . . . . . . . . . . . . 84
Model Selection . . . . . . . . . . . . . . . . . . . . . . . . . . 84
3.2.2 Signi�cant Variables in Linear Regression . . . . . . . . . . . 85
School-Number Model . . . . . . . . . . . . . . . . . . . . . . 86
School-Covariates Model . . . . . . . . . . . . . . . . . . . . . 90
3.2.3 Simplest Main E�ects Model using stepAIC . . . . . . . . . . 91
School-Number Model . . . . . . . . . . . . . . . . . . . . . . 94
School-Covariates Model . . . . . . . . . . . . . . . . . . . . . 99
3.2.4 Investigation of procyear . . . . . . . . . . . . . . . . . . . . 106
3.2.5 Comparison of School-Number and School-Covariates Models . 107
Statistical Theory . . . . . . . . . . . . . . . . . . . . . . . . . 111
Transformed Data . . . . . . . . . . . . . . . . . . . . . . . . 113
Signi�cant Schools . . . . . . . . . . . . . . . . . . . . . . . . 114
3.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
4 Hierarchical Modelling: Mixed E�ects Model 119
4.1 Theory of Linear Multilevel Mixed E�ects Models . . . . . . . . . . . 120
4.2 Hierarchical Model Formulation . . . . . . . . . . . . . . . . . . . . . 122
4.3 Hierarchical Model Selection . . . . . . . . . . . . . . . . . . . . . . . 124
4.3.1 Model Selection by Markov Chain Monte Carlo Sampling . . . 126
4.3.2 Model Selection by Likelihood Ratio Test . . . . . . . . . . . . 128
4.3.3 Model Selection by glmulti . . . . . . . . . . . . . . . . . . . 130
4.4 Analysis of Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
5 Bayesian Hierarchical Modelling 135
5.1 Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
5.1.1 Bayesian Statistics . . . . . . . . . . . . . . . . . . . . . . . . 135
5.1.2 Markov Chain Monte Carlo Simulation and Sampling . . . . . 136
Gibbs Sampling . . . . . . . . . . . . . . . . . . . . . . . . . . 137
5.2 Hierarchical Modelling Using BUGS . . . . . . . . . . . . . . . . . . . 138
5.2.1 The Hierarchical Model . . . . . . . . . . . . . . . . . . . . . . 138
5.2.2 The Program . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
5.2.3 Directed Acyclic Graphs . . . . . . . . . . . . . . . . . . . . . 140
5.2.4 Analysis of BUGS output . . . . . . . . . . . . . . . . . . . . 141
Validity of the Model: Diagnosis of Convergence . . . . . . . . 144
Visualisation of Results . . . . . . . . . . . . . . . . . . . . . . 146
5.3 Hierarchical Modelling Using Stan . . . . . . . . . . . . . . . . . . . . 153
5.3.1 Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
Hamiltonian Monte Carlo . . . . . . . . . . . . . . . . . . . . 153
No-U-Turn Sampler . . . . . . . . . . . . . . . . . . . . . . . . 154
5.3.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
Validity of the Model: Diagnosis of Convergence . . . . . . . . 155
Interpretation of Regression Coe�cients . . . . . . . . . . . . 158
6 Model Validation 159
6.1 Student-level Prediction . . . . . . . . . . . . . . . . . . . . . . . . . 159
6.1.1 Prediction Intervals for lmer . . . . . . . . . . . . . . . . . . . 160
6.1.2 Prediction Intervals from Stan . . . . . . . . . . . . . . . . . . 161
6.1.3 Comparison of lmer and Stan . . . . . . . . . . . . . . . . . . 162
6.2 Analysis of School E�ect . . . . . . . . . . . . . . . . . . . . . . . . . 165
6.3 Heteroscedasticity and School Size . . . . . . . . . . . . . . . . . . . . 169
6.4 Conclusion and Impact . . . . . . . . . . . . . . . . . . . . . . . . . . 171
7 Initial Longitudinal Analysis 173
7.1 Summary Statistics of Data from Sequential Tests . . . . . . . . . . . 173
7.2 Grade 3 and Grade 5 Tests . . . . . . . . . . . . . . . . . . . . . . . . 174
7.2.1 Individual Scores . . . . . . . . . . . . . . . . . . . . . . . . . 175
7.2.2 Di�erence in Scores . . . . . . . . . . . . . . . . . . . . . . . . 175
Simple Linear Regression . . . . . . . . . . . . . . . . . . . . . 178
Hierarchical Modelling using Linear Multilevel Mixed E�ects
Models . . . . . . . . . . . . . . . . . . . . . . . . . 182
7.3 Grade 3, 5 and 7 Tests . . . . . . . . . . . . . . . . . . . . . . . . . . 186
7.3.1 Longitudinal Modelling . . . . . . . . . . . . . . . . . . . . . . 191
8 Conclusion 195
8.1 Summary and Conclusions . . . . . . . . . . . . . . . . . . . . . . . . 195
8.2 Practical Implications and Future Work . . . . . . . . . . . . . . . . . 196
A Coding of Variables 201
A.1 Test Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
A.2 School Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
A.3 Student Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
B Plots 207
B.1 Boxplots of LL Rasch and NN Rasch for Categorical Variables . . . . 207
B.2 Grade 3 and Grade 5 Tests . . . . . . . . . . . . . . . . . . . . . . . 228
C Output 233
C.1 Chapter 3: Initial Model Selection . . . . . . . . . . . . . . . . . . . . 233
C.1.1 Full model with Main E�ects . . . . . . . . . . . . . . . . . . 233
School-Number Model . . . . . . . . . . . . . . . . . . . . . . 233
School-Covariates Model . . . . . . . . . . . . . . . . . . . . . 244
C.1.2 Simplest Main E�ects Model by stepAIC . . . . . . . . . . . . 245
School-Number Model . . . . . . . . . . . . . . . . . . . . . . 245
School-Covariates Model . . . . . . . . . . . . . . . . . . . . . 256
C.2 Chapter 7. Initial Longitudinal Analysis . . . . . . . . . . . . . . . . 258
C.2.1 Grade 3 and Grade 5 Tests . . . . . . . . . . . . . . . . . . . 258
School-Number Model . . . . . . . . . . . . . . . . . . . . . . 258
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
List of Tables
Page
2.2.1 Test variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.2.2 School variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.2.3 Student variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.3.1 Number and percentage of participants in each calendar year . . . 22
2.3.2 Number and percentage of participants in each grade . . . . . . . . 22
2.3.3 Number of participants for each calendar year and grade . . . . . . 23
2.3.4 Number of participants and schools each year . . . . . . . . . . . . 24
2.3.5 Descriptive statistics for raw and Rasch scores for Literacy and Nu-
meracy aspects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.3.6 Coding for cohort numbers . . . . . . . . . . . . . . . . . . . . . . . 28
2.3.7 Number of participants in each cohort . . . . . . . . . . . . . . . . . 29
2.3.8 Mean scores for each cohort in each grade . . . . . . . . . . . . . . . 42
2.3.9 Number of participants by year, grade and number of tests to date . 43
2.3.10 Number of participants with test aspects in each year and grade . . 45
2.3.11 Number and proportion of participants with missing aspects . . . . 46
2.3.12 Number of participants with missing LW by year . . . . . . . . . . . 46
2.3.13 Number of participants with missing scores for aspects . . . . . . . 48
2.3.14 Observed and expected frequencies for the goodness-of-�t χ2 test of
the Binomial model . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
2.4.1 Anomalies in student data . . . . . . . . . . . . . . . . . . . . . . . 51
2.4.2 Number of schools and participants with anomalies in the sum of
scores . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
2.4.3 Number of participants in 2003 and 2004 with Literacy Flag 1 . . . 52
2.4.4 Number of participants in 2001 and 2002 with Literacy Flag 3 . . . 53
ix
2.4.5 Linear regression output from LL vs (LR+LS+LW) for Literacy
Flag 1 tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
2.4.6 Linear regression output from LL vs LR, LS and LW individually
for Literacy Flag 1 tests . . . . . . . . . . . . . . . . . . . . . . . . 58
2.5.1 ANOVA for LL Rasch against p_g_nesb . . . . . . . . . . . . . . . 62
2.5.2 ANOVA for NN Rasch against p_g_nesb . . . . . . . . . . . . . . . 62
2.5.3 ANOVA for NN Rasch against visa_sub_c . . . . . . . . . . . . . . 62
2.5.4 P -value and largest di�erence in group means for categorical vari-
ables against LL Rasch and NN Rasch . . . . . . . . . . . . . . . . 65
2.5.5 P -value and R2 value for quantitative variables against LL Rasch
and NN Rasch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
2.5.6 The standard deviation, proportion of variance explained and the
cumulative proportion for each of the principal components . . . . . 76
3.2.1 Regression output for gpokm and school size from the school-covariates
model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
3.2.2 Regression output for gpokm and school size from the simplest-
stepAIC school-covariates model . . . . . . . . . . . . . . . . . . . . 102
3.2.3 Comparison of models with and without procyear - simplest-stepAIC
school-number model . . . . . . . . . . . . . . . . . . . . . . . . . . 102
3.2.4 Comparison of models with and without procyear - simplest-stepAIC
school-covariates model . . . . . . . . . . . . . . . . . . . . . . . . . 104
3.2.5 Comparison of models with and without p_g_nesb - simplest-stepAIC
school-number model . . . . . . . . . . . . . . . . . . . . . . . . . . 104
3.2.6 Comparison of models with and without p_g_nesb - simplest-stepAIC
school-covariates model . . . . . . . . . . . . . . . . . . . . . . . . . 104
3.2.7 Type II Anova test for the full school-number model . . . . . . . . . 105
3.2.8 Type II Anova test for the full school-covariates model . . . . . . . 105
3.2.9 Summary linear regression output of procyear in the school-number
model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
3.2.10 Summary linear regression output of procyear in the school-covariates
model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
3.2.11 Subset of data to illustrate raw and �tted scores . . . . . . . . . . . 108
3.2.12 Summary output of Raw ∼ Model linear regression . . . . . . . . . 108
3.2.13 Summary output of linear regression of transformed data . . . . . . 114
3.2.14 Comparison of the number of statistically signi�cant over- and under-
performing schools . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
3.2.15 χ2 test for association between the signi�cance groups of schools
and school covariates . . . . . . . . . . . . . . . . . . . . . . . . . . 117
4.3.1 Output of �xed and random e�ects from linear mixed e�ects model 125
4.3.2 Estimates and P -values of �xed and random e�ects estimated by
Markov Chain Monte Carlo sampling . . . . . . . . . . . . . . . . . 127
4.3.3 Estimates and P -values of �xed and random e�ects estimated by
maximum likelihood . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
5.2.1 Summary output of hierarchical model without procyear using
OpenBUGS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
5.3.1 Summary output of hierarchical model with procyear as �xed e�ect
using Stan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
6.1.1 Counts and proportions of students in each performance category
from the lmer model . . . . . . . . . . . . . . . . . . . . . . . . . . 161
6.1.2 Counts and proportions of students in each performance category
from the Stan model . . . . . . . . . . . . . . . . . . . . . . . . . . 162
6.1.3 3×3 table of the counts of students for performance categories under
lmer and Stan models . . . . . . . . . . . . . . . . . . . . . . . . . 162
7.1.1 Counts of students who sat two appropriate sequential tests . . . . . 174
7.2.1 ANOVA between Grade 3 and 5 scores and schoolno . . . . . . . . 177
7.2.2 Linear regression output of school-covariates model . . . . . . . . . 181
7.2.3 Output of �xed and random e�ects from linear mixed e�ects model 183
7.2.4 Estimates and P -values of �xed and random e�ects estimated by
Markov Chain Monte Carlo sampling . . . . . . . . . . . . . . . . . 184
7.2.5 Estimates and P -values of �xed and random e�ects estimated by
maximum likelihood . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
7.3.1 Counts of data for all the categorical predictors in the data of Grade
3, 5 and 7 scores . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
7.3.2 Counts of students in the categories of atsi and school_edu . . . . 193
C.1.1 Linear regression output of school-number model . . . . . . . . . . . 233
C.1.2 Linear regression output of school-covariates model . . . . . . . . . 244
C.1.3 Linear regression output of simplest-stepAIC school-number model . 245
C.1.4 Linear regression output of simplest-stepAIC school-covariates model256
C.2.1 Linear regression output of school-number model . . . . . . . . . . . 258
List of Figures
Page
1.2.1 National Assessment Scale for NAPLAN . . . . . . . . . . . . . . . 3
1.2.2 Exemplar of My School . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.3.1 Number of participants in each school. . . . . . . . . . . . . . . . . 21
2.3.2 Ratio of participants in 2004 compared to 2003 for all schools in all
grades. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.3.3 Histograms of Literacy raw scores over all years and grades. . . . . . 26
2.3.4 Histograms of Literacy Rasch scores over all years and grades. . . . 27
2.3.5 Boxplots of Literacy scores for the cohorts in Grade 3 . . . . . . . . 30
2.3.6 Boxplots of Numeracy scores for the cohorts in Grade 3 . . . . . . . 31
2.3.7 Boxplots of Literacy scores for the cohorts in Grade 5 . . . . . . . . 32
2.3.8 Boxplots of Numeracy scores for the cohorts in Grade 5 . . . . . . . 33
2.3.9 Boxplots of Literacy scores for the cohorts in Grade 7 . . . . . . . . 34
2.3.10 Boxplots of Numeracy scores for the cohorts in Grade 7 . . . . . . . 35
2.3.11 Histograms of raw and Rasch LL and LW Grade 3 scores in Cohort 8 37
2.3.12 Mean aspect score in each multi-year cohort and across grades. . . . 39
2.3.13 Cohort mean scores along grades in each Literacy aspect . . . . . . 40
2.3.14 Cohort mean scores along grades in each Numeracy aspect . . . . . 41
2.3.15 Proportion of participants in each school missing LW. . . . . . . . . 47
2.4.1 Proportion of participants in a school which have Literacy Flag 1
in 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
2.4.2 Proportion of participants in a school which have Literacy Flag 1
in 2004 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
2.4.3 Proportion of participants in a school which have Literacy Flag 3
in 2001 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
xiii
2.4.4 Proportion of participants in a school which have Literacy Flag 3
in 2002 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
2.4.5 Plot of LR+LS+LW versus LL for Literacy Flag 1 tests . . . . . . . 59
2.4.6 Plot of LR+LS+LW versus LL for Literacy Flag 1 tests . . . . . . . 60
2.5.1 Heatmap of the P -values from the χ2 test for independence between
the categorical explanatory variables. . . . . . . . . . . . . . . . . . 63
2.5.2 Boxplot of LL Rasch against disability . . . . . . . . . . . . . . . 67
2.5.3 Boxplot of LL Rasch against procyear . . . . . . . . . . . . . . . . 68
2.5.4 Boxplot of LL Rasch against gradedyear . . . . . . . . . . . . . . . 68
2.5.5 Boxplots of LL Rasch and NN Rasch for gender . . . . . . . . . . . 69
2.5.6 Boxplot of LL Rasch against procyear split into grades . . . . . . . 70
2.5.7 Heatmap of the correlation between the quantitative explanatory
variables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
2.5.8 Barplot of the variances explained by the principal components. . . 75
2.5.9 3D scatter plot of the scores of the �rst three principal components. 75
2.6.1 Flowchart of data sets. . . . . . . . . . . . . . . . . . . . . . . . . . 79
3.2.1 Coe�cient plot for the school-number model . . . . . . . . . . . . . 88
3.2.2 Subset of coe�cient plot for the school-number model . . . . . . . . 89
3.2.3 Coe�cient plot for the school-covariates model . . . . . . . . . . . . 92
3.2.4 Subset of coe�cient plot for the school-covariates model . . . . . . . 93
3.2.5 Coe�cient plot for the simplest-stepAIC school-number model . . . 96
3.2.6 Subset of coe�cient plot for the simplest-stepAIC school-number
model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
3.2.7 Coe�cients for the simplest-stepAIC school-number model plotted
against the regression coe�cients for the school-number model . . . 98
3.2.8 Coe�cient plot for the simplest-stepAIC school-covariates model . . 100
3.2.9 Subset of coe�cient plot for the simplest-stepAIC school-covariates
model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
3.2.10 Coe�cients for the simplest-stepAIC school-covariates model plot-
ted against the regression coe�cients for the school-covariates model 103
3.2.11 Linear regression of original raw mean school e�ects against the
�tted original model mean school e�ects . . . . . . . . . . . . . . . 109
3.2.12 Diagnostic plots to assess validity of assumptions in the linear re-
gression of Raw ∼ Model. . . . . . . . . . . . . . . . . . . . . . . . 110
3.2.13 Plot of residuals versus school size. . . . . . . . . . . . . . . . . . . . 111
3.2.14 Plot of residuals versus school size for linear regression on trans-
formed data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
3.2.15 Prediction intervals of the original and transformed data . . . . . . 116
4.0.1 Hierarchy of the school education system. . . . . . . . . . . . . . . . 120
4.3.1 AIC value for the best 100 models in the exhaustive search of the
main e�ects model . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
4.3.2 The relative weights of model terms . . . . . . . . . . . . . . . . . . 132
5.2.1 Directed acyclic graph for hierarchical model in equation (5.2.2). . . 141
5.2.2 Histogram of parameters' posterior distribution. . . . . . . . . . . . 146
5.2.3 Density plot of parameters' posterior distribution . . . . . . . . . . 147
5.2.4 Traceplots of parameters . . . . . . . . . . . . . . . . . . . . . . . . 148
5.2.5 Running means of parameters in each chain. . . . . . . . . . . . . . 148
5.2.6 Autocorrelation plots of parameters. . . . . . . . . . . . . . . . . . . 149
5.2.7 Crosscorrelations of parameters. . . . . . . . . . . . . . . . . . . . . 150
5.2.8 Highest posterior density plot for the school and student parameters.152
5.3.1 Comparison of NUTS with Metropolis and Gibbs sampling . . . . . 155
5.3.2 Autocorrelation plot of parameters from Stan. . . . . . . . . . . . . 156
6.1.1 Plot of the Stan �tted values against the lmer �tted values . . . . . 164
6.2.1 Ranked proportion of students who are over-performing in a school
plotted with the 95% con�dence interval . . . . . . . . . . . . . . . 165
6.2.2 Ranked proportion of students who are under-performing in a school
plotted with the 95% con�dence interval . . . . . . . . . . . . . . . 166
6.2.3 Normal Q-Q plot of the random e�ects estimates. . . . . . . . . . . 168
6.3.1 Plot of the variance of the residuals for each school against school size170
6.4.1 Student scores for school 115 . . . . . . . . . . . . . . . . . . . . . . 171
7.2.1 Individual Rasch scores for Grades 3 and 5 . . . . . . . . . . . . . . 176
7.2.2 Di�erence versus Average in Grade 3 and Grade 5 scores. . . . . . . 177
7.2.3 Mean di�erence in Grade 3 and Grade 5 values in each school . . . 178
7.3.1 Descriptives for the Grade 3, 5 and 7 data . . . . . . . . . . . . . . 188
7.3.2 Individual Rasch scores for Grades 3, 5 and 7 . . . . . . . . . . . . . 189
7.3.3 Possible trends between Grades 3, 5 and 7 . . . . . . . . . . . . . . 190
7.3.4 Longitudinal and hierarchical structure of the education system. . . 191
B.1.1 Boxplot of LL Rasch against procyear split into grades. . . . . . . 227
B.2.1 Descriptives for the Grade 3 and 5 data - page 1 . . . . . . . . . . . 229
B.2.2 Descriptives for the Grade 3 and 5 data - page 2 . . . . . . . . . . . 230
B.2.3 Descriptives for the Grade 3 and 5 data - page 3 . . . . . . . . . . . 231
Abstract
In order to improve the education system so that students are not only reaching the
minimum standards of literacy and numeracy but are also given the opportunity to
excel, accurate measures of school performance are vital. These measures need to
focus on student progress so that schools and teachers can focus on improving all
students - particularly those most in need.
A current trend in primary and secondary education in Australia is national testing,
with the introduction of the National Assessment Program - Literacy and Numeracy
(NAPLAN). This, along with similar tests like the Basic Skills Tests, measures
student performance and progress over sequential years. The issue now is how to
interpret these results for individuals, schools and educational authorities.
The education system can be modelled with a hierarchical model due to the nesting
of variables at the education system, school, classroom and student levels. Inter-
twined with the hierarchical nature of the model is also the longitudinal aspect of
the data as students sit biennial NAPLAN assessments. We mainly investigate hier-
archical modelling techniques, such as linear multilevel mixed e�ects and a Bayesian
approach, to �t a statistical model which incorporates student and school predictor
variables. In addition to hierarchical modelling methods, we also consider longitu-
dinal techniques.
The objective of this thesis is to investigate and determine what conclusions about
student progress and school performance can be reliably drawn from regular stan-
dardised system-wide assessment, such as NAPLAN. Various models are investigated
and the end result is a model which comprehensively and accurately models the re-
sults of students, from which we can predict students' scores and assess the e�ect of
schools in that way.
xvii
Declaration
I, Jessica Tan, certify that this work contains no material which has been accepted
for the award of any other degree or diploma in any university or other tertiary insti-
tution and to the best of my knowledge and belief, contains no material previously
published or written by another person, except where due reference has been made
in the text. In addition, I certify that no part of this work will, in the future, be
used in a submission for any other degree or diploma in any university or other ter-
tiary institution without the prior approval of the University of Adelaide and where
applicable, any partner institution responsible for the joint-award of this degree.
I give consent to this copy of my thesis, when deposited in the University Library,
being made available for loan and photocopying, subject to the provisions of the
Copyright Act 1968.
I also give permission for the digital version of my thesis to be made available
on the web, via the University's digital research repository, the Library catalogue
and also through web search engines, unless permission has been granted by the
University to restrict access for a period of time.
Signature: ....................... Date: .......................
xix
Acknowledgements
Firstly, I would like to thank my supervisors, Professor Nigel Bean and Dr Jono Tuke,
for their invaluable supervision and support during these two years of my Masters -
at the end of this journey, you are more like friends than supervisors. Both of you
have given generously of your time and mathematical expertise and your thoughts
are insightful and explanations clear, re�ecting the depth and breadth of your knowl-
edge. I accredit you and thank you for teaching me research skills and adding tools
to my virtual toolbox. I consider it a privilege that you both agreed to supervise me.
To my family, you get a big thank you. Thank you, Mum and Dad, for allow-
ing me to spend two years completing this Masters and for your continual support
and encouragement throughout my entire education. Thank you to my brother Dar-
ren for helping me take a break from work on occasion, and to my sisters, Sarah
and Hannah.
Thanks is also due to Dr Murray Thompson, a great friend and mentor from my
school days, who provided a wealth of information about the education theory as-
pect of my project. I would also like to acknowledge Professor John Keeves for his
help and Dr Darmawan for providing the data set.
My �home away from home� has been the School of Mathematical Sciences at the
University of Adelaide and I would like to express my appreciation for the wonderful
sense of community among the o�ce sta�, lecturers and students. A great part of
my enjoyment of university has been because of all my friends and fellow postgrads
- there are too many to name individually - but I would like to particularly thank
Pricey, Michael, Hayden, Sophie, Kale, Ben, Mingmei, Kate M, Kate R, Kyle, Lydia
xxi
A and Max for all the fun and laughs we have had.
�Mathematics is the queen of the sciences.�
(Carl Friedrich Gauss)
�The essence of mathematics is not to make simple things complicated,
but to make complicated things simple.�
(Stan Gudder)
�What lies behind us, and what lies before us are tiny matters,
compared to what lies within us.�
(Ralph Emerson)
Chapter 1
Introduction
1.1 The Problem
There has been substantial interest in policy activities related to outcomes-based
performance indicators [6, 8, 15, 22]. A performance indicator is a summary statis-
tical measurement on an institution or system which is related to the `quality' of
its functioning and may measure di�erent aspects of the system or re�ect di�erent
objectives. One application of performance indicators is in education and the grow-
ing demand for the accountability of teachers and schools. Coupled with this, is the
ranking of schools in school league tables [20, 23]. Many examples of the assess-
ment of education systems based on this structure exist internationally, for example
in England and Scotland [23, 39]. This topic especially applies to the Australian
education system in the last few years with the introduction of the NAPLAN tests
and the controversial My School 1TM
website. As a result of the My School website,
much debate has �ared up over the accuracy and interpretation of the ranking of
Australian schools.
Even with all the limitations of league tables and the associated statistical issues
involved in comparisons of school performance, it is possible that they are useful if
used carefully and in the proper context. However, care must be taken in the making
of conclusions and the question is �what can be sensibly and reliably concluded?�.
1 http://www.myschool.edu.au
1
2 1.2. Background and Literature Review
1.2 Background and Literature Review
To provide context to this problem, some of the research in measuring and esti-
mating school e�ects and the speci�cs of the Australian education system under
consideration will be discussed as background and a literature review.
1.2.1 National Assessment Program - Literacy and Numer-
acy
The National Assessment Program - Literacy and Numeracy (NAPLAN) was �rst
introduced in 2008 as part of the federal government initiative to support national
literacy and numeracy levels. NAPLAN is part of the National Assessment Program
(NAP), the measure through which governments, education authorities and schools
can determine whether or not young Australians are meeting important educational
outcomes [35]. NAP encompasses tests endorsed by the Ministerial Council for Edu-
cation, Early Childhood Development and Youth A�airs (MCEECDYA). Both NAP
and NAPLAN are under the jurisdiction of the Australian Curriculum, Assessment
and Reporting Authority (ACARA) which is �the independent authority responsible
for the development of a national curriculum, a national assessment program and a
national data collection and reporting program that supports 21st century learning
for all Australian students� [1].
NAPLAN replaced a variety of state-based exams, including the Basic Skills Test
in South Australia. All students in Years 3, 5, 7 and 9 of government and non-
government schools take part in the nation-wide standardised NAPLAN tests and
are examined in the domains of Reading, Writing, Language Conventions (Spelling,
Grammar and Punctuation) and Numeracy at their respective year levels. NAPLAN
is designed to test the requirements for literacy and numeracy common amongst the
curricula of each state and territory. NAPLAN tests are developed collaboratively by
the States and Territories, the Australian Government and non-government school
sectors with the aid of eminent assessment and educational measurement experts.
In addition, national protocols for test administration aim to ensure consistency
in administering the tests by all test administration authorities and schools across
Australia [1].
Chapter 1. Introduction 3
The NAPLAN test results are reported via individual student, school and national
reports. These provide information on how students have performed in relation
to other students in the same year group, against their state or territory, the na-
tional average and the National Minimum Standards. Student outcomes are re-
ported against achievement bands for each of the �ve NAPLAN assessment domains
of Reading, Writing, Spelling, Grammar and Punctuation and Numeracy (Figure
1.2.1). The range of the bands, from one to ten, re�ects the increasing complexity
of skills and understandings demonstrated by a student and assessed by NAPLAN
testing as the student progresses from Year 3 to Year 9. For any one year, the full
range of student performance is reported using six of the ten bands [1].
Figure 1.2.1: National Assessment Scale for NAPLAN. [35]
The use of common national assessment scales and reporting bands that span Years
3, 5, 7 and 9 enable the progress of an individual's or group's performance to be
monitored. To ensure that it is possible to compare scores from the current year's
tests with those of the previous years, test di�erences between the years are taken
into account, using a rigorous equating process [1].
Other results which are published in the NAPLAN National Report for each year
level, test domain and state, territory and Australia, are:
A NOTE:
This figure/table/image has been removed to comply with copyright regulations. It is included in the print copy of the thesis held by the University of Adelaide Library.
4 1.2. Background and Literature Review
• NAPLAN results by gender, Indigenous status, language background other
than English status, parental occupation, parental education and location,
• the performance of each state and territory relative to other states, territories
and the whole of Australia,
• participation rates and participation categories,
• comparison between the results of di�erent calendar years in the form of pair-
wise di�erences (classi�ed into average achievement statistically signi�cantly
higher, no statistically signi�cant di�erence and average achievement statisti-
cally signi�cantly lower), and
• cohort gains or the di�erence between results across two years of testing.
NAPLAN results are reported in the form of mean scale scores with variance repre-
sented by standard deviations or con�dence intervals of the mean.
1.2.2 My School
The My School website graphically and numerically publishes the NAPLAN results
for each school in Australia. However, it only displays the average score of a school's
students in the NAPLAN assessments for each domain and year level, the margin of
error at a 90% level of con�dence and compares a particular school to the average for
all Australian schools (Figure 1.2.2). TheMy School website also reports school per-
formance by comparing schools' NAPLAN scores within statistically-similar-school
groups. These school groups are determined by various demographic and socio-
economic factors like remoteness, Indigenous population and proxies of the socio-
economic status of the student population, summarised by the Index of Community
Socio-Educational Advantage (ICSEA) for each school [28, 41].
For each school, the My School website includes
• a school pro�le - information about the student and sta� population,
• information about school �nances,
• a list of `like schools' throughout Australia,
Chapter 1. Introduction 5
Figure 1.2.2: Exemplar of the My School website. [35]
• NAPLAN results in graphs, numbers and bands,
• a breakdown of the percentage of students in each NAP band for each domain,
the data is then compared with the national and `like school' results,
• student progress over time,
• a list of twenty local schools, including detailed results,
• data on vocational education and training program results, and
• the school Index of Community Socio-Economic Advantage (ICSEA) value.
NAPLAN results are reported using averages and their stated purpose is to be able
to measure the school e�ect. However, that concept is de�ned as something known
as value-added measurement.
A NOTE:
This figure/table/image has been removed to comply with copyright regulations. It is included in the print copy of the thesis held by the University of Adelaide Library.
6 1.2. Background and Literature Review
1.2.3 Value-added Measurement
Theory
There has been much discussion in the literature advocating the use of value-added
scores over raw averages [21, 38]. The Organisation of Economic Co-operation and
Development (OECD) de�ned school value-added as:
�The contribution of a school to students' progress towards stated or pre-
scribed education objectives (for example, cognitive achievement). The
contribution is net of other factors that contribute to students' educa-
tional progress.� [28]
Given the above de�nition, value-added modelling was similarly de�ned by the
OECD as:
�A class of statistical models that estimate the contributions of schools to
student progress in stated or prescribed education objectives (eg. cognitive
achievement) measured at at least two points in time.� [28]
Based on this theory, the major recommendation of the Grattan Institute report [28]
is to replace measurement of average school performance with so-called value-added
indices. The idea is to measure student progress as the primary outcome and employ
an appropriate statistical regression model to extract the school-attributable compo-
nent of the improvement. The report concluded that measuring student outcomes at
one time point, which is the current measure of school performance published on the
My School website in the form of NAPLAN scores, and averaging over each school
does not provide a valid measure of school performance. Obviously, this depends
on what one means by a school's performance. If one means the ability of a school
to attract and retain smart students while deterring less smart students, then the
average student outcome is probably a very meaningful measure. But if by school
performance, it refers to the e�ect of the school on the student's learning outcome,
then school averages will be hopelessly biased. The reason is that students are not
randomly allocated to schools, but rather, gifted students tend to concentrate in
some schools while disadvantaged students concentrate in others.
A school's value-added score represents the contribution the school makes to the
progress of its students, and hence, it is claimed that it measures school perfor-
Chapter 1. Introduction 7
mance more accurately, because it is better able to isolate the performance of the
school from other factors that a�ect student performance. The result is a fairer
system that is not biased against schools serving lower socio-economic communities.
School value-added scores are calculated using a statistical model which compares
the progress made by each student between assessments to the progress of other
students with the same initial level of attainment, measuring the contribution the
school makes to that progress and controlling for students' background factors. We
are particularly interested in measuring a school's contribution to student progress
between NAPLAN assessments of literacy and numeracy at Years 3, 5, 7 and 9.
At the moment, cross-sectional measures of student performance are being used
to indicate school performance. However, a school might be drastically improving
the test scores of its students, but be still achieving comparatively low scores in
the national or statewide tests when compared against statistically similar schools.
By the current de�nition of �good versus bad� amid the ranking of schools, such a
school would be classi�ed as a �bad� school. However, this may not be the case - a
more appropriate measure of school performance might be a longitudinal measure
of student improvement. Under such a de�nition, schools which produce improved
students would be justly recognised.
The education system's interest in the ranking of schools, in order to allocate funding
and identify schools which are not producing their expected results, should be based
on this longitudinal measure of improvement. In contrast, parents want to know
which school is best suited to their child. Improvement is still important, but parents
are not interested in averages, only in their individual child and whether, given
the characteristics of their child, a particular school can be expected to produce
better subsequent achievements than any other chosen school or schools. It is well
understood that the education system is based on a hierarchical model of students
in schools and then schools within the education system. These two perspectives
result in completely di�erent models and ways of analysing the problem, one at the
individual student level and the other at the school level of the hierarchical model.
A further advancement is the �contextual value-added � system which, in addition to
adjusting for the prior achievements of an individual student, also attempts to adjust
for such factors as the average prior achievement of a student's peers. Goldstein [23]
8 1.2. Background and Literature Review
states that a value-added system takes account of the di�ering intake achievements of
students entering the school. Explicit or implicit selection procedures, for example,
would a�ect the value-added score.
Gains or value-added data incorporates the �gain� or improvement of students. How-
ever, one problem is that an increase of one unit at the higher end of the assessment
scale is not equivalent to a one unit increase among average marks. It is harder to
measure the improvement or �gain� of students who are already getting close to full
marks.
Darmawan and Keeves [16] de�ne the essence of the value-added approach as the
statistical isolation of the contribution of teachers and schools to growth in student
achievement at a given grade level.
Meyer's paper [34] de�nes two types of value-added indicators, the total and intrinsic
school performance indicators which are appropriate for purposes of school choice
and school accountability, respectively. Meyer uses a two-level model of student
achievement. The �rst level of the model captures the in�uences of student and
family characteristics on growth in student achievement.
PostTestis = θPreTestis + αStudCharis + ηs + εis (1.2.1)
where i indexes individual students and s indexes schools; PostTestis and PreTestis
represent student achievement for a given individual in consecutive grades; StudCharis
represents a set of individual and family characteristics assumed to determine growth
in student achievement (a constant term); εis captures the unobserved student level
determinants of achievement growth; θ and α are model parameters that must be
estimated and ηs is a school-level e�ect that also must be estimated. The param-
eter ηs re�ects the contribution of school s to growth in student achievement after
controlling for all student-level factors such as pre-test and student characteristics.
The second level of the model captures the school-level factors that contribute to
student achievement growth.
ηs = δ1Externals + δ2Internals + us
where ηs is the school e�ect for school s from equation (1.2.1), Externals and
Internals represent all observed school-level characteristics assumed to determine
growth in student achievement, us is the unobserved determinant of total school
Chapter 1. Introduction 9
performance and δ1 and δ2 are estimated parameters. The intrinsic school perfor-
mance indicator, denoted φs, is de�ned to be
φs = δ2Internals + us = ηs − δ1Externals.This basic model is extended to more complicated value-added models by Meyer
[34].
1.2.4 Rasch Scaling
One way to compare scores between students over years is Rasch scaling [7, 33, 48].
Rasch scaling is a well-established method used in many human sciences, particularly
psychometrics, and increasingly in the health profession. In the context of education,
the objective of Rasch scaling is that measures of education variables like results or
scores should have a general meaning independent of the species of instrument (for
example, tests and teachers) used to obtain them so that they are comparable [3].
The Rasch model was �rst introduced by George Rasch in the 1960s. To motivate
the mathematical theory of the Rasch model, suppose there is a single true/false
question - what is the probability that it is answered correctly? The probability that
the question is answered correctly has a Bernoulli distribution but is also dependent
on the level of di�culty of the question and the person's ability. The mathematical
formulation for this probability is
ln
[Pr(xsi = 1)
(1− Pr(xsi = 1))
]= θs − βi (1.2.2)
where Pr(Xsi = 1) is the probability of person s answering item i correctly, θs is
the ability of person s and βi is the di�culty of item i. This formula says that
the di�erence between a person's underlying ability θs and the item's di�culty βi
determines the log-odds of a person answering the item correctly. The parameters
θs and βi are estimated from the true/false response data. Equation (1.2.2) can then
be algebraically rearranged to give an expression for the probability
Pr(xsi = 1|θs, βi) =exp(θs − βi)
1 + exp(θs − βi).
10 1.2. Background and Literature Review
From this probability, the Binomial likelihood function is
L(xs1, xs2, . . . , xsi|θs, βi) =I∏i=1
P (θs, βi)xsi(1− P (θs, βi))
(1−xsi)
where P (θs, βi) denotes Pr(xsi = 1|θs, βi). To estimate the ability of students (θs)
and the item di�culty (βi), an iterative process of joint and conditional maximum
likelihood estimation is used. One such method is the Expectation-Maximisation
(EM) algorithm, an iterative method for �nding maximum likelihood estimates of
parameters in statistical models, where the model depends on unobserved latent
variables. One very important point about Rasch scaling is that the sum of the
estimated item di�culty parameters is zero [7] and this deals with the spare degree
of freedom when �tting the Rasch model. The ability of students is the latent
variable and the Rasch model is based on a log-linear simple item response model
where tests of �t and item parameter estimation can take place without assumptions
about the distribution of the latent variable [12].
In practice, the Rasch model is applied using computer software. A selection of these
Rasch measurement programs are Winsteps, RUMM, Facets, Quest and ConQuest
[42].
With these probabilities Pr(xsi = 1|θs, βi), the expected score (Es) for a test of I
items can be calculated from the predictions made by the Rasch model
Es =I∑i=1
Pr(xsi = 1|θs, βi).
Stemming from the example of answering a single true/false question, all of the above
mathematical formulation is for the dichotomous Rasch model. The dichotomous
model can be extended to the non-dichotomous model, also known as the polytomous
or partial credit model. Rather than having only two possible answers for a question
(for example, true/false, multiple choice questions), there is now a range of marks
which can be awarded to a single question. Expected scores are now calculated by
Esi =
mi∑k=0
Pr(xsi = k|θs, βi)
where k is between 0 and the maximum possible score for item i, mi.
The wide application of Rasch scaling in human sciences, health sciences and market
research is because of its usefulness in test design and comparability in the analysis
Chapter 1. Introduction 11
of tests or surveys. In the area of test design and assessing whether the questions are
appropriate for the target audience, Rasch scaling is used to identify test questions
which give insu�cient information to di�erentiate between students, for example
when a question is so easy that all students answer it correctly or at the other
extreme, no one correctly answers a very hard question. These questions do not
help in determining the ability of students or separating students based on their
ability. Rasch scaling also identi�es test questions which are �odd� in that they
are questions which the less-able students answer correctly and the students with
greater ability answer incorrectly. Such a situation could be when a question is
vaguely worded and guessing the answer has a higher probability of being correct.
Under the National Assessment Program, pilot studies with sample tests are given
to a sample of students, and from their Rasch scores, educational authorities can
assess the questions' level of di�culty and appropriateness.
Once the NAPLAN tests have been administered, scaling of the raw scores under
Rasch scaling enables the results to be benchmarked for comparability. Rasch scaling
compares tests through having common questions between years in the same grade
- this enables the tracking of a cohort's progress over time. Common questions are
also included across grades in the same year, and it is expected that Rasch scores
will increase with grade. For example, suppose the common questions are pitched
at a Year 5 level and are included in the Year 3, Year 5 and Year 7 tests. It is
expected that compared to the Year 5 students, Year 3 students will answer less
of these common questions correctly and Year 7 students will get more of these
common questions correct. Hence due to the presence of these common questions,
the natural progression of total Rasch scores is to increase with the ability of the
student, and therefore the grade of the test.
However, one key point about Rasch scaling which must be understood for correct
interpretation, is that Rasch scaling is completely relative and not absolute in any
way. Rasch scores depend on the choice of scale for the model and only have a
meaningful relative interpretation.
One feature of Rasch scaling is that it standardises and normalises student scores,
using standardised residuals and normalisation to sum to zero. This aids in the
comparison of scores across di�erent grades and years. The actual process of stan-
12 1.2. Background and Literature Review
dardising and normalising is somewhat arbitrary, depending on the computer soft-
ware. As mentioned, there are many Rasch measurement programs [42] and these
operate mainly as �black boxes�. It is possible to investigate how each of the many
software standardises and normalises student scores through the Rasch model, but
that is beyond the scope of this literature review.
In summary, the Rasch model is used in situations where the variable of interest is
latent and is only measured indirectly, and the responses are either dichotomous or
fall into ordered categories. For this reason, Rasch scaling is used for school tests
to separate the ability of test takers and assess the quality of the test, and it is
standard practice for student results to be converted into Rasch scores.
1.2.5 The Estimation of School E�ects
The increasing public demand to hold schools accountable for their e�ect on student
outcomes lends urgency to the task of clarifying statistical issues pertaining to the
study of school e�ects. A school e�ect can be interpreted in two di�erent ways. The
term may refer to the e�ect on a student outcome of a particular policy or practice
or may be the extent to which a particular school modi�es a student's outcome - we
are mainly concerned with the latter de�nition.
Based on Willms and Raudenbush [47], Raudenbush and Willms [39] present a
statistical model that de�nes two di�erent types of school e�ect implicit in a school
accountability system: one appropriate for parents choosing schools for their children
(Type A), the other for agencies evaluating school practice (Type B). Firstly, a
statistical model for school e�ects is
Yij = µ+ Pij + Cij + Sij + eij
where Yij is the outcome for student i in school j; µ is the overall mean, Pij is
the e�ect of school practices (for example, school resources, organizational structure
and instructional leadership) on student i in school j; Cij is the contribution of
school context (for example, the mean socio-economic level of the school's students,
the unemployment rate of the community); Sij is the in�uence of measured student
background variables (for example, pre-entry aptitude or socio-economic status) and
eij is a random error term, including unmeasurable sources of a particular student's
Chapter 1. Introduction 13
outcome, assumed to be statistically independent of P,C and S. In this model, the
in�uence of school practice P and context C is allowed to vary across students within
a school. Technically, this means that the model can include both main e�ects of
school-level variables and interactions between school- and student-level variables.
The Type A e�ect is the discrepancy between a child i's potential outcome in school
j, say Yij(Sij, Cij, Pij, eij) and that child's potential outcomes in school j′, that
is, Yij′(Sij′ , Cij′ , Pij′ , eij′) [38]. Alternatively, Type A e�ects are used to ascertain
the expected output achievement of a particular student conditional on their own
characteristics. It is denoted as
Aij = Pij + Cij.
In contrast, the Type B e�ect is the di�erence between child i's potential outcome
in school j when school practice P ∗ij is in operation, yielding Y ∗ij(Sij, Cij, P∗ij, e
∗ij) and
that child's potential outcomes in school j when school practice Pij is in operation,
that is, yielding Yij(Sij, Cij, Pij, eij), denoted
Bij = Pij.
It can also be interpreted as a measure of those institutional characteristics which
explain di�erences between schools.
Darmawan and Keeves [16] then extend the above model to accommodate classroom
or teacher e�ects by splitting school context (C) into classroom (CC) and school
context (SC). Furthermore, school policies and practices (P ) can be divided into
identi�ed (IP ) and unidenti�ed (UP ). Identi�ed policies and practices (IP ) can be
further subdivided into malleable (MP ) and non-malleable (NP ) polices and prac-
tices. In addition to the speci�cation of Type A and Type B e�ects, Darmawan and
Keeves de�ned Type X e�ects [27] to refer to how well the students in a classroom
perform, when compared to similar students in classrooms and schools with simi-
lar contexts as well as similar non-malleable policies and practices. The remaining
e�ects after controlling for malleable policy and practices are labelled Type Z e�ects.
1.2.6 Hierarchical and Longitudinal Modelling
The hierarchy of students, classes, schools and educational authorities naturally
evokes multilevel or hierarchical modelling. In addition, the gathering of data in the
14 1.2. Background and Literature Review
form of assessment scores over time on the same students creates the longitudinal
aspect of the data. Both hierarchical and longitudinal models are examples of linear
mixed e�ects models, and the vast extent of literature and research on these topics in
general are beyond this project. Some of the selected papers are written by Gelman
et al. [18], Harville [24, 25], Laird and Ware [30], Snijders [43] and Willms and
Raudenbush [47], to name a few.
Hierarchical and longitudinal modelling fall under the heading of multiple regression
y = Xβ + ε,
ε ∼ Nn(0, σ2In)
where y = (y1, y2, . . . , yn)′ is the response vector; X is the model matrix with typ-
ical row x′i = (x1i, x2i, . . . , xpi); β = (β1, β2, . . . , βp)′ is the vector of regression
coe�cients; ε = (ε1, ε2, . . . , εn)′ is the vector of errors; Nn represents the n-variable
multivariate-normal distribution; 0 is an n× 1 vector of zeros and In is the order-n
identity matrix. The regression coe�cients are then classi�ed into �xed or random
e�ects. A �xed e�ects model is one in which the coe�cients do not vary by group
and are constant across individuals. The random e�ects are modelled using prob-
ability distributions of a random variable. A mixed e�ects model is a model that
involves a combination of �xed and random e�ects.
Hierarchical models are either linear or generalised linear models in which the param-
eters are given a probability model. Another de�nition is that the hierarchical linear
model is a random coe�cient model with nested random coe�cients. This second-
level model has parameters of its own which are known as the hyper-parameters of
the model and are estimated from the data. Longitudinal modelling is necessary
when an array of variables have been recorded for each subject at several points in
time.
A longitudinal hierarchical linear model for estimating school e�ects, which has
been widely cited, is by Willms and Raudenbush [47]. The �rst level is a separate
regression of outcomes on student-level background variables within each school and
at each point in time:
Yijt = βjt0 + βjt1Xijt1 + . . .+ βjtK−1XijtK−1 +Rijt,
for student i (i = 1, . . . , nj) in school j (j = 1, . . . , J) at occasion t (t = 1, . . . , T )
Chapter 1. Introduction 15
such that Yijt is the outcome score, βjtk are within-school regression coe�cients, Rijt
are student-level residuals and there are k = 1, . . . , K−1 independent variables Xijtk
which describe the background characteristics of students. The βjt0 are estimates of
the performance for each school j at occasion t, after adjusting for the covariates in
the model.
The second level is a between-occasion model for school j to �nd the intake-adjusted
levels of performance βjt0 based on policy (P ) and context (C) variables and on ωjt,
the time of the tth observation for each school
βjt0 = θj0 + θj1ωt + θ2(Pjt − Pj) + θ3(Cjt − Cj) + Ujt.
Finally, the average e�ectiveness of each school and the variation between schools'
trend in achievement can be estimated respectively by
θj0 = Φ00 + Φ01Pj + Φ02Cj + Vj0,
θj1 = Φ10 + Vj1,
with the hyper-parameters Φ00,Φ01,Φ02 and Φ10.
1.3 Underlying Research Question
The purpose of NAPLAN is to report national and jurisdictional achievements in
literacy and numeracy as well as providing accurate and reliable measures of student
and school performance. However, educational experts question the current use and
interpretation of the published results.
�NAPLAN is perhaps one of the most signi�cant data sets on school-
ing in Australia. However, in its current form, the questions it could
potentially help answer cannot be addressed. The di�culty of comparing
schools statistically is well recognised by those charged with analysing and
reporting NAPLAN results but these professionals are restricted in what
they can (and should) report.
Further, the potential power of the NAPLAN data could be geometrically
advanced if it were included in a broader national research agenda, open
to a larger body of researchers who know what can be done with it.
16 1.4. Outline
But we cannot even begin the task of developing alternative within-school
practices intelligently until we harness our data and research capacity in
a more educationally productive manner.� [29]
This summarises the general thrust of my research objective - to apply mathemat-
ical and statistical modelling and analysis techniques to NAPLAN data, or data of
a similar nature, to investigate and determine what conclusions about school per-
formance can be drawn from such data. In other words, how can we accurately
measure a school's e�ect on student improvement?
1.4 Outline
In Chapter 2, we investigate a data set of the Basic Skills Tests and the univariate
and bivariate descriptive statistics. From a clean data set, we then apply model
selection techniques to identify the signi�cant variables and discuss the results of
simple linear regression models in Chapter 3.
Simple linear regression is extended in Chapters 4 and 5 to a hierarchical model
which is �tted using linear multilevel mixed e�ects models (Chapter 4) and a
Bayesian approach through the BUGS and Stan software (Chapter 5). Chapter
6 then discusses the validation of the model and how we have achieved a well-�tting
model which explains the relationship between student and school covariates, and
can be used for accurate prediction.
Having �t a hierarchical model, Chapter 7 looks at an initial longitudinal model
analysis before some conclusions, the practical implications of our �ndings to the
NAPLAN data in Australia and some ideas for further work that could be performed
in this area, are given in Chapter 8.
Chapter 2
Data Analysis
A subset of the South Australian Basic Skills Test data was procured which contains
54 recorded variables on the test results and background information of 49 341
student identities from 426 schools in the years 1997 to 2005. Other than the
data itself, no further information about the classi�cation of variables or the data
collection method was supplied, and all attempts to contact the data owners were
unsuccessful. To maintain the con�dentiality of the schools and their results, all
school identi�ers were anonymised and hence, we could not contact schools directly
to investigate further. Since it was not possible to go back to the data source, or
the schools themselves, when issues and questions arose regarding the data, it is in
this context that we cannot report any factual explanations.
2.1 The Basic Skills Test Data
The Basic Skills Tests are administered in Grades 3, 5 and 7 and have a Literacy
and a Numeracy component. Each component has sub-tests or aspects - Literacy
is divided into Reading (LR), Spelling (LS) and Writing (LW) while Unit (NU),
Spatial (NS) and Measurement (NM) tests make up the Numeracy component. The
total scores for Literacy and Numeracy are represented by LL and NN respectively.
Importantly, Literacy - Writing (LW) was only introduced in 2001. As a result of
the exploratory data analysis, various issues concerning missing and incomplete data
were observed.
17
18 2.2. The Variables: Background Information
A student is de�ned as the general term for a person who is formally engaged in
learning, in particular, one enrolled in a school. However for each actual test and
its associated year and grade, at least one result is recorded for only a subset of
the students. These students are de�ned to be the participants of a test or in other
words, a participant is a person for which data is recorded for a particular time in
the Basic Skills Test data set which we are considering. A test refers to a physical
Basic Skills test while the recorded numerical scores of the tests are referred to as
scores.
2.2 The Variables: Background Information
In total, there are 54 variables which can be divided into three categories - test vari-
ables, school variables and student variables (Tables 2.2.1 to 2.2.3). An explanation
of the coding of the variables is given in Appendix A.
An individual participant is identi�ed by their combined studentide and schoolno
values. Student ID numbers are speci�ed by the school and do not uniquely de�ne a
student. Hence, it is not possible to track participants by their ID numbers should
they move between schools.
Table 2.2.1: Test variablesVariable Name Descriptionprocyear calendar yearaspect literacy or numeracy aspectnocorrect raw test markstandardsc standardised score under Rasch scalinggradedyear grade of the test
Chapter 2. Data Analysis 19
Table 2.2.2: School variablesVariable Name Descriptionschoolno code number of the schoolgpokm distance from Adelaide General
Post O�ceisolation isolation indexspatial_ar MCEETYA* classi�cation for
Rurality and Remotenessstaff_metr classi�cation by DECS** sta�cap Country Areas Program***x006_enr, x005_enr, x004_enr enrolment numbers in 2006, 2005
& 2004x006_abs, x005_abs, x004_abs absentee rate in 2006, 2005 &
2004x006_beh, x005_beh, x004_beh number of behavioral incidents in
2006, 2005 & 2004x006_scrd, x005_scrd, x004_scrd number of School Cards**** in
2006, 2005 & 2004x006_mob, x005_mob, x004_mob mobility of students in 2006, 2005
& 2004x006_tch, x005_tch, x004_tch number of teachers in 2006, 2005
& 2004x006_tmob, x005_tmob, x004_tmob teacher mobility in 2006, 2005 &
2004
* Ministerial Council on Education, Employment, Training and Youth A�airs
** Department of Education and Children's Services
*** The Country Areas Program is an Australian government program which pro-
vides �nancial help to rural schools.
(https://deewr.gov.au/country-areas-program)
**** The School Card Scheme is a government initiative which provides �nancial
assistance towards educational expenses for eligible families.
(http://www.decd.sa.gov.au/goldbook/pages/school_card/schoolcard/?reFlag=1)
20 2.2. The Variables: Background Information
Table 2.2.3: Student variablesVariable Name Descriptionstudentide student ID numberatsi Aboriginal or Torres Strait Islanderlbote language background other than Englishstatus current status of student within schoolgender male or femaledate_of_bi date of birthaboriginal Aboriginal statusdisability disability statusschool_car individual School Cardoccupation parental occupation groupschool_edu parental school educationnon_school parental non-school educationp_g_gender gender of principal guardian or parentp_g_cultur cultural background of principal guardian or parentp_g_countr parental country of originp_g_nesb parental non-English speaking backgroundcountry_of country of originnesb_code non-English speaking backgroundhome_langu English as home languagecultural_b cultural backgroundvisa_sub_c Australian permanent visa number
Chapter 2. Data Analysis 21
2.3 Descriptive Statistics
2.3.1 Number of Participants in Schools
The number of recorded participants in each school ranges from a minimum of
one participant to a maximum of 671 participants. These numbers are depicted
graphically in Figure 2.3.1. The observed trend is that the number of participants
in a school tends to increase as the school ID number also increases.
●
●●●●●
●●●●●●●●●●●●
●●●●●●●●●
●●●●●●●●
●
●●●
●
●●●●●●●●●●●●●●●
●
●
●●●●●
●
●●●●●
●
●
●
●●●
●
●●●●
●
●●●●
●
●●
●●●●●●●●
●
●●●●
●
●
●●●●
●
●
●
●●
●●
●●●
●
●●●●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●●●●
●●
●●
●●
●●
●
●
●
●
●●
●
●●●●●
●●
●●
●
●
●
●●
●●
●
●
●
●●
●
●
●●
●
●
●
●●
●●
●
●●●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●●
●
●●
●
●
●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●●●
●
●
●
●
●
●●
●
●
●
●●
●●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●●●●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
0
200
400
600
0 200 400 600
School Number
Num
ber
of P
arti
cipa
nts
Figure 2.3.1: Number of participants in each school.
2.3.2 Univariate Statistics
To avoid confusion, LL, LR, LS, LW, NN, NM, NU and NS are called the aspects
of the test and a participant is de�ned to have a result for at least one Literacy or
Numeracy aspect for a given student, year and grade.
22 2.3. Descriptive Statistics
From Table 2.3.1, we observe that the majority of the participants in the data set
are from the calendar years 2000 to 2004 with the two standout years being 2002
and 2003. Each of these two years accounts for almost a third of the total number
of students in the entire data set. There is also over 60% of participants in Grade 3
while Grade 7 constitutes only 4.5% of the data (Table 2.3.2).
Table 2.3.1: Number and percentage of participants in each calendar year(procyear)
procyear Number Percentage1997 7 0.011998 8 0.011999 73 0.112000 9 282 13.392001 9 180 13.242002 19 654 28.342003 21 638 31.202004 9 489 13.682005 13 0.02
Table 2.3.2: Number and percentage of participants in each grade (gradedyear)
gradedyear Number Percentage3 42 375 61.115 23 826 34.367 3 143 4.53
A further break down of the participants into grades in each year is given in Table
2.3.3. We note again the concentration of the data in certain grades and years.
Drop-o� in participants in 2004
An observation is that there is a dramatic drop-o� in the number of participants in
2004, and one possible reason could be the selection of student participation in a
school. Another suggested reason for the drop in numbers could be because there
are less schools involved in the Basic Skills tests, and hence, less students tested.
However, the number of schools in 2004 is very similar to the number of schools in
Chapter 2. Data Analysis 23
Table 2.3.3: Number of participants for each calendar year and grade
procyear gradedyear Number1997 3 61997 5 11998 3 71998 5 11999 3 681999 5 52000 3 9 2722000 5 102001 3 9 1252001 5 522001 7 32002 3 9 7402002 5 9 9062002 7 82003 3 10 9662003 5 10 6442003 7 282004 3 3 1842004 5 3 2042004 7 3 1012005 3 72005 5 32005 7 3
the years 2000 to 2003 (Table 2.3.4). Are all schools giving less Basic Skills tests
in 2004 to cause the number of participants to drop evenly at the same rate or did
some schools dramatically drop while other schools stayed constant? The ratio of
participants in 2004 compared to 2003 across all grades for schools is given in Figure
2.3.2. The unimodal, rather than bimodal, feature of the plot seems to indicate that
the decrease in participants is similar over all schools. The few schools which have
a high ratio of participants all have very few participants in 2003 and is presumably
a result of quite di�erent numbers of students in the respective grades in di�erent
years.
Since the school factor does not seem to be the cause for the decrease in participant
numbers in 2004, we continue to investigate at the school level. Looking at the years
24 2.3. Descriptive Statistics
Table 2.3.4: The number of participants and schools each year
Year No. of Participants No. of Schools1997 7 71998 8 81999 73 622000 9 282 4052001 9 180 4122002 19 654 4152003 21 638 4162004 9 489 4192005 13 13
0
20
40
60
80
100
120
0 1 2 3 4 5 6
Ratio
Cou
nt
Figure 2.3.2: Ratio of participants in 2004 compared to 2003 for all schools in allgrades.
which each individual school participated in the Basic Skills Tests, we notice that
there are many schools who participate in the block of years from 2000 to 2004.
However, thirteen out of the 426 schools do not participate in consecutive years but
skip at least one year.
Chapter 2. Data Analysis 25
Test scores
The main variables of interest - the measured variables - are the participants' scores
for the test aspects. Table 2.3.5 gives the mean, standard deviation, median, in-
terquartile range and observed number for each of the Literacy and Numeracy as-
pects using both the raw and Rasch scores. Note that the mean raw scores vary
signi�cantly since the total available marks vary depending on the aspect and the
Basic Skills test itself. The mean Rasch scores are all similar and comparable since
they have been standardised and normalised under the Rasch model - see Section
1.2.4 for a discussion of Rasch scaling. When comparing the mean to the median
for each of the aspects, all are similar except for LW.
The histogram of LW (Figure 2.3.3) exhibits distinct right skewness and is possibly
bimodal, compared to other Literacy aspects. Looking at the Rasch scores, LW has
a smaller interquartile range of 9.32 compared to the other aspects which range from
10.76 to 13.21. This feature can be observed in Figure 2.3.4.
Table 2.3.5: Descriptive statistics for the raw scores and the Rasch scores for Literacyand Numeracy aspects
Mean Std Deviation Median IQR Number ObservedLL 46.76 16.08 47.00 23.00 59 869LR 23.90 8.75 25.00 12.00 59 949LS 15.63 6.74 15.00 9.00 59 764LW 21.74 14.20 16.00 23.00 48 886NM 8.38 4.02 8.00 5.00 59 947NN 27.36 9.69 27.00 13.00 60 078NS 7.57 2.85 7.00 3.00 59 912NU 11.44 4.42 11.00 6.00 59 944LL Rasch 51.78 7.76 51.98 10.76 59 869LR Rasch 51.89 8.45 52.21 11.45 59 949LS Rasch 52.19 9.45 51.80 12.02 59 764LW Rasch 52.80 8.75 52.95 9.32 48 886NM Rasch 53.67 10.69 53.30 12.73 59 947NN Rasch 53.23 9.31 53.16 12.24 60 078NS Rasch 52.99 10.63 52.89 12.80 59 912NU Rasch 53.54 10.47 53.44 13.21 59 944
26 2.3. Descriptive Statistics
0
1000
2000
3000
4000
5000
0 20 40 60 80 100
LL
Cou
nt
0
1000
2000
3000
4000
5000
0 10 20 30 40 50
LR
Cou
nt
0
1000
2000
3000
4000
5000
6000
0 10 20 30
LS
Cou
nt
0
1000
2000
3000
4000
5000
6000
0 20 40 60
LW
Cou
nt
Figure 2.3.3: Histograms of Literacy raw scores over all years and grades.
Chapter 2. Data Analysis 27
0
2000
4000
6000
8000
20 40 60 80
LL Rasch
Cou
nt
0
2000
4000
6000
20 40 60 80
LR Rasch
Cou
nt
0
2000
4000
6000
8000
20 40 60 80 100
LS Rasch
Cou
nt
0
2000
4000
6000
8000
0 20 40 60 80 100
LW Rasch
Cou
nt
Figure 2.3.4: Histograms of Literacy Rasch scores over all years and grades.
28 2.3. Descriptive Statistics
Cohort Analysis
Students who remain in the school system usually progress from Grade 3 to Grade
5 in two years time and similarly to Grade 7. This progression enables cohorts of
participants to be followed and their scores compared. From the data set, a total of
11 cohorts are de�ned by the years and grades in which participants sat Basic Skills
tests (Table 2.3.6), ordered according to student birth year.
Table 2.3.6: Coding for cohort numbers
Cohort No. Grade 3 Grade 5 Grade 71 - 1997 -2 - 1998 -3 1997 1999 20014 1998 2000 20025 1999 2001 20036 2000 2002 20047 2001 2003 20058 2002 2004 -9 2003 2005 -10 2004 - -11 2005 - -
Table 2.3.7 gives further information on the number of participants in each cohort.
To recap, a participant is a person for which at least one Literacy or Numeracy score
has been recorded for a particular test in the Basic Skills data set. In order to draw
useful conclusions from the statistical analysis of the cohorts' results, we require a
reasonable cohort size at each grade level. We assume that a sample size greater
than one hundred participants is adequate. Hence, cohort 6 is a �three-grade� cohort,
cohorts 7 and 8 are �two-grade� cohorts and cohorts 9 and 10 only have adequate
numbers at the Grade 3 level (these are in bold in Table 2.3.7). All other cohorts
are considered to have insu�cient data.
It is important to note that cohorts are de�ned to contain participants who took
tests in Grade 3, 5 and 7 in the appropriate years. Cohorts are not restrained to
only include participants which took tests in all three grades. Some participants
may only have been recorded for two of the three tests but are still included in
the cohort. If this restriction is imposed, the cohort size would drastically drop,
Chapter 2. Data Analysis 29
Table 2.3.7: Number of participants in each cohort (bold entries indicate cohortswith reasonable numbers of participants)
Cohort procyear gradedyear Number1 1997 5 12 1998 5 13 1997 3 63 1999 5 53 2001 7 34 1998 3 74 2000 5 104 2002 7 85 1999 3 685 2001 5 525 2003 7 286 2000 3 9 2726 2002 5 9 9066 2004 7 3 1017 2001 3 9 1257 2003 5 10 6447 2005 7 38 2002 3 9 7408 2004 5 3 2049 2003 3 10 9669 2005 5 310 2004 3 3 18411 2005 3 7
especially as we cannot track participants by their ID when they change schools.
Statistical conclusions about a cohort's performance would be stronger if it only
contains participants with all three tests, but with the lack of data, we decide to
resort to the looser de�nition of a cohort.
The performance of participants in each grade, cohort and aspect is one of the points
which we wish to consider and compare. The distributions of the Literacy and Nu-
meracy scores are depicted in Figures 2.3.5 to 2.3.10. It can be seen that the shape,
spread and location of the aspects' raw scores vary greatly (as previously identi�ed
from the summary statistics), but the Rasch scores have all been standardised and
normalised. These plots clearly show the e�ect of Rasch scaling on the raw scores.
30 2.3. Descriptive Statistics
●●●●●●●●●●● ●●●●●●●●●●●●●●●●● ●●●●●●●
8302 8207 8677 8912 2657
●
●
●●●●●●●●●●●
●
●●●
●
●
●
●●●
●
●●●
●
●●●●●●
●
●
●●●
●
●
●
●●●●●●
●●
●
●●
●●
●
●●●
●●●
●
●
●
●
●
●●
●●
●●
●●●
●
●●
●
●
●●
●
●
●
●●●
●●●
●
●
●●●●●
●●
●●●●●
●●
●●●●●
●
●
●
●●
●
●●●
●
●●
●
●
●●
●
●●
●●
●●●●●
●●●●
●●
●●●
●●●●●●●●●●●●●●●
●
●
●
●●●●●●●●●●●●●●●●●●●●●●
●
●●●
●
●●
●
●●●●●●●●●●●●●
●
●●●
●
●
●
●
●
●
●
●
●
●●●●●●
●
●
●●
●
●●●
●●
●●●
●
●
●
●●
●
●●●
●
●●●●
●●
●
●●
●●●●
●●●●●●●●●●●●●●
●
●●●●●●
●
● ●
●●●
●●●
●●●●●
●
●●●●●
●●
●●●●
●●
●
●●
●
●●
●
●●
●●●●
●
●
●
●●●●●
●
●●●●●●●
●
●
●
●●
●
●
●
●
●
●
8302 8207 8677 8912 2657
8300 8071 8674 9056 2621
●●●
●
●●●
●
●
●●
●●●●
●
●
●●●●●●
●●
●
●
●●●●●●
●●
●●
●
●●●
●
●
●●●●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●●●●
●●●●
●
●
●
●
●●
●●
●
●
●●●
●
●●
●
●●
●
●●●
●
●●●
●
●●●●●
●●●●
●●
●●
●
●●
●●●
●
●●
●●
●●
●●●●
●
●●
●●
●
●●●
●●
●●
●●
●
●
●
●●
●
●
●
●●
●●●●
●●
●
●
●●●●
●
●
●
●
●●●●●
●
●
●
●
●
●
●
●
●
●
●●●
●●
●●●
●●●●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●●
●●
●
●●
●
●
●●
●●●●
●
●●●●●●●●●●
●●
●●●
●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●●●
●●●●●●
●●
●●
●
●●●●
●●●●
●
●
●
●●●
●
●●
●
●
●●
●●●
●
●
●●
●●●
●●
●●
●●●●●●
●
●●●●●●●●●●●●●
●
●●●●●●●●●●●●
●
●
●
●
●
●
●
●
●
●●
●
●
●●●●●●●
●
●●●
●
●●●●●
●
●●●●●●
●
●●●
●
●●●●●
●
●
●●
●
●
●●●
●
●
●●●●●●
●
●
●●
●
●●
●●●●●●
●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●
●
●●
●●●●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
8300 8071 8674 9056 2621
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
8330 8070 8687 8937 2639
●
●●●●●
●
●●●
●●
●
●
●●●●
●
●
●
●●●●●●
●●●
●●●
●
●
●
●●●
●●
●
●
●●●●
●
●●
●
●●●●
●
●
●
●●●●
●
●●
●
●●
●
●●●
●●
●
●
●●●●
●●●
●
●●●
●●●
●
●
●●●
●
●●
●
●●●●●
●●●●●
●
●●●●●
●
●●
●
●●●●●
●●●●●
●●●●
●●
●
●
●
●
●
●
●●●
●●
●●●●
●
●
●
●
●●
●●
●
●●
●
●●
●●
●●●
●●
●
●
●●●●
●
●●
●
●
●●●●●●
●●●●●●
●●●●
●●●●
●●●●●●●●●●
●
●●●
●●
●●●●●
●●
●●●●●●●
●
●●
●
●●●●●
●
●●
●●
●●
●
●●
●
●●●●●●●●
●
●●●●●
●
●●
●
●●●●
●●
●●●●●
●
●
●●
●
●●●●●●
●
●●●●●●●
●
●
●
●●●
●
●●●●●●●●●●●●●●●●●●●●
●
●●
●
●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●
●
●●
●
●●
●
●●●●●●●●
●
●●●●●●●
●●●
●
●
●●
●
●●
●
●
●●
●●
●●
●●
●●
●
●
●●●●●
●
●●●
●
●●
●●●●●●●
●●●●●
●●●●●
●●
●●●
●
●
●●
●●
●
●●
●●●●●●●●
●
●
●●
●●
●●●●
●
●
●
●
●●
●
●
●●●●
●●●
●●
●●●●●
●
●
●
●
●
●●●●●●●●●
●
●●●●●●●●●●●●●●●
●
●●●●
●
●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●
●
●
●
●
●
●●
●
●●●●●●●
●●
●●
●●●
●
●●●●
●
●●
●●
●
●●●
●●●●
●●
●●●●●
●
●●●●●
●●●●●●●●
●●●●
●
●
●●
●
●●
●
●
●●●
●●
●●
●
●
●●●●
●●●●
●
●●●
●●●●●
●
●●
●●●●●●
●●●●
●●●●
●
●
●●
●●●
●
●●●●●●
●●
●●
●●●●
●
●●
●●
●●●●
●
●●
●●
●
●●
●●●●●●●●●●●●
●●●
●
●
●
●
●
●●
●●
●●●●●
●
●
●
●
●
●●
●
●●
●●●
●●●●
●●
●●●●
●
●●
●●●
●●●●●●●●●
●
●●
●
●●●
●
●●●●●●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●●●●●●●●
●
●
●
●●●●●●●●●●●
●
●●●
●
●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
8330 8070 8687 8937 2639
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●
●●●
●
●●●●●●●●●●●●●●●●●●
●
●●
●●
●
●
●●●●●●
●
●
●
●●●●●●
●
●●●●●●●●●●●●
●
●●●●●●
●
●●
●
●
●
●●●●
●●
●●
●●
●●
●●●●●●
●●●
7254 7693 9072 2642
●
●
●
●
●●●
●●
●
●●●●●●●●
●
●
●
●
●
●
●
●
●
●●
●●●
●●
●●●
●
●
●●●
●●
●●
●
●●●●●
●●●●●
●●
●●
●●●
●●●
●●
●
●●
●●●●●●●
●●●
●●●●
●
●●●●
●
●
●
●
●
●
●
●
●●●●●
●●●●
●●
●●
●
●
●
●
●
●
●●●●●●
●
●●
●●
●
●
●●●●
●●
●●●●●
●
●
●
●
●●●●●●●●
●
●
●
●
●
●
●
●
●●●●●●●●●●●
●●
●●
●
●●
●●●
●
●
●
●●●
●
●●
●
●
●●●●
●●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●●●●●●●●●
●●
●
●
●
●
●
●
●●●
●
●
●
●●
●●
●●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●
●
●●●●
●
●
●●●●●●●
●●
●
●
●●
●●●
●
●
●
●●●●
●
●●
●
●
●●●
●
●●●●
●
●●●
●
●
●●●●●●●
●
●
●●●
●
●●
●
●
●
●●
●
●
●●
●●
●
●
●●●
●
●●
●
●
●
●●●●
●
●●
●
●
●●●
●
●
●●
●
●
●
●
●●
●●
●●
●
●
●●●●
●
●
●
●●
●
●●●
●
●
●●
●
●●
●●
●●●●●●●
●
●●●●●●●
●
●●●
●
●
●
●●
●●
●●●
●
●
●
●●
●
●●●●
●
●
●
●
●
●●●●
●
●●●●●●●●●●●●●
●●●●
●
●●●
●
●
●●
●
●
●●●●●
●●
●
●●●●●●●●
●
●●●●
●
●●
●●●●●
●●
●●●●
●
●●
●
●●●
●●
●
●
●
●●
●●
●
●●
●●●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●●
●
●
●
●●
●
●●
●●●●●●●●●●●●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●●●●●●
●
●
●●
●
●
●
●●●●●●
●
●●
●●
●●
●
●●●●●●
●
●
●
●
●●
●●●●
●
●●●
●●●●●
●
●
●
●●●●●●●●●●●●●●●●●●●●●●●
●
●●
●
●●●
●●
●
●
●●●●
●
●
●
●
●
●
●
●●●
●●●
●
●
●●●●●●●●●
●
●●
●●●●●●
●●●●●●
●●●
●
●
●
●●●
●
●●●
●
●
●●
●
●
●●
●
●●●●●
●
●●●●●●
●
●
●
●●
●
●●●●
●
●
●
●●
●
●●●
●
●
●
●●●
●
●●●●
●●
●
●
●
●
●
●●●●●●
●
●
●●●
●●●
●
●
●●●●
●●●
●●
●●●●●
●
●
●●
●
●
●
●
●●●●●●●
●●●●●
●●●●●●●
●●
●
●●
●
●●
●
●
●●
●●
●●●●●●●
●
●
●
●
●
●●
●
●
7254 7693 9072 2642
LL LL Rasch
LR LR Rasch
LS LS Rasch
LW LW Rasch
0
20
40
60
80
100
0
20
40
60
80
100
0
20
40
60
80
100
0
20
40
60
80
100
6 7 8 9 10 6 7 8 9 10Cohort
Scor
e
Figure 2.3.5: Boxplots of Literacy scores for the cohorts in Grade 3 (the sample sizeis given above each boxplot).
Chapter 2. Data Analysis 31
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●
8365 8113 8691 9109 2618
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●
●●●
●
●●●●
●●●●●●●●●
●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●
●●
●
●●●●●
●
●●●●
●
●
●●
●●●●
●
●
●●
●●●●●●
●
●●
●●
●●
●
●●●
●
●●●●●●●
●
●●●●●●●●●●
8365 8113 8691 9109 2618
●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●● ●●
8336 8209 8683 9086 2619
●●
●
●
●
●
●
●
●●●●
●
●
●●
●
●●
●
●●
●
●
●●
●
●
●●
●
●
●
●
●
●●●
●
●
●●●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●●●
●●●●
●●
●
●●
●
●
●●
●●
●
●●●●●●●●
●
●●●
●●
●●●●●
●
●●●
●
●●●●●●●
●●
●●●●●
●●
●●
●
●●
●●
●●
●
●●●
●●●●●
●
●●
●
●●
●
●●●
●●
●
●
●●
●
●●●●●
●●
●●●
●
●●●●●
●
●●●
●●●●
●
●●●
●
●●●●●
●●●
●
●
●●●
●●●
●●●●●●
●
●●●
●
●●●●●●●●●●
●
●
●
●●●●●●●●●●●●●●●●
●
●●
●
●●●●●●●●
●●
●●
●
●●●●
●●●
●●
●
●●●●●●
●
●
●●●
●
●●●●●●
●
●●●
●
●●●●●●
●
●●
●
●
●
●●●●●
●
●●
●●●●●●
●
●●
●●●
●●
●
●●●●●●●●●●●●
●●
●
●●
●●
● ●●●●
●
●
●●●
●
●●●●●●
●
●●●
●
●
●●
●
●
●●●●●●●
●
●●●
●●
●●●
●
●●●●●●●●●●●
●
●
●
●●●●●●●
●
●●●●●●
●
●
●
●
●●
●
●
●
●●●●
●
●
●
●
●●
●●
●
●
●
●●●●●●●●●●●●●
●
●
●
●●●●●●
●
●●●●●●●
●
●●●
●●
●●●●●●●
●
●●●
●
●
●●●●
●
●
●●
●●●
●●●
●
●
●
●●●
●
●●●●
●●●●●
●
●●
●●●●●
●
●●
●●
●●
●
●●●
●
●●
●
●
●●
●
●
●●
●●●
●●●●
●
●
●●●
●●
●●●●
●●●●●●
●●●
●
●●●●●
●
●●●
●●●
●●●
●●●
●
●●●●●●●
●
●●●●●
●
●●●●●●●
●
●●●
●
●
●●
●●●●
●●●●
●
●
●●●●●●●
●●
●●●●●
8336 8209 8683 9086 2619
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
8386 8101 8745 9078 2611
●●●●●●
●●●●
●●
●
●
●●●●●●
●●
●
●●●●
●●●●●
●
●
●
●●
●
●●●●
●
●●●●●
●
●●●
●●
●
●●
●●●
●
●
●
●●●●●
●
●●
●●
●●●
●
●●●●●
●
●●●●●
●
●
●
●●●
●
●●●●●●
●
●●●
●
●●●
●●
●
●
●
●
●●●●●●
●
●●
●
●●●●●
●
●●●
●
●●●●
●
●●●●●●
●
●
●
●●●●●●●●●●●●●●●
●
●●
●●
●●●●●
●●
●●●●●
●●
●
●
●●●●
●
●●●●●
●●●
●●
●
●●●
●
●●●●●●●●
●
●●●●
●
●●
●●
●●●●●●
●
●●
●
●
●
●●●●
●
●
●
●
●
●●●●●
●●
●●●●●●●●
●●
●●●●
●
●
●
●
●●
●
●●
●
●●●●●
●●
●●●●
●
●
●●●
●●●●
●●●
●
●●
●
●●
●●●●
●●●●
●●●●
●●
●●●
●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●●●
●●
●●
●
●●●
●
●●
●●●
●●●●●●
●
●
●
●●
●●●
●
●
●●
●
●
●
●
●
●●●
●
●●●
●
●
●
●
●
●
●●●●
●●
●
●
●●●●●●●●●●●
●
●●●●●●●●
●
●
●●●
●●
●●
●●●●●●
●
●●●●
●●●●
●
●
●
●●●●●●
●
●●●●●
●
●●●●●
●●●●
●●●●●
●●
●
●
●
●
●●
●●
●
●
●
●●
●●
●●
●●●●●●●●
●
●●
●
●●●●
●
●●
●●
●●●
●●●●●
●
●
●●
●
●●●●●●
●●
●●●
●
●●●●●●
●●
●
●
●
●
●
●●
●●
●●
●
●●●●
●●●
●●
●
●●●
●●
●●●●●●
●
●●●●●
●
●●●●●●●●●●●
●
●
●●●
●●●●●
●●●
●●●●●●●●●
●
●
●●●●●●
●●
●
●
●●●●●●●●●●●●●●●●●●●●
●●●●
●
●●●●●●●●
●
●●●●●
●
●
●●
●
●
●
●
●●●
●
●●●
●●●●●●●
●
●
●
●
●
●●●●
●
●
●
●●
●
●●●●
●●●●
●
●
●●●
●
●●●●
●●●●
●●●●
●●●●●●
●
●
●●●
●
●●●
●
●
●
●●●
●
●●●●●●●●
●
●
●
●●●●●
●
●●
●
●
●●●●●●
●●●●●●
●
●●
●●●●●
●
●●
●●
●
●
●
●●
●
●●●
●●●
●
●
●
●●●●
●
●
●●
●
●●●●
●
●●●●●●
●●●
●
●
●●●●●●
●●
●
●●●
●●●●●●
●
●
●
●
●●
●●●●
●●●●●●●●●●●●●●●●●
●●
●
●●
●
●●●●●●●●●●●
●●●●●
●
●●●●●●●
●
●●
●
●●●●●●●●●●●●●●●●●
●
●●●●●●
●●
●●
●
●●
●
●
●●●
●●
●
●●●●●●●●●
●
●●
●●●●
●
●●●●●●
●●●●●●●●●●
●●●●
●●
●
●
●
●●●
●
●●
●●●●●
●
●●
●
●●
●●●
●●●●●
●
●●
●
●●●
●
●●●●●●●
●
●●●●●●●
●
●●●
●
●●●●
●
●●●●
●
●●●
●
●●●
●●●●
●●
●●
●●●
●●
●●●●●●
●
●●●
●●
●●●●●●●●●●●●●●
●
●●
●
●
●
●●●●●●●●●●●●
●●●
●●●●●●●
●
●●
●●●
●
●●●
●
●
●●
●●●
●
●
●
●
●
●
●●●●
●●
●
●●
●
●●
●
●
●●●
●●
●
●
●
●●●●
●●●
●●
●
●●
●●
●
●●
●
●●
●
●
●●●●●●●●●●
●
●●●●●
●
●●●●●●●●●
●●
●
●●
●●
●●●●●
●●●●●●●●●●●●
●
●●
●●
●●
●●
●●●●●●●●
●●
●●●●●●●●● ●●●
●
●●●
●●
●●
●●●●●
●●●●
●
●●
●●
●●
●●●
●
●
●●●●●
●●
●●
●
●●●●●●●●●
●●●●
●●●
●
●●●●●●●●
●
●
8386 8101 8745 9078 2611
●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●● ●●●●●●●●●●
8350 8069 8726 9107 2612
●●
●
●●●
●
●●
●
●●●●●●●
●●
●●
●●
●●
●●
●●
●
●
●
●●
●
●●●●●
●
●
●
●●
●●
●●●●●
●
●●
●
●●●●
●
●●
●●
●●●●●
●
●●●●●●●●●●●●●●●●
●
●●●●
●
●
●
●●●
●
●●●
●
●●●●
●
●●●●●●●●●●●●●●●●●●●●●●
●●
●●●●●●
●
●●●●●●●●●●
●
●●●●●●●●●●
●●
●
●
●
●
●
●●●●
●
●●●
●
●●
●
●
●●
●
●
●
●●●●
●
●●
●
●●
●
●
●●●●●●●●
●
●
●●
●
●●●
●
●
●●●
●
●●●●
●
●●●●
●●●●●●
●
●●
●
●●
●
●●
●
●●
●
●●●
●●
●●●
●
●
●
●●
●
●●●●●●●●●●●●●●
●
●●
●
●●●●●●●●●●
●
●●●●
●
●●●●●
●●●●●●
●
●
●●●●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●●●●●●●●
●
●●●●
●
●
●●
●
●●●●●●
●
●
●
●●●●●●●●●●●
●
●●
●
●
●●
●
●●●●●●
●
●●●
●●
●
●
●●●●●●
●
●●
●
●
●●●
●
●●
●●
●●
●●
●●
●
●
●
●
●
●●
●
●●
●●●
●
●
●
●●
●
●●●
●●
●
●
●●●●●
●●
●●
●
●●●●●●●●
●●●●
●●
●
●
●●●
●●
●
●
●
●●
●
●●●
●●●●
●●
●●
●●●
●●
●
●●●●●●●
●●
●
●●●●
●
●
●●●●
●
●●●●●●●●●●●●
●●
●
●
●
●
●●
●●●●●
●
●
●●●
●
●●●●●●●
●
●●
●
●
●
●
●●●
●●●
●
●●●●●
●●●
●
●●●
●●●●●
●●
●
●
●●●●●●●●
●●●●●
●●●
●
●
●
●●●●●
●●●
●●
●
●●
●
●
●●●●●
●●●●●
●●●●
●
●●●●●
●
●●●
●
●
●
●●
●●●
●●●
●●
●●
●
●
●
●●●●●
●●
●
●
●
●
●
●
●
●
●●●
●●●●●
●
●
●●
●
●
●●
●
●●
●
●
●●
●●●●●●
●
●
●
●●●●●●
●
●●●●●●●●
●
●●●●
●
●●
●
●●
●●
●●
●●
●
●●
●
●
●
●
●●
●●●●●
●
●
●
●
●●
●●●●
●
●●●●●●●●●● ●●●●●
●
●●●●●
●●
●●●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●
●
●
●
●●●●●
●
●●
●
●●●●●
●
●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●
8350 8069 8726 9107 2612
NM NM Rasch
NN NN Rasch
NS NS Rasch
NU NU Rasch
0
20
40
60
80
100
0
20
40
60
80
100
0
20
40
60
80
100
0
20
40
60
80
100
6 7 8 9 10 6 7 8 9 10Cohort
Scor
e
Figure 2.3.6: Boxplots of Numeracy scores for the cohorts in Grade 3 (the samplesize is given above each boxplot).
32 2.3. Descriptive Statistics
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●
8863 8889 2662
●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●
●
●●●●●
●
●●
●
●●
●
●
●●●
●
●●●●
●●
●●●
●●●
●
●●
●
●●●●●●●●
●
●●●●
●●
●
●●
●
●●●
●●●●●●●
●●
●
●
●
●●●●
●
●●●●●●●
●●●
●●●●●●●●
●
●●●●
●
●
●
●
●
●●
●
●
●●●
●●●●●●
●
●●●
●
●●●
●
●
●●●●●●
●●
●●●●●
●●●
●●●●
●
●●
●●●●
●●●●
●●
●●
●●●
●
●
●●●
●
●
●
●●●
●●●●●●●●●●●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
8863 8889 2662
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●
8907 8912 2669
●●●
●
●●
●
●●●●●
●
●
●●●
●●●●
●
●●
●●●
●●
●●
●●
●●●●●
●●
●●●●●●
●
●●
●●●●●●●
●●●●●
●●
●●
●
●●●●●
●
●
●●
●●●●●●●
●●
●●●●●
●
●●●●●●●●●●●●●●
●
●
●●●●
●●●
●●●●●●●●●●●●●●●
●
●●
●●●
●●
●
●●
●●●●
●●
●
●●●●●●
●
●●●●●●●●
●
●●●●●
●
●
●
●●
●
●●●●
●
●●●●●
●●
●●
●●●●
●●●●●●●●●●●
●
●
●
●
●●●●●●●
●
●●
●
●●
●
●●●
●●
●
●
●
●
●●●●●●●●●
●
●●
●
●
●●●●
●
●●
●
●●
●
●
●●
●
●●
●●●
●
●
●
●●
●●●●●
●●●●
●
●
●●
●●
●
●●●
8907 8912 2669
●●●●●●●●●●●●●●●
8900 8817 2658
●●●●●
●
●●●●●●
●
●
●
●●●●●
●
●●
●
●●
●
●
●
●●●●
●●
●●●
●
●●●●●●
●●
●●
●
●●●●
●●
●●
●
●
●●●
●●●
●●●●
●
●●●
●●●●
●●●
●
●●
●
●●
●●
●●●●●
●●
●
●
●●
●
●
●●●
●
●
●
●●
●
●●●●
●●
●●●●●●
●●●●
●●●●
●
●●●●●●●●
●●●●●
●
●●●
●●
●●●●●●●●●●●●●●●●●
●
●●●●●●●●
●
●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●
●
●●●●●●
●
●●●●●●●●●●●●●●
●
●
●
●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●
●
●●●
●●
●●●●●●●●●●●●●●●●●
●
●●●
●
●●●●●●●●●●●●●●●●●●●●●
●
●●●●●
●
●●●●●●●
●
●●●
●●●
●●●●●●●●●
●
●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●●
●
●●●●●
●●●●
●●●●●●●
●●
●
●
●●●●●●
●
●●●●
●●
●●●
●
●●●●●
●
●●●●●●●●
●
●
●
●●●●●●●
●
●●●
●
●●●●●●●●●●●●●●●●●
●●●
●●●●●●
●
●●●●
●
●●●●●●●●●●
●
●
●●
●●
●
●●●●●●●
●
●●●
●●●●
●●
●
8900 8817 2658
●●●●●
●
●
●
●●
●
●
●●●●
●
●●●●
●
●●●●●
●●
●●●●●●●●●
●●●●●
●
●●●●●
●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●
●●
●
●●
●●●
●
●●●
●●
●
●
●
●●●
●
●●●●
●●
●●●●●●
●●●
●●
●
●●
●●
●●
●●●
●●
●●●●●●●
●●●
●●●●●●●
●●
●●●●●
●●●
●●
●
●●
●●●
●
●●●●●●
●●
●
●
●●●●
●●
●●
●●●
●●●●●
●
●●
●●●
●●●
●
●
●●●●●●●
●●●●
●
●●●●●
●●●
●●
●
●●
●●
●
●●●●
●●●●●
●●●●
●
●
●●
●
●
●
●●●
●
●●●●●●●●●
●●●●●●●●
●
●●
●
●
●●●●●●●●●
●●●●●●●
●●●
●●
●
●
●
●●
●●●●●
●
●●●●●●●
●
●●●●●
●●
●●●●●●●●●●
●●●●●●
●●●●●●●●
●●●●●●
●●●●●●●●●
●
●●●
●●●●●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●●●
●
●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●
●●
●●
●
●●●●●●●●●●●●●●●●
●
●●●
●
●●
●
●●●●●●●●
●●
●●
●●
●●●●●●●●●●●●●
●
●●
●●●●●●
●
●●●●●●●●
●
●
●
●●
●
●
●●●
●
●
●●
●
●●●●●●●●●●●●●●●
●
●
●
●
●
●
●
●●●●
●
●●●
●●●
●●●●
●
●
●●
●
●
●●
●
●
●
●
●
●●●●●
●
●●●
●
●●
●●
●●
●●●
8047 8847 2692
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●●●●
●●
●
●
●●
●●
●
●
●
●
●●
●
●
●●●
●
●
●
●●●
●●
●●●●●●●
●
●●●
●
●
●●●
●●
●●●
●●●●
●●
●
●
●
●●●
●●
●●●●●
●
●
●●
●●●
●●●
●●●
●
●●●
●
●●●●●
●●
●
●
●
●
●
●●●
●
●●●
●
●
●
●●●●
●●
●
●
●●
●
●
●
●
●
●
●●●●●
●
●●●●
●
●
●
●●●
●
●
●
●●
●●●●●●
●●
●●●●●●●
●
●●
●
●
●
●
●●
●
●
●
●●●
●
●●●●
●
●
●
●●●
●●
●●●
●●
●●●●●●●●●●●
●
●
●●●●●●●
●●
●
●●●●●
●●●●
●
●●●●●●
●
●●
●
●●
●
●
●●
●●
●
●●●
●
●●
●
●●
●●
●
●●
●
●
●●●
●
●
●●●
●●
●●
●
●●●●
●
●●●●●
●●
●●●
●●●●
●
●●●
●
●
●
●
●
●●
●
●
●●●●●●●
●●●●●
●●●●●●●
●
●
●
●
●
●
●●
●●
●
●●●●
●
●●●
●●●●●●●●●●●
●
●●
●
●●●
●
●●●
●●●●●
●●●●●●●●●●
●
●
●
●●●●
●
●●
●●●
●●
●●●
●●
●
●●●
●
●
●●●●●
●●●●●●●●
●●●
●●●●●●●●●●
●
●
●●●●
●●●●●●●●●●●
●
●
●
●
●●
●
●●●
●●●●●●
●●●
●
●●●●
●●●●●●●●●●●●●
●●
●
●
●●●●
●●●
●
●●●
●
●●
●
●●●●●●●●●●●●
●●●
●
●
●
●
●
●●●●
●●
●
●●●●●●
●
●
●
●
●●●●
●●●
●
●
●
●
●●●●●●●
●
●●●●●
●●
●●
●●
●
●
●●
●●
●
●
●●●
●●●
●
●●
●
●●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●
●●
●
●
●●●
●
●
●
●
●●
●●●
●●●
●●●●
●●●
●●
●
●
●
●
●
●
●
●
●●●
●●
●
●●●
●●
●
●
●
●●●
●●●●●
●●
●●
●
●●
●
●
●●●
●
●●●●●
●
●
●●
●
●●●●●●●
●●
●●●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●
●●●●●
●●
●●●
8047 8847 2692
LL LL Rasch
LR LR Rasch
LS LS Rasch
LW LW Rasch
0
20
40
60
80
100
0
20
40
60
80
100
0
20
40
60
80
100
0
20
40
60
80
100
6 7 8 6 7 8Cohort
Scor
e
Figure 2.3.7: Boxplots of Literacy scores for the cohorts in Grade 5 (the sample sizeis given above each boxplot).
Chapter 2. Data Analysis 33
●●●●●●●●●
8879 8812 2639
●
●
●●
●●●
●
●●
●●●
●
●
●
●●
●
●●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●●
●
●
●
●
●
●●●
●●
●●
●●●
●●●
●●
●●●
●
●
●
●●
●●
●●
●●
●●
●
●
●●
●
●
●●●
●
●●●
●
●●●●
●
●
●
●
●●
●●●
●●
●●●●
●
●
●●●
●
●●
●●●●●●●
●
●●
●●
●●
●
●
●●●●●
●
●●
●●●●●
●●
●
●●
●●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●●●
●
●●●●
●
●●
●
●●●●●
●
●
●●
●●
●●
●●●●●●●●●●●●●
●●
●
●
●
●
●
●
●
●
●●●
●
●●●
●
●
●●
●
●
●
●
●●
●
●●
●●
●●●
●
●●●●
●●●
●●●
●
●
●
●●●
●●
●●
●
●
●●●●●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●●●●
●
●
●
●
●
●
●
●
●●●●
●●●●
●
●
●●
●●●●
●
●●●●●●●
●●
●●●●●
●
●●●●●●●
●●
●
●●●●●●●●●
●●
●●
●●●
●
●
●
●●
●●●●●●●
●
●●
●
●●
●●
●
●●
●●
●
●●
●
●●
●●
●
●●●●●●
●
●
●●●●●
●
●
●
●
●●
●●●
●●
●
●
●●●
●
●●
●●
●●●●●●●●●●●●●●●●●
●
●
●●
●●
●●●
●●
●
●
●●●
●
●●●
●●●●
●
●
●●●●
●
●
●
●●●
●●
●●
●
●●
●●
●
●
●
●●
●
●
●●
●
●●●●●●
●
●
●
●●●●●
●●●●●●●
●
●●
●●●●●●●●●
●
●●●●●●●
●●
●
●
●●
●●●
●
●
●●
●
●
●●●
●
●
●
●●
●
●
●●●●●●●●
●
●
●
●●●●●●●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●●
●
●●●
●
●●●●●●
●
●●
●
●
●
●●
●●
●●●●
●●
●●
●
●
●
●●
●
●
●●
●
●●
●●
●●
●●
●●●
●
●●●●●●●●
●
●●●
●
●●
●
●
●
●
●●
●
●●
●
●
8879 8812 2639
●●●●●●●●●●● ●●●●●●●●●
●
8920 8832 2671
●●●●
●
●●●●
●●
●●●●
●
●●
●
●
●
●
●
●●
●
●
●●●●
●
●●
●
●
●
●
●●●●
●●
●
●●
●●●
●
●●
●●●●
●●●●
●
●●
●●●●●●●
●
●
●●
●●●
●●●●
●
●●●
●●●
●●●
●
●
●●●●●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●●●●●
●
●●●
●
●
●
●
●
●●
●●●●
●
●●●
●
●●
●●●
●
●
●
●●●
●●
●
●
●●
●●
●
●●●
●●
●●●●●
●
●
●
●
●
●●
●
●●●●●●●●●
●
●●●●●●●●
●●
●●●●
●
●●●●●●●
●
●●●●●●●●●●●●●●●
●
●●●●
●
●●●
●●●●
●
●●
●
●
●●●●●●●
8920 8832 2671
●●●●●●
●●
●●●●
●
●●●●●●
●●
●●
●
●●●●●●●●●●●●
●
●●●●●●●●●●
●
●
●●●●
●●●●●●●
●
●●
●
●●●●●●●●●●
●●
●●●●
●●
●
●
●●●●●●●●●
●
●●●
●●●●
●●●
●●
●●●●
●
●●●
●●
●
●
●●●●●●●
●●●
●●
●●●
●
●
●●
●
●●
●
●●●
●
●●●●●●
●
●
●●●●●
●
●●●●●●
●●●●●●
●●
●●●●●
●●●
●
●●
●●
●●
●●●
●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
8829 8836 2609
●
●
●
●●
●●
●
●
●
●
●●●
●
●●
●●
●
●
●
●
●●
●
●
●●
●●
●
●●●
●
●
●
●●
●
●●●●●
●
●●
●●
●●●
●●
●
●●●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●●●
●
●
●●●
●
●
●
●
●
●●●●
●
●
●●●●●
●
●●●●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●
●●●●●●
●
●●
●
●●●
●●●●●
●●
●
●
●
●
●●
●
●
●●
●
●
●●●
●
●
●●
●●●
●●
●
●
●
●
●●●●●●
●
●
●
●●
●
●●●
●
●●
●●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●●
●
●
●●●●
●
●●
●
●●
●
●
●
●●
●
●
●
●●●●●
●
●
●
●
●
●●●●●●●
●●
●●
●
●●●●●●
●
●●●●
●●
●●
●●
●●
●●
●
●●●
●●●●
●●
●●
●
●
●
●●●
●●
●
●
●
●●
●●
●
●●
●
●
●
●●
●
●●●●
●
●
●
●
●●
●●●
●
●
●
●●●●●●
●
●●
●
●
●
●
●●
●●
●
●●●●
●●
●
●
●●●
●
●●●●●●●●●●●●
●
●●
●
●●
●
●
●●
●
●●
●
●●●●
●
●●●●●●
●
●●●●
●
●●●●●●●
●●
●
●
●
●
●●●
●●
●
●●
●
●
●
●●
●●●●●●●●●●●
●
●●●●●
●●
●●●
●
●
●
●●●●●●●●●
●●
●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●
●
●●●●
●●●
●●●●●●●●
●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●
●
●●●●● ●●
●
●●●●●●●●
●
●
●
●●●●●●
●
●
●
●●●●
●
●●●●
●
●●●●●●●●●●●●●●●●●●●
8829 8836 2609
●●●●●●●●●●●
8896 8859 2635
●
●
●
●●●●●
●●●
●
●●
●
●
●
●●●
●●●●
●
●●●
●
●
●●
●●●●●●
●●
●●
●
●
●
●
●●●
●●
●●●●●●
●●
●
●●●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●●
●
●
●
●
●●●●●
●
●●●●●
●●●
●
●●●
●●
●●●
●●
●
●●●●●●●
●●●●●
●
●●●●
●●
●●●
●
●
●●
●
●
●
●●
●●●
●●●●●●●●●●●
●
●●●
●
●●●●●●●●
●
●
●
●●●
●
●
●●●●●
●●●●●●
●
●●●●●●●●
●
●
●
●●
●●●
●●●●●●
●
●●●●●●●●
●
●
●
●●●●●●●●●●●●●●●
●●
●●●
●
●●●●●●●●●●
●●●●●
●●●●●●●●●●●●●●
●●
●●●●●●●●●●●
●
●●●
●
●●
●
●●●●
●●
●
●●●●●
●
●●●
●●
●●
●●
●●
●
●●
●●
●
●
●●●●
●●
●
●
●
●
●●●
●●
●
●
●
●●
●
●
●●
●
●●
●
●
●●●●●●
●●●●
●●
●●●●●●
●
●●●
●
●
●●●●
●●●
●●
●
●●●●
●●
●
●●●
●
●
●
●
●●●
●●
●●
●
●
●●
●
●●
●●
●
●●●●
●●●●
●●
●●●●
●●
●●●●●
●●●
●
●●●●●●●●●
●●
●●●●●●●●●●
●
●
●
●●
●
●
●●●●
●
●●
●
●●●●●●●●●●●
●
●
●
●●
8896 8859 2635
NM NM Rasch
NN NN Rasch
NS NS Rasch
NU NU Rasch
0
20
40
60
80
100
0
20
40
60
80
100
0
20
40
60
80
100
0
20
40
60
80
100
6 7 8 6 7 8Cohort
Scor
e
Figure 2.3.8: Boxplots of Numeracy scores for the cohorts in Grade 5 (the samplesize is given above each boxplot).
34 2.3. Descriptive Statistics
●●●●
●
●●●●
●●●●
2526
●●●●●●●
●●●●●
●
●
●
●
●
●
●●
●
●●●●●
●
●
●●
●
●
2526
●●●●●●●
2568
●●
●
●●●
●
●
●
●
●●
●●●
●
●
2568
2551
●●●●●●●●●●●●●●●
●
●●●●●●●●●●
●
●
●●
●●●●●●●
●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●
●●
●
●●
●
●
●
●●
●●●●
●●
●
●●●●
●●●
●
●●
2551
●●●●●●●●●●●●●
●●
●
●
2555
●
●
●
●●●●●●
●●
●●●
●
●
●
●●
●
●
●
●●
●
●
●●●●
●
●●●
●
●●●●
2555
LL LL Rasch
LR LR Rasch
LS LS Rasch
LW LW Rasch
0
20
40
60
80
100
0
20
40
60
80
100
0
20
40
60
80
100
0
20
40
60
80
100
6 6Cohort
Scor
e
Figure 2.3.9: Boxplots of Literacy scores for the cohorts in Grade 7 (the sample sizeis given above each boxplot).
Chapter 2. Data Analysis 35
2556●●●●
●
●●●●●●●●●●●●
●
●●●●●●●●●●
●
●
●
●●●●●
●
●●●●
●
●
●
●●●●●●●●●●●
2556
2554
●
●
●●●
●
●●●●●●●●●
●
●●●●●●●●●●●●
●2554
2544●
●●●●
●
●●
●
●
●
●
●●
●●
●
●●
●
●
●●
●●●●
●
●●
●
●●
●●●
●
●
●●
●
●●
●
●
●
●●●●●
●
●
●
●●
●
●
●●●
●●●●
●
2544
2516●
●
●●
●
●●
●●
●●●●●●●●●●●
●●
●●●●●●●
●
●●●
●
●
●
●●
2516
NM NM Rasch
NN NN Rasch
NS NS Rasch
NU NU Rasch
0
20
40
60
80
100
0
20
40
60
80
100
0
20
40
60
80
100
0
20
40
60
80
100
6 6Cohort
Scor
e
Figure 2.3.10: Boxplots of Numeracy scores for the cohorts in Grade 7 (the samplesize is given above each boxplot).
36 2.3. Descriptive Statistics
It is important to note that the Rasch model can result in a symmetric, bell-shaped
distribution of the scores, even when the raw scores are not symmetrically or nor-
mally distributed. An example to illustrate this is given in Figure 2.3.11 where we
look at the literacy scores for Grade 3 participants in Cohort 8. The distribution
of the LL raw scores is skewed to the left, but the Rasch scores are normally dis-
tributed. Another interesting feature is found in the plots of the LW scores (Figure
2.3.11) as an �outlier� peak is highlighted in the Rasch distribution which is not as
apparent in the plot of the raw LW scores. This could be a result of the standard-
isation and normalisation of the scores, as participants with a mark of zero cannot
be scaled by the Rasch model.
Chapter 2. Data Analysis 37
LL LL Rasch
0
200
400
600
800
1000
0 20 40 60 80 0 20 40 60 80Score
Cou
nt
LW LW Rasch
0
500
1000
1500
0 20 40 60 0 20 40 60Score
Cou
nt
Figure 2.3.11: Histograms of the distribution of raw and Rasch LL and LW Grade3 scores in Cohort 8.
38 2.3. Descriptive Statistics
The mean Literacy and Numeracy scores are plotted for each cohort and grade in
Figure 2.3.12. One observation is the increasing trend of mean Rasch marks between
sequential grades. This is caused by the test design and the use of common questions
in the Grade 3, 5 and 7 tests of a particular year. It is expected that students
who have progressed further in their education and are in higher grades, are more
likely to correctly answer the common questions, compared to students in lower
grades, resulting in a higher mean score. Through the use of Rasch scaling, these
common questions allow comparability of results across grades, as the `di�culty' of
all questions is now on a common scale. Even in some instances, the mean raw score
decreases, but the mean Rasch score increases. One example of this is Cohort 5's LS
scores - the mean raw score decreases from Grade 3 to Grade 5, but the mean Rasch
score increases. This illustrates the purpose of Rasch scaling in placing all scores
on a standard scale. At each grade level, the mean scores for the Rasch aspects are
plotted almost on top of each other, which is to be expected from Rasch scaling.
Numeracy tests have lower mean scores than Literacy tests which is due to the lower
total available marks for Numeracy aspects. The total Literacy (LL) and Numeracy
(NN) aspects clearly have larger mean scores as they are the sum of all the other
Literacy and Numeracy aspects respectively.
In a similar manner, Figures 2.3.13 and 2.3.14 compare the progression of mean
scores of each cohort from Grades 3, 5 and 7 in each aspect. This plot compares
cohorts at each grade level and in each aspect. Each cohort would have sat a
di�erent test at each grade, hence the di�erence in the raw scores. Table 2.3.8 gives
the mean Rasch aspect scores for each cohort and grade. Analysis of variance to
test for equality of means concludes that there is a di�erence in the mean scores
in both Literacy and Numeracy (all P -values are < 2e-16). Further investigation
could be done using pair-wise comparisons to identify which cohorts are statistically
signi�cantly di�erent, but the di�erences are small - less than one Rasch mark - so
not very important when compared to overall Rasch scores of 50 to 60 marks.
Note that for the purposes of reproducible research, all R [45] output is recorded
as displayed in the default R workspace and to the same number of decimal places.
For this reason and throughout this thesis, P -values are stated as given in R.
Chapter 2. Data Analysis 39
6 7 8
10
20
30
40
50
60
10
20
30
40
50
60
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
LN
3 4 5 6 7 3 4 5 6 7 3 4 5 6 7
Grade
Mea
ns
Score Type ● Raw Rasch
Aspect ● ● ● ● ● ● ● ●LL LR LS LW NM NN NS NU
Figure 2.3.12: Mean aspect score in each multi-year cohort and across grades.
40 2.3. Descriptive Statistics
●
●
●
●
●
●
●
●●
●
●
●●
●●
●●●
●
●
●●
●
●●
●●
●
●
●
●
●
●●●
LL LL Rasch
LR LR Rasch
LS LS Rasch
LW LW Rasch
102030405060
102030405060
102030405060
102030405060
3 4 5 6 7 3 4 5 6 7
Grade
Mea
ns
Score Type ● Raw Rasch
Cohort ● ● ● ● ●6 7 8 9 10
Figure 2.3.13: Comparing the progression of cohorts' mean scores along grades ineach Literacy aspect.
Chapter 2. Data Analysis 41
● ●●
● ●●
●●●
●
● ●
●
●
●
●●●
●●
●●●
●●
●●
● ● ●●●
●●●●
NM NM Rasch
NN NN Rasch
NS NS Rasch
NU NU Rasch
102030405060
102030405060
102030405060
102030405060
3 4 5 6 7 3 4 5 6 7
Grade
Mea
ns
Score Type ● Raw Rasch
Cohort ● ● ● ● ●6 7 8 9 10
Figure 2.3.14: Comparing the progression of cohorts' mean scores along grades ineach Numeracy aspect.
42 2.3. Descriptive Statistics
Table 2.3.8: Mean scores for each cohort in each grade
Cohort gradedyear LL Rasch LR Rasch LS Rasch LW Rasch6 3 48.27 48.30 48.26 -7 3 49.15 49.04 49.63 48.458 3 49.10 49.33 49.10 50.969 3 48.99 48.76 49.94 49.5210 3 49.87 50.01 50.86 50.186 5 55.87 56.23 55.64 56.227 5 55.31 55.44 56.18 55.268 5 56.68 57.09 57.53 56.986 7 61.22 61.33 61.60 61.26
Cohort gradedyear NM Rasch NN Rasch NS Rasch NU Rasch6 3 49.24 48.87 48.35 49.457 3 50.00 49.30 48.01 50.188 3 52.30 51.14 50.86 51.149 3 49.70 49.47 49.74 49.6110 3 49.27 49.05 48.86 49.326 5 57.42 57.05 56.76 57.407 5 59.26 59.19 59.38 59.498 5 59.01 58.63 59.18 58.416 7 65.46 65.48 65.83 65.44
Chapter 2. Data Analysis 43
Table 2.3.9: Number of participants by year, grade and number of tests to date.Grades are along the rows, with calendar year and the number of tests to date inthe columns.
1997 19981 2 3 4 5 1 2 3 4 5
3 6 0 0 0 0 7 0 0 0 05 1 0 0 0 0 1 0 0 0 07 0 0 0 0 0 0 0 0 0 0
1999 20001 2 3 4 5 1 2 3 4 5
3 68 0 0 0 0 9259 12 1 0 05 1 4 0 0 0 4 6 0 0 07 0 0 0 0 0 0 0 0 0 0
2001 20021 2 3 4 5 1 2 3 4 5
3 9095 30 0 0 0 9696 43 1 0 05 16 36 0 0 0 2506 7381 18 1 07 1 1 1 0 0 1 2 3 1 1
2003 20041 2 3 4 5 1 2 3 4 5
3 10935 31 0 0 0 3167 17 0 0 05 2881 7725 38 0 0 935 2254 15 0 07 16 5 7 0 0 719 631 1744 7 0
20051 2 3 4 5
3 7 0 0 0 05 3 0 0 0 07 3 0 0 0 0
2.3.3 Number of Tests: Longitudinal View
The numbers of participants at each grade level and year and how many tests have
already been taken by a participant at that point in time, including the test at
that particular time point, is given in Table 2.3.9. In cohort 6, there are only 1744
participants who sat all three grades.
It can also be observed that there are a number of participants who do not seem
to follow the normal system and sit multiple tests at the same grade level. These
44 2.3. Descriptive Statistics
participants are represented in the numbers o� the diagonal for each year in Table
2.3.9. This might be explained by the re-use of student identi�cation numbers by
schools or the event that a student re-sits a grade.
2.3.4 Test Aspects
To investigate the data further at the student level, the number of participants with
test aspects in each year and grade is given in Table 2.3.10.
The number of test aspects in each grade and year are almost equal, but there is
a large di�erence between them and the participant numbers in the corresponding
grade and year - hence, there must be missing data for some participants. One
possible reason for missing data could be a student is ill on the day of the test, and
so, no score is recorded. Data entry could be another possible cause for a missing
test score.
As the Literacy and Numeracy aspects are very di�erent from each other, we consider
them separately for a single participant. Hence in total, there are 138 688 �tests�.
Out of these, 52 898 tests have at least one missing aspect.
2.3.5 Missing Test Aspects
As noted, some participants have missing aspect scores. So the question is �Is one
aspect more likely than others to be missing?� or �Is each aspect equally likely to be
missing?�. So conditioning on the participants who have at least one missing aspect,
for each aspect, the proportion of participants missing that aspect is given in Table
2.3.11.
Chapter 2. Data Analysis 45
Table 2.3.10: Number of participants with test aspects in each year and grade
1997LL LR LS LW NM NN NS NU
3 6 5 6 0 5 5 6 65 1 1 1 0 1 1 1 17 0 0 0 0 0 0 0 0
1998LL LR LS LW NM NN NS NU
3 7 6 6 0 5 6 7 75 1 1 1 0 1 1 1 17 0 0 0 0 0 0 0 0
1999LL LR LS LW NM NN NS NU
3 63 58 61 0 57 55 63 625 5 5 5 0 4 4 5 57 0 0 0 0 0 0 0 0
2000LL LR LS LW NM NN NS NU
3 8 302 8 300 8 330 0 8 365 8 336 8 386 8 3505 6 8 10 0 10 10 7 77 0 0 0 0 0 0 0 0
2001LL LR LS LW NM NN NS NU
3 8 207 8 071 8 070 7 254 8 113 8 209 8 101 8 0695 49 47 47 44 43 49 47 467 3 3 3 3 3 3 2 3
2002LL LR LS LW NM NN NS NU
3 8 677 8 674 8 687 7 693 8 691 8 683 8 745 8 7265 8 863 8 907 8 900 8 047 8 879 8 920 8 829 8 8967 6 7 7 7 7 7 6 6
2003LL LR LS LW NM NN NS NU
3 8 912 9 056 8 937 9 072 9 109 9 086 9 078 9 1075 8 889 8 912 8 817 8 847 8 812 8 832 8 836 8 8597 18 20 17 19 18 17 16 18
2004LL LR LS LW NM NN NS NU
3 2 657 2 621 2 639 2 642 2 618 2 619 2 611 2 6135 2 662 2 669 2 658 2 692 2 639 2 671 2 609 2 6357 2 526 2 568 2 551 2 555 2 556 2 554 2 544 2 516
2005LL LR LS LW NM NN NS NU
3 6 7 7 7 7 6 7 75 1 1 1 1 1 2 2 27 2 2 3 3 3 2 3 3
46 2.3. Descriptive Statistics
Table 2.3.11: Number and proportion of participants with missing aspects
Aspect Number ProportionLL 9 475 0.226LR 9 395 0.224LS 9 580 0.228LW 11 088 0.299NN 9 266 0.221NM 9 397 0.224NS 9 432 0.225NU 9 400 0.224
Missing Literacy Writing Results
From Table 2.3.11, a consistent percentage of 22% of participants are missing scores
for LL, LR, LS, NN, NM, NS or NU, but 30% of participants do not have a recorded
LW score. In order to try and explain why the proportion of LW missing is higher
than the other aspects, Table 2.3.12 gives the number of missing LW aspects com-
pared to all participants with a missing aspect, on a yearly basis. The LW aspect
was only introduced in 2001, hence we only have data for missing LW aspects from
2001 to 2005. The average proportion is 0.297 and the proportions in Table 2.3.12
are all larger than the proportions of other missing Literacy aspects in 2001-2005.
Table 2.3.12: The number of participants with missing LW by year
Year Number Missing Total Count Proportion2001 1 879 5 761 0.332002 3 907 12 215 0.322003 3 700 13 278 0.282004 1 600 5 816 0.282005 2 7 0.29
The proportion of the number of missing LW scores out of the total number of
participants in the school is plotted for all schools, combining all data from 2001 to
2005 for each school (Figure 2.3.15).
These results suggest the increased proportion of LW missing is not due to some
schools missing nearly all of the LW scores and the remaining results being around
Chapter 2. Data Analysis 47
0.0
0.2
0.4
0.6
0.8
1.0
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●
●
●
●●
●
●
●●
●●●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●●
●●
●●
●●●●
●●●
●●
●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●●
●
●●
●
●●●
●●●●
●
●
●
●
●
●
●●●●
●
●●
●
●
●
●●
●
●
●
●●
●
●●●●
●●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●●
●●
●
●
●●
●
●●●
●
●
●
●●●●●
●
●
●
●
●
●●
●
●●●●●●●●●●●
●
●
●
●
●●
●●
●
●
●●●
●
●●
●
●●●●●
●●
●
●●
●●
●
●●●
●
●●●
●●●●
●●●●
●
●
●
●
●
●
●●
100 200 300 400 500 600School Number
Pro
port
ion
Size of School●
●
●
●
●
●
(0,10]
(10,20]
(20,50]
(50,100]
(100,300]
(300,560]
Figure 2.3.15: Proportion of participants in each school missing LW.
the usual proportion of 0.22. Instead, there is clustering of schools around the state
average which is higher for LW compared to the other aspects.
Goodness of Fit - Binomial Model
In addition to which aspects are missing, we investigate the distribution of the
number of missing aspect scores. The number of participants with no, one, two,
three and four missing scores for Literacy and Numeracy are given in Table 2.3.13.
The Literacy scores are divided into those before 2001 (Literacy - pre 2001) and
48 2.3. Descriptive Statistics
those after 2001 (Literacy - post 2001) due to the introduction of LW in 2001, which
increases the number of recorded Literacy aspects from three to four aspects.
Table 2.3.13: The number of participants with missing scores for aspects
Number of Literacy - pre 2001 Literacy - post 2001 Numeracymissing scores0 6 796 35 343 43 6511 2 265 18 058 19 5002 277 3 497 3 2043 32 733 3694 - 2 343 2 620
Looking at the counts in Table 2.3.13, we notice that for Literacy-post 2001 and
Numeracy, the counts for having four, that is all, missing scores does not �t a
binomial distribution, but the observed outcome frequencies for having no, one, two
or three missing scores appear to follow an approximate binomial distribution. In
order to test how well the binomial distribution �ts the observed counts, we apply
a Pearson's χ2 test to measure the goodness of �t.
Table 2.3.14 compares the observed and expected counts, and we can see that for
all but the last row, the binomial distribution closely approximates the outcome
frequencies. However, the statistical conclusion of all three χ2 tests is that the
binomial distribution does not model the observed frequencies (P -values of 2.7e-10,
3.1e-271 and 5.9e-62 respectively).
Table 2.3.14: The observed and expected frequencies for the goodness-of-�t χ2 testof the Binomial model
Literacy - pre 2001 Literacy - post 2001 NumeracyNumber Observed Expected Observed Expected Observed ExpectedMissing0 6 796 6 746.8 35 343 34 449.6 43 651 43 190.91 2 265 2 341.8 18 058 19 337.5 19 500 20 215.22 277 270.9 3 497 3 618.2 3 204 3 153.93 32 10.4 733 225.7 369 164.0
Chapter 2. Data Analysis 49
2.4 Cleaning the Data: Forensic Statistics
The correct evaluation, presentation and interpretation of the data is vital, and we
describe the process of asking questions when �strange things� are observed and delv-
ing further into the layers of the data to try and statistically explain the observation
as forensically digging into the data - hence, forensic statistics.
Before any statistical analysis or even descriptive statistics can be done, the data
must be such that no obvious errors can be detected. The cleaning process involves
understanding the data, the data structure and identifying anomalies. We wish
to investigate any discrepancies, with the purpose of possibly �nding a systematic
observation which can be explained in context. We will then be able to either infer
results, given su�cient information, or exclude data. We are limited in what we can
explain and infer with con�dence because of the lack of information on reasons for
missing or incorrect data. This information would normally be supplied by those
who provided the data, but in our case, was unavailable. The individual schools
could not be approached either because all school information was de-identi�ed for
con�dentiality purposes and political reasons.
The initial analysis starts with looking at the blanks or missing data for the variables
schoolno, procyear, studentide, aspect, nocorrect, standardsc and gradedyear.
This then identi�es a group of students who do not have any recorded test scores
for both nocorrect and standardsc. However, eliminating these students from the
data set may cause misinterpretation later when we try to track students and their
tests longitudinally. The lack of scores for students could be explained if students
were exempted or not present for the tests, but they are still recorded in the school
records as being part of the school cohort but with no scores. It is concluded that
the students with no test results must be included as we can still �follow� them
through the years if they have sat other tests. More information about schools will
also be available as a result.
2.4.1 Consistency of School Data
From the table of school variables (Table 2.2.2), the values of these school variables
are checked to be the same for all students in each school and are found to be so.
50 2.4. Cleaning the Data: Forensic Statistics
2.4.2 Consistency of Student Data
As with the school data, we check for consistency across the records of all student
variables (Table 2.2.3) for each individual student. There are 136 437 students
for which their student data is consistent, but three main groups of anomalies are
identi�ed:
• ATSI changes between categories 1 and 2 within and between tests,
• LBOTE changes between categories 1 and 2 within and between tests, and
• other variables have discrepancies.
Out of the identi�ed anomalies, it is decided that changes in ATSI and LBOTE will
be de�ned to be �Inconsistent� and left in the data set for now. Similarly, among the
other variables of school_car, occupation, school_edu, non_school, p_g_gender,
p_g_cultur, p_g_countr, p_g_nesb, country_of, nesb_code, home_langu, cultural_b
and visa_sub_c, any �errors� involving parent information are also deemed not rel-
evant at this point in time. In the end, 13 participants and their data are removed
from the data set - reasons are given in Table 2.4.1.
It could also be possible that schools might recycle student ID numbers and have
more than one student with the same ID at a future point in time. This is a
possibility because in the Basic Skills data set, student identi�cation is not global,
but rather school speci�c and under the jurisdiction of the school - a student is
uniquely identi�ed by the combination of their school ID and student ID numbers.
2.4.3 Score Check
Part of the cleaning of the data involves a check of the results to make sure the sum
of the sub-aspect scores equals the recorded total score for Literacy and Numeracy.
Considering Literacy and Numeracy aspects separately, there are 50 884 participants
whose scores add up correctly, but 34 867 Literacy participants and 39 Numeracy
participants for which the added score does not equal the recorded total. We can
only consider participants for which all scores are recorded - having missing aspect
scores means that it cannot be checked whether the sum total equals the recorded
Chapter 2. Data Analysis 51
Table 2.4.1: Anomalies in student data
schoolno studentide Anomaly109 1243669 status changes from A to L
school_car changes from Y to N within a single test173 920059 disability changes from N to Y within a single test297 1257837 status changes from L to A and school_car changes
from N to Y within a single test297 858523 date_of_bi changes randomly and not in connection to
changes to aboriginal status297 969411 status changes from L to A and school_car changes
with status
361 1127557 changing date_of_bi
399 1051563 status changes between A and L426 862091 status changes from L to A
consistent with above changes are changes inschool_car (A with Y and L with N)
477 1006947 status changes from A to L alternately497 1143261 status changes from A to L within the same test546 1189293 status changes from A to L595 1116261 status changes from L to A alternately
consistent with changes in school_car (N and Y)623 1031323 disability changes within tests
total scores. So, we only consider the data for the participants with all recorded
scores.
We de�ne and categorise these errors to be
• Literacy Flag 1 ⇒ LR + LS + LW < LL,
• Literacy Flag 2 ⇒ LR + LS + LW > LL and LR + LS 6= LL,
• Literacy Flag 3 ⇒ LR + LS + LW > LL and LR + LS = LL,
• Numeracy Flag 1 ⇒ NM + NU + NS < NN, and
• Numeracy Flag 1 ⇒ NM + NU + NS > NN.
The number of participants and schools with these Literacy and Numeracy �ags are
given in Table 2.4.2.
52 2.4. Cleaning the Data: Forensic Statistics
Table 2.4.2: The number of schools and participants with anomalies in the sum ofthe scores
Test Category No. of No. ofSchools Participants
Literacy 1 (LR + LS + LW < LL) 417 18 3072 (LR+ LS + LW > LL & LR + LS 6= LL) 47 523 (LR + LS + LW > LL & LR + LS = LL) 413 16 508
Numeracy 1 (NM + NU + NS < NN) 4 42 (NM + NU + NS > NN) 33 35
The total number of schools is 426 schools, so the fact that such a large majority of
schools are committing Literacy Flags 1 and 3 is a worrying sign. With the relatively
small numbers for Literacy Flag 2 and Numeracy Flags 1 and 2, we attribute these
errors to random noise, possibly incorrect random data entry errors, and can remove
them from the data without much impact. However, the larger cohorts for Literacy
Flags 1 and 3 require further investigation.
Literacy Flag 1 only occurs in 2003 and 2004 (Table 2.4.3), while Literacy Flag 3
occurs in 2001 and 2002 (Table 2.4.4). It seems that when LW was �rst introduced
in 2001, the majority of schools did not include it in their total score. The source
of this error could have been the use of a former spreadsheet which did not take
into account the introduction of LW, since Literacy Flag 3 requires LR+LS=LL and
LR+LS+LW > LL. Then in 2003 and 2004, a structural problem or error led to the
total recorded score being larger than the sum of all the sub-aspects. This could
have been caused by an error in a replacement spreadsheet - no literature has been
found of yet to support this claim - but we were unable to identify the reason for
the error.
Table 2.4.3: The number of participants in 2003 and 2004 with Literacy Flag 1
Year Flagged Total Proportion2003 12 713 12 953 0.9812004 5 590 5 708 0.979
We now consider the distribution of the proportion of errors for each school on a
year-by-year basis to see whether schools are 100% wrong or only have a few errors
Chapter 2. Data Analysis 53
Table 2.4.4: The number of participants in 2001 and 2002 with Literacy Flag 3
Year Flagged Total Proportion2001 5 185 5 295 0.9792002 11 323 11 379 0.995
in the recorded scores for Literacy Flag 1 and Flag 3 (Figures 2.4.1 to 2.4.4).
In Figures 2.4.1 to 2.4.4, the top plot is the proportion of participants in each school
which have Literacy Flag 1 or 3, the middle plot is the proportion of the participants
in each school ordered from lowest to highest (this gives the ordered school index)
and depicted as a step function and the bottom plot is just a truncation of the
middle plot to highlight the proportions which are less than one. These plots exhibit
a certain clustering of points at discrete proportions like 0.5 or 0.75 for example.
These could be explained by a single class or teacher having identi�ed the error in
the recording of the scores and taken measures to correct and �x it. The bottom
plots of Figures 2.4.1, 2.4.2, 2.4.3 and 2.4.4 illustrate this discretisation more clearly.
We can observe from these plots that the majority of schools have 100% of their
participants with Literacy Flag 1 or 3, depending on the year.
54 2.4. Cleaning the Data: Forensic Statistics
0.5
0.6
0.7
0.8
0.9
1.0 ●●●
●
●●●●●●●●●●●●
●
●●●●●●●●●●
●
●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●
●●●●●●
●
●
●
●
●●●●●●●●●●
●
●●●●●●●●●●
●
●
●
●●
●
●●●●
●
●●
●
●●●●
●
●●●●●●
●
●●●
●
●
●
●●●●●
●●
●
●
●●●●●●●●●●
●
●●●
●
●
●
●
●●●●
●
●
●●●
●
●●
●
●
●
●●●●●●●●
●
●●●
●●●
●
●
●
●●
●
●
●
●
●
●
●●●●●●
●
●●●
●
●●●
●
●
●●
●
●
●
●
●●●
●
●
●●
●●
●
●
●
●
●●●
●●●
●●
●
●
●●
●
●●●●●
●●
●●
●●●
●●
●●
●
●●●●●●
●
●●
●
●
●●
●●
●
●●●●
●
●
●
●
●
●●●
●●
●●●●●●
●
●
●
●
●●●
●
●
●
●●
●
●●
●●●
●●
●
●●●
●
●
●●
●●●●
●
●
●
●
●
●●
●
●●●●●●
●
●●●●
●●●
●
●●●●●
●
●
●●
●
●
●●●
●
●
●●●●●●●●●
●
●
●●
●
●
●●●
●
●
●
●●
●● ●●
100 200 300 400 500 600
Literacy Flag 1 − 2003
School Number
Pro
port
ion
0.5
0.6
0.7
0.8
0.9
1.0
●
●
●●●●
●
●●●●
●●●●●●
●●●●●●●●●
●●●●●●●●●
●●●●●●●●●●
●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
100 200 300 400
Literacy Flag 1 − 2003
Ordered School Index
Pro
port
ion
0.5
0.6
0.7
0.8
0.9
1.0
●
●
●●●●
●
●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●
50 100 150
Literacy Flag 1 − 2003
Ordered School Index
Pro
port
ion
Figure 2.4.1: Proportion of participants in a school which have Literacy Flag 1 in2003. Middle plot is of the ordered proportions from lowest to highest, de�ning theordered school index, and the bottom plot shows the subset of the �rst 150 schools.(See text for further details.)
Chapter 2. Data Analysis 55
0.4
0.5
0.6
0.7
0.8
0.9
1.0 ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●
●
●●
●
●●●●●●●●
●
●
●
●●●●●●
●
●●●●●●●
●
●
●●●●
●
●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●
●
●●
●
●●●●
●
●●●●●●
●
●●●●●
●
●●●
●
●
●●●●●●●●
●
●●●●●●
●
●●●●●●●●●●
●
●●
●
●
●
●●
●
●●●●●●●●●●●●●●●
●
●
●
●
●●●
●
●●
●
●
●●●●●●
●
●
●
●●●
●
●●●●
●
●
●
●●●●●●●●●●
●
●●●●●●
●●
●●●●
●
●●●
●
●●●●●●●
●
●●●
●
●●●●
●
●
●●
●●●●●
●
●
●
●
●
●●
●
●●●●●●
●
●
●
●
●●
●●
●
●●●●●●●
●
●
●
●
●●●●●
●
●●●
●
●●●●●●
●
●
●
●
●
●
●
●
●●
●●●●
●
●
●
●●●
●
●●
●
●●●●●●●●●●●●●
●
●●
●
●●
●●
●
●
●●
●
●
●●●●●
●
●●
●
●●
●
●● ●●●●
●
● ●● ●
●
● ●●●●
100 200 300 400 500 600
Literacy Flag 1 − 2004
School Number
Pro
port
ion
0.4
0.5
0.6
0.7
0.8
0.9
1.0
●
●●●●
●
●●●
●●●●
●●●●
●●●●●●●
●●●●●●●●●●●
●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
100 200 300 400
Literacy Flag 1 − 2004
Ordered School Index
Pro
port
ion
0.4
0.5
0.6
0.7
0.8
0.9
1.0
●
●●●●
●
●●●
●●●●
●●●●
●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
20 40 60 80 100
Literacy Flag 1 − 2004
Ordered School Index
Pro
port
ion
Figure 2.4.2: Proportion of participants in a school which have Literacy Flag 1 in2004. Middle plot is of the ordered proportions from lowest to highest, de�ning theordered school index, and the bottom plot shows the subset of the �rst 100 schools.(See text for further details.)
56 2.4. Cleaning the Data: Forensic Statistics
0.4
0.5
0.6
0.7
0.8
0.9
1.0 ●●●●●●●●●●●●●●●●●●●●●●
●
●●●
●
●●●●●●●●●●●●●
●
●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●
●
●●●●●●
●
●●●●●●●●●●●●●●
●
●●●●●●●●
●
●
●
●●●●●●●●●
●
●
●
●●●
●
●●●●
●
●●●
●
●●●●●●●●●●●●●●●●●
●
●
●●
●
●
●●●●●●●
●
●●
●
●●●●●●●●●●●
●
●
●
●
●
●●
●
●●●●
●
●●●●●●●
●
●●●
●
●●
●
●●●●●●●●
●●
●●●●●●
●
●●●●
●
●●●●●●●●●●●●●●●
●
●●
●
●●●●●
●
●
●●●●●●
●
●
●
●
●●●●●
●●
●●
●
●
●●●●●●●●●●●
●
●●
●
●●●●●●
●
●●
●●
●●●●
●
●
●●
●●
●●
●
●
●●●●
●
●
●
●
●●●●
●
●●●●●●●
●
●
●●
●●●●
●
●●●●●●●
●
●●
●●●●●●
●
●●
●
●●●
●
●
●
●●
●
●
●
●●●●●●●● ●●●
●●
● ●● ●●
●●●
100 200 300 400 500 600
Literacy Flag 3 − 2001
School Number
Pro
port
ion
0.4
0.5
0.6
0.7
0.8
0.9
1.0
●
●●
●
●
●●
●●●●●●●
●●●●●●
●●●●●●●●●●●●●
●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
100 200 300 400
Literacy Flag 3 − 2001
Ordered School Index
Pro
port
ion
0.4
0.5
0.6
0.7
0.8
0.9
1.0
●
●●
●
●
●●
●●●
●●●●
●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
20 40 60 80
Literacy Flag 3 − 2001
Ordered School Index
Pro
port
ion
Figure 2.4.3: Proportion of participants in a school which have Literacy Flag 3 in2001. Middle plot is of the ordered proportions from lowest to highest, de�ning theordered school index, and the bottom plot shows the subset of the �rst 90 schools.(See text for further details.)
Chapter 2. Data Analysis 57
0.4
0.6
0.8
1.0 ●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●
●
●●●●●●●●●
●
●●●●●●●●●●●●●●●●
●
●●●●●
●
●●●●●●●●●●●●●●
●
●●●●
●
●●●●●●
●
●
●●●●●●●●
●
●●
●
●●●●●●●●●●●●●●●●●●●●●●●
●
●
●
●●●●●●●●●●●
●●
●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●
●
●●
●
●●●●
●
●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●
●
●
●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●
●
●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●
●●●●●●●
●●●●
●●●●●●
●● ●● ● ●●
100 200 300 400 500 600
Literacy Flag 3 − 2002
School Number
Pro
port
ion
0.4
0.6
0.8
1.0
●
●
●
●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
100 200 300 400
Literacy Flag 3 − 2002
Ordered School Index
Pro
port
ion
0.4
0.6
0.8
1.0
●
●
●
● ●● ●
●●
● ●●
● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●
10 20 30 40 50 60
Literacy Flag 3 − 2002
Ordered School Index
Pro
port
ion
Figure 2.4.4: Proportion of participants in a school which have Literacy Flag 3 in2002. Middle plot is of the ordered proportions from lowest to highest, de�ning theordered school index, and the bottom plot shows the subset of the �rst 60 schools.(See text for further details.)
58 2.4. Cleaning the Data: Forensic Statistics
For the case of Literacy Flag 1, where LR+LS+LW 6= LL, we wish to investigate
whether it is a common shift or scale in the marks or just random noise. The total
sum LR+LS+LW is plotted against the recorded total LL (Figures 2.4.5 and 2.4.6),
and the di�erence between the two quantities ranges from 2 to 12 approximately.
Note that all the data points lie below the line LL = LR+LS+LW as Literacy
Flag 1 is de�ned to be LR+LS+LW < LL. There appears to be a systematic trend
in Literacy Flag 1 due to grade and calendar year - however, this is caused by
the overplotting of the latest category on top of the previous points, so the R
functions alpha (aesthetic in ggplot2 package) and jitter (base package) are
used to assess the density and spread of the colours and evoke transparency. The
resultant conclusion is that there seems to be no identi�able trend due to grade or
calendar year.
The output of the �tted linear regression of LL versus LR+LS+LW is given in
Table 2.4.5. We see that both the intercept and slope parameters are statistically
signi�cant at the 5% signi�cance level. We then �t LL against each of the sub-
aspects separately (Table 2.4.6), and both the intercept and slope parameters are
signi�cant. From the similar magnitudes of the slope coe�cients of LR, LS and LW,
we cannot say that the discrepancy in Literacy Flag 1 is caused by the scores in any
one of the sub-aspects, LR, LS or LW.
Table 2.4.5: Linear regression output from LL vs (LR+LS+LW) for Literacy Flag1 tests
Estimate Std. Error t-value P -valueIntercept 0.3347 0.0467 7.17 < 2.2e-16LR + LS + LW 1.1236 0.0010 1 170.60 < 2.2e-16
Table 2.4.6: Linear regression output from LL vs LR, LS and LW individually forLiteracy Flag 1 tests
Estimate Std. Error t-value P -valueIntercept 0.3501 0.0471 7.43 < 2.2e-16LR 1.1265 0.0023 499.39 < 2.2e-16LS 1.1288 0.0043 260.02 < 2.2e-16LW 1.1102 0.0043 259.62 < 2.2e-16
Chapter 2. Data Analysis 59
20
40
60
80
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
● ●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
● ●
●●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●● ●
●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
● ● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
● ●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
● ●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●● ●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●●
●
●
●●
●
●
●
● ●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
● ●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
● ●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
● ●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
● ●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
● ●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
● ●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
● ●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
● ●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
20 40 60 80 100
Literacy Flag 1
LL
LR
+ L
S +
LW Grade
●
●
●
3
5
7
Figure 2.4.5: Plot of LR+LS+LW versus LL for Literacy Flag 1 tests (coloursrepresent Grades 3, 5 and 7).
60 2.4. Cleaning the Data: Forensic Statistics
20
40
60
80
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
● ●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
● ●
●●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●● ●
●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
● ● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
● ●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
● ●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●● ●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●●
●
●
●●
●
●
●
● ●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
● ●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
● ●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
● ●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
● ●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
● ●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
● ●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
● ●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
● ●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
20 40 60 80 100
Literacy Flag 1
LL
LR
+ L
S +
LW
Year●
●
2003
2004
Figure 2.4.6: Plot of LR+LS+LW versus LL for Literacy Flag 1 tests (coloursrepresent calendar years 2003 and 2004).
Chapter 2. Data Analysis 61
2.4.4 Inference of Data
Based on the �ndings in Section 2.4.3, we can conclude that participants with Liter-
acy Flag 3 should have included their LW score and change the LL scores to re�ect
the addition of the LW score.
For participants with multiple missing aspect scores, no imputation can be made.
However, a single missing score for a participant can be calculated and deduced from
the other corresponding raw scores, with the exception of participants who have a
missing LW score in 2001 or 2002. It is not possible to infer LW from LL in 2001
or 2002, due to the possibility of an inaccurate LL score under Literacy Flag 3. All
changes are made to the raw scores only.
2.5 Bivariate Statistics
We now consider the student and school covariates and how they a�ect the response
variables, which are the raw and Rasch scores. To assess the relationship of the
54 explanatory variables to the scores, we consider the categorical and quantitative
variables separately and apply di�erent methods.
2.5.1 Categorical Variables
In order to determine whether the mean Rasch scores are signi�cantly di�erent
within each categorical variable, a simple linear regression is �t to the response
variable (LL Rasch or NN Rasch) against the categorical variable (isolation,
spatial_ar, staff_metr, atsi, lbote, status, gender, aboriginal, disability,
school_car, occupation, school_edu, non_school, p_g_gender, p_g_nesb, nesb_code,
home_langu or visa_sub_c). All empty and �Inconsistent� entries are removed, and
the baseline group is taken to be the �rst alphabetical or numerical group for each
categorical variable.
To test whether the linear relationship between the categorical variables and LL
Rasch or NN Rasch is statistically signi�cant, analysis of variance (ANOVA) is ap-
plied to each categorical variable. The result is that p_g_nesb, the variable repre-
senting information on parental/guardian non-English speaking background, is not
62 2.5. Bivariate Statistics
signi�cant for both LL and NN Rasch, and the variable visa_sub_c is not found to
have a signi�cant linear relationship with NN Rasch (Tables 2.5.1, 2.5.2 & 2.5.3). All
other variables have an ANOVA P -value less than 0.05 and are signi�cant predictors
of LL Rasch and NN Rasch at the 5% signi�cance level.
Table 2.5.1: ANOVA for LL Rasch against p_g_nesb
Df Sum Sq Mean Sq F -value P -valuep_g_nesb 1 193.31 193.31 3.29 0.0698Residuals 39 633 2 330 172.15 58.79
Table 2.5.2: ANOVA for NN Rasch against p_g_nesb
Df Sum Sq Mean Sq F -value P -valuep_g_nesb 1 178.38 178.38 2.19 0.1387Residuals 39 698 3 230 177.28 81.37
Table 2.5.3: ANOVA for NN Rasch against visa_sub_c
Df Sum Sq Mean Sq F -value P -valuevisa_sub_c 6 238.33 39.72 1.28 0.4544Residuals 3 93.26 31.09
A χ2 test for the independence of the categorical variables is applied to each pair of
variables, and the corresponding P -values give the statistical conclusion to whether
the variables are independent or not. From a heatmap of the P -values (Figure
2.5.1), we see that at a 5% level of signi�cance, many of the variables are correlated
with each other. The shade of the colour indicates the magnitude of the P -value,
with dark purple corresponding to the P -values which are very small and close to
zero and the lightest shade of blue corresponding to P -values which are closer to
the maximum probability of one. The expanse of dark purple indicates that there
are very few pairs of uncorrelated variables - gender is independent of school_edu,
occupation and non_school; procyear is independent of isolation, staff_metr
and spatial_ar; while gradedyear is independent of staff_metr and spatial_ar.
A complete list of all the categorical variables along with their associated P -value
from a linear regression with LL Rasch or NN Rasch and the maximum di�erence
Chapter 2. Data Analysis 63
gend
er
scho
ol_
edu
occu
pati
on
non_
scho
ol
proc
year
hom
e_la
ngu
nesb
_co
de
p_g_
nesb
p_g_
gend
er
scho
ol_
car
disa
bilit
y
abor
igin
al
stat
us
atsi
lbot
e
isol
atio
n
grad
edye
ar
staf
f_m
etr
spat
ial_
ar
spatial_ar
staff_metr
gradedyear
isolation
lbote
atsi
status
aboriginal
disability
school_car
p_g_gender
p_g_nesb
nesb_code
home_langu
procyear
non_school
occupation
school_edu
gender
0 0 0 0 0.61 0 0 0 0 0 0 0 0 0 0 0 0.12 0 0
0 0 0 0 0.32 0 0 0 0 0 0 0 0 0 0 0 0.32 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.02 0 0.32 0.12
0 0 0 0 0.15 0 0 0 0 0 0 0 0 0 0 0 0.02 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.15 0 0.32 0.61
0.77 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0.41 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0.27 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0.27 0.41 0.77 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0.4Value
Color Key
Figure 2.5.1: Heatmap of the P -values from the χ2 test for independence betweenthe categorical explanatory variables.
64 2.5. Bivariate Statistics
in the group means is given in Table 2.5.4. From this table, as before, we see that
only p_g_nesb is not signi�cant at the 5% level of signi�cance (P -value of 0.07 for
LL Rasch and 0.139 for NN Rasch). All other variables seem to be signi�cant or
in�uential in explaining the variance in the Rasch scores according to a linear model.
Looking at the Range of Means column in Table 2.5.4, the large di�erence in the
group means for procyear (11.49 for LL Rasch and 15.78 for NN Rasch) stand out as
surprising, since we would expect Rasch scaling to have resulted in very similar mean
scores across the calendar years 1997 to 2005. Boxplots of the distribution of the LL
Rasch and NN Rasch scores for all categories and variables are used to investigate
and are given in Appendix B, but we illustrate some points and observations with
selected plots. (Note that the na category represents missing data for the categorical
variable.)
Chapter 2. Data Analysis 65
Table 2.5.4: P -value and largest di�erence in group means for each of the categoricalvariables against LL Rasch and NN Rasch (ordered according to the maximumdi�erence in the group means). The extra value in brackets for procyear signify thetrimmed range of means due to small sample sizes in the years 1997-1999 and 2005.
Variable Response P -value Range of meansisolation LL Rasch < 10−16 14.24gradedyear LL Rasch < 10−16 12.24procyear LL Rasch < 10−16 11.49
(7.57)nesb_code LL Rasch < 10−16 9.12disability LL Rasch < 10−16 8.86status LL Rasch < 10−16 8.46atsi LL Rasch < 10−16 6.23aboriginal LL Rasch < 10−16 6.20school_edu LL Rasch < 10−16 5.32spatial_ar LL Rasch < 10−16 5.05non_school LL Rasch < 10−16 4.61occupation LL Rasch < 10−16 4.42school_car LL Rasch < 10−16 4.01gender LL Rasch < 10−16 2.05lbote LL Rasch < 10−16 1.70staff_metr LL Rasch < 10−16 1.39home_langu LL Rasch < 10−16 1.28p_g_gender LL Rasch 4.78e-04 0.36p_g_nesb LL Rasch 0.0698 0.25isolation NN Rasch < 10−16 16.79gradedyear NN Rasch < 10−16 15.80procyear NN Rasch < 10−16 15.78
(8.78)status NN Rasch < 10−16 12.96nesb_code NN Rasch < 10−16 10.97disability NN Rasch < 10−16 9.52atsi NN Rasch < 10−16 7.52aboriginal NN Rasch < 10−16 7.24spatial_ar NN Rasch < 10−16 5.78school_edu NN Rasch < 10−16 5.17school_car NN Rasch < 10−16 4.94occupation NN Rasch < 10−16 4.90non_school NN Rasch < 10−16 4.84lbote NN Rasch < 10−16 2.22home_langu NN Rasch < 10−16 1.64staff_metr NN Rasch < 10−16 1.04gender NN Rasch < 10−16 0.95p_g_gender NN Rasch 1.65e-04 0.46p_g_nesb NN Rasch 0.139 0.24
66 2.5. Bivariate Statistics
Figure 2.5.3 shows that the large range of means is dependent on the sample size of
results. The years with a reasonable sample size are from 2000 to 2004, and so we
report the trimmed range of means to be 7.57, the di�erence between the mean LL
Rasch score in 2004 and 2000. From this plot, we also observe that in the years 2000
to 2004, there is a trend of increasing mean LL Rasch scores. It could be supposed,
just from this observation and plot, that the increasing mean LL Rasch scores is
due to an improving education system over time. However, these samples include
participants from all grades (3, 5 and 7) in a given calendar year. We know that due
to test design, it is expected that participants in higher grades will have a higher
mean Rasch score. So including participants in all grades will increase the mean
Rasch score in the later years, as there are more recorded Grade 5 and 7 students
in 2002, 2003 and 2004 (Section 2.3.2). When Figure 2.5.3 is divided into grades
(Figure 2.5.6), we observe that the range of trimmed means is 1.6 Rasch marks in
Grade 3 and 0.81 Rasch marks in Grade 5. We conclude then that the increase in
students in Grades 5 and 7 and the comparison of students at a lower grade with
students at a higher grade is the cause for the observed increase in mean LL Rasch
scores. A similar observation and explanation applies to NN Rasch scores as well.
Figure 2.5.4 con�rms that there is a di�erence of approximately six Rasch marks
on average between students separated by two years of education. We take this as
a benchmark to compare all di�erences in mean Rasch scores - six Rasch marks is
equivalent to, or worth, two years of education. Using this, we can then say that
having a disability can be equivalent to almost three years of education (Figure
2.5.2).
Looking at the boxplots of gender (Figure 2.5.5), we observe that males have a
higher NN Rasch score on average compared to females (53.7 compared to 52.75),
but females perform better at Literacy on average (52.75 compared to 50.7). The
observed di�erence in Numeracy between genders is less than the observed di�erence
in Literacy, and it would appear that the di�erence in Numeracy is worth approxi-
mately 4 months of education while the di�erence in Literacy is worth approximately
8 months of education.
It is also important to note that the extent of the missing data in each of these
variables and �gures (Figures 2.5.2 to 2.5.6), can be substantial - for example, ap-
Chapter 2. Data Analysis 67
proximately a third of the students have `unknown' gender. This could bias the
estimated di�erence between the categories of each variable.
●
●
●
●
●
●●
●
●
●
●●●●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●●●●●
●●
●
●
●●●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●●●●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●●●●●
●●
●●●
●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●●●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●●●●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●●
●
●
●
●●●●
●
●●●
●
●
●●●
●●
●
●
●
●●
●●●
●
●●
●
●
●●
●
●●
●
●
●
●●●●
●
●
●●
●
●●
●
●●●
●
●
●
●
●●
37236 2401 20232
52.24 43.38 51.94
disability
20
40
60
80
100
N Y NACategories
LL
Ras
ch
Figure 2.5.2: Boxplot of LL Rasch against disability (sample size and mean aregiven above each boxplot).
68 2.5. Bivariate Statistics
●●
●
●●
●●
●
●●●●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●●
●●●
●
●
●●●
●
●
●
●●●
●●
●
●●
●
●●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●●●
●
●
●●
●
●●
●●●●●●●
●
●
●
●●●
●
●
●
●●
●●
●●
●●●●●●●●●●
●
●●●●●
●
●
●
●
●
●●
●
●●●●
●●
●●●●●●
●
●
●
●
●●●
●
●
●●●●
●
●●●
●
●●●●
●●●●●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●●●
●●
●●●●
●
●●
●
●●
●
●
●●●
●
●
●
●
●
●●
●
●●●●
●
●
●●●●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●●●●●●●
●●
●●
●
●●
●
●
●
●
●●●●●●●
●●●
●
●●
●
●●●●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●●●
●●
●●●●●
●
●
●
●
●●
●●●
●
●
●●●
●
●●●
●
●
●●●●●●●
●●●
●
●
●●
●
●
●
●
●●
●●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●
●
●●
●
●
●●●●●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
7 8 68 8308 8259 17546 17819 7845 9
45.7 46.8 45.4 48.27 49.16 52.52 52.15 55.84 56.89
procyear
20
40
60
80
100
1997 1998 1999 2000 2001 2002 2003 2004 2005Categories
LL
Ras
ch
Figure 2.5.3: Boxplot of LL Rasch against procyear (sample size and mean aregiven above each boxplot).
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●
●●
●
●●
●●
●
●●●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●●●●●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●●●●
●●
●
●
●
●●●●●
●
●
●●●
●
●●
●●
●●
●
●
●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●●
●
●
●
●●
●●●●
●
●
●
●
●
●
●
●●●●
●●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●●●●●
●
●●
●
●●●
●●●
●
●●
●●
●●
●
●
●
●
●●
●
●
●
●
●●
●●
●
●
●●
●●●●
●
●
●
●
●
●●
●
●●●
●
●●
●
●
●
●
●
●●●●●
●
●●
●●
●
●●
●
●●●●
●
●●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●●
●
●●
●
●
●
●●●
●
●
●
●
●
●●
●
●●
●●●●●
●
●●
●
●
●●●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●●
●●
●
●●
●
●●●●
●●
●
●●●●
●
●●
●
●
●
●●●●●
●
●
●●●
●
●
●●●
●
●
●
●●●
●
●
●
●
●
●●
●
●●
●●
●
●
●●
●
●
●●
●●
●
●●●●●
●
●●
●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●●
●●
●
●
●
●●●●●●
●
●●●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●●●●
●
●
●
●
●
●
36837 20477 2555
48.94 55.72 61.18
gradedyear
20
40
60
80
100
3 5 7Categories
LL
Ras
ch
Figure 2.5.4: Boxplot of LL Rasch against gradedyear (sample size and mean aregiven above each boxplot).
Chapter 2. Data Analysis 69
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●●●
●●
●●●
●
●
●
●
●
●
●●
●
●
●
●●●
●●
●●●
●●
●●
●
●
●
●●●
●
●●
●
●
●●
●●
●
●●
●
●●
●●
●
●
●●
●●
●
●
●
●
●
●●●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●●
●●
●
●
●
●●
●
●
●
●
●●
●
●●●●●
●●
●
●
●
●
●●●●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●●
●
●
●
●
●●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●●●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●●● ●
●●
●
●●●●●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●●
●
●
●
●●●●
●
●●●
●
●
●●●
●●
●
●
●
●●
●●●
●
●●
●
●
●●
●
●●
●
●
●
●●●●
●
●
●●
●
●●
●
●●●
●
●
●
●
●●
19400 20237 20232
52.75 50.7 51.94
gender
20
40
60
80
100
F M NACategories
LL
Ras
ch
●
●
●●
●●
●
●
●●●
●
●
●●
●
●
●
●●
●
●
●
●●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●●
●●●
●
●
●●
●●
●
●
●
●
●
●●●
●●●
●
●
●●
●
●
●
●
●●●
●
●
●
●
●●●
●
●●
●
●
●
●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●
●
●●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●●
●●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●●
●
●
●●
●●
●●●●●●
●
●●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●●●
●●
●●
●●
●
●●
●
●
●●●●
●
●●
●
●
●
●
●
●
●
●
●●●●
●●●
●
●
●
●
●●●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●●●●
●
●
●●
●
●●●●●●●
●
●●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●
●
●
●
●●
●●
●
●
●
●●
●●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●●
●
●●●●●●
●
●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●●
●
19315 20388 20375
52.75 53.7 53.22
gender
20
40
60
80
100
F M NACategories
NN
Ras
ch
Figure 2.5.5: Boxplots of LL Rasch and NN Rasch for gender (sample size and meanare given above each boxplot).
70 2.5. Bivariate Statistics
●●● ●● ●●● ●●● ●●●● ● ●●● ●● ●●●● ●● ●● ● ● ●●●● ●● ●●●● ● ●●● ●● ● ●●●● ●● ● ●● ●● ●●●● ●● ●● ●● ● ●● ●●● ●●● ●●● ●●● ● ●● ●●● ●● ●● ●●●●● ● ● ●● ●● ●● ● ●●
●●●● ●● ●●● ●●●● ●● ●● ●● ● ●●● ●●●● ●●● ● ●●●●● ●●●●● ●●●●●●●● ●● ●●● ●● ●●● ●●●● ●●●●●●●●●●● ●●●●● ●●● ● ●●● ●●●●●●● ●●●●●●● ●
● ●●● ●● ●● ●● ●● ● ●● ●●● ●● ● ●● ●●●●● ●●●● ●● ●●● ●●●● ●● ●●●● ●●● ●● ●●●●● ●● ●●● ●●●●● ● ●●●● ●●● ●●● ●●●●●● ●●●● ●● ● ●●● ●● ● ●●● ●● ● ●● ● ●● ● ●●
● ● ●●● ●● ●●● ●● ●● ● ●●●●●●● ●● ●●●● ●● ●●
67
6383
0282
0786
7789
1226
576
44.2
646
.31
45.0
248
.27
49.1
549
.148
.99
49.8
752
.33
●
●●●●● ●● ●● ● ●● ●●● ● ●● ●● ●● ●● ●●● ●●●● ● ● ●● ●●● ● ●● ●●● ●●● ●●● ● ●●●● ●●● ●●● ●●
●● ●● ●●●● ●●●●● ●● ●● ●● ●●●● ●●● ●●● ● ●● ●● ● ●●●● ●●● ● ●●●● ● ● ●● ●●●●● ● ●● ●●● ●● ●● ● ● ●●● ●● ●● ● ●● ●●●● ●●● ● ● ●● ●● ●● ●●●● ●●● ●●●● ● ●●●●● ● ●● ●●●● ● ●●●● ● ●● ●●● ●● ● ● ●●● ●
● ●●● ● ●● ●●●● ●●●● ● ●●●● ●● ●● ●●● ● ●● ●●● ● ●● ●●
11
56
4988
6388
8926
621
54.3
950
.25
50.2
552
.86
52.0
855
.87
55.3
156
.68
62.0
4
●●
●●
● ●● ●● ●● ●●●●●● ● ●● ●● ●● ●●●● ● ● ●● ● ● ●●
36
1825
262
53.6
160
.58
57.2
361
.22
68
35
7
20406080100
1997
1998
1999
2000
2001
2002
2003
2004
2005
1997
1998
1999
2000
2001
2002
2003
2004
2005
1997
1998
1999
2000
2001
2002
2003
2004
2005
Cat
egor
ies
LL Rasch
Figure
2.5.6:
BoxplotofLLRasch
against
procyearsplitinto
grades
(sam
plesize
andmeanaregivenaboveeach
boxplot).
Chapter 2. Data Analysis 71
2.5.2 Quantitative Variables
To test the association between the quantitative variables, a Pearson's correlation
test is used between each variable and LL Rasch and NN Rasch separately. All
variables but one have a P -value less than 0.05 for the correlation tests - the stand-
out variable is NN Rasch ∼ x005_tmob (P -value of 0.509).
To visually assess the strength of the correlation between the quantitative explana-
tory variables, Figure 2.5.7 is a heatmap of the correlation values. Most variables
have a positive correlation of varying strength, and the 2004, 2005 and 2006 values
for certain variables form distinguishable 3x3 blocks in the heatmap.
A complete list of all the quantitative variables along with their associated P -value
and R2 value from the linear regression with LL Rasch or NN Rasch, ordered by R2
values, is given in Table 2.5.5. The P -values are very small and indicate that there
is an association between the explanatory variables. However, the R2 values are
also very small and seem to indicate that only a small proportion - less than 5% - of
the variance in the response variables can be explained by a linear relationship with
the quantitative variables. There is a very large sample size of at least 47 500 data
points for each of the explanatory variables, possibly causing the small P -values,
but we also note that the R2 value of the insigni�cant variable x005_tmob is less
than all the other R2 values by at least two orders of magnitude.
The overall conclusion from the bivariate statistics is that at this minimal level of
statistical analysis and testing, most of the explanatory variables are deemed to be
associated with the response variables of LL Rasch and NN Rasch scores.
72 2.5. Bivariate Statistics
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●●●●
●
●
●●●●
●
●
●
●●●
●
●
●
●
●●
●●●
●
●
●
●●●●
●
●
●
●●●
●
●
●
●
●●
●●●
●
●
●
●●●
●
●
●
●
●
●
●
●●●●●
●●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●●●●●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●●●●●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●●●●●●
●
●●
●
●●
●
●
●
●●●●
●
●
●●●●●●●
●
●
●●●●
●
●
●
●●●●
●
●
●●●●●●●
●
●
●●●●
●
●
●
●●●●
●
●
●●●●●●●
●
●
●●●
●
●
●
●
●
●
●
●●●●●
●●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●●●
●
●
●
●●●●
●
●
●
●●●
●
●
●
●
●●
●●●●
●
●
●●●
●
●
●
●
●●●
●
●
●
●
●●
●●●●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● −1
−0.8
−0.6
−0.4
−0.2
0
0.2
0.4
0.6
0.8
1
gpok
mx0
06_
enr
x005
_en
rx0
04_
enr
x006
_ab
sx0
05_
abs
x004
_ab
sx0
06_
beh
x005
_be
hx0
04_
beh
x006
_sc
rdx0
05_
scrd
x004
_sc
rdx0
06_
mob
x005
_m
obx0
04_
mob
x006
_tc
hx0
05_
tch
x004
_tc
hx0
06_
tmob
x005
_tm
obx0
04_
tmob
gpokmx006_enrx005_enrx004_enrx006_absx005_absx004_absx006_behx005_behx004_behx006_scrdx005_scrdx004_scrdx006_mobx005_mobx004_mobx006_tchx005_tchx004_tch
x006_tmobx005_tmobx004_tmob
Figure 2.5.7: Heatmap of the correlation between the quantitative explanatory vari-ables.
Chapter 2. Data Analysis 73
Table 2.5.5: P -value and R2 value for each of the quantitative variables against LLRasch and NN Rasch (ordered according to R2 values)
Variable Response P -value R2 (%)x004_abs LL Rasch < 10−16 4.63x004_mob LL Rasch < 10−16 4.63x006_abs LL Rasch < 10−16 4.38x006_mob LL Rasch < 10−16 4.38x005_abs LL Rasch < 10−16 4.19x005_mob LL Rasch < 10−16 4.19x006_scrd LL Rasch < 10−16 1.23x004_scrd LL Rasch < 10−16 1.13x006_enr LL Rasch < 10−16 1.01x005_scrd LL Rasch < 10−16 0.96x005_enr LL Rasch < 10−16 0.93x006_beh LL Rasch < 10−16 0.89x004_enr LL Rasch < 10−16 0.85gpokm LL Rasch < 10−16 0.81x004_beh LL Rasch < 10−16 0.60x005_beh LL Rasch < 10−16 0.23x004_tmob LL Rasch < 10−16 0.23x006_tmob LL Rasch < 10−16 0.22x006_tch LL Rasch < 10−16 0.21x005_tch LL Rasch < 10−16 0.19x004_tch LL Rasch < 10−16 0.18x005_tmob LL Rasch 0.00842 0.0133x004_abs NN Rasch < 10−16 4.6x004_mob NN Rasch < 10−16 4.61x005_abs NN Rasch < 10−16 4.25x005_mob NN Rasch < 10−16 4.25x006_abs NN Rasch < 10−16 4.13x006_mob NN Rasch < 10−16 4.13x006_scrd NN Rasch < 10−16 1.62x004_scrd NN Rasch < 10−16 1.51x005_scrd NN Rasch < 10−16 1.31x006_beh NN Rasch < 10−16 0.84x004_beh NN Rasch < 10−16 0.71x006_enr NN Rasch < 10−16 0.58gpokm NN Rasch < 10−16 0.57x005_enr NN Rasch < 10−16 0.52x004_enr NN Rasch < 10−16 0.46x005_beh NN Rasch < 10−16 0.28x006_tmob NN Rasch < 10−16 0.25x004_tmob NN Rasch < 10−16 0.21x006_tch NN Rasch 3.93e-13 < 0.1x005_tch NN Rasch 4.21e-11 < 0.1x004_tch NN Rasch 2.55e-10 < 0.1x005_tmob NN Rasch 0.509 8.32e-04
74 2.5. Bivariate Statistics
2.5.3 Principal Component Analysis (PCA)
Background Theory
Principal component analysis (PCA) is a mathematical procedure that uses an or-
thogonal transformation to convert a set of observations of possibly correlated vari-
ables into a set of values of linearly uncorrelated variables called principal compo-
nents. The number of principal components is less than or equal to the number
of original variables. This transformation is de�ned in such a way that the �rst
principal component accounts for as much of the variability in the data as possible,
and each succeeding component in turn has the highest variance possible under the
constraint that it be orthogonal to, and uncorrelated with, the preceding compo-
nents. PCA is a form of multivariate analysis, and its application can often reveal
the internal structure of the data in a way which best explains the variance in the
data. PCA can simplify high-dimensional data to one with less variables by using
the �rst few principal components, so that the dimensionality of the transformed
data is reduced.
Results
The results from the principal component analysis on the continuous explanatory
variables are given in Table 2.5.6. We see that the �rst three components explain
the majority of the variance in the covariates, as the �rst component accounts for
approximately 72.5% and by the third component, a total of 98.5% of the variation.
This result can also be seen graphically (Figure 2.5.8).
The �rst three principal components are primarily constructed from variables per-
taining to enrolment, the number of School Cards issued and the school's distance
from the Adelaide General Post O�ce. Since almost all of the variance is explained
by the �rst three principal components, we look to see if any clusters of points can
be identi�ed in a 3-dimensional scatter plot of the scores of the �rst three prin-
cipal components (Figure 2.5.9). No obvious clusters of data points are observed,
and we conclude that from the principal component analysis, we cannot reduce the
dimensionality of the data to linear combinations of fewer meaningful variables.
Chapter 2. Data Analysis 75
Figure 2.5.8: Barplot of the variances explained by the principal components.
Figure 2.5.9: 3D scatter plot of the scores of the �rst three principal components.
76 2.5. Bivariate Statistics
Table 2.5.6: The standard deviation, proportion of variance explained and the cu-mulative proportion for each of the principal components
Principal Component Standard Dev Proportion Cumulative Proportion1 385.6 0.725 0.7252 176.5 0.152 0.8773 148.7 0.108 0.9854 37.6 0.007 0.9925 26.7 0.003 0.9956 19.2 0.002 0.9977 16.1 0.001 0.9988 10.8 5.7e-4 0.99939 6.8 2.2e-4 0.999510 6.2 2e-4 0.999711 4.9 1e-4 0.999812 3.6 6.3e-5 0.999913 1.9 1.7e-5 114 1.5 1.1e-5 115 1.2 7.5e-6 116 1.1 5.6e-5 117 0.1 5.1e-8 118 0.09 4e-8 119 0.07 2.6e-8 120 5.4e-8 1.4e-20 121 0 0 122 0 0 1
Chapter 2. Data Analysis 77
2.6 Final Data Sets
Having started with the original Basic Skills data subset and through the use of
forensic statistics, the data has been cleaned. Our �ndings and actions can be
summarised in the speci�ed order:
1. remove participants with more than one missing aspect score for either a Lit-
eracy or Numeracy participant, as inadequate information is available to infer
these missing scores,
2. remove the Numeracy scores of participants with Numeracy Flags 1 and 2,
attributing these to random error,
3. remove the Literacy scores of participants with Literacy Flag 2, attributing
these to random error,
4. remove the Literacy scores of participants with Literacy Flag 1 as we found
no explanation, and there is no method to rectify this discrepancy,
5. add the LW score to the LL scores of participants with Literacy Flag 3 as the
nature of the �ag was explained (Section 2.4.3), and
6. for participants who have one missing score which is not LW in 2001 or 2002,
we infer or impute the missing scores (note that we cannot infer LW from LL
in 2001 or 2002 due to the possibility of an incorrect LL score under Literacy
Flag 3).
Figure 2.6.1 diagramatically illustrates the above process and the data sets.
The steps above discuss the corrections and imputations of the raw scores. However,
the Rasch scores are the main response variables and picked to be so because Rasch
scaling enables results to be compared across grades and years. Consider the Literacy
scores. Some Rasch scores can be corrected by �rstly imputing the single missing
raw scores and then attributing the Rasch score based on the Rasch score of students
with the same raw score. Unfortunately, an observation of the recorded scores is
that although we can correct for Literacy Flag 3 and change the corresponding LL
scores, the Rasch LL scores have been calculated on the �incorrect� LL scores, rather
78 2.6. Final Data Sets
than on the correct individual aspect scores for LR, LS and LW. This is veri�ed by
looking at the LL Rasch scores for all participants in the same grade and cohort
who received the same �incorrect� LL score - they all have the same LL Rasch
score independent of the di�ering raw scores for LR, LS and LW. So the imputed
LL Rasch scores would be as inaccurate as the recorded Rasch scores. In fact, we
cannot impute any LL Rasch scores because their calculation used original data,
and all LL Rasch scores are �tted to �incorrect� or inaccurate (apparently) data.
As a result, the Numeracy scores, after participants with Numeracy Flags 1 and
2 have been removed, is the most reliable data, and so we choose to restrict all
modelling to the Numeracy data.
After this investigation of the data, we would have liked to pursue various data
questions further with the data source, but since that was not possible, the data was
cleaned and corrected as much as possible in preparation for statistical modelling.
Chapter 2. Data Analysis 79
Redu
ced
Data
Data
Set
1Da
ta S
et 2
Orig
inal
Rem
ove
Mul
tiple
M
issin
g As
pect
s
Rem
ove
Num
erac
y Fl
ags
1 &
2
Rem
ove
Lite
racy
Flag
s 1
& 2
Corre
ct L
itera
cy F
lag
3Im
pute
Miss
ing
Mar
ks
138
688
50 8
8467
392
100
144
Figure
2.6.1:
Flowchartof
datasets.
Chapter 3
Initial Model Selection
From the principal component analysis in Chapter 2, we know that the explanatory
variables are highly correlated with each other. Unfortunately, as highlighted in
Chapter 2, we were not able to contact those responsible for collecting the data to
discuss which of the variables could be dropped based on logical reasons. With this
knowledge in the background, we will perform a simple multiple linear regression as
an initial primary assessment of the data. This will provide meaningful insight into
the physical variables themselves and will allow the data to speak for itself.
There is data recorded for 54 explanatory variables. However, we want to be able to
select the variables which are only statistically signi�cant and important in terms
of their relationship with the response variable, NN Rasch scores. We do this by
looking for missing data and whether the data values for each variable make logical
sense, before using standard statistical model selection techniques. This will then
result in the simplest and most relevant model for the Rasch scores.
3.1 Manual Reduction of Data and Predictor Vari-
ables
Looking at the data itself, certain variables and data provide either insu�cient
information, irrelevant information or exhibit discrepancies. Hence, we can remove
them from the list of possible regression variables. Each of the following paragraphs
81
82 3.1. Manual Reduction of Data and Predictor Variables
give reasons for the exclusion of variables and data.
Participants with Numeracy Flags 1 and 2 (Section 2.4.3) are removed from the
data set and similarly, participants with missing NN scores are also removed, as the
response variable is either incorrect or not recorded.
Categorical Variables
The categorical cap (Country Areas Program) variable has only one de�ned group,
`Y' (yes), and the non-entries could represent either `N' (no) or be missing data
- this information is not known. The variable status is also removed due to the
ill-de�ned categories for the status of students in the education system (Appendix
A.3). Students' date of birth (date_of_bi) is also deemed to give no further useful
information. The visa_sub_c variable is removed because the majority of values
are missing.
Certain variables on parental and cultural background are removed because of the
presence of missing data, for example, 54 217 out of 60 039 entries for nesb_code are
missing. For this reason, cultural_b, p_g_cultur, p_g_countr and country_of
are also excluded. For the recorded values which we have, these variables identify
the country of origin of a student, however, similar information on nationality can
be gleaned from the p_g_nesb variable which indicates the non-English speaking
background of the principal parent or guardian in the form of a binary variable.
Quantitative Variables
All the year-speci�c 2004-2006 variables are removed because we observed that the
x00y{4,5,6}_abs and x00y{4,5,6}_mob columns are exactly the same for all par-
ticipants. The number of School Cards for a school can also exceed the enrolment
number in a particular year, which does not seem plausible. In addition, a large
proportion of the values are missing for these variables, and data from 2004 to 2006
is not very relevant as we only have substantial numbers of recorded test data in the
years 2000 to 2004.
With the presence of missing data for school variables, schools either have all or no
data for a particular variable. With more complete data, teacher mobility would
Chapter 3. Initial Model Selection 83
be an important variable to include as a predictor in the model as it is politically
important and informative about school demography. There are some schools which
struggle to retain teachers - sometimes due to location or the student population -
and this is re�ected in higher teacher mobility. However, it is important to note that
teacher mobility is not a consistent variable over time. There is a lack of correlation
between teacher mobility in 2004, 2005 and 2006 (Figure 2.5.7), and so we cannot
use data for one year out of 2004, 2005 or 2006 to predict teacher mobility in another
of those years, let alone predict and extrapolate for teacher mobility in previous or
subsequent years.
Missing Data
Missing data is observed in school and student variables. Schools 19, 20, 21, 22, 23,
122 and 174 have no data for at least half of the variables and are not considered
in the model because of lack of data. In addition, 20 350 participants have missing
data for a majority of the variables - primarily in the student covariates - and are
removed, further reducing the �nal data set.
The remaining data set has 39 683 rows and 20 columns containing NN Rasch and
19 covariates - schoolno, procyear, gradedyear, gpokm, isolation, spatial_ar,
staff_metr, atsi, lbote, gender, aboriginal, disability, school_car, occupation,
school_edu, non_school, p_g_gender, p_g_nesb and home_langu.
3.2 Statistical Reduction of Predictor Variables
Having manually pruned the data set, we wish to �nd the statistically simplest model
which will best explain the relationship between the student and school variables
and the NN Rasch scores. We pursue two alternative methods - �rstly looking at
signi�cant predictor variables in a simple linear regression model and then using a
step-wise selection process based on Akaike's Information Criterion.
More complicated hierarchical modelling and model selection will be discussed in
Chapter 4 and Chapter 5.
84 3.2. Statistical Reduction of Predictor Variables
3.2.1 Theory
Linear Regression
Suppose we have data
(x1, y1), (x2, y2), . . . , (xn, yn),
where x represents the vector of k predictor variables. The classical linear regression
model can be written as
yi = Xiβ + εi
= β1Xi1 + . . . βkXik + εi for i = 1, . . . , n,
where the error εi is independently distributed εi ∼ N(0, σ2) and β is the vector of
regression coe�cients. The linear regression model in matrix form is
yi ∼ N(Xiβ, σ2) for i = 1, . . . , n,
or
y ∼ Nn(Xβ, σ2I)
where y is a vector of length n, X is a n × k matrix of predictors, β is a column
vector of length k and I is the n× n identity matrix.
It is well-known statistical theory that the least squares estimate of β is
β = (XTX)−1XTy.
Model Selection
As before, suppose we have data
(x1, y1), (x2, y2), . . . , (xn, yn)
and we wish to choose a suitable model which is parsimonious and well-�tting.
Ideally, we want to choose the smallest well-�tting model and to do this, we need to
balance the number of parameters to avoid unnecessarily increasing the complexity
of the model but at the same time, ensuring the model �ts the data. Exclusion of
important terms clearly leads to an incorrect model, which can then cause misleading
conclusions. On the other hand, including unnecessary terms diminishes the value
of the model as a simpli�cation of the data and can lead to over-�tting.
In linear regression and generalised linear regression models, the model selection
Chapter 3. Initial Model Selection 85
problem is often stated as a variable selection problem. We have a response variable
y and a set of predictors x = {x1, . . . , xn}. We wish to divide x into two groups,
x = (xA,xI), the active and inactive predictors, such that the distribution of y|xAis the same as the distribution of y|(xA,xI). Hence, all the information about y is
explained by the active predictors.
General model selection is concerned with not just �nding the active predictors
xA but also building the model itself. This includes de�ning the model predictors
and selecting distributions and other modelling assumptions. Initially, we address
limited variable selection by assuming the linear model and �nd the active variables
xA or, equivalently, delete the inactive variables xI . Subsets of the full model are
taken and for each, some chosen information criterion of model quality is calculated
and optimized.
Another important criterion for model selection is the satisfaction of the principle
of marginality. The principle of marginality states that when an interaction term is
included in a model, all implied lower order interactions and main e�ects must also be
included to be a sensible model. In general, it is wrong to test, estimate or interpret
the main e�ects of explanatory variables separate to a signi�cant interaction term or
to model interaction e�ects when main e�ects that are marginal to the interactions
are deleted. While such models are interpretable, they ignore the e�ects of the
marginal main e�ects and lack applicability. This principle of marginality must be
satis�ed in any �nal model as a result of model selection.
3.2.2 Signi�cant Variables in Linear Regression
We start by �tting a linear regression of NN Rasch against the main e�ects and
identify multicollinearity between the covariates of gpokm, isolation, spatial_ar,
school_eduInconsistent and home_languY with the schoolno variable. Multi-
collinearity occurs when two or more predictor variables in a multiple regression
model are highly correlated. This correlation then impacts on the ability to perform
the matrix inversion required for computing regression coe�cients, and parameter es-
timates cannot be reliably computed for highly correlated variables. Multicollinear-
ity is also a problem because it increases the standard errors of the coe�cients and
increased standard errors in turn could result in the false conclusion of the statistical
86 3.2. Statistical Reduction of Predictor Variables
insigni�cance of a variable, when it is actually signi�cant.
In our data, collinearity is explained by the schools partitioning each category
of the school covariates, and as a result, schoolno gives the same information
as all the school covariates combined together. The R function alias expresses
the school covariates as a linear combination of schoolno categories, home_languY
and school_eduInconsistent, and this con�rms multicollinearity. To remove the
collinearity with home_languY, the variable home_langu is removed from the data
set. Data where school_edu is inconsistent is also removed as any conclusions for
this category are not reliable due to the inconsistency of the data.
On this basis, we consider two di�erent models - the school-number model which in-
cludes schoolno but excludes all school covariates, gpokm, isolation, spatial_ar
and staff_metr, and the school-covariates model where the school covariates re-
place schoolno in the model.
School-Number Model
Consider the full main e�ects model with schoolno - this model is henceforth known
as the school-number model.
School-Number Model
NN Rasch = schoolno + procyear + gradedyear + atsi + lbote + gender
+ aboriginal + disability + school_car + occupation + school_edu +
non_school + p_g_gender + p_g_nesb.
The linear regression of schoolno and the other main e�ects gives 10 signi�cant
variables (schoolno, gradedyear, atsi, lbote, gender, disability, school_car,
occupation, school_edu, non_school) and 50 categories which are signi�cant (Ap-
pendix C, Table C.1.1). The non-signi�cant variables are procyear, aboriginal,
p_g_gender and p_g_nesb and would be excluded from the model based on signi�-
cance. After the removal of the non-signi�cant variables, the model which shall be
referred to as the signi�cant-predictors school-number model is
Chapter 3. Initial Model Selection 87
Signi�cant-Predictors School-Number Model
NN Rasch = schoolno + gradedyear + atsi + lbote + gender + disability +
school_car + occupation + school_edu + non_school.
Out of the 400 schools, only 33 schools are signi�cantly di�erent from the inter-
cept school 27 at the 5% signi�cance level. Figure 3.2.1 plots all coe�cients of the
linear regression other than schoolno and the 95% con�dence intervals of the �tted
estimates. All of the regression coe�cients are compared to the baseline category or
intercept of a female student in school 27 and grade 3 in 1998 who does not come
from an Aboriginal, Torres Strait Islander or non-English speaking background, is
not identi�ed as having a disability or School Card and their primary guardian or
parent is female from an English speaking background whose status of occupation,
school education and non-school education is not stated. The signi�cant categories
are coloured blue and the non-signi�cant categories are coloured red in Figure 3.2.1.
This �gure plots all the regression coe�cients on a common scale from which we
can observe the relative e�ect of each of the levels and predictor variables on the
baseline category. Compared to all other parameters, the procyear categories have
wide error bars, and the widest error bars for 1999 and 2005 are due to the lack of
data in those years as discussed in Chapter 2. All other regression coe�cients have
a relatively small standard error compared to those three years. To better see the
majority of points and error bars around the line of no deviation from the intercept,
Figure 3.2.2 is Figure 3.2.1 restricted to the range of -5 to 5.
88 3.2. Statistical Reduction of Predictor Variables
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
Y
1
Inconsistent
Y
M
5
7
1
Inconsistent
5678
12348
M
Y
1999200020012002200320042005
Y
1234
aboriginalatsi
disabilitygender
gradedyearlbote
non_school
occupationp_
g_gender
p_g_
nesbprocyear
school_car
school_edu
−20 −15 −10 −5 0 5 10 15Value
Cat
egor
ies Significant
●
●
No
Yes
Figure 3.2.1: Coe�cient plot for the school-number model. (See text for de�nitionof baseline category. The Inconsistent category of the atsi and lbote variablesrepresents students with identi�ed anomalies in the data.)
Chapter 3. Initial Model Selection 89
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
Y
1
Inconsistent
Y
M
5
7
1
Inconsistent
5678
12348
M
Y
1999200020012002200320042005
Y
1234
aboriginalatsi
disabilitygender
gradedyearlbote
non_school
occupationp_
g_gender
p_g_
nesbprocyear
school_car
school_edu
−5 −4 −3 −2 −1 0 1 2 3 4 5Value
Cat
egor
ies Significant
●
●
No
Yes
Figure 3.2.2: Subset of the coe�cient plot for the school-number model. The x-axisranges from -5 to 5. (See text for de�nition of baseline category. The Inconsis-tent category of the atsi and lbote variables represents students with identi�edanomalies in the data.)
90 3.2. Statistical Reduction of Predictor Variables
School-Covariates Model
We �t the alternative school-covariates model, replacing schoolno with the school
covariates of gpokm, isolation, spatial_ar, staff_metr and school size. School
size is de�ned to be the number of recorded participants for a school in the data set
and is denoted by a di�erent font as it is a deduced variable and not an observed
variable.
School-Covariates Model
NN Rasch= procyear+ gradedyear+ gpokm+ isolation+ spatial_ar staff_metr
+ atsi+ lbote+ gender+ aboriginal+ disability+ school_car+ occupation
+ school_edu + non_school + p_g_gender + p_g_nesb + school size.
The signi�cant variables are gradedyear, gpokm, isolation, spatial_ar, atsi,
lbote, gender, aboriginal, disability, school_car, occupation, school_edu,
non_school and school size (Appendix C, Table C.1.2). The model which contains
only the signi�cant predictor variables is known as the signi�cant-predictors school-
covariates model.
Signi�cant-Predictors School-Covariates Model
NN Rasch = gradedyear + gpokm + isolation + spatial_ar + atsi + lbote +
gender + aboriginal + disability + school_car + occupation + school_edu
+ non_school + school size.
Figure 3.2.3 plots all coe�cients of categorical variables in the linear regression and
the 95% con�dence intervals of the �tted estimates. All of the regression coe�cients
are compared to the baseline category or intercept of a female student in grade 3 in
1998 who does not come from an Aboriginal, Torres Strait Islander or non-English
speaking background, is not identi�ed as having a disability or School Card and their
primary guardian or parent is female from an English speaking background whose
status of occupation, school education and non-school education is not stated. The
characteristics of the reference school is one with an isolation factor of 1, spatial
value of 1.1 (metropolitan) and metro sta� classi�cation. The signi�cant categories
are coloured blue and the non-signi�cant categories are coloured red in Figure 3.2.3.
Chapter 3. Initial Model Selection 91
This �gure plots all the regression coe�cients on a common scale from which we
can observe the relative e�ect of each of the levels and predictor variables on the
baseline category. To better see the majority of points and error bars around the
line of no deviation from the intercept, Figure 3.2.4 is Figure 3.2.3 restricted to the
range of -5 to 5.
In addition to the categorical variables, the regression output for the two continu-
ous predictors, gpokm and school size, are given in Table 3.2.1. Both variables are
signi�cant at the 5% signi�cance level.
Table 3.2.1: Regression output for gpokm and school size from the school-covariatesmodel
Estimate Std. Error t-value P -valuegpokm -0.0095 0.0020 -4.7041 2.6e-06school size 0.0027 0.0006 4.4995 6.9e-06
3.2.3 Simplest Main E�ects Model using stepAIC
We have now established the full main e�ects linear model and observed that certain
variables are not signi�cant. Removing these variables from the model gives us the
simplest model based on the signi�cance of variables as stated in Section 3.2.2 for
both the school-number and school-covariates models.
Step-wise model selection is another method to eliminate the variables which are
not statistically signi�cant and to achieve the simplest model. In general, step-wise
model selection is a combination of forward and backward selection.
Suppose we wish to decrease or minimise a model selection criterion. Under the
forward selection algorithm, the initial model is the null model with no predictors
or the model with all pre-determined necessary predictors included. The algorithm
continues to add predictors until adding another variable increases the criterion of
interest. If a predictor decreases the criterion, it is included in the model, and this
process continues until no further variables are added. Similarly, the backward selec-
tion algorithm starts with the most complicated model which includes all predictors
and continues to remove variables until the criterion of interest is increased by the
92 3.2. Statistical Reduction of Predictor Variables
●
●
●
●
●
●
●
●●
●●
●●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
Y
1
Inconsistent
Y
M
5
7
1.522.533.544.555.566.57
1
Inconsistent
5
6
7
8
12348
M
Y
1999200020012002200320042005
Y
1
2
3
4
2.2.1
2.2.2
3.1
3.2
C
aboriginalatsi
disabilitygender
gradedyearisolation
lbotenon_
schooloccupation
p_g_
genderp_
g_nesb
procyearschool_
carschool_
eduspatial_
arstaff_
metr
−20 −15 −10 −5 0 5 10 15 20Value
Cat
egor
ies Significant
●
●
No
Yes
Figure 3.2.3: Coe�cient plot for the school-covariates model. (See text for de�nitionof baseline category. The Inconsistent category of the atsi and lbote variablesrepresents students with identi�ed anomalies in the data.)
Chapter 3. Initial Model Selection 93
●
●
●
●
●●
●●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
Y
1
Inconsistent
Y
M
5
7
1.522.533.544.555.566.57
1
Inconsistent
5
6
7
8
12348
M
Y
1999200020012002200320042005
Y
1
2
3
4
2.2.1
2.2.2
3.1
3.2
C
aboriginalatsi
disabilitygender
gradedyearisolation
lbotenon_
schooloccupation
p_g_
genderp_
g_nesb
procyearschool_
carschool_
eduspatial_
arstaff_
metr
−5 −4 −3 −2 −1 0 1 2 3 4 5Value
Cat
egor
ies Significant
●
●
No
Yes
Figure 3.2.4: Subset of coe�cient plot for the school-covariates model. The x axisranges from -5 to 5. (See text for de�nition of baseline category. The Inconsis-tent category of the atsi and lbote variables represents students with identi�edanomalies in the data.)
94 3.2. Statistical Reduction of Predictor Variables
removal of a variable. At this point, the model contains only signi�cant variables.
Step-wise model selection is a combination of both forward and backward selection
in which at each step, either the e�ect of the addition of a predictor or the removal of
a predictor on the criterion is considered. This process also applies to the situation
where the model selection criterion is maximised to indicate a �good� model.
We choose to use stepAIC which is a step-wise model selection process based on
minimising Akaike's Information Criterion (AIC). Akaike's Information Criterion
[2] is a measure of the relative goodness of �t of a statistical model. In the general
case, Akaike's Information Criterion is used to compare models of the same type,
for example, regression models. AIC is de�ned to be
AIC = 2k − 2 ln(L) (3.2.1)
where k is the number of parameters in the statistical model and L is the maximised
value of the likelihood function for the estimated model. As with all model selection,
there needs to be a balance between the model �t and a penalty for model complexity
and the number of parameters. This is accounted for in equation (3.2.1) where
D = −2 ln(L) is the deviance and is a measure of model �t and 2k compensates for
the number of estimated parameters. The model with the smallest AIC is chosen as
the best model.
A few important points to consider when using stepAIC and interpreting the results
is that AIC can be applied to compare nested models, and the process of �tting
models is by maximum likelihood. It is also important to note that often there is
no one single best model, and it is wise to consider all models which are within two
units of the minimal AIC [40].
With this statistical theory, the stepAIC function in R is contained in the MASS
package and performs step-wise model selection by AIC. It takes a model object
and uses it as the initial model in the step-wise search. The default direction for
stepAIC is a backward step-wise search.
School-Number Model
From the school-number model, the variables p_g_nesb and p_g_gender are re-
moved by stepAIC to give the simplest-stepAIC school-number model.
Chapter 3. Initial Model Selection 95
Simplest-stepAIC School-Number Model
NN Rasch = schoolno + procyear + gradedyear + atsi + lbote + gender
+ aboriginal + disability + school_car + occupation + school_edu +
non_school.
Table C.1.3 in Appendix C gives the linear regression output, and we can see that the
variables procyear and aboriginal are not signi�cant but are still included. The
model with only signi�cant predictors is the signi�cant-predictors simplest-stepAIC
school-number model.
Signi�cant-Predictors Simplest-stepAIC School-Number Model
NN Rasch = schoolno + gradedyear + atsi + lbote + gender + disability +
school_car + occupation + school_edu + non_school. (3.2.2)
Figure 3.2.5 plots all coe�cients of the simple linear regression model other than
schoolno and the 95% con�dence intervals of the �tted estimates. All of the re-
gression coe�cients are compared to the baseline category or intercept of a female
student in school 27 and grade 3 in 1998 who does not come from an Aboriginal,
Torres Strait Islander or non-English speaking background, is not identi�ed as hav-
ing a disability or School Card and their primary guardian or parent is female from
an English speaking background whose status of occupation, school education and
non-school education is not stated. The signi�cant categories are coloured blue and
the non-signi�cant categories are coloured red in Figure 3.2.5. This �gure plots all
the regression coe�cients on a common scale from which we can observe the relative
e�ect of each of the levels and the predictor variables on the baseline category. To
better see the majority of points and error bars around the line of no deviation from
the intercept, Figure 3.2.6 is Figure 3.2.5 restricted to the range of -5 to 5.
The model in equation (3.2.2) is the same as the signi�cant-predictors school-number
model in Section 3.2.2 after the manual removal of non-signi�cant variables. We can
see from Figure 3.2.7 that there is no observable di�erence between the coe�cients
of the categories and variables in both the full school-number model and the full
simplest-stepAIC school-number model.
96 3.2. Statistical Reduction of Predictor Variables
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
Y
1
Inconsistent
Y
M
5
7
1
Inconsistent
5
6
7
8
12348
1999200020012002200320042005
Y
1
2
3
4
aboriginalatsi
disabilitygender
gradedyearlbote
non_school
occupationprocyear
school_car
school_edu
−20 −15 −10 −5 0 5 10 15Value
Cat
egor
ies Significant
●
●
No
Yes
Figure 3.2.5: Coe�cient plot for the simplest-stepAIC school-number model. (Seetext for de�nition of baseline category. The Inconsistent category of the atsi andlbote variables represents students with identi�ed anomalies in the data.)
Chapter 3. Initial Model Selection 97
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
Y
1
Inconsistent
Y
M
5
7
1
Inconsistent
5
6
7
8
12348
1999200020012002200320042005
Y
1
2
3
4
aboriginalatsi
disabilitygender
gradedyearlbote
non_school
occupationprocyear
school_car
school_edu
−5 −4 −3 −2 −1 0 1 2 3 4 5Value
Cat
egor
ies Significant
●
●
No
Yes
Figure 3.2.6: Subset of coe�cient plot for the simplest-stepAIC school-numbermodel. (See text for de�nition of baseline category. The Inconsistent categoryof the atsi and lbote variables represents students with identi�ed anomalies in thedata.)
98 3.2. Statistical Reduction of Predictor Variables
●
●
●●
●●
●
●
●
●
●●
●●
●
●
●
●
●●●●●
●●●
●●
●●
●
−10
0
10
20
30
40
50
−10 0 10 20 30 40 50Coefficients − School−Number model
Coe
ffic
ient
s −
Sim
ples
t−st
epA
IC S
choo
l−N
umbe
r m
odel
Significant●
●
No
Yes
Figure 3.2.7: Coe�cients for the simplest-stepAIC school-number model plottedagainst the regression coe�cients for the school-number model. The dotted line isthe line y=x.
Chapter 3. Initial Model Selection 99
School-Covariates Model
From the school-covariates model, the variables staff_metr, p_g_gender and p_g_nesb
are removed by stepAIC to give the simplest-stepAIC school-covariates model.
Simplest-stepAIC School-Covariates Model
NN Rasch = procyear + gradedyear + gpokm + isolation + spatial_ar + atsi
+ lbote + gender + aboriginal + disability + school_car + occupation +
school_edu + non_school + school size.
Table C.1.4 in Appendix C gives the linear regression output and the procyear vari-
able is not signi�cant. The model with only signi�cant predictors is the signi�cant-
predictors simplest-stepAIC school-covariates model.
Signi�cant-Predictors Simplest-stepAIC School-Covariates Model
NN Rasch = gradedyear + gpokm + isolation + spatial_ar + atsi + lbote +
gender + aboriginal + disability + school_car + occupation + school_edu
+ non_school + school size. (3.2.3)
Figure 3.2.8 plots all coe�cients of categorical variables in the linear regression
and the 95% con�dence intervals of the �tted estimates. All of the regression co-
e�cients are compared to the baseline category or intercept of a female student in
grade 3 in 1998 who does not come from an Aboriginal Torres Strait Islander or non-
English speaking background, is not identi�ed as having a disability or School Card
and their primary guardian's or parent's status of occupation, school education and
non-school education is not stated. The characteristics of the reference school is one
with an isolation factor of 1 and spatial value of 1.1 (metropolitan). The signi�cant
categories are coloured blue and the non-signi�cant categories are coloured red in
Figure 3.2.8. This �gure plots all the regression coe�cients on a common scale from
which we can observe the relative e�ect of each of the levels and predictor variables
on the baseline category. To better see the majority of points and error bars around
the line of no deviation from the intercept, Figure 3.2.9 is Figure 3.2.8 restricted to
the range of -5 to 5.
100 3.2. Statistical Reduction of Predictor Variables
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
Y
1
Inconsistent
Y
M
5
7
1.52
2.53
3.54
4.55
5.56
6.57
1
Inconsistent
5
6
7
8
1
2
3
4
8
1999200020012002200320042005
Y
1
2
3
4
2.2.1
2.2.2
3.1
3.2
aboriginalatsi
disabilitygender
gradedyearisolation
lbotenon_school
occupationprocyear
school_carschool_edu
spatial_ar
−20 −15 −10 −5 0 5 10 15 20Value
Cate
gorie
s Significant
●
●
No
Yes
Figure 3.2.8: Coe�cient plot for the simplest-stepAIC school-covariates model. (Seetext for de�nition of baseline category. The Inconsistent category of the atsi andlbote variables represents students with identi�ed anomalies in the data.)
Chapter 3. Initial Model Selection 101
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
Y
1
Inconsistent
Y
M
5
7
1.52
2.53
3.54
4.55
5.56
6.57
1
Inconsistent
5
6
7
8
1
2
3
4
8
1999200020012002200320042005
Y
1
2
3
4
2.2.1
2.2.2
3.1
3.2
aboriginalatsi
disabilitygender
gradedyearisolation
lbotenon_school
occupationprocyear
school_carschool_edu
spatial_ar
−5 −4 −3 −2 −1 0 1 2 3 4 5Value
Cate
gorie
s Significant
●
●
No
Yes
Figure 3.2.9: Subset of coe�cient plot for the simplest-stepAIC school-covariatesmodel. The x axis ranges from -5 to 5. (See text for de�nition of baseline category.The Inconsistent category of the atsi and lbote variables represents students withidenti�ed anomalies in the data.)
102 3.2. Statistical Reduction of Predictor Variables
In addition to the categorical variables, the regression output for the two continu-
ous predictors, gpokm and school size, are given in Table 3.2.2. Both variables are
signi�cant at the 5% signi�cance level.
Table 3.2.2: Regression output for gpokm and school size from the simplest-stepAICschool-covariates model
Estimate Std. Error t-value P -valuegpokm -0.0095 0.0020 -4.7313 2.2e-06school size 0.0027 0.0006 4.5364 5.8e-06
The model in equation (3.2.3) is the same as the signi�cant-predictors school-
covariates model in Section 3.2.2 after the manual removal of non-signi�cant vari-
ables. We can see from Figure 3.2.10 that there is no important di�erence between
the coe�cients of the categories and variables in both the full school-covariates
model and the full simplest-stepAIC school-covariates model.
We observe that although procyear is included in the simplest stepAIC model for
both the school-number and school-covariates models, it is not a signi�cant variable
- none of the calendar years have a signi�cant e�ect. Comparison of the models - one
not including and the other including procyear - using ANOVA (Tables 3.2.3 and
3.2.4), indicate that there is a signi�cant di�erence between the two models, and
procyear should be included in the simplest model. For comparison, p_g_nesb is a
variable which is not included in the simplest model by stepAIC, and the ANOVA
tables (Tables 3.2.5 and 3.2.6) say that it should not be included. This di�erence in
the ANOVA results support the inclusion of procyear by stepAIC.
Table 3.2.3: Comparison of models with and without procyear - simplest-stepAICschool-number model. Model 1 is without procyear and model 2 is with procyear
Model Res.Df RSS Df Sum of Sq F -value P -value1 19 733 894 796.912 19 726 890 826.32 7 3 970.59 12.56 < 10−16
Chapter 3. Initial Model Selection 103
●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●●●●
●
●●
●●●
●●
●0
20
40
0 20 40Coefficients − School−Covariates model
Coe
ffic
ient
s −
Sim
ples
t−st
epA
IC S
choo
l−C
ovar
iate
s m
odel
Significant●
●
No
Yes
Figure 3.2.10: Coe�cients for the simplest-stepAIC school-covariates model plottedagainst the regression coe�cients for the school-covariates model. The dotted lineis the line y=x.
104 3.2. Statistical Reduction of Predictor Variables
Table 3.2.4: Comparison of models with and without procyear - simplest-stepAICschool-covariates model. Model 1 is without procyear and model 2 is with procyear
Model Res.Df RSS Df Sum of Sq F -value P -value1 20 092 1 000 816.992 20 085 996 914.10 7 3 902.89 11.23 < 10−16
Table 3.2.5: Comparison of models with and without p_g_nesb - simplest-stepAICschool-number model. Model 1 is without p_g_nesb and model 2 is with p_g_nesb.
Model Res.Df RSS Df Sum of Sq F -value P -value1 19 726 890 826.322 19 725 890 778.47 1 47.85 1.06 0.3033
Table 3.2.6: Comparison of models with and without p_g_nesb - simplest-stepAICschool-covariates model. Model 1 is without p_g_nesb and model 2 is withp_g_nesb.
Model Res.Df RSS Df Sum of Sq F -value P -value1 20 085 996 914.102 20 084 996 911.18 1 2.93 0.06 0.8082
Sequential analysis of variance using Anova from the car package in R, reports Type
II hypothesis tests. Pairs of models are compared, and it tests the addition of one
of the predictors to a model that includes all the other predictors. For the school-
number model, aboriginal, p_g_gender and p_g_nesb are not signi�cant variables
(P -values of 0.1123, 0.8148 and 0.2995 respectively - Table 3.2.7), and similarly,
staff_metr, p_g_gender and p_g_nesb are not signi�cant predictors in the school-
covariates model (P -values of 0.5363, 0.3368 and 0.7863 respectively - Table 3.2.8),
both of which agree with the conclusion from the stepAIC model selection.
Chapter 3. Initial Model Selection 105
Table 3.2.7: Type II Anova test for the full school-number model
Variable Sum Sq Df F -value P -valueschoolno 118 172.64 401 6.53 < 2.2e-16procyear 3 984.70 7 12.60 3.1e-16gradedyear 273 038.52 2 3 022.88 < 2.2e-16atsi 417.13 2 4.62 0.0099lbote 393.94 2 4.36 0.0128gender 5 604.44 1 124.10 < 2.2e-16aboriginal 113.87 1 2.52 0.1123disability 64 839.04 1 1 435.70 < 2.2e-16school_car 794.79 1 17.60 2.7e-05occupation 826.27 5 3.66 0.0026school_edu 5 586.92 4 30.93 < 2.2e-16non_school 3 138.21 4 17.37 3.1e-14p_g_gender 2.48 1 0.05 0.8148p_g_nesb 48.62 1 1.08 0.2995Residuals 890 775.99 19 724
Table 3.2.8: Type II Anova test for the full school-covariates model
Variable Sum Sq Df F -value P -valueprocyear 3 916.88 7 11.27 2.5e-14gradedyear 282 113.13 2 2 841.66 < 2.2e-16gpokm 1 098.43 1 22.13 2.6e-06isolation 6 395.00 12 10.74 < 2.2e-16spatial_ar 971.62 4 4.89 0.0006staff_metr 18.99 1 0.38 0.5363atsi 781.25 2 7.87 0.0004lbote 545.45 2 5.49 0.0041gender 6 577.81 1 132.51 < 2.2e-16aboriginal 190.61 1 3.84 0.0501disability 74 593.08 1 1 502.72 < 2.2e-16school_car 1 537.15 1 30.97 2.7e-08occupation 4 369.33 5 17.60 < 2.2e-16school_edu 11 022.63 4 55.51 < 2.2e-16non_school 6 626.15 4 33.37 < 2.2e-16p_g_gender 45.80 1 0.92 0.3368p_g_nesb 3.65 1 0.07 0.7863school size 1 004.94 1 20.25 6.9e-06Residuals 996 845.64 20 082
106 3.2. Statistical Reduction of Predictor Variables
3.2.4 Investigation of procyear
Based on the theory of Rasch modelling, there should not be a signi�cant di�erence
in NN Rasch scores across di�erent years - the Rasch model should take that into
account and normalise all results. This is observed as none of the calendar years
have a signi�cant e�ect.
When we look at the estimated parameters for procyear in Tables C.1.3 and C.1.4
in Appendix C, we observe that the years which are most di�erent from the baseline
year of 1998 are 1999 and 2005. We observe that these are the years with small
cohorts (Table 2.3.1, Section 2.3.2), and it is permissible for them to be removed
from the data. When the data from 1998, 1999 and 2005 are removed from the
overall data set, the variable procyear is now signi�cant in 2002 and 2003 in the
linear regression of the school-number and the school-covariates models (Tables 3.2.9
and 3.2.10).
Table 3.2.9: Summary linear regression output of procyear in the school-numbermodel
Estimate Std. Error t-value P -valueIntercept 46.5944 3.023 15.413 < 2e-16
procyear2001 0.1181 0.2730 0.432 0.6655procyear2002 1.1456 0.2615 4.380 1.19e-05procyear2003 0.5230 0.2639 1.982 0.0475procyear2004 -0.1221 0.2859 -0.427 0.6695
......
......
...
This result then indicates that having taken into account the theory of Rasch mod-
elling, which states that there should be no statistically signi�cant �uctuation in
NN Rasch scores over years, and the observed increasing trend in the Rasch scores
of the data which is due to the in�ux of Grade 5 and Grade 7 students in the later
years (Section 2.5), there is still some yearly variation which is identi�ed by the
model. We can model this variation of procyear as a random e�ect with mean zero
to satisfy the theory of Rasch modelling. This will be explored further in mixed
e�ects models (Chapter 4) and hierarchical Bayesian models (Chapter 5).
Chapter 3. Initial Model Selection 107
Table 3.2.10: Summary linear regression output of procyear in the school-covariatesmodel
Estimate Std. Error t-value P -valueIntercept 49.8429 0.2894 172.206 < 2e-16
procyear2001 0.1967 0.2795 0.704 0.4816procyear2002 1.2656 0.268 4.723 2.34e-06procyear2003 0.7541 0.2670 2.824 0.0047procyear2004 0.1588 0.2893 0.549 0.5831
......
......
...
3.2.5 Comparison of School-Number and School-Covariates
Models
We have now two models - the school-number model and the school-covariates model.
The school-number model identi�es schools whose mean scores are signi�cantly dif-
ferent from the average NN Rasch mark, but from this model, we do not have any
information, for example, about the location or the level of government funding
of these schools. This information is contained in the school covariates of gpokm,
isolation, spatial_ar and staff_metr and may be important in determining the
reason why certain schools are under- or over-performing. The school-covariates
model contains more information and hence, has increased usefulness for making
conclusions and decisions at the school level. For this reason, we disregard the
school-number model from this point in favour of the school-covariates model.
However, the school factor or school e�ect is not included in the school-covariates
model, and we wish to still be able to identify individual schools. The school-
covariates model we are referring to here and for the rest of this section is the
signi�cant-predictors simplest-stepAIC school-covariates model in equation (3.2.3)
�t on data excluding procyears 1998, 1999 and 2005 (as established in Sections
3.2.3 and 3.2.4). To identify individual schools, we calculate the mean �tted score
of each school from the school-covariates model and consider that to be the school
e�ect from the school-covariates model. The data for a small worked example is
given in Table 3.2.11, and each row represents the data for an individual student in
school 27.
108 3.2. Statistical Reduction of Predictor Variables
Table 3.2.11: Subset of data to illustrate raw and �tted scores
School Number Raw Scores Fitted Model Scores27 42.03732 45.8631527 46.55742 50.9644327 48.64362 50.6049127 47.39190 52.6054427 57.89244 50.75956
Mean 48.50454 50.1595
From Table 3.2.11, the mean school e�ect from the raw data is 48.5 and the mean
school e�ect from the school-covariates model is 50.16. These means shall be known
as the original raw mean school e�ect and the original model mean school e�ect
respectively for school 27 and are calculated for all schools in the data from which
the school-covariates model was �tted. This calculated data of original raw mean
school e�ects and original model mean school e�ects is called original to distinguish
it from the transformed data considered later.
To test whether the original model mean school e�ects are signi�cant predictors
of the original raw mean school e�ects, we do a linear regression of original raw
mean school e�ects against original model mean school e�ects (Table 3.2.12 and
Figure 3.2.11). From the summary output, the original model mean school e�ect is
a signi�cant predictor at a 5% signi�cance level.
Table 3.2.12: Summary output of Raw ∼ Model linear regression
Estimate Std. Error t-value P -valueIntercept -5.6243 2.8476 -1.98 0.0489Model 1.1084 0.0538 20.59 < 2e-16
The regression line has a slope which is signi�cantly di�erent from one at the 5%
signi�cance level (test statistic of 2.01 and P -value of 0.045). Hence, there is a
signi�cant di�erence between the least squares line and the line Raw = Model.
To assess the validity of the assumptions of the linear regression, the diagnostic
plots are included in Figure 3.2.12. Standard assumption checking for linearity,
Chapter 3. Initial Model Selection 109
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●
●●
●
●
●
●
●
●●
●●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●
●
● ●
●
●
● ●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
● ● ●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●●●●
●
●
●
●
●
●
●
●
20
30
40
50
60
70
40 50 60Model
Raw
Figure 3.2.11: Linear regression of original raw mean school e�ects against the �ttedoriginal model mean school e�ects (the black line represents the line Raw = Model).
homoscedasticity and normality can be made from these diagnostic plots, and there
is no reason to suppose that the linear regression assumptions are invalid as a result.
However, when we plot the residuals against school size, there is stark heteroscedas-
ticity, and the data points form the shape of a funnel in Figure 3.2.13. As school
size increases, variability in the di�erence in the original raw and model mean school
e�ects decreases. It would appear that school size can explain some of the variance
observed in the residuals, and we investigate this conjecture further. Our objective is
to �nd a weighted transformation by school size of the original raw and model mean
school e�ects such that heteroscedasticity in the residuals is reduced somewhat, if
not completely.
110 3.2. Statistical Reduction of Predictor Variables
35 40 45 50 55 60 65
−20
−10
010
Fitted values
Res
idua
ls
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
● ●
●
●
●
●
●●●
●
●●
●
●
●●
●
●
●
●
●●●●
●
●●
●●● ●
●●●
●●●
●
●●●
●
●●
●
●
●
●●
●●
●●
●
●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
● ●
●●
● ●●
●
●
●●
●
● ●●
●●
●
● ●
●
● ●
●
●●
●
●●●
●
●● ●●
●●●
● ●
●
●
●●
●●●
●
●
●
●
●●
●●
●
●
●●●
●● ●●
●●
●
●
●
●●
●●
●
●
●
●
●●
●●●
●●●
●●●
●●●
●●●●
●
●●●●
●●
●
●●●●
●
●
●
●●
●●
●●
● ●●
●
●
●
●●
●●
●●●●
●
●
●
●
●
●
●
●
●
●●●●●●
●●
●
●
●
●
●●●
●●
●
●●●
●●
●
●
●
●
●●
●●
●
● ●●●
●
●
●●●●● ●
●
●●
●●●●
●
●●●
●
●
●●
●
●●●●
●
●●●
●
●●●
●●●●
●
●
●●
●
●●
●
●●●
●●
●●●●
●
●
●
●
●●
●
●●
●●
●●●●●
●
●
●●
●● ●
●
●●
●
●
●
●●●●
●●
●
●●
●
●●●●
●●
●
●
● ●
●●●●
Residuals vs Fitted
105
14 249
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
● ●
●
●
●
●
● ●●
●
●●
●
●
●●
●
●
●
●
●●
●●
●
●●
●●
●●●
●●
●●●
●
●●●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●●●
●
●
●●
●
● ●●
●
●
●
●●
●
●●
●
●●
●
●●
●
●
● ●●●
● ●●
●●
●
●
●●
●●●
●
●
●
●
●●
●●
●
●
●
●●
●●●●
●●
●
●
●
●●
●●
●
●
●
●
●●
●●●
●●
●
●●●
●●
●
●●
● ●
●
●●●●
●●
●
●●
●●●
●
●
●●
●●
●●
●●●
●
●
●
●●
●●
● ●●●
●
●
●
●
●
●
●
●
●
●●●●
●●
●●
●
●
●
●
●●●
●●
●
● ●●
●●
●
●
●
●
●●
●●
●
●●●●
●
●
●●
●●●●
●
●●
●●
●●
●
●●●
●
●
●●
●
●●●●
●
●●●
●
●●●
●●●●
●
●
●●
●
●●
●
●●●
●●
●●●●
●
●
●
●
●●
●
●●
●●
●●●●●
●
●
●●
●● ●
●
●●
●
●
●
●● ●●
●●
●
●●
●
●● ●●
●●
●
●
●●
●●●●
−3 −2 −1 0 1 2 3
−6
−4
−2
02
4
Theoretical Quantiles
Stan
dard
ized
res
idua
ls
Normal Q−Q
105
14249
35 40 45 50 55 60 65
0.0
0.5
1.0
1.5
2.0
2.5
Fitted values
Stan
dard
ized
res
idua
ls
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
● ●●●
●
● ●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●● ●●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●
● ●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●●
●●
●
●
●
● ●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
● ●
●
●
●●●
●
●●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●●●
●●
●
●
●
●
●
●
● ●
●
●●
●●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●●●
●
●
●●●
●
●
●
●●●
●
●●
●●●
●
●●
●
●
●
●
● ●●
●
●
●●●
●
●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●●
●
●
● ●
●●
●●
Scale−Location105
14 249
0.00 0.02 0.04 0.06
−8
−4
02
4
Leverage
Stan
dard
ized
res
idua
ls
●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●●
●
●
●
●
●●●
●
●●
●
●
●●
●
●
●
●
●●●●
●
●●
●●●●
●●●
●●●
●
●●●
●
●●
●
●
●
●●
●●●●
●
●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●●
●●●●
●
●●
●
●●●●
●
●
●●
●
●●
●
●●
●
●●●
●
●●●●●●
●●●
●
●●●
●●●
●
●
●
●
●●
●●
●
●
●●●●●●●●
●
●
●
●
●●
●●
●
●
●
●
●●
●●●
●●●
●●●●●●
●●●●
●
●●●●
●●
●
●●●●
●
●
●
●●●
●
●●●●
●●
●
●
●●
●●●●●●
●
●
●
●
●
●
●
●
●
●●●●●●
●●
●
●
●
●
●●●●
●
●
●●●
●●
●
●
●
●
●●●●
●
●●●●
●
●
●●●●●●
●
●●●
●●●
●
●●●●
●
●●
●
●●●●
●
●●●
●
●●●
●●●●
●
●
●●
●
●●●
●●●
●●
●●●●
●
●
●
●
●●●
●●
●●
●● ●●●
●●
●●
●●●●
●●
●
●
●
●●●●
●●
●
●●●
●●●●
●●
●
●
● ●
●●●●
Cook's distance
1
0.5
0.5
1
Residuals vs Leverage
105
14
4
Figure 3.2.12: Diagnostic plots to assess validity of assumptions in the linear regres-sion of Raw ∼ Model.
Chapter 3. Initial Model Selection 111
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
● ●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●●
●
●
●●
● ●
●●
−20
−10
0
10
0 100 200 300 400 500School Size
Lin
ear
Reg
ress
ion
Res
idua
ls
Figure 3.2.13: Plot of residuals versus school size.
Statistical Theory
For homoscedastic data, the linear model is mathematically represented in matrix
form, as
Y = Xβ + ε
where ε is a random vector with
E(ε) = 0 and V ar(ε) = σ2I.
In the case that data is heteroscedastic, the assumption
V ar(ε) = σ2I
becomes
V ar(ε) = σ2V
where V is a known n× n positive de�nite, symmetric matrix.
112 3.2. Statistical Reduction of Predictor Variables
Suppose we wish to estimate β. The ordinary least squares (OLS) estimate is
βOLS = (XTX)−1XTy.
βOLS is an unbiased linear estimator, but because the assumption of the Gauss-
Markov Theorem does not hold, it is not the best linear unbiased estimator. To
derive the best linear unbiased estimator, the problem must be transformed into
one for which the Gauss-Markov Theorem can be applied directly.
Theorem 3.2.1. Gauss-Markov Theorem [11]
Consider the linear regression model Y = Xβ + ε. Suppose E(Y ) = η = Xβ and
V ar(Y ) = σ2I. If aTY is an unbiased linear estimator for λTη then
V ar(aTY ) ≥ V ar(λT η)
with equality if and only if
a = X(XTX)−1XTλ.
Consider the regression model
Y = Xβ + ε
where ε is a random vector with
E(ε) = 0 and V ar(ε) = σ2V.
Pre-multiplying by V −12 gives
Y∗ = X∗β + ε∗
where
Y∗ = V −12Y , X∗ = V −
12X, and ε∗ = V −
12ε.
Now applying the rules for linear transformations of random variables, we �nd
E(ε∗) = E(V −12ε)
= V −12E(ε)
= V −120
= 0
Chapter 3. Initial Model Selection 113
and
V ar(ε∗) = V ar(V −12ε)
= V −12V ar(ε){V − 1
2}T
= σ2V −12V V −
12
= σ2I.
Hence, the Gauss-Markov Theorem applies, and the best linear unbiased estimator
is therefore
β = (XT∗ X∗)
−1XT∗ y∗
with
V ar(β) = σ2(XT∗ X∗)
−1.
Substituting for X∗ and y∗ produces the generalised least squares estimates
βGLS = (XTV −1X)−1XTV −1y
and
V ar(βGLS) = σ2(XTV −1X)−1.
Transformed Data
To account for the possible in�uence of school size on the model, we multiply both
the original raw mean school e�ects and the original model mean school e�ects by
the square root of school size to give the transformed raw mean school scores and
transformed model mean school scores. The linear regression of the transformed data
is given in Table 3.2.13. Now when the residuals are plotted again against the school
size (Figure 3.2.14), we observe the roughly constant variance of the residuals and
the reduced heteroscedasticity.
We have observed variance due to school size, however, school size does not com-
pletely explain this observed variance. Alternative methods like generalised least
squares could be used to transform the data and reduce the heteroscedasticity, but
ultimately, this variance should be modelled to identify all in�uential variables, not
only school size. This variation can be represented and modelled as a random e�ect
in a mixed e�ects model, which will be explained and implemented in Chapter 4.
114 3.2. Statistical Reduction of Predictor Variables
Table 3.2.13: Summary output of linear regression Raw*sqrt(school size) ∼Model*sqrt(school size)
Estimate Std. Error t-value P -valueIntercept 1.2545 2.9206 0.43 0.6678transformed model 0.9975 0.0055 180.57 < 2e-16
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
● ●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
−50
0
50
100
0 100 200 300 400 500School size
Res
idua
ls
Figure 3.2.14: Plot of residuals versus school size for linear regression on data trans-formed by multiplying with square root of school size.
Signi�cant Schools
Our point of interest is to be able to identify schools which are under-performing or
over-performing compared to the other �average� schools at a statistically signi�cant
level. These schools would be of interest to investigate and see what programs are
being implemented to the advantage of students in the over-performing schools and
what can be done to aid under-performing schools.
Chapter 3. Initial Model Selection 115
From the original linear regression model in Table 3.2.12, 95% prediction bands are
calculated and plotted in Plot (A) of Figure 3.2.15. From this plot, it is clear that
the majority of schools fall within the 95% prediction bands and are represented
by the red dots in Figure 3.2.15. Eight schools are signi�cantly over-performing
and lie above the prediction interval (green dots in Figure 3.2.15) and ten schools
are signi�cantly under-performing and lie below the prediction interval (blue dots).
In comparison, the 95% prediction bands from the linear regression on the trans-
formed data identify nine signi�cantly over-performing schools (green dots), three
of which are signi�cant schools from the original linear regression. In addition, thir-
teen schools are signi�cantly under-performing (blue dots), and there is an overlap
of seven schools in both the original linear regression model and the transformed
linear regression model. Plot (B) of Figure 3.2.15 is Plot (A) of Figure 3.2.15 but
with the colours of the data points representing the signi�cant schools under the
transformed linear regression. The transformed linear model better satis�es the
assumption of homoscedasticity and has narrower prediction bands on the trans-
formed scale and more signi�cant schools. Table 3.2.14 is a 3×3 table of the numberof signi�cant schools under each of the two linear regression models, and Plot (C)
of Figure 3.2.15 highlights the three and seven schools which are signi�cant under
both linear regression models.
Table 3.2.14: The number of schools which are average, statistically signi�cantlyover-performing and under-performing under the original linear regression modeland the transformed linear regression model
Transformed ModelOriginal Model Average Over UnderAverage 371 6 6Over 5 3 0Under 3 0 7
To try and explain any common, underlying reasons or characteristics shared by
the signi�cant schools, we test whether the school covariates have a signi�cant re-
lationship with the signi�cance of schools - over, average and under - using χ2
tests to test for association. The school covariates under consideration are gpokm,
isolation, spatial_ar, staff_metr, school size and school type. Schools are clas-
116 3.2. Statistical Reduction of Predictor Variables
●
●
●
●
● ●
●
●
●
●
●●
●
●
●
● ●
●
●●
●
●● ●
●
●
●●
●●●
●●
●
●
●●
●
●●●
●
●
●
●
●
●● ●●
●
●
●●
●●
●
●● ●
●●
●
● ● ●●● ●●
●
●●● ●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●●
●
●●●
●
●
●●
●
●
●●
●
●
●
●●
● ●●
●
●●
●●
●
●●
●
●
●
●
●
●●
● ●●
●●
●●●
●●
●
●
●●●●
●
●●
●●
●●
●
●●
●●● ●
●●
●
●
●
●
●
●
●●●
●
●
●●●●
●●
●
●●●
●
●●●
●●●●
●
●
●●●
●●●
●
●
●●●
●●●
● ●●●
●
●● ●●
●●
●●
●●●●
●●
●● ●
●
● ●●
●
●
●●●
●●
●● ●●●●
●
●
●
●●
●
●
●
●
● ●●●
●●●●
●
●
●●
●●●
●
● ●●●
●
●●
●● ●●
●● ●
●●● ●
●●●
●●
●●●●
●
●
●
●
● ●●●
● ●●●●
●
● ●●●●●
●
●
●●
●
●
●●
●
● ● ●●
●
●●●
●
●●●●●
●
●●●●●●●
●
●
●
●●
● ●●
● ●
●●
●
●●
●
●
●●
●
●●
●●●
●
●● ●●●●
●●
●
● ●●●●●● ●●
●●
●
●●
●●●
20
40
60
40 50 60Model
Raw
(A) Performance●
●
●
Average
Over
Under
●
●
●
●
● ●
●
●
●
●
●●
●
●
●
● ●
●
●●
●
●● ●
●
●
●●
●●●
●●
●
●
●●
●
●●●
●
●
●
●
●
●● ●●
●
●
●●
●●
●
●● ●
●●
●
● ● ●●● ●●
●
●●● ●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●●
●
●●●
●
●
●●
●
●
●●
●
●
●
●●
● ●●
●
●●
●●
●
●●
●
●
●
●
●
●●
● ●●
●●
●●●
●●
●
●
●●●●
●
●●
●●
●●
●
●●
●●● ●
●●
●
●
●
●
●
●
●●●
●
●
●●●●
●●
●
●●●
●
●●●
●●●●
●
●
●●●
●●●
●
●
●●●
●●●
● ●●●
●
●● ●●
●●
●●
●●●●
●●
●● ●
●
● ●●
●
●
●●●
●●
●● ●●●●
●
●
●
●●
●
●
●
●
● ●●●
●●●●
●
●
●●
●●●
●
● ●●●
●
●●
●● ●●
●● ●
●●● ●
●●●
●●
●●●●
●
●
●
●
● ●●●
● ●●●●
●
● ●●●●●
●
●
●●
●
●
●●
●
● ● ●●
●
●●●
●
●●●●●
●
●●●●●●●
●
●
●
●●
● ●●
● ●
●●
●
●●
●
●
●●
●
●●
●●●
●
●● ●●●●
●●
●
● ●●●●●● ●●
●●
●
●●
●●●
20
40
60
40 50 60Model
Raw
(B) Performance●
●
●
Average
Over
Under
●
●
●
●
● ●
●
●
●
●
●●
●
●
●
● ●
●
●●
●
●● ●
●
●
●●
●●●
●●
●
●
●●
●
●●●
●
●
●
●
●
●● ●●
●
●
●●
●●
●
●● ●
●●
●
● ● ●●● ●●
●
●●● ●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●●
●
●●●
●
●
●●
●
●
●●
●
●
●
●●
● ●●
●
●●
●●
●
●●
●
●
●
●
●
●●
● ●●
●●
●●●
●●
●
●
●●●●
●
●●
●●
●●
●
●●
●●● ●
●●
●
●
●
●
●
●
●●●
●
●
●●●●
●●
●
●●●
●
●●●
●●●●
●
●
●●●
●●●
●
●
●●●
●●●
● ●●●
●
●● ●●
●●
●●
●●●●
●●
●● ●
●
● ●●
●
●
●●●
●●
●● ●●●●
●
●
●
●●
●
●
●
●
● ●●●
●●●●
●
●
●●
●●●
●
● ●●●
●
●●
●● ●●
●● ●
●●● ●
●●●
●●
●●●●
●
●
●
●
● ●●●
● ●●●●
●
● ●●●●●
●
●
●●
●
●
●●
●
● ● ●●
●
●●●
●
●●●●●
●
●●●●●●●
●
●
●
●●
● ●●
● ●
●●
●
●●
●
●
●●
●
●●
●●●
●
●● ●●●●
●●
●
● ●●●●●● ●●
●●
●
●●
●●●
20
40
60
40 50 60Model
Raw
(C) Significance●
●
Once or none
Twice
Figure 3.2.15: (A) 95% prediction interval for original linear regression. Colours in-dicate schools which are considered average, statistically signi�cant over-performingand under-performing. (B) 95% prediction interval for the transformed scores.Colours indicate schools which are considered average, statistically signi�cant over-performing and under-performing. (C) Plot of which schools are signi�cantly di�er-ent in both the original linear regression model and the transformed linear regressionmodel.
Chapter 3. Initial Model Selection 117
si�ed into three di�erent types - boys-only, girls-only and co-education - and we
de�ne a boys-only school to be one which has the male gender recorded for over
90% of its students. The analogous de�nition is used for girls-only schools, and all
other schools are classi�ed as co-education. The χ2 values, degrees of freedom and
P -values are given in Table 3.2.15. At a 5% signi�cance level, we retain the null
hypothesis of independence between the signi�cance of schools and the covariate,
and none of the covariates seem to be outstanding in explaining the classi�cation of
schools into signi�cantly under-performing, average or signi�cantly over-performing
schools.
Table 3.2.15: Output from χ2 tests for association between the signi�cance groupsof schools and school covariates
Covariate χ2-value df P -valueisolation 23.4402 24 0.494spatial_ar 8.2187 8 0.4124staff_metr 1.7381 2 0.4194school type 6.2077 4 0.1842
For the continuous variables of gpokm and school size, the results of logistic regression
state that gpokm is not a signi�cant predictor of the signi�cance of schools (P -value
of 0.353) but school size is a signi�cant predictor (P -value of 0.00077) at the 5%
signi�cance level.
3.3 Discussion
We have established that the school-number model should not be considered as it
does not adequately model the school e�ect. The school-covariates model has its own
issues as the assumption of homoscedasticity is not satis�ed between the residuals
and school size. However, taking a suitable transformation of the school-covariates
model, we can identify the performance of schools based on mean scores and compare
how a school performs to how we expect them to perform from the model. This
mean-scores approach is similar to the comparison and ranking of statistically similar
schools on theMy School website. A more comprehensive and fundamental approach
118 3.3. Discussion
to the performance of schools is to look at a student-centric measure of performance.
Looking at how a school improves an individual student highlights the suitability
of a school for the student's characteristics and could potentially aid parents in
knowing which school is best for their child. This student-centric approach to the
performance of schools shall be discussed in Chapter 6.
The limitations of the school-number and school-covariates models can be addressed
by a hierarchical model which incorporates both models through having a school-
level regression in addition to a student-level model. The structure of a hierarchical
model can also model the high variability at the school level which we identi�ed
by the heteroscedasticity with school size. As a result, a hierarchical model is an
improvement and shall be discussed in Chapters 4 and 5.
Chapter 4
Hierarchical Modelling: Linear
Multilevel Mixed E�ects Models
The structure of students in classes in schools and then in education systems under
educational authorities naturally evokes multilevel or hierarchical modelling (Figure
4.0.1). The measured variable is the test result, and there are various factors at
the student, school and system levels. In the Basic Skills data set, we have student
covariates which are at the individual level and school covariates at the group level.
This structure of individuals within prescribed groups can be modelled using linear
multilevel mixed e�ects models. A vast array of literature can be found on mixed
e�ects models, and some selected papers and books include Gelman et al. [18],
Harville [24, 25], Laird and Ware [30], Snijders [43], Willms and Raudenbush [47],
Pinheiro and Bates [37] and Venables and Ripley [46].
The advantage of a hierarchical model is that the model combines both the school
number and the school covariates into a single model, and no information is lost at
either the student or school level. Thus, the school e�ect is explained and modelled
by the school covariates in a hierarchical model.
119
120 4.1. Theory of Linear Multilevel Mixed E�ects Models
SYSTEM
SCHOOL
CLASS
STUDENT
TEST
TEST RESULT
PARENT
Figure 4.0.1: Hierarchy of the school education system.
4.1 Theory of Linear Multilevel Mixed E�ects Mod-
els
We start with multiple linear regression as explained in Section 3.2.1 and to recap,
the matrix form of multiple linear regression is
y = Xβ + ε
ε ∼ Nn(0, σ2In)
where y = (y1, y2, . . . , yn)′ is the response vector; X is the model matrix with typ-
ical row xi = (xi1, xi2, . . . , xip); β = (β1, β2, . . . , βp)′ is the vector of regression
coe�cients; ε = (ε1, ε2, . . . , εn)′ is the vector of errors; Nn represents the n-variable
multivariate-normal distribution; 0 is an n× 1 vector of zeros and In is the order-n
identity matrix.
For any linear regression, the regression coe�cients are classi�ed as either �xed or
random e�ects. A �xed e�ects model is one in which the coe�cients are constant
across individuals [18]. The random e�ects coe�cients are modelled using proba-
bility distributions of a random variable. A mixed e�ects model is one which has a
combination of �xed and random e�ects.
As the simplest case, consider a single predictor. As stated before, students are
the individuals, and they are nested within groups, or schools, in our case. We
could �t a linear regression across all schools which collates all student information
Chapter 4. Hierarchical Modelling: Mixed E�ects Model 121
together and loses vital information about individual schools and their di�erentiating
characteristics which could help explain students' results. This is known as complete-
pooling
yi = α + βxi + εi
for student i (i = 1, . . . , N where N is the total number of students in the data)
and constants α and β. Another method would be to �t a linear model within each
school and have 426 linear models, one for each school. This is a no-pooling model
yi = αj[i] + βj[i]xi + εi
where i = 1, . . . , N , j[i] represents the group j of individual i and the αj[i]'s and
βj[i]'s are classic least squares estimates.
Instead, we use the nested structure of the data - students nested in schools - to �t
a linear regression where the intercept and/or slope can vary with each school and
incorporate the school e�ect into the model. The varying intercept and/or slope
can be modelled as a probability distribution or as a linear model itself based on
the school covariates. This is a mixed e�ects model and speci�cally, a hierarchical
model.
A mixed e�ects model is classi�ed as a hierarchical model when it is a random
coe�cient model with nested random coe�cients. The group-level model has pa-
rameters of its own which are known as the hyper-parameters of the model and are
estimated from the data. The data is structured into groups and coe�cients vary by
group. With grouped data, a regression that includes indicators for groups is called
a varying-intercept model because it can be interpreted as a model with a di�erent
intercept within each group
yi = αj[i] + βxi + εi (4.1.1)
where j[i] represents the group j of individual i. The slope can vary with constant
intercept to give the varying-slope model, and the varying slopes are interactions
between the continuous predictor x and the group-level indicators
yi = α + βj[i]xi + εi. (4.1.2)
The combination of equations (4.1.1) and (4.1.2) gives the varying-intercept, varying-
122 4.2. Hierarchical Model Formulation
slope model
yi = αj[i] + βj[i]xi + εi.
The model we consider is the varying-intercept model
yi = αj[i] + βxi + εi
where the αj's can be modelled by assigning a probability distribution, in this case,
the normal distribution
αj ∼ N(µα, σ2α),
for j = 1, . . . , J , with mean µα and standard deviation σα estimated from the data.
However, the group-level model is not solely restricted to a probability distribution
but can also be written as a separate regression in the form
αj = γ0 + γ1uj + ηj with ηj ∼ N(0, σ2α),
and uj is a group-level predictor. Adding a group-level predictor improves inference
for group coe�cients αj.
In summary, the model formulation is
yi = αj[i] + βxi + εi for i = 1, . . . , N
αj = γ0 + γ1uj + ηj for j = 1, . . . , J
εi ∼ N(0, σ2y)
ηj ∼ N(0, σ2α).
4.2 Hierarchical Model Formulation
The model we consider is one where the varying intercept represents the school e�ect
and the individual-level predictors are the student covariates. The school e�ect is
then modelled at the group-level by the school covariates. Considering an individual
student i, the multilevel model is
Chapter 4. Hierarchical Modelling: Mixed E�ects Model 123
NNRaschi = schoolj[i] + β1procyeari + β2gradedyeari + β3atsii + β4lbotei + β5genderi
+ β6aboriginali + β7disabilityi + β8school_cari + β9occupationi
+ β10school_edui + β11non_schooli + εi
schoolj = γ0 + γ1gpokmj + γ2isolationj + γ3spatial_arj + ηj (4.2.1)
εi ∼ N(0, σ2y)
ηj ∼ N(0, σ2α)
for student i = 1, . . . , N and school j = 1, . . . , J , with N = 20 124 and J = 401.
The beauty of hierarchical modelling is that we model the student predictor vari-
ables and take into account variation by school. Therefore, we can also incorporate
the school covariates into modelling the school variation. This hierarchical model
combines both the school number and school covariates into a single model, and no
information is lost at either the student or school level.
However, in such a hierarchical model, the school is a random e�ect and is modelled
by a distribution. As a result, this hierarchical model can no longer identify school
e�ects, over and above those already modelled by the school covariates, and in
particular, signi�cant school e�ects. To identify signi�cantly under-performing or
signi�cantly over-performing schools, school would need to be treated as a �xed or
treatment e�ect in the model. This was represented in the school-number model
(Chapter 3), but the disadvantage of the school-number model was its inability to
explain school e�ect using the school covariates. The collinearity between schoolno
and the school covariates (Section 3.2.5) meant that we could not combine both the
schoolno and the school covariates into a single model. Should we continue with
the school-number model, we would now be interested in modelling students' scores
over time at the same school, rather than between schools. This would eliminate
the need for the explanatory nature of the school covariates of location as a school
does not generally change its location over time.
A solution in the form of modelling both the student and school covariates in a
hierarchical model introduces a random e�ect for school but restricts us to no longer
being able to distinctly identify under-performing or over-performing schools.
Our aim now is to use statistical modelling techniques to achieve a well-�tting model
124 4.3. Hierarchical Model Selection
which explains the relationship between NN Rasch scores and the covariates. With
this model, we hope to identify the signi�cant predictors of NN Rasch scores and
build a comprehensive and reliable model of a school's performance based at the
individual-level of a school's students' scores.
4.3 Hierarchical Model Selection
We �t this model using the linear mixed e�ects regression function lmer in the lme4
package in R. The random e�ect is schoolno and the summary lmer regression
output is given in Table 4.3.1.
We can see from Table 4.3.1 that the summary of a linear mixed e�ects model �t
by lmer provides estimates of the �xed e�ects parameters, standard errors for those
parameters and t-values, but no P -values have been given to compare and assess
the signi�cance of variables. Douglas Bates, the author of lmer, argues in [4, 32]
why it is not correct to state P -values for a linear mixed e�ects model. Bates is of
the opinion that calculating the P -values for �xed e�ects terms in a mixed e�ects
model is not reliable. That is because with unbalanced, multilevel data, the degrees
of freedom of the denominator used to penalise uncertainty are unknown, that is, we
are uncertain about how uncertain we should be. Bates contends that alternative
inferential approaches make P -values unnecessary.
Chapter 4. Hierarchical Modelling: Mixed E�ects Model 125
Table 4.3.1: Output of �xed and random e�ects from linear mixed e�ects model
Fixed E�ects Estimate Std. Error t-valueIntercept 49.587147 0.348477 142.30procyear2001 0.155524 0.272450 0.57procyear2002 1.161221 0.260959 4.45procyear2003 0.570628 0.262967 2.17procyear2004 -0.083015 0.284865 -0.29gradedyear5 8.836288 0.120601 73.27gradedyear7 16.477341 0.418110 39.41atsi1 -2.135234 0.901806 -2.37atsiInconsistent -2.313706 0.777260 -2.98lbote1 -0.916066 0.266568 -3.44lboteInconsistent -0.501266 0.195160 -2.57genderM 1.089922 0.096272 11.32aboriginalY -1.518347 0.866750 -1.75disabilityY -8.182025 0.213560 -38.31school_carY -0.827144 0.189066 -4.37occupation1 0.771733 0.292393 2.64occupation2 0.677161 0.245864 2.75occupation3 0.673836 0.238969 2.82occupation4 0.168770 0.236223 0.71occupation8 0.079052 0.242659 0.33school_edu1 -1.723365 0.372996 -4.62school_edu2 -1.370120 0.322740 -4.25school_edu3 -0.409514 0.311441 -1.31school_edu4 0.311442 0.313765 0.99non_school5 0.653118 0.276486 2.36non_school6 1.544894 0.303423 5.09non_school7 2.274865 0.324388 7.01non_school8 0.632840 0.266736 2.37gpokm -0.007462 0.004947 -1.51isolation1.5 -0.104003 2.095854 -0.05isolation2 -2.062810 2.264896 -0.91isolation2.5 0.814912 2.251190 0.36isolation3 1.978547 2.479518 0.80isolation3.5 0.126070 2.425124 0.05isolation4 0.855229 2.589191 0.33isolation4.5 1.772173 2.908747 0.61isolation5 2.137099 3.498097 0.61isolation5.5 1.943485 3.666863 0.53isolation6 -0.043261 4.498165 -0.01isolation6.5 9.709074 7.077226 1.37isolation7 -1.888134 6.969833 -0.27spatial_ar2.2.1 0.634194 2.054559 0.31spatial_ar2.2.2 0.567557 2.270862 0.25spatial_ar3.1 1.609032 2.527954 0.64spatial_ar3.2 3.014347 2.973478 1.01
Random E�ects Name Variance Std Devschoolno Intercept 5.8022 2.4088Residual 45.2202 6.7246
126 4.3. Hierarchical Model Selection
4.3.1 Model Selection by Markov Chain Monte Carlo Sam-
pling
An alternative method to calculate and estimate the P -values is to use Markov Chain
Monte Carlo (MCMC) sampling. The function pvals.fnc (languageR package)
calculates the P -values which are given in the pMCMC column, and the highest
posterior density intervals for the �xed and random e�ects coe�cients (Table 4.3.2).
A 100(1 - α)% highest posterior density (HPD) interval is a region for which the
posterior probability of that region is 100(1 - α)%, and the minimum density of
any point within that region is equal to or larger than the density of any point
outside that region, for signi�cance level α. For the �xed e�ects parameters, anti-
conservative P -values based on the t-value calculated from the upper bound for the
degrees of freedom, are also included in the column P -value.
From Table 4.3.2, the signi�cant �xed e�ects are procyear, gradedyear, atsi,
lbote, gender, disability, school_car, occupation, school_edu and non_school
at the 5% signi�cance level.
Hence, the �nal model from MCMC sampling is
NN Rasch = procyear + gradedyear + atsi + lbote + gender + disability +
school_car + occupation + school_edu + non_school.
To compare which variables are signi�cant, it is a standard �rule-of-thumb� practice
[18] to consider a variable whose t-value has an absolute value less than two, as
signi�cant. Based on this measure, the �signi�cant� predictors in Table 4.3.1 agree
with the �nal MCMC sampling model.
Chapter 4. Hierarchical Modelling: Mixed E�ects Model 127
Table 4.3.2: Estimates and P -values of �xed and random e�ects estimated byMarkov Chain Monte Carlo sampling
Estimate MCMCmean HPD95lower HPD95upper pMCMC P -valueIntercept 49.59 49.60 48.92 50.24 0.00 0.00procyear2001 0.16 0.15 -0.36 0.69 0.58 0.57procyear2002 1.16 1.16 0.67 1.69 0.00 0.00procyear2003 0.57 0.57 0.06 1.08 0.03 0.03procyear2004 -0.08 -0.08 -0.64 0.46 0.78 0.77gradedyear5 8.84 8.83 8.60 9.07 0.00 0.00gradedyear7 16.48 16.46 15.69 17.34 0.00 0.00atsi1 -2.14 -2.16 -3.87 -0.35 0.01 0.02atsiInconsistent -2.31 -2.35 -3.87 -0.82 0.00 0.00lbote1 -0.92 -0.92 -1.44 -0.41 0.00 0.00lboteInconsistent -0.50 -0.50 -0.89 -0.13 0.01 0.01genderM 1.09 1.09 0.91 1.28 0.00 0.00aboriginalY -1.52 -1.53 -3.20 0.16 0.08 0.08disabilityY -8.18 -8.19 -8.62 -7.79 0.00 0.00school_carY -0.83 -0.83 -1.20 -0.45 0.00 0.00occupation1 0.77 0.77 0.20 1.33 0.01 0.01occupation2 0.68 0.69 0.21 1.16 0.00 0.01occupation3 0.67 0.68 0.23 1.16 0.01 0.00occupation4 0.17 0.17 -0.30 0.62 0.47 0.48occupation8 0.08 0.06 -0.42 0.53 0.80 0.74school_edu1 -1.72 -1.75 -2.45 -0.98 0.00 0.00school_edu2 -1.37 -1.39 -2.00 -0.75 0.00 0.00school_edu3 -0.41 -0.41 -1.00 0.22 0.18 0.19school_edu4 0.31 0.31 -0.30 0.92 0.33 0.32non_school5 0.65 0.66 0.11 1.21 0.02 0.02non_school6 1.54 1.55 0.96 2.15 0.00 0.00non_school7 2.27 2.29 1.67 2.94 0.00 0.00non_school8 0.63 0.64 0.12 1.18 0.02 0.02gpokm -0.01 -0.01 -0.02 0.00 0.09 0.13isolation1.5 -0.10 -0.08 -3.92 3.62 0.97 0.96isolation2 -2.06 -2.05 -6.00 2.12 0.32 0.36isolation2.5 0.81 0.86 -3.04 5.13 0.67 0.72isolation3 1.98 2.06 -2.24 6.76 0.37 0.42isolation3.5 0.13 0.21 -3.95 4.81 0.92 0.96isolation4 0.86 0.94 -3.74 5.61 0.70 0.74isolation4.5 1.77 1.88 -3.35 7.12 0.47 0.54isolation5 2.14 2.25 -4.02 8.59 0.48 0.54isolation5.5 1.94 2.07 -4.57 8.66 0.53 0.60isolation6 -0.04 0.04 -7.88 8.18 0.99 0.99isolation6.5 9.71 9.90 -3.21 23.08 0.14 0.17isolation7 -1.89 -1.49 -13.80 11.05 0.82 0.79spatial_ar2.2.1 0.63 0.59 -3.30 4.15 0.75 0.76spatial_ar2.2.2 0.57 0.51 -3.53 4.68 0.81 0.80spatial_ar3.1 1.61 1.55 -3.16 6.07 0.50 0.52spatial_ar3.2 3.01 3.01 -2.40 8.41 0.27 0.31Groups Name Std.Dev. MCMCmedian MCMCmean HPD95lower HPD95upperschoolno Intercept 2.41 2.17 2.17 1.99 2.37Residual 6.72 6.74 6.74 6.67 6.80
128 4.3. Hierarchical Model Selection
4.3.2 Model Selection by Likelihood Ratio Test
One alternative inferential method to calculate P -values is to perform a likelihood
ratio test for each term in the lmer object. The function p.values.lmer (written by
Christopher Moore [31]) takes a lmer object and iteratively �ts models via maximum
likelihood which are reduced by each �xed e�ect and compares them to the full
model, yielding a vector of P -values based on χ2(1). Note that the accuracy of the
resulting P -values depends on having a large sample.
From Table 4.3.3, the signi�cant �xed e�ects are procyear, gradedyear, atsi,
lbote, gender, disability, school_car, occupation, school_edu and non_school,
the same variables as from the Markov Chain Monte Carlo sampling. This gives the
�nal model of
NN Rasch = procyear + gradedyear + atsi + lbote + gender + disability +
school_car + occupation + school_edu + non_school.
Chapter 4. Hierarchical Modelling: Mixed E�ects Model 129
Table 4.3.3: Output of �xed and random e�ects from linear mixed e�ects regressionwith P -values calculated from comparing nested models �t by maximum likelihood
Fixed E�ects Estimate Std. Error t-value P -valueLRTIntercept 49.587147 0.348477 142.300000 0.000procyear2001 0.155524 0.272450 0.570000 0.566procyear2002 1.161221 0.260959 4.450000 0.000procyear2003 0.570628 0.262967 2.170000 0.029procyear2004 -0.083015 0.284865 -0.290000 0.776gradedyear5 8.836288 0.120601 73.270000 0.000gradedyear7 16.477341 0.418110 39.410000 0.000atsi1 -2.135234 0.901806 -2.370000 0.017atsiInconsistent -2.313706 0.777260 -2.980000 0.003lbote1 -0.916066 0.266568 -3.440000 0.001lboteInconsistent -0.501266 0.195160 -2.570000 0.010genderM 1.089922 0.096272 11.320000 0.000aboriginalY -1.518347 0.866750 -1.750000 0.079disabilityY -8.182025 0.213560 -38.310000 0.000school_carY -0.827144 0.189066 -4.370000 0.000occupation1 0.771733 0.292393 2.640000 0.008occupation2 0.677161 0.245864 2.750000 0.006occupation3 0.673836 0.238969 2.820000 0.005occupation4 0.168770 0.236223 0.710000 0.474occupation8 0.079052 0.242659 0.330000 0.757school_edu1 -1.723365 0.372996 -4.620000 0.000school_edu2 -1.370120 0.322740 -4.250000 0.000school_edu3 -0.409514 0.311441 -1.310000 0.187school_edu4 0.311442 0.313765 0.990000 0.321non_school5 0.653118 0.276486 2.360000 0.018non_school6 1.544894 0.303423 5.090000 0.000non_school7 2.274865 0.324388 7.010000 0.000non_school8 0.632840 0.266736 2.370000 0.018gpokm -0.007462 0.004947 -1.510000 0.123isolation1.5 -0.104003 2.095854 -0.050000 0.961isolation2 -2.062810 2.264896 -0.910000 0.352isolation2.5 0.814912 2.251190 0.360000 0.709isolation3 1.978547 2.479518 0.800000 0.411isolation3.5 0.126070 2.425124 0.050000 0.953isolation4 0.855229 2.589191 0.330000 0.732isolation4.5 1.772173 2.908747 0.610000 0.530isolation5 2.137099 3.498097 0.610000 0.530isolation5.5 1.943485 3.666863 0.530000 0.586isolation6 -0.043261 4.498165 -0.010000 0.991isolation6.5 9.709074 7.077226 1.370000 0.164isolation7 -1.888134 6.969833 -0.270000 0.788spatial_ar2.2.1 0.634194 2.054559 0.310000 0.755spatial_ar2.2.2 0.567557 2.270862 0.250000 0.803spatial_ar3.1 1.609032 2.527954 0.640000 0.518spatial_ar3.2 3.014347 2.973478 1.010000 0.300
Random E�ects Name Variance Std Devschoolno Intercept 5.8022 2.4088Residual 45.2202 6.7246
130 4.3. Hierarchical Model Selection
4.3.3 Model Selection by glmulti
An R package for easy automated model selection with generalised linear mod-
els is glmulti [9, 10]. From a list of explanatory variables, the function glmulti
investigates all possible unique models including main e�ects and up to pair-wise
interaction terms. In the speci�cation of these models, restrictions can be set to
exclude speci�c terms, enforce marginality or control the complexity of the candi-
date models. The function glmulti returns the best n models and the value for the
chosen criterion, whether it be Akaike's Information Criterion (AIC) or the Bayesian
Information Criterion (BIC). One can then directly select the �best� model or build
a con�dence set of models from which model-averaged parameter estimates and pre-
dictions can be made. In addition, glmulti can not only implement exhaustive
screening but can also select models using a compiled genetic algorithm method
which is particularly useful in the event of large candidate model sets.
Applying glmulti to the following main e�ects model
NN Rasch = procyear + gradedyear + atsi + lbote + gender + aboriginal +
disability+ school_car + occupation + school_edu + non_school + gpokm
+ isolation + spatial_ar + (1 | schoolno),
where (1| schoolno) is the notation for a varying intercept for each school, a set of
eight models are deemed the best after an exhaustive search.
• NN Rasch = gradedyear + disability + occupation + non_school +
isolation + gpokm,
• NN Rasch = gradedyear + disability + occupation + non_school +
isolation ,
• NN Rasch = gradedyear + disability + occupation + non_school,
• NN Rasch = gradedyear + disability + occupation + non_school +
gpokm,
• NN Rasch = gradedyear + disability + occupation + non_school +
spatial_ar + gpokm,
Chapter 4. Hierarchical Modelling: Mixed E�ects Model 131
• NN Rasch = gradedyear + disability + occupation + non_school +
spatial_ar,
• NN Rasch = gradedyear + disability + occupation + non_school +
isolation + spatial_ar + gpokm,
• NN Rasch = gradedyear + disability + occupation + non_school +
isolation + spatial_ar.
The Akaike Information Criterion pro�le or the ranked AIC values of the models
are plotted in Figure 4.3.1. The horizontal line is drawn at two AIC units above
the best model. A common rule of thumb is that models below this horizontal line
are worth considering as highlighted in the important points about using stepAIC
(Section 3.2.3). We see from Figure 4.3.1 that there are eight best models which are
the ones given above, and all other models have a much larger AIC value.
Figure 4.3.2 plots for each term, its estimated importance or relative evidence weight
computed as the sum of the relative evidence weights of all models in which the term
appears. We see from this plot that the test variable gradedyear and the student
variables of disability, occupation and non_school appear in all of the eight best
models and hence, have an importance of one. The school covariates of isolation,
gpokm and spatial_ar are also included in the best models but in only four out of
the eight models for each individual variable. This gives these variables a relative
importance of 0.5, and all other variables have a relative importance of zero.
When glmulti is applied to the main e�ects plus pair-wise interactions model, there
are hundreds of models with the same minimum AIC value in the con�dence set of
�best� models. Since there are too many models to sensibly analyse and interpret as
the optimal model, we achieve no further useful information from the model with
pair-wise interactions.
132 4.3. Hierarchical Model Selection
●●●●●●●●
●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
0 20 40 60 80 100
1356
0013
5800
1360
0013
6200
1364
0013
6600
1368
00
IC profile
Best models
Supp
ort
( ai
c )
Figure 4.3.1: Value of information criterion (AIC) for the best 100 models in theexhaustive search of the main e�ects model.
school_car
gender
aboriginal
lbote
procyear
atsi
school_edu
isolation
spatial_ar
gpokm
gradedyear
disability
occupation
non_school
Model−averaged importance of terms
0.0
0.2
0.4
0.6
0.8
1.0
Figure 4.3.2: The relative weights or importance of model terms.
Chapter 4. Hierarchical Modelling: Mixed E�ects Model 133
4.4 Analysis of Results
From the P -values calculated by MCMC sampling and the likelihood ratio test,
we identi�ed the signi�cant variables to be procyear, gradedyear, atsi, lbote,
gender, disability, school_car, occupation, school_edu and non_school. Note
that the school covariates have large standard errors which indicate high variability
between the Rasch scores of di�erent schools. Recall from Section 2.5, that our
univariate analysis established the benchmark of six Rasch marks being equivalent
to, or worth, two years of education. Looking at the coe�cients for gradedyear5
and gradedyear7, it seems like the hierarchical model �t by lmer equates two years
of education to approximately eight Rasch marks. We can now interpret the regres-
sion coe�cients in terms of the number of years of education, for example, having a
disability is equivalent to being behind by two years of education.
As discussed in Section 4.3, model selection is not conclusive from the lmer out-
put through the usual procedure of comparing P -values to a signi�cance level. We
can use alternative methods like Markov Chain Monte Carlo simulation or the like-
lihood ratio test to calculate posterior probabilities and P -values as illustrated.
Using Markov Chain Monte Carlo simulation and sampling is essentially a Bayesian
technique, and so rather than restricting ourselves to just using Markov Chain
Monte Carlo techniques to calculate posterior probabilities, we can take advantage
of Bayesian statistics to �t a Bayesian hierarchical model. We have just considered
hierarchical modelling using linear multilevel mixed e�ects models but now con-
sider a Bayesian approach, not only to assess model �t and selection but to �t the
hierarchical model itself.
Chapter 5
Bayesian Hierarchical Modelling
We wish to apply a multilevel or hierarchical model using Bayesian techniques such
as Bayesian Inference Using Gibbs Sampling (BUGS) [36]. The challenge in �tting
multilevel models is estimating the data-level regression, along with the group-level
model. The most direct way of doing this is through Bayesian inference, a statistical
method that treats the group-level model as �prior information� in estimating the
individual- or student- level coe�cients.
The individual- or student-level coe�cients are estimated using Markov Chain Monte
Carlo simulation and sampling methods. Some common and reasonably simple
methods are the random walk Metropolis-Hastings algorithm or Gibbs sampling.
5.1 Theory
5.1.1 Bayesian Statistics
Bayesian statistics is based on Bayes' Theorem which states that the joint probability
mass or density function for parameter θ and data y can be written as the product
of the prior distribution p(θ) and the likelihood p(y|θ)
p(θ,y) = p(θ)p(y|θ).
Using the de�nition of conditional probability, Bayes' Theorem can be expressed in
135
136 5.1. Theory
terms of the posterior distribution p(θ|y)
p(θ|y) =p(θ,y)
p(y)
=p(θ)p(y|θ)
p(y)
∝ p(θ)p(y|θ).
This form of Bayes' Theorem eliminates the need to deduce the joint probability
distribution which is often quite complicated - only the prior distribution and like-
lihood need to be known. We wish to calculate the posterior distribution of the
parameters of interest, and inferences are typically summarised by random draws
from the posterior distribution.
The simplest form of Bayesian inference uses an un-informative prior, often in the
form of a prior distribution which is uniformly distributed. However, if information
is known about p(θ), an informative prior can be speci�ed and will improve the
estimate of p(θ|y).
5.1.2 Markov Chain Monte Carlo Simulation and Sampling
Markov Chain Monte Carlo (MCMC) simulation is an algorithmic method which
is based on drawing values of parameter θ from a prior distribution and then up-
dating those draws until they converge to an equilibrium distribution which is the
target posterior distribution p(θ|y). The samples are drawn sequentially, with the
distribution of the sampled draws depending on the last value drawn; hence, the
draws form a Markov Chain. Recall that a Markov Chain is a sequence of random
variables θ(1), θ(2), . . . , for which, at any step n, the distribution of θ(n) given all
previous θ's depends only on the most recent value, θ(n−1). This is known as the
memoryless property. By this method, the distributions are improved at each step
in the simulation and converge to the true distribution.
Two important cases of MCMC simulation are the Metropolis-Hastings algorithm
which takes a random walk through the space of the parameters, and the Gibbs
sampler, a special case of the Metropolis-Hastings algorithm, which updates the
parameters one at a time, or in batches, using conditional distributions.
We mainly use the Gibbs sampler for our analysis.
Chapter 5. Bayesian Hierarchical Modelling 137
Gibbs Sampling
Gibbs sampling is the name given to a family of iterative algorithms that are used in
the program BUGS (Bayesian Inference Using Gibbs Sampling) [36] (Section 5.2.2)
and other programs to �t Bayesian models. The basic idea of Gibbs sampling is
to partition the set of unknown parameters and then simulate them one at a time,
or one group at a time, with each parameter or group of parameters simulated
conditionally on all the others. The algorithm is e�ective because in a wide range of
problems, estimating separate parts of a model is relatively easy, even if it is di�cult
to see how to estimate all the parameters simultaneously.
Consider the simple two-variable case where we have a pair of random variables
(X, Y ). The Gibbs sampler generates a sample from the marginal distributions
fX(x) (and then fY (y)) by sampling instead from the conditional distribution fX|Y (x|y)
(and then fY |X(y|x)), assuming that they are easily known. This is done by gener-
ating a �Gibbs sequence� of random variables
Y ′0 , X′0, Y
′1 , X
′1, . . . , Y
′k , X
′k.
The initial value Y ′0 = y′0 is chosen, and the rest are obtained iteratively by alter-
nately generating values from
X ′j ∼ fX|Y (x|Y ′j = y′j)
Y ′j+1 ∼ fY |X(y|X ′j = x′j).
This is known as Gibbs sampling, and under certain regularity conditions, the dis-
tribution of X ′k converges to fX(x), the true marginal distribution of X as k →∞.
Thus, for k large enough, the �nal observation X ′k = x′k is e�ectively a sample point
from fX(x).
This algorithm can be extended to the estimation of a parameter vector which is
divided into components or sub-vectors. For the interested reader, texts such as
Gelman [17] and many others, explain this topic to a greater depth.
138 5.2. Hierarchical Modelling Using BUGS
5.2 Hierarchical Modelling through Bayesian Infer-
ence Using Gibbs Sampling (BUGS)
5.2.1 The Hierarchical Model
In Chapter 4, we discussed the structure of a hierarchical model in general. That
structure still applies here, but in a Bayesian hierarchical model, the regression
parameters are now estimated using Bayesian methods.
To recap, in a hierarchical model, there are the individual or student level and the
group or school level. We have the e�ect of school j being modelled and explained by
a linear regression of the spatial covariates (and potentially other school-level vari-
ables) for each school. The hierarchical model given in equation (4.2.1) of Chapter
4 is
NNRaschi = schoolj[i] + β1procyeari + β2gradedyeari + β3atsii + β4lbotei
+ β5genderi + β6aboriginali + β7disabilityi + β8school_cari
+ β9school_edui + β10non_schooli + β11occupationi + εi
schoolj = γ0 + γ1gpokmj + γ2isolationj + γ3spatial_arj + ηj (5.2.1)
εi ∼ N(0, σ2y)
ηj ∼ N(0, σ2α)
for student i = 1, . . . , N and school j = 1, . . . , J where N = 20 124 and J = 401.
It is natural to model this problem hierarchically, with observable outcomes mod-
elled conditionally on certain parameters which themselves are given a probabilistic
speci�cation in terms of hyper-parameters. More speci�cally, the di�erence between
the hierarchical model in equation (5.2.1) and a Bayesian hierarchical model is that
all parameters are now modelled by distributions, and we wish to estimate these
distributions. The regression coe�cients, βs and γs, are given independent normal
prior distributions, and the hyper-parameters of σy and σα are given independent
uniform prior distributions from which their posterior distributions are computed
using the conditional probability de�nition of Bayes' Theorem and Gibbs Sampling.
These random variables σα and σy are then simulated from their posterior distri-
butions and used in the sampling from the posterior distribution for the Bayesian
Chapter 5. Bayesian Hierarchical Modelling 139
model. We assign uniform un-informative prior distributions to the hyper-parameter
standard deviations following Gelman [18].
In mathematical notation, the Bayesian hierarchical model is
NNRaschi = schoolj[i] + β1procyeari + β2gradedyeari + β3atsii + β4lbotei
+ β5genderi + β6aboriginali + β7disabilityi + β8school_cari
+ β9school_edui + β10non_schooli + β11occupationi + εi
schoolj = γ0 + γ1gpokmj + γ2isolationj + γ3spatial_arj + ηj (5.2.2)
εi ∼ N(0, σ2y)
ηj ∼ N(0, σ2α)
σy ∼ U(0, 1000)
σα ∼ U(0, 1000).
5.2.2 The Program
To implement this Bayesian hierarchical model, we use BUGS [36], a software pack-
age for performing Bayesian Inference Using Gibbs Sampling. In BUGS, the user
speci�es a complex, statistical model by stating the relationships between the re-
lated variables and the prior distributions for parameters. The software's system
then determines an appropriate MCMC scheme based on the Gibbs sampler for
analysing the speci�ed model. The power of BUGS is that it can break down the
analysis of arbitrarily large and complex structures into a sequence of relatively
simple computations which is then solved using a range of algorithms in BUGS.
There are two main versions of BUGS - WinBUGS and OpenBUGS. One of the main
di�erences between OpenBUGS and WinBUGS is the way in which the respective
systems make algorithmic decisions. WinBUGS de�nes one algorithm for each pos-
sible computation type, whereas there is no limit to the number of algorithms that
OpenBUGS can make use of, allowing for greater �exibility and extensibility. For
this reason, OpenBUGS is our software of choice, and we only state the OpenBUGS
results, as for every model and analysis we consider, the equivalent WinBUGS out-
put is very similar.
140 5.2. Hierarchical Modelling Using BUGS
5.2.3 Directed Acyclic Graphs
In BUGS, models are speci�ed as Directed Acyclic Graphs (DAGs). A directed
acyclic graph is a graph formed by a collection of vertices (nodes) and directed edges
(arrows), each edge connecting one vertex to another, such that it is not possible
to return to a vertex after leaving it. DAGs are used to describe pictorially a wide
class of statistical models through describing the essential structure of the model
and the local relationship between quantities, without using numerous equations.
This is achieved by abstraction, as the details of distributional assumptions and
deterministic relationships are hidden.
The notation for a DAG in BUGS is that each quantity or variable in the model cor-
responds to a node, and edges between nodes represent direct dependencies between
variables. Rectangular nodes denote known constants, which may be in the form
of data, and elliptical nodes represent either deterministic relationships or stochas-
tic quantities which require a distributional assumption. Stochastic dependence
and functional dependence are denoted by single-edged arrows and double-edged
arrows, respectively. Repetitive structures, such as for-loops, are represented by
`panels' which may be nested if the model is hierarchical.
The DAG for our general hierarchical model in equation (5.2.2) is given in Figure
5.2.1.
From this diagram, we see that we have data for the variables of gradedyear, atsi,
lbote, gender, aboriginal, disability, school_car, school_edu, non_school,
gpokm, isolation and spatial_ar. The panels also clearly de�ne the school and
student variables and illustrate the repetitive structure of the data and model.
Although we have data for procyear, according to the theory of Rasch modelling,
procyear should not have a signi�cant e�ect on the NN Rasch scores. This issue
has been discussed previously in Section 1.2.4 and Section 3.2.4, for your reference.
As a result, we �rstly consider the model with the exclusion of procyear in equation
(5.2.3) and then incorporate it as a �xed e�ect which is normally distributed with
mean zero and a hyper-parameter standard deviation σp. The theory of Rasch
modelling justi�es the zero mean for procyear, and we see what e�ect this has on
the �t of the model in the later section of this thesis.
Chapter 5. Bayesian Hierarchical Modelling 141
for(school j IN 1 : J)
for(student i IN 1 : N)
sigma.p
sigma.a
spatial[j]
isolation[j]gpokm[j]
a.hat[j]
school card[i]
occupation[i]aboriginal[i]
non school[i]
school edu[i]
gender[i]
disability[i]
lbote[i]atsi[i]
gradedyear[i]
procyear[i]sigma.ymu[i]
a[school[i]]
NN Rasch[i]
Figure 5.2.1: Directed acyclic graph for hierarchical model in equation (5.2.2).
NNRaschi = schoolj[i] + β1gradedyeari + β2atsii + β3lbotei + β4genderi
+ β5aboriginali + β6disabilityi + β7school_cari
+ β8school_edui + β9non_schooli + β10occupationi + εi
schoolj = γ0 + γ1gpokmj + γ2isolationj + γ3spatial_arj + ηj (5.2.3)
εi ∼ N(0, σ2y)
ηj ∼ N(0, σ2α)
σy ∼ U(0, 1000)
σα ∼ U(0, 1000).
5.2.4 Analysis of BUGS output
Using OpenBUGS with three chains of 2000 iterations each and a burn-in of 1000
iterations, we �t the model in equation (5.2.3), and the summary output is given in
Table 5.2.1. Table 5.2.1 gives the estimated mean and standard deviation for the
142 5.2. Hierarchical Modelling Using BUGS
model parameters. For these estimates, the 95% posterior density interval is given
by the 2.5% and 97.5% columns.
There are two di�culties associated with MCMC methods that are not present
with an independent sampling method. The �rst problem is determining when a
randomly initialized Markov chain has converged to its equilibrium distribution.
The second problem is whether the subsequent draws from the Markov chain are
correlated. Hence, we need to �rst answer whether the model is valid by considering
the convergence of the Markov chain, before looking at the results of the �tted
model.
Chapter 5. Bayesian Hierarchical Modelling 143
Table 5.2.1: Summary output of hierarchical model without procyear using Open-BUGS
mean sd 2.5% 25% 50% 75% 97.5% R Ne�
Intercept 50.13 0.25 49.64 49.97 50.13 50.30 50.64 1.01 170gradedyear5 8.81 0.10 8.61 8.74 8.80 8.88 9.02 1.00 1 900gradedyear7 15.85 0.40 15.09 15.58 15.85 16.12 16.66 1.00 840atsi1 -2.04 0.93 -3.96 -2.64 -2.01 -1.39 -0.29 1.03 96atsiInconsistent -2.40 0.80 -3.98 -2.93 -2.39 -1.87 -0.84 1.00 600lbote1 -1.02 0.26 -1.54 -1.20 -1.02 -0.85 -0.51 1.00 1 400lboteInconsistent -0.42 0.20 -0.79 -0.55 -0.42 -0.29 -0.04 1.00 3 000genderM 1.10 0.10 0.91 1.04 1.10 1.16 1.29 1.01 450aboriginalY -1.61 0.89 -3.28 -2.23 -1.66 -1.04 0.19 1.03 100disabilityY -8.18 0.21 -8.60 -8.32 -8.18 -8.04 -7.76 1.01 330school_carY -0.92 0.18 -1.27 -1.04 -0.92 -0.80 -0.58 1.00 1 200occupation1 0.80 0.28 0.25 0.61 0.80 1.00 1.34 1.00 1 100occupation2 0.69 0.24 0.22 0.54 0.70 0.86 1.14 1.00 960occupation3 0.69 0.23 0.22 0.52 0.69 0.85 1.13 1.00 970occupation4 0.19 0.23 -0.28 0.03 0.20 0.36 0.62 1.01 340occupation8 0.09 0.24 -0.39 -0.08 0.09 0.25 0.54 1.00 570school_edu1 -1.69 0.38 -2.44 -1.95 -1.69 -1.43 -0.94 1.04 64school_edu2 -1.35 0.33 -1.97 -1.58 -1.35 -1.11 -0.70 1.05 51school_edu3 -0.40 0.32 -1.01 -0.62 -0.41 -0.17 0.22 1.06 45school_edu4 0.33 0.32 -0.28 0.10 0.33 0.56 0.97 1.06 44non_school5 0.63 0.28 0.09 0.43 0.63 0.81 1.19 1.03 86non_school6 1.51 0.31 0.91 1.30 1.51 1.72 2.10 1.02 96non_school7 2.23 0.32 1.61 2.00 2.23 2.45 2.87 1.02 110non_school8 0.61 0.27 0.10 0.42 0.63 0.79 1.15 1.02 90gpokm -0.01 0.00 -0.02 -0.01 -0.01 -0.01 0.00 1.08 33isolation1.5 0.41 1.80 -3.90 -0.65 0.52 1.66 3.63 1.38 9isolation2 -1.52 1.97 -5.77 -2.74 -1.48 -0.20 2.18 1.32 10isolation2.5 1.37 1.90 -3.09 0.23 1.52 2.73 4.66 1.41 9isolation3 2.66 2.03 -1.87 1.37 2.74 4.09 6.37 1.41 9isolation3.5 0.81 1.96 -3.57 -0.38 1.00 2.14 4.14 1.47 8isolation4 1.58 2.12 -3.03 0.22 1.81 3.12 5.03 1.44 8isolation4.5 2.54 2.48 -2.99 0.95 2.98 4.29 6.43 1.41 9isolation5 3.00 3.06 -3.52 1.09 3.36 5.20 8.18 1.27 12isolation5.5 2.73 3.15 -4.10 0.73 3.19 4.91 8.11 1.30 11isolation6 0.83 3.93 -7.44 -1.68 1.01 3.51 8.11 1.17 16isolation6.5 10.56 6.27 -2.02 6.61 10.77 14.67 22.18 1.04 63isolation7 -0.87 6.16 -13.52 -4.88 -0.58 3.44 10.36 1.15 18spatial_ar2.2.1 0.12 1.72 -2.97 -1.01 0.02 1.08 4.28 1.37 10spatial_ar2.2.2 -0.02 1.74 -3.21 -1.17 -0.13 1.07 3.66 1.42 9spatial_ar3.1 1.01 1.96 -2.45 -0.39 0.89 2.25 5.21 1.26 13spatial_ar3.2 2.61 2.45 -1.91 0.83 2.49 4.24 7.74 1.07 35σy 6.74 0.03 6.67 6.71 6.74 6.76 6.81 1.00 1 800σα 2.41 0.12 2.19 2.33 2.41 2.48 2.65 1.00 2 900
144 5.2. Hierarchical Modelling Using BUGS
Validity of the Model: Diagnosis of Convergence
Markov Chain Monte Carlo methods generate samples from a target distribution
only after it has converged to an equilibrium distribution. In practice, we look at
diagnostics to monitor whether the Markov chains have converged. Methods to do
this include the numerical measures of R and Ne� which are given in Table 5.2.1, as
well as diagnostic plots.
Potential Scale Reduction
One way to monitor whether a chain has converged to the equilibrium distribution
is to compare its behaviour to other randomly initialised chains. This is the moti-
vation for the potential scale reduction statistic R as de�ned by Gelman and Rubin
[19]. The R statistic measures the ratio of the average variance of samples within
each chain to the variance of the pooled samples across chains. If all chains are at
equilibrium, these variances will be the same and R will be one, but if the chains
have not converged to a common distribution, the R statistic will be greater than
one.
Suppose we have M Markov Chains denoted by θm for m = 1, . . . ,M , each of which
has N simulation draws to give θ(n)m (n = 1, . . . , N). The between-sample variance
estimate is
B =N
M − 1
M∑m=1
(θ(•)m − θ(•)• )2,
where
θ(•)m =1
N
N∑n=1
θ(n)m and θ(•)• =1
M
M∑m=1
θ(•)m ,
and the within-sample variance is
W =1
M
M∑m=1
s2m,
where
s2m =1
N − 1
N∑n=1
(θ(n)m − θ(•)m )2.
The variance estimator is then de�ned to be
ˆvar+(θ|y) =n− 1
NW +
1
NB,
Chapter 5. Bayesian Hierarchical Modelling 145
where y is the data. From this, the potential scale reduction statistic is de�ned by
R =
√ˆvar+(θ|y)
W.
This potential scale reduction approaches 1 as N →∞.
Looking at Table 5.2.1, the R values for the student variables are close to one,
but for the school variables, R ranges from 1.08 to 1.59. The discrepancy from a
preferred value of one indicates that the estimates of the school parameters are not
very reliable.
E�ective Sample Size
The second technical di�culty posed by MCMC methods is that the samples will
typically be autocorrelated within a chain. This leads to under-estimation of the
standard error of the posterior parameters. The amount by which autocorrelation
within Markov Chains leads to under-estimation of the standard error can be mea-
sured by the e�ective sample size.
Suppose we have the same situation of M Markov Chains denoted by θm for m =
1, . . . ,M , each of which has N simulation draws to give θ(n)m (n = 1, . . . , N). We start
with the observation that if the N simulation draws within each chain were truly
independent, then the between-sample variance B would be an unbiased estimate
of the posterior variance var(θ|y), and we would have a total of MN independent
simulations for the M chains. In general, however, the simulations of θ within each
chain will be autocorrelated. We thus de�ne the e�ective number of independent
draws or e�ective sample size as
Ne� = MNˆvar+(θ|y)
Bwith ˆvar+(θ|y) and B de�ned as for the potential scale reduction. If M is small,
B will have a high sampling variability, and Ne� is a fairly crude estimate. We
actually report min(Ne�,MN) to avoid claims that the simulation is more e�cient
than random sampling. If desired, more precise measures of simulation accuracy
and the e�ective number of draws could be constructed based on autocorrelations
within each chain.
Looking at Table 5.2.1, the e�ective sample size is reasonable for most of the student
parameters and both standard deviation parameters, but only ranges from eight to
sixty-three for the school parameters. The parallels between Ne� and R agree to give
146 5.2. Hierarchical Modelling Using BUGS
the message that the school parameters are not accurately estimated by OpenBUGS.
Visualisation of Results
As part of the analysis of the OpenBUGS model, various diagnostic plots are pro-
duced for each of the estimated parameters. In the OpenBUGS model, the param-
eters are the model coe�cients for the student variables and the school variables,
the mean school e�ect for each school (αj) and the standard deviation for student
and school e�ect, σy and σα respectively. For each type of diagnostic plot, a brief
description and an example of an undesirable plot versus a desirable plot will be
given along with a discussion of the results.
Histograms: Histograms of the posterior distributions of all estimated model pa-
rameters, combined from all chains, are plotted to visualise the sampling distribution
of the parameters (Figure 5.2.2). All histograms are roughly symmetric with varying
means.
schoolno27
schoolno28
schoolno29
0
100
200
300
0
100
200
0
100
200
300
44 48 52 56
44 48 52 56
48 50 52 54 56value
coun
t
Figure 5.2.2: Histogram of parameters' posterior distribution.
Density Plots: Similar to the histograms, density plots of the posterior distribution
of the parameters allow for ease of comparison between chains - an individual density
is plotted for each chain (Figure 5.2.3). The density plots of all except seventeen
Chapter 5. Bayesian Hierarchical Modelling 147
of the school e�ects from school 576 to school 639, are nearly identical for all three
chains.
schoolno30
schoolno31
schoolno33
0.0
0.1
0.2
0.0
0.1
0.2
0.0
0.1
0.2
0.3
42.5 45.0 47.5 50.0 52.5
48 51 54
47.5 50.0 52.5 55.0 57.5value
dens
ity
Chain
1
2
3
isolation4
isolation4.5
isolation5
0.0
0.1
0.2
0.3
0.00
0.05
0.10
0.15
0.20
0.00
0.05
0.10
0.15
−3 0 3 6
0 5
0 5 10value
dens
ity
Chain
1
2
3
(a) good (b) bad
Figure 5.2.3: Density plot of parameters' posterior distribution (each chain is adi�erent colour).
Traceplots: Traceplots give a time series representation of the estimated param-
eters. From the traceplots, we wish to see good mixing between the three chains.
Some examples are given in Figure 5.2.4, and the left hand plot is representative
of well-mixed chains as the traceplots of all three chains are plotted on top of each
other. The right hand plot though shows a parameter whose estimates are not well-
mixed as the three coloured chains are distinctly separate. Such plots are for schools
576 to 639, which corresponds to our observations about the di�erent density plots.
Running Means: Another diagnostic plot is the running means plot, which is a
time series of the running mean of a chain for each of the individual chains. The
horizontal line denotes the mean of the chain, and convergence is shown by the chain
converging to the mean line as the number of iterations increase (Figure 5.2.5). By
1000 iterations, some parameters still have not converged to the mean but are slowly
approaching convergence.
148 5.2. Hierarchical Modelling Using BUGS
schoolno35
schoolno37
schoolno39
45.0
47.5
50.0
52.5
30
35
40
45
48
50
52
54
56
0 250 500 750 1000
0 250 500 750 1000
0 250 500 750 1000Iteration
value
Chain
1
2
3
isolation4
isolation4.5
isolation5
−3
0
3
6
0
5
0
5
10
0 250 500 750 1000
0 250 500 750 1000
0 250 500 750 1000Iteration
value
Chain
1
2
3
(a) good (b) bad
Figure 5.2.4: Traceplots of parameters (each chain is a di�erent colour).
1 2 3
49.0
49.5
50.0
50.5
51.0
46.0
46.5
47.0
47.5
46
47
48
49
schoolno54schoolno55
schoolno57
0 250 500 750 10000 250 500 750 10000 250 500 750 1000Iteration
Runn
ing M
ean
Chain
1
2
3
1 2 3
−4
−2
0
2
−2
0
2
4
−4
−2
0
2
isolation7spatial2.2.1
spatial2.2.2
0 250 500 750 10000 250 500 750 10000 250 500 750 1000Iteration
Runn
ing M
ean
Chain
1
2
3
(a) good (b) bad
Figure 5.2.5: Running means of parameters in each chain.
Chapter 5. Bayesian Hierarchical Modelling 149
Autocorrelation Plots: Autocorrelation plots of each parameter, bounded be-
tween -1 and 1 are given in Figure 5.2.6. We expect an exponential decrease in
autocorrelation as lag increases. However, some parameters are very highly corre-
lated even at a lag of �fty which raises concerns about the reliability of these results
from OpenBUGS. High autocorrelation within chains is an indication that the pa-
rameter space may not be thoroughly explored by Gibbs Sampling, and so, estimates
of the parameters are unreliable. For this reason, in Section 5.3, we investigate an
alternative method which reduces the correlation in the sampling method.
1 2 3
−1.0
−0.5
0.0
0.5
1.0
−1.0
−0.5
0.0
0.5
1.0
−1.0
−0.5
0.0
0.5
1.0
schoolno50schoolno51
schoolno52
0 10 20 30 40 50 0 10 20 30 40 50 0 10 20 30 40 50Lag
Autoc
orrela
tion
Chain
1
2
3
1 2 3
−1.0
−0.5
0.0
0.5
1.0
−1.0
−0.5
0.0
0.5
1.0
−1.0
−0.5
0.0
0.5
1.0
occ4occ8
edu1
0 10 20 30 40 50 0 10 20 30 40 50 0 10 20 30 40 50Lag
Autoc
orrela
tion
Chain
1
2
3
(a) good (b) bad
Figure 5.2.6: Autocorrelation plots of parameters.
150 5.2. Hierarchical Modelling Using BUGS
Crosscorrelation Plot: Previously, we looked at the autocorrelation plots of each
parameter in each chain - now the correlations between all the parameters are given
in Figure 5.2.7. We observe that there is high correlation between the school vari-
ables, and it occurs in blocks for both the school and student variables.
gradedyear5gradedyear7
atsi1atsiInconsistent
lbote1lboteInconsistent
genMaborY
disYschoolcardY
occ1occ2occ3occ4occ8edu1edu2edu3edu4non5non6non7non8
Interceptgpokm
isolation1.5isolation2
isolation2.5isolation3
isolation3.5isolation4
isolation4.5isolation5
isolation5.5isolation6
isolation6.5isolation7
spatial2.2.1spatial2.2.2
spatial3.1spatial3.2
grad
edye
ar5
grad
edye
ar7
atsi
1at
siIn
cons
iste
ntlb
ote1
lbot
eInc
onsi
sten
tge
nMab
orY
disY
scho
olca
rdY
occ1
occ2
occ3
occ4
occ8
edu1
edu2
edu3
edu4
non5
non6
non7
non8
Inte
rcep
tgp
okm
isol
atio
n1.5
isol
atio
n2is
olat
ion2
.5is
olat
ion3
isol
atio
n3.5
isol
atio
n4is
olat
ion4
.5is
olat
ion5
isol
atio
n5.5
isol
atio
n6is
olat
ion6
.5is
olat
ion7
spat
ial2
.2.1
spat
ial2
.2.2
spat
ial3
.1sp
atia
l3.2
−0.5
0.0
0.5
value
Figure 5.2.7: Crosscorrelations of parameters.
Chapter 5. Bayesian Hierarchical Modelling 151
Highest Posterior Density Interval (HPD) Plot: Recall from Section 4.3.1
that the highest posterior density interval is the interval of values that contains
100(1 - α)% of the posterior probability and also has the characteristic that the
density within the region is never lower than that outside. The 95% highest posterior
density intervals of the school and student parameters are plotted in Figure 5.2.8,
also known as a caterpillar plot. These could be used to assess the signi�cance of
variables depending on whether the value of zero is contained in the HPD interval.
152 5.2. Hierarchical Modelling Using BUGS
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
Intercept
gpokm
isolation1.5
isolation2
isolation2.5
isolation3
isolation3.5
isolation4
isolation4.5
isolation5
isolation5.5
isolation6
isolation6.5
isolation7
spatial2.2.1
spatial2.2.2
spatial3.1
spatial3.2
0 20 40Value
Par
amet
er
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
gradedyear5
gradedyear7
atsi1
atsiInconsistent
lbote1
lboteInconsistent
genM
aborY
disY
schoolcardY
occ1
occ2
occ3
occ4
occ8
edu1
edu2
edu3
edu4
non5
non6
non7
non8
0 10Value
Par
amet
er
Figure 5.2.8: Highest posterior density plot for the school and student parameters.
Chapter 5. Bayesian Hierarchical Modelling 153
We could go on to interpret the coe�cients of the OpenBUGS output and their
signi�cance from the HPD intervals, but based on the validation of the model,
the estimates may not be very reliable. So, we investigate a way to avoid high
autocorrelation.
5.3 Hierarchical Modelling Using Stan
In 2012, the software Stan was released [5]. Stan is based on the No-U-Turn Sam-
pler (NUTS), an extension of the Hamiltonian Monte Carlo (HMC) method which
is a MCMC algorithm that avoids the random walk behaviour and sensitivity to
correlated parameters which a�ect the more common MCMC methods.
5.3.1 Theory
Hamiltonian Monte Carlo
For complicated models with many parameters, simple methods such as the random-
walk Metropolis-Hastings algorithm or Gibbs sampling may require an unacceptably
long time to converge to the target equilibrium distribution. This is due to the ten-
dency of these methods to explore the parameter space via ine�cient random walks.
Hamiltonian Monte Carlo (HMC) is able to suppress such ine�cient behaviour by
transforming the problem into one of simulating Hamiltonian dynamics, rather than
sampling from a target distribution.
HMC is a MCMC method based on simulating the Hamiltonian dynamics of a �c-
tional physical system in which the parameter θ represents the position of a particle
in K-dimensional space, and potential energy is de�ned to be the negative, unnor-
malised log probability. Each sample in the Markov Chain is generated by starting
at the last sample, applying a random momentum to determine initial kinetic en-
ergy, then simulating the path of the particle in the �eld. This is the evolution over
time of the Hamiltonian dynamics of this system and requires the gradient of the
log-posterior probability. Although HMC is more e�cient than standard MCMC
algorithms, the e�ciency of HMC is reliant on the choice of at least two parameters,
a step size ε and the number of steps L for which to run a simulated Hamiltonian
154 5.3. Hierarchical Modelling Using Stan
system. In particular, if L is too small, the algorithm exhibits undesirable ran-
dom walk behaviour, while if L is too large, the algorithm wastes computation. A
poor choice of either of these parameters will result in a dramatic drop in HMC's
e�ciency.
No-U-Turn Sampler
The No-U-Turn Sampler (NUTS) is a MCMC algorithm which extends HMC and
eliminates the need to set a number of steps L. Standard HMC runs the simulation
for a �xed number of discrete steps of a �xed step size, but NUTS adjusts the number
of steps on each iteration and allows varying step size per parameter. NUTS uses a
recursive algorithm to build a set of likely candidate points that span a wide swath
of the target distribution, stopping automatically when it starts to double back and
retrace its steps. NUTS uses a geometric criterion that stops a trajectory when
it begins to head back in the direction of the initial state; hence the name of no
U-turns. This prevents correlation between estimates. This criterion is based on the
dot product between the current momentum and the vector from the initial position
to the current position. Once a trajectory is stopped, NUTS uses slice sampling to
select a state along the trajectory as the next proposal. The details of the NUTS
algorithm can be found in Ho�man and Gelman [26].
NUTS eliminates HMC's dependence on the number-of-steps parameter but retains
and sometimes improves HMC's ability to generate e�ectively independent samples
e�ciently. With NUTS, it is possible to obtain the e�ciency of HMC without
spending time and e�ort hand-tuning HMC's parameters.
The advantage of NUTS is its e�ciency in e�ectively generating independent sam-
ples. A quantitative comparison of NUTS and Metropolis and Gibbs sampling al-
gorithms was done on a highly-correlated 250-dimensional multivariate normal dis-
tribution in Ho�man and Gelman [26]. The samples are generated from the data
in Ho�man and Gelman [26], and the �rst two dimensions are plotted in Figure
5.3.1. The right-most plot is of 1000 independent draws from the highly-correlated
distribution, and we observe the e�cient exploration of the parameter space. One
million samples by Metropolis and Gibbs sampling are given in the �rst and second
left plots, and the Metropolis samples are very concentrated with the Gibbs samples
Chapter 5. Bayesian Hierarchical Modelling 155
being slightly more extensive. These plots were of one million samples, but with
only 1000 samples generated by NUTS (second-from-the-right plot), NUTS is able
to generate many e�ectively independent samples and explore the space relatively
well compared to Metropolis and Gibbs sampling. The extent to which the param-
eter space is explored by independent samples and those by NUTS is very similar
and supports the claim that NUTS can generate e�ectively independent samples.The No-U-Turn Sampler
Figure 5.3.1: Comparison of NUTS with Metropolis and Gibbs sampling. (the �rsttwo dimensions are plotted on the two axes) [26]
NUTS's ability to operate e�ciently without user intervention makes it well suited
for use in inference programs such as BUGS which until now, has largely relied
on much less e�cient algorithms, such as Gibbs sampling. NUTS is used as the
fundamental inference algorithm for parameters in the program Stan.
5.3.2 Results
Validity of the Model: Diagnosis of Convergence
After using Stan to �t the hierarchical model with procyear as a �xed main e�ect,
the results from three chains of 2000 iterations each, are given in Table 5.3.1. The
measures for convergence are again, R which is exactly or close to one for all vari-
ables, and Ne� is adequately large, the smallest e�ective sample size being 306. All
the diagnostic plots indicate the Markov Chains have converged to the equilibrium
distribution, and the autocorrelation is now reasonable for all variables. The diag-
A NOTE:
This figure/table/image has been removed to comply with copyright regulations. It is included in the print copy of the thesis held by the University of Adelaide Library.
156 5.3. Hierarchical Modelling Using Stan
nostic plots are not included here, but to illustrate the reduction in autocorrelation,
Figure 5.3.2 is the plot of the previously highly-correlated variable in Figure 5.2.6(b)
but now from Stan, and we observe the improvement. The overall conclusion is that
convergence can be assumed, and the estimates are reliable.
1 2 3
−1.0
−0.5
0.0
0.5
1.0
−1.0
−0.5
0.0
0.5
1.0
−1.0
−0.5
0.0
0.5
1.0
occ2occ3
occ4
0 10 20 30 40 50 0 10 20 30 40 50 0 10 20 30 40 50Lag
Aut
ocor
rela
tion
Chain
1
2
3
Figure 5.3.2: Autocorrelation plot of parameters from Stan.
Chapter 5. Bayesian Hierarchical Modelling 157
Table 5.3.1: Summary output of hierarchical model with procyear as �xed e�ectusing Stan
mean se mean sd 2.5% 25% 50% 75% 97.5% Ne� RIntercept 51.1 0.9 1.3 45.7 47.6 49.1 50.8 52.3 1 025 1.01procyear2001 0.02 0.01 0.24 -0.41 -0.14 0.01 0.18 0.54 443 1.01procyear2002 1.01 0.01 0.24 0.59 0.85 1.00 1.16 1.53 332 1.01procyear2003 0.43 0.01 0.23 0.02 0.28 0.42 0.58 0.92 306 1.01procyear2004 -0.20 0.01 0.25 -0.64 -0.36 -0.21 -0.04 0.32 381 1.01gradedyear5 8.84 0.00 0.12 8.61 8.76 8.84 8.92 9.08 1 055 1.00gradedyear7 16.39 0.01 0.42 15.58 16.10 16.39 16.68 17.21 1 006 1.00atsi1 -2.00 0.04 0.89 -3.67 -2.62 -1.99 -1.37 -0.22 506 1.00atsiInconsistent -2.30 0.02 0.77 -3.79 -2.83 -2.30 -1.80 -0.76 1 061 1.00lbote1 -0.96 0.01 0.26 -1.44 -1.14 -0.96 -0.78 -0.42 1 023 1.00lboteInconsistent -0.52 0.01 0.19 -0.91 -0.65 -0.51 -0.39 -0.15 1 484 1.00genderM 1.09 0.00 0.10 0.90 1.03 1.09 1.16 1.29 3 000 1.00aboriginalY -1.84 0.04 0.85 -3.47 -2.43 -1.86 -1.26 -0.19 504 1.00disabilityY -8.17 0.01 0.22 -8.58 -8.31 -8.17 -8.02 -7.75 1 368 1.00school_carY -0.84 0.00 0.19 -1.20 -0.97 -0.83 -0.70 -0.49 1 553 1.00occupation1 0.79 0.01 0.28 0.24 0.59 0.78 0.98 1.35 1 077 1.00occupation2 0.69 0.01 0.24 0.24 0.53 0.68 0.86 1.15 1 036 1.00occupation3 0.69 0.01 0.23 0.23 0.53 0.69 0.85 1.13 815 1.00occupation4 0.19 0.01 0.23 -0.25 0.03 0.19 0.35 0.65 804 1.00occupation8 0.12 0.01 0.24 -0.36 -0.05 0.11 0.28 0.57 911 1.00school_edu1 -1.81 0.01 0.37 -2.52 -2.06 -1.82 -1.57 -1.05 709 1.01school_edu2 -1.42 0.01 0.32 -2.03 -1.63 -1.42 -1.21 -0.79 517 1.01school_edu3 -0.47 0.01 0.30 -1.06 -0.67 -0.47 -0.27 0.17 453 1.01school_edu4 0.27 0.01 0.31 -0.33 0.05 0.27 0.47 0.89 447 1.01non_school5 0.70 0.01 0.27 0.17 0.51 0.70 0.87 1.23 379 1.01non_school6 1.60 0.01 0.30 1.00 1.40 1.59 1.80 2.18 411 1.01non_school7 2.32 0.02 0.31 1.68 2.11 2.33 2.52 2.92 363 1.01non_school8 0.66 0.01 0.26 0.17 0.50 0.67 0.84 1.18 373 1.01gpokm 3.10 0.24 9.55 -15.51 -3.43 3.17 9.35 22.28 1 539 1.00isolation1.5 2.34 0.26 9.30 -16.34 -3.76 2.46 8.53 20.48 1 237 1.01isolation2 3.12 0.30 10.10 -16.26 -3.68 2.97 9.86 23.30 1 171 1.00isolation2.5 2.51 0.32 9.04 -15.47 -3.40 2.38 8.82 19.75 802 1.00isolation3 2.55 0.26 9.85 -16.12 -4.18 2.27 8.85 22.83 1 397 1.00isolation3.5 3.10 0.26 9.74 -15.17 -3.60 3.22 9.75 22.32 1 396 1.00isolation4 2.50 0.33 9.59 -16.08 -4.24 2.62 8.81 21.81 863 1.00isolation4.5 2.98 0.37 10.20 -17.43 -3.75 3.01 9.77 23.09 775 1.00isolation5 2.75 0.29 10.13 -16.49 -4.25 2.32 9.50 23.35 1 184 1.00isolation5.5 2.00 0.30 9.73 -17.60 -4.52 1.92 8.68 21.93 1 052 1.01isolation6 2.82 0.31 9.51 -15.70 -3.77 2.98 9.12 21.69 916 1.00isolation6.5 2.96 0.28 9.21 -15.02 -3.25 2.88 9.20 20.42 1 080 1.00isolation7 2.93 0.25 9.86 -15.78 -3.96 2.86 9.75 22.01 1 550 1.00spatial_ar2.2.1 2.68 0.30 9.85 -16.34 -3.85 2.47 9.06 22.62 1 051 1.00spatial_ar2.2.2 2.95 0.29 10.04 -17.86 -3.36 3.10 9.38 22.22 1 166 1.00spatial_ar3.1 2.67 0.27 9.54 -15.90 -3.75 2.74 9.11 21.13 1 214 1.00spatial_ar3.2 3.01 0.29 9.57 -15.96 -3.47 3.31 9.21 21.57 1 109 1.00σα 2.47 0.00 0.10 2.31 2.40 2.47 2.54 2.69 1 416 1.00σy 6.73 0.00 0.03 6.66 6.70 6.73 6.75 6.80 3 000 1.00σp 0.92 0.02 0.64 0.34 0.55 0.74 1.08 2.44 805 1.00
158 5.3. Hierarchical Modelling Using Stan
Interpretation of Regression Coe�cients
Looking at the regression coe�cients (Table 5.3.1), we can determine which predictor
variables are signi�cant by considering the 95% highest posterior density interval
given by the 2.5% and 97.5% bounds and seeing whether zero is contained within the
interval. Based on this, the variables of procyear, occupation and school_edu are
signi�cant, as well as all the school covariates, gpokm, isolation and spatial_ar.
However, all the school covariates have large standard errors, creating 95% HPD
intervals with widths of approximately 37 units.
Comparing the Stan regression coe�cients (Table 5.3.1) to those given by lmer
(Table 4.3.1), the student predictor variables of gradedyear, atsi, lbote, gender,
aboriginal, disability, school_car, occupation, school_edu and non_school
have very similar estimates, some of them di�ering at the second decimal place.
The school predictor variables, however, vary greatly between lmer and Stan, but
recall that the standard errors are very high and notice that the estimated lmer
coe�cients are contained within the 95% HPD intervals. We can also estimate the
between-students and between-schools standard deviation by σy = 6.73 and σα =
2.47 respectively.
Recall from Section 2.5 that our univariate analysis established the benchmark of six
Rasch marks being equivalent to two years of education. Looking at the coe�cients
for gradedyear5 and gradedyear7, it seems like it is now approximately eight Rasch
marks which are equivalent to two years of education. We can now interpret the
regression coe�cients in terms of years of education.
Chapter 6
Model Validation
At this point, we have �t the hierarchical model
NNRaschi = schoolj[i] + β1procyeari + β2gradedyeari + β3atsii + β4lbotei
+ β5genderi + β6aboriginali + β7disabilityi + β8school_cari
+ β9school_edui + β10non_schooli + β11occupationi + εi
schoolj = γ0 + γ1gpokmj + γ2isolationj + γ3spatial_arj + ηj (6.0.1)
εi ∼ N(0, σ2y)
ηj ∼ N(0, σ2α)
for student i = 1, . . . , N and school j = 1, . . . , J where N = 20 124 and J = 401,
using two di�erent methods - lmer's linear multilevel mixed e�ects and Stan's No-
U-Turn sampler. In Chapter 5, we �tted the models, assessed model �t and the
signi�cance of variables, and interpreted regression coe�cients. However, we wish
to validate the models and use them to predict students' scores.
6.1 Student-level Prediction
One method of assessing the e�ect of school is to analyse the performance of the
students in the school. From our �tted hierarchical model, suppose we calculate a
�tted score for a student. The error associated with this �tted score is represented
by a prediction interval calculated for a student's score in their speci�ed school and
with their student characteristics as given in the data set. In the education system,
159
160 6.1. Student-level Prediction
we would expect a narrow expected range of marks around a student's observed
score. As a result, we calculate a 50% prediction interval, compare the observed
NN Rasch score to the 50% prediction interval, and say that if a student's observed
score lies below the lower bound, they are under-performing, if their observed score
lies above the upper bound, they are over-performing and if their observed score
lies within the interval, they are performing as expected. This classi�es a student
as lying under, over or within their prediction interval, and for each school, the
number of students in each of these categories can be counted. From the de�nition
of prediction intervals, we expect approximately 50% of students to lie within their
50% prediction intervals.
6.1.1 Prediction Intervals for lmer
To calculate the prediction intervals for students from the lmer model, �rst recall
the statistical theory of prediction intervals based on linear regression.
Consider the model
Y = Xβ + Zb+ ε
where Xβ represents the �xed e�ects of the student variables, Zb represents the
random e�ects of the school variables and ε is noise. From the lmer output, we
have the estimated parameters β, b, ˆV ar(b) and ˆV ar(ε). Suppose we wish to
predict values for an individual with data X0 and Z0. The predicted value would be
given by
Y0 = X0β + Z0b+ ε
and the estimated expected value is
E[Y0] = X0β + Z0b.
Assuming independence, the estimated variance is
ˆV ar(Y0) = V ar(X0β + Z0b+ ε)
= V ar(X0β) + V ar(Z0b) + V ar(ε)
= X0V ar(β)XT0 + Z0V ar(b)Z
T0 + V ar(ε)
where we have estimates for V ar(β), V ar(b) and V ar(ε). The approximate 50%
prediction interval is taken to be 0.67 standard deviations from the predicted value
(Y0 − 0.67
√ˆV ar(Y0), Y0 + 0.67
√ˆV ar(Y0)).
Chapter 6. Model Validation 161
The cut-o� value of 0.67 corresponds to a 25%-50%-25% strati�cation of a standard
normal distribution.
From the hierarchical model �tted in lmer (Section 4.3), we calculate prediction
intervals for each student in our data set. In reference to the statistical theory
outlined above, the data for the predicted values is X0 = X since we are predicting
for the same data on which the model was �tted. We then classify each student into
under-performing, over-performing or within, depending on whether their observed
NN Rasch score falls below, above or in the 50% prediction interval. The counts
and proportions of students in each of the three categories is given in Table 6.1.1.
Table 6.1.1: Counts and proportions of students in each performance category fromthe lmer model
Category Count ProportionOver 4 429 0.22Within 11 131 0.55Under 4 564 0.23
From the column of proportions in Table 6.1.1, we observe that approximately 50%
of students' observed scores fall within their 50% prediction bands which is to be
expected. It is important to note that the �z-score� of 0.67 is only an approximation,
as it is not reasonable to calculate a t-test statistic for the same reason why P -values
are not calculated in lmer (Section 4.3).
6.1.2 Prediction Intervals from Stan
To calculate prediction intervals for students' scores in Stan, we simulate 3000 scores
from the model for each of the 20 124 students in the data set. These simulations
give a posterior distribution for the score of each student, and from the posterior
distribution, we take the 50% posterior density interval as the 50% prediction interval
for a student's score.
The same classi�cation of students in Section 6.1.1 as under-performing, over-performing
or within, is applied to the observed data and the prediction intervals from Stan.
Table 6.1.2 gives the counts and proportions of students in each of the three cate-
gories.
162 6.1. Student-level Prediction
Table 6.1.2: Counts and proportions of students in each performance category fromthe Stan model
Category Count ProportionOver 4 653 0.23Within 10 686 0.53Under 4 785 0.24
As before, approximately 50% of students' observed scores fall within their 50%
prediction bands.
6.1.3 Comparison of lmer and Stan
We have results for the performance of students from lmer and Stan and now wish
to assess model agreement. Table 6.1.3 gives the 3×3 table of the counts of studentsunder each of the performance categories for both models.
Table 6.1.3: 3×3 table of the counts of students for performance categories underlmer and Stan models
Stanlmer Over Within UnderOver 4 403 26 0Within 250 10 641 240Under 0 19 4 545
The values along the diagonals of Table 6.1.3 are those students who are classi�ed
into the same group under both models. The o�-diagonal entries are of interest as
they are students for which their classi�cation di�ers depending on lmer or Stan.
This identi�es a di�erence between the models in the calculation of �tted values or
prediction intervals, and hence, the classi�cation of certain students.
From the calculated prediction intervals, we obtain the number of students in the
performance categories of Over, Within and Under for each individual school. We
then use this data and a χ2 test to identify whether the school has a signi�cant e�ect
on the number of over-performing, under-performing and within students. A more
Chapter 6. Model Validation 163
appropriate test would be Fisher's exact test because of the small counts for some
variables in schools, but Fisher's exact test is intractable for the large 3 × 401 table
created by three performance categories and 401 schools.
For the lmer model, Pearson's χ2 test gives a test statistic of 688.92 and a P -value of
0.9981. Similarly for the Stan model, Pearson's χ2 test gives a test statistic of 680.93
and a P -value of 0.9991. We conclude that school is not a signi�cant predictor of
the counts in each of the performance categories since we retain the null hypothesis
at a 5% signi�cance level for both models.
Another way to compare models is to compare the �tted values for student scores
under the lmer and Stan models. Figure 6.1.1 plots the Stan �tted values against
the lmer �tted values, and we observe the close alignment between the �tted values.
The colours of Figure 6.1.1 are de�ned from Table 6.1.3 where students are coloured
depending on whether they are classi�ed as Over (O), Within (W) or Under (U) by
the lmer (L) and Stan (S) models.
The top plot in Figure 6.1.1 is of all students who have no discrepancy in classi�ca-
tion, and the data points lie very close to the line of equality. Looking at the bottom
plot which contains the students whose classi�cations di�er, we see that the �tted
values of lmer and Stan agree well in most cases, with only a few points standing
out. These plots indicate that the lmer and Stan models agree in general in terms
of �tted values for students. It is the calculated prediction intervals which appear
to change the classi�cation of students, and since we have commented on the fact
that the lmer prediction intervals are only approximate, we choose the Stan model
over lmer, and all further analysis is with the results from the Stan model.
164 6.1. Student-level Prediction
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●●
●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●●
●●●●
●●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●●●●
●
●
●
●
●
●
●●●●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●
●●
●●
●●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●●
●●
●
●
●
●
●
●●●
●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●
●●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●●●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●●
●●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●●●
●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●
●●●●
●
●
●
●●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●●
●●
●●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●●
●●
●
●
●
●●●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●●
●
●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●●
●
●●
●
●
●●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●●●●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●●●
●
●
●
●●
●
●
●●
●
●●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●●●
●
●
●
●
●●●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●●
●●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●●●
●
●
●●●●
●
●
●
●●
●
●
●●
●
●
●●
●●
●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●●●
●
●
●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●●●
●
●
●●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●●●●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●●●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●●
●
●●●●
●
●
●●
●
●
●
●
●●
●●
●
●
●
●
●
●●●●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●●
●
●
●●●●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●●
●
●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●●
●●
●●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●●
●
●
●
●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●
●
●●●
●
●
●
●●
●●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●●
●
●
●●
●
●
●●●
●
●
●●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●●●●
●
●●
●
●
●
●
●●
●●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●●●
●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●●
●●●●
●
●
●
●
●●
●
●●
●●
●
●
●
●●
●
●
●
●
●●
●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●●●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●●
●
●●●
●
●
●●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●
●
●
●●
●
●
●●
●
●
●
●
●●
●●●
●●●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●●
●
●●
●
●
●●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●
●●
●●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●●
●
●●●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●●●
●
●
●
●●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●●
●
●●
●
●
●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●●
●●
●
●●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●●●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●●●
●
●●
●
●●
●●●
●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●●●
●●
●
●●●●●●●
●●●●●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●●●●
●●
●●●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●●●
●
●●●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●●
●
●
●●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●●
●●
●
●
●●●●
●
●●
●
●●
●
●●
●●
●●●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●●
●●
●
●
●●
●●
●
●
●●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●●
●●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●●
●●
●
●●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●●●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●●●
●
●
●●●
●●●
●●●
●
●
●●●●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●
●
●●
●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●●
●
●
●
●
●●●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●●
●
●●
●
●●
●
●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●●●●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●●●
●●
●
●
●
●
●●●
●●
●●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●●
●
●●●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●●●
●
●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●●
●
●
●
●●
●
●
●
●
●●
●●
●●●
●●●
●
●●
●
●
●
●
●
●
●
●
●●●
●●
●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●●
●
●
●
●
●
●
●●
●
●
●●
●●●
●●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●●
●
●●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●●●
●
●●
●
●
●
●
●
●
●
●●
●●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●●●
●
●●
●
●●●
●
●
●●
●
●●
●
●
●●●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●
●●
●
●●●
●
●
●●
●
●
●
●
●
●●
●●●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●●
●
●
●
●
●
●
●●●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●●
●●●●●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●●●
●●
●●●
●●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●●
●●●
●
●
●●
●
●
●
●
●●●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●●●
●●
●
●●
●
●●
●
●
●
●
●●●
●
●●●
●
●
●●
●●
●●
●
●●
●
●
●
●
●
●●●
●
●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●●●●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●●●
●●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●●
●●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●●●
●●●
●
●●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●●
●
●
●●●●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●●
●
●
●
●●●
●●●
●
●
●
●
●●●●
●●
●
●●●
●●
●
●
●
●
●●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●●
●
●●
●●
●
●
●
●●
●
●
●●
●
●
●
●●●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●●
●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●●
●●
●●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●
●●●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●●
●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●●
●●
●
●
●
●●
●
●
●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●●●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●●
●
●
●
●●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●
●
●
●
●
●●●
●●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●●
●●●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●●
●
●
●
●●
●●
●●
●●
●
●
●●●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●●●●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●●
●●
●
●●
●
●
●●
●
●●●
●
●●
●●
●
●
●
●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●●●
●●
●●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●●
●●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●●●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●●●
●
●●
●
30
40
50
60
70
30 40 50 60 70lmer fitted values
Stan
fitt
ed v
alue
s
Categories●
●
●
SOLOSULUSWLW
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●●
●●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
30
40
50
60
70
30 40 50 60 70lmer fitted values
Stan
fitt
ed v
alue
s
Categories●
●
●
●
SOLWSULWSWLOSWLU
Figure 6.1.1: Plot of the Stan �tted values against the lmer �tted values colouredby classi�cation cateogories.
Chapter 6. Model Validation 165
6.2 Analysis of School E�ect
We have divided students into three categories - over-performing, within and under-
performing. If we look more closely at the proportion of students in these categories
for each school, we can rank schools in order of their proportion of over-performing
students. Figure 6.2.1 plots these ranked non-zero proportions and the 95% con�-
dence intervals of the proportion.
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
100
200
300
400
−0.2 0.0 0.2 0.4 0.6 0.8 1.0 1.2Proportion of Students
Ran
k
Stan: Over−performing Students of Schools
Figure 6.2.1: Ranked proportion of students who are over-performing in a schoolplotted with a 95% con�dence interval. The line y = 0.25 is given for reference andthe colour green highlights schools whose con�dence interval lies completely above0.25 while the colour red highlights schools whose con�dence interval lies completelybelow 0.25.
We would expect that approximately 25% of students in a school would be over-
performing, deduced from the 50% prediction interval, and a vertical line is drawn
at 0.25 to illustrate how schools perform with respect to this expectation. If a
school's 95% con�dence interval lies above the line at 0.25, we can conclude that that
166 6.2. Analysis of School E�ect
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
100
200
300
400
−0.2 0.0 0.2 0.4 0.6 0.8 1.0 1.2Proportion of Students
Ran
kStan: Under−performing Students of Schools
Figure 6.2.2: Ranked proportion of students who are under-performing in a schoolplotted with a 95% con�dence interval. The line y = 0.25 is given for reference andthe colour red highlights schools whose con�dence interval lies completely above 0.25while the colour green highlights schools whose con�dence interval lies completelybelow 0.25.
particular school has a signi�cant proportion of over-performing students. These
schools are highlighted in green in Figure 6.2.1, and there are three such schools.
Note that these three schools have over-performing students as their total student
population, giving a proportion of one and a non-existent con�dence interval. In
the case where the proportion of over-performing students is either zero or one, the
exact binomial calculation can be used instead.
If a school's 95% con�dence interval for the proportion of over-performing students
lies completely below the reference point of 0.25, we conclude that that particular
school has a signi�cantly low number of over-performing students. These schools
are coloured red in Figure 6.2.1, and there are 43 such schools.
Chapter 6. Model Validation 167
We also consider the proportion of under-performing students in schools in Figure
6.2.2. There are four red schools with a signi�cantly higher proportion of under-
performing students, and 32 green schools which have a signi�cantly lower proportion
of under-performing students. The four red schools are small schools, and either
75% or all the students are under-performing. Minimising the number of under-
performing students in a school is an aim, like having a signi�cant number of over-
performing students.
In Section 6.1, we performed a χ2 test on all schools and their counts of Over, Within
and Under students. The P -value of 0.9991 led us to the conclusion that school
does not have an overall signi�cant e�ect on the strati�cation of students in the
performance categories. We then look at a χ2 test on the proportion of counts being
(Over, Within, Under) = (0.25, 0.5, 0.25) in each individual school. This is to assess
the goodness of �t of a multinomial distribution for the proportion of counts in each
school. To perform multiple hypothesis testing, there are a number of methods, one
of which is to calculate Bonferroni adjusted P -values. Bonferroni adjusted P -values
correct for familywise error rate, but are very conservative. Another method is to
convert the P -values to q-values which take into account the positive false discovery
rate (see Storey [44] for the statistical theory of q-values). The calculated q-values
are all larger than 0.05, so at the 5% signi�cance level, we conclude that there is
no signi�cant association between school and the number of over-performing, within
and under-performing students.
This method answers the question of whether the proportion of over-performing or
under-performing students in a school is signi�cantly di�erent from the expected
value of 0.25. From the χ2 test however, we cannot conclude that signi�cant pro-
portions are due to a school e�ect. This then provides the avenue of investigating
more subtle issues and the e�ect of other student and school covariates.
The conclusion that school does not have a signi�cant e�ect is an indication that the
model is working and �tting the data well. The validation of the model con�rms that
as we expect, schools are performing as they should, and the model has satisfactorily
captured and explained the relationship between students' NN Rasch scores and their
student and school characteristics. As stated, each school has a random e�ect to
explain its e�ect on a student's expected performance, the variation of this random
168 6.2. Analysis of School E�ect
e�ect (standard deviation of 2.41) could be further explained by predictive factors
not presently contained in the data set, for example, pedological philosophies of the
teachers at the schools. This unexplained random e�ect varies between schools and
is de�ned to be γ0 in equation (6.0.1), that is, the di�erence in the observed and
estimated school-level linear regression. These γ0s can be extracted from the model
�t and a normal Q-Q plot of the random e�ects estimates is given in Figure 6.2.3.
●
●
●●
●
●●●●●●●●
●●●●●
●●●●●●●
●●●●●●●
●●●●●●●●●●
●●●●●●●●●●
●●●●●●●●●
●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●
●●●●●●●●●●●●●
●●●●●●●●●●●●●●●
●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●
●●●●●●●●●●
●●●●●●●●●
●●●●●●●●●
●●●●●●●●
●●●●●●●
●●●
●
●●
●
●
−4
0
4
8
−2 0 2Theoretical Quantiles
Sam
ple
Qua
ntile
s
Figure 6.2.3: Normal Q-Q plot of the random e�ects estimates.
We observe that the points in the normal Q-Q plot follow quite a strong linear
relationship, especially for the majority of the points. There is the possibility of
the highest, rightmost point being an outlying school, and such a school could be
investigated further.
The fact that the random e�ects for school are normally distributed, as assumed for
the model, is further con�rmation that we have a well-�tting and accurate model.
We wish to explain this currently �unexplained� random e�ect, and to do so, would
Chapter 6. Model Validation 169
require more data. Ideally, the most meaningful data would be data on variables
which can be changed and controlled. Our current data is of variables over which
students, schools or education authorities have no control, for example, they cannot
change or control the disability status of a student, their parental occupation or the
location of the school. It would be of great interest to have policy-controlled data,
for example, the level of sta�ng at a school, teacher mobility or the implementation
of a particular teaching program, so that we can analyse the e�ect these investments
and controlled factors have on student performance.
6.3 Heteroscedasticity and School Size
In Section 3.2.5, we looked at the heteroscedastic nature of the residuals from the
model �tted on transformed data and school size. A similar analysis is done here
where the variance of the student residuals for each school is plot against school size
(Figure 6.3.1). For each school, the variance of the student residuals is de�ned to
be
MSE =
∑nj
i=1(yi − yi)2nj − 1
for student i = 1, . . . , nj and school j = 1, . . . , J where nj is the number of students
in school j, yi is the observed student score and yi is the �tted or expected student
score from the model.
There is a distinct funnel-shape to the plot which indicates heteroscedasticity be-
tween the residuals and school size. This relationship between the residuals and
school size suggests that we need to account for school size in the model. One
solution is to stratify schools into small schools and large schools and then �t the hi-
erarchical model on each of these groups of schools separately to reduce the in�uence
of school size. Another idea would be to incorporate school size into the original
model to hopefully weight the model or compensate for the di�erence in school size.
Finally, the predicted quartile interval for the students of a school may be adjusted
by the school's variance of residuals to account for the variation between schools.
All of these suggested methods are avenues for future research in addressing this
heteroscedasticity.
170 6.3. Heteroscedasticity and School Size
●
●●●
●
●●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●● ●
●●●
●
●
●
●●
● ●
●
●
●●●
●●
●● ●
●
●
●
●
●
●
●
●
●●
●●
●●
●
●
●● ●
●●
● ●
●
●
●
●
●
●
●
●
● ●
●
●●
●
●●
●
●
● ●●
●
●
●●
●
●●
●
●
●
●●
●
●
●
●
●●
●
● ●
●● ●●
●
●
●●
●
●
● ●
● ●●●
●
●●
●
●
●
●● ●
● ●● ● ●●
● ●●●
●
●
●●
●
●
●
●●
●●●
●●
●●
●●
●●
●
● ●
●●
●●
●●
●●
●
●●
●●●
●
●
●
●●●
●● ●
●
●
●
●
●●
●●
●
●●
●
●●
●●●
●● ● ● ● ●●
●
●
●
●●
●
●
● ● ●●
●●
●●●
●● ● ●●● ●
●
●●●
●
●●
●
●
●●
●● ●●
● ●●
●●
●●● ●
●●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●● ●
● ●●●
●
●
● ●●
●●●
●●●
●●
● ●●
●
●
● ● ●●●●●
●●●●
●●
●●
●●
●●
●●● ●●●
●
●● ●
●
●●●
●●
●
●● ●
●●
●
●●
●
●
●● ●● ●
●●
●●
●
●
●●
●●●
●●
●
●●● ● ●●
●●
●● ●
●●
●●
●●
●
●●
●
●
●●
●●●
0
200
400
600
800
0 50 100 150 200School Size
Var
ianc
e of
Res
idua
lsStan
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●● ●
●
●
●
●
●
●
●
●●
●●
●●
●
●
●
●
●
●
●
●●
●
●● ●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●● ●
● ●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
● ●
●
●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
0
50
100
150
0 50 100 150 200School Size
Var
ianc
e of
Res
idua
ls
Stan
Figure 6.3.1: Plot of the variance of the residuals for each school against school size(the bottom plot has the point at (1,778) removed to zoom in on the other points).
Chapter 6. Model Validation 171
6.4 Conclusion and Impact
It is also important to note that the classi�cation of students can highlight the
exceeding of expectations due to background factors. An over-performing student is
not necessarily the student with the highest score in a school; it could be a student
who has a socially disadvantaged background, but their scores are not hindered and
they are classed as over-performing. Figure 6.4.1 gives the example of school 115
where there is a student who is over-performing with a score of 67.34.
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
40
50
60
70
80
StanModel
Scor
e Indicator●
●
●
WithinOverUnder
Figure 6.4.1: Student scores for school 115 coloured by whether the students areclassi�ed as over-performing, within or under-performing.
From this analysis, we can accurately predict a student's score given their student
and school characteristics. This student-centric approach would be of particular
use to anyone who wishes to determine which school is best for an individual, for
example a parent for their child, as we can calculate their predicted score for each
school under the �nal Stan model and compare results.
172 6.4. Conclusion and Impact
By the end of this chapter, we have achieved a model which is the �best�, out of
all we considered. The di�erence between the results which are reported from NA-
PLAN scores by the My School website's rankings is that for each school, My School
agglomerates all the students' results into a single measure or average and then
compares the school to other �statistically similar� schools. In stark contrast, our
methods takes an individual student, compares them to other students, classifying
them into over-performing, within or under-performing and then identi�es whether
a school has a signi�cant proportion of students in each category. The power of this
model is its emphasis on students and their individual performance to give a person-
alised model aggregated to schools. This makes the model very versatile as it models
and assesses the performance of students and schools, based on the performance of
their students.
Chapter 7
Initial Longitudinal Analysis
In Chapter 1, we discussed the motivation for value-added measurement. This
concept is related to the longitudinal aspect of the data. It is the usual practice for
students to remain in the school system and progress from Grade 3 to Grade 5 to
Grade 7, and during the course of their education, sit all three tests. Hence, there is
an underlying commonality or correlation between tests if they are sat by the same
person. Longitudinal modelling incorporates this extended structure of repeated
measures over time. For these students, we can take the di�erence in sequential
scores and analyse the improvement of students over time to see whether the �value�
added to their education is due to a school e�ect.
In the Basic Skills Test data set, individual students are uniquely identi�ed by the
combination of their school ID number and student number (Section 2.2). As a
result, we cannot track students who move between schools but are restricted to
tracking students only within a school.
7.1 Summary Statistics of Data from Sequential Tests
Students can either sit two sequential tests - Grades 3 and 5 or Grades 5 and 7 -
or three sequential tests - Grades 3, 5 and 7. First, we clean the data by removing
students who have multiple scores recorded for the same grade. For example, a
student has a Grade 3 result in 2000 and a Grade 3 result in 2001 as they have
re-sat the Grade 3 test. As we are primarily interested in the progress of students
173
174 7.2. Grade 3 and Grade 5 Tests
between sequential tests, that is, between Grades 3 and 5 and Grades 5 and 7, we
disregard these repeated scores. Some students have scores recorded for Grade 3
and Grade 7 but not Grade 5. The di�erence in these scores are not comparable
to the di�erence in scores of sequential tests and so are removed from the data set.
Finally, there are 31 students who sit sequential tests which do not occur biennially,
for example, they skip a year and sit tests in consecutive years, or they are held
back a year and sit another test three years after the previous one. These students
are also removed during the process of cleaning the data.
From the cleaned data set, 8 357 students from 401 schools have two appropriate
sequential test entries, and the break-down into whether they are Grade 3 and Grade
5 scores or Grade 5 and Grade 7 scores is given in Table 7.1.1.
Table 7.1.1: Counts of students who sat two appropriate sequential tests
Type CountsGrades 3 & 5 8 143Grades 5 & 7 214
Total 8 357
In the cleaned data set, there are now 487 students from 158 schools with three
appropriate recorded scores. These scores can be broken down into two sets of two
sequential tests, or by taking a more fundamental longitudinal analysis approach,
another recorded score at a third time point (Grade 7) can be added to the longi-
tudinal model. This data will be addressed in Section 7.3.
For the moment, we shall use only the data of Grade 3 and Grade 5 test scores to
illustrate statistical methods for the simplest case as proof-of-concept for how this
time-dependent data can be analysed.
7.2 Grade 3 and Grade 5 Tests
In the data of all students who sat a Grade 3 and Grade 5 test two years apart, there
are 8 143 students from 400 schools. There are seventy-nine schools which have less
than �ve students recorded in this data set of matched Grade 3 and Grade 5 scores,
Chapter 7. Initial Longitudinal Analysis 175
and as it may be di�cult to estimate the school e�ect from such small sample sizes,
we remove these schools from the data to give 7 907 students from 321 schools. The
univariate summary statistics of this data set are given in Appendix B.2.
7.2.1 Individual Scores
The most important relationship to plot for longitudinal data on multiple subjects is
the trend of the response over time by student. We plot the actual Rasch scores for
each individual student in each school to see the trend of scores over grades (Figure
7.2.1). Each panel is a school, and the connected data points are for each individual
student. The axes are constant for all the panels, which allows for examination
of the time trends within students and for comparison of these patterns between
schools. Figure 7.2.1 plots the data for only schools 33 to 213, but similar plots can
be done for all schools and their students. An observation is that Rasch scores can
sometimes decrease over time and between tests. These are highlighted in red in
Figure 7.2.1, and in total, there are 458 students who exhibit this decrease in the
scores of two sequential tests.
Reasons for this decrease in Rasch scores are various, but we are interested in
whether school e�ect is a cause. We investigate this further by using the di�er-
ence in scores for each student.
7.2.2 Di�erence in Scores
For each student, we calculate the di�erence in NN Rasch scores between Grade 3
and Grade 5. This is a measure of a student's change over time. In accordance with
the theory of Rasch modelling, we expect an increase in Rasch scores from Grade 3
to Grade 5 to Grade 7.
Figure 7.2.2 plots for each individual student, the di�erence in the Grade 3 and
Grade 5 scores against the average of their Grade 3 and Grade 5 scores. From this
plot, we see that there is no observable trend between how well students do on tests
and how much they improve between tests. Good students can still improve greatly,
and overall improvement does not seem to be directly related to the ability of the
student, as measured by their average Rasch scores.
176 7.2. Grade 3 and Grade 5 Tests
●●
●
●
●●
●
●●
●● ●
●
●●
●
●
●● ●●
●
● ●
●
●
● ●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
● ●●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●●●
●
●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
● ●●
●
●
●
●
●
●
●●
●
●
●●
●
●
● ● ●
●
●●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●●
●●
● ●●
●
● ●●
●
●
●
●
●●
●
●●
●
●
●
● ●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●● ●
●
●●
●●
●●
●
●
●
●
●●
●
●●
●
●
●●
●● ●
●
●● ●
●●● ●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●● ●● ●
●
●●
●●●
●●●
●
●●
●
●●
●
●
●
●
●
●
●
● ●
●●●●
●
●
●
●
●
●●
●
● ●●
●
● ●
●
●
●●
●
●
● ●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●● ●
●
●●
●
●
●
●
●●
●
●
●
●
●●●
●
●●
●
●
●●
● ●●
●
●
●
●●
●
●
●●
●●
●●
●
●
●●
●
●
●
●
●
● ●
●
●●●
●
●
●
●
●
●●
●
●●
●●●
●
●
●●
●●
●
●
●
●
●●
●
●
●● ●●
●
●
●
●
●●
●
●
●● ●●
●
●
●
●
●
●●
●
●
● ●
●
●
●●
●
●●
●
● ●●
●
●●
●
●
●●
●
●● ●
● ●●●
●
●
●●
●
●
●
●● ●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
● ●●
● ●
●
●
●
●
●
●
● ●●●●
●
●
●
●
●● ●●
●
●
●●
●
●●
●
●●
●●
●
●
●
●
●●
●
●●
●
●●
●
● ●
● ●
●●● ●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●●●●
●
●
●●●
● ●
●
●●●
●●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●●
●●
●●
● ●
●
●
●●
●
●●
●
●●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●●
●
●●● ●●●
●
●
●
●●
●
●
●
●
●
● ●
●
●
●
●
●●
●●
●
●
● ●● ●●
●
●
●
●
●
●
●
●●
●●
●●
●●
●
●
●
●
●
●●
●
●●
●●●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●●
● ●●●
●●
●●
●
●●●
●
●
●
●
●●
●
●
●
●●
●●● ●
●●
●●
●
●●
●●
●
●●
●
●
●●
●
●
●
●
●
●● ●
●
●●
●●
●●
●
●●
●
●
●
●
●
●
●
●●
●●
●
●●
●● ●●
●●
●
●
●
●
●●
●
●
●
●●
●
●
●●●
●
●
●●
●
●
●
●
●● ●
●●
●
●
●
●●
●
●
●●
●
●●● ●
● ●●
●
●●
●●●●
●
●●
●
●
●●
●
●●
●●
●
●●
●
●
●
●
●● ●●
●● ●
●
●●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●● ●
●●
● ●●
●
●
●●
●●
●
●●
●●
●●
● ●● ●
● ●●●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●
● ●
● ●
●
●
● ●
●●
●
●
●
●
●
●
● ●
●
● ● ●●
●
●
●●
●●●
● ●
●
●
●
●
●
●● ●●
●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●●●
●
●
●
●●
●
●
●
●
●
●
●● ●●●
●
●●●
●
●
● ●
●●
●
●
● ●
●●
●
●
●●
●
●
●
●
●
●●
●● ●
● ●●
●●
●●●
●
●
●
●
● ●
●
●
●●
●●
●●
●
●●
●
●
●● ●● ●● ●●
●
●
●
●
●
●
●● ●
●
●
●
●
● ●
●●
●
●●
●
●
●● ●
●
●●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●●
●●
●●
●
●
●
●●
●
●
●
● ●
●
●
● ●
●
●
●
●
●
●
●
●●●
●
● ●
●
● ●●
●
●
●●
●
●
●
●
● ●
●● ●
●
●●
●●
●
● ●●
●
●●● ●●●
● ●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
33 41 49 61 67 73 75 77 79 82
84 85 89 91 94 95 96 97 102 105
106 110 113 114 120 123 125 127 128 130
134 135 142 146 147 148 149 151 152 153
155 157 160 161 162 164 167 168 169 171
173 182 183 184 185 186 187 188 189 190
191 192 194 199 200 201 205 206 207 213
40
60
80
40
60
80
40
60
80
40
60
80
40
60
80
40
60
80
40
60
80
3 5 3 5 3 5 3 5 3 5 3 5 3 5 3 5 3 5 3 5Graded year
NN
Ras
ch Indicator●
●
Decrease
Increase
Figure 7.2.1: Plot of the Rasch scores of students where each panel is a school andthe colour of the points denote an increase or decrease from Grade 3 to Grade 5.
As each individual student sits two tests and has two scores recorded, it is a matched-
pairs design. We then take the di�erence to form a single sample of the di�erences
and test to see whether the school means are signi�cantly di�erent from each other
using ANOVA (Table 7.2.1). Figure 7.2.3 is a plot of the mean di�erence in each
school, and from this plot, there are no schools which have a negative mean di�er-
ence.
From Table 7.2.1, the P -value is less than 0.05, and at the 5% signi�cance level,
schoolno is a signi�cant predictor of the di�erence in Grade 3 and Grade 5 scores.
Chapter 7. Initial Longitudinal Analysis 177
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●
●●
●
●● ●
●
●
●●
●
●
●
●● ●
●
●
●
●
●
●
●
●
●
● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●●
●
●
●
●
● ●
●
●
●●●
●●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
● ●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●
● ●
●
●●
●
●
●
●
● ●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●● ●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●●
● ●
●●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●●
●
●●
●
●
●
●●●
●
●
●
●
●●
●●
●
●
● ●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
● ●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●●
●
●
●
●●
●
●
●
●●
● ●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●
● ●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
● ●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
● ●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
● ●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●●
●●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●●
●● ●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
● ●
●
●
●●
●
●
●
●
● ●
●
●
●
●●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●●
●
●●● ●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
● ●
●
●●
●
●
●
●
● ●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
● ●●
●
● ●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
● ●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●●
●●
● ●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
● ●●
●
●
●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●
●
●●
●
●
●
●
●●
● ●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●●
● ●●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
● ●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
● ● ●
●
●●
●
●
●
●
●
●
●
●
●
●● ●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
● ●
●
●
●●
● ●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●● ●
●
●
●●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
● ●
●
●
●
●●
●●●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●● ●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●
● ●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●● ●
●
●
●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●●
●
●
●
●
●
●●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●● ●
●
●
●
●
●
●
●
●
●●●
●●
●●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●
●
●
● ●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
● ●●
●
●
● ●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●
●●
●●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●
●●
●
●
●●
●●
●●
●
●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●●
●●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●
● ●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●●
●
●●
●
●
●
● ●
●●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●
●
●●
●
●●
●
●
●
●●
●
●●
●
●●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●●
●●
●
●
● ●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●
●
●
●●
●
●●
●
●●
● ●
●
●
●
● ●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●●
● ●
●
●
●
●●
●
●
●
●
●●
●
●
●
● ●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
● ●
●
●●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●●
●
●●
●●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●●
●
●
●
●●
●●
●
●●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
● ●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
● ●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●●●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●● ●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
● ●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
−20
0
20
40
20 40 60 80Average in Grade 3 and 5 scores
Diff
eren
ce in
Gra
de 3
and
5 s
core
s
Figure 7.2.2: Di�erence versus Average in Grade 3 and Grade 5 scores.
Table 7.2.1: Summary output of ANOVA between Grade 3 and 5 scores andschoolno
Df Sum Sq Mean Sq F -value P -valueschoolno 320 35 087 109.65 3.301 < 2.2e-16Residuals 7 586 251 997 33.22
178 7.2. Grade 3 and Grade 5 Tests
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●●
●
●
●
●●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
5
10
15
100 200 300 400 500 600schoolno
mea
nGrades 3−5
Figure 7.2.3: Plot of the mean di�erence in Grade 3 and Grade 5 values in eachschool.
Simple Linear Regression
Having established that schoolno is a signi�cant predictor, we wish to assess which
other student, school or test predictor variables are signi�cant in predicting this
di�erence in scores. We �t a simple linear regression as given in the model below
where Grade 3 Rasch is the Grade 3 NN Rasch score of the student and proc is the
value for procyear for a student's Grade 3 test.
(Grade 5 - Grade 3)= Grade 3 Rasch+ schoolno+ proc+ gpokm+ isolation+
spatial_ar+ staff_metr+ atsi+ lbote+ gender+ aboriginal+ disability
+ school_car + occupation + school_edu + non_school + p_g_gender +
p_g_nesb + home_langu.
Chapter 7. Initial Longitudinal Analysis 179
As in Chapter 3, there is collinearity between the schoolno variable and the school
covariates of gpokm, isolation, spatial_ar and staff_metr which results in the
un-identi�ability of the above model. We then split the model up into the school-
number model and the school-covariates model, and the summary output of the
linear regressions are given in Table C.2.1 (Appendix C) and Table 7.2.2 respec-
tively.
School-Number Model
Under the school-number model, all of the regression coe�cients are compared to
the baseline category or intercept of a female student in school 33 who sits the
Grade 3 test in 2000 and does not come from an Aboriginal, Torres Strait Islander
or non-English speaking background, English is their home language and they are
not identi�ed as having a disability or School Card. Their primary guardian or
parent is female from an English speaking background whose status of occupation,
school education and non-school education are not stated. From Table C.2.1 in
Appendix C, there are six schools out of the 321 schools which are signi�cant, and
the signi�cant predictors are Grade 3 Rasch, proc, gender and disability at the
5% signi�cance level. This is another example of how the test variables of Grade 3
Rasch and proc are signi�cant even though there should be no statistically signi�-
cant di�erence according to the theory of Rasch modelling. The other two variables
which are signi�cant are disability and gender. It is reasonable that having a
disability is signi�cant as it is a condition which greatly a�ects a student's progress
educationally and has the potential to either �uctuate over time or have a greater
in�uence at di�erent stages of their lives. It could be possible that a disability has a
greater e�ect on younger students as they are still developing and learning to cope
with their disability. From the regression coe�cient of -1.9197 in Table C.2.1 in
Appendix C, having a disability results in lower Rasch scores by almost two units,
as expected. Gender is also statistically signi�cant with males having Rasch scores
which are 0.342 units higher on average (P -value of 0.0326) than females. Recall
that we are considering NN Rasch scores in this analysis, and boys generally perform
better at Numeracy compared to females. The signi�cance of gender also indicates
that over time, boys not only perform better, but their rate of improvement is also
higher - the development rate di�ers between males and females.
180 7.2. Grade 3 and Grade 5 Tests
School-Covariates Model
Under the school-covariates model, all of the regression coe�cients are compared to
the baseline category or intercept of a female student who sits the Grade 3 test in
2000 and does not come from an Aboriginal, Torres Strait Islander or non-English
speaking background, is not identi�ed as having a disability or School Card and their
primary guardian or parent is female from an English speaking background whose
status of occupation, school education and non-school education is not stated. The
characteristics of the reference school is one with an isolation factor of 1, spatial
value of 1.1 (metropolitan) and metro sta� classi�cation. From Table 7.2.2, the
signi�cant variables are Grade 3 Rasch, proc, aboriginal, disability, school_car
and non_school at the 5% signi�cance level.
As before, we can combine both the school-number model and the school-covariates
model using hierarchical modelling, whether through linear multilevel mixed e�ects
models or Bayesian techniques.
Chapter 7. Initial Longitudinal Analysis 181
Table 7.2.2: Linear regression output of school-covariates model
Estimate Std. Error t-value P -valueIntercept 28.0276 0.6411 43.71 0.0000Grade 3 Rasch -0.3986 0.0109 -36.55 0.0000proc2001 1.7782 0.2406 7.39 0.0000proc2002 -0.1400 0.2743 -0.51 0.6097atsi1 0.9698 1.5676 0.62 0.5362atsiInconsistent -0.4929 0.9614 -0.51 0.6082lbote1 -0.1969 0.6569 -0.30 0.7644lboteInconsistent -0.3536 0.3026 -1.17 0.2426genderM 0.2896 0.1596 1.81 0.0697aboriginalY -2.9556 1.4501 -2.04 0.0416disabilityY -1.8251 0.3821 -4.78 0.0000school_carY -1.5561 0.6101 -2.55 0.0108occupation1 0.3448 0.4905 0.70 0.4820occupation2 0.4459 0.4055 1.10 0.2715occupation3 0.2811 0.3940 0.71 0.4757occupation4 -0.1912 0.3905 -0.49 0.6244occupation8 -0.3818 0.4015 -0.95 0.3416school_edu1 0.2839 0.6268 0.45 0.6506school_edu2 -0.2707 0.5572 -0.49 0.6270school_edu3 0.2073 0.5294 0.39 0.6953school_edu4 0.5201 0.5358 0.97 0.3317non_school5 0.3512 0.4772 0.74 0.4618non_school6 0.5174 0.5201 0.99 0.3199non_school7 1.2320 0.5479 2.25 0.0246non_school8 0.1218 0.4611 0.26 0.7917p_g_genderM -0.1446 0.2196 -0.66 0.5103p_g_nesbY 0.0276 0.3895 0.07 0.9436home_languY 0.6151 0.4089 1.50 0.1326gpokm -0.0045 0.0032 -1.40 0.1620isolation1.5 0.5138 1.4396 0.36 0.7212isolation2 -3.0177 3.2234 -0.94 0.3492isolation2.5 -1.5420 3.2291 -0.48 0.6330isolation3 -1.2058 3.3116 -0.36 0.7158isolation3.5 -1.9639 3.2790 -0.60 0.5493isolation4 -2.7259 3.3290 -0.82 0.4129isolation4.5 -2.6123 3.4033 -0.77 0.4428isolation5 -2.0635 3.6523 -0.56 0.5721isolation5.5 -2.4696 3.6641 -0.67 0.5003isolation6 -4.1386 4.0324 -1.03 0.3048spatial_ar2.2.1 -0.5314 1.4191 -0.37 0.7081spatial_ar2.2.2 0.6797 1.5766 0.43 0.6664spatial_ar3.1 2.0616 1.7254 1.19 0.2322spatial_ar3.2 3.0254 2.0259 1.49 0.1354staff_metrC 2.1576 2.8412 0.76 0.4477
182 7.2. Grade 3 and Grade 5 Tests
Hierarchical Modelling using Linear Multilevel Mixed E�ects Models
Similar to the hierarchical model in Section 4.2, the model we consider is one where
the varying intercept represents the school e�ect and the individual-level predictors
are the student covariates. The school e�ect is then modelled at the group level by
the school covariates. Considering an individual student i, the multilevel model is
(Grade5− Grade3)i = schoolj[i] + β1proci + β2atsii + β3lbotei + β4genderi
+ β5aboriginali + β6disabilityi + β7school_cari
+ β8school_edui + β9non_schooli + β10p_g_genderi
+ β11p_g_nesbi + β12home_langui + εi
schoolj = γ0 + γ1gpokmj + γ2isolationj + γ3spatial_arj + ηj
εi ∼ N(0, σ2y)
ηj ∼ N(0, σ2α)
for student i = 1, . . . , 7907 and school j = 1, . . . , 321. This model is �t using
lmer, and the summary output is given in Table 7.2.3. P -values can be calculated
using MCMC sampling (Table 7.2.4) or the likelihood ratio test (Table 7.2.5). From
these tables, the signi�cant variables are Grade 3 Rasch, proc, gender, aboriginal,
disability and school_car at the 5% signi�cance level, and these are similar to the
signi�cant variables of the school-covariates model. Under the school-number model,
we have discussed the signi�cance of Grade 3 Rasch, proc, gender and disability,
and the same reasoning would apply. Looking at the other signi�cant variables and
their regression coe�cients, being Aboriginal or having a School Card is associated
with lower NN Rasch scores. Scores will be lowered on average by 2.8 units if a
student is Aboriginal, by 1.9 units if a student has a disability, by 1.2 units if a
student has a School Card and by 0.3 if their gender is female. These variables
are similar to disability as they are characteristics which have a great impact on
the progress of students' education, and their e�ect could �uctuate over time, in
particular the allocation of a School Card.
Chapter 7. Initial Longitudinal Analysis 183
Table 7.2.3: Output of �xed and random e�ects from linear mixed e�ects model
Fixed E�ects Estimate Std. Error t-valueIntercept 28.355383 0.660856 42.91Grade 3 Rasch -0.405707 0.010997 -36.89proc2001 1.773802 0.243842 7.27proc2002 -0.149153 0.277424 -0.54atsi1 0.795269 1.541512 0.52atsiInconsistent -0.415879 0.942732 -0.44lbote1 -0.110775 0.647401 -0.17lboteInconsistent -0.403400 0.297961 -1.35genderM 0.318460 0.156389 2.04aboriginalY -2.824746 1.422893 -1.99disabilityY -1.868215 0.376738 -4.96school_carY -1.234884 0.609795 -2.03occupation1 0.117941 0.486178 0.24occupation2 0.316111 0.405328 0.78occupation3 0.154364 0.394346 0.39occupation4 -0.200597 0.391764 -0.51occupation8 -0.363845 0.402761 -0.90school_edu1 0.433085 0.624708 0.69school_edu2 -0.117734 0.555700 -0.21school_edu3 0.295951 0.530276 0.56school_edu4 0.541175 0.535819 1.01non_school5 0.304333 0.470355 0.65non_school6 0.413619 0.512525 0.81non_school7 1.024595 0.541995 1.89non_school8 0.067121 0.453768 0.15p_g_genderM -0.263181 0.224004 -1.17p_g_nesbY -0.089162 0.388065 -0.23home_languY 0.642543 0.411745 1.56gpokm -0.004032 0.004364 -0.92isolation1.5 0.569176 1.946172 0.29isolation2 -0.271445 2.108274 -0.13isolation2.5 0.494194 2.103445 0.23isolation3 0.802914 2.312771 0.35isolation3.5 0.188738 2.261286 0.08isolation4 -0.674492 2.413397 -0.28isolation4.5 -0.389159 2.643973 -0.15isolation5 -0.106571 3.185713 -0.03isolation5.5 -0.400420 3.277458 -0.12isolation6 -2.481292 3.857674 -0.64spatial_ar2.2.1 -0.442645 1.915104 -0.23spatial_ar2.2.2 0.633213 2.128294 0.30spatial_ar3.1 2.109029 2.311667 0.91spatial_ar3.2 3.045106 2.729152 1.12
Random E�ects Name Variance Std Devschoolno Intercept 1.7769 1.3330Residual 22.1866 4.7103
184 7.2. Grade 3 and Grade 5 Tests
Table 7.2.4: Estimates and P -values of �xed and random e�ects estimated byMarkov Chain Monte Carlo sampling
Estimate MCMCmean HPD95lower HPD95upper pMCMC P -valueIntercept 28.36 28.30 27.01 29.60 0.00 0.00Grade 3 Rasch -0.41 -0.40 -0.43 -0.38 0.00 0.00proc 2001 1.77 1.77 1.30 2.27 0.00 0.00proc 2002 -0.15 -0.15 -0.67 0.41 0.58 0.59atsi1 0.80 0.82 -2.11 3.83 0.60 0.61atsiInconsistent -0.42 -0.44 -2.21 1.47 0.64 0.66lbote1 -0.11 -0.13 -1.37 1.14 0.84 0.86lboteInconsistent -0.40 -0.40 -0.99 0.20 0.19 0.18genderM 0.32 0.31 0.00 0.62 0.05 0.04aboriginalY -2.82 -2.84 -5.65 -0.07 0.05 0.05disabilityY -1.87 -1.86 -2.60 -1.10 0.00 0.00school_carY -1.23 -1.30 -2.51 -0.11 0.03 0.04occupation1 0.12 0.16 -0.80 1.10 0.75 0.81occupation2 0.32 0.34 -0.47 1.12 0.41 0.44occupation3 0.15 0.18 -0.59 0.95 0.65 0.70occupation4 -0.20 -0.19 -0.95 0.56 0.62 0.61occupation8 -0.36 -0.36 -1.16 0.42 0.36 0.37school_edu1 0.43 0.41 -0.83 1.58 0.51 0.49school_edu2 -0.12 -0.14 -1.22 0.93 0.80 0.83school_edu3 0.30 0.29 -0.76 1.29 0.59 0.58school_edu4 0.54 0.54 -0.54 1.53 0.31 0.31non_school5 0.30 0.30 -0.62 1.22 0.52 0.52non_school6 0.41 0.42 -0.56 1.43 0.42 0.42non_school7 1.02 1.05 -0.00 2.11 0.05 0.06non_school8 0.07 0.06 -0.84 0.92 0.88 0.88p_g_genderM -0.26 -0.25 -0.67 0.21 0.27 0.24p_g_nesbY -0.09 -0.07 -0.82 0.71 0.86 0.82home_languY 0.64 0.64 -0.17 1.44 0.12 0.12gpokm -0.00 -0.00 -0.01 0.00 0.31 0.36isolation1.5 0.57 0.58 -2.98 4.21 0.75 0.77isolation2 -0.27 -0.33 -4.21 3.51 0.87 0.90isolation2.5 0.49 0.55 -3.19 4.49 0.78 0.81isolation3 0.80 0.85 -3.46 5.00 0.69 0.73isolation3.5 0.19 0.27 -3.92 4.34 0.91 0.93isolation4 -0.67 -0.60 -5.07 3.71 0.77 0.78isolation4.5 -0.39 -0.37 -5.17 4.39 0.87 0.88isolation5 -0.116 -0.00 -5.89 5.66 0.99 0.97isolation5.5 -0.40 -0.33 -6.34 5.51 0.91 0.90isolation6 -2.48 -2.33 -9.39 4.85 0.52 0.52spatial_ar2.2.1 -0.44 -0.48 -3.95 3.08 0.78 0.82spatial_ar2.2.2 0.63 0.62 -3.30 4.59 0.76 0.77spatial_ar3.1 2.11 2.06 -2.21 6.50 0.33 0.36spatial_ar3.2 3.05 3.03 -1.91 8.19 0.24 0.26Groups Name Std.Dev. MCMCmedian MCMCmean HPD95lower HPD95upperschoolno Intercept 1.33 1.13 1.13 0.92 1.33Residual 4.71 4.74 4.74 4.63 4.85
Chapter 7. Initial Longitudinal Analysis 185
Table 7.2.5: Output of �xed and random e�ects from linear mixed e�ects regressionwith P -values calculated from comparing nested models �t by maximum likelihood
Fixed E�ects Estimate Std. Error t-value P -valueLRTIntercept 28.355383 0.660856 42.910000 0.000Grade 3 Rasch -0.405707 0.010997 -36.890000 0.000proc 2001 1.773802 0.243842 7.270000 0.000proc 2002 -0.149153 0.277424 -0.540000 0.585atsi1 0.795269 1.541512 0.520000 0.601atsiInconsistent -0.415879 0.942732 -0.440000 0.651lbote1 -0.110775 0.647401 -0.170000 0.858lboteInconsistent -0.403400 0.297961 -1.350000 0.176genderM 0.318460 0.156389 2.040000 0.042aboriginalY -2.824746 1.422893 -1.990000 0.046disabilityY -1.868215 0.376738 -4.960000 0.000school_carY -1.234884 0.609795 -2.030000 0.039occupation1 0.117941 0.486178 0.240000 0.794occupation2 0.316111 0.405328 0.780000 0.425occupation3 0.154364 0.394346 0.390000 0.683occupation4 -0.200597 0.391764 -0.510000 0.607occupation8 -0.363845 0.402761 -0.900000 0.360school_edu1 0.433085 0.624708 0.690000 0.492school_edu2 -0.117734 0.555700 -0.210000 0.823school_edu3 0.295951 0.530276 0.560000 0.577school_edu4 0.541175 0.535819 1.010000 0.309non_school5 0.304333 0.470355 0.650000 0.514non_school6 0.413619 0.512525 0.810000 0.414non_school7 1.024595 0.541995 1.890000 0.055non_school8 0.067121 0.453768 0.150000 0.880p_g_genderM -0.263181 0.224004 -1.170000 0.246p_g_nesbY -0.089162 0.388065 -0.230000 0.828home_languY 0.642543 0.411745 1.560000 0.117gpokm -0.004032 0.004364 -0.920000 0.336isolation1.5 0.569176 1.946172 0.290000 0.765isolation2 -0.271445 2.108274 -0.130000 0.886isolation2.5 0.494194 2.103445 0.230000 0.804isolation3 0.802914 2.312771 0.350000 0.716isolation3.5 0.188738 2.261286 0.080000 0.930isolation4 -0.674492 2.413397 -0.280000 0.779isolation4.5 -0.389159 2.643973 -0.150000 0.880isolation5 -0.106571 3.185713 -0.030000 0.979isolation5.5 -0.400420 3.277458 -0.120000 0.905isolation6 -2.481292 3.857674 -0.640000 0.516spatial_ar2.2.1 -0.442645 1.915104 -0.230000 0.810spatial_ar2.2.2 0.633213 2.128294 0.300000 0.758spatial_ar3.1 2.109029 2.311667 0.910000 0.348spatial_ar3.2 3.045106 2.729152 1.120000 0.250
Random E�ects Name Variance Std Devschoolno Intercept 1.7769 1.3330Residual 22.1866 4.7103
186 7.3. Grade 3, 5 and 7 Tests
7.3 Grade 3, 5 and 7 Tests
In the cleaned data of students with test results in Grades 3, 5 and 7, all separated
by two years, there are 487 students from 158 schools. When schools with less than
�ve students are removed, there are now 173 students from 20 schools - this is not
much data at all, but this is not surprising, as after the data has been cleaned and
missing data removed, it is highly unlikely that there will be many students in each
school who sat all three tests. The lack of data is also explained by only having one
cohort available with three tests (Table 2.3.7), and all the students sit the Grade 3
test in 2000, Grade 5 test in 2002 and Grade 7 test in 2004.
The univariate counts of the categorical covariates are given in Table 7.3.1. Some
groups of the categorical variables have a small number of data entries which is
important to keep in mind, when making inferences. The proportions of data in
each group of a category are reasonable and what we would expect based on what
the groups represent in real life and past counts from this data set (Section 2.5
and Section 7.2). Some groups of the categorical variables are not even included in
this relatively small data set, for example, some spatial_ar, isolation and lbote
categories.
Figure 7.3.1 includes a plot of the number of students in each school and histograms
of the di�erences in scores between Grade 3 and 5 scores and Grade 5 and 7 scores.
There are not many students in any school, with the maximum number of students
in a school being 19. We also observe that there are some negative di�erences in
scores for both the di�erence in Grade 3 and 5 tests and the Grade 5 and 7 tests.
These students who decrease in Rasch scores between at least one pair of tests over
time, are highlighted in Figure 7.3.2; there are eighteen such students.
We wish to identify students who have a decrease in results as they progress from
Grade 3 to Grade 5 and from Grade 5 to Grade 7. They are bucking the expected
trend of improvement and an increase in Rasch scores for further grades. For three
sequential grades, there are four di�erent possible �trends� (Figure 7.3.3) - the usual
and desired trend is the one in part (A) of Figure 7.3.3 where there is a progression
and increase in scores over time. Any of the other three trends are cause for concern.
As we are considering Rasch scores and they have a continuous scale, we do not
expect scores to remain equal in di�erent grades. Hence, we exclude these scenarios
Chapter 7. Initial Longitudinal Analysis 187
Table 7.3.1: Counts of data for all the categorical predictors in the data of Grade 3,5 and 7 scores
Variable Group Countisolation 1 125
3.5 214 85 75.5 12
spatial_ar 1.1 1252.2.2 293.1 19
staff_metr M 125C 48
atsi 2 1681 3Inconsistent 2
lbote 2 147Inconsistent 26
gender F 88M 85
aboriginal N 170Y 3
disability N 168Y 5
school_car N 170Y 3
home_langu N 170Y 3
Variable Group Countoccupation 0 17
1 12 143 64 118 1NA 123
school_edu 0 171 22 33 164 12NA 123
non_school 0 185 86 87 28 14NA 123
p_g_gender F 132M 41
p_g_nesb N 166Y 7
as highly unlikely with Rasch scores and have four possible trends. These trends are
highlighted in Figure 7.3.2 and in total, there are 13 students which experience a
downward trend between Grade 3 and Grade 5 and four students who have a lower
Grade 7 score than in Grade 5. However, there are no students who have both a
decrease from Grade 3 to Grade 5 and Grade 5 to Grade 7 - that is, no occurrence
of trend (D).
188 7.3. Grade 3, 5 and 7 Tests
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
10
15
400 500 600School Number
Num
ber
of S
tude
nts
0
5
10
15
−10 0 10 20Difference in Grade 3 and 5 scores
coun
t
0
5
10
15
0 10 20Difference in Grade 5 and 7 scores
coun
t
Figure 7.3.1: Plots of the number of students in each school and the distribution ofthe di�erences in scores between Grade 3 and 5 and Grade 5 and 7.
Chapter 7. Initial Longitudinal Analysis 189
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
358 451 461 490 500 527 533 547 548 550
552 556 567 571 584 593 599 600 608 64930
40
50
60
70
80
90
30
40
50
60
70
80
90
3 5 7 3 5 7 3 5 7 3 5 7 3 5 7 3 5 7 3 5 7 3 5 7 3 5 7 3 5 7Graded year
NN R
asch Indicator
●
●
Decrease
Increase
Figure 7.3.2: Plot of the Rasch scores of students where each panel is a school andthe colour of the points denote an increase or decrease between grades over time.
190 7.3. Grade 3, 5 and 7 Tests
(A) (B)
(C) (D)
Figure 7.3.3: (A) Increasing trend from Grade 3 to Grade 5 and Grade 7 (B) Increasebetween Grades 3 and 5 but a decrease between Grades 5 and 7 (C) Decrease betweenGrades 3 and 5 but an increase between Grades 5 and 7 (D) Decreasing trend fromGrade 3 to Grade 5 and Grade 7.
Chapter 7. Initial Longitudinal Analysis 191
7.3.1 Longitudinal Modelling
For each grade, there is the hierarchical structure of the school and student covariates
in�uencing the NN Rasch score as discussed in Chapters 4, 5 and 6. Having three
tests, Grades 3, 5 and 7, there is the added element of time as this hierarchical
structure is repeated at each time point (Figure 7.3.4) in a longitudinal data set.
SYSTEM
SCHOOL
CLASS
STUDENT
TEST
TEST RESULT
PARENT
Grade 3SYSTEM
SCHOOL
CLASS
STUDENT
TEST
TEST RESULT
PARENT
Grade 5SYSTEM
SCHOOL
CLASS
STUDENT
TEST
TEST RESULT
PARENT
Grade 7
Figure 7.3.4: Longitudinal and hierarchical structure of the education system.
With students' results in Grades 3, 5 and 7, these scores can be broken down into
two di�erences between pairs of sequential tests and analysed in the same way as for
the Grade 3 and Grade 5 tests in Section 7.2. However, these separate individual
analyses would not incorporate the underlying factor of the student which links
Grades 3, 5 and 7. We can utilise longitudinal modelling techniques which exploit
the time-dependency structure of the data
The full longitudinal and hierarchical model is
NNRaschit = schoolj[i]t + β1tatsiit + β2tlboteit + β3tgenderit + β4taboriginalit
+ β5tdisabilityit + β6tschool_carit + β7toccupationit
+ β10tschool_eduit + β11tnon_schoolit + β12tp_g_genderit
+ β13tp_g_nesbit + β14thome_languit + εit
schooljt = γ0t + γ1tgpokmjt + γ2tisolationjt + γ3tspatial_arjt + ηjt (7.3.1)
εit ∼ N(0, σ2y)
ηjt ∼ N(0, σ2α)
for student i = 1, . . . , 173, school j = 1, . . . , 20 and time t = 1, 2, 3 where t = 1 is
Grade 3, t = 2 is Grade 5 and t = 3 is Grade 7.
However for the data we have, individual students are uniquely identi�ed by the
192 7.3. Grade 3, 5 and 7 Tests
combination of their student ID and school number. This then means that within
this data set of students who have sat all three tests, a student cannot have moved
school in that time, else, they could not be tracked over time. Hence, the school-
level variables of schoolno, gpokm, isolation and spatial_ar are not dependent
on time, and the model becomes
NNRaschit = schoolj[i] + β1tatsiit + β2tlboteit + β3tgenderit + β4taboriginalit
+ β5tdisabilityit + β6tschool_carit + β7toccupationit
+ β10tschool_eduit + β11tnon_schoolit + β12tp_g_genderit
+ β13tp_g_nesbit + β14thome_languit + εit
schoolj = γ0 + γ1gpokmj + γ2isolationj + γ3spatial_arj + ηj
εit ∼ N(0, σ2y)
ηj ∼ N(0, σ2α).
This longitudinal model can be �t in lmer. However, with only 173 students in this
data set and nine student variables with 22 categories in total, in addition to three
school variables with 17 categories in total, there is a strati�cation problem when
trying to �t the longitudinal, hierarchical model or even a simple linear regression.
One illustrative example of the insu�ciency of the data in terms of quantity and
quality when trying to �t this complicated, multilevel and longitudinal model is give
in Table 7.3.2 which is of the counts in the intersection of each of the categories of
atsi (rows) and school_edu (columns). There are no students who have an atsi
value of 1 or Inconsistent and at the same time, a school_edu value of 0, 1, 2, 3 or
4. The only occurrences are when school_edu is missing (NA), and these missing
data are removed when �tting a linear regression model. There are other pair-wise
combinations of covariates which exhibit this lack of data in the intersection of
categories, not only atsi and school_edu. Hence, it is not possible to �t regression
models with all the covariates included in the model, due to the lack of data when
it is strati�ed over all the variables' categories.
Ideally, we wish to �t a longitudinal, hierarchical model using all the student and
school variables, but are not able to with our lack of data. We could reduce the
number of estimated parameters by excluding variables, and a simple linear regres-
sion of the student and school covariates
Chapter 7. Initial Longitudinal Analysis 193
Table 7.3.2: Counts of students in the categories of atsi and school_edu
atsi 0 1 2 3 4 NA2 51 6 9 48 36 3541 0 0 0 0 0 9Inconsistent 0 0 0 0 0 6
NN Rasch = atsi + lbote + gender + aboriginal + disability + school_car
+ p_g_gender + p_g_nesb + home_langu + gpokm + isolation + spatial_ar
highlights the linear dependence between atsi1 and aboriginal which both in-
dicate the nine Aboriginal students. The variable aboriginal is then excluded
from the model, as it does not provide any further information about the Aboriginal
status of students which is not already contained in atsi. There is also collinearity
between spatial_ar and isolation. We would choose to include spatial_ar over
isolation as we do not need both school measures of location, and the de�nition of
spatial_ar is clearer and more meaningful. The issue with removing variables from
the model and then �tting the longitudinal, hierarchical model, if it were even possi-
ble with our data, is that the reliability of the estimated parameters is reduced. We
cannot be assured that this is a good model since variables are removed pre-model-
�tting and analysis, solely to reduce collinearity and the over-parameterisation of
the model. We would be limited by the variables which we can include in the model
to try and best explain the linear relationship with NN Rasch scores.
In the situation of having appropriate and su�cient data, we would �t the model
in equation (7.3.1) in lmer and then Stan, having established that Stan is the most
appropriate method compared to BUGS (Chapter 5). These methods and programs
would be applied to the longitudinal, hierarchical model to give and assess the
signi�cance of the parameter estimates. From the �tted models, we could validate
the model using methods as outlined in Chapter 6. Essentially, we would apply all
statistical methods as for the hierarchical model to the longitudinal, hierarchical
model.
The longitudinal, hierarchical model is an improvement on the hierarchical model
as it incorporates repeated measures and accounts for the common characteristics of
194 7.3. Grade 3, 5 and 7 Tests
the student. Given su�cient data, we can use this more complicated, but improved,
model to measure the performance of schools through the improvement of their
students.
Chapter 8
Conclusion
8.1 Summary and Conclusions
The objective of this thesis was to accurately model and measure the school e�ect
on student improvement. This was achieved on an example data set, a subset
of the Basic Skills Test data which was the precursor assessment program to the
National Assessment Program - Literacy and Numeracy (NAPLAN) tests in South
Australia. An analysis of the basic descriptive statistics uncovered various structural
and incidental anomalies at di�erent levels of the data. The use of statistics in a
forensic manner proved to be integral in investigating these anomalies, trying to
explain what we could, infer with reason and caution and to �nally achieve a heavily
reduced data set which was clean and able to be used for modelling.
The process of statistical model selection pruned the variables down further to
�fteen variables - procyear, gradedyear, gpokm, isolation, spatial_ar, atsi,
lbote, gender, aboriginal, disability, school_car, occupation, school_edu
and non_school. The issue of collinearity between the schoolno variable and the
school covariates was resolved using hierarchical modelling which more importantly,
modelled the natural multilevel structure of students within schools. In addition,
the hierarchical model estimated the school e�ect as a linear regression of location
variables. This hierarchical model, however, could not identify individual schools
and their e�ects as statistically signi�cant.
The bulk of the modelling in this thesis was �tting hierarchical models using three
195
196 8.2. Practical Implications and Future Work
di�erent methods - linear multilevel mixed e�ects models and two Bayesian ap-
proaches. The hierarchical linear mixed e�ects model was �t using the lmer package
inR, and one of the three model selection techniques was to calculate P -values using
Markov Chain Monte Carlo sampling. This method identi�ed the signi�cant �xed ef-
fects to be procyear, gradedyear, atsi, lbote, gender, disability, school_car,
occupation, school_edu and non_school.
Since the Markov Chain Monte Carlo simulation and sampling method is essen-
tially a Bayesian technique, we decided to �t the hierarchical model itself using
Bayesian Inference using Gibbs Sampling (BUGS) and assessed the �t of the model.
Unfortunately, some of the variables were highly correlated, and this was causing
imprecise estimates of the parameters. To further improve model �t, we used the
recently-released program Stan, a package for obtaining Bayesian inference using the
No-U-Turn sampler, a variant of the Hamiltonian Monte Carlo method. This im-
proved model-�tting approach achieved converged parameter estimates of increased
reliability.
The �nal model from Stan was validated and shown to be a well-�tting model which
accurately predicts students' results, given student and school characteristics. From
this model, school is not a signi�cant predictor, and our conclusion is that from the
data, schools are performing as they should. The usefulness of the Stan model lies
in its explanatory ability to identify signi�cant predictors, account for the variation
in the scores and ultimately, model the data.
If the purpose of the model was to classify schools, more school data with variables
which are not correlated would allow for school to be a treatment e�ect so that we
can identify signi�cant schools. This is future work, given suitable data. It would be
best if these extra school variables were ones that are potentially subject to control
by the school, or education authorities.
8.2 Practical Implications and Future Work
To link this work on the Basic Skills Test data back to the NAPLAN tests, we have
developed techniques which can be applied to the NAPLAN data to predict the
performance of students. These techniques incorporate the hierarchical and longi-
Chapter 8. Conclusion 197
tudinal structure of the data, and so dependencies are accounted for and modelled,
leading to sensible and reliable conclusions. If these statistical methods and models
were applied to the current NAPLAN data, the relevant education system authori-
ties, for example, the Australian Curriculum, Assessment and Reporting Authority
(ACARA), could use the results to approach schools and see which programs are
being implemented to the advantage of students and what can be done to aid other
schools. Used appropriately, this information could be used to raise the performance
of many schools. Should a speci�c teaching initiative be implemented in a subset of
schools, these statistical models can identify the e�ect of these additional programs
and measure the resultant improvement of the students. This could then be used to
assess the �value� of the di�erent initiatives.
Recall that there is also the perspective of the parent who is primarily interested
in knowing, given the characteristics of their child, which school is best suited to
their child, and are there any particular schools which can be expected to produce
better subsequent achievements than other schools? The breakdown of the model
into student and school level explanatory variables, gives the e�ect of each of the
individual variables on the overall school mean. This personalises the school e�ect
to also incorporate an individual student's characteristics.
Having addressed both the school and student levels of the hierarchy of the education
system, the repeated measures of tests at increasing grades for each student is the
longitudinal aspect which is now available in the NAPLAN data, four years after
it �rst started in 2008. As students sit NAPLAN tests in Grades 3, 5, 7 and 9, an
individual student will only be recorded every two years, for example, a student who
sits the Grade 3 test in 2008 will have their Grade 5 test recorded in 2010. As of
December 2012, there are at most, only two recorded test results for each student.
This is very similar data to the Grades 3 and 5 longitudinal data in Chapter 7.
One restriction with the longitudinal Basic Skills Test data was only being able to
identify students uniquely by both their school and student ID numbers - we could
not identify students who had changed schools between tests. However, the mobility
of students is an important issue which can be modelled from the NAPLAN data
which assigns to each student, a unique ID number for the duration of their edu-
cation. For any longitudinal measure, how students who change and move between
198 8.2. Practical Implications and Future Work
schools are modelled must be resolved. For example, what proportion of a student's
results or improvement should be attributed to the current and the former school is
a question that should be addressed. Goldstein et al. [20] states
�There is also the problem of accounting for students who change schools,
who may have particular characteristics, and there is almost no research
into this problem.�
This student mobility is a vital consideration, because a large proportion of students
change schools at some point (especially the transition between Year 7 and 9 or Year
5 and 7, that is, from primary to secondary education, depending on the state or
territory), and so valuable information is lost if they are discarded from the data
analysis.
The focus of this thesis was primarily at the student and school levels because
of the data available. However, there are other levels in education's hierarchical
structure, and the statistical techniques discussed in this thesis can be widely applied
to compare the performance of students, whether it be between classes, between
the private and government school sectors or between denominational versus non-
denominational schools.
One �nal area of interest is the noticeable drop in student participation in certain
schools. An article by Save Our Schools [14] on November the 26th, 2012 says that
although the average percentage of students who have withdrawn from the NAPLAN
tests remains low in all states and for Australia overall, these low averages disguise
some very high withdrawal rates in many schools.
�The average withdrawal rates across Australia in 2011 and 2012 were
one to two percent for the di�erent Year levels tested. In contrast, 276
schools had withdrawal rates of 10% or more in 2011 according to the
My School website. Seventy-eight schools had over 25% withdrawn and
32 schools had withdrawal rates of between 75 and 100%. . .
Schools with high withdrawal rates include both government and private
schools . . . Private schools accounted for the large proportion of schools
with very high withdrawal rates. Seventy-three per cent of all schools that
had 25% or more of their students withdrawn from NAPLAN were private
schools.�
Chapter 8. Conclusion 199
This �worrying� trend may have roots in something deeper at an education level,
but at a data management and analysis level, the rapid growth in withdrawal rates
poses a threat to the reliability of NAPLAN results for inter-school comparisons,
inter-jurisdictional comparisons and trends in indicators of student achievement [13].
The increasing numbers of students being withdrawn or absent from NAPLAN will
a�ect the reliability of the results as exempt students, those students who are not
assessed, are deemed to be below minimum national standards and still included
in the NAPLAN results. Changes in participation rates could a�ect the results of
individual schools, sub-groups of students such as Indigenous and low socio-economic
status students and state or territory results as well as trends over time. This needs
to be taken into account in the interpretation of future NAPLAN results.
As shown in this thesis, statistical modelling techniques can be applied to an ed-
ucation data set, such as the Basic Skills Tests or NAPLAN data to draw reliable
conclusions about school performance and can potentially be used to answer future
research questions in this area.
Appendix A
Coding of Variables
(c) represents categorical variables
(q) represents quantitative, continuous variables
A.1 Test Variables
aspect literacy or numeracy aspect (c)
LL, LR, LS, LW, NN, NS, NU, NM
gradedyear grade of the test (c) - 3, 5 or 7
nocorrect raw test mark (q)
procyear calendar year (c) - 1997 - 2005
standardsc standardised score under Rasch scaling (q)
A.2 School Variables
x00y{4,5,6}_abs absentee rate in 2004, 2005 and 2006 respectively
(q)
x00y{4,5,6}_beh number of behavioral incidents in 2004, 2005 and
2006 respectively (q)
201
202 A.2. School Variables
cap Country Areas Program* (c)
Y Yes
x00y{4,5,6}_enr enrolment numbers in 2004, 2005 and 2006 respec-
tively (q)
gpokm distance from Adelaide General Post O�ce in km
(q)
isolation isolation index (c)
1, 1.5, 2, 2.5, 3, 3.5, 4, 4.5, 5, 5.5, 6, 6.5, 7
(further information about the categories of
this variable is not available)
x00y{4,5,6}_mob mobility of students in 2004, 2005 and 2006 respec-
tively (q)
schoolno code number of the school (c)
x00y{4,5,6}_scrd number of School Cards** in 2004, 2005 and 2006
respectively (q)
spatial_ar MCEETYA*** classi�cation for Rurality and Re-
moteness (c)
1.1 Metropolitan
2.2.1 Large Provincial Towns
2.2.2 Small Provincial Towns
3.1 Remote Areas
3.2 Very Remote Areas
* The Country Areas Program is an Australian government program which provides
�nancial help to rural schools.
(https://deewr.gov.au/country-areas-program)
** The School Card Scheme is a government initiative which provides �nancial
assistance towards educational expenses for eligible families.
(http://www.decd.sa.gov.au/goldbook/pages/school_card/schoolcard/?reFlag=1)
*** Ministerial Council on Education, Employment, Training and Youth A�airs
Appendix A. Coding of Variables 203
staff_metr classi�cation by DECS**** sta� (c)
M Metro
C Country
x00y{4,5,6}_tch number of teachers in 2004, 2005 and 2006 respec-
tively (q)
x00y{4,5,6}_tmob teacher mobility in 2004, 2005 and 2006 respec-
tively (q)
**** Department of Education and Children's Services
A.3 Student Variables
aboriginal Aboriginal status (c)
Y Yes
N No
atsi Aboriginal or Torres Strait Islander (c)
1 Yes
2 No
country_of country of origin (c) - country abbreviations (e.g. TAIW)
cultural_b cultural background (c) - country abbreviations (e.g.
SAUD)
date_of_bi date of birth
disability disability status (c)
Y Yes
N No
gender male or female (c)
M Male
F Female
204 A.3. Student Variables
home_langu English as home language (c)
Y Yes
N No
lbote language background other than English (c)
1 Yes
2 No
nesb_code non-English speaking background (c)
A Students of ATSI origin who identify as ATSI and
who speak an ATSI language (including Aborigi-
nal English). Exclude ATSI students who do not
speak an ATSI language.
P1 Permanent resident students born overseas with
at least one parent/guardian from a non-English
speaking background. (This includes children adopted
by English speaking families who have maintained
a cultural or linguistic link with their country of
origin.)
P2 Permanent resident students born in Australia with
at least one parent/guardian born overseas and
from a non-English speaking background.
P3 Permanent resident students born in Australia, not
included in the previous two de�nitions, who have
maintained an identity and family link with a non-
English speaking language or culture.
TR Temporary resident - students who are not per-
manent residents in Australia and who come from
non-English speaking countries
non_school parental non-school education (c)
0 Not stated/unknown
Appendix A. Coding of Variables 205
5 Certi�cate I to IV
6 Advanced diploma/Diploma
7 Bachelor degree or above
8 No non-school quali�cation
occupation parental occupation group (c)
0 Not stated/unknown
1 Senior management in large business organisations,
government administration and defence and qual-
i�ed professionals
2 Other business managers, arts/media/sportspersons
and associate professionals
3 Trades and advanced/intermediate clerical, sales
and service sta�
4 Other occupations like machinery operators, hos-
pitality sta�, o�ce or sales assistants, labourers
and related workers
8 Not in paid work in the last 12 months
p_g_countr parental country of origin (c) - country abbreviations
(e.g. ZIMB)
p_g_cultur cultural background of parent/primary guardian (c) -
country abbreviations (e.g. RUSS)
p_g_gender gender of principal guardian or parent (c)
M Male
F Female
p_g_nesb parental non-English speaking background (c)
Y Yes
N No
206 A.3. Student Variables
school_car individual School Card (c)
Y Yes
N No
school_edu parental school education (c)
0 Not stated/unknown
1 Year 9 or equivalent or below
2 Year 10 or equivalent
3 Year 11 or equivalent
4 Year 12 or equivalent
status current status of student within school (c)
A Active
B Bus only
C Exclusion placement
D Deceased
E External
F Future
H Home education
L Left
M Medical placement
N Not active
O Out of scope
R Not in census
T Alternative placement
V Permanent exemption
X Never attended
studentide student ID number (c)
visa_sub_c Australian permanent visa sub-class (c)
Appendix B
Plots
B.1 Boxplots of LL Rasch and NN Rasch for Cate-
gorical Variables
The sample size and sample mean are given above each boxplot. The na category
represents missing data for the categorical variable.
207
●●
●
●●
●●
●
●●●●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●●
●●●
●
●
●●●
●
●
●
●●●
●●
●
●●
●
●●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●●●
●
●
●●
●
●●
●●●●●●●
●
●
●
●●●
●
●
●
●●
●●
●●
●●●●●●●●●●
●
●●●●●
●
●
●
●
●
●●
●
●●●●
●●
●●●●●●
●
●
●
●
●●●
●
●
●●●●
●
●●●
●
●●●●
●●●●●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●●●
●●
●●●●
●
●●
●
●●
●
●
●●●
●
●
●
●
●
●●
●
●●●●
●
●
●●●●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●●●●●●●
●●
●●
●
●●
●
●
●
●
●●●●●●●
●●●
●
●●
●
●●●●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●●●
●●
●●●●●
●
●
●
●
●●
●●●
●
●
●●●
●
●●●
●
●
●●●●●●●
●●●
●
●
●●
●
●
●
●
●●
●●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●
●
●●
●
●
●●●●●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
7 8 68 8308 8259 17546 17819 7845 9
45.7 46.8 45.4 48.27 49.16 52.52 52.15 55.84 56.89
procyear
20
40
60
80
100
1997 1998 1999 2000 2001 2002 2003 2004 2005Categories
LL
Ras
ch
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●●
●
●
●
●
●
●●●
●
●
●●●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●●●
●●●●
●
●
●
●●
●
●
●●
●●
●
●●●●●●●●
●
●●●
●●
●●●●●
●
●●●
●
●●●●●●●
●
●
●●●●●
●●
●●
●
●●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●●
●●
●
●
●●
●●●●
●
●●●
●
●●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●●
●●
●
●
●
●●
●●
●
●
●
●●
●
●●●●●●
●
●
●
●
●
●
●
●
●●●
●
●
●●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●●
●
●
●
●●
●
●●
●●
●
●●
●
●●
●●●●●
●
●●●●
●
●
●
●●
●
●
●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●●●
●
●
●
●
●●●●●●
●
●
●●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●●●
●
●●●●
●●
●
●●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●●
●
●●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●●●
●
●●
●
●●
●
●
●●●
●
●●●
●●●
●●
●
●●
●●
●●●●
●
●
●
●
●
●
●
●●●
●
●
●
●●
●●
●
●●
●●●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●●
●
●
●●●●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●●●
●●
●
●
●●●
●●
●●●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●●●
●
●
●
●
●
●
●●●●
●
●
●
●
●●
●●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●
●
●
●●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●●
●●
●●●
●●
●
●
●
●
●●●
●
●
●
●
●●●●●●●
●
●
●●●
●
●●●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
6 7 59 8346 8261 17610 17935 7844 10
47.37 46.69 42.64 48.88 49.32 54.14 54.27 57.66 58.42
procyear
20
40
60
80
100
1997 1998 1999 2000 2001 2002 2003 2004 2005
Categories
NN
Ras
ch
208 B.1. Boxplots of LL Rasch and NN Rasch for Categorical Variables
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●
●●
●
●●
●●
●
●●●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●●●●●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●●●●
●●
●
●
●
●●●●●
●
●
●●●
●
●●
●●
●●
●
●
●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●●
●
●
●
●●
●●●●
●
●
●
●
●
●
●
●●●●
●●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●●●●●
●
●●
●
●●●
●●●
●
●●
●●
●●
●
●
●
●
●●
●
●
●
●
●●
●●
●
●
●●
●●●●
●
●
●
●
●
●●
●
●●●
●
●●
●
●
●
●
●
●●●●●
●
●●
●●
●
●●
●
●●●●
●
●●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●●
●
●●
●
●
●
●●●
●
●
●
●
●
●●
●
●●
●●●●●
●
●●
●
●
●●●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●●
●●
●
●●
●
●●●●
●●
●
●●●●
●
●●
●
●
●
●●●●●
●
●
●●●
●
●
●●●
●
●
●
●●●
●
●
●
●
●
●●
●
●●
●●
●
●
●●
●
●
●●
●●
●
●●●●●
●
●●
●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●●
●●
●
●
●
●●●●●●
●
●●●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●●●●
●
●
●
●
●
●
36837 20477 2555
48.94 55.72 61.18
gradedyear
20
40
60
80
100
3 5 7Categories
LL
Ras
ch
●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●●
●●
●
●
●●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●●
●
●●
●
●
●
●
●
●●●●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●●●●
●
●●●
●
●●●●●●●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●●
●
●●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●●
●
●●
●
●
●
●●●
●
●
●●
●
●
●●
●
●
●
●●
●●●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●●●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●●●●
●
●●●●
●
●●●
●●
●
●
●●
●
●
●
●●●●●●
●●●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●●
●
●
●●●●
●
●
●
●
●●●
●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●●●●●
●●
●●
●●
●●●●
●
●●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●●●●●
●
●
●
●
●
●
●
●
●●
●
●●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●●●
●
●
●
●
●
●●
●●
●●
●
●●
●
●
●
●
●●
●●●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●●●
●
●
●
●
●
●
●●
●
●
●●●●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●●
●●
●
●
●●
●
●
●●●●●
●
●●
●●
●
●●
●●●●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●●●
●
●●●
●
●
●
●
●
●
●
●●●
●●
●
●
●
●●
●●●
●●●●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●●●●●
●
●
●
●
37005 20490 2583
49.65 58.17 65.45
gradedyear
20
40
60
80
100
3 5 7Categories
NN
Ras
ch
Appendix B. Plots 209
●●
●
●
●
●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●●●●●
●
●
●
●
●
●●
●
●●●
●●
●
●●
●
●●●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●●●●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●●●
●
●●
●
●
●
●●●●
●
●●
●
●
●●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●●●
●
●
●●●
●
●
●
●
●
●
●
●●●●
●
●●●●
●●
●
●
●●●●
●
●
●●●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●●
●
●●
●
●
●●
●
●
●
●●●●●
●●●●●
●
●
●
●●●●●
●
●
●
●
●
●
●
●
●
●●
●
●●
●●
●
●
●
●
●●●●
●
●
●●●
●
●
●
●
●
●●
●
●
●●
●
●●
●●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
● ●
●●
39848 1392 135 8 85 1174145 1029 1174 920 3545 2348 4213 910
52.15 50.05 46.28 43.74 37.91 51.7252.1 48.85 51.88 51.93 51.16 50.12 51.33 51.14
isolation
20
40
60
80
100
1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6 6.5 7 NACategories
LL
Ras
ch
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●●●●●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●●
●●●
●●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●●●
●●
●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●●●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●●
●
●
●
●
●●●●
●
●●●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●●●
●●
●
●
●●
●●
●●●
●
●
●
●
●
●
●
●●●
●
●
●●
●
●●
●
●●●●
●●●
●
●
●●
●
●
●
●
●●●●●●●
●
●
●●
●●
●
●
●
●●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●
●
●●
●
●
●●●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●●●●
●
●●●
●
●
●
●
●
●
●●●●●
●
●
●
●●
●
●●
●
●●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●
●
●●
●
●
●●
●
●
●●
●●
●
●●
●
●
●
●●
●
●
●●●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●●●●
●
●
●
●
●
●
●●●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
39815 1423 141 9 117 1194164 1039 1194 932 3598 2331 4282 914
53.47 51.55 45.89 43.91 37.53 54.353.87 50.17 54.18 54.32 53.11 52.08 52.59 53.41
isolation
20
40
60
80
100
1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6 6.5 7 NACategories
NN
Ras
ch
210 B.1. Boxplots of LL Rasch and NN Rasch for Categorical Variables
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●●●●●
●
●
●
●
●
●●
●
●●●
●●
●
●●
●
●●●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●●●●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●●●
●
●●
●
●
●
●●●●
●
●●
●
●
●●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●●●
●
●
●●●
●
●
●
●
●
●
●
●●●●
●
●●●●
●●
●
●
●●●●
●
●
●●●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●●
●
●●
●
●
●●
●
●
●
●
●
●●●
●●●●●
●
●
●
●
●
●
●●●
●●
●
●●
●
●
●
●
●
●
●●
●
●●●
●●
●
●
●●
●
●●
●●
●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●●
●
●
●
●●
●
●
●
●●
●●●
●
●
●
●●
●
●
●
●
●
●
●
●●
●●
39489 7581 9767 2451 568 13
52.16 51.53 50.97 50.84 47.11 52.8
spatial_ar
20
40
60
80
100
1.1 2.2.1 2.2.2 3.1 3.2 NACategories
LL
Ras
ch
●
●
●●
●
●●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●●●●●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●●
●●●
●●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●●●
●●
●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●●●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●●
●
●
●
●
●●●●
●
●●●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●●
●
●
●
●●
●
●●●
●●
●
●
●●
●●
●●●
●
●
●
●
●
●
●
●●●
●
●
●●
●
●●
●
●●●●
●●●
●
●
●●
●
●
●
●
●●●●●●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●●●●
●
●●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●●
●
●
●
●●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
39445 7646 9891 2470 612 14
53.49 53.14 52.77 52.7 47.71 54.61
spatial_ar
20
40
60
80
100
1.1 2.2.1 2.2.2 3.1 3.2 NACategories
NN
Ras
ch
Appendix B. Plots 211
●
●
●
●
●●
●
●
●
●
●
●●
●●
●●●
●
●●●
●
●●
●●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●●●
●●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●●●●●
●
●
●
●
●
●●
●
●●●
●●
●
●●
●
●●●
●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●●●●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●●●
●
●●
●
●
●
●●●●
●
●●
●
●
●●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●●●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●●●
●
●
●
●
●
●
●
●●●●
●
●●●●
●●
●
●
●●●●
●
●
●●●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
16049 43807 13
50.77 52.16 52.8
staff_metr
20
40
60
80
100
C M NACategories
LL
Ras
ch
●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●●●
●●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●●●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●●●●●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●●●
●●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●●●
●●
●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●●●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●●
●
●
●●
●
●●●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●●●
●
●●
●
●
●
●●
●
●
●
●●
●
●●●
●●
●
●
●●
●●
●●●
●
●
●
●
●
●
●
●●●
●
●
●●
●
●●
●
●●●●
●●●
●
●
●●
●
●
●
●
●●●●●●●
●
●
●●
●●
●
●
●
●●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●
16275 43789 14
52.47 53.51 54.61
staff_metr
20
40
60
80
100
C M NACategories
NN
Ras
ch
212 B.1. Boxplots of LL Rasch and NN Rasch for Categorical Variables
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●●●●
●
●
●
●
●
●●
●
●
●
●
●●
●●
●
●
●●
●
●●
●
●
●
●●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●●
●
●
●
●●
●●
●
●●●●
●
●
●
●
●
●
●●
●
●●
●●●●●
●
●●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●●●
●
●
●
●
●
●●
●
●
●●
●
●●●
●
●
●
●
●
●
●●●
●
●
●●●●
●●
●
●
●●
●
●
●●●●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●●●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●●●●
●
●
●
1936 57529 404
45.77 52 49.82
atsi
20
40
60
80
100
1 2 NACategories
LL
Ras
ch
●
●
●
●
●●●●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●●●
●●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●●●●●
●●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●●●●●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●●●●
●●
●●●●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●●
●●●
●●
●●
●●
●
●
●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●●
●●●●
●●
●
●
●
●
●●●
●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●●
●
●●●●
●
●●
●
●
●
●
●
●
●
●●
●
●●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●●
●
●
●●
●●
●●●
●
●
●
●
●
●
●
●●●
●●
●
●
●
●
●●
●●
●
●●
●
●●●●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●●●●●●●
●
●
●●
●●
●
●
●
●●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
1934 57730 414
45.97 53.49 50.69
atsi
20
40
60
80
100
1 2 NACategories
NN
Ras
ch
Appendix B. Plots 213
●
●●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●●
●
●
●
●
●
●
●
●●●
●●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●●●●●●●
●
●
●
●
●
●●●
●
●
●●
●●
●
●●
●●
●
●●
●
●
●
●●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●●●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●●
●
●
●
●●●●
●
●●
●●
●
●●●●
●
●
●●
●
●
●
●●
●
●●
●
●●●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●●
●●●●
●
●
●
●
●
●●
●
●
●●●
●●
●
●
●
●
●
●●●●
●
●
●●●
●●
●
●●
●
●
●●●●●
●
●
●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
2445 51606 5818
50.07 51.76 52.7
lbote
20
40
60
80
100
1 2 NACategories
LL
Ras
ch
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●●●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●
●●●
●●
●
●●●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●●●●
●
●
●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●●
●
●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●●
●
●
●
●
●
●
●
●●
●●●
●●
●●●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●●●
●●
●
●
●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●
●
●●●
●
●●●
●
●
●
●
●
●
●
●●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●●●●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●●●
●
●
●
●●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●●●●
●
●
●●
●●
●
●
●●
●●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●●
●●●
●
●
●
●
●
●
2494 51712 5872
51.01 53.24 54.13
lbote
20
40
60
80
100
1 2 NACategories
NN
Ras
ch
214 B.1. Boxplots of LL Rasch and NN Rasch for Categorical Variables
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●●●●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
●●
●
●●●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●●
●●
●
●●
●
●●
●●●
●
●
●
●
●
●●
●
●●
●
●●●●
●
●
●
●●●
●
●
●●
●●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●
●
●●
●●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●●●●
●
●
●●
●
●
●
●
●
●
●●
●
●●●
●
●●
●
●
●
● ●
●●
●
●●●●●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●●
●
●
●
●●●●
●
●●●
●
●
●●●
●●
●
●
●
●●
●●●
●
●●
●
●
●●
●
●●
●
●
●
●●●●
●
●
●●
●
●●
●
●●●
●
●
●
●
●●
11321 88 1 9 28179 32 2 5 20232
50.24 52.82 57.38 49.11 52.29 50.09 48.93 51.18 51.94
status
20
40
60
80
100
A B E H L N T X NACategories
LL
Ras
ch
●
●●
●
●
●●●●
●●
●
●
●
●
●
●
●●●
●●●
●
●●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●●
●
●
●●●●●
●
●
●
●
●●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●●●
●
●
●
●
●●
●●
●
●●
●
●
●
●
●●
●●●
●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●●
●
●
●
●●
●
●●●●
●
●
●
●
●●
●
●●
●●●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●●
●●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●●
●●
●
●●
●●●
●
●
●
●
●
●
●●●
●●●
●
●
●
●●●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●●
●●
●
●●
●
●
●
●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●
●
●
●
●●●
●●
●●●
●
●●●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●●
●
●●●
●
●
●●
●
●
●
●
●●
●
●●●
●
●
●●
●●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●●
●●
●●●
●
●
●
●
●
●
●
●●
●●
●●
●
●
●●
●
●●
●
●●
●
●●●●
●●●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●
●
●
●
●●
●●
●
●
●
●●
●●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●●
●
●●●●●●
●
●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●●
●
11421 95 1 6 28140 32 2 6 20375
50.89 56.84 54.48 48.17 54.18 54.27 61.13 50.89 53.22
status
20
40
60
80
100
A B E H L N T X NACategories
NN
Ras
ch
Appendix B. Plots 215
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●●●
●●
●●●
●
●
●
●
●
●
●●
●
●
●
●●●
●●
●●●
●●
●●
●
●
●
●●●
●
●●
●
●
●●
●●
●
●●
●
●●
●●
●
●
●●
●●
●
●
●
●
●
●●●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●●
●●
●
●
●
●●
●
●
●
●
●●
●
●●●●●
●●
●
●
●
●
●●●●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●●
●
●
●
●
●●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●●●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●●● ●
●●
●
●●●●●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●●
●
●
●
●●●●
●
●●●
●
●
●●●
●●
●
●
●
●●
●●●
●
●●
●
●
●●
●
●●
●
●
●
●●●●
●
●
●●
●
●●
●
●●●
●
●
●
●
●●
19400 20237 20232
52.75 50.7 51.94
gender
20
40
60
80
100
F M NACategories
LL
Ras
ch
●
●
●●
●●
●
●
●●●
●
●
●●
●
●
●
●●
●
●
●
●●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●●
●●●
●
●
●●
●●
●
●
●
●
●
●●●
●●●
●
●
●●
●
●
●
●
●●●
●
●
●
●
●●●
●
●●
●
●
●
●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●
●
●●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●●
●●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●●
●
●
●●
●●
●●●●●●
●
●●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●●●
●●
●●
●●
●
●●
●
●
●●●●
●
●●
●
●
●
●
●
●
●
●
●●●●
●●●
●
●
●
●
●●●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●●●●
●
●
●●
●
●●●●●●●
●
●●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●
●
●
●
●●
●●
●
●
●
●●
●●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●●
●
●●●●●●
●
●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●●
●
19315 20388 20375
52.75 53.7 53.22
gender
20
40
60
80
100
F M NACategories
NN
Ras
ch
216 B.1. Boxplots of LL Rasch and NN Rasch for Categorical Variables
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●
●●●●
●
●●
●
●●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●●
●
●●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●
●●
●
●
●●
●
●●
●
●●
●●
●
●
●●
●
●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●●
●●
●
●
●●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●●●●●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●●
●
●
●
●●●●
●
●●●
●
●
●●●
●●
●
●
●
●●
●●●
●
●●
●
●
●●
●
●●
●
●
●
●●●●
●
●
●●
●
●●
●
●●●
●
●
●
●
●●
38354 1283 20232
51.91 45.71 51.94
aboriginal
20
40
60
80
100
N Y NACategories
LL
Ras
ch
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●●
●
●
●
●●●
●
●
●●●
●
●●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●●
●●●●
●
●
●
●●
●
●●●
●●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●●●
●
●●●
●
●●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●●
●●
●
●●●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●●
●●●●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●
●●
●
●●●●
●●●●●
●
●
●
●
●●
●●
●
●
●
●●●
●
●
●
●
●●●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●●●●
●
●●●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●
●
●
●
●●
●●
●
●
●
●●
●●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●●
●
●●●●●●
●
●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●●
●
38416 1287 20375
53.47 46.24 53.22
aboriginal
20
40
60
80
100
N Y NACategories
NN
Ras
ch
Appendix B. Plots 217
●
●
●
●
●
●●
●
●
●
●●●●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●●●●●
●●
●
●
●●●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●●●●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●●●●●
●●
●●●
●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●●●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●●●●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●●
●
●
●
●●●●
●
●●●
●
●
●●●
●●
●
●
●
●●
●●●
●
●●
●
●
●●
●
●●
●
●
●
●●●●
●
●
●●
●
●●
●
●●●
●
●
●
●
●●
37236 2401 20232
52.24 43.38 51.94
disability
20
40
60
80
100
N Y NACategories
LL
Ras
ch
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●●●●●
●●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●●●●●
●
●●●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●●
●
●●●
●●
●●
●●
●
●
●
●●
●
●
●
●
●●
●●
●
●
●●●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●●
●
●
●
●
●●
●
●●●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●●●●
●●●
●
●
●
●
●
●
●●
●
●●
●
●
●●
●
●●●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●●●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●●
●
●
●
●
●
●●
●
●
●●●
●●
●●
●
●
●●
●
●
●
●●
●
●●
●
●●●●
●●●●●
●
●
●
●
●●
●●
●
●
●
●●●●
●●
●
●
●●●●●
●
●●●
●●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●
●
●
●
●●
●●
●
●
●
●●
●●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●●
●
●●●●●●
●
●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●●
●
37226 2477 20375
53.83 44.32 53.22
disability
20
40
60
80
100
N Y NACategories
NN
Ras
ch
218 B.1. Boxplots of LL Rasch and NN Rasch for Categorical Variables
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●●
●●
●
●●
●
●
●●
●●●●●
●
●
●●
●
●
●
●●
●
●●
●
●●●●
●
●
●
●
●
●●●
●
●●
●●
●
●●●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●●
●●
●
●●
●●
●●
●●●●
●
●
●
●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●●●
●
●●
●●
●
●●
●
●
●●●●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●●
●
●
●
●
●
●●
●
●●
●
●
●●●
●
●●
●●
●
●
●●●●●
●
●
●●
●
●●●●●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●●
●
●
●
●●●●
●
●●●
●
●
●●●
●●
●
●
●
●●
●●●
●
●●
●
●
●●
●
●●
●
●
●
●●●●
●
●
●●
●
●●
●
●●●
●
●
●
●
●●
36852 2785 20232
51.99 47.98 51.94
school_car
20
40
60
80
100
N Y NACategories
LL
Ras
ch
●
●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●
●
●
●●●●●
●●
●
●
●
●
●●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●●
●●
●
●
●
●●
●
●
●
●
●●
●●
●●●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●●
●●
●●
●
●●
●
●●●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●●
●
●
●●
●●
●
●
●●●
●
●●●
●
●
●●●
●
●●
●
●●●
●
●●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●●●
●
●
●●
●
●
●●
●
●●
●
●●
●
●
●
●
●●
●
●●●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●●
●●
●●●
●●
●●
●
●
●●●●
●
●●
●
●●
●
●●●●
●●●●●
●
●
●
●
●
●●
●
●
●
●
●●●
●●
●
●
●●●●●
●
●●●
●●
●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●●
●●
●
●
●●●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●
●
●
●
●●
●●
●
●
●
●●
●●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●●
●
●●●●●●
●
●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●●
●
36896 2807 20375
53.59 48.65 53.22
school_car
20
40
60
80
100
N Y NACategories
NN
Ras
ch
Appendix B. Plots 219
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●●●●●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●●●
●
●
●●
●
●
●
●●
●
●●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●●●●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●●●●
●
●
●
●
●●
●
●
●●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●●
●
●●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●
●
●●●●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●●●
●
●
●
●●●
●●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●●
●
●●●
●
●
●
●●●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●●●
●●
●●
●
●
●
●●
●
●
●
●
●
●●●
3062 2100 3611 3853 4669 3581 38993
50.84 54.89 53.24 52.59 51.08 50.47 51.68
occupation
20
40
60
80
100
0 1 2 3 4 8 NACategories
LL
Ras
ch
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●●●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●●●●
●
●● ●
●●●
●●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●●●
●●
●●●
●●
●
●●●
●
●
●
●●●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●●●
●●
●
●
●●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●●
●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●●
●
●
●
●●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●●●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●●●
●●
●●●●●
●
●●
●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●●
●●
●
●
●
●
●
●●●
●●
●
●●
●
●
●
●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●●●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●●
●●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●●●●●
●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●
3050 2069 3653 3858 4611 3622 39215
52.23 56.65 55 54.08 52.57 51.75 53.1
occupation
20
40
60
80
100
0 1 2 3 4 8 NACategories
NN
Ras
ch
220 B.1. Boxplots of LL Rasch and NN Rasch for Categorical Variables
●
●●
●
●
●●●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●●
●●
●
●●
●
●
●
●●●
●
●
●
●
●●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●●
●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●●●
●
●●
●
●●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●
●
●●●●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●●●
●
●
●
●●●
●●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●●
●
●●●
●
●
●
●●●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●●
●●
●●
●
●
●
●●
●
●
●
●
●
●●●
3224 961 3332 5784 7585 38983
51.1 48.44 50.17 51.66 53.76 51.7
school_edu
20
40
60
80
100
0 1 2 3 4 NACategories
LL
Ras
ch
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●●
●
●
●●●
●●●
●
●
●
●●
●
●●●
●
●●●
●
●
●
●
●
●●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●●●●●●●●●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●●
●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●●
●
●
●
●●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●●●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●●●
●●
●●●●●
●
●●
●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●●
●●
●
●
●
●
●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●●●●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●●
●●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●●●●●
●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●
●
●●
●
3236 985 3300 5761 7569 39227
52.55 50.15 51.48 53.23 55.32 53.11
school_edu
20
40
60
80
100
0 1 2 3 4 NACategories
NN
Ras
ch
Appendix B. Plots 221
●●●●
●
●●●
●●
●
●
●
●●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●●
●
●●
●
●●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●
●
●●●●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●●
●●●●
●
●
●
●
●●●
●
●
●
●●●
●●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●●
●
●●●
●
●
●
●●●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●●
●●
●●
●
●
●
●●
●
●
●
●
●
●●●
3875 4387 2095 2417 7695 39400
50.97 51.94 53.77 55.51 50.9 51.69
non_school
20
40
60
80
100
0 5 6 7 8 NACategories
LL
Ras
ch
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●●
●
●
●
●●●
●●
●
●
●●
●
●
●●●
●
●
●
●●●●●●
●
●●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●●●●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●●●
●
●
●
●●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●
●●●
●
●
●●
●●
●●
●
●
●●
●●●●●●●●
●
●●
●
●●
●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●●
●
●
●
●●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●●●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●●
●●
●●●●●
●
●●
●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●●
●●
●
●
●
●
●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●●●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●●
●●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●●●●●
●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●
●
●●
●
3883 4405 2077 2391 7687 39635
52.33 53.32 55.58 57.17 52.43 53.11
non_school
20
40
60
80
100
0 5 6 7 8 NACategories
NN
Ras
ch
222 B.1. Boxplots of LL Rasch and NN Rasch for Categorical Variables
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●●
●
●●
●
●
●●
●●●●
●
●
●●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●
●●●
●●
●
●
●
●●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●●
●
●
●●
●●●
●
●
●●
●●
●
●
●
●
●
●●
●●●●●
●
●
●
●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
● ●
●●
●
●●●●●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●●
●
●
●
●●●●
●
●●●
●
●
●●●
●●
●
●
●
●●
●●●
●
●●
●
●
●●
●
●●
●
●
●
●●●●
●
●
●●
●
●●
●
●●●
●
●
●
●
●●
33096 6534 20239
51.65 52.01 51.94
p_g_gender
20
40
60
80
100
F M NACategories
LL
Ras
ch
●
●
●●
●
●
●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●●
●●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●
●●●●●●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●●●●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●●●
●●●
●
●
●
●
●
●
●●
●●
●●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●●●●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●●
●
●
●
●
●
●
●
●
●●
●●
●●
●
●●●●
●●●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●●●●●
●
●●●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●
●
●
●
●●
●●
●
●
●
●●
●●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●●
●
●●●●●●
●
●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●●
●
33163 6533 20382
53.16 53.62 53.22
p_g_gender
20
40
60
80
100
F M NACategories
NN
Ras
ch
Appendix B. Plots 223
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●
●
●●
●●●●
●
●●
●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●
●
●●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●●
●●●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●●●●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●●●
●●●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●
●●●●●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●●
●
●
●
●●●●
●
●●●
●
●
●●●
●●
●
●
●
●●
●●●
●
●●
●
●
●●
●
●●
●
●
●
●●●●
●
●
●●
●
●●
●
●●●
●
●
●
●
●●
36377 3258 20234
51.69 51.94 51.94
p_g_nesb
20
40
60
80
100
N Y NACategories
LL
Ras
ch
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●●
●●
●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●●
●
●
●
●●
●
●
●●●
●
●●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●●●●
●
●
●
●●●●●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●●
●
●●●
●
●
●
●
●
●●●
●
●
●
●
●●
●●●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●●●
●●
●
●
●●
●
●
●
●
●●
●
●●●
●●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●
●
●
●
●●
●●
●
●
●
●●
●●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●●
●
●●●●●●
●
●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●●
●
36394 3306 20378
53.22 53.46 53.22
p_g_nesb
20
40
60
80
100
N Y NACategories
NN
Ras
ch
224 B.1. Boxplots of LL Rasch and NN Rasch for Categorical Variables
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●●●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●●
●●
●
●
●
●
●
●
●
●●●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●●
●
●
●
●
●
●●
●
●
●
●
●●
●●
●
●●
●
●●
●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●●●●
●●
●
●●
●
●●●●
●
●
●●
●
●
●
●●
●
●●
●●●
●
●●
●
●
●●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●●●
●
●
●●●
●●
●
●
●
●
●
●
●●●
●
●
●●●●
●●
●
●
●
●●
●●●●●
●
●
●
●●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
643 723 2588 1747 53 54115
43.75 52.87 52.18 52.24 52.05 51.83
nesb_code
20
40
60
80
100
A P1 P2 P3 TR NACategories
LL
Ras
ch
●
●●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●●
●
●●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●
●●●●
●●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●●
●
●
●
●
●
●●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●●●
●●
●●●●●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●●●
●●
●
●
●
●●
●●
●
●●
●
●
●
●
●●●
●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●●●
●●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●●
●●
●
●
●
●
●
●
●●●
●●
●
●
●
●
●●
●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●●●●●
●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●
●
●
●●
●
657 732 2622 1759 52 54256
44.07 54.68 53.55 53.51 55.04 53.3
nesb_code
20
40
60
80
100
A P1 P2 P3 TR NACategories
NN
Ras
ch
Appendix B. Plots 225
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●●
●●●●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●●●●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●●
●
●●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●
●●
●
●
●
●
●●
●
●●
●●●
●
●
●●●
●
●
●
●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●●
●
●
●
●
●
●●●
●
●●
●●
●
●
●●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●●
●
●
●●
●
●●●●●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●●
●
●
●
●●●●
●
●●●
●
●
●●●
●●
●
●
●
●●
●●●
●
●●
●
●
●●
●
●●
●
●
●
●●●●
●
●
●●
●
●●
●
●●●
●
●
●
●
●●
36216 3420 20233
51.82 50.54 51.94
home_langu
20
40
60
80
100
N Y NACategories
LL
Ras
ch
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●●●●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●●
●●●●
●
●
●
●●●●●●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●●●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
●●●
●
●
●
●
●●
●●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●
●●
●
●●●●
●●●●
●
●
●
●
●●
●●
●
●
●
●●●
●
●
●
●
●●●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●
●●
●
●●
●
●
●
●●●●
●●●●
●●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●
●
●
●
●●
●●
●
●
●
●●
●●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●●
●
●●●●●●
●
●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●●
●
36227 3475 20376
53.38 51.74 53.22
home_langu
20
40
60
80
100
N Y NACategories
NN
Ras
ch
226 B.1. Boxplots of LL Rasch and NN Rasch for Categorical Variables
Appendix B. Plots 227
●● ●●● ● ●● ●●●●●● ●●● ●● ●●● ●●●● ● ●●● ●● ●●● ●●● ●● ●●● ●● ●●●● ●● ●●●● ●● ● ● ●●●● ●●●● ●●●● ● ●● ●● ●● ●●●●● ●●●●●●●●● ●●●●● ●●●●● ●●●● ●●●●●●●● ●●●●●●●●● ●● ●●●●● ●●● ●● ●
●●● ●●● ●● ●●●● ● ●●● ● ●● ●●● ● ●●● ●●● ●●● ●●●● ●● ●● ●● ●●●● ●● ●● ●●● ●●●● ●● ●● ●●●● ●● ● ●●● ● ●● ●●●● ●● ● ●● ●● ● ●●● ● ●●● ● ●●●●● ●● ●● ● ●●●● ●●● ● ● ●● ● ●●●●● ● ● ●●● ●●●● ● ●●● ●●● ●●●● ●●● ● ●● ●●●● ●● ●● ●● ●● ● ●●●● ●● ●●● ●●● ●●● ●●●●●● ● ● ●●● ●● ●●● ●●● ●●●●●● ●●●● ●●● ●●●● ●
●● ●● ●● ●● ●● ● ●● ● ●● ●● ●●● ● ●● ●●● ●●●●●● ●●●●●● ●●● ●●● ●●● ●● ●●●● ●●● ● ● ●●● ●● ●● ●● ●● ●●● ● ●●● ●● ●●●●●● ● ●●●● ● ●● ●● ●●● ●● ● ●● ●●● ●● ●●● ●● ●● ●●● ●● ● ●● ●●● ●●●●● ●● ● ●● ●●● ● ●● ●●●●●● ●● ●●●●● ●●●● ● ●● ●●● ● ● ●●●●●●● ●●● ●● ●●● ●●
● ● ●●● ●●●● ●●● ●● ● ●● ●●● ●● ●● ● ●● ● ●●●● ● ●●●● ● ● ●●● ●●●● ● ●● ●●●● ●●● ● ●●●● ●●● ● ● ●●●●●●●● ●●●● ●●● ●●●●●
●● ●●●● ●● ●●●●●● ●● ●●● ●●●●●●●● ● ●●● ●●●
56
5583
3682
0986
8390
8626
196
43.7
645
.85
42.1
148
.87
49.3
51.1
449
.47
49.0
552
.54
● ●●
● ●● ● ●●●●●●● ● ●● ●● ● ●● ●● ●● ●● ●●●●●●● ●●●● ●● ●● ● ●●● ●●● ●● ●● ●●● ●● ● ●●● ●● ●●●●●● ●●● ● ●●● ● ● ●●●●● ●●●●● ● ● ●●●● ●● ●● ●●
●● ●● ●● ●● ●●● ●● ●● ●● ●●●● ●●●● ●● ●●● ●●●●●● ●●●● ●● ● ● ● ●● ●● ● ●●●● ●● ● ●● ● ●●●● ● ●● ●● ●● ●● ●● ●●● ●● ●●●●● ●●●● ●●●●●●●● ● ●● ●●● ●●●●● ●● ● ●●● ●●● ●●● ●● ●●● ●● ●●●●● ●●● ● ●●●●● ●●● ●●● ●●●● ●
11
410
4989
2088
3226
712
65.4
51.7
49.9
853
.82
51.5
857
.05
59.1
958
.63
59.3
5
●
●
● ●●●● ●● ●●●● ●● ●● ●●●● ●●● ●●●●●●●
37
1725
542
55.9
964
.24
62.5
365
.48
75.1
4
35
7
20406080100
1997
1998
1999
2000
2001
2002
2003
2004
2005
1997
1998
1999
2000
2001
2002
2003
2004
2005
1997
1998
1999
2000
2001
2002
2003
2004
2005
Cat
egor
ies
NN Rasch
Figure
B.1.1:Boxplotof
LLRasch
against
procyearsplitinto
grades.
228 B.2. Grade 3 and Grade 5 Tests
B.2 Grade 3 and Grade 5 Tests
Before we investigate the progress in NN Rasch scores between grades, univariate
summary statistics of this data set are given in the form of plots and counts for each
of the explanatory variables (Figures B.2.1, B.2.2 and B.2.3).
We can see from these plots that there is no large imbalance between the groups of
the categorical variables, and the proportions of data in each group of a category
are reasonable and what we would expect based on what the groups represent in
real life.
Appendix B. Plots 229
●●●●●●●●●●●●
●●
●
●●
●
●●●●●●
●
●●●●
●●●●
●●
●●
●●
●●●●
●●●●
●●●●●●●●●
●●
●
●●
●
●●●●
●●●
●
●
●
●
●●
●●●●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●●●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●
●
●●●
●●
●
●●●
●●
●●
●
●
●
●
●●
●
●●●
●
●●
●
●●●
●●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●
●●
●
●
●●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
0
50
100
100 200 300 400 500 600School Number
Num
ber
of S
tude
nts
0
1000
2000
3000
0 100 200 300 400 500 600 700 800 900gpokm
Cou
nt
●●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●●●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●●
●
●
●
●
●
●●
●●
●
●
●●●
●
●
●
●
●
●
●●●●●
●
●
●●
●
●
●●
●
●●
●
●●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●
●●
●●
●
5210 600 170 119 144 537 209 498 152 231 17 7 13
−20
0
20
40
1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6 7 NAisolation
Diff
eren
ce in
Gra
de 3
and
5 s
core
s
●●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●●●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●●
●
●
●
●
●
●●
●●
●
●
●●●
●
●
●
●
●
●
●●●●●
●
●
●●
●
●
●●
●
●●
●
●●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●●●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
5183 1006 1226 416 76
−20
0
20
40
1.1 2.2.1 2.2.2 3.1 3.2spatial_ar
Diff
eren
ce in
Gra
de 3
and
5 s
core
s
●●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●
●●●●
●
●
●
●
●
●
●●
●●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●●●
●
●
●
●
●
●
●
●
●●●●●
●
●
●●
●
●
●
●●
●
●
●●
●
●●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●●●●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
5817 2090
−20
0
20
40
M Cstaff_metr
Diff
eren
ce in
Gra
de 3
and
5 s
core
s
●
●
●
●
●●●
●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●●
●
●
●●
●
●●●●
●
●●
●
●
●
●
●
●●●
●
●
●●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●●●●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
7675 155 77
−20
0
20
40
2 1 Inconsistentatsi
Diff
eren
ce in
Gra
de 3
and
5 s
core
s
Figure B.2.1: Plots of the number of students in each school, the distribution ofgpokm values and boxplots for the distribution of the di�erence in Grade 3 andGrade 5 scores in each category of isolation, spatial_ar, staff_metr and atsi.
230 B.2. Grade 3 and Grade 5 Tests
●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●●●
●●
●
●
●
●●●
●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●●
●
●●
●
●
●
●
●
6662 197 1048
−20
0
20
40
2 1 Inconsistentlbote
Diff
eren
ce in
Gra
de 3
and
5 s
core
s
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●●●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
3847 4060
−20
0
20
40
F Mgender
Diff
eren
ce in
Gra
de 3
and
5 s
core
s
●
●
●
●
●●●
●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●●
●●
●
●
●●
●
●●●●
●
●●
●
●
●
●
●
●●●
●
●
●●
●
●
●
●●
●
●
●
●●●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●●●●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
7729 178
−20
0
20
40
N Yaboriginal
Diff
eren
ce in
Gra
de 3
and
5 s
core
s
●
●
●
●
●●●
●
●●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●●
●
●●●●
●
●●
●
●
●
●
●
●●
●
●
●●●
●
●
●●●
●
●●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●●●●●●●●●●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
7497 410
−20
0
20
40
N Ydisability
Diff
eren
ce in
Gra
de 3
and
5 s
core
s
●
●
●
●
●
●●●
●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●●
●
●
●●
●
●●●●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●●●
●
●●
●
●
●
●
●
●●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●●●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
7776 131
−20
0
20
40
N Yschool_car
Diff
eren
ce in
Gra
de 3
and
5 s
core
s
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●
●
●●
●
●
●
●
●
●
● ●
●●●
●
●●
●●
●
●
●
●●●
●
●
●●●
●
●
●
●
●
●●●
●●
●
●
●●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●●●●●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
488 390 736 775 922 659 3937
−20
0
20
40
0 1 2 3 4 8 NAoccupation
Diff
eren
ce in
Gra
de 3
and
5 s
core
s
Figure B.2.2: Boxplots of the distribution of the di�erence in Grade 3 and Grade5 scores in each category of lbote, gender, aboriginal, disability, school_carand occupation.
Appendix B. Plots 231
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●●●●
●
●
●●
●
●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●●●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
506 198 643 1167 1442 3951
−20
0
20
40
0 1 2 3 4 NAschool_edu
Diff
eren
ce in
Gra
de 3
and
5 s
core
s
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
● ●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●●●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
621 855 379 468 1551 4033
−20
0
20
40
0 5 6 7 8 NAnon_school
Diff
eren
ce in
Gra
de 3
and
5 s
core
s
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●●
●
●
●
●●●●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●
●
●
●●
●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
6631 1 1275
−20
0
20
40
F Inconsistent Mp_g_gender
Diff
eren
ce in
Gra
de 3
and
5 s
core
s
●
●
●
●
●●●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●●●●
●●
●
●
●
●
●
●●●
●
●
●●
●
●
●
●●
●
●
●●●
●
●●
●
●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●
●●●●●●●
●●
●
●
●
●
●
7229 677 1
−20
0
20
40
N Y NAp_g_nesb
Diff
eren
ce in
Gra
de 3
and
5 s
core
s
●
●
●
●
●●●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●●●
●●
●
●
●
●
●
●●●
●
●
●●
●
●
●
●●
●
●●●
●
●●
●●
●
●●
●
●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
7257 650
−20
0
20
40
N Yhome_langu
Diff
eren
ce in
Gra
de 3
and
5 s
core
s
0
200
400
600
−25 0 25Difference in Grade 3 and 5 scores
coun
t
Figure B.2.3: Boxplots of the distribution of the di�erence in Grade 3 and Grade5 scores in each category of school_edu, non_school, p_g_gender, p_g_nesb,home_langu and a histogram of the di�erences.
Appendix C
Output
C.1 Chapter 3: Initial Model Selection
C.1.1 Full model with Main E�ects
School-Number Model
Table C.1.1: Linear regression output of school-number
model
Estimate Std. Error t-value P -valueIntercept 47.7486 5.6792 8.41 0.0000schoolno28 0.1571 4.5314 0.03 0.9724schoolno29 4.5469 5.6263 0.81 0.4190schoolno30 3.8556 5.6407 0.68 0.4943schoolno31 3.7444 3.9395 0.95 0.3419schoolno33 4.9522 3.5406 1.40 0.1619schoolno35 5.9197 4.5155 1.31 0.1899schoolno37 1.4145 3.8369 0.37 0.7124schoolno39 5.4732 3.6301 1.51 0.1316schoolno41 1.5968 3.7544 0.43 0.6706schoolno42 4.0787 4.0731 1.00 0.3167schoolno43 0.2424 3.6884 0.07 0.9476schoolno46 -1.0136 7.3728 -0.14 0.8906schoolno47 12.1347 7.3682 1.65 0.0996schoolno48 6.2669 3.6287 1.73 0.0842schoolno50 1.2494 3.5813 0.35 0.7272
Continued
233
234 C.1. Chapter 3: Initial Model Selection
Estimate Std. Error t-value P -valueschoolno51 0.5469 3.8338 0.14 0.8866schoolno52 -4.0872 7.3676 -0.55 0.5791schoolno54 8.8168 3.9379 2.24 0.0252schoolno55 5.1139 3.8348 1.33 0.1824schoolno57 13.3749 5.6262 2.38 0.0175schoolno58 3.4265 3.9379 0.87 0.3842schoolno61 3.4908 3.3335 1.05 0.2950schoolno62 1.6237 5.6275 0.29 0.7729schoolno63 -3.7926 3.7516 -1.01 0.3121schoolno65 9.1776 3.5815 2.56 0.0104schoolno67 -3.5299 3.3798 -1.04 0.2963schoolno68 6.2562 4.0871 1.53 0.1259schoolno69 6.0980 3.8337 1.59 0.1117schoolno70 6.5003 3.6864 1.76 0.0779schoolno72 8.2400 3.7550 2.19 0.0282schoolno73 2.0625 3.4007 0.61 0.5442schoolno75 0.1222 3.5394 0.03 0.9725schoolno76 -1.6420 3.6864 -0.45 0.6560schoolno77 -0.9603 3.5050 -0.27 0.7841schoolno78 -3.4845 4.2555 -0.82 0.4129schoolno79 0.8353 3.5394 0.24 0.8134schoolno82 -7.3652 5.6290 -1.31 0.1907schoolno83 3.3035 3.3995 0.97 0.3312schoolno84 0.9581 4.2538 0.23 0.8218schoolno85 1.0542 3.7516 0.28 0.7787schoolno87 -0.5206 3.6865 -0.14 0.8877schoolno89 6.9128 3.2100 2.15 0.0313schoolno90 -0.9232 3.7504 -0.25 0.8056schoolno91 6.0872 3.5416 1.72 0.0857schoolno94 1.4965 3.6833 0.41 0.6845schoolno95 4.7751 3.4469 1.39 0.1660schoolno96 7.8580 3.4465 2.28 0.0226schoolno98 5.5181 3.6279 1.52 0.1283schoolno99 8.6240 3.4225 2.52 0.0117schoolno100 0.8888 3.5417 0.25 0.8018schoolno101 10.5028 7.3660 1.43 0.1539schoolno102 0.5603 3.5405 0.16 0.8743schoolno103 8.0802 5.6484 1.43 0.1526schoolno105 5.1498 3.3474 1.54 0.1240schoolno106 2.9847 3.3635 0.89 0.3749schoolno107 2.2185 3.5802 0.62 0.5355schoolno108 5.9548 3.6844 1.62 0.1061
Continued
Appendix C. Output 235
Estimate Std. Error t-value P -valueschoolno109 5.1012 3.5799 1.42 0.1542schoolno111 4.7210 3.5047 1.35 0.1780schoolno112 7.9641 3.4216 2.33 0.0199schoolno114 6.7200 4.0735 1.65 0.0990schoolno115 -5.7713 3.4333 -1.68 0.0928schoolno116 2.9168 4.0724 0.72 0.4738schoolno117 2.6615 3.8340 0.69 0.4876schoolno118 3.9719 3.5792 1.11 0.2671schoolno120 1.1958 3.2566 0.37 0.7135schoolno123 5.0767 3.2870 1.54 0.1225schoolno124 6.2976 3.5409 1.78 0.0753schoolno125 3.9511 3.2491 1.22 0.2240schoolno126 0.0335 3.4733 0.01 0.9923schoolno127 4.4203 3.4471 1.28 0.1997schoolno128 3.6990 3.3458 1.11 0.2689schoolno129 4.9709 3.3466 1.49 0.1375schoolno130 2.4257 3.4234 0.71 0.4786schoolno134 7.0893 3.2580 2.18 0.0296schoolno135 3.9610 3.2409 1.22 0.2216schoolno138 -5.0144 4.0715 -1.23 0.2181schoolno139 1.4080 4.9215 0.29 0.7748schoolno140 6.9575 3.9388 1.77 0.0773schoolno141 10.7094 3.5815 2.99 0.0028schoolno142 11.9450 3.2842 3.64 0.0003schoolno143 -5.3684 3.9388 -1.36 0.1729schoolno146 1.7381 3.2767 0.53 0.5958schoolno147 4.7648 3.2156 1.48 0.1384schoolno148 5.5869 3.3825 1.65 0.0986schoolno149 5.0012 3.2953 1.52 0.1291schoolno150 0.2350 3.4224 0.07 0.9452schoolno151 5.6582 4.5129 1.25 0.2099schoolno152 -0.4698 3.2063 -0.15 0.8835schoolno153 7.2582 3.2649 2.22 0.0262schoolno154 -8.4475 4.5135 -1.87 0.0613schoolno155 5.1327 4.9110 1.05 0.2960schoolno157 2.2857 3.4000 0.67 0.5014schoolno158 -0.4009 4.9421 -0.08 0.9354schoolno159 -5.9718 3.7055 -1.61 0.1071schoolno160 7.6924 3.2655 2.36 0.0185schoolno161 6.6706 3.1782 2.10 0.0358schoolno162 1.7818 3.1960 0.56 0.5772schoolno164 5.7277 3.2045 1.79 0.0739
Continued
236 C.1. Chapter 3: Initial Model Selection
Estimate Std. Error t-value P -valueschoolno166 -0.8678 3.4733 -0.25 0.8027schoolno167 4.2271 3.5822 1.18 0.2380schoolno168 7.1711 3.4221 2.10 0.0361schoolno169 0.6549 3.2952 0.20 0.8425schoolno170 -29.5639 7.4356 -3.98 0.0001schoolno171 4.7492 3.3637 1.41 0.1580schoolno172 1.8342 3.7509 0.49 0.6248schoolno173 0.4599 3.6852 0.12 0.9007schoolno175 -4.0080 4.9128 -0.82 0.4146schoolno182 9.0645 3.3067 2.74 0.0061schoolno183 3.2030 3.5804 0.89 0.3710schoolno184 6.1008 3.3059 1.85 0.0650schoolno185 1.8111 3.2004 0.57 0.5715schoolno186 -0.1246 3.2492 -0.04 0.9694schoolno187 2.7968 3.2057 0.87 0.3830schoolno188 -1.3095 3.2216 -0.41 0.6844schoolno189 0.6575 3.1614 0.21 0.8353schoolno190 2.5114 3.2222 0.78 0.4358schoolno191 0.5747 3.2102 0.18 0.8579schoolno192 0.9603 3.2418 0.30 0.7671schoolno193 -2.0599 3.6314 -0.57 0.5705schoolno194 1.4172 3.2957 0.43 0.6672schoolno197 3.0334 3.8340 0.79 0.4288schoolno199 4.5873 3.2412 1.42 0.1570schoolno200 0.2154 3.2888 0.07 0.9478schoolno201 3.8909 3.2000 1.22 0.2240schoolno205 2.4210 3.2109 0.75 0.4509schoolno206 -1.9494 5.6268 -0.35 0.7290schoolno207 3.7387 3.4003 1.10 0.2716schoolno213 4.9947 3.1745 1.57 0.1157schoolno214 3.1983 3.2111 1.00 0.3193schoolno219 2.2386 3.1499 0.71 0.4773schoolno220 3.8820 4.9137 0.79 0.4295schoolno221 0.1310 3.2434 0.04 0.9678schoolno222 -3.0544 3.3832 -0.90 0.3666schoolno223 -1.7803 3.3084 -0.54 0.5905schoolno228 1.3550 3.1706 0.43 0.6691schoolno229 2.6077 3.5859 0.73 0.4671schoolno230 4.8285 3.1958 1.51 0.1308schoolno232 6.7412 3.6835 1.83 0.0672schoolno234 -1.4607 3.2485 -0.45 0.6530schoolno236 1.9408 3.1100 0.62 0.5326
Continued
Appendix C. Output 237
Estimate Std. Error t-value P -valueschoolno237 5.8013 3.1872 1.82 0.0687schoolno239 6.7499 4.2542 1.59 0.1126schoolno244 5.4242 3.4474 1.57 0.1156schoolno245 6.7909 3.6874 1.84 0.0655schoolno246 -1.1589 3.1270 -0.37 0.7109schoolno247 4.4078 3.1496 1.40 0.1617schoolno248 2.8513 3.4236 0.83 0.4049schoolno249 2.7977 3.1270 0.89 0.3710schoolno251 4.3967 3.5076 1.25 0.2101schoolno252 3.1747 3.1399 1.01 0.3120schoolno253 2.8194 3.2559 0.87 0.3865schoolno256 0.5576 3.5554 0.16 0.8754schoolno257 1.6125 3.2570 0.50 0.6206schoolno259 3.6128 3.1053 1.16 0.2447schoolno261 0.1234 3.3064 0.04 0.9702schoolno262 1.7145 3.1902 0.54 0.5910schoolno263 3.1970 3.3659 0.95 0.3422schoolno267 0.8550 3.1282 0.27 0.7846schoolno268 4.6606 3.1125 1.50 0.1343schoolno270 4.2796 3.2748 1.31 0.1913schoolno271 5.9590 3.3658 1.77 0.0767schoolno273 1.8308 3.2050 0.57 0.5678schoolno274 -0.2437 3.9405 -0.06 0.9507schoolno277 5.4636 3.1103 1.76 0.0790schoolno278 -6.5504 3.2663 -2.01 0.0449schoolno279 1.3768 3.1819 0.43 0.6652schoolno282 2.3303 3.1962 0.73 0.4660schoolno283 0.8498 3.1271 0.27 0.7858schoolno284 2.7437 3.2424 0.85 0.3975schoolno287 -2.6342 3.1702 -0.83 0.4060schoolno294 6.7520 3.1577 2.14 0.0325schoolno295 4.3675 3.1267 1.40 0.1625schoolno296 3.7361 3.3829 1.10 0.2694schoolno297 4.1978 3.0987 1.35 0.1755schoolno298 0.5533 3.1710 0.17 0.8615schoolno299 3.4217 3.1079 1.10 0.2709schoolno300 0.6895 3.1206 0.22 0.8251schoolno304 3.3616 3.3197 1.01 0.3113schoolno305 4.6618 3.2570 1.43 0.1524schoolno308 1.1713 3.0792 0.38 0.7036schoolno310 5.1310 3.9403 1.30 0.1929schoolno311 1.7612 3.1286 0.56 0.5735
Continued
238 C.1. Chapter 3: Initial Model Selection
Estimate Std. Error t-value P -valueschoolno312 -2.2225 3.2222 -0.69 0.4904schoolno313 -0.8512 3.1449 -0.27 0.7867schoolno315 1.7648 3.4508 0.51 0.6091schoolno316 1.4131 3.5077 0.40 0.6871schoolno318 3.2811 3.1608 1.04 0.2993schoolno319 -4.0144 3.2011 -1.25 0.2098schoolno320 3.9318 3.2666 1.20 0.2287schoolno322 -3.0411 3.1750 -0.96 0.3382schoolno323 5.6739 3.2216 1.76 0.0782schoolno324 7.8083 3.2426 2.41 0.0160schoolno325 3.8914 3.1586 1.23 0.2180schoolno326 3.0823 3.1396 0.98 0.3262schoolno330 5.1860 3.1517 1.65 0.0999schoolno333 3.1121 3.8358 0.81 0.4172schoolno334 1.4260 3.1345 0.45 0.6492schoolno335 -1.1864 3.2219 -0.37 0.7127schoolno336 3.5505 3.2780 1.08 0.2788schoolno337 2.4189 3.0970 0.78 0.4348schoolno338 2.1043 3.6846 0.57 0.5679schoolno339 3.2929 3.6272 0.91 0.3640schoolno343 5.0027 3.1580 1.58 0.1132schoolno344 5.9441 3.1442 1.89 0.0587schoolno350 3.2758 3.0729 1.07 0.2864schoolno351 2.7616 3.1724 0.87 0.3840schoolno353 2.7985 3.0802 0.91 0.3636schoolno354 3.5926 3.1871 1.13 0.2597schoolno355 -2.5078 3.1791 -0.79 0.4302schoolno356 4.8283 3.1648 1.53 0.1271schoolno357 3.8749 3.1490 1.23 0.2185schoolno358 2.9551 3.0746 0.96 0.3365schoolno359 3.1912 3.0581 1.04 0.2967schoolno361 -1.0107 3.1713 -0.32 0.7500schoolno362 -0.3901 3.3191 -0.12 0.9064schoolno363 5.1625 3.7527 1.38 0.1689schoolno364 0.5717 3.6858 0.16 0.8767schoolno365 1.7795 3.0878 0.58 0.5644schoolno366 0.9063 3.1363 0.29 0.7726schoolno369 1.6739 3.2103 0.52 0.6021schoolno371 1.3169 3.2652 0.40 0.6867schoolno372 7.5559 3.3324 2.27 0.0234schoolno373 6.2729 3.1047 2.02 0.0433schoolno374 1.0516 3.0659 0.34 0.7316
Continued
Appendix C. Output 239
Estimate Std. Error t-value P -valueschoolno376 2.5931 3.1555 0.82 0.4112schoolno377 1.6561 3.1029 0.53 0.5935schoolno378 0.1973 3.1724 0.06 0.9504schoolno380 4.2199 3.1170 1.35 0.1758schoolno381 2.4499 3.1298 0.78 0.4338schoolno382 1.3117 3.3630 0.39 0.6965schoolno385 1.3966 3.0686 0.46 0.6490schoolno386 -1.5732 3.2130 -0.49 0.6244schoolno387 4.6276 3.1126 1.49 0.1371schoolno388 -0.0468 3.0824 -0.02 0.9879schoolno389 2.6390 3.1723 0.83 0.4055schoolno390 0.3540 3.1322 0.11 0.9100schoolno391 0.2360 3.1166 0.08 0.9396schoolno392 5.3734 3.1324 1.72 0.0863schoolno393 3.0122 3.0984 0.97 0.3310schoolno395 1.2291 3.1223 0.39 0.6938schoolno396 1.6342 3.1779 0.51 0.6071schoolno398 1.5470 3.1946 0.48 0.6282schoolno399 3.9328 3.1778 1.24 0.2159schoolno400 0.7742 3.0856 0.25 0.8019schoolno401 4.2432 3.0766 1.38 0.1678schoolno403 14.3568 3.9423 3.64 0.0003schoolno405 1.3110 3.1006 0.42 0.6724schoolno406 1.3256 3.1742 0.42 0.6762schoolno407 6.0747 3.1059 1.96 0.0505schoolno410 -0.5258 3.2520 -0.16 0.8715schoolno413 5.4367 3.1088 1.75 0.0803schoolno414 -4.3770 3.1899 -1.37 0.1700schoolno415 2.3525 3.1951 0.74 0.4616schoolno416 3.4705 3.0982 1.12 0.2627schoolno417 2.2290 3.1199 0.71 0.4750schoolno418 1.0450 3.0641 0.34 0.7331schoolno419 0.6430 3.0879 0.21 0.8350schoolno420 0.2528 3.1272 0.08 0.9356schoolno421 3.2875 3.1649 1.04 0.2989schoolno422 4.7373 3.2757 1.45 0.1481schoolno423 -1.0888 3.0946 -0.35 0.7250schoolno425 5.5177 3.1182 1.77 0.0768schoolno426 -3.3561 3.1369 -1.07 0.2847schoolno427 0.3736 3.1896 0.12 0.9068schoolno428 3.7068 3.0642 1.21 0.2264schoolno430 1.5252 3.1098 0.49 0.6238
Continued
240 C.1. Chapter 3: Initial Model Selection
Estimate Std. Error t-value P -valueschoolno431 1.9002 3.1456 0.60 0.5458schoolno432 4.7849 3.0830 1.55 0.1207schoolno434 6.0440 4.9166 1.23 0.2190schoolno435 -0.5413 3.0796 -0.18 0.8605schoolno437 2.1655 4.5132 0.48 0.6314schoolno439 1.4502 3.2408 0.45 0.6545schoolno441 5.0041 3.0773 1.63 0.1039schoolno442 -0.0845 3.6273 -0.02 0.9814schoolno444 -1.6722 3.0577 -0.55 0.5845schoolno445 3.2054 3.2025 1.00 0.3169schoolno446 4.8288 3.3384 1.45 0.1481schoolno447 3.0954 3.0810 1.00 0.3151schoolno448 0.9600 3.2172 0.30 0.7654schoolno450 6.0598 3.9443 1.54 0.1245schoolno451 5.3828 3.3093 1.63 0.1038schoolno453 5.1076 3.0642 1.67 0.0956schoolno454 4.6462 3.1824 1.46 0.1443schoolno455 -0.0602 3.1216 -0.02 0.9846schoolno456 3.4684 3.1310 1.11 0.2680schoolno457 3.1505 3.0614 1.03 0.3034schoolno458 2.6376 3.1329 0.84 0.3998schoolno459 2.3616 3.6288 0.65 0.5152schoolno460 4.9692 3.0581 1.62 0.1042schoolno461 -1.1094 3.1442 -0.35 0.7242schoolno465 2.0878 3.2158 0.65 0.5162schoolno466 3.7381 3.0915 1.21 0.2266schoolno467 2.3920 3.4751 0.69 0.4913schoolno468 2.5430 3.0494 0.83 0.4043schoolno469 1.6296 3.1190 0.52 0.6013schoolno470 2.6142 3.1150 0.84 0.4014schoolno471 -1.5927 3.1080 -0.51 0.6083schoolno472 5.5645 3.0910 1.80 0.0718schoolno473 4.1886 3.2360 1.29 0.1956schoolno474 -1.1872 3.1203 -0.38 0.7036schoolno475 -0.2483 3.1750 -0.08 0.9377schoolno476 0.9584 3.1779 0.30 0.7630schoolno477 -0.8998 3.0810 -0.29 0.7702schoolno478 5.1094 3.0992 1.65 0.0992schoolno479 3.2154 3.1036 1.04 0.3002schoolno480 3.0265 3.0780 0.98 0.3255schoolno482 2.6452 3.1417 0.84 0.3998schoolno483 5.2972 3.1056 1.71 0.0881
Continued
Appendix C. Output 241
Estimate Std. Error t-value P -valueschoolno484 -1.2949 3.0573 -0.42 0.6719schoolno485 5.3582 3.0621 1.75 0.0802schoolno486 5.1688 3.0448 1.70 0.0896schoolno489 5.8512 3.0896 1.89 0.0583schoolno490 2.3584 3.1490 0.75 0.4539schoolno491 3.1782 3.0670 1.04 0.3001schoolno492 2.4884 3.0975 0.80 0.4218schoolno493 4.4993 3.1250 1.44 0.1500schoolno494 -0.0883 3.0853 -0.03 0.9772schoolno495 4.8883 3.0632 1.60 0.1106schoolno496 3.8424 3.2235 1.19 0.2333schoolno497 2.7840 3.2769 0.85 0.3956schoolno499 -4.0485 4.5115 -0.90 0.3695schoolno500 2.5901 3.2378 0.80 0.4237schoolno502 1.9598 3.1033 0.63 0.5277schoolno503 0.4524 3.4034 0.13 0.8942schoolno504 3.3198 3.3103 1.00 0.3159schoolno505 4.1517 3.1551 1.32 0.1882schoolno508 3.6686 3.0655 1.20 0.2314schoolno509 4.0238 3.0875 1.30 0.1925schoolno510 5.5255 3.1609 1.75 0.0805schoolno512 -1.3319 3.0446 -0.44 0.6618schoolno515 5.5230 3.1286 1.77 0.0775schoolno516 1.5992 3.1055 0.51 0.6066schoolno517 1.3896 3.0803 0.45 0.6519schoolno519 -1.6443 3.0904 -0.53 0.5947schoolno521 5.1379 3.0881 1.66 0.0962schoolno522 3.0168 3.0541 0.99 0.3233schoolno523 4.7379 3.1181 1.52 0.1287schoolno524 0.8445 3.0710 0.27 0.7833schoolno525 0.4174 3.2852 0.13 0.8989schoolno526 3.7348 3.0453 1.23 0.2201schoolno527 7.9240 3.0575 2.59 0.0096schoolno528 7.7967 3.1289 2.49 0.0127schoolno529 3.8457 3.1242 1.23 0.2184schoolno531 2.7117 3.0615 0.89 0.3758schoolno532 2.9516 3.1353 0.94 0.3465schoolno533 2.4911 3.0663 0.81 0.4166schoolno534 3.7111 3.3227 1.12 0.2641schoolno536 -0.4293 3.1811 -0.13 0.8926schoolno537 -1.9230 7.3784 -0.26 0.7944schoolno538 3.3116 3.1305 1.06 0.2901
Continued
242 C.1. Chapter 3: Initial Model Selection
Estimate Std. Error t-value P -valueschoolno539 7.1896 3.0825 2.33 0.0197schoolno540 5.6380 3.1829 1.77 0.0765schoolno541 4.0501 3.9392 1.03 0.3039schoolno542 2.2298 3.0681 0.73 0.4674schoolno543 1.6655 3.0881 0.54 0.5897schoolno545 7.6255 3.0832 2.47 0.0134schoolno546 7.5842 3.0694 2.47 0.0135schoolno547 2.8910 3.0486 0.95 0.3430schoolno548 2.6776 3.0409 0.88 0.3786schoolno549 1.0646 4.0805 0.26 0.7942schoolno550 3.6627 3.2884 1.11 0.2654schoolno551 2.4242 3.2961 0.74 0.4621schoolno552 6.3693 3.0542 2.09 0.0370schoolno553 5.0309 3.1524 1.60 0.1105schoolno554 -1.7526 3.1632 -0.55 0.5796schoolno555 -1.2211 3.0681 -0.40 0.6906schoolno556 6.7398 3.0587 2.20 0.0276schoolno557 3.8511 3.0671 1.26 0.2093schoolno558 4.3927 3.0603 1.44 0.1512schoolno559 0.3754 3.1071 0.12 0.9038schoolno561 -0.8744 3.0870 -0.28 0.7770schoolno564 1.3889 3.1969 0.43 0.6640schoolno566 6.4768 3.0596 2.12 0.0343schoolno567 2.6849 3.0493 0.88 0.3786schoolno568 4.9512 3.1105 1.59 0.1115schoolno569 3.0122 3.0932 0.97 0.3302schoolno571 5.2225 3.0512 1.71 0.0870schoolno573 5.6567 3.0832 1.83 0.0666schoolno574 4.4645 3.0659 1.46 0.1454schoolno575 3.0786 3.1864 0.97 0.3340schoolno576 4.5801 3.0528 1.50 0.1336schoolno578 0.2534 3.0747 0.08 0.9343schoolno581 4.9604 3.0531 1.62 0.1042schoolno584 4.4964 3.0811 1.46 0.1445schoolno593 7.1204 3.0890 2.31 0.0212schoolno595 1.7282 3.1294 0.55 0.5808schoolno596 1.5750 3.0952 0.51 0.6109schoolno597 1.7715 3.1233 0.57 0.5706schoolno599 1.8692 3.3204 0.56 0.5735schoolno600 0.9382 3.1206 0.30 0.7637schoolno608 1.8778 3.1328 0.60 0.5489schoolno614 3.9615 3.2064 1.24 0.2167
Continued
Appendix C. Output 243
Estimate Std. Error t-value P -valueschoolno623 8.1360 3.2739 2.49 0.0130schoolno624 3.7130 3.0691 1.21 0.2264schoolno628 4.3240 4.9277 0.88 0.3802schoolno637 1.3941 7.3773 0.19 0.8501schoolno639 0.6531 3.1128 0.21 0.8338schoolno649 3.3143 3.0412 1.09 0.2758schoolno650 2.8320 3.0410 0.93 0.3517procyear1999 -7.4803 6.2054 -1.21 0.2280procyear2000 -1.1543 4.8209 -0.24 0.8108procyear2001 -1.0363 4.8162 -0.22 0.8296procyear2002 -0.0091 4.8153 -0.00 0.9985procyear2003 -0.6325 4.8150 -0.13 0.8955procyear2004 -1.2816 4.8161 -0.27 0.7902procyear2005 4.8264 5.6888 0.85 0.3962gradedyear5 8.8438 0.1211 73.03 0.0000gradedyear7 16.4989 0.4210 39.19 0.0000atsi1 -2.0826 0.9092 -2.29 0.0220atsiInconsistent -1.9754 0.7824 -2.52 0.0116lbote1 -0.7619 0.2969 -2.57 0.0103lboteInconsistent -0.4326 0.2114 -2.05 0.0408genderM 1.0748 0.0965 11.14 0.0000aboriginalY -1.3835 0.8713 -1.59 0.1123disabilityY -8.1355 0.2147 -37.89 0.0000school_carY -0.7971 0.1900 -4.20 0.0000occupation1 0.7338 0.2961 2.48 0.0132occupation2 0.6044 0.2495 2.42 0.0154occupation3 0.6260 0.2426 2.58 0.0099occupation4 0.1599 0.2398 0.67 0.5049occupation8 0.1782 0.2466 0.72 0.4699school_edu1 -1.6512 0.3769 -4.38 0.0000school_edu2 -1.3228 0.3263 -4.05 0.0001school_edu3 -0.4219 0.3151 -1.34 0.1805school_edu4 0.2808 0.3172 0.89 0.3760non_school5 0.6838 0.2784 2.46 0.0140non_school6 1.5297 0.3052 5.01 0.0000non_school7 2.2191 0.3262 6.80 0.0000non_school8 0.6472 0.2688 2.41 0.0161p_g_genderM 0.0331 0.1415 0.23 0.8148p_g_nesbY -0.2325 0.2241 -1.04 0.2995
244 C.1. Chapter 3: Initial Model Selection
School-Covariates Model
Table C.1.2: Linear regression output of school-covariates
model
Estimate Std. Error t-value P -valueIntercept 52.0977 4.9951 10.43 0.0000procyear1999 -9.6148 6.4424 -1.49 0.1356procyear2000 -2.8535 4.9975 -0.57 0.5680procyear2001 -2.5657 4.9929 -0.51 0.6073procyear2002 -1.4896 4.9919 -0.30 0.7654procyear2003 -1.9823 4.9916 -0.40 0.6913procyear2004 -2.5689 4.9925 -0.51 0.6069procyear2005 2.6786 5.9040 0.45 0.6501gradedyear5 8.6832 0.1235 70.33 0.0000gradedyear7 16.1488 0.4296 37.59 0.0000gpokm -0.0095 0.0020 -4.70 0.0000isolation1.5 0.0667 0.7499 0.09 0.9292isolation2 -2.3020 1.1960 -1.92 0.0543isolation2.5 1.8365 1.1942 1.54 0.1241isolation3 4.0460 1.2784 3.16 0.0016isolation3.5 1.8044 1.2401 1.46 0.1457isolation4 2.1361 1.2956 1.65 0.0992isolation4.5 3.8550 1.3818 2.79 0.0053isolation5 3.2281 1.6227 1.99 0.0467isolation5.5 2.8284 1.6350 1.73 0.0837isolation6 1.5005 1.9891 0.75 0.4507isolation6.5 12.9792 5.3562 2.42 0.0154isolation7 3.4680 3.0958 1.12 0.2626spatial_ar2.2.1 0.2797 0.7323 0.38 0.7025spatial_ar2.2.2 -0.2527 0.8225 -0.31 0.7586spatial_ar3.1 1.5225 0.9615 1.58 0.1133spatial_ar3.2 3.5129 1.1601 3.03 0.0025staff_metrC -0.5350 0.8650 -0.62 0.5363atsi1 -2.5740 0.9317 -2.76 0.0057atsiInconsistent -2.7961 0.8021 -3.49 0.0005lbote1 -0.9101 0.3009 -3.03 0.0025lboteInconsistent -0.4461 0.2177 -2.05 0.0405genderM 1.1482 0.0997 11.51 0.0000aboriginalY -1.7597 0.8980 -1.96 0.0501disabilityY -8.5044 0.2194 -38.76 0.0000school_carY -1.0799 0.1941 -5.56 0.0000occupation1 1.0991 0.2971 3.70 0.0002
Continued
Appendix C. Output 245
Estimate Std. Error t-value P -valueoccupation2 1.1309 0.2469 4.58 0.0000occupation3 0.9514 0.2386 3.99 0.0001occupation4 0.1673 0.2356 0.71 0.4777occupation8 -0.2730 0.2412 -1.13 0.2578school_edu1 -2.5957 0.3774 -6.88 0.0000school_edu2 -1.9995 0.3248 -6.16 0.0000school_edu3 -0.6940 0.3122 -2.22 0.0262school_edu4 0.1199 0.3154 0.38 0.7038non_school5 0.6393 0.2828 2.26 0.0238non_school6 1.9020 0.3111 6.11 0.0000non_school7 2.9241 0.3313 8.83 0.0000non_school8 0.6556 0.2725 2.41 0.0161p_g_genderM 0.1324 0.1378 0.96 0.3368p_g_nesbY -0.0612 0.2256 -0.27 0.7863school size 0.0027 0.0006 4.50 0.0000
C.1.2 Simplest Main E�ects Model by stepAIC
School-Number Model
Table C.1.3: Linear regression output of simplest-
stepAIC school-number model
Estimate Std. Error t-value P -valueIntercept 47.6956 5.6789 8.40 0.0000schoolno28 0.1298 4.5312 0.03 0.9771schoolno29 4.5563 5.6261 0.81 0.4180schoolno30 3.8292 5.6405 0.68 0.4972schoolno31 3.7557 3.9389 0.95 0.3404schoolno33 4.9393 3.5403 1.40 0.1630schoolno35 5.9071 4.5153 1.31 0.1908schoolno37 1.3998 3.8366 0.36 0.7152schoolno39 5.4714 3.6300 1.51 0.1318schoolno41 1.6201 3.7529 0.43 0.6660schoolno42 4.0759 4.0730 1.00 0.3170schoolno43 0.2474 3.6882 0.07 0.9465schoolno46 -1.0366 7.3725 -0.14 0.8882
Continued
246 C.1. Chapter 3: Initial Model Selection
Estimate Std. Error t-value P -valueschoolno47 12.1278 7.3679 1.65 0.0998schoolno48 6.2516 3.6286 1.72 0.0849schoolno50 1.2425 3.5811 0.35 0.7286schoolno51 0.5476 3.8336 0.14 0.8864schoolno52 -4.0675 7.3666 -0.55 0.5808schoolno54 8.8114 3.9378 2.24 0.0253schoolno55 5.1184 3.8347 1.33 0.1820schoolno57 13.3651 5.6259 2.38 0.0175schoolno58 3.4324 3.9376 0.87 0.3834schoolno61 3.4959 3.3334 1.05 0.2943schoolno62 1.6111 5.6273 0.29 0.7746schoolno63 -3.7869 3.7513 -1.01 0.3128schoolno65 9.1532 3.5814 2.56 0.0106schoolno67 -3.5359 3.3797 -1.05 0.2955schoolno68 6.2356 4.0870 1.53 0.1271schoolno69 6.0993 3.8336 1.59 0.1116schoolno70 6.4951 3.6863 1.76 0.0781schoolno72 8.2562 3.7539 2.20 0.0279schoolno73 2.0540 3.4005 0.60 0.5458schoolno75 0.1120 3.5391 0.03 0.9748schoolno76 -1.6383 3.6861 -0.44 0.6567schoolno77 -0.9693 3.5049 -0.28 0.7821schoolno78 -3.4953 4.2553 -0.82 0.4114schoolno79 0.8270 3.5392 0.23 0.8152schoolno82 -7.3763 5.6288 -1.31 0.1901schoolno83 3.2914 3.3992 0.97 0.3329schoolno84 0.9529 4.2537 0.22 0.8227schoolno85 1.0530 3.7515 0.28 0.7789schoolno87 -0.5449 3.6863 -0.15 0.8825schoolno89 6.9095 3.2099 2.15 0.0314schoolno90 -0.9300 3.7503 -0.25 0.8042schoolno91 6.0765 3.5414 1.72 0.0862schoolno94 1.4872 3.6831 0.40 0.6864schoolno95 4.7626 3.4467 1.38 0.1670schoolno96 7.8500 3.4463 2.28 0.0228schoolno98 5.5130 3.6278 1.52 0.1286schoolno99 8.6220 3.4224 2.52 0.0118schoolno100 0.8799 3.5416 0.25 0.8038schoolno101 10.5297 7.3650 1.43 0.1528schoolno102 0.5488 3.5403 0.16 0.8768schoolno103 7.9413 5.6467 1.41 0.1596schoolno105 5.1454 3.3473 1.54 0.1243
Continued
Appendix C. Output 247
Estimate Std. Error t-value P -valueschoolno106 2.9852 3.3634 0.89 0.3748schoolno107 2.2188 3.5801 0.62 0.5354schoolno108 5.9581 3.6842 1.62 0.1059schoolno109 5.0911 3.5797 1.42 0.1550schoolno111 4.7218 3.5047 1.35 0.1779schoolno112 7.9655 3.4214 2.33 0.0199schoolno114 6.7180 4.0734 1.65 0.0991schoolno115 -5.9170 3.4304 -1.72 0.0846schoolno116 2.9028 4.0721 0.71 0.4760schoolno117 2.6718 3.8338 0.70 0.4859schoolno118 3.9554 3.5790 1.11 0.2691schoolno120 1.1938 3.2565 0.37 0.7139schoolno123 5.0875 3.2868 1.55 0.1217schoolno124 6.2868 3.5407 1.78 0.0758schoolno125 3.9472 3.2490 1.21 0.2244schoolno126 0.0260 3.4731 0.01 0.9940schoolno127 4.4244 3.4471 1.28 0.1993schoolno128 3.6925 3.3456 1.10 0.2698schoolno129 4.9571 3.3463 1.48 0.1385schoolno130 2.4341 3.4231 0.71 0.4770schoolno134 7.1105 3.2571 2.18 0.0290schoolno135 3.9581 3.2408 1.22 0.2220schoolno138 -5.0161 4.0714 -1.23 0.2180schoolno139 1.3848 4.9212 0.28 0.7784schoolno140 6.9682 3.9384 1.77 0.0769schoolno141 10.7102 3.5807 2.99 0.0028schoolno142 11.9442 3.2841 3.64 0.0003schoolno143 -5.3560 3.9384 -1.36 0.1739schoolno146 1.7535 3.2756 0.54 0.5924schoolno147 4.7589 3.2155 1.48 0.1389schoolno148 5.5767 3.3822 1.65 0.0992schoolno149 4.9914 3.2951 1.51 0.1298schoolno150 0.2331 3.4221 0.07 0.9457schoolno151 5.6633 4.5126 1.25 0.2095schoolno152 -0.4730 3.2062 -0.15 0.8827schoolno153 7.2519 3.2648 2.22 0.0263schoolno154 -8.4579 4.5132 -1.87 0.0609schoolno155 5.1231 4.9108 1.04 0.2969schoolno157 2.2855 3.4000 0.67 0.5015schoolno158 -0.5163 4.9404 -0.10 0.9168schoolno159 -5.9615 3.7051 -1.61 0.1076schoolno160 7.6868 3.2655 2.35 0.0186
Continued
248 C.1. Chapter 3: Initial Model Selection
Estimate Std. Error t-value P -valueschoolno161 6.6646 3.1781 2.10 0.0360schoolno162 1.7743 3.1959 0.56 0.5788schoolno164 5.7251 3.2043 1.79 0.0740schoolno166 -0.8863 3.4731 -0.26 0.7986schoolno167 4.2336 3.5820 1.18 0.2373schoolno168 7.1609 3.4219 2.09 0.0364schoolno169 0.6536 3.2951 0.20 0.8428schoolno170 -29.7337 7.4337 -4.00 0.0001schoolno171 4.7435 3.3635 1.41 0.1585schoolno172 1.8272 3.7507 0.49 0.6261schoolno173 0.4634 3.6850 0.13 0.8999schoolno175 -4.1648 4.9103 -0.85 0.3963schoolno182 9.0594 3.3065 2.74 0.0062schoolno183 3.1964 3.5799 0.89 0.3719schoolno184 6.0953 3.3058 1.84 0.0652schoolno185 1.8039 3.2003 0.56 0.5730schoolno186 -0.1367 3.2489 -0.04 0.9664schoolno187 2.7856 3.2055 0.87 0.3849schoolno188 -1.3074 3.2215 -0.41 0.6849schoolno189 0.6503 3.1613 0.21 0.8370schoolno190 2.5044 3.2221 0.78 0.4370schoolno191 0.5663 3.2100 0.18 0.8600schoolno192 0.9753 3.2413 0.30 0.7635schoolno193 -2.0791 3.6312 -0.57 0.5669schoolno194 1.3729 3.2953 0.42 0.6770schoolno197 3.0255 3.8338 0.79 0.4300schoolno199 4.5792 3.2411 1.41 0.1577schoolno200 0.2107 3.2886 0.06 0.9489schoolno201 3.8907 3.1999 1.22 0.2240schoolno205 2.4312 3.2108 0.76 0.4489schoolno206 -1.9548 5.6266 -0.35 0.7283schoolno207 3.6986 3.4000 1.09 0.2767schoolno213 4.9901 3.1743 1.57 0.1160schoolno214 3.1633 3.2108 0.99 0.3245schoolno219 2.2419 3.1498 0.71 0.4766schoolno220 3.8708 4.9135 0.79 0.4308schoolno221 0.0739 3.2428 0.02 0.9818schoolno222 -3.0539 3.3831 -0.90 0.3667schoolno223 -1.8056 3.3082 -0.55 0.5852schoolno228 1.3483 3.1705 0.43 0.6707schoolno229 2.6178 3.5858 0.73 0.4654schoolno230 4.8136 3.1957 1.51 0.1320
Continued
Appendix C. Output 249
Estimate Std. Error t-value P -valueschoolno232 6.7392 3.6834 1.83 0.0673schoolno234 -1.4697 3.2483 -0.45 0.6509schoolno236 1.9394 3.1098 0.62 0.5329schoolno237 5.7902 3.1869 1.82 0.0693schoolno239 6.7441 4.2540 1.59 0.1129schoolno244 5.4167 3.4473 1.57 0.1161schoolno245 6.7847 3.6872 1.84 0.0658schoolno246 -1.1548 3.1269 -0.37 0.7119schoolno247 4.3991 3.1495 1.40 0.1625schoolno248 2.8150 3.4233 0.82 0.4109schoolno249 2.7923 3.1269 0.89 0.3719schoolno251 4.3642 3.5074 1.24 0.2134schoolno252 3.1714 3.1399 1.01 0.3125schoolno253 2.7887 3.2557 0.86 0.3917schoolno256 0.6151 3.5545 0.17 0.8626schoolno257 1.6155 3.2569 0.50 0.6199schoolno259 3.6149 3.1052 1.16 0.2444schoolno261 0.1222 3.3063 0.04 0.9705schoolno262 1.7179 3.1901 0.54 0.5902schoolno263 3.2034 3.3657 0.95 0.3412schoolno267 0.8557 3.1281 0.27 0.7844schoolno268 4.6739 3.1115 1.50 0.1331schoolno270 4.2378 3.2744 1.29 0.1956schoolno271 5.9480 3.3655 1.77 0.0772schoolno273 1.8192 3.2048 0.57 0.5703schoolno274 -0.2258 3.9403 -0.06 0.9543schoolno277 5.4525 3.1101 1.75 0.0796schoolno278 -6.5546 3.2662 -2.01 0.0448schoolno279 1.3446 3.1817 0.42 0.6726schoolno280 -0.4351 3.4000 -0.13 0.8982schoolno282 2.3352 3.1961 0.73 0.4650schoolno283 0.8504 3.1270 0.27 0.7857schoolno284 2.7336 3.2423 0.84 0.3992schoolno287 -2.6446 3.1701 -0.83 0.4041schoolno294 6.7458 3.1576 2.14 0.0327schoolno295 4.3402 3.1265 1.39 0.1651schoolno296 3.7182 3.3827 1.10 0.2717schoolno297 4.1823 3.0985 1.35 0.1771schoolno298 0.5467 3.1709 0.17 0.8631schoolno299 3.4152 3.1078 1.10 0.2718schoolno300 0.6570 3.1203 0.21 0.8332schoolno304 3.3648 3.3196 1.01 0.3108
Continued
250 C.1. Chapter 3: Initial Model Selection
Estimate Std. Error t-value P -valueschoolno305 4.6654 3.2569 1.43 0.1520schoolno308 1.1465 3.0790 0.37 0.7096schoolno310 5.1124 3.9402 1.30 0.1945schoolno311 1.7520 3.1285 0.56 0.5755schoolno312 -2.2264 3.2221 -0.69 0.4896schoolno313 -0.8578 3.1448 -0.27 0.7850schoolno315 1.7199 3.4505 0.50 0.6182schoolno316 1.4264 3.5075 0.41 0.6842schoolno318 3.2718 3.1608 1.04 0.3006schoolno319 -4.0323 3.2009 -1.26 0.2078schoolno320 3.9354 3.2665 1.20 0.2283schoolno322 -3.0682 3.1748 -0.97 0.3338schoolno323 5.6675 3.2214 1.76 0.0785schoolno324 7.7782 3.2424 2.40 0.0165schoolno325 3.8617 3.1584 1.22 0.2215schoolno326 3.0736 3.1395 0.98 0.3276schoolno330 5.1585 3.1515 1.64 0.1017schoolno333 3.0346 3.8350 0.79 0.4288schoolno334 1.4320 3.1336 0.46 0.6477schoolno335 -1.1942 3.2217 -0.37 0.7109schoolno336 3.5465 3.2777 1.08 0.2793schoolno337 2.3608 3.0964 0.76 0.4458schoolno338 2.1193 3.6843 0.58 0.5651schoolno339 3.2839 3.6271 0.91 0.3653schoolno343 4.9970 3.1578 1.58 0.1136schoolno344 5.9199 3.1439 1.88 0.0597schoolno350 3.2475 3.0726 1.06 0.2906schoolno351 2.7575 3.1722 0.87 0.3847schoolno353 2.7936 3.0801 0.91 0.3644schoolno354 3.5830 3.1870 1.12 0.2609schoolno355 -2.5098 3.1790 -0.79 0.4298schoolno356 4.8066 3.1646 1.52 0.1288schoolno357 3.8552 3.1489 1.22 0.2209schoolno358 2.9451 3.0744 0.96 0.3381schoolno359 3.1284 3.0574 1.02 0.3062schoolno361 -1.0123 3.1710 -0.32 0.7495schoolno362 -0.4200 3.3189 -0.13 0.8993schoolno363 5.1733 3.7526 1.38 0.1680schoolno364 0.5278 3.6854 0.14 0.8861schoolno365 1.7708 3.0877 0.57 0.5663schoolno366 0.8973 3.1361 0.29 0.7748schoolno369 1.6698 3.2101 0.52 0.6030
Continued
Appendix C. Output 251
Estimate Std. Error t-value P -valueschoolno371 1.3240 3.2651 0.41 0.6851schoolno372 7.5513 3.3323 2.27 0.0235schoolno373 6.2829 3.1040 2.02 0.0430schoolno374 1.0364 3.0658 0.34 0.7353schoolno376 2.5771 3.1553 0.82 0.4141schoolno377 1.6481 3.1027 0.53 0.5953schoolno378 0.1893 3.1722 0.06 0.9524schoolno380 4.2161 3.1169 1.35 0.1762schoolno381 2.4430 3.1297 0.78 0.4351schoolno382 1.2892 3.3628 0.38 0.7014schoolno385 1.3968 3.0685 0.46 0.6490schoolno386 -1.6821 3.2113 -0.52 0.6004schoolno387 4.6132 3.1125 1.48 0.1383schoolno388 -0.0479 3.0823 -0.02 0.9876schoolno389 2.6012 3.1720 0.82 0.4122schoolno390 0.3592 3.1321 0.11 0.9087schoolno391 0.2273 3.1164 0.07 0.9419schoolno392 5.3213 3.1320 1.70 0.0893schoolno393 3.0063 3.0983 0.97 0.3319schoolno395 1.2243 3.1222 0.39 0.6950schoolno396 1.6182 3.1778 0.51 0.6106schoolno398 1.5466 3.1945 0.48 0.6283schoolno399 3.9162 3.1777 1.23 0.2178schoolno400 0.7737 3.0854 0.25 0.8020schoolno401 4.2432 3.0765 1.38 0.1678schoolno403 14.3479 3.9421 3.64 0.0003schoolno405 1.2966 3.1004 0.42 0.6758schoolno406 1.3192 3.1740 0.42 0.6777schoolno407 6.0722 3.1056 1.96 0.0506schoolno410 -0.5611 3.2517 -0.17 0.8630schoolno413 5.4084 3.1085 1.74 0.0819schoolno414 -4.3832 3.1898 -1.37 0.1694schoolno415 2.3385 3.1949 0.73 0.4642schoolno416 3.4502 3.0981 1.11 0.2654schoolno417 2.2172 3.1197 0.71 0.4773schoolno418 1.0364 3.0639 0.34 0.7352schoolno419 0.6477 3.0878 0.21 0.8339schoolno420 0.2428 3.1270 0.08 0.9381schoolno421 3.2805 3.1648 1.04 0.2999schoolno422 4.6411 3.2743 1.42 0.1564schoolno423 -1.1250 3.0943 -0.36 0.7162schoolno425 5.5126 3.1181 1.77 0.0771
Continued
252 C.1. Chapter 3: Initial Model Selection
Estimate Std. Error t-value P -valueschoolno426 -3.3656 3.1368 -1.07 0.2833schoolno427 0.3725 3.1895 0.12 0.9070schoolno428 3.7065 3.0641 1.21 0.2264schoolno430 1.5350 3.1096 0.49 0.6216schoolno431 1.8948 3.1455 0.60 0.5469schoolno432 4.7913 3.0827 1.55 0.1201schoolno434 6.0664 4.9164 1.23 0.2172schoolno435 -0.5344 3.0791 -0.17 0.8622schoolno437 2.1535 4.5130 0.48 0.6332schoolno439 1.4394 3.2405 0.44 0.6569schoolno441 5.0018 3.0772 1.63 0.1041schoolno442 -0.0914 3.6271 -0.03 0.9799schoolno444 -1.6984 3.0576 -0.56 0.5786schoolno445 3.1921 3.2023 1.00 0.3189schoolno446 4.7216 3.3368 1.41 0.1571schoolno447 3.1140 3.0802 1.01 0.3121schoolno448 0.9052 3.2165 0.28 0.7784schoolno450 6.0471 3.9442 1.53 0.1253schoolno451 5.3899 3.3092 1.63 0.1034schoolno453 5.1086 3.0642 1.67 0.0955schoolno454 4.6219 3.1822 1.45 0.1464schoolno455 -0.0900 3.1213 -0.03 0.9770schoolno456 3.4586 3.1308 1.10 0.2693schoolno457 3.1273 3.0612 1.02 0.3070schoolno458 2.6263 3.1327 0.84 0.4018schoolno459 2.3046 3.6283 0.64 0.5253schoolno460 4.9541 3.0580 1.62 0.1052schoolno461 -1.1123 3.1441 -0.35 0.7235schoolno465 2.0958 3.2157 0.65 0.5146schoolno466 3.7297 3.0913 1.21 0.2276schoolno467 2.3839 3.4748 0.69 0.4927schoolno468 2.5353 3.0493 0.83 0.4057schoolno469 1.6219 3.1188 0.52 0.6030schoolno470 2.5993 3.1149 0.83 0.4040schoolno471 -1.5854 3.1079 -0.51 0.6100schoolno472 5.5347 3.0907 1.79 0.0734schoolno473 4.1856 3.2358 1.29 0.1959schoolno474 -1.1859 3.1203 -0.38 0.7039schoolno475 -0.2782 3.1748 -0.09 0.9302schoolno476 0.9492 3.1777 0.30 0.7652schoolno477 -0.9017 3.0809 -0.29 0.7698schoolno478 5.1010 3.0992 1.65 0.0998
Continued
Appendix C. Output 253
Estimate Std. Error t-value P -valueschoolno479 3.2097 3.1034 1.03 0.3010schoolno480 3.0172 3.0778 0.98 0.3269schoolno482 2.6385 3.1416 0.84 0.4010schoolno483 5.2869 3.1055 1.70 0.0887schoolno484 -1.2960 3.0572 -0.42 0.6716schoolno485 5.3252 3.0618 1.74 0.0820schoolno486 5.1551 3.0446 1.69 0.0904schoolno489 5.8045 3.0891 1.88 0.0603schoolno490 2.3393 3.1488 0.74 0.4575schoolno491 3.1687 3.0668 1.03 0.3015schoolno492 2.4389 3.0971 0.79 0.4310schoolno493 4.4446 3.1245 1.42 0.1549schoolno494 -0.0913 3.0852 -0.03 0.9764schoolno495 4.8883 3.0632 1.60 0.1105schoolno496 3.8203 3.2233 1.19 0.2359schoolno497 2.7242 3.2764 0.83 0.4057schoolno499 -4.0607 4.5113 -0.90 0.3681schoolno500 2.5866 3.2377 0.80 0.4244schoolno502 1.9420 3.1031 0.63 0.5314schoolno503 0.4335 3.4033 0.13 0.8986schoolno504 3.3446 3.3101 1.01 0.3123schoolno505 4.1188 3.1548 1.31 0.1917schoolno508 3.6405 3.0654 1.19 0.2350schoolno509 4.0209 3.0874 1.30 0.1928schoolno510 5.4792 3.1605 1.73 0.0830schoolno512 -1.3521 3.0445 -0.44 0.6570schoolno515 5.4852 3.1283 1.75 0.0795schoolno516 1.5929 3.1053 0.51 0.6080schoolno517 1.3724 3.0801 0.45 0.6559schoolno519 -1.6646 3.0902 -0.54 0.5901schoolno521 5.1300 3.0879 1.66 0.0967schoolno522 2.9797 3.0538 0.98 0.3292schoolno523 4.7345 3.1179 1.52 0.1289schoolno524 0.8219 3.0709 0.27 0.7890schoolno525 0.4222 3.2850 0.13 0.8977schoolno526 3.6854 3.0449 1.21 0.2261schoolno527 7.8937 3.0573 2.58 0.0098schoolno528 7.7792 3.1287 2.49 0.0129schoolno529 3.7919 3.1237 1.21 0.2248schoolno531 2.7061 3.0614 0.88 0.3767schoolno532 2.9375 3.1351 0.94 0.3488schoolno533 2.5026 3.0656 0.82 0.4143
Continued
254 C.1. Chapter 3: Initial Model Selection
Estimate Std. Error t-value P -valueschoolno534 3.7029 3.3226 1.11 0.2651schoolno536 -0.4485 3.1809 -0.14 0.8879schoolno537 -1.9351 7.3781 -0.26 0.7931schoolno538 3.3089 3.1304 1.06 0.2905schoolno539 7.1565 3.0823 2.32 0.0203schoolno540 5.6395 3.1827 1.77 0.0764schoolno541 4.0435 3.9390 1.03 0.3047schoolno542 2.1985 3.0678 0.72 0.4736schoolno543 1.6588 3.0880 0.54 0.5912schoolno545 7.6262 3.0831 2.47 0.0134schoolno546 7.5591 3.0692 2.46 0.0138schoolno547 2.8631 3.0484 0.94 0.3476schoolno548 2.6712 3.0408 0.88 0.3797schoolno549 1.1338 4.0794 0.28 0.7811schoolno550 3.6286 3.2882 1.10 0.2698schoolno551 2.4262 3.2960 0.74 0.4617schoolno552 6.3619 3.0539 2.08 0.0372schoolno553 5.0251 3.1523 1.59 0.1109schoolno554 -1.7544 3.1631 -0.55 0.5791schoolno555 -1.2070 3.0676 -0.39 0.6940schoolno556 6.7217 3.0586 2.20 0.0280schoolno557 3.7858 3.0664 1.23 0.2170schoolno558 4.3892 3.0601 1.43 0.1515schoolno559 0.3728 3.1069 0.12 0.9045schoolno561 -0.9191 3.0866 -0.30 0.7659schoolno564 1.3350 3.1963 0.42 0.6762schoolno566 6.4576 3.0594 2.11 0.0348schoolno567 2.6820 3.0492 0.88 0.3791schoolno568 4.9458 3.1104 1.59 0.1118schoolno569 3.0043 3.0930 0.97 0.3314schoolno571 5.1969 3.0510 1.70 0.0885schoolno573 5.6541 3.0831 1.83 0.0667schoolno574 4.4532 3.0657 1.45 0.1464schoolno575 3.0355 3.1860 0.95 0.3407schoolno576 4.5824 3.0526 1.50 0.1333schoolno578 0.2445 3.0745 0.08 0.9366schoolno581 4.9465 3.0530 1.62 0.1052schoolno584 4.4659 3.0809 1.45 0.1472schoolno593 7.0630 3.0885 2.29 0.0222schoolno595 1.7208 3.1293 0.55 0.5824schoolno596 1.5451 3.0950 0.50 0.6176schoolno597 1.7531 3.1232 0.56 0.5746
Continued
Appendix C. Output 255
Estimate Std. Error t-value P -valueschoolno599 1.8717 3.3203 0.56 0.5730schoolno600 0.9324 3.1205 0.30 0.7651schoolno608 1.8767 3.1328 0.60 0.5491schoolno614 3.9622 3.2063 1.24 0.2166schoolno623 8.1361 3.2738 2.49 0.0130schoolno624 3.7002 3.0689 1.21 0.2279schoolno628 4.3604 4.9275 0.88 0.3762schoolno637 1.3828 7.3771 0.19 0.8513schoolno639 0.6321 3.1127 0.20 0.8391schoolno649 3.3126 3.0411 1.09 0.2760schoolno650 2.8238 3.0409 0.93 0.3531procyear1999 -7.3968 6.2040 -1.19 0.2332procyear2000 -1.0910 4.8204 -0.23 0.8209procyear2001 -0.9709 4.8156 -0.20 0.8402procyear2002 0.0545 4.8147 0.01 0.9910procyear2003 -0.5724 4.8144 -0.12 0.9054procyear2004 -1.2166 4.8155 -0.25 0.8005procyear2005 4.8278 5.6882 0.85 0.3960gradedyear5 8.8460 0.1211 73.06 0.0000gradedyear7 16.5014 0.4210 39.19 0.0000atsi1 -2.0708 0.9090 -2.28 0.0227atsiInconsistent -1.9754 0.7823 -2.52 0.0116lbote1 -0.8875 0.2688 -3.30 0.0010lboteInconsistent -0.5128 0.1956 -2.62 0.0088genderM 1.0740 0.0965 11.13 0.0000aboriginalY -1.3763 0.8711 -1.58 0.1141disabilityY -8.1348 0.2147 -37.89 0.0000school_carY -0.7994 0.1900 -4.21 0.0000occupation1 0.7467 0.2955 2.53 0.0115occupation2 0.6149 0.2491 2.47 0.0136occupation3 0.6320 0.2425 2.61 0.0092occupation4 0.1628 0.2397 0.68 0.4970occupation8 0.1776 0.2465 0.72 0.4713school_edu1 -1.6505 0.3765 -4.38 0.0000school_edu2 -1.3199 0.3263 -4.05 0.0001school_edu3 -0.4206 0.3151 -1.33 0.1819school_edu4 0.2754 0.3172 0.87 0.3853non_school5 0.6797 0.2783 2.44 0.0146non_school6 1.5241 0.3051 5.00 0.0000non_school7 2.2135 0.3262 6.79 0.0000non_school8 0.6443 0.2687 2.40 0.0165
256 C.1. Chapter 3: Initial Model Selection
School-Covariates Model
Table C.1.4: Linear regression output of simplest-
stepAIC school-covariates model
Estimate Std. Error t-value P -valueIntercept 52.0345 4.9936 10.42 0.0000procyear1999 -9.4744 6.4401 -1.47 0.1413procyear2000 -2.7752 4.9959 -0.56 0.5786procyear2001 -2.4876 4.9914 -0.50 0.6182procyear2002 -1.4163 4.9904 -0.28 0.7766procyear2003 -1.9102 4.9902 -0.38 0.7019procyear2004 -2.4984 4.9910 -0.50 0.6167procyear2005 2.7650 5.9031 0.47 0.6395gradedyear5 8.6885 0.1233 70.45 0.0000gradedyear7 16.1636 0.4294 37.65 0.0000gpokm -0.0095 0.0020 -4.73 0.0000isolation1.5 0.0513 0.7490 0.07 0.9454isolation2 -2.8263 0.8268 -3.42 0.0006isolation2.5 1.3203 0.8363 1.58 0.1144isolation3 3.5421 0.9538 3.71 0.0002isolation3.5 1.2990 0.9061 1.43 0.1517isolation4 1.6305 0.9882 1.65 0.0990isolation4.5 3.3442 1.1102 3.01 0.0026isolation5 2.7441 1.4021 1.96 0.0503isolation5.5 2.3524 1.4190 1.66 0.0974isolation6 0.9823 1.8149 0.54 0.5883isolation6.5 12.4128 5.2948 2.34 0.0191isolation7 2.9338 2.9986 0.98 0.3279spatial_ar2.2.1 0.2784 0.7322 0.38 0.7038spatial_ar2.2.2 -0.2642 0.8223 -0.32 0.7480spatial_ar3.1 1.4888 0.9604 1.55 0.1211spatial_ar3.2 3.5158 1.1594 3.03 0.0024atsi1 -2.5664 0.9315 -2.76 0.0059atsiInconsistent -2.8092 0.8019 -3.50 0.0005lbote1 -0.9289 0.2663 -3.49 0.0005lboteInconsistent -0.4597 0.1994 -2.31 0.0212genderM 1.1475 0.0997 11.50 0.0000aboriginalY -1.7650 0.8978 -1.97 0.0493disabilityY -8.5013 0.2193 -38.76 0.0000
Continued
Appendix C. Output 257
Estimate Std. Error t-value P -valueschool_carY -1.0801 0.1940 -5.57 0.0000occupation1 1.1197 0.2963 3.78 0.0002occupation2 1.1458 0.2462 4.65 0.0000occupation3 0.9535 0.2385 4.00 0.0001occupation4 0.1710 0.2355 0.73 0.4679occupation8 -0.2811 0.2411 -1.17 0.2437school_edu1 -2.5863 0.3771 -6.86 0.0000school_edu2 -1.9975 0.3247 -6.15 0.0000school_edu3 -0.7029 0.3121 -2.25 0.0243school_edu4 0.1094 0.3153 0.35 0.7287non_school5 0.6498 0.2826 2.30 0.0215non_school6 1.8989 0.3110 6.11 0.0000non_school7 2.9271 0.3312 8.84 0.0000non_school8 0.6571 0.2723 2.41 0.0158school size 0.0027 0.0006 4.54 0.0000
258 C.2. Chapter 7. Initial Longitudinal Analysis
C.2 Chapter 7. Initial Longitudinal Analysis
C.2.1 Grade 3 and Grade 5 Tests
School-Number Model
Table C.2.1: Linear regression output of school-number
model
Estimate Std. Error t-value P -valueIntercept 27.5375 2.2463 12.26 0.0000Grade 3 Rasch -0.4165 0.0116 -35.98 0.0000schoolno41 -3.3498 3.4570 -0.97 0.3326schoolno61 2.6747 2.7700 0.97 0.3343schoolno67 -3.8928 3.1762 -1.23 0.2204schoolno73 2.4323 2.8618 0.85 0.3954schoolno75 0.4934 2.9885 0.17 0.8689schoolno77 2.3795 2.8656 0.83 0.4064schoolno79 1.1626 3.4486 0.34 0.7361schoolno84 -1.0347 5.1748 -0.20 0.8415schoolno85 3.1634 3.4496 0.92 0.3592schoolno89 4.2497 2.6961 1.58 0.1151schoolno91 0.0796 3.1811 0.03 0.9800schoolno94 -1.4702 3.4528 -0.43 0.6703schoolno95 3.4377 2.9869 1.15 0.2498schoolno96 5.2491 2.9926 1.75 0.0795schoolno102 -0.2318 3.4515 -0.07 0.9464schoolno105 3.0337 2.9895 1.01 0.3103schoolno106 0.5589 2.9884 0.19 0.8517schoolno114 3.9561 5.1851 0.76 0.4455schoolno120 1.5995 2.5894 0.62 0.5368schoolno123 2.9313 3.1716 0.92 0.3554schoolno125 1.5546 2.8678 0.54 0.5878schoolno127 2.9639 3.1671 0.94 0.3494schoolno128 0.2100 3.1683 0.07 0.9472schoolno130 1.4211 3.4478 0.41 0.6802schoolno134 -3.2760 2.9953 -1.09 0.2741schoolno135 5.0405 3.1729 1.59 0.1122schoolno142 2.5581 2.7671 0.92 0.3553schoolno146 -0.6618 2.8725 -0.23 0.8178schoolno147 2.2232 2.6935 0.83 0.4092schoolno148 -3.2607 2.9913 -1.09 0.2758
Continued
Appendix C. Output 259
Estimate Std. Error t-value P -valueschoolno149 2.7406 2.9859 0.92 0.3588schoolno152 0.5063 2.5522 0.20 0.8428schoolno153 5.2729 2.6935 1.96 0.0504schoolno155 2.1262 5.1698 0.41 0.6809schoolno157 -0.2197 3.4549 -0.06 0.9493schoolno160 2.1037 2.6958 0.78 0.4352schoolno161 3.2662 2.5153 1.30 0.1942schoolno162 0.5663 2.5190 0.22 0.8221schoolno164 3.3550 2.5883 1.30 0.1950schoolno167 -1.8634 3.4544 -0.54 0.5896schoolno168 1.2154 3.4479 0.35 0.7245schoolno169 0.0965 3.0079 0.03 0.9744schoolno171 -0.7345 3.1678 -0.23 0.8167schoolno173 -6.1143 5.2076 -1.17 0.2404schoolno182 5.9586 2.7650 2.15 0.0312schoolno183 -2.5980 3.9579 -0.66 0.5116schoolno184 4.5057 2.9870 1.51 0.1315schoolno185 -1.0891 3.1678 -0.34 0.7310schoolno186 -4.4783 2.6943 -1.66 0.0966schoolno187 2.7415 2.5546 1.07 0.2833schoolno188 -0.3549 2.9960 -0.12 0.9057schoolno189 -0.2038 2.4876 -0.08 0.9347schoolno190 1.9854 2.6605 0.75 0.4556schoolno191 3.0232 2.6940 1.12 0.2618schoolno192 5.1594 3.9585 1.30 0.1925schoolno194 2.6734 2.8638 0.93 0.3506schoolno199 0.0507 2.6910 0.02 0.9850schoolno200 2.6400 2.9954 0.88 0.3782schoolno201 2.3024 2.9943 0.77 0.4420schoolno205 1.0756 2.8699 0.37 0.7078schoolno207 4.8783 3.4521 1.41 0.1577schoolno213 0.3102 2.5525 0.12 0.9033schoolno214 2.1754 2.5949 0.84 0.4019schoolno219 0.8520 2.5551 0.33 0.7388schoolno220 -3.9159 5.1839 -0.76 0.4501schoolno222 -5.9487 3.1750 -1.87 0.0611schoolno223 -2.8115 2.7843 -1.01 0.3127schoolno228 -0.4804 2.6968 -0.18 0.8586schoolno229 3.2888 3.1914 1.03 0.3028schoolno230 0.4597 2.7739 0.17 0.8684schoolno234 -1.7129 3.4496 -0.50 0.6195schoolno236 1.2898 2.3547 0.55 0.5839
Continued
260 C.2. Chapter 7. Initial Longitudinal Analysis
Estimate Std. Error t-value P -valueschoolno237 2.9109 2.5915 1.12 0.2614schoolno239 5.2102 3.9528 1.32 0.1876schoolno245 6.8807 3.9601 1.74 0.0824schoolno246 -2.9716 2.4639 -1.21 0.2279schoolno247 2.4799 2.5172 0.99 0.3246schoolno248 5.6010 3.9723 1.41 0.1586schoolno249 -0.4386 2.4422 -0.18 0.8575schoolno252 -0.4268 2.4887 -0.17 0.8638schoolno253 0.1520 2.9913 0.05 0.9595schoolno257 3.3020 2.9887 1.10 0.2693schoolno259 3.0165 2.4241 1.24 0.2135schoolno261 -1.9133 2.9898 -0.64 0.5222schoolno262 -0.9195 2.6395 -0.35 0.7276schoolno263 6.3809 5.1852 1.23 0.2186schoolno267 1.2429 2.3954 0.52 0.6039schoolno268 2.0100 2.4337 0.83 0.4089schoolno271 1.3086 2.7859 0.47 0.6386schoolno273 1.0984 2.6370 0.42 0.6770schoolno274 0.0431 5.1819 0.01 0.9934schoolno277 3.7277 2.4450 1.52 0.1274schoolno278 -0.9703 2.8728 -0.34 0.7356schoolno279 2.5167 2.8620 0.88 0.3793schoolno282 2.9430 2.7001 1.09 0.2758schoolno283 -2.8740 2.5883 -1.11 0.2669schoolno287 -1.2622 3.2250 -0.39 0.6955schoolno294 -1.5972 2.6954 -0.59 0.5535schoolno295 0.6431 2.4867 0.26 0.7960schoolno296 0.5594 3.1758 0.18 0.8602schoolno297 2.9898 2.3317 1.28 0.1998schoolno298 1.3354 2.5925 0.52 0.6065schoolno299 3.1302 2.3754 1.32 0.1877schoolno300 -0.6422 2.5183 -0.26 0.7987schoolno304 0.1323 3.1747 0.04 0.9668schoolno305 3.0189 3.4476 0.88 0.3813schoolno308 -1.1533 2.3099 -0.50 0.6176schoolno310 0.0723 3.9662 0.02 0.9855schoolno311 1.3373 2.8626 0.47 0.6404schoolno312 -1.3073 2.6436 -0.49 0.6210schoolno313 -2.5934 2.5188 -1.03 0.3033schoolno315 2.8474 3.5101 0.81 0.4173schoolno316 -2.0683 3.4634 -0.60 0.5504schoolno318 -1.5782 2.9950 -0.53 0.5983
Continued
Appendix C. Output 261
Estimate Std. Error t-value P -valueschoolno319 -0.7724 2.7046 -0.29 0.7752schoolno322 3.4180 2.8717 1.19 0.2340schoolno323 -0.8214 2.7672 -0.30 0.7666schoolno324 6.5210 2.9911 2.18 0.0293schoolno325 -2.3735 2.5545 -0.93 0.3529schoolno326 -0.7416 2.5484 -0.29 0.7711schoolno330 -0.7797 2.6508 -0.29 0.7687schoolno333 1.2772 5.1932 0.25 0.8057schoolno334 0.5099 2.4767 0.21 0.8369schoolno335 1.4205 2.8746 0.49 0.6212schoolno336 -1.0440 2.7784 -0.38 0.7071schoolno337 3.2699 2.3945 1.37 0.1722schoolno338 -0.5275 3.9514 -0.13 0.8938schoolno339 4.4279 3.9513 1.12 0.2625schoolno343 1.0283 2.5917 0.40 0.6916schoolno344 2.6537 2.5873 1.03 0.3051schoolno350 3.6331 2.3257 1.56 0.1184schoolno351 -0.0157 2.4789 -0.01 0.9950schoolno353 1.8000 2.3266 0.77 0.4392schoolno354 3.1660 2.6401 1.20 0.2305schoolno355 -2.5254 2.4740 -1.02 0.3074schoolno356 1.8654 2.5896 0.72 0.4713schoolno357 2.1963 2.5525 0.86 0.3896schoolno358 0.5802 2.3163 0.25 0.8022schoolno359 0.7079 2.2723 0.31 0.7554schoolno361 -0.5219 2.7725 -0.19 0.8507schoolno363 3.7967 5.1777 0.73 0.4634schoolno364 -5.2662 5.2125 -1.01 0.3124schoolno365 -0.8284 2.3531 -0.35 0.7248schoolno366 -0.6231 2.5229 -0.25 0.8049schoolno369 5.5249 2.7754 1.99 0.0466schoolno371 4.1493 2.9882 1.39 0.1651schoolno373 3.3298 2.3808 1.40 0.1620schoolno374 0.9295 2.3059 0.40 0.6869schoolno376 1.3170 2.5555 0.52 0.6063schoolno377 0.9459 2.4059 0.39 0.6942schoolno378 -3.1593 2.7022 -1.17 0.2424schoolno380 -2.0147 2.4383 -0.83 0.4087schoolno381 -4.0718 2.5180 -1.62 0.1060schoolno382 -2.8964 3.1733 -0.91 0.3614schoolno385 2.5849 2.2933 1.13 0.2598schoolno386 -0.6000 2.8133 -0.21 0.8311
Continued
262 C.2. Chapter 7. Initial Longitudinal Analysis
Estimate Std. Error t-value P -valueschoolno387 2.0395 2.4212 0.84 0.3997schoolno388 -1.2362 2.3246 -0.53 0.5949schoolno389 1.3567 2.7101 0.50 0.6167schoolno390 1.0721 2.5202 0.43 0.6706schoolno391 0.1244 2.4700 0.05 0.9598schoolno392 1.3712 2.6414 0.52 0.6037schoolno393 -0.2929 2.3794 -0.12 0.9021schoolno395 -0.3901 2.4434 -0.16 0.8732schoolno396 1.3171 2.6400 0.50 0.6179schoolno399 3.0204 3.4577 0.87 0.3824schoolno400 -0.0092 2.3482 -0.00 0.9969schoolno401 1.7118 2.3497 0.73 0.4663schoolno403 11.1963 5.1872 2.16 0.0310schoolno405 1.8706 2.3773 0.79 0.4314schoolno406 -1.6182 2.9883 -0.54 0.5882schoolno407 2.8655 2.4994 1.15 0.2517schoolno410 1.9125 2.8792 0.66 0.5066schoolno413 4.4833 2.4644 1.82 0.0690schoolno414 -3.0873 2.7708 -1.11 0.2653schoolno415 2.9989 2.6346 1.14 0.2551schoolno416 1.4984 2.4439 0.61 0.5398schoolno417 2.0393 2.5890 0.79 0.4309schoolno418 1.8476 2.3114 0.80 0.4242schoolno419 2.1880 2.3452 0.93 0.3509schoolno420 0.6673 2.4957 0.27 0.7892schoolno421 -1.6063 2.9905 -0.54 0.5912schoolno422 3.9975 3.1802 1.26 0.2088schoolno423 0.4103 2.5295 0.16 0.8711schoolno425 -1.3802 2.4729 -0.56 0.5768schoolno426 -2.5182 2.5257 -1.00 0.3188schoolno427 1.9077 2.6571 0.72 0.4728schoolno428 -1.2431 2.3091 -0.54 0.5904schoolno430 2.0290 2.4648 0.82 0.4105schoolno432 0.5115 2.3942 0.21 0.8309schoolno435 0.6297 2.3330 0.27 0.7873schoolno439 -3.9880 3.0016 -1.33 0.1841schoolno441 3.7547 2.3543 1.59 0.1108schoolno444 -0.6981 2.2630 -0.31 0.7578schoolno445 3.3367 2.8725 1.16 0.2455schoolno446 -1.7607 3.9803 -0.44 0.6583schoolno447 3.8297 2.2866 1.67 0.0941schoolno448 1.3689 3.9607 0.35 0.7297
Continued
Appendix C. Output 263
Estimate Std. Error t-value P -valueschoolno450 0.2587 3.9688 0.07 0.9480schoolno451 2.5859 2.8803 0.90 0.3694schoolno453 0.5193 2.3181 0.22 0.8227schoolno455 -0.9567 2.4078 -0.40 0.6911schoolno456 3.0562 2.9868 1.02 0.3063schoolno457 1.4699 2.3020 0.64 0.5232schoolno458 1.7954 2.5159 0.71 0.4755schoolno459 3.2305 3.9736 0.81 0.4163schoolno460 -1.2273 2.2630 -0.54 0.5876schoolno461 2.7391 2.9901 0.92 0.3597schoolno465 4.5362 3.1771 1.43 0.1534schoolno466 -1.2428 2.4195 -0.51 0.6075schoolno467 6.2157 3.4610 1.80 0.0726schoolno468 2.1862 2.2339 0.98 0.3278schoolno469 -0.7715 2.6948 -0.29 0.7747schoolno470 1.5107 2.5872 0.58 0.5593schoolno471 1.2497 2.4465 0.51 0.6095schoolno472 5.0792 2.3568 2.16 0.0312schoolno473 4.2033 3.9582 1.06 0.2883schoolno474 1.3590 2.4452 0.56 0.5784schoolno475 -1.9910 2.6445 -0.75 0.4516schoolno476 -0.2802 2.9872 -0.09 0.9253schoolno477 -0.5544 2.3725 -0.23 0.8153schoolno478 0.2334 2.4042 0.10 0.9227schoolno479 0.6391 2.3653 0.27 0.7870schoolno480 1.7883 2.3432 0.76 0.4454schoolno482 -3.3103 2.5145 -1.32 0.1881schoolno483 2.0194 2.3899 0.84 0.3982schoolno484 -2.2105 2.2557 -0.98 0.3272schoolno485 0.1771 2.3019 0.08 0.9387schoolno486 2.1944 2.2211 0.99 0.3232schoolno489 2.6977 2.3603 1.14 0.2531schoolno490 1.7925 2.5306 0.71 0.4788schoolno491 2.0123 2.3416 0.86 0.3902schoolno492 2.1482 2.4500 0.88 0.3807schoolno493 3.8626 2.4530 1.57 0.1154schoolno494 0.8286 2.4065 0.34 0.7306schoolno495 1.9613 2.2566 0.87 0.3848schoolno496 1.1156 2.9965 0.37 0.7097schoolno497 2.9363 3.4626 0.85 0.3965schoolno500 1.5351 3.1809 0.48 0.6294schoolno502 1.8110 2.4928 0.73 0.4676
Continued
264 C.2. Chapter 7. Initial Longitudinal Analysis
Estimate Std. Error t-value P -valueschoolno503 1.9001 5.1764 0.37 0.7136schoolno504 0.0095 3.4646 0.00 0.9978schoolno505 2.0652 2.9932 0.69 0.4903schoolno508 2.4774 2.3541 1.05 0.2927schoolno509 0.1011 2.4388 0.04 0.9669schoolno510 2.1893 2.7677 0.79 0.4290schoolno512 0.3353 2.2485 0.15 0.8815schoolno515 1.6483 2.5526 0.65 0.5185schoolno516 -0.3873 2.4647 -0.16 0.8752schoolno517 -0.1081 2.2958 -0.05 0.9625schoolno519 -0.1811 2.3597 -0.08 0.9388schoolno521 0.1620 2.3868 0.07 0.9459schoolno522 1.2776 2.2444 0.57 0.5692schoolno523 0.8915 2.9871 0.30 0.7654schoolno524 1.0831 2.3433 0.46 0.6440schoolno525 3.7363 3.1716 1.18 0.2389schoolno526 3.4716 2.2250 1.56 0.1188schoolno527 2.8924 2.2925 1.26 0.2071schoolno528 1.8903 2.5143 0.75 0.4522schoolno529 1.8843 2.4664 0.76 0.4449schoolno531 1.7228 2.2890 0.75 0.4517schoolno533 2.7973 2.2838 1.22 0.2207schoolno534 -2.7546 3.1829 -0.87 0.3868schoolno536 -1.3990 3.1725 -0.44 0.6592schoolno538 1.7310 2.7643 0.63 0.5312schoolno539 -0.0128 2.3321 -0.01 0.9956schoolno540 4.2372 2.9908 1.42 0.1566schoolno542 1.9247 2.2912 0.84 0.4009schoolno543 2.8559 2.3706 1.20 0.2284schoolno545 4.2953 2.3414 1.83 0.0667schoolno546 3.1228 2.2892 1.36 0.1726schoolno547 3.4903 2.2415 1.56 0.1195schoolno548 1.8169 2.2229 0.82 0.4138schoolno549 2.7925 5.2275 0.53 0.5932schoolno550 1.7735 3.0053 0.59 0.5551schoolno551 0.9170 3.1687 0.29 0.7723schoolno552 1.8953 2.2626 0.84 0.4023schoolno553 1.8434 2.5320 0.73 0.4666schoolno555 -0.6048 2.3312 -0.26 0.7953schoolno556 6.7524 2.2922 2.95 0.0032schoolno557 -0.6380 2.6431 -0.24 0.8093schoolno558 3.9261 2.2595 1.74 0.0824
Continued
Appendix C. Output 265
Estimate Std. Error t-value P -valueschoolno559 1.2927 2.5494 0.51 0.6121schoolno561 -0.7649 2.4159 -0.32 0.7515schoolno564 -1.0407 2.7765 -0.37 0.7078schoolno566 3.9125 2.3059 1.70 0.0898schoolno567 0.9013 2.2534 0.40 0.6892schoolno569 0.7463 2.5151 0.30 0.7667schoolno571 2.0048 2.2692 0.88 0.3770schoolno573 -0.1241 2.4639 -0.05 0.9598schoolno574 2.6848 2.2846 1.18 0.2400schoolno575 -0.8015 3.1700 -0.25 0.8004schoolno576 2.1865 2.2723 0.96 0.3360schoolno578 -1.6135 2.3536 -0.69 0.4930schoolno581 1.1152 2.3111 0.48 0.6294schoolno584 0.1694 2.3860 0.07 0.9434schoolno593 3.4802 2.4229 1.44 0.1510schoolno595 -1.2036 2.5559 -0.47 0.6377schoolno596 2.2955 2.4324 0.94 0.3454schoolno597 -1.3452 3.4554 -0.39 0.6971schoolno599 3.3456 3.4953 0.96 0.3385schoolno600 0.2587 2.5362 0.10 0.9188schoolno608 3.8193 2.5669 1.49 0.1369schoolno614 0.9885 2.8780 0.34 0.7313schoolno624 0.9655 2.6356 0.37 0.7142schoolno639 -0.0909 2.5622 -0.04 0.9717schoolno649 2.0530 2.2254 0.92 0.3563schoolno650 -1.6855 2.2292 -0.76 0.4496proc2001 1.8125 0.2553 7.10 0.0000proc2002 -0.1265 0.2926 -0.43 0.6656atsi1 0.5884 1.6028 0.37 0.7136atsiInconsistent -0.2467 0.9642 -0.26 0.7981lbote1 0.0167 0.6735 0.02 0.9802lboteInconsistent -0.4385 0.3045 -1.44 0.1499genderM 0.3420 0.1600 2.14 0.0326aboriginalY -2.5945 1.4686 -1.77 0.0774disabilityY -1.9197 0.3893 -4.93 0.0000school_carY -0.8841 0.6374 -1.39 0.1655occupation1 0.0024 0.5094 0.00 0.9963occupation2 0.1996 0.4318 0.46 0.6438occupation3 0.0196 0.4223 0.05 0.9630occupation4 -0.2211 0.4194 -0.53 0.5980occupation8 -0.2960 0.4304 -0.69 0.4916school_edu1 0.5570 0.6535 0.85 0.3940
Continued
266 C.2. Chapter 7. Initial Longitudinal Analysis
Estimate Std. Error t-value P -valueschool_edu2 -0.0149 0.5843 -0.03 0.9796school_edu3 0.3174 0.5594 0.57 0.5705school_edu4 0.5122 0.5645 0.91 0.3643non_school5 0.3458 0.4822 0.72 0.4733non_school6 0.4142 0.5253 0.79 0.4304non_school7 0.9254 0.5568 1.66 0.0966non_school8 0.0999 0.4647 0.22 0.8297p_g_genderM -0.2987 0.2373 -1.26 0.2082p_g_nesbY -0.1481 0.4038 -0.37 0.7138home_languN -0.5579 0.4374 -1.28 0.2023
Bibliography
[1] Australian Curriculum Assessment & Reporting Authority (ACARA).
http://www.acara.edu.au/default.asp.
[2] Hirotugu Akaike. Information theory and an extension of the maximum likeli-
hood principle. Second International Symposium on Information Theory, pages
267�281, 1973.
[3] Sivakumar Alagumalai, David D. Curtis, and Njora Hungi, editors. Applied
Rasch Measurement: A Book of Exemplars. Springer, Netherlands, 2005.
[4] Douglas Bates. P-values and all that Lmer. https://stat.ethz.ch/pipermail/r-
help/2006-May/094765.html, 2006.
[5] Stan: A C++ Library for Probability and Sampling. http://mc-stan.org/, 2012.
[6] Sheila M. Bird, David Cox, Vern T. Farewell, Harvey Goldstein, Tim Holt, and
Peter C. Smith. Performance indicators: good, bad, and ugly. Journal of the
Royal, 168(1):1�27, January 2005.
[7] Trever G. Bond and Christine M. Fox. Applying the Rasch Model. Lawrence
Erlbaum Associates Publishers, New Jersey, 2001.
[8] Geo� Brindley. Outcomes-based assessment in practice: some examples and
emerging insights. Language Testing, 18(4):393�407, October 2001.
[9] Vincent Calcagno. Glmulti: Model selection and multimodel inference made
easy, http://cran.r-project.org/package=glmulti, 2012.
267
268 Bibliography
[10] Vincent Calcagno and Claire Mazancourt. Glmulti : An R Package for Easy
Automated Model Selection with ( Generalized ) Linear Models. Journal of
Statistical Software, 34(12), 2010.
[11] George Casella and Roger L. Berger. Statistical Inference. Duxbury Press,
2002.
[12] Karl Bang Christensen. Latent Covariates in Generalized Linear Models: A
Rasch Model Approach. In Advances in Statistical Methods for the Health Sci-
ences2, pages 95�108. Birkhauser Boston, 2007.
[13] Trevor Cobbold. Big Increase in Students Withdrawn from NA-
PLAN Tests, http://www.saveourschools.com.au/league-tables/big-increase-
in-students-withdrawn-from-naplan-tests, 2012.
[14] Trevor Cobbold. Many Schools Have High Withdrawal Rates from NA-
PLAN, http://www.saveourschools.com.au/league-tables/many-schools-have-
high-withdrawal-rates-from-naplan, 2012.
[15] Bert P. M. Creemers and Gerry J. Reezigt. School E�ectiveness and School
Improvement: Sustaining Links. School E�ectiveness and School Improvement,
8(4):396�429, 1997.
[16] Igusti Darmawan and John P Keeves. Accountability of teachers and schools:
A value-added approach. International Education Journal, 7(2):174�188, 2006.
[17] Andrew Gelman, John B. Carlin, Hal S. Stern, and Donald B. Rubin. Bayesian
Data Analysis. Chapman and Hall/ CRC Texts in Statistical Science, second
edition, 2004.
[18] Andrew Gelman and Jennifer Hill. Data Analysis Using Regression and Multi-
level/Hierarchical Models. Cambridge University Press, 2006.
[19] Andrew Gelman and Donald B. Rubin. Inference from iterative simulation
using multiple sequences. Statistical science, 7(4):457�472, 1992.
[20] Harvey Goldstein. League tables and their limitations: statistical issues in com-
parisons of institutional performance. Journal of the Royal Statistical Society.
Series A (Statistics in Society), 159(3):385�443, 1996.
Bibliography 269
[21] Harvey Goldstein. Using examination results as indicators of school and college
performance. Journal of the Royal Statistical Society. Series A, 159(1):149�163,
1996.
[22] Harvey Goldstein. Methods in school e�ectiveness research. School e�ectiveness
and school improvement, 8(4):369�395, 1997.
[23] Harvey Goldstein and George Leckie. School league tables: what can they really
tell us? Signi�cance, 5(2):67�69, June 2008.
[24] David A. Harville. Extension of the Gauss-Markov Theorem to Include the
Estimation of Random E�ects. The Annals of Statistics, 4(2):384�395, 1976.
[25] David A. Harville. Maximum Likelihood Approaches to Variance Component
Estimation and to Related Problems. Journal of the Americal Statistical Asso-
ciation, 72(358):320�338, 1977.
[26] Matthew D. Ho�man and Andrew Gelman. The No-U-Turn Sampler : Adap-
tively Setting Path Lengths in Hamiltonian Monte Carlo (pre-print). (2011):1�
30.
[27] Njora Hungi. Measuring School E�ects Across Grades. Shannon Research Press,
South Australia, 2003.
[28] Ben Jenson. Measuring What Matters: Student Progress. Grattan Institute,
January 2010.
[29] James G. Ladwig. What NAPLAN Doesn't Address (But Could and Should).
Professional Voice : The NAPLAN Debate, 8(1):35�40, May 2010.
[30] Nan M. Laird and James H. Ware. Random-E�ects Models for Longitudinal
Data. Biometrics, 38(4):963�974, December 1982.
[31] Christopher Moore. Linear mixed-e�ects regression P-
values in R: A likelihood ratio test Function.
http://blog.lib.umn.edu/moor0554/canoemoore/2010/09/lmer_p-
values_lrt.html, 2010.
270 Bibliography
[32] Douglas Bates. lme vs Lmer. https://stat.ethz.ch/pipermail/r-sig-mixed-
models/2009q3/002912.html, 2009.
[33] Geo� N. Masters. A Rasch Model for Partial Credit Scoring. Psychometrika,
47(2):149�174, February 1982.
[34] Robert H. Meyer. Value-Added Indicators of School Performance: A Primer.
Economics of Education Review, 16(3):283�301, 1997.
[35] National Assessment Program (NAP). http://www.nap.edu.au/.
[36] OpenBUGS. http://www.openbugs.info/w/.
[37] Jose C. Pinheiro and Douglas M. Bates. Mixed-E�ects Models in S and S-plus.
Springer, 2000.
[38] Stephen W. Raudenbush. What are value-added models estimating and what
does this imply for statistical practice? Journal of Educational and Behavioral
Statistics, 29(1):121 �129, 2004.
[39] Stephen W. Raudenbush and J. Douglas Willms. The Estimation of School
E�ects. Journal of Educational and Behavioral Statistics, 20(4):307�335, 1995.
[40] Brian D. Ripley. Model Choice. Technical report.
[41] My School. http://www.myschool.edu.au/.
[42] James Sick. Rasch Analysis Software Programs,
http://jalt.org/test/PDF/Sick4.pdf. Shiken: JALT Testing & Evaluation
SIG Newsletter, 13(November):13�16, 2009.
[43] Tom Snijders. Analysis of longitudinal data using the hierarchical linear model.
Quality and Quantity, 30(4):405�426, November 1996.
[44] John D. Storey. A direct approach to false discovery rates. Journal of the Royal
Statistical Society: Series B (Statistical Methodology), 64(3):479�498, August
2002.
[45] R Core Team. R: A Language and Environment for Statistical Computing,
2012.
Bibliography 271
[46] William Venables and Brian Ripley. Modern Applied Statistics with S. Springer,
2002.
[47] J. Douglas Willms and Stephen W. Raudenbush. A Longitudinal Hierarchical
Linear Model for Estimating School E�ects and Their Stability. Journal of
Educational Measurement, 26(3):209�232, 1989.
[48] M. Wu and R. Adams. Applying the Rasch Model to Psycho-social Measure-
ment: A practical approach. Educational Measurement Solutions, Melbourne,
2007.