-
41
Odyssey 2014:The Speaker and Language Recognition Workshop16-19
June 2014, Joensuu, Finland
Exploring some limits of Gaussian PLDA modelingfor i-vector
distributions
Pierre-Michel Bousquet, Jean-François Bonastre, Driss
Matrouf
University of Avignon - LIA, France{pierre-michel.bousquet,
driss.matrouf, jean-francois.bonastre}@univ-avignon.fr
AbstractGaussian-PLDA (G-PLDA) modeling for i-vector
basedspeaker verification has proven to be competitive versus
heavy-tailed PLDA (HT-PLDA) based on Student’s t-distribution,when
the latter is much more computationally expensive. How-ever, its
results are achieved using a length-normalization,which projects
i-vectors on the non-linear and finite surfaceof a hypersphere.
This paper investigates the limits of linearand Gaussian G-PLDA
modeling when distribution of data isspherical. In particular,
assumptions of homoscedasticity arequestionable: the model assumes
that the within-speaker vari-ability can be estimated by a unique
and linear parameter. Anon-probabilistic approach is proposed,
competitive with state-of-the-art, which reveals some limits of the
Gaussian modelingin terms of goodness of fit. We carry out an
analysis of residue,which finds out a relation between the
dispersion of a speaker-class and its location and, thus, shows
that homoscedasticityassumptions are not fulfilled.
1. IntroductionIntroduced in [1], the i-vector representation of
speech utter-ances provides a feature vector of low dimension (less
than600), independent of the length of the utterance. A
speakerverification system using these features and a simple
classifieroutperforms the previous approaches, like Joint Factor
Anal-ysis (JFA) [2, 3]. The Bayesian generative model designedto
provide a consistent probabilistic framework for i-vectors isthe
PLDA, introduced in [4] for face recognition and adaptedfor speaker
verification in [5, 6]. The first approach, GaussianPLDA (G-PLDA),
assumed that speaker and residual compo-nents have Gaussian
distributions. To deal with severe within-class distortions and
increase the robustness to outliers, a spe-cific PLDA approach for
modeling i-vector distributions wasintroduced in [5], based on
Student’s t-distribution and referredto as heavy-tailed PLDA.
Student’s t-distribution has heaviertails compared to the
exponentially-decaying tails of a Gaus-sian, providing a better
representation of the speaker and resid-ual subspaces, including
the outliers [7].
More recently, a pre-conditioning before any i-vector Gaus-sian
modeling has been introduced [8, 9]. I-vectors are whitenedand
length-normalized, projecting data onto a spherical surface.It is
shown [8] that this technique improves Gaussianity of thei-vectors.
Also, it involves a number of properties related to in-tersession
compensation [9, 10] and contributes strongly to themodel
efficiency [11]. Using these normalized i-vectors, per-formance of
the Gaussian and Heavy-Tailed PLDA models arecomparable, the former
being much faster both in training andin testing.
The goal of this paper is to study and reveal some limits ofthe
G-PLDA modeling when applied in the i-vector field.
Afterlength-normalization (LN), data are closer to Gaussianity
as-sumptions but they lie on the non-linear and finite surface ofa
hypersphere. This point caught our attention. We considerthat this
discrepancy between a non-linear data-manifold and alinear
framework could reveal some limits of G-PLDA for mod-eling
i-vectors distributions. In particular, homoscedasticity ofthe
within-class variability is questionable, when data distribu-tion
is spherical: PLDA assumes that it exists a unique
(thusspeaker-independent) and linear parameter of within-class
vari-ability. We consider that spherical distributions cannot
fulfillsuch an assumption.
First, we propose a new approach for estimating the
PLDAmetaparameters. Based on the pre-conditioning procedure and
adeterministic estimate of parameters, this non-probabilistic
ap-proach helps to assess ability of a maximum likelihood
(ML)-based approach to improve the goodness of fit, and reveals
somelack of compliance of i-vector distributions with G-PLDA
as-sumptions.
Second, we analyze homoscedasticity of G-PLDA model-ing after
LN. Any significant relation between the within-classvariability of
a speaker and another class parameter would vio-late the assumption
of homoscedasticity. We compute posteriorlikelihoods of speaker and
residual factors for each speaker ofour development corpus and
confirm the concern about linearityand homoscedasticity
assumptions.
2. Gaussian-PLDA2.1. Modeling
Introduced in [4], Gaussian Probabilistic Linear
DiscriminantAnalysis (PLDA) is a generative i-vector model. The
most com-mon PLDA model in speaker verification assumes that each
p-dimensional i-vector w of a speaker s can be decomposed as
w = µ+ Φys + ε (1)
The mean vector µ is a global offset, Φ is a p× r matrix
whosecolumns provide a basis for the eigenvoice subspace, the
r-dimensional vector ys is the speaker factor and � is the
residualterm. Therefore, the speaker-specific part µ + Φys
representsthe between-speaker variability and is assumed to be tied
acrossall utterances of the same speaker. G-PLDA assumes that
alllatent variables are statistically independent. Standard
normalprior is assumed for the speaker factor ys and normal prior
forthe residual term � with mean 0 and full covariance matrix Λ.The
maximum of likelihood (ML) point estimates of the modelparameters
are obtained from a large collection of development
-
42
data using an expectation-maximization (EM) algorithm as
in[4].
Note that this approach is the simplified version of the
orig-inal PLDA model: eigenchannels have been removed, as pro-posed
in [5], since i-vectors are of sufficiently low dimension(400 to
600 usually) and since PLDA modeling does not showmajor improvement
with eigenchannels.
2.2. Verification score
The speaker verification score, given the two i-vectors w1 andw2
involved in a trial, is the likelihood-ratio
score = logP (w1,w2|θtar)P (w1,w2|θnon)
(2)
where the hypothesis θtar states that inputs w1 and w2 are
fromthe same speaker and the hypothesis θnon states they are
fromdifferent speakers. Likelihoods of (2) can be decomposed as
P (w1,w2|θtar) =∫y
∏i=1,2
P (wi|µ+ Φy,Λ)P (y|0, I) dy
P (w1,w2|θnon) =∏
i=1,2
∫y
P (wi|µ+ Φy,Λ)P (y|0, I) dy
(3)
For the G-PLDA case, all the marginal likelihoods are
Gaussianand the score (2) can be evaluated analytically [12]
score (w1,w2) = logN([
w1w2
];
[µµ
],
[Σtot ΣacΣac Σtot
])− logN
([w1w2
];
[µµ
],
[Σtot 0
0 Σtot
])(4)
whereN () denotes normal density function and{Σtot = ΦΦ
t+Λ
Σac = ΦΦt (5)
The matrix Σtot consists of a probabilistic total covariance
ma-trix, adding variabilities of the decomposition to take into
ac-count underlying assumptions of the probabilistic model.
Thematrix Σac is the covariance matrix of speaker-dependent
factor(across-class). It is worth noting that computing this score
withtest vectors depends solely on the knowledge of metaparame-ters
Σtot and Σac (speaker and residual factors of i-vectors oftest have
not to be computed).
3. ConditioningA pre-processing step applied before any i-vector
modeling hasbeen introduced in [8, 9], following WCC Normalization
andcosine-scoring technique of [2]. I-vectors are whitened
andlength-normalized, in order to make them more Gaussian. Themost
commonly used whitening technique is a standardizationand the
transformation applied to an i-vector w can be summa-rized as
follows:
w← A− 1
2 (w − µ)∥∥∥A− 12 (w − µ)∥∥∥ (6)First, data are standardized
according to the mean µ and a vari-ability matrix A of a training
corpus. Then they are length-normalized, confining the i-vectors to
the hypersphere of unit
radius. Parameters are computed for the i-vectors present in
thetraining corpus and applied to test i-vectors. The matrix A
canbe the total covariance matrix Σ or, as proposed in [2, 10],
thewithin-class covariance matrix. We denote in this article LΣ,LW
these transformations (Length-normalization of standard-ized
vectors according to Σ or W).
In [8], it is shown that this technique improves Gaussian-ity of
i-vectors. It reduces the gap between the underlying as-sumptions
on the data distribution and the real distribution andalso reduces
the dataset shift between development and trial i-vectors [13].
Performance of a G-PLDA system with this pre-conditioning is
competitive versus the HT-PLDA, when the lat-ter shows a
significant higher complexity. It is shown in [11]that this
procedure mainly contributes to model efficiency. Asproposed in
[9], its two steps can be iterated. As a result, i-vectors tend to
be simultaneously A-standardized and length-normalized (magnitude
1), involving a number of properties re-lated to intersession
compensation. Some of them are detailedin [9, 10].
4. Proposed approach4.1. Deterministic estimation of G-PLDA
parameters
G-PLDA scoring of (4) is based solely on the determination
ofmetaparameters Σtot and Σac. Factors ys and ε have not to
becomputed. Given a training corpus T comprised of i-vectors ofS
speakers, we denote by Σ the total covariance matrix of Tand B the
between-class covariance matrix of T defined by
B =
S∑s=1
nsn
(ws − µ) (ws − µ)t (7)
where ns is the number of utterances for speaker s, n is the
totalnumber of utterances of T , ws is the mean i-vector of s and
µrepresents the overall mean of T . Let W denote the within-class
covariance matrix of T , equal to Σ−B.Singular value decomposition
of B provides a matrix P whosecolumns are the eigenvectors of B
sorted by decreasing orderof eigenvalues and a corresponding
eigenvalue diagonal matrix∆, such that B = P∆Pt. Given a rank r
< p, the r-rangeprincipal between-class variability can be
summarized into thep× p covariance matrix B1:r defined by
B1:r = P1:r∆1:rP1:r (8)
where P1:r denotes the p × r matrix comprised of the first
rcolumns of P and ∆1:r the r × r top-left block of ∆.
To estimate G-PLDA metaparameters Φ and Λ, we proposethe
following procedure:
• Apply LW-Conditioning of Equation (6), eventually
it-erated.
• Replace EM-ML based estimates of speaker and
residualparameters by the following direct expressions:{
Σtot = Σ
Σac = B1:r(9)
4.2. Justification of the approach and conformity to G-PLDA
assumptions
Ignoring the probabilistic constraints of G-PLDA, we presumethat
the most relevant metaparameters Σtot and Σac (by onlyconsidering
information of the training set and assuming that
-
43
exists a r-range eigenvoices subspace) are the total
covariancematrix Σ and the part B1:r of within-speaker variability
due tothe principal axes of B. In order to assess conformity of
thisnon-probabilistic approach to G-PLDA assumptions (standard-ity
of ys, statistical independence between ys and ε, Gaussian-ity of
factors), its factors have first to be expressed. As matrixP of (8)
is orthogonal, we use the equality
I = PPt = P1:rPt1:r + P(r+1):pP
t(r+1):p (10)
where I is the p × p identity matrix and P(r+1):p is the p ×(p−
r) matrix comprised of the last (p− r) columns of P, todecompose an
i-vector w of a speaker s as follows:
w = µ+[P1:rP
t1:r (ws − µ)
]+[P(r+1):pP
t(r+1):p (ws − µ) + w −ws
](11)
where ws is the mean i-vector of speaker s. Factor and
matricialparameters can be expressed as
ys = ∆− 1
21:r P
t1:r (ws − µ)
Φ = P1:r∆121:r
ε = P(r+1):pPt(r+1):p (ws − µ) + w −ws
(12)
A straightforward computation shows that the covariance ma-trix
of µ + Φys is equal to B1:r , as desired. Note that com-puting
factors ys and ε requires the knowledge of the speakermean-vector
ws, which is unknown for test vectors. G-PLDAassumes that the
speaker factor ys is standardized and that alllatent variables are
statistically independent. Considering solelydata from the
development corpus, a straightforward computa-tion shows that ys,
as defined in (12), has a mean equal to 0and a covariance matrix
equal to the identity matrix I. More-over, to obtain covariance
matrices of (5) from (3), only nullityof the covariance between ys
and ε (a necessary condition ofindependence) is required. As ys and
ε are centered, this valueis equal to
E[ysε
t] = ∆− 121:r Pt1:rBP(r+1):pPt(r+1):p+ E
[∆− 1
21:r P
t1:r (ws − µ) (w −ws)t
](13)
As Pt1:rBP(r+1):p = 0, the first term is null. The second
term
is equal to ∆− 1
21:r P
t1:rEs [(ws − µ) Ew∈s [w −ws]] = 0,
since Ew∈s [w −ws] = 0.Normal priors are assumed for speaker and
session fac-
tors. Gaussian shape of factors can be estimated by analyz-ing
square-norms of their standardized versions [8]. Indeed,
thesquare-length of vectors drawn from a standard Gaussian
dis-tribution follows a χ2 distribution with number of degrees
offreedom equal to the dimension of the vector. Figure 1
presentshistograms of square-norm distributions of standardized
fac-tors ys (left panel) and ε (right panel). As the model
assumesys ∼ N (0, I) and ε ∼ N (0,Λ) , the standardized versionsof
ys and ε are assumed to be ys and Λ−
12 ε. Thus, Figure 1
displays ‖ys‖2 (left panel) and∥∥∥Λ− 12 ε∥∥∥2 (right panel) for
the
development set (thick line) and the evaluation set NIST
SRE10male telephone (thin line). Also, Figure 1 depicts the pdfs
ofχ2 distributions with r and p degrees of freedom (dashed
line).Three systems are considered:
no L-norm
LW EM-ML
LW deterministic
ys ε
ys ε
ys ε
0 50 100 150 200
0.00
00.
010
0.02
00.
030
devevalchi2
0 50 100 150 200
0.00
00.
010
0.02
00.
030
devevalchi2
0 50 100 150 200
0.00
00.
010
0.02
00.
030
devevalchi2
400 500 600 700 800
0.00
00.
004
0.00
8
devevalchi2
400 500 600 700 800
0.00
00.
004
0.00
8
devevalchi2
400 500 600 700 800
0.00
00.
004
0.00
8
devevalchi2
Figure 1: Histograms of the square norm distributions of G-PLDA
factors. Graphs also depict the pdfs of χ2 distributionswith r
degrees of freedom for ys and p for ε.
1. No LN procedure has been carried out before G-PLDAmodeling.
Development and evaluation i-vectors arecentered only, by
subtracting the mean vector of the de-velopment set,
2. LW-normalization (2 iterations) followed by EM-MLbased G-PLDA
modeling.
3. LW-normalization (2 iterations) followed by the pro-posed
deterministic approach.
Three observations could be gathered from this analysis.
First,G-PLDA factors extracted from initial i-vectors have a
non-symmetric and heavy-tailed distribution. Moreover, a
severedataset shift between development and evaluation i-vectors
oc-curs. Second, after LN, distributions better match the χ2
distri-butions. In particular, non-Gaussian behavior and dataset
shiftof the residual factor are significantly reduced. Lastly,
his-tograms of the two post-LN approaches are similar, for
bothfactors. This last result shows that the deterministic
estimateimproves the Gaussian shape of factors and reduces the
mis-match between development and evaluation datasets in a
similarmanner to ML technique.
5. Experimental setupEvaluation was performed on NIST SRE08 core
conditions 6and 7 male only, corresponding to telephone-telephone
(re-spectively All and English-only) enrollment-verification
tri-als, SRE10 extended condition 5 male only, corresponding
totelephone-telephone enrollment-verification trials and SRE12core
conditions 4 and 5 male only, corresponding to telephone
-
44
Table 1: Comparison of performance between deterministic and
EM-ML approaches, without or with pre-conditioning.Conditioning No
length-norm. LΣ LWParameter estimation EM-ML deterministic EM-ML
EM-ML deterministicNIST-SRE male evaluation minDCF EER(%) minDCF
EER(%) minDCF EER(%) minDCF EER(%) minDCF EER(%)2008 tel. all (det
6) 0.317 6.52 0.342 7.44 0.277 5.24 0.279 4.92 0.289 4.922008 tel.
eng. (det 7) 0.180 3.19 0.176 4.10 0.121 1.82 0.116 1.59 0.128
1.592010 tel. extended (det 5 ext) 0.599 5.97 0.601 6.76 0.496 2.40
0.483 2.28 0.487 2.282012 tel. with added noise (det 4) 0.463 5.37
0.463 5.30 0.438 3.53 0.412 3.21 0.452 3.212012 tel. noisy
environment (det 5) 0.673 7.27 0.639 19.88 0.419 2.48 0.394 2.24
0.433 2.40
with added noise and noisy environment. These last two
con-ditions enable to evaluate the proposed approach on noisy
ut-terances. EM-ML-based and deterministic estimations of G-PLDA
metaparameters have been evaluated with LIA con-figuration : the
feature extraction, 512-components GMM-UBM functionalities and
i-vector extraction configurations forNIST SRE08, SRE10 evaluations
are described in [11]. ForSRE12 evaluation, a gender-dependent LIA
i-vector extractorwas trained on data from NIST SRE04,05,06,08,10,
Switch-board II Phases 2 and 3, Switchboard Cellular Parts 1 and 2,
giv-ing 1879 speakers in 25024 segments of speech. PLDA modelwas
trained by merging two datasets: first, i-vectors of the
samedataset than for extraction, second, i-vectors of target
speakersSRE12 and their noisy versions. For each clean segment of
atarget speaker, two noisy versions (6dB and 15dB) are gener-ated,
following the method suggested in [14]. Channel factorε of G-PLDA
is kept full and speaker factor is fixed to its op-timal value in
terms of performance. For SRE12 multi-cut en-rollment, scoring is
performed by averaging all the enrollmenti-vectors of each target
speaker after LN. The size of i-vectorsis 400.
6. ResultsTable 1 summarizes results of the tests performed on
the NISTSRE conditions described above, in terms of Equal Error
Rateand normalized minimum Detection Cost Function as definedby
NIST for SRE08 and SRE10 evaluations (DCF 2010 is ap-plied for
SRE12). Evaluations have been carried without or withLN, and with
EM-ML or deterministic approach. Also, resultsof LΣ-conditioning
(standardization according to the total co-variance matrix followed
by LN) are reported in columns 5,6 ofTable 1, since this
transformation is the most commonly imple-mented. Comparison of
systems without or with conditioningrecalls the necessity of LN to
handle i-vectors in a Gaussianframework: it yields 49% and 24%
relative improvements inaverage EER and minDCF. Comparison of
systems with LΣ orLW conditioning recalls the relevance of the
latter, remarkedin [10]. LW-conditioning leads to 8.5% and 3.6%
relative im-provements in average EER and minDCF.
Without LN, the deterministic approach degrades perfor-mance.
Accuracy degradation can even be considerable (seeEER of last
condition, SRE12 noisy environment). EM-MLapproach brings the
expected improvement in performance ina probabilistic context.
However, after LW conditioning, de-terministic approach yields
similar performance across the re-ported conditions, relative to
EM-ML, in terms of EER asminDCF (with a slight gap in terms on
minDCF, except for lastconditions in noisy environment). It even
outperforms, in termsof EER, the commonly used LΣ-based second
system.
To better assess independence of these results to the up-stream
configuration, we carried out the same NIST SRE10evaluation
condition with i-vectors we already used in [10],
provided by BUT laboratory [15]. EERs and minDCFsof EM-ML vs
deterministic approach are respectively equalto (1.04%, 0.31) vs
(1.04%, 0.30) for male set, and to(1.78%, 0.33) vs (1.75%, 0.33)
for female set, confirming theprevious observations.
Results of the deterministic approach recall the relevanceof LW
conditioning. After this procedure, i-vectors fit betterthe
Gaussian model. But they also reveal some limits of G-PLDA
statistical modeling, in terms of goodness of fit. Thenext section
addresses this issue.
7. Limits of Gaussian linear modeling fori-vectors
While Gaussian PLDA applied to whitened and length-normalized
i-vectors has shown its efficiency, previous out-comes raise an
issue which relates to the relevance of a lin-ear Gaussian approach
with i-vectors. Model training con-sists of fitting a parametric
model to the training set. TheLW-conditioning and deterministic
estimation are built solelyon covariance parameters of this latter,
ignoring the aims ofGaussianity and generalization to new
observations, in partic-ular from unknown speakers. Moreover, the
eigenvoice ba-sis is orthogonal, which can limit the accuracy of
the esti-mate. By increasing metaparameter likelihoods, EM-ML
al-gorithm should enhance the Gaussian modeling. Previous re-sults
show that EM-ML based approach does not bring the ex-pected
improvement of performance. After LN, data are closerto
probabilistic assumptions but they lie on the non-linear andfinite
surface of a hypersphere. We consider that this inconsis-tency
between a non-linear data-manifold and a linear Gaussianframework
could reveal some limits of G-PLDA modeling fori-vectors. In
particular, the assumption that there exists a lin-ear and
speaker-independent metaparameter Λ of within-classvariability is
questionable.
7.1. Heteroscedasticity of residue
G-PLDA model assumes homoscedasticity of speaker-classes(i.e.,
that their distributions share the common covariance ma-trix Λ). It
is noticed in [16] that two Gaussian distributionslaying on a
p-sphere can only be homoscedastic if the meanfeature vector of one
class is the same as that of the others upto a sign 1. Equality of
covariance, when model is homoscedas-tic, assumes that the modeling
errors (the residue between theactual variability of a class s and
the metaparameter Λ) are un-correlated, normally distributed and
that their variances do notvary with the effects being modeled. Any
significant relationbetween this residue and another class
parameter would violatethe assumption of random prior. Such a
relation could penalizethe goodness of fit of G-PLDA modeling.
1except when the common covariance matrix is the identity
matrixor a multiple of the identity matrix.
-
45
llk(ys|M) llk(ys|M) llk(ys|M) llk(ys|M)
llk(ε
s|M)
llk(ε
s|M)
llk(ε
s|M)
llk(ε
s|M)
no L-norm LΣ LW EM-ML LW deterministic
l
l
l
ll
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
ll
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
l
ll
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
l
l
l
l
l
ll
l
l l
l
l
ll
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l l
l
l
l
l
l
l
l
l
ll
l
l
l
ll
l
l l
l
l
l
l
l
l
l
l
ll
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
l
l
l
ll
l
l
l
l
l
l
l
l
l
l
ll ll l
lll
l
l l
l
l
l
l
l
lll
l
l
l
l
lll
l
l
l
l
l
l
l
l
l
l
l
l
l l
l
ll
l
l
ll
l
l
l
ll
l
l
l
l
l
l
l
lll
l
l
l
l
l
l
l
ll
l
l
l
ll
l
l
l
l
l
l
ll
l
l
l l
l
ll
l
l
l
l
l
ll
l
ll l
l
ll
l
ll l
ll
l
l
l
l
l
l l
l
ll
l
l
ll
l l
l
ll
l
l
l
l
l
l
ll
l
l l
l
l
l
ll
l
l
l
l
l
l
l
l
ll
l
l
l
ll l
l
l
l
l
ll
l
l
l
l
l
l
l l
ll
l
ll
ll
l
l
l
l
ll
ll
l
l
l
l
l l
l
ll
l
l
l
ll
l
l
l
l
l
l
l
l ll
l
l
l
l
l
l
l
ll
l
l
l
l
ll
lll
ll
l
ll
l
ll
l
l
l
l
l
ll
l
l
l
l
l
l
ll
ll
ll
ll
l
l
l
l
l
ll
l
l
l
l
l
l
l
l
l
ll
l l
l
l
l
l
l
l
l
l
l
l
ll
ll
l
l
l
l
l
ll
l
ll l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
ll
l
ll
l
l
l
ll
l
l l
l
l
l
l
ll
l
l
l
l
l
l
l
l
l l
ll
l l
ll
l
ll
l ll
l l
l
l
l
l
l
l
ll
ll
ll
llll
ll
l
ll
l
l
ll
l
l
ll
l
ll
l
l
l
ll
l
l
l
l
lll
lll l
l
ll
l
l
l l
l
l
l
ll
l
ll
l
l
ll l
l
ll
l
l
ll
l
l
l
l
l
l
l
ll
lll
l
l
l
lll
l
l
l
l
l
l
l
l
l
l
ll
l
ll
ll
l ll
l
l l
ll
l
l
l
ll
l
l
l
l
l
l
l
l
l
l
l
l
ll
ll
l
l
l
l
l
l
l
ll
l
l
lll
l
l l
l
l
l
l
l
l
ll
ll
l
ll
ll
ll l
l
l
ll ll
l
ll
ll
l
l
ll l
l
l
ll
ll
l
l
l
l
l
l
l
l
ll
l l
l
l
l
ll
l
l
l
ll
l
ll
l
l
l
ll l
l
l
ll
l
l
l
ll
ll
l
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
l
l
l
lll l
l
l
l
l
l
l
l
l
l
l
l
l ll
l
ll
l
ll
l
l
l
l
l
l ll ll
ll
l
l
l
l
ll
ll
l
l
l
l
l
l
l
ll
l ll
l
l ll
l
l
l
l
l
l
l
ll
l
l
l
l
l
lll
l
l
l
ll
ll
l
l
l
l
ll
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
l
l
ll
l
l
l l
ll
l
lll
l
ll
lll
l
l
ll
l
l
l
l
l
l
l
l l
l
l
ll l
l
ll
l
l
l
ll
l
l
l
ll
l
l l
l
l
l
l
l
l
lll
l
l
l
l
l l
l
l
l
ll
l
lll
l
l
l
l
ll
l
l
ll
ll
l
ll
l l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l ll
ll
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l ll l
l
l
l
l
lll
ll
l
l
l
ll
l
llll
l
l
l
l
l
l
l
l
l
l
l
l
ll
l
ll
l
ll
l
lll
ll
l
lll
l
l
l
l
ll
l
l
ll
l
ll
l
l
l
l
l
l
l
l ll
l
l
l
ll
l
l
l
l
l
l
l
l
l
ll
l
ll
l
l
l
l
l
ll
l
l
l
ll
ll
l
l
ll
l
l
ll
l
ll
l
ll
l
ll
l
ll
l
ll
l
l
l
l
lll
l
l
l
l
ll l
l
ll
l
ll
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
lll
ll
ll
l
l
l
l
l
l
l
l
l
ll
l
ll
l
l
l
l
l
l
l
l
l
l
l
l
l ll l
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
l
ll
lll
ll
l
l
l
ll
ll
l
l
l
l
l
l
l
l
ll
ll
ll
l
l
ll
l
ll
l
l
l
ll
l
l
l
l
l
l
l l
l
l
ll
l
l
l
l
ll
l
-350 -300 -250 -200 -150 -100
-110
0-9
00-7
00
R² = 0.04
l
l
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
l
l
ll l
l
l
ll
l
l
l
ll
l
ll
l l
l
l
l
ll
l
l
l
l
ll
l
l
ll
ll
l
l
l
ll l
l
ll lll
l
l
l
l
l
l
l
l
ll
l
l
l
l
l
ll
l l
l
l l
l
l
l
l
l
l
l
ll
ll
ll
l ll
l
ll
l
l
l
ll
l
l
l
l
l
l
ll
l
l
l
l
l
lll
l
l
ll
ll
l
l
l
l
ll
l
l
ll
l
l
l
ll
l
ll
l
l
l
ll
ll l l
l
l
l
l
l
ll
l
l
l
l
l
l
l
ll
ll
l l
ll
l
ll
l
ll
l
l
ll
l
l
ll
l
l
lll
l
l
l
l
l
ll
l
ll
l
l
l
l
l
ll
l
l
l
l
l
ll
l
ll
l
l
l
l
l
l
ll
l
l
ll
ll
l
l
l
l
l
l
l
ll
l
l
ll
l
ll
ll
l
l
l
l
l
l
l
l
ll l
ll
l
ll
lll
l
l
l
ll l
l
l
l
l
l
l
l
l
l
l
ll
l
l
l
l
l
l
l
l
l
l
l
l
l l
l
l l
l
l
l
l
ll
l
ll
l
l ll
l
l
l
l
l
l
ll
l
l
l
l
l
l
ll
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
l l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
ll
l l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
ll l
l l
l
l
l
l
l
l
l
l
lll
l
ll
l
ll
l
l
l
ll
l
ll
l
ll
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
ll
l
ll
l
l
l
l
l
ll
l
ll
l
l
l
ll
l
ll
l
ll
l
ll
l
l
l
l
l
l
l
l
l
l
l
l
l
l ll
l
l
ll
l
ll
l
l
l l
l
l
l
l
l
l
l
l
l
lll
l
l
ll l
ll
l
l
l
l
l
l
l
ll
ll
l
l
l
l l
l
ll
l
l
l
l
l
ll
l
l
l
l
l
l l
l
l
l
l
l
l
l
ll
lll
l
l
l
l
l
l
l
l
l
l
ll
l
l
l
ll
l
l
l
l
l
ll
l
lll
l
l
ll
l
l l
l
l
l ll
ll
l l
l
l
l
l
l
l
l l
l
l
l
l
l
l
l
l
l
l
ll
l
l
l
ll
ll
l
l
l
l
l
l
lll
l
l
l
lll
l
l
l
l
l
ll
l
l
l
l
l
l
l
ll
lll
llll
l
l
l
ll
l
ll
l
ll
l
ll
l
lll
l
l
l
l
ll
l
lll
l
l l
ll
ll
l
l l
lll
l
l
l l
ll l l
l
ll
l
l
l
ll
l
l
ll
ll
l
l
l
l
l
l l l
l
l
l
l
ll
l
l
l
ll l
l
l
ll
l
l
l
ll
lll
l
ll
l
lll
l
l
l
l
l
l
l
lll l
l
l l
l
l
ll
l
ll
l
l
l
ll
ll l
l
l
l
ll l
l
l
l l
ll
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
l
l
l
l
l l
l
l
ll
l
ll lll
l
l
l
ll
l l
l
l l
l l
l
l
l l
l
ll
ll
l
l lll
l
ll
l
ll
l
ll
lll
l
ll
l
l
ll
ll l
ll
l
l
l
l
l
l
l
l
ll
l
l
l
l
ll
lll
l l
l
l
l
ll
l
l
l
l
l l
l
l
ll
l
ll
l
l
l
ll
ll l
l l
ll
l
l
l
l
l
l
l
l
l l
l
l
ll
l
l
ll
l
l
ll
l
ll
ll
l
l
ll
l
l
l
l
l
l
ll
l
l
l
l
ll
ll l
l
ll
l
l
l
ll
l
l
l
l
lll
l
l
l
l
l
l
ll
l
ll
lll
l
l ll
ll l
l
l
l
l
l
ll
ll
ll
lll
l
l
l
l
l
l
l
l
ll
l
l ll l
l
l
ll
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
ll
l
l
l
l
l
l
ll
ll
lll
l
l
l
l
l
lll
l
l
l l
l
l
l
l
l
l
l
l l
ll
l
l ll
l
ll
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
l
l
l
l
l
l
l
l
l l
l
l
l
l ll
l
l
ll
l
l
l
ll
l
l
l
l
l
l
ll
l
l
ll
l
l
l
ll
l l
l
ll
l
ll
l
l
l
l
l l
l
ll
l
l l l
ll
l
l
l
ll
l l
ll
l
l
l
l
l
ll
l
l
l
l
lll
ll
l
ll
l
l
l
l
lll
l
l
l
l
ll
l l
l
l l
l
l
l l
l
l
l
l
l
l
l
l
l
ll
l
l
l
ll
l
l
l
l
ll
l
l
l
l
ll
l
l
l lll
l l
l
l
l
l ll
ll
l
l l ll
l
l
l
l
l
l
l l
l
l l
l
l
l
l
l
l l
ll
l
l
ll
l
l
-250 -200 -150 -100
1100
1150
1200
R² = 0.59
l
l
l
l
ll l
l
l
l
l
lll
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
l
l
ll
ll
l
l
l
l
l
l
lll
l
ll
l
ll
l
ll
l
l
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
l
l
ll
l
l
l
l l
l
l
l
ll
l
l
l
l
l
l
l
l
l
l
l l
ll
ll
ll
l
l
l
l
l
ll
l
l
l
ll
l
ll
l
ll
ll
l
l
l
l
l
l l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
l
l
l
l
ll
ll
l
l
l
l
l l
l
l
l
l
l
ll
ll l
l
l
l
l
l
l
l
l
l
ll
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
ll
ll
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
ll l
l
l
l
l
l
l
l
l l
l
ll
ll
l
l
l
l
l
l
l
l
ll
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
l
l
ll
l
l
l
l
l
l
l
l
ll
l
ll
l
l
l
l
ll
l
ll
l
l
l
ll
l
ll
ll
l
ll
l
l
l
l
l
l
l
l
l
l
l
l
l
ll
l l
l
l
l
l
l l
l
ll
l
l
ll
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l l
l l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
l
l
l
l
l
ll
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
l
l
l l
l l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
l
l
l
l
l
ll
l
l
l
l
l
l
l
l l
l
l
l
l
l l
ll
l
ll
l
l
l
l
l
l
l
l
ll
l
ll
ll
ll
l
ll
l
l
l
ll
l
l
l
l
l
l
ll
l
l
l
l
l
ll
l
l
l
l
l
l
ll
l l
l
l
l
l
l
ll
ll
l
ll
l
l
l
ll
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
l
l
l
l
l
l
l
ll
l
ll
l
l
l
l
ll
l
l
l
l
l
l
ll
l
ll
l
l
l
l
l
l
l
l
ll
l
l
l
l
lll
l
l
l
lll
l
l
l
ll
ll
l
l
l
l
l
l
l l
l
ll
l
l
l l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
l
l
l
l
l
lll
l
l ll
l
l
l ll
ll l
l
l
l
l
l
l
l
l
l l
l
l
l lll ll
l
l
l
l
l
l
l
l
l
lll
l
l
ll
l
ll
l
l
ll
l l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
l
l
l
l
ll
l l
ll
l
l
l
l
ll
l
l l
l
l
l
l
l
l
ll
l
l lll
l
l
l
l
l
l
l
l
l
ll
l
ll
l
l
l
l
l
l
l
l
ll
l
ll
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
l
l
ll
l
l
l
l l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
ll
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
l
l
l
l
l
l
l
l
l
ll
l
ll
l
l
l
l
l
l
l
l
l
l
l
l
lll
l
l
l
l
l l
l l
l
l
ll
l
ll
ll
l
l
l
l
l
l
l
ll
l
l
l
l l
l
l
l
l
ll
l
ll
ll
ll
l
l
l
l
l
l
l
l
ll
l
l
ll
l
l
l
l
ll
l
l
l l
l
l
l
l
l
l
l
ll
ll
l
l
ll
l
l
l
l
l
l
l
l
l
ll
l
l l
l
l
l
l
l
l
l
l
l
ll
l
l
ll
ll
l
l
l
l l
l
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
l
l
l
l
l
ll
l
l
l
ll
ll
l l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
l
l l
llll
l
ll
l
l
l
l
l
l
ll
l
l
l
ll
ll
l
l
l
l
ll
l
l
l
l
l
l
l
ll
l
l
l
l
l
l
l
l l
l
l
l
ll
l
l
ll
ll
l
l
l
l
l
ll
l
l
l
l
l
ll
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
l
l
lll
l
l
l
l
l
l
l
-180 -160 -140 -120 -100
1100
1150
1200
1250
R² = 0.64
l
l
l
l
ll
l
l
l
l
l
ll
l
l
l
l
l
l
l
l
l
l
l
l
l
l
ll
ll
l
ll
l
l
l
l
l
l
l
ll
l
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
l
ll
ll
l
l
l
l
l
l
l
ll
l
l
l
l
l
l
l
ll
ll
l
l
l
l
ll l
l
l
l
l
l
ll
l
l
l
l
l
l
l
l
l
l
l l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
ll
ll
l
ll
l
l
ll
l
l
l
ll
l
ll
l
l
llll
l
l
l
l
l
ll
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l l
l
l
l
l
l
lll
l
l
l
l
l
lll
l
l
l
l
l
ll
lll
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l l
l
l
l
l
l
l
l
l
l
l
ll
l
l
l
ll
l
l
l
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
l
l
l
l
l
l
ll
l
l
l
l
l
l
l
l
l
ll
ll
ll
ll
l
l
ll
ll
ll
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l l
l
l
l
l
ll
l
ll
l
l l ll
l
l l
l
l
l
ll
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
ll l
ll
l
ll
l
ll
l
ll
l
l
l
l
l
ll
l
l ll
ll
l
l
l
l
l
l
l
l
l
l
l
l
l
l l
l l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
l
l
ll
l
l
l
l
l
l
l
l
l
l
ll
l
l
l
l
l
l l
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
ll
l
l
l
l
l
l
l
l
ll
l
l
l
l
l
l
ll
ll
l
l
l
l
l
l l
ll
ll
l
l
l
l
l
l
l
l
ll
l
l
l
l
l
l
l
l
l
l
ll
ll l
l
l
l
l
l
l
l
l
ll
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l l
l
ll
ll
l
l
l
l
l
l
ll
l
l
ll
l
l
l
l
l
l
ll
l
l
l
l
l
ll
ll
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
l
l
l
l
l
l
l l
l
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
l
l
ll
l l
l
l l
l
l
l
l
l
l
l
l
l
l
ll
l
l
l
l
l
l
ll
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
ll
l
ll
l
l
l
l
l
l
l l
l
ll
l
l
l
ll
l
ll
l
l
l
l
ll
l
ll
l
l
l
l
l
l
l
l
lll
ll
l
ll
l
l
ll
llll
l
l
ll
l
l
l
l
ll
l
l
l ll
ll
l
l
ll
l
l
l
l l
l
l
l
l
l
l
l
l
ll
l
l
l
l l
l
l
l
l
ll
l
l
l
ll
l
l
l
l
l
l
ll
l
ll
l
l
l
l
l
l
ll
l
lll
l
l
l
l
lll
ll
l
l
l
l
ll
ll
l
l
l
l
l
l
l
l
l
ll
l
l l l l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l ll l
l
l
l
l
l
l
ll
l
l
l
l
l
l
l
l
l
ll
l
l ll l
l
l
l
l
l
l
ll
l
ll
ll
l
ll
l
ll
l
ll
l
l
l
ll
ll
l
l
l
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
l
ll
l
l
l
ll
l
l l
l
l
l
l
l
l
l
ll
ll
l
ll
l
l
l
l
l
l
l
ll
l
l
ll
l
l
l
l
l
l
l
ll
l
l
ll
l
l
l
ll
ll
ll
l
ll
l
l
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
l l
l ll
l
l
l
ll
l
l
l
ll
l
l
l
l
ll
l
ll
l
l
l
ll
l
l
l
l
l
l
l
ll
lll
l
l
l
l
l
l
l
l
ll
ll
l
l
l
l
ll
l
ll
l
l
l l
l
l
l
ll
l
l
ll
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
ll
l
l
l
l
l
l
l
l
l
l l
lll
l
l
l
l
ll
l
l
ll
l
ll
l
l
l
l
l
l
l
l
ll
l
l
l
l
ll
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l l
l
ll l
l
l
l
l
l
ll
l
l
l
l
l
ll
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
ll
ll
l
ll
l
l
ll
l
l
l
l
l
l
l
l l
l
ll
ll
l
l
l
l
l
l
ll
l
l
l
l
l
l
ll
l
l
l
ll
l
l
l
l
l
ll
ll l
l
l ll
l
l
l
l
l ll
l
l
l
l
l
ll
l
l
ll
l
ll
l
l
l
ll
l
l
l
ll l
l
l
l
l
l
l
l
llll
l
l
l
l
l
ll
l
l
l
l
lll
l
l
l
l
l
l
l
l
l
l
l
l
l
ll
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
-160 -140 -120 -100
1100
1150
1200
1250
1300
R² = 0.8
l l l l l l l l l l l l l l l l l l l ll l
ll l
ll l l l l
l l l ll l l l l l l l l
l
l
l ll
l l
ll
l l l l
10 20 30 40 50 60
0.0
0.2
0.4
0.6
0.8
1.0
Min. number of sessions per speaker
R²
ll l
l ll l l
l l ll l l l l l
l l l l l l
l l l l l l l l
l l ll l
l l ll l l l l l
l l ll l
l l l
ll
l l
10 20 30 40 50 60
0.0
0.2
0.4
0.6
0.8
1.0
Min. number of sessions per speaker
R²
l ll l
l l l l l l l ll l
l ll
l l l l l ll l l l l
l ll l
l l l ll l
l ll l l l l
ll l l l
l ll
l
l l l
10 20 30 40 50 60
0.0
0.2
0.4
0.6
0.8
1.0
Min. number of sessions per speaker
R²
l ll l l l l l
l l l ll l
l l l l l l l l l l l l l l l ll l
l l l l l ll l l l l l l l l l
l ll l l
l l l l
10 20 30 40 50 60
0.0
0.2
0.4
0.6
0.8
1.0
Min. number of sessions per speaker
R²
Figure 2: Likelihoods of speaker and residual factors computed
on PLDA development speaker-classes, without or with
length-normalization and using the different approaches. Figure
also depicts the coefficient of determination R2 between the
likelihoodsas a function of the minimal number of utterances per
speaker.
7.1.1. Property of length-normalized vectors
First, we present a simple relation between the variance of
atraining speaker-class and the position of its centroı̈d, once
allthe i-vectors are length-normalized.
Let {wsi }nsi=1 denote the set of ns i-vectors of a speaker sand
ws its mean vector. The variance of this sample is equal to:
1
ns
ns∑i=1
‖wsi −ws‖2 =1
ns
ns∑i=1
‖wsi ‖2 − ‖ws‖2 (14)
= 1− ‖ws‖2 (15)
as all the vectors have a norm of 1. This equality showsthat a
link exists between the position of a speaker-class andits
variance. G-PLDA model assumes that for any speaker and,thus, for
any speaker-dependent term µ+Φys, the within-classvariability can
be expressed by a unique speaker-independentcovariance matrix Λ.
Equation (15) shows a dependency be-tween the dispersion of a class
and its position in regard to theorigin, expressed by ‖ws‖2. To
better assess this phenomenonand observe whether or not it occurs
with G-PLDA class-factorsys and ε, the next paragraph analyzes
heteroscedasticity of theresidue ε.
7.1.2. Heteroscedasticity
Given a G-PLDA model M = (µ,Φ,Λ) computed on a de-velopment
corpus T , we denote by {ws,i}nsi=1 the collection ofns i-vectors
from speaker s of T and ys, {εs,i}nsi=1 the corre-sponding factors
provided byM. The model assumes that, fori = 1 to ns
ws,i = µ+ Φys + εs,i (16)
We denote by llk (ys|M) the posterior log-likelihood of ys
ac-cording toM, equal to
llk (ys|M) = logN (ys| 0, I)
= − r2
log (2π)− 12‖ys‖2 (17)
Let llk (ε|M) denote the average posterior log-likelihood of
theresidue over all observations of s according toM, given by
llk (ε|M) = 1ns
∑ilogN (ws,i|µ+ Φys,Λ)
= −p2
log 2π − 12
log |Λ| − 12ns∑
i(ws,i − µ−Φys)
t Λ−1 (ws,i − µ−Φys)(18)
Figure 2 top panel displays llk (ys|M) and llk (ε|M)
ofspeaker-classes from our development corpus T . Analysis hasbeen
carried out for systems 1, 3, 4, 5 of section 6:
withoutnormalization then (LΣ, EM-ML), (LW, EM-ML) and
(LW,deterministic). Without normalization (first graph), no
relationoccurs between the two likelihoods. The coefficient of
deter-mination R2, which indicates how well data points fit a line,
isindicated in the figure and close to 0. The least-square line
isdisplayed and close to be horizontal. If variances of Λ and ε
ex-actly match, llk (ε|M) is equal to − p
2log 2π − 1
2log |Λ| − p
2.
This value is plotted by a horizontal dashed line. Figure 2
showsthat the series llk (ε|M) corresponds to its theoretical value
be-fore length-normalization.
After any LN technique (graphs 2 to 4), a strong linear
re-lation occurs between the two likelihoods. All the R2
exceed0.59. It can be objected that this result is due to the less
in-formative training speaker-classes (training speakers with
lowamount of utterances). Figure 2 bottom panel displays the R2
series between the likelihoods (y-axis), only for speakers witha
minimal number of utterances (x-axis). The intensity of therelation
remains high and even slightly increases with the min-imal amount
of utterances, thus for the main training classes interms of
information.
This significant relation between likelihoods of
G-PLDAclass-factors ys and εs entails heteroscedasticity of the
residue.The likelihood llk (ε|M) is equal, up to a constant, to a
linearfunction of Λ−1. This relation shows that the residual
varianceis depending on the class position, which prevents an
overalllinear parameter Λ to be optimal. It should be replaced
by
-
46
a class-dependent parameter Λs, at least taking into account‖µ+
Φys‖ to fit to the actual class-dispersion.
8. ConclusionThis paper investigates the limits of linear and
homoscedasticGaussian PLDA modeling applied to spherical i-vector
distribu-tions. First, we propose a new deterministic approach for
esti-mating PLDA metaparameters. This non-probabilistic
approachenables to assess ability of a maximum likelihood
(ML)-basedapproach to improve the goodness of fit. It turns out
that resultsof this non-probabilistic approach are similar to those
of theEM-ML approach in terms of EER, and close in terms of
min-imal DCF. We consider the inefficiency of the latter approachas
an empirical evidence towards lack of compliance with themodel
assumptions. Therefore, we carried out an analysis of
ho-moscedasticity, revealing a dependency between the dispersionof
a class and its position in regard to the origin. This
significantrelation violates clearly the assumption of
homoscedasticity.
Instead of dealing with severe within-class distortions
andoutlying observations by proposing non-Gaussian prior, asdone in
heavy-tailed PLDA, Gaussian PLDA draws on length-normalization to
bring data onto a surface of high Gaussian like-lihood.
Length-normalization makes the development and triali-vector
distributions more similar and more Gaussian shaped,but i-vectors
lie on the non-linear and finite surface of a hyper-sphere. As
noticed in [16], assumptions of linearity and ho-moscedasticity
cannot be fulfilled when data share a commonnorm and, thus, G-PLDA
model cannot be optimal.
Also, these findings confirm the benefit of the LW
con-ditioning. By standardizing data according to a
within-classvariability, techniques like WCCN [2] or LW [10] move
thisvariability towards an isotropic model σI [17]. As remarked
in[10], this model does not favor any principal direction of
ses-sion variability nor dependence between directions and,
thus,alleviates the concern about heteroscedasticity.
9. PerspectivesAdvances in i-vector modeling could be achieved
by pursuingthe HT-PLDA approach, which attempts to find out
adequatepriors, or by preserving length-normalization (as this
techniquehas underlined importance of the directional information
inthe i-vector space) then modeling such representations
usingspherical distributions. The difficulty associated with
spheri-cal representations prompts researchers to model spherical
datausing Gaussian distributions. Approximations of
covariancematrix have been proposed, based on “unscented
transforms”[18] or first order Taylor expansion of the non-linear
length-normalization [19]. However, the analysis of
homoscedastic-ity presented in this paper shows that the overall
within-classvariability parameter should be replaced by a
class-dependentparameter taking into account the local position of
the class tofit to its actual dispersion. Such a non-linear model
induces acomplex density by passing the within-class variability
parame-ter through a non-linear function. This model is harder to
workwith in practice. It requires further research, to marginalize
overthe hidden variables and provide a low time consuming
scoringphase.
10. References[1] Najim Dehak, Read Dehak, Patrick Kenny, Niko
Brum-
mer, Pierre Ouellet, and Pierre Dumouchel, “Support Vec-
tor Machines versus Fast Scoring in the Low-DimensionalTotal
Variability Space for Speaker Verification,” in Inter-national
Conference on Speech Communication and Tech-nology, 2009, pp.
1559–1562.
[2] Najim Dehak, Patrick Kenny, Reda Dehak, Pierre Du-mouchel,
and Pierre Ouellet, “Front-End Factor Analy-sis for Speaker
Verification,” IEEE Transactions on Au-dio, Speech, and Language
Processing, vol. 19, no. 4, pp.788–798, 2011.
[3] Patrick Kenny, Pierre Ouellet, Najim Dehak, VishwaGupta, and
Pierre Dumouchel, “A study of inter-speakervariability in speaker
verification,” IEEE Transactions onAudio, Speech, and Language
Processing, vol. 16, no. 5,pp. 980–988, 2008.
[4] Simon J.D. Prince and James H. Elder, “Probabilistic lin-ear
discriminant analysis for inferences about identity,”in
International Conference on Computer Vision. IEEE,2007, pp.
1–8.
[5] Patrick Kenny, “Bayesian speaker verification with
heavy-tailed priors,” in Speaker and Language RecognitionWorkshop
(IEEE Odyssey), 2010.
[6] Mohammed Senoussaoui, Patrick Kenny, Niko Brummer,Edward de
Villiers, and Pierre Dumouchel, “Mixtureof PLDA models in i-vector
space for gender indepen-dent speaker recognition,” in
International Conference onSpeech Communication and Technology,
2011.
[7] Zia Khan and Frank Dellaert, “Robust generative sub-space
modeling: The subspace t distribution,” Tech. Rep.,GVU Center,
College of Computing, Georgia, 2004.
[8] Daniel Garcia-Romero and Carol Y. Espy-Wilson, “Anal-ysis of
i-vector length normalization in speaker recogni-tion systems,” in
International Conference on SpeechCommunication and Technology,
2011, pp. 249–252.
[9] Pierre-Michel Bousquet, Driss Matrouf, and Jean-François
Bonastre, “Intersession compensation and scor-ing methods in the
i-vectors space for speaker recogni-tion,” in International
Conference on Speech Communi-cation and Technology, 2011, pp.
485–488.
[10] Pierre-Michel Bousquet, Anthony Larcher, Driss
Matrouf,Jean-François Bonastre, and Oldřich Plchot,
“Variance-Spectra based Normalization for I-vector Standard
andProbabilistic Linear Discriminant Analysis,” in Speakerand
Language Recognition Workshop (IEEE Odyssey),2012.
[11] Pierre-Michel Bousquet, Jean-François Bonastre, andDriss
Matrouf, “Identify the benefits of the differentsteps in an
i-vector based speaker verification system,” inCIARP, Springer,
Ed., 2013, vol. Part II, pp. 278–285.
[12] Simon J.D. Prince, Computer Vision: Models Learningand
Inference, Cambridge University Press, 2012, Inpress.
[13] Carlos Vaquero, “Dataset Shift in PLDA based
SpeakerVerification,” in Speaker and Language RecognitionWorkshop
(IEEE Odyssey), 2012.
[14] R. Saeidi and al., “I4u submission to NIST sre 2012:A
large-scale collaborative effort for noise-robust
speakerverification,” in International Conference on Speech
Com-munication and Technology, 2013.
-
47
[15] Pavel Matejka, Ondrej Glembeck, Fabio Castaldo, M.J.Alam,
Oldrich Plchot, Patrick Kenny, Lukas Burget, andJan Cernocky,
“Full-covariance UBM and heavy-tailedPLDA in i-vector speaker
verification,” in InternationalConference on Speech Communication
and Technology,2011, pp. 4828–4831.
[16] Onur C. Hamsici and Aleix M. Martinez,
“Spherical-homoscedastic distributions: The equivalency of
sphericaland normal distributions in classification,” The Journal
ofMachine Learning Research, vol. 8, pp. 1583–1623, 2007.
[17] Michael E. Tipping and Christopher M. Bishop, “Mix-tures of
probabilistic principal component analyzers,”Neural computation,
vol. 11, no. 2, pp. 443–482, 1999.
[18] Patrick Kenny, Themos Stafylakis, Pierre Ouellet, Md.
Ja-hangir Alam, and Pierre Dumouchel, “PLDA for speakerverification
with utterances of arbitrary duration,” in Inter-national
Conference on Speech Communication and Tech-nology, 2013.
[19] Sandro Cumani, Oldřich Plchot, and Pietro Laface,
“Prob-abilistic linear discriminant analysis of i-vector poste-rior
distributions,” in IEEE International Conference onAcoustics,
Speech, and Signal Processing, ICASSP, 2013.
