How Familiar Does That Sound? Cross-Lingual RepresentationalSimilarity Analysis of Acoustic Word Embeddings

Badr M. Abdullah Iuliia Zaitova Tania AvgustinovaBernd Möbius Dietrich Klakow

Department of Language Science and Technology (LST)Saarland Informatics Campus, Saarland University, Germany

Corresponding author: babdullah@lsv.uni-saarland.de

Abstract

How do neural networks “perceive” speechsounds from unknown languages? Does thetypological similarity between the model’straining language (L1) and an unknown lan-guage (L2) have an impact on the model rep-resentations of L2 speech signals? To an-swer these questions, we present a novel ex-perimental design based on representationalsimilarity analysis (RSA) to analyze acousticword embeddings (AWEs)—vector represen-tations of variable-duration spoken-word seg-ments. First, we train monolingual AWE mod-els on seven Indo-European languages withvarious degrees of typological similarity. Wethen employ RSA to quantify the cross-lingualsimilarity by simulating native and non-nativespoken-word processing using AWEs. Our ex-periments show that typological similarity in-deed affects the representational similarity ofthe models in our study. We further discuss theimplications of our work on modeling speechprocessing and language similarity with neuralnetworks.

1 Introduction

Mastering a foreign language is a process that re-quires (human) language learners to invest time andeffort. If the foreign language (L2) is very distantfrom our native language (L1), not much of ourprior knowledge of language processing would berelevant in the learning process. On the other hand,learning a language that is similar to our nativelanguage is much easier since our prior knowledgebecomes more useful in establishing the correspon-dences between L1 and L2 (Ringbom, 2006). Insome cases where L1 and L2 are closely relatedand typologically similar, it is possible for an L1speaker to comprehend L2 linguistic expressionsto a great degree without prior exposure to L2. Theterm receptive multilingualism (Zeevaert, 2007) hasbeen coined in the sociolinguistics literature to de-scribe this ability of a listener to comprehend utter-

ances of an unknown but related language withoutbeing able to produce it.

Human speech perception has been an activearea of research in the past five decades whichhas produced a wealth of documented behavioralstudies and experimental findings. Recently, therehas been a growing scientific interest in the cog-nitive modeling community to leverage the recentadvances in speech representation learning to for-malize and test theories of speech perception usingcomputational simulations on the one hand, and toinvestigate whether neural networks exhibit similarbehavior to humans on the other hand (Räsänenet al., 2016; Alishahi et al., 2017; Dupoux, 2018;Scharenborg et al., 2019; Gelderloos et al., 2020;Matusevych et al., 2020b; Magnuson et al., 2020).

In the domain of modeling non-native speechperception, Schatz and Feldman (2018) have shownthat a neural speech recognition system (ASR) pre-dicts Japanese speakers’ difficulty with the Englishphonemic contrast /l/-/ô/ as well as English speak-ers’ difficulty with the Japanese vowel length dis-tinction. Matusevych et al. (2021) have shownthat a model of non-native spoken word processingbased on neural networks predicts lexical process-ing difficulty of English-speaking learners of Rus-sian. The latter model is based on word-level repre-sentations that are induced from naturalistic speechdata known in the speech technology communityas acoustic word embeddings (AWEs). AWE mod-els map variable-duration spoken-word segmentsonto fixed-size representations in a vector spacesuch that instances of the same word type are (ide-ally) projected onto the same point in space (Levinet al., 2013). In contrast to word embeddings innatural language processing (NLP), an AWE en-codes information about the acoustic-phonetic andphonological structure of the word, not its semanticcontent.

Although the model of Matusevych et al. (2021)has shown similar effects to what has been ob-

arX

iv:2

109.

1017

9v1

[cs

.CL

] 2

1 Se

p 20

21

Native EncoderLanguage ( )

Non-native EncoderLanguage ( )

Non-native Encoder Language ( )

a simulation of a native speaker's phonetic perceptual space

a simulation of a non-native speaker's phonetic perceptual space

a simulation of a non-native speaker's phonetic perceptual space

Spoken-word stimuli from native speakers

of language ( )

Figure 1: An illustrated example of our experimental design. A set of N spoken-word stimuli from languageλ are embedded using the encoder F (λ) which was trained on language λ to obtain a (native) view of the data:X(λ/λ) ∈ RD×N . Simultaneously, the same stimuli are embedded using encoders trained on other languages,namely F (α) and F (β), to obtain two different (non-native) views of the data: X(λ/α) and X(λ/β). We quantifythe cross-lingual similarity between two languages by measuring the association between their embedding spacesusing the representational similarity analysis (RSA) framework.

served in behavioral studies (Cook et al., 2016),it remains unclear to what extent AWE modelscan predict a facilitatory effect of language simi-larity on cross-language spoken-word processing.In this paper, we present a novel experimental de-sign to probe the receptive multilingual knowledgeof monolingual AWE models (i.e., trained with-out L2 exposure) using the representational simi-larity analysis (RSA) framework. In a controlledexperimental setup, we use AWE models to sim-ulate spoken-word processing of native speakersof seven languages with various degrees of typo-logical similarity. We then employ RSA to char-acterize how language similarity affects the emer-gent representations of the models when tested onthe same spoken-word stimuli (see Figure 1 foran illustrated overview of our approach). Our ex-periments demonstrate that neural AWE models ofdifferent languages exhibit a higher degree of rep-resentational similarity if their training languagesare typologically similar.

2 Proposed Methodology

A neural AWE model can be formally described asan encoder function F : A −→ RD, where A is the(continuous) space of acoustic sequences and Dis the dimensionality of the embedding. Given anacoustic word signal represented as a temporal se-quence of T acoustic eventsA = (a1,a2, ...,aT ),

where at ∈ Rk is a spectral vector of k coefficients,an embedding is computed as

x = F(A;θ) ∈ RD (1)

Here, θ are the parameters of the encoder, whichare learned by training the AWE model in a mono-lingual supervised setting. That is, the trainingspoken-word segments (i.e., speech intervals corre-sponding to spoken words) are sampled from utter-ances of native speakers of a single language wherethe word identity of each segment is known. Themodel is trained with an objective that maps differ-ent spoken segments of the same word type ontosimilar embeddings. To encourage the model toabstract away from speaker variability, the trainingsamples are obtained from multiple speakers, whilethe resulting AWEs are evaluated on a held-out setof speakers.

Our research objective in this paper is to studythe discrepancy of the representational geometryof two AWE encoders that are trained on differentlanguages when tested on the same set of (mono-lingual) spoken stimuli. To this end, we train AWEencoders on different languages where the train-ing data and conditions are comparable across lan-guages with respect to size, domain, and speakervariability. We therefore have access to severalencoders {F (α),F (β), . . . , ,F (ω)}, where the su-perscripts {α, β, . . . , ω} denote the language of the

training samples.Now consider a set of N held-out spoken-word

stimuli produced by native speakers of languageλ: A(λ)

1:N = {A(λ)1 , . . . ,A

(λ)N }. First, each acoustic

word stimulus in this set is mapped onto an em-bedding using the encoder F (λ), which yields amatrix X(λ/λ) ∈ RD×N . Since the encoder F (λ)

was trained on language λ, we refer to it as thenative encoder and consider the matrix X(λ/λ) asa simulation of a native speaker’s phonetic per-ceptual space. To simulate the phonetic percep-tual space of a non-native speaker, say a speakerof language α, the stimuli A(λ)

1:N are embeddedusing the encoder F (α), which yields a matrixX(λ/α) ∈ RD×N . Here, we read the superscript no-tation (λ/α) as word stimuli of language λ encodedby a model trained on language α. Thus, the twomatrices X(λ/λ) and X(λ/α) represent two differ-ent views of the same stimuli. Our main hypothesisis that the cross-lingual representational similaritybetween emergent embedding spaces should reflectthe acoustic-phonetic and phonological similaritiesbetween the languages λ and α. That is, the moredistant languages λ and α are, the more dissimilartheir corresponding representation spaces are. Toquantify this cross-lingual representational simi-larity, we use Centered Kernel Alignment (CKA)(Kornblith et al., 2019). CKA is a representation-level similarity measure that emphasizes the dis-tributivity of information and therefore it obviatesthe need to establish the correspondence mappingbetween single neurons in the embeddings of twodifferent models. Moreover, CKA has been shownto be invariant to orthogonal transformation andisotropic scaling which makes it suitable for ouranalysis when comparing different languages andlearning objectives. Using CKA, we quantify thesimilarity between languages λ and α as

sim(λ, α) := CKA(X(λ/λ),X(λ/α)) (2)

Here, sim(λ, α) ∈ [0, 1] is a scalar that mea-sures the correlation between the responses ofthe two encoders, i.e., native F (λ) and non-nativeF (α), when tested with spoken-word stimuliA(λ)

1:N .sim(λ, α) = 1 is interpreted as perfect asso-ciation between the representational geometryof models trained on languages λ and α whilesim(λ, α) = 0 indicates that no association canbe established. Likewise, we obtain another non-native view X(λ/β) of the stimuli A(λ)

1:N using an-other non-native encoder F (β). We then quan-

tify the cross-lingual representational similarity be-tween languages λ and β as

sim(λ, β) := CKA(X(λ/λ),X(λ/β)) (3)

If sim(λ, α) > sim(λ, β), then we interpretthis as an indication that the native phonetic per-ceptual space of language λ is more similar tothe non-native phonetic perceptual space of lan-guage α, compared to that of language β. Notethat while CKA(., .) is a symmetric metric (i.e.,CKA(X,Y) = CKA(Y,X)), our establishedsimilarity metric sim(., .) is not symmetric (i.e.,sim(λ, α) : 6= sim(α, λ)). To estimate sim(α, λ),we use word stimuli of language α and collect thematrices X(α/α) and X(α/λ). Then we compute

sim(α, λ) := CKA(X(α/α),X(α/λ)) (4)

When we apply the proposed experimentalpipeline across M different languages, the effectof language similarity can be characterized by con-structing a cross-lingual representational similaritymatrix (xRSM) which is an asymmetric M ×Mmatrix where each cell represents the correlation(or agreement) between two embedding spaces.

3 Acoustic Word Embedding Models

We investigate three different approaches of train-ing AWE models that have been previously intro-duced in the literature. In this section, we formallydescribe each one of them.

3.1 Phonologically Guided Encoder

The phonologically guided encoder (PGE) is asequence-to-sequence model in which the networkis trained as a word-level acoustic model (Abdul-lah et al., 2021). Given an acoustic sequenceA and its corresponding phonological sequenceϕ = (ϕ1, . . . , ϕτ ),1 the acoustic encoder F istrained to takeA as input and produce an AWE x,which is then fed into a phonological decoder Gwhose goal is to generate the sequenceϕ (Fig. 2–a).The objective is to minimize a categorical cross-entropy loss at each timestep in the decoder, whichis equivalent to

L = −τ∑i=1

log PG(ϕi|ϕ<i,x) (5)

1We use the phonetic transcription in IPA symbols as thephonological sequence ϕ.

Acoustic WordEmbedding

(a) (b) (c)

Figure 2: A visual illustration of the different learning objectives for training AWE encoders: (a) phonologicallyguided encoder (PGE): a sequence-to-sequence network with a phonological decoder, (b) correspondence auto-encoder (CAE): a sequence-to-sequence network with an acoustic decoder, and (c) contrastive siamese encoder(CSE): a contrastive network trained via triplet margin loss.

where PG is the probability of the phone ϕi attimestep i, conditioned on the previous phone se-quence ϕ<i and the embedding x. The intuition ofthis objective is that spoken-word segments of thesame word type would have identical phonologicalsequences, thus they are expected to end up nearbyin the embedding space.

3.2 Correspondence AutoencoderIn the correspondence autoencoder (CAE) (Kam-per, 2019), each training acoustic segment A ispaired with another segment that corresponds tothe same word typeA+ = (a+1 ,a

+2 , ...,a

+T+). The

acoustic encoder F then takesA as input and com-putes an AWE x, which is then fed into an acousticdecoder H that attempts to reconstruct the corre-sponding acoustic sequence A+ (Fig. 2–b). Theobjective is to minimize a vector-wise mean squareerror (MSE) loss at each timestep in the decoder,which is equivalent to

L = −T+∑i=1

||a+i −Hi(x)||2 (6)

where a+i is the ground-truth spectral vector attimestep i andHi(x) is the reconstructed spectralvector at timestep i as a function of the embeddingx. It has been hypothesized that learning the corre-spondence between two acoustic realizations of thesame word type encourages the model to build upspeaker-invariant word representations while pre-serving phonetic information (Matusevych et al.,2020a).

3.3 Contrastive Siamese EncoderThe contrastive siamese encoder (CSE) is basedon a siamese network (Bromley et al., 1994) and

has been explored in the AWEs literature with dif-ferent underlying architectures (Settle and Livescu,2016; Kamper et al., 2016). This learning objec-tive differs from the first two as it explicitly mini-mizes/maximizes relative distances between AWEsof the same/different word types (Fig. 2–c). First,each acoustic word instance is paired with anotherinstance of the same word type, which yields thetuple (A,A+). Given the AWEs of the instancesin this tuple (xa,x+), the objective is then to mini-mize a triplet margin loss

L = max[0,m+ d(xa,x+)− d(xa,x−)

](7)

Here, d : RD × RD → [0, 1] is the cosine distanceand x− is an AWE that corresponds to a differentword type sampled from the mini-batch such thatthe term d(xa,x−) is minimized. This objectiveaims to map acoustic segments of the same wordtype closer in the embedding space while pushingaway segments of other word types by a distancedefined by the margin hyperparameter m.

4 Experiments

4.1 Experimental DataThe data in our study is drawn from the Global-Phone speech (GPS) database (Schultz et al., 2013).GPS is a multilingual read speech resource wherethe utterances are recorded from native speakers(with self-reported gender labels) in a controlledrecording environment with minimal noise. There-fore, the recording conditions across languages arecomparable which enables us to conduct a cross-linguistic comparison. We experiment with sevenIndo-European languages: Czech (CZE), Polish(POL), Russian (RUS), Bulgarian (BUL), Brazil-ian Portuguese (POR), French (FRA), and German

(DEU). We acknowledge that our language sampleis not typologically diverse. However, one of ourobjectives in this paper is to investigate whetherthe cross-lingual similarity of AWEs can predictthe degree of mutual intelligibility between relatedlanguages. Therefore, we focus on the Slavic lan-guages in this sample (CZE, POL, RUS, and BUL),which are known to be typologically similar andmutually intelligible to various degrees.

To train our AWE models, we obtain time-aligned spoken-word segments using the MontrealForced Aligner (McAuliffe et al., 2017). Then,we sample 42 speakers of balanced gender fromeach language. For each language, we sample~32k spoken-word segments that are longer than 3phonemes in length and shorter than 1.1 seconds induration (see Table 2 in Appendix A for word-levelsummary statistics of the data). For each word type,we obtain an IPA transcription using the grapheme-to-phoneme (G2P) module of the automatic speechsynthesizer, eSpeak. Each acoustic word segmentis parametrized as a sequence of 39-dimensionalMel-frequency spectral coefficients where framesare extracted over intervals of 25ms with 10msoverlap.

4.2 Architecture and Hyperparameters

Acoustic Encoder We employ a 2-layer recurrentneural network with a bidirectional Gated Recur-rent Unit (BGRU) of hidden state dimension of512, which yields a 1024-dimensional AWE.

Training Details All models in this study aretrained for 100 epochs with a batch size of 256using the ADAM optimizer (Kingma and Ba, 2015)and an initial learning rate (LR) of 0.001. The LRis reduced by a factor of 0.5 if the mean averageprecision (mAP) for word discrimination on thevalidation set does not improve for 10 epochs. Theepoch with the best validation performance duringtraining is used for evaluation on the test set.

Implementation We build our models using Py-Torch (Paszke et al., 2019) and use FAISS (John-son et al., 2017) for efficient similarity search dur-ing evaluation. Our code is publicly available onGitHub.2

4.3 Quantitative Evaluation

We evaluate the models using the standard intrin-sic evaluation of AWEs: the same-different worddiscrimination task (Carlin et al., 2011). This task

2https://github.com/uds-lsv/xRSA-AWEs

Encoder typeLanguage

PGE CAE CSE

CZE 78.3 76.1 82.9POL 67.6 63.5 73.8RUS 64.3 57.7 71.0BUL 72.1 68.9 78.4POR 74.5 72.2 80.4FRA 65.6 64.5 68.5DEU 67.9 70.3 75.8

Table 1: mAP performance on evaluation sets.

aims to assess the ability of a model to determinewhether or not two given speech segments corre-spond to the same word type, which is quantifiedusing a retrieval metric (mAP) reported in Table 1.

5 Representational Similarity Analysis

Figure 3 shows the cross-lingual representationalsimilarity matrices (xRSMs) across the three differ-ent models using the linear CKA similarity metric.3

Warmer colors indicate a higher similarity betweentwo representational spaces. One can observe strik-ing differences between the PGE-CAE models onthe one hand, and the CSE model on the other hand.The PGE-CAE models yield representations thatare cross-lingually more similar to each other com-pared to those obtained from the contrastive CSEmodel. For example, the highest similarity scorefrom the PGE model is sim(RUS, BUL) = 0.748,which means that the representations of the Rus-sian word stimuli from the Bulgarian model F (BUL)

exhibit the highest representational similarity to therepresentations of the native Russian model F (RUS).On the other hand, the lowest similarity score fromthe PGE model is sim(POL, DEU) = 0.622, whichshows that the German model’s view of the Polishword stimuli is the view that differs the most fromthe view of the native Polish model. Likewise, thehighest similarity score we observe from the CAEmodel is sim(BUL, RUS) = 0.763, while the low-est similarity is sim(DEU, BUL) = 0.649. If wecompare these scores to those of the contrastiveCSE model, we observe significant cross-languagedifferences as the highest similarity scores aresim(POL, RUS) = sim(POR, RUS) = 0.269, whilethe lowest is sim(BUL, DEU) = 0.170. This dis-crepancy between the contrastive model and the

3The xRSMs obtained using non-linear CKA with an RBFkernel are shown in Figure 6 in Appendix B, which show verysimilar trends to those observed in Figure 3.

CZE POL RUS BUL POR FRA DEU

CZE

POL

RUS

BUL

POR

FRA

DEU 0.687 0.680 0.659 0.649 0.660 0.658

0.739 0.735 0.705 0.710 0.737 0.717

0.737 0.722 0.735 0.736 0.726 0.705

0.742 0.731 0.763 0.731 0.706 0.692

0.745 0.735 0.761 0.721 0.695 0.690

0.759 0.721 0.721 0.693 0.700 0.681

0.762 0.747 0.736 0.706 0.717 0.703

CZE POL RUS BUL POR FRA DEU

CZE POL RUS BUL POR FRA DEU

0.2

0.3

0.4

0.5

0.6

0.7

0.8CZE

POL

RUS

BUL

POR

FRA

DEU 0.180 0.195 0.193 0.183 0.177 0.182

0.193 0.220 0.200 0.202 0.197 0.179

0.234 0.266 0.269 0.254 0.234 0.202

0.207 0.241 0.238 0.198 0.190 0.170

0.220 0.257 0.234 0.212 0.189 0.178

0.248 0.269 0.245 0.219 0.205 0.179

0.264 0.244 0.225 0.208 0.203 0.180

CZE POL RUS BUL POR FRA DEU

0.683 0.675 0.658 0.663 0.658 0.661

0.696 0.691 0.683 0.700 0.679 0.663

0.684 0.705 0.721 0.718 0.688 0.649

0.719 0.724 0.745 0.700 0.691 0.638

0.723 0.720 0.748 0.694 0.681 0.624

0.731 0.712 0.722 0.669 0.668 0.622

0.745 0.733 0.734 0.676 0.699 0.651

Figure 3: The cross-lingual representational similarity matrix (xRSM) for each model: PGE (Left), CAE (Middle),and CSE (Right). Each row corresponds to the language of the spoken-word stimuli while each column correspondsto the language of the encoder. Note that the matrices are not symmetric. For example in the PGE model, the cellat row CZE and column POL holds the value of sim(CZE, POL) = CKA(X(CZE/CZE),X(CZE/POL)) = 0.745, whilethe cell at row POL and column CZE holds the value of sim(POL, CZE) = CKA(X(POL/POL),X(POL/CZE)) = 0.731.

other models suggests that training AWE modelswith a contrastive objective hinders the ability ofthe encoder to learn high-level phonological ab-stractions and therefore contrastive encoders aremore sensitive to the cross-lingual variation duringinference compared to their sequence-to-sequencecounterparts.

5.1 Cross-Lingual Comparison

To get further insights into how language sim-ilarity affects the model representations of non-native spoken-word segments, we apply hierarchi-cal clustering on the xRSMs in Figure 3 usingthe Ward algorithm (Ward, 1963) with Euclideandistance. The result of the clustering analysis isshown in Figure 4. Surprisingly, the generatedtrees from the xRSMs of the PGE and CAE mod-els are identical, which could indicate that thesetwo models induce similar representations whentrained on the same data. Diving a level deeperinto the cluster structure of these two models, weobserve that the Slavic languages form a pure clus-ter. This could be explained by the observationthat some of the highest pair-wise similarity scoresamong the PGE models are observed betweenRussian and Bulgarian, i.e., sim(RUS, BUL) =0.748 and sim(BUL, RUS) = 0.745, and Czechand Polish, i.e., sim(CZE, POL) = 0.745. Thesame trend can be observed in the CAE models:i.e., sim(RUS, BUL) = 0.761, sim(BUL, RUS) =0.763, and sim(CZE, POL) = 0.762. Within theSlavic cluster, the West Slavic languages Czechand Polish are grouped together, while the Rus-sian (East Slavic) is first grouped with Bulgarian(South Slavic) before joining the West Slavic group

to make the Slavic cluster. Although Russian andBulgarian belong to two different Slavic branches,we observe that this pair forms the first sub-clusterin both trees, at a distance smaller than that of theWest Slavic cluster (Czech and Polish). At first,this might seem surprising as we would expect theWest Slavic cluster to be formed at a lower dis-tance given the similarities among the West Slaviclanguages which facilitates cross-language speechcomprehension, as documented by sociolinguisticstudies (Golubovic, 2016). However, Russian andBulgarian share typological features at the acoustic-phonetic and phonological levels which distinguishthem from West Slavic languages.4 We furtherelaborate on these typological features in §6. Eventhough Portuguese was grouped with French in thecluster analysis, which one might expect given thatboth are Romance languages descended from Latin,it is worth pointing out that the representations ofthe Portuguese word stimuli from the Slavic mod-els show a higher similarity to the representationsof the native Portuguese model compared to theseobtained from the French model (with only twoexceptions, the Czech PGE model and Polish CAEmodel). We also provide an explanation of whythis might be the case in §6.

The generated language cluster from the CSEmodel does not show any clear internal structurewith respect to language groups since all clusterpairs are grouped at a much higher distance com-pared to the PGE and CAE models. Furthermore,the Slavic languages in the generated tree do not

4Note that our models do not have access to word orthogra-phy. Thus, the similarity cannot be due to the fact that Russianand Bulgarian use Cyrillic script.

0.0

0.2

0.4

CzechPolishRussianBulgarianPortugueseFrench

German

0.0

0.2

0.4

CzechPolishRussianBulgarianPortugueseFrench

German

Portuguese

Russian

0.0

0.5

1.0

Polish

French

German

Czech

Bulgarian

Slavic Romance Germanic

Figure 4: Hierarchical clustering analysis on the cross-lingual representational similarity matrices (using linearCKA) of the three models: PGE (Left), CAE (Middle), and CSE (Right).

form a pure cluster since Portuguese was placedinside the Slavic cluster. We also do not observethe West Slavic group as in the other two modelssince Polish was grouped first with Russian, and notCzech. We believe that this unexpected behavior ofthe CSE models can be related to the previously at-tributed to the poor performance of the contrastiveAWEs in capturing word-form similarity (Abdullahet al., 2021).

Moreover, it is interesting to observe that Ger-man seems to be the most distant language to theother languages in our study. This observationholds across all three encoders since the represen-tations of the German word stimuli by non-nativemodels are the most dissimilar compared to therepresentations of the native German model.

5.2 Cross-Model Comparison

To verify our hypothesis that the models trainedwith decoding objectives (PGE and CAE) learnsimilar representations when trained on the samedata, we conduct a similarity analysis between themodels in a setting where we obtain views of thespoken-word stimuli from native models, then com-pare these views across different learning objec-tives while keeping the language fixed using CKAas we do in our cross-lingual comparison. Theresults of this analysis is shown in Figure 5. Weobserve that across different languages the pairwiserepresentational similarity of the PGE-CAE models(linear CKA = 0.721) is very high in comparisonto that of PGE-CSE models (linear CKA = 0.239)and CAE-CSE models (linear CKA = 0.230).5 Al-though the non-linear CKA similarity scores tendto be higher, the general trend remains identical inboth measures. This finding validates our hypoth-esis that the PGE and CAE models yield similar

5CKA scores are averaged over languages.

representation spaces when trained on the samedata even though their decoding objectives oper-ate over different modalities. That is, the decoderof the PGE model aims to generate the word’sphonological structure in the form of a sequenceof discrete phonological units, while the decoderof the CAE model aims to generate an instanceof the same word represented as a sequence of(continuous) spectral vectors. Moreover, the se-quences that these decoders aim to generate vary inlength (the mean phonological sequence length is~6 phonemes while mean spectral sequence lengthis ~50 vectors). Although the CAE model has noaccess to abstract phonological information of theword-forms it is trained on, it seems that this modellearns non-trivial knowledge about word phono-logical structure as demonstrated by the represen-tational similarity of its embeddings to those ofa word embedding model that has access to wordphonology (i.e., PGE) across different languages.

6 Discussion

6.1 Relevance to Related Work

Although the idea of representational similarityanalysis (RSA) has originated in the neuroscienceliterature (Kriegeskorte et al., 2008), researchers inNLP and speech technology have employed a simi-lar set of techniques to analyze emergent represen-tations of multi-layer neural networks. For exam-ple, RSA has previously been employed to analyzethe correlation between neural and symbolic repre-sentations (Chrupała and Alishahi, 2019), contex-tualized word representations (Abnar et al., 2019;Abdou et al., 2019; Lepori and McCoy, 2020; Wuet al., 2020), representations of self-supervisedspeech models (Chung et al., 2021) and visually-grounded speech models (Chrupała et al., 2020).We take the inspiration from previous work and

0.2

0.3

0.4

0.5

0.6

0.7

0.8

PGE vs.CAE

PGE vs.CSE

CAE vs.CSE

Cros

s-m

odel

CK

A (l

inea

r)

0.239

0.721

PGE vs.CAE

PGE vs.CSE

CAE vs.CSE

0.401

0.757

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Cros

s-m

odel

CK

A (R

BF

0.5)

0.230

0.1

0.397

Figure 5: Cross-model CKA scores: linear CKA (left)and non-linear CKA (right). Each point in this plot isthe within-language representational similarity of twomodels trained with different objectives when tested onthe same (monolingual) word stimuli. Each red point isthe average CKA score per model pair.

apply RSA in a unique setting: to analyze the im-pact of typological similarity between languageson the representational geometry. Our analysis fo-cuses on neural models of spoken-word processingthat are trained on naturalistic speech corpora. Ourgoal is two-fold: (1) to investigate whether or notneural networks exhibit predictable behavior whentested on speech from a different language (L2),and (2) to examine the extent to which the trainingstrategy affects the emergent representations of L2spoken-word stimuli. To the best of our knowledge,our study is the first to analyze the similarity ofemergent representations in neural acoustic modelsfrom a cross-linguistic perspective.

6.2 A Typological Discussion

Given the cross-linguistic nature of our study, wechoose to discuss our findings on the representa-tional similarity analysis from a language typologypoint of view. Language typology is a sub-fieldwithin linguistics that is concerned with the studyand categorization of the world’s languages basedon their linguistic structural properties (Comrie,1988; Croft, 2002). Since acoustic word embed-dings are word-level representations induced fromactual acoustic realizations of word-forms, we fo-cus on the phonetic and phonological properties ofthe languages in our study. We emphasize that ourgoal is not to use the method we propose in this pa-per as a computational approach for phylogeneticreconstruction, that is, discovering the historicalrelations between languages. However, typologi-cal similarity usually correlates with phylogeneticdistance since languages that have diverged from acommon historical ancestor inherited most of their

typological features from the ancestor language(see Bjerva et al. (2019) for a discussion on howtypological similarity and genetic relationships in-teract when analyzing neural embeddings).

Within the language sample we study in this pa-per, four of these languages belong to the Slavicbranch of Indo-European languages. Comparedto the Romance and Germanic branches of Indo-European, Slavic languages are known to be re-markably more similar to each other not only atlexical and syntactic levels, but also at the pre-lexical level (acoustic-phonetic and phonologicalfeatures). These cross-linguistically shared fea-tures between Slavic languages include rich conso-nant inventories, phonemic iotation and complexconsonant clusters. The similarities at differentlinguistic levels facilitate spoken intercomprehen-sion, i.e., the ability of a listener to comprehendan utterance (to some degree) in a language that isunknown, but related to their mother tongue. Sev-eral sociolinguistic studies of mutual intelligibilityhave reported a higher degree of intercomprehen-sion among speakers of Slavic languages comparedto other language groups in Europe (Golubovic,2016; Gooskens et al., 2018). On the other hand,and even though French and Portuguese are bothRomance languages, they are less mutually intel-ligible compared to Slavic language pairs as theyhave diverged in their lexicons and phonologicalstructures (Gooskens et al., 2018). This shows thatcross-language speech intelligibility is not mainlydriven by shared historical relationships, but bycontemporary typological similarities.

Therefore, and given the documented phonologi-cal similarities between Slavic languages, it is notsurprising that Slavic languages form a pure clus-ter in our clustering analysis over the xRSMs oftwo of the models we investigate. Furthermore, thegrouping of Czech-Polish and Russian-Bulgarianin the Slavic cluster can be explained if we con-sider word-specific suprasegmental features. Be-sides the fact that Czech and Polish are both WestSlavic languages that form a spatial continuumof language variation, both languages have fixedstress. The word stress in Czech is always on theinitial syllable, while Polish has penultimate stress,that is, stress falls on the syllable preceding thelast syllable of the word (Sussex and Cubberley,2006). On the other hand, Russian and Bulgarianlanguages belong to different Slavic sub-groups—Russian is East Slavic while Bulgarian is South

Slavic. The representational similarity betweenRussian and Bulgarian can be attributed to the ty-pological similarities between Bulgarian and EastSlavic languages. In contrast to West Slavic lan-guages, the stress in Russian and Bulgarian is free(it can occur on any syllable in the word) and mov-able (it moves between syllables within morpholog-ical paradigms). Slavic languages with free stresstend to have a stronger contrast between stressedand unstressed syllables, and vowels in unstressedsyllables undergo a process that is known as vowel-quality alternation, or lexicalized vowel reduction(Barry and Andreeva, 2001). For example, con-sider the Russian word ruka (‘hand’). In the singu-lar nominative case the word-form is phoneticallyrealized as ruká [rU"ka] while in the singular ac-cusative case rúku ["rukU] (Sussex and Cubberley,2006). Here, the unstressed high back vowel /u/is realized as [U] in both word-forms. Althoughvowel-quality alternations in Bulgarian follow dif-ferent patterns, Russian and Bulgarian vowels inunstressed syllables are reduced in temporal dura-tion and quality. Therefore, the high representa-tional similarity between Russian and Bulgariancould be explained if we consider their typologicalsimilarities in word-specific phonetics and prosody.

The Portuguese language presents us with an in-teresting case study of language variation. From aphylogenetic point of view, Portuguese and Frenchhave diverged from Latin and therefore they areboth categorized as Romance languages. The clus-tering of the xRSMs of the PGE and CAE modelsgroups Portuguese and French together at a higherdistance compared to the Slavic group, while Por-tuguese was grouped with Slavic languages whenanalyzing the representations of the contrastiveCSE model. Similar to Russian and Polish, sibilantconsonants (e.g., /S/, /Z/) and the velarized (dark)L-sound (i.e., /ë/) are frequent in Portuguese. Wehypothesize that contrastive training encouragesthe model to pay more attention to the segmentalinformation (i.e., individual phones) in the speechsignal at the expense of phonotactics (i.e., phonesequences). Given that contrastive learning is a pre-dominant paradigm in speech representation learn-ing (van den Oord et al., 2018; Schneider et al.,2019), we encourage further research to analyzewhether or not speech processing models trainedwith contrastive objectives exhibit a similar behav-ior to that observed in human listeners and closelyexamine their plausibility for cognitive modeling.

6.3 Implications on Cross-Lingual TransferLearning

A recent study on zero-resource AWEs has shownthat cross-lingual transfer is more successful whenthe source (L1) and target (L2) languages are morerelated (Jacobs and Kamper, 2021). We conducteda preliminary experiment on the cross-lingual worddiscrimination performance of the models in ourstudy and observed a similar effect. However, wealso observed that cross-lingual transfer using thecontrastive model is less effective compared to themodels trained with decoding objectives. Fromour reported RSA analysis, we showed that thecontrastive objective yields models that are cross-lingually dissimilar compared to the other objec-tives. Therefore, future work could investigate thedegree to which our proposed RSA approach pre-dicts the effectiveness of different models in a cross-lingual zero-shot scenario.

7 Conclusion

We presented an experimental design based on rep-resentational similarity analysis (RSA) wherebywe analyzed the impact of language similarity onrepresentational similarity of acoustic word em-beddings (AWEs). Our experiments have shownthat AWE models trained using decoding objec-tives exhibit a higher degree of representationalsimilarity if their training languages are typolog-ically similar. We discussed our findings from atypological perspective and highlighted pre-lexicalfeatures that could have an impact on the models’representational geometry. Our findings provideevidence that AWE models can predict the facilita-tory effect of language similarity on cross-languagespeech perception and complement ongoing effortsin the community to assess their utility in cognitivemodeling. Our work can be further extended byconsidering speech segments below the word-level(e.g., syllables, phonemes), incorporating seman-tic representations into the learning procedure, andinvestigating other neural architectures.

Acknowledgements

We thank the anonymous reviewers for their con-structive comments and insightful feedback. Wefurther extend our gratitude to Miriam Schulz andMarius Mosbach for proofreading the paper. Thisresearch is funded by the Deutsche Forschungsge-meinschaft (DFG, German Research Foundation),Project ID 232722074 – SFB 1102.

ReferencesMostafa Abdou, Artur Kulmizev, Felix Hill, Daniel M.

Low, and Anders Søgaard. 2019. Higher-order com-parisons of sentence encoder representations. InProceedings of the 2019 Conference on EmpiricalMethods in Natural Language Processing and the9th International Joint Conference on Natural Lan-guage Processing (EMNLP-IJCNLP), pages 5838–5845, Hong Kong, China. Association for Computa-tional Linguistics.

Badr M. Abdullah, Marius Mosbach, Iuliia Zaitova,Bernd Möbius, and Dietrich Klakow. 2021. DoAcoustic Word Embeddings Capture PhonologicalSimilarity? An Empirical Study. In Proc. Inter-speech 2021, pages 4194–4198.

Samira Abnar, Lisa Beinborn, Rochelle Choenni, andWillem Zuidema. 2019. Blackbox meets blackbox:Representational similarity & stability analysis ofneural language models and brains. In Proceedingsof the 2019 ACL Workshop BlackboxNLP: Analyzingand Interpreting Neural Networks for NLP, pages191–203, Florence, Italy. Association for Computa-tional Linguistics.

Afra Alishahi, Marie Barking, and Grzegorz Chru-pała. 2017. Encoding of phonology in a recurrentneural model of grounded speech. In Proceedingsof the 21st Conference on Computational NaturalLanguage Learning (CoNLL 2017), pages 368–378,Vancouver, Canada. Association for ComputationalLinguistics.

William Barry and Bistra Andreeva. 2001. Cross-language similarities and differences in spontaneousspeech patterns. Journal of the International Pho-netic Association, 31(1):51–66.

Johannes Bjerva, Robert Östling, Maria Han Veiga,Jörg Tiedemann, and Isabelle Augenstein. 2019.What do language representations really represent?Computational Linguistics, 45(2):381–389.

Jane Bromley, Isabelle Guyon, Yann LeCun, EduardSäckinger, and Roopak Shah. 1994. Signature veri-fication using a" siamese" time delay neural network.In Proc. NIPS.

Michael A Carlin, Samuel Thomas, Aren Jansen, andHynek Hermansky. 2011. Rapid evaluation ofspeech representations for spoken term discovery. InProc. Interspeech.

Grzegorz Chrupała and Afra Alishahi. 2019. Corre-lating neural and symbolic representations of lan-guage. In Proceedings of the 57th Annual Meet-ing of the Association for Computational Linguis-tics, pages 2952–2962, Florence, Italy. Associationfor Computational Linguistics.

Grzegorz Chrupała, Bertrand Higy, and Afra Alishahi.2020. Analyzing analytical methods: The case ofphonology in neural models of spoken language. In

Proceedings of the 58th Annual Meeting of the Asso-ciation for Computational Linguistics, pages 4146–4156, Online. Association for Computational Lin-guistics.

Yu-An Chung, Yonatan Belinkov, and James Glass.2021. Similarity analysis of self-supervised speechrepresentations. In ICASSP 2021-2021 IEEE Inter-national Conference on Acoustics, Speech and Sig-nal Processing (ICASSP), pages 3040–3044. IEEE.

Bernard Comrie. 1988. Linguistic typology. AnnualReview of Anthropology, 17:145–159.

Svetlana V Cook, Nick B Pandža, Alia K Lancaster,and Kira Gor. 2016. Fuzzy nonnative phonolexicalrepresentations lead to fuzzy form-to-meaning map-pings. Frontiers in Psychology, 7:1345.

William Croft. 2002. Typology and Universals. Cam-bridge University Press.

Emmanuel Dupoux. 2018. Cognitive science in theera of artificial intelligence: A roadmap for reverse-engineering the infant language-learner. Cognition,173:43–59.

Lieke Gelderloos, Grzegorz Chrupała, and Afra Al-ishahi. 2020. Learning to understand child-directedand adult-directed speech. In Proceedings of the58th Annual Meeting of the Association for Compu-tational Linguistics, pages 1–6, Online. Associationfor Computational Linguistics.

Jelena Golubovic. 2016. Mutual intelligibility in theSlavic language area. Groningen: Center for Lan-guage and Cognition.

Charlotte Gooskens, Vincent J van Heuven, Jelena Gol-ubovic, Anja Schüppert, Femke Swarte, and Ste-fanie Voigt. 2018. Mutual intelligibility betweenclosely related languages in Europe. InternationalJournal of Multilingualism, 15(2):169–193.

Christiaan Jacobs and Herman Kamper. 2021. Multi-lingual Transfer of Acoustic Word Embeddings Im-proves When Training on Languages Related to theTarget Zero-Resource Language. In Proc. Inter-speech, pages 1549–1553.

Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2017.Billion-scale similarity search with GPUs. IEEETransactions on Big Data.

H. Kamper, W. Wang, and Karen Livescu. 2016.Deep convolutional acoustic word embeddings us-ing word-pair side information. In Proc. ICASSP.

Herman Kamper. 2019. Truly unsupervised acousticword embeddings using weak top-down constraintsin encoder-decoder models. In Proc. ICASSP.

Diederik P. Kingma and Jimmy Ba. 2015. Adam: Amethod for stochastic optimization. In Proc. ICLR.

Simon Kornblith, Mohammad Norouzi, Honglak Lee,and Geoffrey Hinton. 2019. Similarity of neuralnetwork representations revisited. In InternationalConference on Machine Learning, pages 3519–3529.PMLR.

Nikolaus Kriegeskorte, Marieke Mur, and Peter A Ban-dettini. 2008. Representational similarity analysis-connecting the branches of systems neuroscience.Frontiers in systems neuroscience, 2:4.

Michael Lepori and R. Thomas McCoy. 2020. PickingBERT’s brain: Probing for linguistic dependenciesin contextualized embeddings using representationalsimilarity analysis. In Proceedings of the 28th Inter-national Conference on Computational Linguistics,pages 3637–3651, Barcelona, Spain (Online). Inter-national Committee on Computational Linguistics.

Keith Levin, Katharine Henry, Aren Jansen, and KarenLivescu. 2013. Fixed-dimensional acoustic embed-dings of variable-length segments in low-resourcesettings. In Proc. ASRU.

James S Magnuson, Heejo You, Sahil Luthra, Mon-ica Li, Hosung Nam, Monty Escabi, Kevin Brown,Paul D Allopenna, Rachel M Theodore, NicholasMonto, et al. 2020. Earshot: A minimal neural net-work model of incremental human speech recogni-tion. Cognitive science, 44(4):e12823.

Yevgen Matusevych, Herman Kamper, and SharonGoldwater. 2020a. Analyzing autoencoder-basedacoustic word embeddings. In Bridging AI and Cog-nitive Science Workshop@ ICLR 2020.

Yevgen Matusevych, Herman Kamper, Thomas Schatz,Naomi Feldman, and Sharon Goldwater. 2021. Aphonetic model of non-native spoken word process-ing. In Proceedings of the 16th Conference of theEuropean Chapter of the Association for Computa-tional Linguistics: Main Volume, pages 1480–1490,Online. Association for Computational Linguistics.

Yevgen Matusevych, Thomas Schatz, Herman Kam-per, Naomi Feldman, and Sharon Goldwater. 2020b.Evaluating computational models of infant phoneticlearning across languages. In Proc. CogSci.

Michael McAuliffe, Michaela Socolof, Sarah Mihuc,Michael Wagner, and Morgan Sonderegger. 2017.Montreal Forced Aligner: Trainable text-speechalignment using Kaldi. In Interspeech.

Adam Paszke, Sam Gross, Francisco Massa, AdamLerer, James Bradbury, Gregory Chanan, TrevorKilleen, Zeming Lin, Natalia Gimelshein, LucaAntiga, et al. 2019. Pytorch: An imperative style,high-performance deep learning library. In Proc.NeuRIPS.

Okko Räsänen, Tasha Nagamine, and Nima Mesgarani.2016. Analyzing distributional learning of phone-mic categories in unsupervised deep neural net-works. Annual Conference of the Cognitive ScienceSociety (CogSci). Cognitive Science Society (U.S.).Conference, 2016:1757–1762.

Håkan Ringbom. 2006. Cross-linguistic similarity inforeign language learning. Multilingual Matters.

Odette Scharenborg, Nikki van der Gouw, Martha Lar-son, and Elena Marchiori. 2019. The representa-tion of speech in deep neural networks. In Interna-tional Conference on Multimedia Modeling, pages194–205. Springer.

Thomas Schatz and Naomi H Feldman. 2018. Neu-ral network vs. hmm speech recognition systems asmodels of human cross-linguistic phonetic percep-tion. In Proceedings of the conference on cognitivecomputational neuroscience.

Steffen Schneider, Alexei Baevski, Ronan Collobert,and Michael Auli. 2019. wav2vec: UnsupervisedPre-Training for Speech Recognition. In Proc. Inter-speech 2019, pages 3465–3469.

Tanja Schultz, Ngoc Thang Vu, and Tim Schlippe.2013. GlobalPhone: A multilingual text and speechdatabase in 20 languages. In Proc. ICASSP.

Shane Settle and Karen Livescu. 2016. Discrimina-tive acoustic word embeddings: Recurrent neuralnetwork-based approaches. In Proc. IEEE SpokenLanguage Technology Workshop (SLT).

Roland Sussex and Paul Cubberley. 2006. The slaviclanguages. Cambridge University Press.

Aäron van den Oord, Yazhe Li, and Oriol Vinyals.2018. Representation learning with contrastive pre-dictive coding. ArXiv, abs/1807.03748.

Joe H Ward. 1963. Hierarchical grouping to optimizean objective function. Journal of the American sta-tistical association, 58(301):236–244.

John Wu, Yonatan Belinkov, Hassan Sajjad, Nadir Dur-rani, Fahim Dalvi, and James Glass. 2020. Similar-ity analysis of contextual word representation mod-els. In Proceedings of the 58th Annual Meetingof the Association for Computational Linguistics,pages 4638–4655.

Ludger Zeevaert. 2007. Receptive multilingualism. JDTen Thije & L. Zeevaert (Eds.), 6:103–137.

Appendices

A Experimental Data Statistics

Table 2 shows a word-level summary statistics ofour experimental data extracted from the Global-Phone speech daabase (GPS).

B Representational Similarity withNon-Linear CKA

We provide the cross-lingual representational simi-larity matrices (xRSMs) constructed by applyingthe non-linear CKA measure with a radial basisfunction (RBF) in Figure 6 below. We observesimilar trends to those presented in Figure 3.

Applying hierarchical clustering on the xRSMsin Figure 6 yields the language clusters shown inFigure 7. We observe that the clusters are iden-tical to those shown in Figure 4, except for theCSE model where Portuguese is no longer in theSlavic group. However, Portuguese remains thenon-Slavic language that is the most similar toSlavic languages.

Lang.Language

group#Trainspkrs

#Trainsamples

#Evalsamples

#Phones/word(mean ± SD)

Word dur. (sec)(mean ± SD)

Token-Typeratio

CZE West Slavic 42 32,063 9,228 6.77 ± 2.28 0.492 ± 0.176 0.175POL West Slavic 42 31,507 9,709 6.77 ± 2.27 0.488 ± 0.175 0.192RUS East Slavic 42 31,892 9,005 7.56 ± 2.77 0.496 ± 0.163 0.223BUL South Slavic 42 31,866 9,063 7.24 ± 2.60 0.510 ± 0.171 0.179POR Romance 42 32,164 9,393 6.95 ± 2.36 0.526 ± 0.190 0.134FRA Romance 42 32,497 9,656 6.24 ± 1.95 0.496 ± 0.163 0.167DEU Germanic 42 32,162 9,865 6.57 ± 2.47 0.435 ± 0.178 0.150

Table 2: Summary statistics of our experimental data.

CZE POL RUS BUL POR FRA DEU

CZE

POL

RUS

BUL

POR

FRA

DEU 0.687 0.680 0.659 0.649 0.660 0.658

0.739 0.735 0.705 0.710 0.737 0.717

0.737 0.722 0.735 0.736 0.726 0.705

0.742 0.731 0.763 0.731 0.706 0.692

0.745 0.735 0.761 0.721 0.695 0.690

0.759 0.721 0.721 0.693 0.700 0.681

0.762 0.747 0.736 0.706 0.717 0.703

CZE POL RUS BUL POR FRA DEU

CZE POL RUS BUL POR FRA DEU

0.2

0.3

0.4

0.5

0.6

0.7

0.8CZE

POL

RUS

BUL

POR

FRA

DEU 0.180 0.195 0.193 0.183 0.177 0.182

0.193 0.220 0.200 0.202 0.197 0.179

0.234 0.266 0.269 0.254 0.234 0.202

0.207 0.241 0.238 0.198 0.190 0.170

0.220 0.257 0.234 0.212 0.189 0.178

0.248 0.269 0.245 0.219 0.205 0.179

0.264 0.244 0.225 0.208 0.203 0.180

CZE POL RUS BUL POR FRA DEU

0.683 0.675 0.658 0.663 0.658 0.661

0.696 0.691 0.683 0.700 0.679 0.663

0.684 0.705 0.721 0.718 0.688 0.649

0.719 0.724 0.745 0.700 0.691 0.638

0.723 0.720 0.748 0.694 0.681 0.624

0.731 0.712 0.722 0.669 0.668 0.622

0.745 0.733 0.734 0.676 0.699 0.651

0.702 0.698 0.676 0.674 0.676 0.678

0.709 0.706 0.694 0.712 0.695 0.674

0.700 0.725 0.734 0.735 0.701 0.659

0.728 0.739 0.755 0.713 0.698 0.645

0.736 0.738 0.762 0.708 0.689 0.636

0.746 0.728 0.736 0.683 0.675 0.630

0.763 0.746 0.747 0.692 0.708 0.663

0.674 0.670 0.649 0.627 0.640 0.645

0.737 0.730 0.705 0.708 0.734 0.710

0.735 0.720 0.730 0.728 0.712 0.694

0.744 0.732 0.759 0.726 0.698 0.689

0.753 0.743 0.765 0.722 0.694 0.691

0.763 0.724 0.717 0.683 0.683 0.67

0.769 0.752 0.737 0.703 0.712 0.696

0.293 0.307 0.306 0.292 0.277 0.294

0.32 0.349 0.327 0.329 0.32 0.292

0.35 0.384 0.386 0.377 0.345 0.292

0.33 0.367 0.375 0.322 0.312 0.282

0.338 0.377 0.373 0.332 0.31 0.275

0.361 0.388 0.362 0.314 0.304 0.262

0.4 0.377 0.365 0.331 0.324 0.281

Figure 6: The cross-lingual representational similarity matrix (xRSM) for each model: PGE (Left), CAE (Middle),and CSE (Right), obtained using the non-linear CKA measure. Each row corresponds to the language of thespoken-word stimuli while each column corresponds to the language of the encoder.

Bulgarian

Portuguese

0.0

0.2

0.40.0

0.2

0.4

CzechPolishRussianBulgarianPortugueseFrench

German

CzechPolishRussianBulgarianPortugueseFrench

German

Russian

0.0

0.5

1.0

Polish

French

German

Czech

Slavic Romance Germanic

Figure 7: Hierarchical clustering analysis on the cross-lingual representational similarity matrices (using non-linearCKA) of the three models: PGE (Left), CAE (Middle), and CSE (Right).