Meta-Learning for Domain Generalization in Semantic Parsing

Bailin Wang, Mirella Lapata and Ivan TitovInstitute for Language, Cognition and Computation

School of Informatics, University of Edinburghbailin.wang@ed.ac.uk, {mlap, ititov}@inf.ed.ac.uk

Abstract

The importance of building semantic parserswhich can be applied to new domains andgenerate programs unseen at training has longbeen acknowledged, and datasets testing out-of-domain performance are becoming increas-ingly available. However, little or no atten-tion has been devoted to learning algorithmsor objectives which promote domain general-ization, with virtually all existing approachesrelying on standard supervised learning. Inthis work, we use a meta-learning frameworkwhich targets zero-shot domain generalizationfor semantic parsing. We apply a model-agnostic training algorithm that simulates zero-shot parsing by constructing virtual train andtest sets from disjoint domains. The learningobjective capitalizes on the intuition that gra-dient steps that improve source-domain perfor-mance should also improve target-domain per-formance, thus encouraging a parser to gen-eralize to unseen target domains. Experimen-tal results on the (English) Spider and ChineseSpider datasets show that the meta-learning ob-jective significantly boosts the performance ofa baseline parser.

1 Introduction

Semantic parsing is the task of mapping naturallanguage (NL) utterances to executable programs.While there has been much progress in this area,earlier work has primarily focused on evaluatingparsers in-domain (e.g., tables or databases) andoften with the same programs as those providedin training (Finegan-Dollak et al., 2018). A muchmore challenging goal is achieving domain gener-alization, i.e., building parsers which can be suc-cessfully applied to new domains and are ableto produce complex unseen programs. Achiev-ing this generalization goal would, in principle,let users query arbitrary (semi-)structured data onthe Web and reduce the annotation effort requiredto build multi-domain NL interfaces (e.g., Apple

database: farm

Please show the different statuses of cities and the averagepopulation of cities with each status.

SELECT Status ,  avg(Population) FROM City GROUP BY Status

database: concert singer

Show all countries and the number of singers in each country.

SELECT Country ,  count(*) FROM Singer GROUP BY Country

Test

Train

Figure 1: Zero-shot semantic parsing: at training time,a parser observes instances for the database concertsinger. At test time, it needs to generate SQL for ques-tions pertaining to the unseen database farm.

Siri or Amazon Alexa). Current parsers strug-gle in this setting; for example, we show in Sec-tion 5 that a modern parser trained on the chal-lenging Spider dataset (Yu et al., 2018b) has a gapof more than 25% in accuracy between in- andout-of-domain performance. While the importanceof domain generalization has been previously ac-knowledged (Cai and Yates, 2013; Chang et al.,2020), and datasets targetting zero-shot (or out-of-domain) performance are becoming increasinglyavailable (Pasupat and Liang, 2015; Wang et al.,2015; Zhong et al., 2017; Yu et al., 2018b), little orno attention has been devoted to studying learningalgorithms or objectives which promote domaingeneralization.

Conventional supervised learning simply as-sumes that source- and target-domain data orig-inate from the same distribution, and as a resultstruggles to capture this notion of domain gener-alization for zero-shot semantic parsing. Previousapproaches (Guo et al., 2019b; Wang et al., 2020;Herzig and Berant, 2018) facilitate domain gener-alization by incorporating inductive biases in themodel, e.g., designing linking features or functionswhich should be invariant under domain shifts. Inthis work, we take a different direction and improve

arX

iv:2

010.

1198

8v2

[cs

.CL

] 1

2 A

pr 2

021

the domain generalization of a semantic parser bymodifying the learning algorithm and the objec-tive. We draw inspiration from meta-learning (Finnet al., 2017; Li et al., 2018a) and use an objec-tive that optimizes for domain generalization. Thatis, we consider a set of tasks, where each task is azero-shot semantic parsing task with its own sourceand target domains. By optimizing towards bettertarget-domain performance on each task, we en-courage a parser to extrapolate from source-domaindata and achieve better domain generalization.

Specifically, we focus on text-to-SQL parsingwhere we aim at translating NL questions toSQL queries and conduct evaluations on unseendatabases. Consider the example in Figure 1, aparser needs to process questions to a new databaseat test time. To simulate this scenario during train-ing, we synthesize a set of virtual zero-shot parsingtasks by sampling disjoint source and target do-mains1 for each task from the training domains.The objective we require is that gradient steps com-puted towards better source-domain performancewould also be beneficial to target-domain perfor-mance. One can think of the objective as consistingof both the loss on the source domain (as in stan-dard supervised learning) and a regularizer, equalto the dot product between gradients computedon source- and target-domain data. Maximizingthis regularizer favours finding model parametersthat work not only on the source domain but alsogeneralize to target-domain data. The objective isborrowed from Li et al. (2018a) who adapt a Model-Agnostic Meta-Learning (MAML; Finn et al. 2017)technique for domain generalization in computervision. In this work, we study the effectiveness ofthis objective in the context of semantic parsing.This objective is model-agnostic, simple to incorpo-rate and does not require any changes in the parsingmodel itself. Moreover, it does not introduce newparameters for meta-learning.

Our contributions can be summarized as follows.

• We handle zero-shot semantic parsing by ap-plying a meta-learning objective that directlyoptimizes for domain generalization.

• We propose an approximation of the meta-learning objective that is more efficient andallows more scalable training.

• We perform experiments on two text-to-SQLbenchmarks: Spider and Chinese Spider. Our

1We use the terms domain and database interchangeably.

new training objectives obtain significant im-provements in accuracy over a baseline parsertrained with conventional supervised learning.

• We show that even when parsers are aug-mented with pre-trained models, e.g., BERT,our method can still effectively improve do-main generalization in terms of accuracy.

Our code is available at https://github.com/berlino/tensor2struct-public.

2 Related Work

Zero-Shot Semantic Parsing Developing aparser that can generalize to unseen domains hasattracted increased attention in recent years. Pre-vious work has mainly focused on the sub-task ofschema linking as a means of promoting domaingeneralization. In schema linking, we need to rec-ognize which columns or tables are mentioned ina question. For example, a parser would decideto select the column Status because of the wordstatuses in Figure 1. However, in the setting ofzero-shot parsing, columns or tables might be men-tioned in a question without ever being observedduring training.

One line of work tries to incorporate inductivebiases, e.g., domain-invariant n-gram matchingfeatures (Guo et al., 2019b; Wang et al., 2020),cross-domain alignment functions (Herzig and Be-rant, 2018), or auxiliary linking tasks (Chang et al.,2020) to improve schema linking. However, in thecross-lingual setting of Chinese Spider (Min et al.,2019), where questions and schemas are not in thesame language, it is not obvious how to designsuch inductive biases like n-gram matching fea-tures. Another line of work relies on large-scale un-supervised pre-training on massive tables (Herziget al., 2020; Yin et al., 2020) to obtain better repre-sentations for both questions and database schemas.Our work is orthogonal to these approaches andcan be easily coupled with them. As an example,we show in Section 5 that our training procedurecan improve the performance of a parser already en-hanced with n-gram matching features (Guo et al.,2019b; Wang et al., 2020).

Our work is similar in spirit to Givoli and Re-ichart (2019), who also attempts to simulate sourceand target domains during learning. However, theiroptimization updates on virtual source and targetdomains are loosely connected by a two-step train-ing procedure where a parser is first pre-trained on

virtual source domains and then fine-tuned on vir-tual target domains. As we will show in Section 3,our training procedure does not fine-tune on virtualtarget domains but rather, uses them to evaluate agradient step (for every batch) on source domains.This is better aligned with what is expected of theparser at test time: there will be no fine-tuning onreal target domains at test time so there should notbe any fine-tuning on simulated ones at train timeeither. Moreover, Givoli and Reichart (2019) treatthe division of training domains to virtual train andtest domains as a hyper-parameter, which is possi-ble for a handful of domains, but problematic whendealing with hundreds of domains as is the case fortext-to-SQL parsing.

Meta-Learning for NLP Meta-learning hasbeen receiving soaring interest in the machine learn-ing community. Unlike conventional supervisedlearning, meta-learning operates on tasks, insteadof data points. Most previous work (Vinyals et al.,2016; Ravi and Larochelle, 2017; Finn et al., 2017)has focused on few-shot learning where meta-learning helps address the problem of learning tolearn fast for adaptation to a new task or domain.Applications of meta-learning in NLP are cast in asimilar vein and include machine translation (Guet al., 2018) and relation classification (Obamuyideand Vlachos, 2019). The meta-learning frameworkhowever is more general, with the algorithms orunderlying ideas applied, e.g., to continual learn-ing (Gupta et al., 2020), semi-supervised learn-ing (Ren et al., 2018), multi-task learning (Yuet al., 2020) and, as in our case, domain generaliza-tion (Li et al., 2018a).

Very recently, there have been some applicationsof MAML to semantic parsing tasks (Huang et al.,2018; Guo et al., 2019a; Sun et al., 2019). Theseapproaches simulate few-shot learning scenarios intraining by constructing a pseudo-task for each ex-ample. Given an example, similar examples are re-trieved from the original training set. MAML thenencourages strong performance on the retrievedexamples after an update on the original example,simulating test-time fine-tuning. Lee et al. (2019)use matching networks (Vinyals et al., 2016) to en-able one-shot text-to-SQL parsing where tasks formeta-learning are defined by SQL templates, i.e., aparser is expected to generalize to a new SQL tem-plate with one example. In contrast, the tasks weconstruct for meta-learning aim to encourage gener-alization across domains, instead of adaptation to

a new task with one (or few) examples. One cleardifference lies in how meta-train and meta-test setsare constructed. In previous work (e.g., Huang et al.2018), these come from the same domain whereaswe simulate domain shift and sample different setsof domains for meta-train and meta-test.

Domain Generalization Although the notion ofdomain generalization has been less explored insemantic parsing, it has been studied in other ar-eas such as computer vision (Ghifary et al., 2015;Zaheer et al., 2017; Li et al., 2018b). Recentwork (Li et al., 2018a; Balaji et al., 2018) em-ployed optimization-based meta-learning to handledomain shift issues in domain generalization. Weemploy the meta-learning objective originally pro-posed in Li et al. (2018a), where they adapt MAMLto encourage generalization in unseen domains (ofimages). Based on this objective, we propose acheap alternative that only requires first-order gra-dients, thus alleviating the overhead of computingsecond-order derivatives required by MAML.

3 Meta-Learning for DomainGeneralization

We first formally define the problem of domaingeneralization in the context of zero-shot text-to-SQL parsing. Then, we introduce DG-MAML,a training algorithm that helps a parser achievebetter domain generalization. Finally, we proposea computationally cheap approximation thereof.

3.1 Problem DefinitionDomain Generalization Given a natural lan-guage question Q in the context of a relationaldatabase D, we aim at generating the correspond-ing SQL P . In the setting of zero-shot parsing,we have a set of source domains Ds where labeledquestion-SQL pairs are available. We aim at de-veloping a parser that can perform well on a set ofunseen target domainsDt. We refer to this problemas domain generalization.

Parsing Model We assume a parameterized pars-ing model that specifies a predictive distributionpθ(P |Q,D) over all possible SQLs. For domaingeneralization, a parsing model needs to properlycondition on its input of questions and databasessuch that it can generalize well to unseen domains.

Conventional Supervised Learning Assumingthat question-SQL pairs from source domains andtarget domains are sampled i.i.d from the same

distribution, the typical training objective of super-vised learning is to minimize the loss function ofthe negative log-likelihood of the gold SQL query:

LB(θ) = −1

N

N∑i=1

log pθ(P |Q,D) (1)

where N is the size of mini-batch B. Since a mini-batch is randomly sampled from all training sourcedomainsDs, it usually contains question-SQL pairsfrom a mixture of different domains.

Distribution of Tasks Instead of treating seman-tic parsing as a conventional supervised learningproblem, we take an alternative view based on meta-learning. Basically, wea re interested in a learningalgorithm that can benefit from a distribution ofchoices of source and target domains, denoted byp(τ), where τ refers to an instance of a zero-shotsemantic parsing task that has its own source andtarget domains.

In practice, we usually have a fixed set of train-ing source domains Ds. We construct a set of vir-tual tasks τ by randomly sampling disjoint sourceand target domains from the training domains. In-tuitively, we assume that divergences between thetest and training domains during the learning phaseare representative of differences between trainingand actual test domains. This is still an assumption,but considerably weaker compared to the i.i.d. as-sumption used in conventional supervised learning.Next, we introduce the training algorithm calledDG-MAML motivated by this assumption.

3.2 Learning to Generalize with DG-MAML

Having simulated source and target domains foreach virtual task, we now need a training algo-rithm that encourages generalization to unseen tar-get domains in each task. For this, we turn tooptimization-based meta-learning algorithms (Finnet al., 2017; Nichol et al., 2018; Li et al., 2018a)and apply DG-MAML (Domain Generalizationwith Model-Agnostic Meta-Learning), a variantof MAML (Finn et al., 2017) for this purpose. Intu-itively, DG-MAML encourages the optimization inthe source domain to have a positive effect on thetarget domain as well.

During each learning episode of DG-MAML,we randomly sample a task τ which has its ownsource domain Dτs and target domain Dτt . For thesake of efficiency, we randomly sample mini-batchquestion-SQL pairs Bs and Bt from Dτs and Dτt ,

respectively, for learning in each task. DG-MAMLconducts optimization in two steps, namely meta-train and meta-test.

Meta-Train DG-MAML first optimizes param-eters towards better performance in the virtualsource domain Dτs by taking one step of stochasticgradient descent (SGD) from the loss under Bs.

θ′ ← θ − α∇θLBs(θ) (2)

where α is a scalar denoting the learning rate ofmeta-train. This step resembles conventional su-pervised learning where we use stochastic gradientdescent to optimize the parameters.

Meta-Test We then evaluate the resulting param-eter θ′ in the virtual target domainDt by computingthe loss under Bt, which is denoted as LBt(θ′).

Our final objective for a task τ is to minimizethe joint loss on Ds and Dt:

Lτ (θ) = LBs(θ) + LBt(θ′)= LBs(θ) + LBt(θ − α∇θLBs(θ))

(3)

where we optimize towards the better source andtarget domain performance simultaneously. Intu-itively, the objective requires that the gradient stepconducted in the source domains in Equation (2) bebeneficial to the performance of the target domainas well. In comparison, conventional supervisedlearning, whose objective would be equivalent toLBs(θ) + LBt(θ), does not pose any constraint onthe gradient updates. As we will elaborate shortly,DG-MAML can be viewed as a regularization ofgradient updates in addition to the objective of con-ventional supervised learning.

We summarize our DG-MAML training processin Algorithm 1. Basically, it requires two stepsof gradient update (Step 5 and Step 7). Note thatθ′ is a function of θ after the meta-train update.Hence, optimizingLτ (θ) with respect to θ involvesoptimizing the gradient update in Equation (2) aswell. That is, when we update the parameters θin the final update of Step 7, the gradients needto back-propagate though the meta-train updatesin Step 5. The update function in Step 7 could bebased on any gradient descent algorithm. In thiswork we use Adam (Kingma and Ba, 2015).

Comment Note that DG-MAML is differentfrom MAML (Finn et al., 2017) which is typicallyused in the context of few-shot learning. In ourcase, it encourages domain generalization duringtraining, and does not require an adaptation phase.

Algorithm 1 DG-MAML Training Algorithm

Require: Training databases DRequire: Learning rate α

1: for step← 1 to T do2: Sample a task τ of (Dτs ,Dτt ) from D3: Sample mini-batch Bτs from Dτs4: Sample mini-batch Bτt from Dτt5: Meta-train update:

θ′ ← θ − α∇θLBτs (θ)6: Compute meta-test objective:

Lτ (θ) = LBs(θ) + LBt(θ′)7: Final Update:

θ ← Update(θ,∇θLτ (θ))8: end for

3.3 Analysis of DG-MAMLTo give an intuition of the objective in Equation (3),we follow previous work (Nichol et al., 2018; Liet al., 2018a) and use the first-order Taylor seriesexpansion to approximate it:

Lτ (θ) =LBs(θ) + LBt(θ′)=LBs(θ) + LBt(θ − α∇θLBs(θ))≈LBs(θ) + LBt(θ)−α(∇θLBs(θ) · ∇θLBt(θ))

(4)

where in the last step we expand the function LBsat θ. The approximated objective sheds light onwhat DG-MAML optimizes. In addition to mini-mizing the losses from both source and target do-mains, which are LBs(θ) + LBt(θ), DG-MAMLfurther tries to maximize ∇θLBs(θ) · ∇θLBt(θ),the dot product between the gradients of sourceand target domain. That is, it encourages gradientsto generalize between source and target domainwithin each task τ .

3.4 First-Order ApproximationThe final update in Step 7 of Algorithm 1 requiressecond-order derivatives, which may be problem-atic, inefficient or non-stable with certain classesof models (Mensch and Blondel, 2018). Hence,we propose an approximation that only requirescomputing first-order derivatives.

First, the gradient of the objective in Equation (3)can be computed as:

∇θLτ (θ) =∇θθ′∇θ′LBt(θ′) +∇θLBs(θ)

=(I − α∇2

θLBs(θ))∇θ′LBt(θ′)

+∇θLBs(θ)(5)

where I is an identity matrix and ∇2θLBs(θ) is the

Hessian of LBs at θ. We consider the alternativeof ignoring this second-order term and simply as-sume that ∇θθ

′ = I . In this variant, we simplycombine gradients from source and target domains.We show in the Appendix that this objective canstill be viewed as maximizing the dot product ofgradients from source and target domain.

The resulting first-order training objective,which we refer to as DG-FMAML, is inspiredby Reptile, a first-order meta-learning algo-rithm (Nichol et al., 2018) for few-shot learning. Atwo-step Reptile would compute SGD on the samebatch twice while DG-FMAML computes SGD ontwo different batches, Bs and Bt, once. To put itdifferently, DG-FMAML tries to encourage cross-domain generalization while Reptile encouragesin-domain generalization.

4 Semantic Parser

In general, DG-MAML is model-agnostic and canbe coupled with any semantic parser to improveits domain generalization. In this work, we usea base parser that is based on RAT-SQL (Wanget al., 2020), which currently achieves state-of-the-art performance on Spider.2

Formally, RAT-SQL takes as input question Qand schema S of its corresponding database. Thenit produces a program which is represented as an ab-stract syntax tree T in the context-free grammar ofSQL (Yin and Neubig, 2018). RAT-SQL adopts theencoder-decoder framework for text-to-SQL pars-ing. It has three components: an initial encoder,a transformer-based encoder and an LSTM-baseddecoder. The initial encoder provides initial repre-sentations, denoted as Qinit and Sinit for the ques-tion and the schema, respectively. A relation-awaretransformer (RAT) module then takes the initial rep-resentations and further computes context-awarerepresentations Qenc and Senc for the question andthe schema, respectively. Finally, a decoder gener-ates a sequence of production rules that constitutethe abstract syntax tree T based on Qenc and Senc.To obtain Qinit and Sinit, the initial encoder couldeither be 1) LSTMs (Hochreiter and Schmidhuber,1997) on top of pre-trained word embeddings, likeGloVe (Pennington et al., 2014), or 2) pre-trainedcontextual embeddings like BERT (Devlin et al.,

2We re-implemented RAT-SQL, and added a componentfor value prediction so that our base parsers can be evaluatedby execution accuracy.

2019). In our work, we will test the effectivenessof our method for both variants.

As shown in Wang et al. (2020), the encodingsQenc and Senc, which are the output of the RATmodule, heavily rely on schema-linking features.These features are extracted from a heuristic func-tion that links question words to columns and tablesbased on n-gram matching, and they are readilyavailable in the conventional mono-lingual settingof the Spider dataset. However, we hypothesizethat the parser’s over-reliance on these features isspecific to Spider, where annotators were shownthe database schema and asked to formulate queries.As a result, they were prone to re-using terms fromthe schema verbatim in their questions. This wouldnot be the case in a real-world application whereusers are unfamiliar with the structure of the un-derlying database and free to use arbitrary termswhich would not necessarily match column or tablenames (Suhr et al., 2020). Hence, we will also eval-uate our parser in the cross-lingual setting whereQ and S are not in the same language, and suchfeatures would not be available.

5 Experiments

To evaluate DG-MAML, we integrate it with a baseparser and test it on zero-shot text-to-SQL tasks.By designing an in-domain benchmark, we alsoshow that the out-of-domain improvement doesnot come at the cost of in-domain performance.We also present some analysis to show how DG-MAML affects domain generalization.

5.1 Datasets and Metrics

We evaluate DG-MAML on two zero-shot text-to-SQL benchmarks, namely, (English) Spider (Yuet al., 2018b) and Chinese Spider (Min et al., 2019).Chinese Spider is a Chinese version of Spider thattranslates all NL questions from English to Chineseand keeps the original English database. It intro-duces the additional challenge of encoding cross-lingual correspondences between Chinese and En-glish.3 In both datasets, we report exact set matchaccuracy, following Yu et al. (2018b). We alsoreport execution accuracy in the Spider dataset.

5.2 Baselines

Two kinds of features are widely used in recentsemantic parsers to boost domain generalization:

3Please see the appendix for details of the datasets.

schema-linking features (as mentioned in Sec-tion 4) and pre-trained emebddings such as BERT.To show that our method can still achieve additionalimprovements, we compare with strong baselinesthat are integrated with schema-linking featuresand pre-trained embeddings. In the analysis (Sec-tion 5.6), we will also show the effect of our methodwhen both features are absent in the base parsers.

5.3 Implementation and HyperparametersOur base parser is based on RAT-SQL (Wang et al.,2020), which is implemented in PyTorch (Paszkeet al., 2019). For English questions and schemas,we use GloVe (Pennington et al., 2014) and BERT-base (Devlin et al., 2019) as the pre-trained em-beddings for encoding. For Chinese questions, weuse Tencent embeddings (Song et al., 2018) andMultilingual-BERT (Devlin et al., 2019). In allexperiments, we use a batch size of Bs = Bt = 12and train for up to 20,000 steps. See the Appendixfor details on other hyperparameters.

5.4 Main ResultsOur main results on Spider and Chinese Spider arelisted in Table 1 and 2, respectively.

Non-BERT Models DG-MAML boosts the per-formance of non-BERT base parsers on Spider andChinese Spider by 2.1% and 4.5% respectively,showing its effectiveness in promoting domain gen-eralization. In comparison, the performance marginfor DG-MAML is more significant in the cross-lingual setting of Chinese Spider. This is presum-ably due to the fact that heuristic schema-linkingfeatures, which help promote domain generaliza-tion for Spider, are not applicable in Chinese Spi-der. We will present more analysis on this in Sec-tion 5.6.

BERT Models Most importantly, improvementson both datasets are not cancelled out when thebase parsers are augmented with pre-trained repre-sentations. On Spider, the improvements broughtby DG-MAML remain roughly the same when thebase parser is integrated with BERT-base. As aresult, our base parser augmented with BERT-baseand DG-MAML achieves the best execution accu-racy compared with previous models. On ChineseSpider, DG-MAML helps the base parser with mul-tilingual BERT achieve a substantial improvement.Overall, DG-MAML consistently boosts the per-formance of the base parser, and is complementaryto using pre-trained representations.

Model Dev Test

Set Match AccuracySyntaxSQLNet (Yu et al., 2018a) 18.9 19.7Global-GNN (Bogin et al., 2019) 52.7 47.4IRNet (Guo et al., 2019b) 55.4 48.5RAT-SQL (Wang et al., 2020) 62.7 57.2Our Models

Base Parser 56.4 -Base Parser + DG-MAML 58.5 -

With BERT-base:SyntaxSQLNet + BERT-base (Guo et al., 2019b) 25.0 25.4IRNet + BERT-base (Guo et al., 2019b) 61.9 54.7BRIDGE + BERT-base (Lin et al., 2020) 65.5 58.2RAT-SQL + BERT-base 66.0♦ -Our Models

Base Parser + BERT-base 66.8 63.3Base Parser + BERT-base + DG-MAML 68.9 65.2

With BERT-large:RYANSQL + BERT-large (Choi et al., 2020) 70.6 60.6RAT-SQL + BERT-large (Wang et al., 2020) 69.7 65.6Execution AccuracyGAZP + Distil-BERT (Zhong et al., 2020) 59.2 53.5BRIDGE + BERT-base (Lin et al., 2020) 65.3 59.9Our Models

Base Parser + BERT-base 66.8 64.1Base Parser + BERT-base + DG-MAML 69.3 66.1

Table 1: Accuracy (%) on the development and testsets of Spider. The first half shows set match accuracyfor both non-BERT and BERT models; the second halfshows execution accuracy of BERT models. Due to thenumber of model submissions constraint enforced bythe Spider team, we only evaluate our BERT modelson the test set for now. The number with ♦ is producedby running the code of Wang et al. (2020).

5.5 In-Domain vs. Out-of-Domain

To confirm that the base parser struggles when ap-plied out-of-domain, we construct an in-domain set-ting and measure the gap in performance. This set-ting also helps us address a natural question: doesusing DG-MAML hurt in-domain performance?This would not have been surprising as the parseris explicitly optimized towards better performanceon unseen target domains.

To answer these questions, we create a new splitof Spider. Specifically, for each database from thetraining and development set of Spider, we include80% of its question-SQL pairs in the new trainingset and assign the remaining 20% to the new testset. As a result, the new split consists of 7702training examples and 1991 test examples. Whenusing this split, the parser is tested on databasesthat all have been seen during training. We evaluatethe non-BERT parsers with the same metric of setmatch for evaluation.

Does the parser struggle out-of-domain? Asin-domain and out-of-domain setting have differ-

Model Dev Test

SyntaxSQLNet (Yu et al., 2018a) 16.4 13.3Our Models

Base Parser 31.0 23.0Base Parser + DG-MAML 35.5 26.8

With Multilingual BERT (M-BERT):RAT-SQL + M-BERT (Anonymous) 41.4 37.3RYANSQL + M-BERT (Choi et al., 2020) 41.3 34.7Our Models

Base Parser + M-BERT 47.0 44.3Base Parser + M-BERT + DG-MAML 50.1 46.9

Table 2: Set match accuracy (%) on the developmentand test sets of Chinese Spider.

ent splits, and thus do not use the same test set,the direct comparison between them only servesas a proxy to illustrate the effect of domain shift.We show that, despite the original split of out-of-domain setting containing a larger number oftraining examples (8659 vs 7702), the base parsertested in-domain achieves a much better perfor-mance (78.2%) than its counterpart tested out-of-domain (56.4%). This suggests that the domainshift genuinely hurts the base parser.

Does DG-MAML hurt in-domain perfor-mance? We study DG-MAML in the in-domainsetting to see if it hurts in-domain performance.Somewhat surprisingly, we instead observe amodest improvement (+1.1%) over the base parser.This suggests that DG-MAML, despite optimizingthe model towards domain generalization, captures,to a certain degree, a more general notion ofgeneralization or robustness, which appearsbeneficial even in the in-domain setting.

5.6 Additional Experiments and Analysis

We first discuss additional experiments on linkingfeatures and DG-FMAML, and then present furtheranalysis probing how DG-MAML works. As thetest sets for both datasets are not publicly available,we will use the development sets.

Linking Features As mentioned in Section 2,previous work addressed domain generalizationby focusing on the sub-task of schema linking. ForSpider, where questions and schemas are both inEnglish, Wang et al. (2020) leverage n-gram match-ing features which improve schema linking andsignificantly boost parsing performance. However,in Chinese Spider, it is not easy and obvious howto design such linking heuristics. Moreover, aspointed out by Suhr et al. (2020), the assumption

Model Dev (%)

SpiderBase Parser 55.6 ± 0.5

+ DG-FMAML 56.8 ± 1.2+ DG-MAML 58.0 ± 0.8

Base Parser without Features 38.2 ± 1.0+ DG-FMAML 41.8 ± 1.5+ DG-MAML 43.5 ± 0.9

Chinese SpiderBase Parser 29.7 ± 1.1

+ DG-FMAML 32.5 ± 1.3+ DG-MAML 34.3 ± 0.9

Table 3: Accuracy (and ±95% confidence interval) onthe development sets of Spider and Chinese Spider.

that columns/tables are explicitly mentioned is notgeneral enough, implying that exploiting matchingfeatures would not be a good general solution todomain generalization. Hence, we would like tosee whether DG-MAML can be beneficial whenthose features are not present.

Specifically, we consider a variant of the baseparser that does not use this feature, and train itwith conventional supervised learning and withDG-MAML for Spider. As shown4 in Table 3,we confirm that those features have a big impact onthe base parser. More importantly, in the absenceof those features, DG-MAML boosts the perfor-mance of the base parser by a larger margin. Thisis consistent with the observation that DG-MAMLis more beneficial for Chinese Spider than Spider,in the sense that the parser would need to rely moreon DG-MAML when these heuristics are not inte-grated or not available for domain generalization.

Effect of DG-FMAML We investigate the effectof the first-order approximation in DG-FMAML tosee if it would provide a reasonable performancecompared with DG-MAML. We evaluate it onthe development sets of the two datasets, see Ta-ble 3. DG-FMAML consistently boosts the perfor-mance of the base parser, although it lags behindDG-MAML. For a fair comparison, we use thesame batch size for DG-MAML and DG-FMAML.However, because DG-FMAML uses less memory,it could potentially benefit from a larger batch size.In practice, DG-FMAML is twice faster to trainthan DG-MAML, see Appendix for details.

4Some results in Table 3 differ from Table 1. The formerreports dev set performance over three runs, while the lattershows the best model, selected based on dev set performance.

Model Precision Recall F1

SpiderBase Parser 70.0 70.4 70.2Base Parser + DG-MAML 73.8 70.6 72.1

Chinese SpiderBase Parser 61.5 60.4 61.0Base Parser + DG-MAML 66.8 61.2 63.9

Table 4: Performance (%) of column prediction on thedevelopment sets of Spider and Chinese Spider.

Probing Domain Generalization Schema link-ing has been the focus of previous work on zero-shot semantic parsing. We take the opposite direc-tion and use this task to probe the parser to see if it,at least to a certain degree, achieves domain gen-eralization due to improving schema linking. Wehypothesize that improving linking is the mecha-nism which prevents the parser from being trappedin overfitting the source domains.

We propose to use ‘relevant column recognition’as a probing task. Specifically, relevant columnsrefer to the columns that are mentioned in SQLqueries. For example, the SQL query “Select Sta-tus, avg(Population) From City Groupby Status” inFigure 1 contains two relevant columns: ‘Status’and ‘Population’. We formalize this task as a bi-nary classification problem. Given a NL questionand a column from the corresponding database,a classifier should predict whether the column ismentioned in the gold SQL query. We hypothesizethat representations from the DG-MAML parserwill be more predictive of relevance than those ofthe baseline, and the probing classifier will detectthis difference in the quality of the representations.

We first obtain the representations for NL ques-tions and schemas from the parsers and keep themfixed. The binary classifier is then trained basedonly on these representations. For classifier train-ing we use the same split as the Spider dataset,i.e., the classifier is evaluated on unseen databases.Details of the classifier are provided in the Ap-pendix. The results are shown in Table 4. Theclassifier trained on the parser with DG-MAMLachieves better performance. This confirms ourhypothesis that using DG-MAML makes the parserhave better encodings of NL questions and databaseschemas and that this is one of the mechanisms theparsing model uses to ensure generalization.

6 Conclusions

The task of zero-shot semantic parsing has beengaining momentum in recent years. However, previ-

ous work has not proposed algorithms or objectivesthat explicitly promote domain generalization. Werely on the meta-learning framework to encouragedomain generalization. Instead of learning from in-dividual data points, DG-MAML learns from a setof virtual zero-shot parsing tasks. By optimizingtowards better target-domain performance in eachsimulated task, DG-MAML encourages the parserto generalize better to unseen domains.

We conduct experiments on two zero-shot text-to-SQL parsing datasets. In both cases, using DG-MAML leads to a substantial boost in performance.Furthermore, we show that the faster first-orderapproximation DG-FMAML can also help a parserachieve better domain generalization.

Acknowledgements

We thank Bo Pang, Tao Yu, Qingkai Min and Yue-feng Shi for their help with the evaluation. Wewould like to thank the anonymous reviewers fortheir valuable comments, and Jackie Cheung forpointing out a typo in Eq 4 in the draft version. Wegratefully acknowledge the support of the EuropeanResearch Council (Titov: ERC StG BroadSem678254; Lapata: ERC CoG TransModal 681760)and the Dutch National Science Foundation (NWOVIDI 639.022.518).

ReferencesYogesh Balaji, Swami Sankaranarayanan, and Rama

Chellappa. 2018. Metareg: Towards domain gen-eralization using meta-regularization. In Advancesin Neural Information Processing Systems 31: An-nual Conference on Neural Information ProcessingSystems 2018, NeurIPS 2018, December 3-8, 2018,Montréal, Canada, pages 1006–1016.

Ben Bogin, Matt Gardner, and Jonathan Berant. 2019.Global reasoning over database structures for text-to-SQL parsing. In Proceedings of the 2019 Con-ference on Empirical Methods in Natural LanguageProcessing and the 9th International Joint Confer-ence on Natural Language Processing (EMNLP-IJCNLP), pages 3659–3664, Hong Kong, China. As-sociation for Computational Linguistics.

Qingqing Cai and Alexander Yates. 2013. Large-scalesemantic parsing via schema matching and lexiconextension. In Proceedings of the 51st Annual Meet-ing of the Association for Computational Linguis-tics (Volume 1: Long Papers), pages 423–433, Sofia,Bulgaria. Association for Computational Linguis-tics.

Shuaichen Chang, Pengfei Liu, Yun Tang, Jing Huang,Xiaodong He, and Bowen Zhou. 2020. Zero-shot

text-to-sql learning with auxiliary task. In TheThirty-Fourth AAAI Conference on Artificial Intelli-gence, AAAI 2020, The Thirty-Second Innovative Ap-plications of Artificial Intelligence Conference, IAAI2020, The Tenth AAAI Symposium on EducationalAdvances in Artificial Intelligence, EAAI 2020, NewYork, NY, USA, February 7-12, 2020, pages 7488–7495. AAAI Press.

DongHyun Choi, Myeong Cheol Shin, EungGyun Kim,and Dong Ryeol Shin. 2020. Ryansql: Recursivelyapplying sketch-based slot fillings for complex text-to-sql in cross-domain databases. arXiv preprintarXiv:2004.03125.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2019. BERT: Pre-training ofdeep bidirectional transformers for language under-standing. In Proceedings of the 2019 Conferenceof the North American Chapter of the Associationfor Computational Linguistics: Human LanguageTechnologies, Volume 1 (Long and Short Papers),pages 4171–4186, Minneapolis, Minnesota. Associ-ation for Computational Linguistics.

Catherine Finegan-Dollak, Jonathan K. Kummerfeld,Li Zhang, Karthik Ramanathan, Sesh Sadasivam,Rui Zhang, and Dragomir Radev. 2018. Improvingtext-to-SQL evaluation methodology. In Proceed-ings of the 56th Annual Meeting of the Associationfor Computational Linguistics (Volume 1: Long Pa-pers), pages 351–360, Melbourne, Australia. Asso-ciation for Computational Linguistics.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017.Model-agnostic meta-learning for fast adaptation ofdeep networks. In Proceedings of the 34th Inter-national Conference on Machine Learning, ICML2017, Sydney, NSW, Australia, 6-11 August 2017,volume 70 of Proceedings of Machine Learning Re-search, pages 1126–1135. PMLR.

Muhammad Ghifary, W. Bastiaan Kleijn, MengjieZhang, and David Balduzzi. 2015. Domain general-ization for object recognition with multi-task autoen-coders. In 2015 IEEE International Conference onComputer Vision, ICCV 2015, Santiago, Chile, De-cember 7-13, 2015, pages 2551–2559. IEEE Com-puter Society.

Ofer Givoli and Roi Reichart. 2019. Zero-shot seman-tic parsing for instructions. In Proceedings of the57th Annual Meeting of the Association for Com-putational Linguistics, pages 4454–4464, Florence,Italy. Association for Computational Linguistics.

Jiatao Gu, Yong Wang, Yun Chen, Victor O. K. Li,and Kyunghyun Cho. 2018. Meta-learning for low-resource neural machine translation. In Proceed-ings of the 2018 Conference on Empirical Methodsin Natural Language Processing, pages 3622–3631,Brussels, Belgium. Association for ComputationalLinguistics.

Daya Guo, Duyu Tang, Nan Duan, Ming Zhou, andJian Yin. 2019a. Coupling retrieval and meta-learning for context-dependent semantic parsing. InProceedings of the 57th Annual Meeting of the As-sociation for Computational Linguistics, pages 855–866, Florence, Italy. Association for ComputationalLinguistics.

Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao,Jian-Guang Lou, Ting Liu, and Dongmei Zhang.2019b. Towards complex text-to-SQL in cross-domain database with intermediate representation.In Proceedings of the 57th Annual Meeting of theAssociation for Computational Linguistics, pages4524–4535, Florence, Italy. Association for Compu-tational Linguistics.

Gunshi Gupta, Karmesh Yadav, and Liam Paull. 2020.La-maml: Look-ahead meta learning for continuallearning. arXiv preprint arXiv:2007.13904.

Jonathan Herzig and Jonathan Berant. 2018. Decou-pling structure and lexicon for zero-shot semanticparsing. In Proceedings of the 2018 Conference onEmpirical Methods in Natural Language Processing,pages 1619–1629, Brussels, Belgium. Associationfor Computational Linguistics.

Jonathan Herzig, Pawel Krzysztof Nowak, ThomasMüller, Francesco Piccinno, and Julian Eisenschlos.2020. TaPas: Weakly supervised table parsing viapre-training. In Proceedings of the 58th AnnualMeeting of the Association for Computational Lin-guistics, pages 4320–4333, Online. Association forComputational Linguistics.

Sepp Hochreiter and Jürgen Schmidhuber. 1997.Long short-term memory. Neural computation,9(8):1735–1780.

Po-Sen Huang, Chenglong Wang, Rishabh Singh, Wen-tau Yih, and Xiaodong He. 2018. Natural languageto structured query generation via meta-learning. InProceedings of the 2018 Conference of the NorthAmerican Chapter of the Association for Compu-tational Linguistics: Human Language Technolo-gies, Volume 2 (Short Papers), pages 732–738, NewOrleans, Louisiana. Association for ComputationalLinguistics.

Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, JayantKrishnamurthy, and Luke Zettlemoyer. 2017. Learn-ing a neural semantic parser from user feedback. InProceedings of the 55th Annual Meeting of the As-sociation for Computational Linguistics (Volume 1:Long Papers), pages 963–973.

Diederik P. Kingma and Jimmy Ba. 2015. Adam: Amethod for stochastic optimization. In 3rd Inter-national Conference on Learning Representations,ICLR 2015, San Diego, CA, USA, May 7-9, 2015,Conference Track Proceedings.

Dongjun Lee, Jaesik Yoon, Jongyun Song, Sanggil Lee,and Sungroh Yoon. 2019. One-shot learning for text-to-sql generation.

Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M.Hospedales. 2018a. Learning to generalize: Meta-learning for domain generalization. In Proceed-ings of the Thirty-Second AAAI Conference on Ar-tificial Intelligence, (AAAI-18), the 30th innovativeApplications of Artificial Intelligence (IAAI-18), andthe 8th AAAI Symposium on Educational Advancesin Artificial Intelligence (EAAI-18), New Orleans,Louisiana, USA, February 2-7, 2018, pages 3490–3497. AAAI Press.

Fei Li and H. V. Jagadish. 2014. Constructing aninteractive natural language interface for relationaldatabases. Proceedings of the VLDB Endowment,8(1):73–84.

Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C.Kot. 2018b. Domain generalization with adversarialfeature learning. In 2018 IEEE Conference on Com-puter Vision and Pattern Recognition, CVPR 2018,Salt Lake City, UT, USA, June 18-22, 2018, pages5400–5409. IEEE Computer Society.

Xi Victoria Lin, Richard Socher, and Caiming Xiong.2020. Bridging textual and tabular data for cross-domain text-to-SQL semantic parsing. In Findingsof the Association for Computational Linguistics:EMNLP 2020, pages 4870–4888, Online. Associa-tion for Computational Linguistics.

Arthur Mensch and Mathieu Blondel. 2018. Differen-tiable dynamic programming for structured predic-tion and attention. In Proceedings of the 35th In-ternational Conference on Machine Learning, ICML2018, Stockholmsmässan, Stockholm, Sweden, July10-15, 2018, volume 80 of Proceedings of MachineLearning Research, pages 3459–3468. PMLR.

Qingkai Min, Yuefeng Shi, and Yue Zhang. 2019. Apilot study for Chinese SQL semantic parsing. InProceedings of the 2019 Conference on EmpiricalMethods in Natural Language Processing and the9th International Joint Conference on Natural Lan-guage Processing (EMNLP-IJCNLP), pages 3652–3658, Hong Kong, China. Association for Computa-tional Linguistics.

Alex Nichol, Joshua Achiam, and John Schulman.2018. On first-order meta-learning algorithms.arXiv preprint arXiv:1803.02999.

Abiola Obamuyide and Andreas Vlachos. 2019.Model-agnostic meta-learning for relation classifica-tion with limited supervision. In Proceedings of the57th Annual Meeting of the Association for Com-putational Linguistics, pages 5873–5879, Florence,Italy. Association for Computational Linguistics.

Panupong Pasupat and Percy Liang. 2015. Compo-sitional semantic parsing on semi-structured tables.In Proceedings of the 53rd Annual Meeting of theAssociation for Computational Linguistics and the7th International Joint Conference on Natural Lan-guage Processing (Volume 1: Long Papers), pages1470–1480, Beijing, China. Association for Compu-tational Linguistics.

Adam Paszke, Sam Gross, Francisco Massa, AdamLerer, James Bradbury, Gregory Chanan, TrevorKilleen, Zeming Lin, Natalia Gimelshein, LucaAntiga, Alban Desmaison, Andreas Köpf, EdwardYang, Zachary DeVito, Martin Raison, Alykhan Te-jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang,Junjie Bai, and Soumith Chintala. 2019. Py-torch: An imperative style, high-performance deeplearning library. In Advances in Neural Informa-tion Processing Systems 32: Annual Conferenceon Neural Information Processing Systems 2019,NeurIPS 2019, December 8-14, 2019, Vancouver,BC, Canada, pages 8024–8035.

Jeffrey Pennington, Richard Socher, and ChristopherManning. 2014. GloVe: Global vectors for wordrepresentation. In Proceedings of the 2014 Confer-ence on Empirical Methods in Natural LanguageProcessing (EMNLP), pages 1532–1543, Doha,Qatar. Association for Computational Linguistics.

Ana-Maria Popescu, Oren Etzioni, , and Henry Kautz.2003. Towards a theory of natural language inter-faces to databases. In Proceedings of the 8th Inter-national Conference on Intelligent User Interfaces,pages 149–157.

Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton,and Christopher D. Manning. 2020. Stanza: Apython natural language processing toolkit for manyhuman languages. In Proceedings of the 58th An-nual Meeting of the Association for ComputationalLinguistics: System Demonstrations, pages 101–108, Online. Association for Computational Linguis-tics.

Sachin Ravi and Hugo Larochelle. 2017. Optimiza-tion as a model for few-shot learning. In 5th Inter-national Conference on Learning Representations,ICLR 2017, Toulon, France, April 24-26, 2017, Con-ference Track Proceedings. OpenReview.net.

Mengye Ren, Eleni Triantafillou, Sachin Ravi, JakeSnell, Kevin Swersky, Joshua B. Tenenbaum, HugoLarochelle, and Richard S. Zemel. 2018. Meta-learning for semi-supervised few-shot classification.In 6th International Conference on Learning Rep-resentations, ICLR 2018, Vancouver, BC, Canada,April 30 - May 3, 2018, Conference Track Proceed-ings. OpenReview.net.

Yan Song, Shuming Shi, Jing Li, and Haisong Zhang.2018. Directional skip-gram: Explicitly distinguish-ing left and right context for word embeddings. InProceedings of the 2018 Conference of the NorthAmerican Chapter of the Association for Compu-tational Linguistics: Human Language Technolo-gies, Volume 2 (Short Papers), pages 175–180, NewOrleans, Louisiana. Association for ComputationalLinguistics.

Alane Suhr, Ming-Wei Chang, Peter Shaw, and Ken-ton Lee. 2020. Exploring unexplored generalizationchallenges for cross-database semantic parsing. In

Proceedings of the 58th Annual Meeting of the Asso-ciation for Computational Linguistics, pages 8372–8388, Online. Association for Computational Lin-guistics.

Yibo Sun, Duyu Tang, Nan Duan, Yeyun Gong, Xi-aocheng Feng, Bing Qin, and Daxin Jiang. 2019.Neural semantic parsing in low-resource settingswith back-translation and meta-learning.

Lappoon R. Tang and Raymond J. Mooney. 2000. Au-tomated construction of database interfaces: Inter-grating statistical and relational learning for seman-tic parsing. In 2000 Joint SIGDAT Conference onEmpirical Methods in Natural Language Processingand Very Large Corpora, pages 133–141.

Oriol Vinyals, Charles Blundell, Tim Lillicrap, KorayKavukcuoglu, and Daan Wierstra. 2016. Matchingnetworks for one shot learning. In Advances inNeural Information Processing Systems 29: AnnualConference on Neural Information Processing Sys-tems 2016, December 5-10, 2016, Barcelona, Spain,pages 3630–3638.

Bailin Wang, Richard Shin, Xiaodong Liu, OleksandrPolozov, and Matthew Richardson. 2020. RAT-SQL:Relation-aware schema encoding and linking fortext-to-SQL parsers. In Proceedings of the 58th An-nual Meeting of the Association for ComputationalLinguistics, pages 7567–7578, Online. Associationfor Computational Linguistics.

Yushi Wang, Jonathan Berant, and Percy Liang. 2015.Building a semantic parser overnight. In Proceed-ings of the 53rd Annual Meeting of the Associationfor Computational Linguistics and the 7th Interna-tional Joint Conference on Natural Language Pro-cessing (Volume 1: Long Papers), pages 1332–1342,Beijing, China. Association for Computational Lin-guistics.

Navid Yaghmazadeh, Yuepeng Wang, Isil Dillig, andThomas Dillig. 2017. Sqlizer: Query synthesis fromnatural language. In International Conference onObject-Oriented Programming, Systems, Languages,and Applications, ACM, pages 63:1–63:26.

Pengcheng Yin and Graham Neubig. 2018. TRANX: Atransition-based neural abstract syntax parser for se-mantic parsing and code generation. In Proceedingsof the 2018 Conference on Empirical Methods inNatural Language Processing: System Demonstra-tions, pages 7–12, Brussels, Belgium. Associationfor Computational Linguistics.

Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Se-bastian Riedel. 2020. TaBERT: Pretraining for jointunderstanding of textual and tabular data. In Pro-ceedings of the 58th Annual Meeting of the Asso-ciation for Computational Linguistics, pages 8413–8426, Online. Association for Computational Lin-guistics.

Tao Yu, Michihiro Yasunaga, Kai Yang, Rui Zhang,Dongxu Wang, Zifan Li, and Dragomir Radev.2018a. SyntaxSQLNet: Syntax tree networks forcomplex and cross-domain text-to-SQL task. In Pro-ceedings of the 2018 Conference on Empirical Meth-ods in Natural Language Processing, pages 1653–1663, Brussels, Belgium. Association for Computa-tional Linguistics.

Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga,Dongxu Wang, Zifan Li, James Ma, Irene Li,Qingning Yao, Shanelle Roman, Zilin Zhang,and Dragomir Radev. 2018b. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. InProceedings of the 2018 Conference on EmpiricalMethods in Natural Language Processing, pages3911–3921, Brussels, Belgium. Association forComputational Linguistics.

Tianhe Yu, Saurabh Kumar, Abhishek Gupta, SergeyLevine, Karol Hausman, and Chelsea Finn. 2020.Gradient surgery for multi-task learning. NeurIPS.

Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh,Barnabás Póczos, Ruslan Salakhutdinov, andAlexander J. Smola. 2017. Deep sets. In Advancesin Neural Information Processing Systems 30: An-nual Conference on Neural Information ProcessingSystems 2017, December 4-9, 2017, Long Beach,CA, USA, pages 3391–3401.

John M. Zelle and Raymond J. Mooney. 1996. Learn-ing to parse database queries using inductive logicprogramming. In Proceedings of the Thirteenth Na-tional Conference on Artificial Intelligence - Volume2, pages 1050–1055.

Victor Zhong, Mike Lewis, Sida I. Wang, and LukeZettlemoyer. 2020. Grounded adaptation for zero-shot executable semantic parsing. In Proceedings ofthe 2020 Conference on Empirical Methods in Nat-ural Language Processing (EMNLP), pages 6869–6882, Online. Association for Computational Lin-guistics.

Victor Zhong, Caiming Xiong, and Richard Socher.2017. Seq2sql: Generating structured queriesfrom natural language using reinforcement learning.CoRR, abs/1709.00103.

A Analysis of DG-FMAML

Similarly, we use the first-order Taylor expansionto analyze the gradients of DG-FMAML:

∇θLτ (θ) = ∇θLBs(θ) +∇θ′LBt(θ′)=∇θLBs(θ) +∇θ′LBt(θ − α∇θLBs(θ))≈∇θLBs(θ) +∇θ′LBt(θ)+

α∇2θ′LBt(θ)∇θLBs(θ)

=∇θLBs(θ) +∇θ′LBt(θ)+α∇θ′

(∇θ′LBt(θ) · ∇θLBs(θ)

)

where in Step 3 we expand the gradient func-tion ∇θ′LBt at θ. In DG-FMAML, there is nogradients back-propogating from θ′ to θ, so we cantreat ∇θLBs(θ) and ∇θ′LBt(θ′) as two indepen-dent functions with θ and θ′ denoting their param-eters respectively.

In Step 4, the first two terms ∇θLBs(θ) +∇θ′LBt(θ) can be viewed as the gradient of ap-plying θ to both source and target domains. Thelast term can be viewed as maximize the dot prod-uct between gradients of source and target domainwith respect to θ′. In the same spirit as DG-MAML,DG-FMAML also tries to encourage the gradientsto generalize between source and target domains.

B Datasets

Spider Spider consists of 10,181 examples (ques-tions and SQL pairs) from 206 databases, includ-ing 1,659 examples taken from the Restaurants(Popescu et al., 2003; Tang and Mooney, 2000),GeoQuery (Zelle and Mooney, 1996), Scholar(Iyer et al., 2017), Academic (Li and Jagadish,2014), Yelp and IMDB (Yaghmazadeh et al., 2017)datasets. We follow their split and use 8,659 exam-ples (from 146 databases) for training, and 1,034examples (from 20 databases) as our developmentset. The remaining 2,147 examples from 40 testdatabases are held out and kept by the authors forevaluation.

Chinese Spider Chinese Spider is a Chinese ver-sion of Spider that translates all NL questions fromEnglish to Chinese and keeps the original Englishdatabase. It simulates the real-life scenario whereschemas for most relational databases in industryare written in English while NL questions fromusers could be in any other language. Follow-ing Min et al. (2019), we use the same split oftrain/development/test as the Spider dataset.

C Hyperparameters

Base Parser We stack 6 relation-aware self-attention layers for encoding. Within them, weset the number of attention heads to be 8 and usedropout rate 0.1. Word embeddings for Englishquestions, column and table names are shared andheld fixed except for the 50 most common wordsin the training set. Word embeddings for Chinesequestions are also fixed, except for the 50 most com-mon words in the training set. As noted in Wanget al. (2020), RAT-SQL went through an extensive

hyperparameter sweep for non-BERT RAT-SQL,which partially explains why our non-BERT baseparser is not as good as it in Spider. However, afterthe integration of BERT representations, our baseparser slightly outperforms RAT-SQL, as shown inthe main paper.

Preprocessing A major difference between ourbase parser and RAT-SQL (Wang et al., 2020)is the way of preprocessing. During preprocess-ing, input questions, column names and tablenames in schemas are tokenized and lemmatizedby Stanza (Qi et al., 2020) which can handle bothEnglish and Chinese.

Learning Rates We use the learning rate of α =5 × 10−4 for meta-train. For the final update ofparameters, we use Adam (Kingma and Ba, 2015)with the learning rate 6×10−4. We manually searchfor the best meta-train learning rates from 1×10−4

to 9 × 10−4 with the step size 2 × 10−4, basedon performance on the development set. Otherhyperparameters are not tuned. For the learningrate of final update (not α of meta-train), we use thesame scheduler as Wang et al. (2020). Specifically,during the first 500 steps, the learning rate linearlyincreases from 0 to 6× 10−4. Then, it is annealedto 0 with 6× 10−4(1− step−500

9500 )−0.5.

Hardware and Model Size Our non-BERTmodels are trained using NVIDIA GeForce RTX2080, which has a memory size of 11GB. The baseparser has around 10 million parameters, wherearound 1.5 million parameters are pre-trained em-beddings that are mostly fixed during training. ForBERT models, we first find the best hyperparame-ters using GeForce RTX 2080 with a small batchsize; then we train them using V100 to save cost.

Average Runtime The average training time forthe non-BERT base parser, DG-MAML and DG-FMAML are 10, 24, 13 hours per run. For BERTmodels, the numbers are 36, 68, 42 hours per run.

C.1 Loss Curve

In Figure 2, we show the loss curves of the modelson the two datasets during training. In compari-son, DG-MAML helps to reduce the gap betweentraining and validation loss.

D Classifier for Probing DomainGeneralization

The classifier takes the input of a pair of (column,question) and outputs a binary label indicatingwhether the column is relevant. As explained in thepaper, we retrieve the representations of columnsand questions from a pre-trained parser. We denotethe representation of a column as c ∈ Rk, and therepresentation of a question as q ∈ Rn×k where nis the number of words in the question and k is thesize of encoding.

For each pair of (c, q), we first align the col-umn c softly with the question q using an attentionfunction, and obtain an aligned representation tfor the column. Then we compute a score of rele-vance based on the aligned representation. Finally,a probability p of relevance is computed through asigmoid function σ.

t = Attention(c, q)

score = MLP(c, t)

p = σ(score)

(6)

0 2 4 6 8 10Traning Steps (103)

0

10

20

30

40

50

Loss

Baseline-trainBaseline-valDG-MAML-trainDG-MAML-val

(a) Loss curves on the Spider dataset.

0 2 4 6 8 10Traning Steps (103)

0

10

20

30

40

50

Loss

Baseline-trainBaseline-valDG-MAML-trainDG-MAML-val

(b) Loss curves on the Chinese Spider dataset.

Figure 2: Comparison of losses when the parser is trainined with conventaional supervised learning (Baseline) andDG-MAML.