On the readiness of quantum optimization machines for industrial applications

Alejandro Perdomo-Ortiz,1, 2, 3, ∗ Alexander Feldman,4 Asier Ozaeta,5 Sergei V.Isakov,6 Zheng Zhu,7 Bryan O’Gorman,1, 8, 9 Helmut G. Katzgraber,7, 10, 11 Alexander

Diedrich,12 Hartmut Neven,13 Johan de Kleer,4 Brad Lackey,14, 15, 16 and Rupak Biswas17

1Quantum Artificial Intelligence Lab., NASA Ames Research Center, Moffett Field, California 94035, USA2USRA Research Institute for Advanced Computer Science (RIACS), Mountain View California 94043, USA

3Department of Computer Science, University College London, WC1E 6BT London, UK4Palo Alto Research Center, 3333 Coyote Hill Road, Palo Alto, California 94304, USA

5QC Ware Corp., 125 University Ave., Suite 260, Palo Alto, California 94301, USA6Google Inc., 8002 Zurich, Switzerland

7Department of Physics and Astronomy, Texas A&M University, College Station, Texas 77843-4242, USA8Berkeley Center for Quantum Information and Computation, Berkeley, California 94720 USA

9Department of Chemistry, University of California, Berkeley, California 94720 USA101QB Information Technologies (1QBit), Vancouver, British Columbia, Canada V6B 4W4

11Santa Fe Institute, 1399 Hyde Park Road, Santa Fe, New Mexico 87501, USA12Fraunhofer IOSB-INA, Lemgo, Germany

13Google Inc., Venice, California 90291, USA14Joint Center for Quantum Information and Computer Science,University of Maryland, College Park, Maryland 20742, USA

15Departments of Computer Science and Mathematics,University of Maryland, College Park, Maryland 20742, USA

16Mathematics Research Group, National Security Agency, Ft. George G. Meade, Maryland 20755, USA17Exploration Technology Directorate, NASA Ames Research Center, Moffett Field, California 94035, USA

(Dated: September 1, 2017)

There have been multiple attempts to demonstrate that quantum annealing and, in particular, quantum anneal-ing on quantum annealing machines, has the potential to outperform current classical optimization algorithmsimplemented on CMOS technologies. The benchmarking of these devices has been controversial. Initially,random spin-glass problems were used, however, these were quickly shown to be not well suited to detectany quantum speedup. Subsequently, benchmarking shifted to carefully crafted synthetic problems designedto highlight the quantum nature of the hardware while (often) ensuring that classical optimization techniquesdo not perform well on them. Even worse, to date a true sign of improved scaling with the number problemvariables remains elusive when compared to classical optimization techniques. Here, we analyze the readinessof quantum annealing machines for real-world application problems. These are typically not random and havean underlying structure that is hard to capture in synthetic benchmarks, thus posing unexpected challenges foroptimization techniques, both classical and quantum alike. We present a comprehensive computational scalinganalysis of fault diagnosis in digital circuits, considering architectures beyond D-wave quantum annealers. Wefind that the instances generated from real data in multiplier circuits are harder than other representative randomspin-glass benchmarks with a comparable number of variables. Although our results show that transverse-fieldquantum annealing is outperformed by state-of-the-art classical optimization algorithms, these benchmark in-stances are hard and small in the size of the input, therefore representing the first industrial application ideallysuited for near-term quantum annealers.

I. INTRODUCTION

Quantum annealing [1–7] has been proposed as the mostnatural quantum computing framework to tackle combinato-rial optimization problems, where finding the configurationthat minimizes an application-specific cost function is at thecore of the computational task. Despite multiple studies [8–20], a definite detection of quantum speedup [13, 21] re-mains elusive. Random spin-glass benchmarks [13] have beenshown to be deficient in the detection of quantum speedup[11, 14], which is why the community has shifted to carefully-crafted synthetic benchmarks [19, 20]. While these have

∗Electronic address: alejandro.perdomoortiz@nasa.gov

shown that quantum annealing has a constant speedup overstate-of-the-art classical optimization techniques, their valuefor real-world applications remains controversial.

Although the first proposal for a quantum annealing imple-menting combinatorial optimization problems with real con-strains as they appear in real-world application was proposedclose to a decade ago [22], the question of whether a quan-tum annealer can have a quantum speedup on any real-worldapplications remains an open one. From the many applica-tions implemented in quantum annealers (see for example,Refs. [17, 23–28]), fault diagnosis has been one of the leadingcandidates to benchmark the performance of D-Wave devicesas optimizers [26, 29]. From the range of circuit model-basedfault diagnosis problems [30] we restrict our attention here tocombinational circuit fault diagnosis (CCFD), which in con-trast to sequential circuits, does not have any memory com-ponents and the output is entirely determined by the present

arX

iv:1

708.

0978

0v1

[qu

ant-

ph]

31

Aug

201

7

2

inputs.Using CCFDs, we illustrate the challenges and the readi-

ness of quantum annealers for solving real-world problems byproviding a comprehensive computational scaling analysis ofa real-world application. We compare quantum Monte Carlo(QMC) simulations and quantum annealing experiments onthe D-Wave Systems Inc. D-Wave 2X quantum annealer toseveral state-of-the-art classical solvers on conventional com-puter hardware. More specifically, in this work we strive toanswer or discuss the following questions:

1. What is the payoff of investing in the construction of spe-cialized quantum hardware that natively matches the connec-tivity and interactions (e.g., many-body terms in higher-orderHamiltonians) dictated by the cost function of an actual appli-cation?

2. What could be the impact in the computational scaling ofdifferent annealing schedules or the addition of more complexdriver, such as non-stoquastic Hamiltonians?

3. Does quantum Monte Carlo reproduce the computationalscaling of the current generation of D-Wave quantum anneal-ing machines?

We discuss the importance of each of these algorithmic andarchitectural design aspects related to each of the questionsabove, from an application-centric and physics-focused per-spective. It is demonstrated that CCFD instances based onBoolean multiplier circuits are harder than other representa-tive random spin-glass benchmarks. This makes the diagno-sis of Boolean multipliers a prime application for benchmark-ing quantum annealing architectures. Although we demon-strate that quantum annealing with a transverse-field Hamil-tonian is outperformed by state-of-the-art tailored algorithms[31], CCFD instances are ideal industrial application prob-lems for testing incremental improvements in near-term quan-tum annealers and novel quantum algorithmic strategies foroptimization problems.

II. BENCHMARK PROBLEM

To benchmark quantum annealers with different physicalhardware specifications, we generate a family of multipliercircuits of varying size. The circuit size is determined by thesize of two binary numbers of bit-lengths n and m, respec-tively, to be multiplied. Figure 1 illustrates the layout of themultiplication circuit for two binary numbers, each of lengthk.

The optimization problem consists in diagnosing the healthstatus of each of the gates in the circuit, given an observa-tion vector consisting of inputs and outputs, as illustrated inFig. 2. For the generation of the problem instances, we focuson problems where the output is not consistent with the multi-plication of the two input numbers and therefore the system isexpected to have at least one fault. Under the assumption thatall the gates have the same failure probability, the problem offinding the most probable diagnosis is reduced to finding the

valid diagnoses with the minimal number of faults (see Ap-pendix C for details). It is important to note that all CCFDinstances used in this study were randomly generated by in-jecting a number of faults equivalent to the number of outputsin the circuit [(n + m) for a (n-bit) ×(m-bit) multiplier cir-cuit]. After the random fault injection of cardinality (n+m),a random input is generated and the corresponding output isobtained by propagation of the input under the correspondingfault injection. Hence, we guarantee that every random in-put/output pair generated this way has at least one solution.The simpler strategy of generating random input/output vec-tors can lead to problems that do not have a solution underthe diagnosis model. In the case of instances with many validminimal solutions, we count all the ones found by the stochas-tic algorithms in the estimation of the success probability.

From a computational complexity perspective, the CCFDproblem is NP-hard [33], and it corresponds to the minimiza-tion task we aim to solve either with quantum annealing on theD-Wave 2X device (DW2X) at NASA, a continuous time ver-sion [34] of simulated quantum annealing (SQA) [6, 12, 13]as a QMC-based solver, or other classical optimization tech-niques, such as simulated annealing (SA) [35, 36], paralleltempering Monte Carlo (modified as a solver) [37–39] com-bined with isoenergetic cluster updates [40] (PTICM), or cur-rent specialized SAT-based solvers tailored for this CCFDproblem described in Appendix B.

To perform a scaling analysis it is key to be able to gener-ate a data set with varying input size and with a high intrin-sic hardness such that classical solvers have a harder time,increasing the chances that our instances fall into the hardasymptotic regime for both classical and quantum approaches.This has been one of the challenges for benchmarking earlyquantum annealing devices, where the first proposals [12, 13]were convenient but turned out to be too easy for benchmark-ing purposes [11]. More recently, benchmarking have focusedon carefully-designed synthetic problems [16, 19, 20]. How-ever, as we shall demonstrate in Sec. III B, CCFD-based prob-lems are the hardest benchmarking problems currently avail-able.

III. RESULTS AND DISCUSSION

A. Benchmarking real-world applications

Figure 3 summarizes the main challenges when bench-marking applications with quantum annealing devices. Thefirst step consist of translating the standard format describingthe rules and constrains of the minimization problem into apseudo-Boolean polynomial function HP(sP), with domainsP ∈ {+1,−1}NP and co-domain in R. Appendix C detailsthe construction of HP(sP) for this problem of minimal faultdiagnosis in combinational circuits. The task to be solved con-sists in finding, within the search space with 2NP possible so-lutions, the assignment s∗P that minimizes HP(sP). Becausethe pseudo-Boolean function is a polynomial expression in thebinary variables sP, this optimization problem is known as a

3

a2

b4

a2

b3

a2

b2

bk

HA HA HA

FA FA FA

FA FA FA

FA FA FA

FA FA HA

bk

a1

b2

b1a1b1a2b1a3

a1

b2

b1ak

ak b2

b3ak

a2ak−1

bk

bkak

a1

b4

b3

a1

b4

b3

ak−1

ak−1

ak−1

bk−1ak

p4

p3

p2

p1

pk

pk+1pk+2p2k−1p2k

FIG. 1: Multiplier circuits used to generate our CCFD benchmark instances. In this example the multiplication of two numbers representedas k-digit binary numbers, a1a2 · · · ak and b1b2 · · · bk is shown, resulting in a product output of length 2k, corresponding to p1p2 · · · p2k. HAand FA denote half-adder and full-adder circuit modules, respectively.

ci

i1

i2

co

Σ

0

1

ANDOR

AND

XOR

XOR1

0

0

0

FIG. 2: Example of a model-based fault-diagnosis in combinationalcircuits (CCFD) on a small full adder circuit. In this work, the CCFDoptimization problem consists in finding a smallest set of Booleangate outputs that, when stuck-at-one, match an input/output observa-tion vector. This setting where one restricts the expected fault behav-ior of the gates is known as the strong-fault model (see, for example,Ref. [32]). Although in principle each gate can have a characteristicfault mode, without loss of generality, we adopted stuck-at-one asthe fault mode for all gates. Generalizations to other common faultmodes and multiple fault modes per gate are detailed in Appendix C.In this example, the flagged XOR gate is faulty, because its nominalbehavior should yield an output equal to zero. The diagnosis explainsthe input {i1 = 0, i2 = 0, ci = 0} and the apparently anomalousoutput {c0 = 0,Σ = 1}.

PUBO problem which stands for polynomial unconstrainedbinary optimization problem. Note that sometimes these prob-lems are also referred to as HOBOs, i.e, higher-order binaryoptimization problem. The specific case of a quadratic func-tion leads to the known QUBO [41] which is the type that

is natively implemented in D-Wave quantum annealers. SeeAppendix. A for more details on the quantum annealing im-plementation.

In the case of the benchmark of multiplier circuits, the stan-dard problem description format is a list of propositional logicformulas similar to the ones given in Fig. 3, corresponding tothe nominal behavior of each gate within the full-adder circuitillustrated in Fig. 2. For the case of the strong-fault model[32] considered here, one needs to add specific propositionallogic formulas associated with the expected behavior wheneach gate is faulty. Without loss of generality, and for thepurpose of the benchmark generated here, we considered thatwhenever any gate fails, it would be in a stuck-at-one modeor equivalently, in propositional logic, fi ⇒ zi. Here fi de-notes the health variable associated with the i-th gate and zi itscorresponding gate output. Note that fi = 1 means faulty andfi = 0 nominal. Extensions to other fault modes are describedin Appendix. C. In the specific mapping considered here, sPcontains the health variables, along with variables specifyingthe values for each of the internal wires within the multipliercircuit.

A generic classical solver such as SA or PTICM cantackle the optimization problem in th PUBO representationdirectly because one can easily evaluate HP(sP). As shownin Sec. III C, working in this PUBO representation is the pre-ferred approach from the application perspective. As men-tioned in the explicit mapping construction in Sec. C, HP(sP)is a polynomial with at most quartic degree, independent ofthe circuit size. A quantum annealer capable of implement-ing such quartic polynomials can certainly aim at solving theproblem in this representation. Given the possibility of suchexperimental designs (see, for example, Ref. [42]), we alsoconsider hypothetical quantum annealers that we study using

4

PUBO(P)Propositionallogic

input:i1input:i2input:cioutput:cooutput:sum

z1==i1&&i2z2==(i1==!i2)z3==z2&&cisum ==(z2==!ci)co ==z1||z3

Efficientmappingoftheproblem

EfficientlocalityreductiontoQUBO

1

2

sisjsk

sisjsksl

sisjsk

sisjsksl

⌘⌘

QUBOgraphcomplexbutsparseenoughallowingforembeddingtonear-termorexistingquantumhardware

3

Smallprecisionrequirements.4

Quantumannealer

QUBO(Q)

{f ,x,a} ! {s}Q

{f ,x} ! {s}P

(a) (b)

(c) (d)

FIG. 3: Generation of benchmarks driven with real-world scenar-ios flowchart. The first challenge consists in finding an efficienttranslation of the natural description of the optimization probleminto a pseudo-Boolean function [panel (a)], HP(sP), with domainsP ∈ {+1,−1}NP and co-domain in R. The resulting polynomialunconstrained binary optimization (PUBO) problem consists in find-ing assignments s∗P which minimizes the quartic-degree polynomialHP(sP) [panel (b)]. This specific degree, known as the locality ofthe Hamiltonian (here, 4-local), arises from the effective local inter-actions between gates with two input variables xi,1 and xi,2, the wireoutput zi, and the health variable, fi, associated to each gate. Byadding ancilla qubits, (a), the quartic (4-local) sisjsksl and cubic(3-local) sisjsk terms can be reduced to an effective 2-local Hamil-tonian defining an effective quadratic unconstrained binary optimiza-tion (QUBO) version of the problem instance [pabel (d)]. Finally,minor-embedding can be used to embed the QUBO into the physicalhardware – in this case the chimera structure (see Fig. 7) of the D-Wave quantum annealers [panel (c)]. The cost of the embedding isan additional overhead in the number of qubits. While the “proposi-tional logic” panel contains the description of the full-adder circuit inFig. 2, the remaining are realistic representations of one of the small-est instances from our multiplier circuit with 23, 33, and 72 qubits(or spin variables), for its PUBO, QUBO, and DW2X representation,respectively. In this work, we assess the impact on the performanceof each of these representations and also perform experiments on theDW2X. Steps 1 – 4 denote some of the desiderata for an applica-tion to be a potential candidate for benchmarking next generationof quantum annealers. Note that while the D-Wave device requiresthe embedding of a QUBO. However, future hardware implementa-tions might include k-local interactions with k > 2. Therefore, weperform the classical simulations both in the QUBO and PUBO rep-resentations to compare both approaches.

SQA to assess the impact in the performance of working witha quantum annealer that can natively solve the PUBO prob-lem. Unfortunately, no such devices exist to date and there isan overhead in representing the quartic (4-local) and cubic (3-local) monomials in HP(sP) with a resulting only-quadraticexpression (2-local). The contraction techniques [43] used toreduce the locality incur an overhead of variables by introduc-ing ancillas (for a tutorial of a specific practical example seeRef. [22]). This is not desirable because it increases the searchspace from 2NP to 2NQ , with NQ the number of variables sQin the resulting new quadratic expressionsHQ(sQ) as the newrepresentation fromHP(sP). sQ is now the union of the healthvariables f , the wires x, and the ancilla set a. The overhead islinear in our case as shown in Fig. 9. The next challenge pre-sented in Fig. 3 towards implementing a real-world applica-tion is that most likely there will be quadratic term inHQ(sQ)representing qubit-qubit interactions not present in the phys-ical hardware. This will be the case unless one specificallydesign the layout of the quantum annealer hardware to matchthe resulting connectivity graph dictated directly by the ap-plication through HQ(sQ). Representing the logical graph

withing another graph is called the minor-embedding prob-lem [44]. For the case of the connectivity graph pre-definedin the D-Wave devices, also known as chimera graph, we usethe heuristic solver developed in Ref. [45]. As can be seenin Fig. 9, the overhead is linear given the relatively sparsityof the graphs resulting from the multiplier circuits. This isan encouraging result given that the overhead for an all-to-allconnectivity graph embedded onto the chimera architecture isquadratic in the number of variables. A much larger problemthan minor embedding when embedding an application onto alimited connectivity hardware graph is parameter setting. Forexample, there is no rule of thumb as to how strong the cou-plers for a set of ferromagnetically-coupled qubits defininga physical qubit should be. A sweet-spot value is expected,however it is not easy to determine or predict in the most gen-eral setting. In this work, and for all the experiments on the D-Wave 2X, we used the strategy proposed in Ref. [46] for bothsetting the strength of the ferromagnetic couplers and for theselection of gauges. The final challenge when embedding ap-plications is the requirement that the pseudo-Boolean functionto be minimized has a low precision requirement because ana-

5

log quantum annealing machines operate on a limited preci-sion dictated by the intrinsic noise and finite dynamical rangeof parameters found in these devices.

Summarizing, from our experience with applications, theCCFD instances considered here are the best candidate tomatch each one of the aforementioned requirements. Themapping from propositional logic to PUBO is compact andefficient given that in the digital circuits considered here allthe input, outputs, health variables and wires are all binaryvariables, the resulting QUBO graph is sparse enough that theoverhead to embed onto hardware is linear, and the randomly-generated instances have a higher intrinsic hardness com-pared to other random spin-glass previously studied, as willbe shown in Sec. III B.

B. Hardness compared to other random spin-glassbenchmarks

Figure 4 addresses the hardness of instances embedded inthe chimera topology (C) generated from the CCFD data setby comparing to random spin-glass problems used to bench-mark the performance of D-Wave quantum annealers [seeEq. (A1) for the actual Hamiltonian to be minimized]. Bi-modal instances were the first to be used in benchmarkingstudies [12, 13] and are the simplest to generate. For these,the available couplers in the D-Wave 2X are randomly chosento be Jij ∈ {±1}with biases hi = 0. The reason why randombimodal instances are too easy for quantum and classical algo-rithms alike is their high degeneracy resulting in a large num-ber of floppy spins. To overcome this problem, Refs. [14, 47]introduced couplers distributed according to Sidon sets [48]combined with post-selection procedures. These naturally in-crease the hardness of problems by reducing degeneracy to aminimum and removing floppy spins. For the case of Sidoninstances [47] the values of the couplers Jij are randomly se-lected from the set {±5,±6,±7}, with hi = 0. Planted/Cinstances correspond to an attempt to increase the hardness ofrandom spin-glass instances (see Ref. [16]), but with a knownsolution. For the data shown, we asked the main author inRef. [16], if he could provide us with the hardest set of in-stances he could generate; the only restriction being that theywould need to be generated randomly and not being post-selected for hardness as the rest of all the other families ofinstances here. The attempt consisted of drawing the couplersfrom a continuous distribution instead of from a discrete dis-tribution as the one in the original paper, Ref. [16], or as in thecase of the Bimodal and Sidon set considered here.

Figure 4(a) illustrates that already for approximately 600variables the CCFD/C instances are at least one order of mag-nitude harder than Sidon/C which is the hardest set amongthe random spin-glass problems. Figure 4(b) summarizes theasymptotic scaling of each of these problem types, clearly sep-arating our CCFD instances from any of the random spin-glassinstances, with Sidon and Bimodal having roughly the samescaling. Here we assume that the time-to-solution TTS in µscan be fit to TTS ∼ exp(b

√N)] with N the number of vari-

ables. This conclusion is independent of the percentile consid-

(a)

(b)

FIG. 4: Hardness comparison of the chimera representation of theCCFD instances with other representative random spin glass prob-lems from the literature. (a) For consistency, all the time-to-solutions(TTS) in µs were obtained with PTICM with the same single-coreprocessors. (b) Note the steeper scaling [larger value for b from a fitof the TTS to TTS ∼ exp(b

√N)] for the CCFD problems com-

pared to the other classes of random spin-glass benchmarks. Datapoints correspond to the median values extracted from a bootstrap-ping statistical analysis from 100 instances per problem size, witherror bars indicating the 90% confidence intervals (CI). The differentinstance classes are described in the main text.

6

ered as shown in Fig. 10 in Appendix. F. Our results also showthat the attempt to make hard planted instances did not provideany additional hardness compared to the other random spin-glass problems, at least when they are evaluated with PTICM.Therefore, the CCFD/C instances are not only harder in termsof computational effort, according to TTS, but also from ascaling perspective.

The data set Bimodal/CCFD provides insights as to whythese instances are hard. There are three options of why theseinstances are intrinsically harder than any other random dataset explored here. One option is that the underlying CCFDgraph defined by the QUBO problem for each multiplier typehas some sort of nontrivial long-range correlations or a muchhigher dimensionality in such a way that the problems, whenminor-embedded onto the chimera lattice, become harder thantypical chimera instances. Another explanation relies on thecharacteristic value of biases h and coupler values J in theHamiltonian [Eq. (A1)] that in a nontrivial way define a verycomplex-to-traverse energy landscape. Furthermore, therecould be interplay between the two aforementioned options.To address this question, we generate Bimodal instances onthe native QUBO graph defined by multiplier circuits of vary-ing sizes, denoted here as “Bimodal/CCFD.” If the underly-ing graph contain features that intrinsically “host” hard in-stances, then one would expect that both the scaling and TTScould be different than those on the chimera graph. Figure4 shows that the Bimodal/CCFD instances happen to be eveneasier than the Bimodal instances embedded onto the chimeragraph. This means that the intrinsic hardness of these CCFDinstances most likely is related to the structure and the rela-tionship between the specific biases h and couplers J valuesdefining them. Further studies are being performed to studythis in more detail.

C. Scaling analysis: Application vs physics perspective

Fig. 5 provides insights about the CCFD instances from aphysics and from the application perspective. While in theformer we analyze the scaling of computational resourcesvia the time-to-solution TTS using the number of variables√N , in the latter we analyze the resource requirements by

the application-specific variables, namely the type of multi-plier used. The physics perspective here aims to answer ques-tions about the performance of quantum annealing comparedto other classical solvers on a comparable footing, ignoringfor a moment that the instances were generated from a specificapplication. For example, we compare here the performanceof quantum annealing to other classical and alternative quan-tum solvers on instances represented on a chimera graph (C);similar to previous extensive benchmarking work on syntheticrandom spin-glass instances. We go beyond such studies andprovide as well insights on the performance of quantum an-nealing for the QUBO (Q) instances on their native graph dic-tated by the CCFD application and also on hypothetical quan-tum annealer devices capable of natively encoding up to quar-tic interactions (P).

For the physics scaling analysis we chose to plot the TTS

(a)

(b)

FIG. 5: Scaling analysis from a (a) physics, and (b) an application-centric perspective. The TTS is plotted as a function of

√N , with

N the number of spin variables in each of the problem representa-tions [PUBO (P), QUBO (Q), or chimera (C)]. Panel (b) correspondsto

√Ngates, with Ngates the number of gates regardless if we are

considering symmetric multipliers, mult[n-n] as in Fig. 1, or asym-metric ones (mult[n-m]). The legend for the data sets depicted inpanel (a) is shared with panel (b), with SAT-based results only ap-pearing in panel (b). SQA/C runs were performed with an optimizedlinear schedule, as well as the DW2X schedule, marked with “ls” and“dws” subscripts, respectively, (details in Appendix D).

computational effort as a function of√N , with N being

the problem size in terms of number of spins, regardless ofwhether the problem to be minimized is in a PUBO (P),

7

FIG. 6: Asymptotic scaling analysis. The asymptotic scaling expo-nent bapp refers to the multiplier representation, whereas bphys refersto the physical representation of the problem. Data points correspondto the median values extracted from a bootstrapping statistical anal-ysis from 100 instances per problem size, with error bars indicatingthe 90% confidence intervals (CI).

QUBO(Q), or chimera (C) format. This selection is motivatedby the linear relation between any pair of problem sizes NP,NQ or NC (see Fig. 9) and the fact that the scaling for prob-lems on the quasi-planar Chimera graphs is expected to be astretched exponential, largely due to its tree width∼ √NC, incontrast to a tree width ∼ N characteristic of fully-connectedgraphs [49].

The analysis from the application perspective aims for in-sights on the performance where the sole purpose is to findthe solution to the CCFD problem. Here, it is natural to plotthe TTS computational effort as a function of a characteristicproperty of the circuit scaling with the problem size, regard-less if one considers a symmetric multiplier, mult[n-n] or anasymmetric one, i.e., mult[n-m]. We chose this quantity to bethe number of gates in the circuit, Ngates (or more precisely√Ngates), which is justified given the linear relationship be-

tween Ngates ∝ NP ∝ NC illustrated in Fig. 9, and theexpected stretched exponential behaviour in

√NC discussed

above for chimera graphs.a. Limited quantum speedup — Figure 5(a) compares

the single-core computational effort of SA, PTICM, SQA(with both linear and DW2X schedules), and the experimentalresults obtained with the DW2X quantum annealer. Repre-sented with diamond symbols in Fig. 6 and with values onthe right axis, we plot the asymptotic analysis performed byconsidering only the four largest sizes from each of the datasets. From this scaling analysis and the value of the mainscaling exponent b (slopes of curves in Fig. 5) for the chimera

instances SA/C and DW2X, it can be seen that we also findhere limited quantum speedup (without optimizing annealingschedules) [21] as found for the benchmarks on synthetic in-stances used in the study by the Google Inc. [19]. From thisphysics perspective, there seems to be even a quantum advan-tage when comparing with SA at the PUBO level, SA/P, whichhappen to have a better scaling than both of their quadraticcounterparts, SA/Q and SA/C. The values are close enoughthat one has to be careful because the real bphys (DW2X)might be larger than the calculated in our analysis due tosub-optimal annealing time [13, 21, 50]. On the other hand,note that the quantum advantage at the level of the same rep-resentation where bphys(DW2X) � bphys(SA/C) also holdsagainst any of the optimized SQA/C implementations, eitherwith a linear or DW2X schedule. Although we believe thatit is very unlikely that sub-optimal time can change our lim-ited quantum speedup conclusion because bphys(DW2X) �bphys(SA/C), we note that optimized SQA corroborates theseclaims. This scaling advantage already yields a difference ofapproximately six orders of magnitude on a single-core CPUin the TTS between DW2X and SA/C for the largest problemstudied (mult[4-4]).

b. SQA vs DW2X and impact of the annealing schedule— From a computational prefactor perspective note that thecomputational effort for the DW2X is smaller by six to eightorders of magnitude than the SQA/C implementations with ei-ther linear or the D-wave schedule. It is important to note thatthe TTS in Figs. 5(a) and 5(b) is for SQA as a classical com-putational solver. For a fair comparison of the scaling of SQAto that of a physical quantum annealer such as the DW2X,the SQA TTS results must be divided by N to account forthe intrinsic parallelism in quantum annealing, as illustratedin Fig. 8. Further analysis of the scaling comparison of SQAand the DW2X device can be found in Appendix D.

Figures 5(a) and 5(b) illustrate that the selection of a poorschedule (the D-wave schedule in this case in comparison tosimpler linear one) can have a significant impact in the com-putational efficiency of SQA as as classical computationalsolver. As discussed in Appendix D, most likely the differ-ence is only at the level of a prefactor and most likely it isnot a scaling advantage. Whether there are schedules that canchange the asymptotic scaling remains an open question. InAppendix D we also discuss that although there seems to bea scaling advantage of the DW2X over the SQA simulations,the results are also inconclusive given that the scaling of theDW2X might be slightly different due to any sub-optimal an-nealing times. We leave it to future work to optimize the an-nealing time of the DW2X because it is beyond the scopeof this work given the sizable computational requirementsneeded.

c. Quantum annealing performance for Hamiltonianswith higher-order interactions — A question not addressedto date is the performance comparison between quantum an-nealing architectures with 2-local and k-local (k > 2) interac-tions within the scope of real-world applications. For exam-ple, the CCFD mapping used in this work (see Appendix Cfor details) natively contains cubic (3-local) and quartic (4-local) interactions and one might think a quantum annealer

8

natively encoding those might have an advantage over 2-localterms. Perhaps one of the most remarkable findings in thisstudy from our SQA simulations is that working directly witha Hamiltonian containing such quartic interactions does notseem to help quantum annealing with a transverse field, be-cause bphys(SQA/Q) < bphys(SQA/P). Note that this result isin contrast to the behavior of the classical algorithms consid-ered here. In the case of SA there seems to be an advantagefor solving the instances in the PUBO representation over theQUBO [bphys(SA/Q) > bphys(SA/P)]. In the case of PTICM,bphys(PTICM/Q) ≈ bphys(PTICM/P). These remarks on thephysics scaling have a significant impact on the scaling fromthe application perspective. Note that while in all the classicalmethods there is a clear preference to solve the problem in thePUBO representation, the case of SQA shows no advantagefor the quantum annealer in the PUBO representation. On thecontrary, as shown in Fig. 11 for the higher percentiles abovethe median, there seems to be a slight preference of SQA/Qover SQA/P not only in the absolute value of computationaleffort measured in TTS [see last data point in Fig. 11(d)], butalso in scaling terms.

The insight to be extracted from the SQA simulations in thecontext of this CCFD application is that simply adding higher-order terms would not necessarily imply any enhancement inthe performance. This result is striking for two reason.

First, because in the application scaling we plot theTTS results vs

√Ngates, then when changing representa-

tions from PUBO to QUBO, there is a natural tendency forbapp(Q)/bapp(P) > bphys(Q)/bphys(P). This is because NQ isalways greater thanNP, and therefore even in the case of com-parable physics scaling slopes as is the case of PTICM withbphys(PTICM/Q) ≈ bphys(PTICM/P) this would imply that

TTSPTICM/P ∼ ebphys

√NP ∼ ebphys

√αP←g

√Ngates ,

while

TTSPTICM/Q ∼ ebphys

√NQ ∼ ebphys

√αQ←PαP←g

√Ngates ,

valid in the asymptotic limit Ngates � 1 of interest here.Therefore,

bQapp = bphys√αQ←PαP←g > bPapp = bphys

√αP←g.

Here we have used that, in this limit, NP ∼ αP←gNgates andNQ ∼ αQ←PNP, and both, αQ←P and αP←g are greater than1, as shown in Fig. 9.

Second, the penalties of the locality reduction ancillaschange the energy scale and it is expected that stochasticsolvers such as SA (which heavily depend on the barriers inthe energy landscape) also suffer from the new QUBO energylandscape with taller barriers. This is indeed what we observebecause bphys(SA/Q)> bphys(SA/P). Note that PTICM seemsto be more resilient to these barriers and, as discussed before,bphys(PTICM/Q) ≈ bphys(PTICM/P). Both of these drivingforces would imply that the application perspective scalingof quantum annealing working in the PUBO representationshould be better than in the QUBO representation and it isnot what we observe here. This second explanation is reason-able and a good indication that SQA is doing a good job at

not 11feeling” these taller barriers, something that could beexplained by means of quantum tunneling.

From the first argument it follows that bapp(SQA/Q) ≈bapp(SQA/P), implying that bphys(SQA/Q) < bphys(SQA/P)which is quite distinctive and different from what we observein the classical approaches. It is clear that SQA is having aharder time traversing the PUBO energy landscape and find-ing the ground state in this representation, despite the smallerproblem size. One plausible explanation is that the transverse-field implementation is not powerful enough to take advantageof the compactness of the PUBO energy landscape. We thusemphasize that any development of new architectures with k-local couplers with k > 2 should be accompanied by develop-ments in more sophisticated driver Hamiltonians, such as theinclusion of non-stoquastic fluctuations.

d. Impact of the limited connectivity — Here we addressthe issues that occur with limited-connectivity hardware (seeFig. 3). From the physics scaling perspective, Fig. 5(a) andFig. 6 show that there are no major effects in solving the prob-lems with the QUBO or with the chimera representation. Thisseems to be a common feature across classical and quantumapproaches. Following the argument just previously made inthe case of the PUBO vs QUBO discussion, we show thatbphys(Q) ≈ bphys(C) and NC ∼ αC←QNQ, implies thatbapp(C) > bapp(Q). Here, αC←Q = 3.5026 from Fig. 9.Although it has always been expected that more connectivityshould be better, to the best of our knowledge this is the firstdemonstration that having a quantum annealing device withmore connectivity can have a significant impact when solvingreal-world applications. Our results, within the context of theCCFD application, show that the advantage here is not sim-ply an overall prefactor improvement in the TTS but that animportant asymptotic scaling advantage is expected as well.

e. Comparison of quantum annealing with generic andtailored algorithms for CCFD — The main question thatmotivated this study was if quantum annealing can efficientlysolve CCFD problems. Figures 5(b) and 6 show that fromthe application perspective the scaling of the DW2X quan-tum annealer and of any of the SQA variants considered heredoes not look favorable for quantum annealing. In fact, theDW2X does not even scale better than simulated annealing(SA/P). One of the major challenges for devices with a smallfinite graph degree connectivity (such as the DW2X with thechimera topology) is that to solve real-world applications itcarries all the qubit overhead from the transformations PUBOto QUBO over to chimera. However, the application scal-ing can be improved. For example, reducing the number ofqubits needed to represent the application would most likelyimprove the scaling performance. In Sec. C we present an al-ternative and more efficient mapping that we aim to explorein further studies. It is important to note that the new map-ping improves the performance of SA and PTICM accord-ingly. Note that from all the approaches, the most efficientwith the best scaling is a SAT-based solver developed by ourteam for this study which excels in this strong-fault model-based diagnosis of multiplier circuits. The SAT-based solverdoes not depend on any of the PUBO, QUBO or Chimera rep-resentation because it has the advantage of constructing its

9

own variable representation and set of satisfiability constrainsdirectly from the propositional logic level shown in panel (a)of Fig. 3. Although all other classical stochastic solvers used(SA, PTICM, and SQA) might work directly with the proposi-tional logic as well, the evaluation of the cost function wouldbe highly nonlocal compared to the evaluation of the differ-ence in energy required for the Metropolis update in the caseof the polynomial evaluation. By nonlocal we mean that if wewere to work with only the fault variables and use the propo-sitional logic instead of constructing the PUBO and includingthe internal wire variables, then we would need to propagatefrom inputs all the way through each gates and their healthstatus assignment to obtain the predicted outputs and subse-quently an effective energy that can be used in the Metropolisupdate. For every pair considered in an Metropolis update, thewhole process need to be applied and later subtracted. In con-trast, in any of the polynomial representation, because all thefaults and wires variables are considered, the evaluation of theenergy difference can be applied very efficiently by only con-sidering the few terms that change the energy by the respec-tive variable flip. Most importantly, because the main point ofthis contribution is to compare with algorithms that could beimplemented in QA architectures, we did not explore this im-plementation. We leave it as an open question whether algo-rithms like SA can have a better scaling on that propositionallogic representation.

IV. CONCLUSIONS

Regardless of the substantial efforts in benchmarking, thestudy of early generation quantum annealers has been doneexclusively with synthetic spin-glass benchmarks. However,comprehensive studies comparing several quantum and classi-cal algorithmic approaches, including state-of-the-art tailoredsolvers for real-world applications, had been missing. In thiswork we present a comprehensive benchmarking study on aconcrete application, namely the diagnosis of faults in digitalcircuits, referred in the main text as CCFD. More specifically,we provide insights on the performance of quantum annealingin the context of the CCFD instances by performing an asymp-totic scaling analysis involving five different approaches:quantum annealing experiments on the DW2X compared tothree classical (SA, PTICM, and a CCFD-tailored SAT-basedsolver), and extensive QMC simulations, most of them onthree different problem representations (PUBO, QUBO onthe native CCFD graphs, and QUBO on the DW2X chimeratopology), for instances of multiplier circuits of varying size.It is important to note that by asymptotic analysis we refer toconclusions drawn from the largest problem sizes accessibleto us to experiment with in each of these approaches.

We have analyzed the problem with two foci: a physics per-spective and an application-centric perspective. The empha-sis of the physics perspective is similar to previous represen-tative benchmark studies [8–20, 50], that aim at probing thecomputational resources of quantum annealing, and to answerquestions such as whether it is even possible in synthetic datasets to prove an asymptotic quantum speedup, or to address

the role of quantum tunneling, among other open questionsin the field. Within our physics perspective we add severalissues not thoroughly considered in other benchmark studies.For example, what is the impact in the computational scalingof solving the problem directly with Hamiltonians nativelyencoding many-body interactions beyond pairwise as thosenaturally appearing in real-world applications? What is theimpact in the scaling from solving the problem instances on(hypothetical) physical hardware with different qubit connec-tivity constrains, e.g., by comparing the quantum annealingperformance on connectivity graphs dictated by the CCFD in-stances and the minor-embedded representation in the DW2XChimera graph?

From this physics perspective we show that our instancesare hardest when compared to any of the proposed randomspin-glass instances (see Fig.. 4). Intrinsic hardness is oneof the long sought-after features when performing benchmarkstudies [11, 16, 20], therefore making our CCFD instancescurrently the best candidate for benchmarking the next gener-ation of quantum annealers. In particular, because these prob-lems stem from real-world applications, in contrast to randomsynthetic benchmarks on the native D-wave’s chimera graphwhich have been dulled not only for giving an advantage to thehardware but also for lacking practical importance [51, 52].

We also address the question of whether SQA can repro-duce the scaling of the DW2X for the CCFD application. Al-though the results in Fig. 8 might lead to the conclusion thatclearly SQA has a different scaling than the DW2X, the factthat most likely the DW2X is running at a sub-optimal anneal-ing time might be distorting the scaling and resulting in a bet-ter apparent scaling. More extensive studies with enough datapoints — where one can optimize for the optimal annealingtime — might reveal the real scaling of the device. Althoughthis is, in principle, feasible on quantum annealers, the mainchallenge might rely on SQA simulations which are already atthe limit of what is computationally feasible. In Appendix Dwe discuss the apparent different scaling of SQA with a lin-ear schedule compared to SQA with the same schedule as theD-Wave device and the challenges on drawing any meaning-ful conclusions about the difference in scaling between theDW2X and SQA. From the choice of schedule perspective,we find that within SQA as a solver, the linear schedule seemsto be more efficient, but most likely not bringing any scalingadvantage.

When compared on the same representation (either native-QUBO or Chimera-QUBO) we show that both, SQA and theDW2X have a limited quantum speedup by showing a scal-ing advantage over SA. We arrive at this conclusion assumingthe DW2X scaling obtained here is not drastically affected bythe non-optimal annealing time, which is very unlikely dueto the large difference in the slopes between SA and SQAand DW2X. These results confirm the presence of quantumtunneling in the DW2X; a quantum speedup restricted to se-quential algorithms [21] similar to the Google Inc. study onthe weak-strong clusters instances [19]. One important high-light here is that ours is the first demonstration on instancesgenerated from a concrete real-world application and wherethe multi-spin co-tunneling needs to happen more often on

10

the strongly ferromagnetically coupled physical qubits encod-ing the logical units from the original QUBO problem. Al-though it is encouraging to see that such co-tunneling eventsseem to be happening in the hardware at the problem sizesconsidered here, the minor-embedding mapping logical vari-ables into physical qubits in the hardware usually involves thegeneration of long “chains.” Further studies need to be per-formed using larger instances to see if this advantage remains,and where longer “chains” with 10 or more qubits would bemore frequent. The comparison against other generic solverslike PTICM or tailored solvers like the SAT-based solver de-veloped here, were not favorable for our SQA simulations andDW2X experiments. It is important to consider that both,SQA simulations and DW2X experiments, were done withstoquastic Hamiltonians as the only ones available in currenthardware. It is expected that non-stoquastic Hamiltonians willbring a boost in performance [53, 54], although it is an openquestion if they will have any asymptotic scaling advantage.

The application-centric perspective is more challenging andraises the bar significantly for quantum annealers. Here, wefind that the tailored SAT-based algorithm performs best. Wenote that the performance is even better that the PTICM algo-rithm, which is currently the state-of-the-art in the field [55].Although the results using quantum optimization approachesas seen from the application perspective are not that encour-aging, our study suggests next steps to be taken in the field ofquantum optimization. First, there is a clear need for higher-connectivity devices. Second, our SQA results suggest thatadding higher-order qubit interactions [42, 56] to new hard-ware might require also the addition of more complex drivingHamiltonians.

This rather extensive study should be considered as a base-line for future application studies. We do emphasize, however,that the conclusions should be interpreted within the contextof the particular CCFD application. Furthermore, the resultsare for the specific case of conventional QA with a trans-verse field driver. The poor performance of quantum anneal-ing should be seen as an incentive for the community to ad-dress important missing ingredients in the search for quantumadvantage for real-world applications. Other variable efficientmappings (as shown in Appendix C 2) could also provide aperformance boost. We note that the latter should also pro-vide an advantage for classical solvers, because larger systemscould be studied. A detailed performance comparison of ourCCFD benchmarks to other mapping strategies [29] will bedone in a subsequent study.

Further adding other features, such as better control of theannealing schedules via “seeding” of solutions [57], and thesubsequent developments of classical-quantum hybrid heuris-tic strategies [57–60] will likely lead to breakthroughs inquantum optimization. However, more simulations are neededto guide the design of new machines.

Acknowledgments

The work of A.P.O. was supported in part by the AFRLInformation Directorate under grant F4HBKC4162G001, the

Office of the Director of National Intelligence (ODNI),and the Intelligence Advanced Research Projects Activity(IARPA), via IAA 145483. Z.Z. and H.G.K. acknowl-edge support from the National Science Foundation (GrantNo. DMR-1151387). The work of H.G.K. and Z.Z. is sup-ported in part by the Office of the Director of National Intelli-gence (ODNI), Intelligence Advanced Research Projects Ac-tivity (IARPA), via MIT Lincoln Laboratory Air Force Con-tract No. FA8721-05-C-0002. The views and conclusionscontained herein are those of the authors and should not beinterpreted as necessarily representing the official policies orendorsements, either expressed or implied, of ODNI, IARPA,AFRL, or the U.S. Government. The U.S. Government is au-thorized to reproduce and distribute reprints for Governmen-tal purpose notwithstanding any copyright annotation thereon.We thank Delfina Garcia-Pintos for the implementation of thelibraries used in the bootstrapping analysis of the slopes andtheir confidence intervals of our asymptotic analysis. We alsothank Tayo Oguntebi for initial support and discussions re-lated to this CCFD application. We also thank Catherine Mc-Geoch for feedback on an earlier version of this manuscript.

Appendix A: Quantum Annealing for combinatorialoptimization problems

The quantum hardware employed consists of 144 unit cellswith eight qubits each, as characterized in Refs. [8, 61]. Post-fabrication characterization determined that only 1097 qubitsfrom the 1152 qubit array can be reliably used for computa-tion, as shown in Fig. 7. The array of coupled superconductingflux qubits is, effectively, an artificial Ising spin system withprogrammable spin-spin couplings and magnetic fields. It isdesigned to solve instances of the following (NP-hard [62])classical optimization problem: given a set of local longitudi-nal fields {hi} and an interaction matrix {Jij}, find an assign-ment s∗ = s∗1s

∗2 · · · s∗N , that minimizes the objective function

E : {−1,+1}N → R, where

E(sC) =∑

1≤i≤N

hisi +∑

1≤i<j≤N

Jijsisj . (A1)

Here, |hi| ≤ 2, |Jij | ≤ 1, and si ∈ {+1,−1}. The sub-script “C” is to emphasize that the spins are within the chimeragraph, and to differentiate these from the other two represen-tations studied in the paper at the PUBO (sP) and QUBO (sQ)level, respectively.

Finding the optimal set of variables s∗ is equivalent tofinding the ground state of the corresponding Ising classicalHamiltonian,

Hp =

N∑1≤i≤N

hiσzi +

N∑1≤i<j≤N

Jijσzi σ

zj , (A2)

where σzi is a Pauli z matrix acting on the ith spin.Experimentally, the time-dependent quantum Hamiltonian

implemented in the superconducting-qubit array via

H(τ) = A(τ)Hb +B(τ)Hp, τ = t/ta, (A3)

11

with Hb = −∑i σ

xi the transverse-field driving Hamiltonian

responsible for quantum tunneling between the classical statesconstituting the computational basis, which is also an Eigen-basis of Hp. The time-dependent functions A(τ) and B(τ)are such that A(0) � B(0) and A(1) � B(1). In Fig.8(a),we plot these functions as implemented in the experiment. tadenotes the time elapsed between the preparation of the initialstate and the measurement, referred to hereafter as the anneal-ing time.

QA as an algorithmic strategy to solve classical optimiza-tion problems exploits quantum fluctuations and the adiabatictheorem of quantum mechanics. This theorem states thata quantum system initialized in the ground state of a time-dependent Hamiltonian remains in the instantaneous groundstate if the Hamiltonian changes sufficiently slow. Becausethe ground state of Hp encodes the solution to the optimiza-tion problem, the idea behind QA is to adiabatically pre-pare this ground state by initializing the quantum system inthe easy-to-prepare ground state of Hb, which correspondsto a superposition of all 2N states of the computational ba-sis, and then slowly interpolating to the problem Hamiltonian,H(τ = 1) ≈ Hp.

In a realistic experimental implementation, the quantumprocessor will operate at a finite temperature, and in addi-tion to thermal fluctuations, other types of noise are unavoid-able, leading to dissipation processes not captured in H(t).Deviations from adiabaticity affecting the performance of thequantum algorithm seem to be a delicate balance between thequantum coherence effects and the interaction with the envi-ronment, responsible for, e.g., thermal excitation (relaxation)processes out of (into) the ground state [23, 63].

Determining the optimum value of ta is an important andnontrivial problem in itself. To the best of our knowledge thisquestion related to the scaling of ta in a noisy environmentis still largely unexplored, with progress only in the case ofcanonical models [64]. From an experimental standpoint, themain limitation is the limited size of the available quantum de-vices, but now with new generation of devices with more than2000 qubits the question is within reach in the case of syn-thetic data sets [50]. Studying this question within the contextof real-world applications is now within reach. We will leavethis study for future work.

Appendix B: Methods

a. Simulated Quantum Annealing (SQA) — QMC sim-ulations are performed using a variant of the continuous timeQMC algorithm [34], which we refer to here and in the maintext as SQA. We build clusters in the imaginary time directionin the same way as done in Ref. [34]. However, because herewe study frustrated systems, we do not build clusters in thespatial directions. To flip segments of finite imaginary timeextent, we use the Metropolis algorithm [65]. This algorithmwas also used in previous benchmark studies of the D-Wavedevices [12, 13].

In the SQA simulations we use a linear schedule. We fix

FIG. 7: Device architecture and qubit connectivity. The array ofsuperconducting quantum bits is arranged in 12 × 12 unit cells thatconsist of 8 quantum bits each. Within a unit cell, each of the 4 qubitson the left-hand partition (LHP) connects to all 4 qubits on the right-hand partition (RHP), and vice versa. A qubit in the LHP (RHP) alsoconnects to the corresponding qubit in the LHP (RHP) of the unitscells above and below (to the left and right of) it. Edges betweenqubits represent couplers with programmable coupling strengths. Weshow only the 1097 functional qubits out of the 1152 qubit array.

the diagonal interaction strengthB(τ) = 1 and vary the trans-verse field strength as A(τ) = Γ0(1 − τ), where τ is the an-nealing time and Γ0 is the initial transverse field strength; seeFig. 8. We use different values of Γ0 for different problem rep-resentations. Γ0 = 0.8 for PUBO, Γ0 = 1.6 for QUBO, andΓ0 = 6 for instances on the chimera graph. These represen-tations are referred as “P,” “Q,” and “C,” respectively, in themain text. In addition, we also implement the A(τ) and B(τ)annealing schedule used in the DW2X device, as depicted inFig. 8.

12

b. Estimation of the time-to-solution (TTS) — Forstochastic algorithms, the time-to-solution depends on the de-sired confidence, i.e., the probability P required such that thesolver produces the target solution. For example, in all previ-ous studies the level of certainty required from the solver was99%, i.e., P = 0.99, and the relevant metric, denoted R99 isthe number of repetitions needed such that the probability thatthe solution is found is at least once is 99%. Let us denoteby ps the success probability to obtain the target solution in asingle execution or repetition of the solver. Because the prob-ability F of not observing the solution after RP repetitions isF = (1 − ps)

RP = 1 − P , the number of repetitions R99

needed to obtain the desired solution with probability at least99% is

R99 = d log(1− 0.99)

log(1− ps)e. (B1)

Therefore, the time-to-solution (TTS) under this criteria is theproduct of R99 times the time it takes to perform one execu-tion or each repetition, trep:

TTS = trepR99. (B2)

For the DW2X, trep was set to the annealing time of 5 µs. ForSA, QMC, and PTICM, trep can be estimated as:

trep = tsuNMCSopt, (B3)

with MCSopt the optimal number of Monte Carlo sweeps(MCS), i.e., the number of MCS that minimizes the TTSand tsu the time it takes to make a MC spin update. In thecase of PTICM, the values of MCSopt include a factor of 120coming from the 4 replicas and 30 temperatures consideredin our implementation. Additionally, we multiply by a fac-tor of 1.2 to account an estimated 20% overhead coming fromother steps in the PT implementation and not present in SA,such as swaps of configurations and cluster updates. Herewe optimize the TTS per instance to obtain the best scal-ing for each algorithm. The computational effort of the al-gorithm optimization of this procedure compared to optimiz-ing over different annealing times as proposed in Ref. [13]should yield comparable scaling results. We prefer this ap-proach because it required the same computational effort asanalyzing the data at different annealing times and it providesmore reliable information on the intrinsic difficulty of eachinstances. For example, in the limit of very large annealingtime, where all the instances have probability one, most in-stances have the same computational effort and it is not possi-ble to identify which instances are intrinsically harder. This isrelated to the problem with reported DW2X scaling for smallinstances, for which the minimum available annealing time isgreater than optimal. Note that ps is a function of the numberof MCS, in the same way that it is a function of the anneal-ing time in the case of the DW2X. Therefore, we estimateps for different values for the number of MCS, calculate R99

and from the values considered we select the optimum. Sinceone MCS involves N updates [66], with N the total numberof spins in the problem, then to calculate the computationaleffort we need to multiply by N and by the effective time

it takes to perform and evaluate each of these updates. Thevalue tsu is different for each of the algorithms (e.g., SA vsSQA) and for each of the different representations (QUBO,PUBO, or chimera). The times estimated and used for thecase of the CCFD instances are: tSA/Psu = t

PTICM/Psu = 5.5 ns,

tSA/Qsu = t

PTICM/Qsu = 3.42 ns, tSA/Csu = t

PTICM/Csu = 2.6 ns,

tSQAls/Psu = 1.08µs, tSQAls/Q

su = 1.88µs, tSQAls/Csu = 1.81µs,

tSQAdws/Csu = 48.8µs. These times were used for all figures

with the exception of Fig. 4 and Fig. 8. For the case of Fig. 4,and to give the best performance for each data set, we opti-mized for R99 as described above for every instance of eachof the CCFD and random spin-glass data sets. Given that allthe data sets were run with PTICM and under the same com-putational resources, we plotted directly the wall-clock timerequired after the aforementioned optimization of MCSopt.

To capture the computational scaling of SQA as a simulatorof a hypothetical quantum annealer [denoted as SQA(q)] weused the same optimal values of MCSopt used for SQA butwe do not multiply by the factor of N . In this way we takeinto account the intrinsic parallelism of quantum annealers.The prefactor tsu is changed as well to an arbitrary constantparameter, denoted tSQA(q) we can tune to make all the linesin Fig. 8 to have a similar TTS as that value obtained by theDW2X device. The values used here were tSQA(q)ls = 5 nsand tSQA(q)dws

= 1.3 ns.Because the SAT-based solver described below is signifi-

cantly different from the other stochastic solvers mentionedabove, to estimate the TTS we run the SAT-solver 1000 timesper instance and compute the TTS for each run. From thisdistribution of TTS values, we pick the 99% percentile as theTTS value we report since it matches the definition of thetime needed to observe the desired solution

c. SAT-based solver tailored for CCFD — The SAT-based model-based diagnosis solver is implemented as fol-lows. First it adds a tree-adder to the fault-augmented circuitto enforce the cardinality of the fault. Second, the formulais converted to Conjunctive Normal Form (CNF). Finally, aSAT solver is called n times, first for computing all zero-cardinality faults, then for all single-faults, etc., until a faultof cardinality n is found. For our implementation we haveused the highly optimized SAT-solver LINGELING [55]. It is adeterministic SAT-solver that uses highly-optimized Booleansearch enhancements, including symbolic optimization, oc-currence lists, literal stack, and clause distillation, etc.

d. DW2X programming details — When programminga quantum annealer to solve real-world applications, the pro-cess of minor-embedding introduces many other parametersthat do not exist when benchmarking quantum annealing witha random spin-glass benchmark. One common misconceptionis that implementing real-world applications is harder becauseof the minor-embedding procedure. Although more efficientembedding strategies are always desirable, we want to empha-size here that it is not the main challenge when programmingthe device since heuristic algorithms solve this problem rea-sonably well [45]. It is also important to note here that the NP-hardness of finding the smallest minor-embedding (with re-spect to number of qubits) is largely moot, because the small-est minor-embedding is often far from optimal in terms of per-

13

formance. For example, from our experience, sometimes itis preferable to have an embedding that uses more physicalqubits but that has shorter “chains” representing logical vari-ables. In our work we generated 100 embeddings per instanceregardless of the problem size.

The main challenge (the curse of limited connectivity [67,68] due to quantum annealers having a bounded number ofcouplers per qubit) does not lie in the minor embedding prob-lem, but rather in the setting of the additional parameters oncethe minor-embedding has been chosen. Although proposalsexist to cope with this challenge [46, 69, 70], the optimal set-ting of parameters is a largely open problem and one of themost important ones affecting the performance of quantumannealers as optimizers [46]. In this work, we use the strategyproposed in Ref. [46] to set the strength JF of the ferromag-netic couplers, which enforce the embedding, and for gaugeselection.

In addition to setting JF, we must also distribute the logicalbiases {hi} and couplings {Ji,j} over the available physicalbiases and couplings {hk} and {Jk,l}. The key considerationin parameter setting is the noise level of the programmableparameters of the quantum device. The noise margin of theD-Wave 2X machine is hj < 0.05 for biases and Jk,l < 0.1for couplers in a normalized, hardware-embedded problem,with the difference due to the difference in dynamic range.

We aim to divide the logical parameters as much as possi-ble over the corresponding physical parameters subject to thisprecision limit using the following heuristic, which is similarto but distinct from that of Ref. [70]. Consider a logical biashi that corresponds to Ni hardware qubits. If hi/Ni is greaterthan the 0.05 noise threshold, then each physical qubit j isgiven a bias hj = hi/Ni. If not, we consider the ni hard-ware qubits within the chain that have nonzero inter-chaincouplings. If hi/ni is greater than the threshold, we evenlydistribute the logical bias amongst these ni physical qubits.Finally, if neither of these strategies exceed the threshold, weassign the logical bias completely to hardware qubits withthe lowest number of intra-chain couplings, breaking ties uni-formly at random. The remaining hardware qubits within thechain are given a bias of zero. We distribute the logical cou-plers Ji,j in a similar way. Suppose that the chains for logicalqubits i and j have Ni,j physical couplers between them. IfJi,j/Ni,j is greater than 0.1 (noise threshold), we evenly dis-tribute the logical coupling amongst the Ni,j available physi-cal couplers. Otherwise, the logical coupling is completely as-signed to a single physical coupler uniformly at random fromthe Ni,j options.

Appendix C: Mapping of minimal-cardinality fault diagnosisfor combinational digital circuits to PUBO and QUBO

In this section we describe in detail two mappings of thefault diagnosis problem to QUBO, via a mapping to PUBO.The original instance consists of a set of m gates, each with aspecified hard fault model. Excluding the inputs and outputsto the circuit, let x = (xi)

ni=1 ∈ {0, 1}

n indicate the valueon every wire in the circuit. For gate i, let yi ∈ {0, 1}∗ be

the values of the input wires and zi the value of the outputwire. These are not new variables but rather alternative waysof referring to the variables x. For example, if wire i is theoutput of gate j and the first input into gate k, then xi, zj ,and yk,1 all refer to the same variable. Let gi(yi) ∈ {0, 1}be the Boolean function indicating the action of gate i, andFi(yi, zi) ∈ {0, 1} be the predicate indicating whether thecombined input yi and output zi are consistent with the faultmodel for gate i. Several examples for gi and Fi are given inTables I and II, respectively.

TABLE I: Example gates and their representation as polynomials.For details see the main text.

Gate gi(yi)

OR yi,1 + yi,2 − yi,1yi,2AND yi,1yi,2

XOR yi,1 + yi,2 − 2yi,1yi,2

EQ 1− yi,1 − yi,2 + 2yi,1yi,2

BUFFER yi,1

NOT 1− yi,1NOR 1− yi,1 − yi,2 + yi,1yi,2

NAND 1− yi,1yi,2

Bian et al. [29] have also used fault diagnosis as a testbed for benchmarking novel techniques in quantum annealing.They used Satisfiability Modulo Theory to automatically gen-erate functions representing the cost function and constraints,whereas here we do so manually, as described in this section.Their approach is further differentiated from the present oneby their use of problem decomposition and locally-structuredembedding.

Note that we describe the mapping to pseudo-Boolean poly-nomials over variables taking the values {0, 1}, while theHamiltonians in physical quantum annealers directly repre-sent functions of variables taking the values ±1, i.e., Isingspins. The two representations are equivalent with the follow-ing transformation:

b = (1− s)/2, s = 1− 2b, (C1)

for b ∈ {0, 1} and s ∈ {±1}, with the latter being the con-ventionally used for physical implementations on quantum an-nealers, as in, e.g., Eq. (A1). Note that the substitutions leavethe degree and connectivity of the polynomials unchanged.

TABLE II: Example fault models and their predicates as polynomi-als. For details see the main text.

Fault model Fi(yi, zi)

Stuck at 1 zi

Stuck at 0 1− ziStuck at 0 or 1 1

Stuck at first input EQ(zi, yi,1)

Stuck at first input or 0 1− yi,1(1− zi)

14

1. Explicit mapping

For each gate i, introduce an additional variable fi thatindicates whether or not that gate is faulty. Assuming thatf = (fi)

Ngates

i=1 is consistent with xi, the number of faults issimply

Hnumfaults(f) =

Ngates∑i=1

H(i)numfaults(fi) =

Ngates∑i=1

fi. (C2)

The consistency with the fault model is enforced by thepenalty function

Hfaultset(x, f) =

Ngates∑i=1

H(i)faultset(yi, zi, fi),

H(i)faultset(yi, zi, fi) = λ

(i)faultsetfi [1− Fi(yi, zi)] .

(C3)

Finally, we must also constrain the system to the appropriatebehavior when there is no fault:

Hgate(x, f) =

Ngates∑i=1

H(i)gate(x, f),

H(i)gate(yi, zi, fi) = λ

(i)gate(1− fi)XOR[gi(yi), zi].

(C4)

The overall cost function isH(x, f) = Hnumfaults(f) +Hfaultset(x, f) +Hgate(x, f)

=

Ngates∑i=1

H(i)(yi, zi, fi),

(C5)

where

H(i)(yi, zi, fi) = Hnumfaults(fi) +Hfaultset(yi, zi, fi)

+Hgate(yi, zi, fi).

(C6)

Note that, in general, this function is quartic. Using two an-cilla bits per gate, the usual gadgets [22, 71] can be used toreduce this to quadratic as needed. Depending on the circuit,some ancilla bits may be reused to reduce the degree of theterms corresponding to more than one gate. For example, ifthe input yi = (yi,1yi,2) to gate i happens to be the same in-put to another gate j, then a single ancilla bit correspondingto yi,1yi,2 may be used for both gates. In this work, we useexactly two ancilla bits per gate, corresponding to the con-junctions yi,1yi,2 and zifi.

The explicit mapping is easily extended to the case of ν > 1input-output pairs. Instead of the single x, we have a copy xιfor each input-output pair, and use a single set of shared faultvariables f . Hnumfaults remains exactly the same as above,while now there are copies of Hfaultset and Hgate for eachinput-output pair:

H(i)faultset(yi, zi, fi) =

ν∑ι=1

H(i,ι)faultset(yi,ι, zi,ι, fi),

H(i,ι)faultset(yi,ι, zi,ι, fi) = λ

(i)faultsetfi [1− Fi(yi,ι, zi,ι)] ;

(C7)

and

H(i)gate(yi, zi, fi) =

ν∑ι=1

H(i,ι)gate(yi,ι, zi,ι, fi),

H(i,ι)gate(yi,ι, zi,ι, fi) = λ

(i)gate(1− fi)XOR [gi(yi,ι), zi,ι] ;

(C8)

where yi,ι and zi,ι are input and output bits for gate i in xι,and yi and zi contain all ν copies thereof.

The explicit mapping is also easily extended further to thecase of µ > 1 fault modes. For each gate i, we use µ faultvariables fi = (fi,α)

µα=1, corresponding to the fault modes

(Fi,α)µα=1. Considering fi =

∑µα=1 fi,α as a function of fi

(rather than a separate bit on its own), Hnumfaults and Hgate

remain unchanged from the single-fault case, even with mul-tiple input-output pairs. Now there are µ copies of Hfaultset:

H(i)faultset(yi, zi, fi) =

ν∑ι=1

µ∑α=1

H(i,ι,α)faultset(yi,ι, zi,ι, fi,α),

H(i,ι,α)faultset(yi,ι, zi,ι, fi,α) = λ

(i)faultsetfi,α [1− Fi,α(yi,ι, zi,ι)] .

(C9)

Finally, to penalize situations in which more than one fault bitis set per gate, we add

H(i)multfault(fi) = λ

(i)multfault

µ−1∑α=1

µ∑β=α+1

fi,αfi,β . (C10)

So long as λ(i)multfault > νλ(i)gate, Hmultfault will outweigh the

potentially negativeHgate as needed. For each gate i, ν(1+µ)ancilla bits suffice, corresponding to the conjunction of thebits yi,ι for each input-output pair ι and to the conjunctionzi,ιfi,α for every ι and mode α.

When the fault modes considered are simply stuck at 1 orstuck at 0, i.e. Fi(yi, zi) = Fi(zi) = zi or 1−zi, respectively,we can use the alternative

H(i,ι)gate = λ

(i)gate {1 + fi[1− 2Fi(zi,ι)]}XOR[gi(yi,ι), zi,ι],

(C11)where fi =

∑µα=1 as before. When Fi is linear in zi, this

expression is quadratic in gi, zi, and fi, so that it suffices toreduce gi to linear using a single ancilla bit corresponding tothe conjunction of the input bits yi. Overall, only ν ancillabits are needed per gate.

2. Implicit mapping

Having the fault bits f are not necessary. Here we showhow to construct the requisite energy functions using just thewire bits x. Note that Hfaultset was used only to enforce con-sistency of the fault bits with the wire bits, and so is obvi-ated by the omission of the former. Recall that we would liketo find the assignment of values to the wires that minimizesthe number of faults while being consistent with the nomi-nal gates and fault models. Therefore, we need a function

15

H(i)numfaults that is zero when zi = gi(yi) and is one when

zi 6= gi(yi) and Fi(yi, zi). Its behavior when zi 6= gi(yi)and not Fi(yi, zi) only need be non-negative; penalizing thatcase is left to Hgate. The following meets our needs:

Hnumfaults(x) =

Ngates∑i=1

H(i)numfaults(yi, zi)

=

Ngates∑i=1

Fi(yi, zi)XOR[gi(yi), zi].

(C12)

To penalize the case when the output zi of gate i is inconsis-tent with the input yi but not in a way allowed by the faultmodel, we use

H(i)faultset(yi, zi) = λ

(i)gate[1− Fi(yi, zi)]XOR[gi(yi), zi].

(C13)The overall energy function for each gate is simply

H(i)(yi, zi) = H(i)numfaults(yi, zi) +H

(i)gate(yi, zi). (C14)

Each H(i) is cubic, and can be reduced to quadratic using asingle ancilla bit. As with the explicit mapping, in certaincases a single ancilla may be shared among multiple gates.

For a single input-output pair, the implicit mapping natu-rally generalizes to multiple fault modes, by considering acombined fault mode that is the conjunction of the multipleones, i.e., using Fi = OR(Fi,1, . . . , Fi,µ). Some examples,e.g., stuck at one or first input, are shown in Table II. This doesnot apply to multiple input-output pairs because it does not en-force that all copies are subject to the same fault mode. Forparticular gates and sets of fault models, it is likely most effi-cient to use a modification of the explicit mapping, as shownfor the stuck at 0 and stuck at 1 cases above.

3. Logical penalty weights

Without loss of generality, in this work we have chosenonly one penalty weight λ for both λ

(i)gate, which penalizes

a mismatch between the input and output of a gate in theabsence of a fault, and λ

(i)faultset, which enforces the fault

model. That is, λ(i)gate = λ(i)faultset = λ for all i. Setting

λ = Ngates + 1 suffices to guarantee that the global min-ima correspond to a valid diagnosis, i.e., those solutions (x, f)such that Hgate(x, f) = Hfaultset(x, f) = 0. Any valid diag-nosis has energy H(x, f) = Hnumfaults at most Ngates, soany violation of the constraints incurring a penalty at leastλ = Ngates + 1 yields a total energy greater than that of anyvalid diagnosis.

A weaker condition to require of the penalty weight λ issimply that the ground state of H is a valid diagnosis. That is,an invalid state (i.e., one that violates at least one of the modelconstraints) may have lower total energy than some valid state,but not than a minimum-fault valid state. One simple upper-bound on the minimum number of faults is the number of out-puts, which thus also serves as a sufficient lower bound on λ.

In the case of the multiplier circuits with k-bit and l-bit inputs,the length of the outputs in simply k + l bits, which is muchsmaller than Ngates.

Nevertheless, a much lower value of λ may suffice in prac-tice for a particular set of instances. It is desirable to usethe smallest λ possible, because when the coefficients ofthe Hamiltonian are rescaled for a hardware implementation,larger values of λ lead to higher precision requirements, whichmay not be met by limited-precision devices. For the genera-tion of the PUBO expressions in the circuits considered here,up to mult8-8, we used a value of λ = 4, regardless of thesize of the circuit. With the help of the complete SAT-basedsolver, we checked that, for all the instances studied here, thisvalue sufficed to ensure that the ground state corresponds to avalid diagnosis.

However, we did generate observations, not included in thisstudy, for which λ = 4 was insufficient. This was extremelyrare, from no such examples in the smaller circuits to at mostone in five hundred for the largest circuits. Because we usedthe first hundred randomly generated instances for each size,λ = 4 sufficed for every instance used; this was highly likelythough not guaranteed.

A more common event that we had to filter in the instancegeneration was the appearance of random instances where theminimal solution contains no faults. These are easy to elim-inate since one can easily verify whether the output corre-sponds to the multiplication of the inputs and therefore thesolution to our problem is trivial with a minimal fault cardi-nality of zero. It is interesting to note that in diagnosis tasksuch instances are still valuable, since one considers not onlythe minimal cardinality but also the runners up could providevaluable information about the circuit. For example, it couldbe the case that there is indeed a fault in the circuit but the out-put observations still match the desired output, but the faultcan only be unmasked for example, by using another obser-vation in the circuit. The problem of selecting the best inputsto probe faults in circuits is another interesting NP-hard prob-lems in its own. We focus here in the minimal cardinality case,given an input/output pairs.

4. PUBO to QUBO reduction

The cost function of the CCFD problem is initially ex-pressed as a pseudo-Boolean expression (i.e., PUBO) of de-gree greater than two. We then transform the higher-degreePUBO expression into a quadratic one by using a conjunctiongadget. The conjunction gadget introduces an ancilla bit qi,jthat corresponds to a conjunction of two bits qi and qj in thePUBO, replaces all occurrences of the qiqj with qi,j , and addsa penalty function so that in any ground state of the QUBOexpression the ancilla bit is appropriately set, qi,j = qiqj . Weuse the penalty function [22, 71, 72]

Hancilla = δ(3qi,j + qiqj − 2qiqi,j − 2qjqi,j), (C15)

which is zero when qi,j = qiqj and at least δ otherwise, whereδ > 0 is the penalty weight. The penalty weight δ needs to belarge enough that states violating the ancilla constraint have

16

energy much larger than the ground energy of the originalPUBO expression, thus preserving the low-energy spectrum.As with the logical penalty λ, we would like δ to be as smallas is necessary in order to minimize the precision needed toimplement the cost function on a hardware device. For eachlogic gate in the CCFD problem, we determined that the fol-lowing values to be best, as a multiple of the logical penaltyweight λ:

δAND = δOR = 2.5λ; δXOR = 2λ. (C16)

Appendix D: SQA versus DW2X

Here we address in more details the question of whetherSQA has the same scaling as the DW2X device and the com-parison of the two schedules used in the SQA simulations.The linear schedule tends to underestimate the scaling expo-nents for easy problems and small systems sizes, when therequired number of sweeps is small. This is because theremight not be enough QMC time to remove the segments inthe imaginary time direction that have different spin values.

For the DW schedule (dws) we cut the first 10% of theschedule. First, the initial part of the D-Wave schedule is notnecessary because it is very easy to equilibrate QMC when thetransverse field strength is large enough. Second, that leadsto shorter simulation times as it takes roughly the same timeto run the first 10% of the schedule as to run the rest of theschedule. This is because the SQA simulation time is roughlyproportional to the transverse field strength and, in the firstpart of the schedule, the transverse field is largest. Strictlyspeaking, one probably can cut more than 10% of the initialschedule. One can also cut some fraction of the schedule at theend, but that will not improve simulation times significantly.However, that could lead to a different scaling for easy prob-lems and small small problem sizes. This difference in scalingthen could be fictitious and it might even disappear for largersystems sizes.

Therefore, it is difficult to make any conclusive statementsabout the apparent difference in scaling and significant fur-ther work is required to address this issue with more certainty.Besides emphasizing that such comparison are not straight-forward, these further simulations and parameter fine-tuningis beyond the scope of this work.

(a)

(c)

(b)

FIG. 8: (a) Details for the different annealing schedules used inthis work. Panels (b) and (c) show a comparison of the DW2X ex-perimental results and SQA simulations of hypothetical quantum an-nealing devices with a DW2X-like [SQA(q)dws] and with a linear an-nealing schedule [SQA(q)ls]. Data points correspond to the medianvalues extracted from a bootstrapping analysis from 100 instancesper problem size, with error bars indicating the 90% confidence in-tervals (CI).

17

Appendix E: Qubit resources for numerical simulation andexperiments

(a)

(b)

(c)

FIG. 9: Qubit resources for each of the problem representation[(a) PUBO, (b) QUBO and (c) chimera (DW2X)] considered in ourbenchmarking study of the CCFD instances. Data points correspondto the median values extracted from a bootstrapping statistical anal-ysis from 100 instances per problem size, with error bars indicatingthe 90% confidence intervals (CI).

18

Appendix F: Intrinsic hardness of the CCFD instances compared to other random spin-glass problems

(a)

(c)

(b)

(d)

FIG. 10: Comparison of CCFD-based benchmark problems against other random spin-glass benchmark classes, at different percentiles. Datapoints correspond to the specific percentile value extracted from a bootstrapping statistical analysis from 100 instances per problem size, witherror bars indicating the 90% confidence intervals (CI).

19

Appendix G: Scaling analysis from the application-centric perspective

2 - 2 3 - 3 4 - 4 5 - 50XlWiSlier WySe

100

102

104

106

108

1010

1012

1014T

T6

(μs

)

2 - 2 3 - 3 4 - 4 5 - 50XlWiSlier WySe

100

102

104

106

108

1010

1012

1014

TT

6 (μs

)

2 - 2 3 - 3 4 - 4 5 - 50XlWiSlier WySe

100

102

104

106

108

1010

1012

1014

TT

6 (μs

)

2 - 2 3 - 3 4 - 4 5 - 50XlWiSlier WySe

100

102

104

106

108

1010

1012

1014

1016T

T6

(μs

)

DW2X

64Als/C

64Als/3

64Als/4

64Adws/C

6A/C

6A/3

6A/4

3TIC0/C

3TIC0/3

3TIC0/4

6AT-bDsed

(a)

(c)

(b)

(d)

(a)

(c)

(b)

(d)

FIG. 11: Scaling analysis from the application-centric perspective at different percentile levels. (a) 25th, (b) 50th, (c) 60th, and (d) 75thpercentile. Data points correspond to the specific percentile value extracted from a bootstrapping statistical analysis from 100 instances perproblem size, with error bars indicating the 90% confidence intervals (CI).

20

Appendix H: Scaling analysis from the physics perspective

(a)

(c)

(b)

(d)

FIG. 12: Scaling analysis from the physics perspective at different percentile levels. (a) 25th, (b) 50th, (c) 60th, and (d) 75th percentile. Datapoints correspond to the specific percentile value extracted from a bootstrapping statistical analysis from 100 instances per problem size, witherror bars indicating the 90% confidence intervals (CI).

21

[1] T. Kadowaki and H. Nishimori, Quantum annealing in thetransverse Ising model, Phys. Rev. E 58, 5355 (1998).

[2] A. B. Finnila, M. A. Gomez, C. Sebenik, C. Stenson, and J. D.Doll, Quantum annealing: A new method for minimizing multi-dimensional functions, Chem. Phys. Lett. 219, 343 (1994).

[3] E. Farhi, J. Goldstone, S. Gutmann, J. Lapan, A. Lundgren,and D. Preda, A quantum adiabatic evolution algorithm appliedto random instances of an NP-complete problem, Science 292,472 (2001).

[4] G. Santoro, E. Martonak, R. Tosatti, and R. Car, Theory ofquantum annealing of an Ising spin glass, Science 295, 2427(2002).

[5] A. Das and B. K. Chakrabarti, Quantum Annealing and RelatedOptimization Methods (Edited by A. Das and B.K. Chakrabarti,Lecture Notes in Physics 679, Berlin: Springer, 2005).

[6] G. E. Santoro and E. Tosatti, TOPICAL REVIEW: Optimizationusing quantum mechanics: quantum annealing through adia-batic evolution, J. Phys. A 39, R393 (2006).

[7] A. Das and B. K. Chakrabarti, Quantum Annealing and AnalogQuantum Computation, Rev. Mod. Phys. 80, 1061 (2008).

[8] M. W. Johnson, M. H. S. Amin, S. Gildert, T. Lanting,F. Hamze, N. Dickson, R. Harris, A. J. Berkley, J. Johansson,P. Bunyk, et al., Quantum annealing with manufactured spins,Nature 473, 194 (2011).

[9] N. G. Dickson, M. W. Johnson, M. H. Amin, R. Harris,F. Altomare, A. J. Berkley, P. Bunyk, J. Cai, E. M. Chapple,P. Chavez, et al., Thermally assisted quantum annealing of a16-qubit problem, Nat. Commun. 4, 1903 (2013).

[10] S. Boixo, T. Albash, F. M. Spedalieri, N. Chancellor, and D. A.Lidar, Experimental signature of programmable quantum an-nealing, Nat. Commun. 4, 2067 (2013).

[11] H. G. Katzgraber, F. Hamze, and R. S. Andrist, GlassyChimeras Could Be Blind to Quantum Speedup: Designing Bet-ter Benchmarks for Quantum Annealing Machines, Phys. Rev.X 4, 021008 (2014).

[12] S. Boixo, T. F. Rønnow, S. V. Isakov, Z. Wang, D. Wecker,D. A. Lidar, J. M. Martinis, and M. Troyer, Evidence for quan-tum annealing with more than one hundred qubits, Nat. Phys.10, 218 (2014).

[13] T. F. Rønnow, Z. Wang, J. Job, S. Boixo, S. V. Isakov,D. Wecker, J. M. Martinis, D. A. Lidar, and M. Troyer, Definingand detecting quantum speedup, Science 345, 420 (2014).

[14] H. G. Katzgraber, F. Hamze, Z. Zhu, A. J. Ochoa, andH. Munoz-Bauza, Seeking Quantum Speedup Through SpinGlasses: The Good, the Bad, and the Ugly, Phys. Rev. X 5,031026 (2015).

[15] B. Heim, T. F. Rønnow, S. V. Isakov, and M. Troyer, Quantumversus classical annealing of Ising spin glasses, Science 348,215 (2015).

[16] I. Hen, J. Job, T. Albash, T. F. Rønnow, M. Troyer, and D. A. Li-dar, Probing for quantum speedup in spin-glass problems withplanted solutions, Phys. Rev. A 92, 042325 (2015).

[17] E. G. Rieffel, D. Venturelli, B. O’Gorman, M. B. Do, E. M.Prystay, and V. N. Smelyanskiy, A case study in programminga quantum annealer for hard operational planning problems,Quant. Inf. Proc. 14, 1 (2015).

[18] S. Boixo, V. N. Smelyanskiy, A. Shabani, S. V. Isakov, M. Dyk-man, V. S. Denchev, M. H. Amin, A. Y. Smirnov, M. Mohseni,and H. Neven, Computational multiqubit tunnelling in pro-grammable quantum annealers, Nat. Comm. 7, 10327 (2016).

[19] V. S. Denchev, S. Boixo, S. V. Isakov, N. Ding, R. Babbush,

V. Smelyanskiy, J. Martinis, and H. Neven, What is the Com-putational Value of Finite Range Tunneling?, Phys. Rev. X 6,031015 (2016).

[20] J. King, S. Yarkoni, J. Raymond, I. Ozfidan, A. D. King, M. M.Nevisi, J. P. Hilton, and C. C. McGeoch, Quantum Anneal-ing amid Local Ruggedness and Global Frustration (2017),(arXiv:quant-phys/1701.04579).

[21] S. Mandra, Z. Zhu, W. Wang, A. Perdomo-Ortiz, and H. G.Katzgraber, Strengths and weaknesses of weak-strong clus-ter problems: A detailed overview of state-of-the-art classi-cal heuristics versus quantum approaches, Phys. Rev. A 94,022337 (2016).

[22] A. Perdomo, C. Truncik, I. Tubert-Brohman, G. Rose, andA. Aspuru-Guzik, Construction of model hamiltonians for adi-abatic quantum computation and its application to finding low-energy conformations of lattice protein models, Phys. Rev. A78, 012320 (2008).

[23] A. Perdomo-Ortiz, N. Dickson, M. Drew-Brook, G. Rose, andA. Aspuru-Guzik, Finding low-energy conformations of latticeprotein models by quantum annealing, Sci. Rep. 2, 571 (2012).

[24] F. Gaitan and L. Clark, Ramsey numbers and adiabatic quantumcomputing, Phys. Rev. Lett. 108, 010501 (2012).

[25] O’Gorman, B., Babbush, R., Perdomo-Ortiz, A., Aspuru-Guzik, A., and Smelyanskiy, V., Bayesian network structurelearning using quantum annealing, Eur. Phys. J. Special Top-ics 224, 163 (2015).

[26] Perdomo-Ortiz, A., Fluegemann, J., Narasimhan, S., Biswas,R., and Smelyanskiy, V.N., A quantum annealing approachfor fault detection and diagnosis of graph-based systems, Eur.Phys. J. Special Topics 224, 131 (2015).

[27] K. M. Zick, O. Shehab, and M. French, Experimental quantumannealing: case study involving the graph isomorphism prob-lem, Scientific Reports 5, 11168 EP (2015).

[28] F. Neukart, G. Compostella, C. Seidel, D. von Dollen,S. Yarkoni, and B. Parney, Optimizing traffic flow us-ing quantum annealing and classical machine learning,arXiv:1708.01625 (2017).

[29] Z. Bian, F. Chudak, R. Israel, B. Lackey, W. G. Macready,and A. Roy, Mapping constrained optimization problemsto quantum annealing with application to fault diagnosis,arXiv:1603.03111 (2016).

[30] A. Feldman, G. Provan, and A. van Gemund, Approximatemodel-based diagnosis using greedy stochastic search, Journalof Artificial Intelligence Research 38, 371 (2010).

[31] S. Mandra, Z. Zhu, W. Wang, A. Perdomo-Ortiz, andH. G. Katzgraber, Strengths and Weaknesses of Weak-StrongCluster Problems: A Detailed Overview of State-of-the-art Classical Heuristics vs Quantum Approaches (2016),(arXiv:1604.01746).

[32] A. Feldman, G. Provan, and A. van Gemund, in Abstraction,Reformulation, and Approximation: 7th International Sympo-sium, SARA 2007, Whistler, Canada, July 18-21, 2007. Pro-ceedings, edited by I. Miguel and W. Ruml (Springer BerlinHeidelberg, Berlin, Heidelberg, 2007), pp. 139–154.

[33] T. Eiter and G. Gottlob, The complexity of logic-based abduc-tion, Journal of the ACM 42, 3 (1995).

[34] Rieger, H. and Kawashima, N., Application of a continuoustime cluster algorithm to the two-dimensional random quantumising ferromagnet, Eur. Phys. J. B 9, 233 (1999).

[35] S. Kirkpatrick, C. D. Gelatt, Jr., and M. P. Vecchi, Optimizationby simulated annealing, Science 220, 671 (1983).

22

[36] S. V. Isakov, I. N. Zintchenko, T. F. Rønnow, and M. Troyer,Optimized simulated annealing for Ising spin glasses, Comput.Phys. Commun. 192, 265 (2015), (see also ancillary material toarxiv:cond-mat/1401.1084).

[37] K. Hukushima and K. Nemoto, Exchange Monte Carlo methodand application to spin glass simulations, J. Phys. Soc. Jpn. 65,1604 (1996).

[38] H. G. Katzgraber, S. Trebst, D. A. Huse, and M. Troyer,Feedback-optimized parallel tempering Monte Carlo, J. Stat.Mech. P03018 (2006).

[39] J. J. Moreno, H. G. Katzgraber, and A. K. Hartmann, Findinglow-temperature states with parallel tempering, simulated an-nealing and simple Monte Carlo, Int. J. Mod. Phys. C 14, 285(2003).

[40] Z. Zhu, A. J. Ochoa, and H. G. Katzgraber, Efficient ClusterAlgorithm for Spin Glasses in Any Space Dimension, Phys. Rev.Lett. 115, 077201 (2015).

[41] A. Lucas, Ising formulations of many NP problems, Front.Physics 12, 5 (2014).

[42] N. Chancellor, S. Zohren, and P. A. Warburton, Circuit de-sign for multi-body interactions in superconducting quantumannealing systems with applications to a scalable architecture,npj Quantum Information 3, 21 (2017).

[43] E. Boros and P. L. Hammer, Pseudo-boolean optimization, Dis-crete Appl. Math. 123, 155 (2002).

[44] V. Choi, Minor-embedding in adiabatic quantum computation:II. minor-universal graph design, Quantum Information Pro-cessing 10, 343 (2011), ISSN 1570-0755.

[45] J. Cai, B. Macready, and A. Roy, A practical heuristic for find-ing graph minors, arXiv:1406.2741 (2014).

[46] A. Perdomo-Ortiz, J. Fluegemann, R. Biswas, and V. N.Smelyanskiy, A performance estimator for quantum anneal-ers: Gauge selection and parameter setting, arXiv:1503.01083(2015).

[47] Z. Zhu, A. J. Ochoa, F. Hamze, S. Schnabel, and H. G. Katz-graber, Best-case performance of quantum annealers on nativespin-glass benchmarks: How chaos can affect success proba-bilities, Phys. Rev. A 93, 012317 (2016).

[48] S. Sidon, Ein Satz uber trigonometrische Polynome und seineAnwendung in der Theorie der Fourier-Reihen, MathematischeAnnalen 106, 536 (1932).

[49] A. Selby, Efficient subgraph-based sampling of isingtype mod-els with frustration, arXiv:1409.3934 (2014).

[50] T. Albash and D. A. Lidar, Evidence for a limited quantumspeedup on a quantum annealer, arXiv:1705.07452 (2017).

[51] S. Aaronson, Google, d-wave, and the caseof the factor - 10**8 speedup for what?,http://www.scottaaronson.com/blog/?p=2555 (2015).

[52] S. Aaronson, Insert d-wave post here,http://www.scottaaronson.com/blog/?p=3192 (2015).

[53] L. Hormozi, E. W. Brown, G. Carleo, and M. Troyer, Nonsto-quastic hamiltonians and quantum annealing of an ising spinglass, Phys. Rev. B 95, 184416 (2017).

[54] H. Nishimori and K. Takada, Exponential enhancement of theefficiency of quantum annealing by non-stoquastic hamiltoni-ans, Frontiers in ICT 4, 2 (2017).

[55] A. Biere, SPLATZ, LINGELING, PLINGELING, TREENGELING,YALSAT entering the SAT competition 2016, SAT COMPETI-

TION 2016 p. 44 (2016).[56] J. Strand, A. Przybysz, D. Ferguson, and K. Zick, Zzz coupler

for native embedding of max-3sat problem instances in quan-tum annealing hardware, Bulletin of the American Physical So-ciety (2017).

[57] A. Perdomo-Ortiz, S. E. Venegas-Andraca, and A. Aspuru-Guzik, A study of heuristic guesses for adiabatic quantum com-putation, Quantum Inf. Process. 10, 33 (2010), ISSN 1570-0755, 1573-1332.

[58] N. Chancellor, Modernizing quantum annealing using localsearches, New Journal of Physics 19, 023024 (2017).

[59] N. Chancellor, Modernizing quantum annealing ii: Genetic al-gorithms and inference, arXiv:1609.05875 (2017).

[60] H. Karimi, G. Rosenberg, and H. G. Katzgraber, Effective op-timization using sample persistence: A case study on quan-tum annealers and various monte carlo optimization methods,arXiv:1706.07826 (2017).

[61] R. Harris, M. W. Johnson, T. Lanting, A. J. Berkley, J. Johans-son, P. Bunyk, E. Tolkacheva, E. Ladizinsky, N. Ladizinsky,T. Oh, et al., Experimental investigation of an eight-qubit unitcell in a superconducting optimization processor, Phys. Rev. B.82, 024511 (2010).

[62] F. Barahona, On the computational complexity of ising spinglass models, Journal of Physics A: Mathematical and General15, 3241 (1982).

[63] T. Albash, S. Boixo, D. A. Lidar, and P. Zanardi, Quantum adi-abatic markovian master equations, New J. Phys. 14, 123016(2012).

[64] V. N. Smelyanskiy, D. Venturelli, A. Perdomo-Ortiz, S. Knysh,and M. I. Dykman, Quantum annealing via environment-mediated quantum diffusion, Phys. Rev. Lett. 118, 066802(2017).

[65] S. V. Isakov and R. Moessner, Interplay of quantum and thermalfluctuations in a frustrated magnet, Phys. Rev. B 68, 104409(2003).

[66] H. G. Katzgraber, Introduction to Monte Carlo Methods (2009),(arXiv:0905.1629).

[67] M. Benedetti, J. Realpe-Gomez, R. Biswas, and A. Perdomo-Ortiz, Quantum-assisted learning of graphical models with ar-bitrary pairwise connectivity, arXiv:1609.02542 (2016).

[68] A. Perdomo-Ortiz, M. Benedetti, J. Realpe-Gomez, andR. Biswas, Opportunities and challenges for quantum-assistedmachine learning in near-term quantum computers, In prepara-tion (2017).

[69] V. Choi, Minor-embedding in adiabatic quantum computation:I. the parameter setting problem, arXiv:0804.4884 (2008).

[70] K. L. Pudenz, Parameter setting for quantum annealers,arXiv:1611.07552 (2016).

[71] R. Babbush, A. Perdomo-Ortiz, B. O’Gorman, W. Macready,and A. Aspuru-Guzik, Construction of energy functions for lat-tice heteropolymer models: A case study in constraint satisfac-tion programming and adiabatic quantum optimization, Adv.Chem. Phys. 155, 201 (2014).

[72] R. Babbush, B. O’Gorman, and A. Aspuru-Guzik, Resource ef-ficient gadgets for compiling adiabatic quantum optimizationproblems, Annalen der Physik 525, 877 (2013).