16
The spinon Fermi surface U(1) spin liquid in a spin-orbit-coupled triangular lattice Mott insulator YbMgGaO 4 Yao-Dong Li 1 , Yuan-Ming Lu 2 , and Gang Chen 1,3* 1 State Key Laboratory of Surface Physics, Department of Physics, Center for Field Theory & Particle Physics, Fudan University, Shanghai, 200433, China 2 Department of Physics, The Ohio State University, Columbus, OH, 43210, United States and 3 Collaborative Innovation Center of Advanced Microstructures, Nanjing, 210093, China (Dated: August 24, 2017) Motivated by the recent progress on the spin-orbit-coupled triangular lattice spin liquid candi- date YbMgGaO4, we carry out a systematic projective symmetry group analysis and mean-field study of candidate U(1) spin liquid ground states. Due to the spin-orbital entanglement of the Yb moments, the space group symmetry operation transforms both the position and the orientation of the local moments, and hence brings different features for the projective realization of the lattice symmetries from the cases with spin-only moments. Among the eight U(1) spin liquids that we find with the fermionic parton construction, only one spin liquid state, that was proposed and analyzed in Yao Shen, et. al., Nature 540, 559-562 (2016) and labeled as U1A00 in the present work, stands out and gives a large spinon Fermi surface and provides a consistent explanation for the spectro- scopic results in YbMgGaO4. Further connection of this spinon Fermi surface U(1) spin liquid with YbMgGaO4 and the future directions are discussed. Finally, our results may apply to other spin- orbit-coupled triangular lattice spin liquid candidates, and more broadly, our general approach can be well extended to spin-orbit-coupled spin liquid candidate materials. I. INTRODUCTION The interplay between strong spin-orbit coupling (SOC) and strong electron correlation has attracted a significant attention in recent years 1 . At the mean time, the abundance of strongly correlated materials with 5d and 4f electrons, such as iridates and rare-earth mate- rials 1,2 , brings a fertile arena to explore various emer- gent and exotic phases that arise from such an inter- play 332 . The recently discovered quantum spin liquid (QSL) candidate YbMgGaO 4 33 , where the rare-earth Yb atoms form a perfect triangular lattice, is an ideal sys- tem that involves strong spin-orbital entanglement in the strong Mott insulating regime of the Yb electrons 3441 . In YbMgGaO 4 , the thirteen 4f electrons of the Yb 3+ ions are well localized and form a spin-orbit-entangled total moment J with J =7/2 34,35 . The eight-fold de- generacy of the J =7/2 moment is further split by the D 3d crystal electric fields. The resulting ground state Kramers doublet of the Yb 3+ ion, whose two-fold de- generacy is protected by the time-reversal symmetry, is well separated from the excited doublets and is re- sponsible for the low-temperature magnetic properties of YbMgGaO 4 . No signature of time-reversal symmetry breaking is observed for YbMgGaO 4 down to the lowest measured temperature 3638 . Applying the recent theoret- ical result on spin-orbit-coupled Mott insulators 42 , two of us and collaborators have proposed YbMgGaO 4 to be the first QSL candidate in the spin-orbit-coupled Mott in- sulator with odd electron fillings 3436,39 . More broadly, YbMgGaO 4 represents a new class of rare-earth materi- als where the strong spin-orbit entanglement of the local moments meets with the geometrical frustration of the triangular lattice such that exotic quantum phases may be stabilized. Apart from the absence of magnetic ordering, the heat capacity was found to be C v T 0.7 at low tempera- tures 33,34,37,43 , and is close to the well-known T 2/3 heat capacity 4446 . The latter was the one obtained within a random phase approximation for the spinon-gauge cou- pling in a spinon Fermi surface U(1) QSL 4446 . More substantially, the broad continuum 36,37 of the magnetic excitation with a clear dispersion for the upper excitation edge agrees reasonably with the particle-hole continuum of the spinon Fermi surface 36 . However, due to the scat- tering with the phonon degrees of freedom, the thermal transport measurement in YbMgGaO 4 was unable to ex- tract the intrinsic magnetic contribution to the thermal conductivity 43 . Partly motivated by the spin liquid be- haviors in YbMgGaO 4 and more broadly by the families of rare-earth magnets with identical structures, in this paper, we carry out a systematic projective symmetry group (PSG) analysis for a triangular lattice Mott insu- lator with spin-orbital-entangled local moments. Unlike the cases for the spin-only moments in the pioneering work by X.-G. Wen 47 , the space group symmetry opera- FIG. 1. (a) The intralayer symmetries of the R ¯ 3m space group for YbMgGaO4 35 . (b) The same lattice symmetry group with a different complete set of elementary transformations. Here S6 C -1 3 I . The bold arrow is the axis for the C2 rotation (see Appendix). arXiv:1612.03447v4 [cond-mat.str-el] 23 Aug 2017

arXiv:1612.03447v4 [cond-mat.str-el] 23 Aug 2017 · 2017. 8. 24. · 1 and T 2, one two-fold rota-tion, C 2, one three-fold rotation, C 3, and one spatial in-version I (see Fig.1(a))35,39

  • Upload
    others

  • View
    1

  • Download
    0

Embed Size (px)

Citation preview

  • The spinon Fermi surface U(1) spin liquid in a spin-orbit-coupledtriangular lattice Mott insulator YbMgGaO4

    Yao-Dong Li1, Yuan-Ming Lu2, and Gang Chen1,3∗1State Key Laboratory of Surface Physics, Department of Physics,

    Center for Field Theory & Particle Physics, Fudan University, Shanghai, 200433, China2Department of Physics, The Ohio State University, Columbus, OH, 43210, United States and

    3Collaborative Innovation Center of Advanced Microstructures, Nanjing, 210093, China(Dated: August 24, 2017)

    Motivated by the recent progress on the spin-orbit-coupled triangular lattice spin liquid candi-date YbMgGaO4, we carry out a systematic projective symmetry group analysis and mean-fieldstudy of candidate U(1) spin liquid ground states. Due to the spin-orbital entanglement of the Ybmoments, the space group symmetry operation transforms both the position and the orientation ofthe local moments, and hence brings different features for the projective realization of the latticesymmetries from the cases with spin-only moments. Among the eight U(1) spin liquids that we findwith the fermionic parton construction, only one spin liquid state, that was proposed and analyzedin Yao Shen, et. al., Nature 540, 559-562 (2016) and labeled as U1A00 in the present work, standsout and gives a large spinon Fermi surface and provides a consistent explanation for the spectro-scopic results in YbMgGaO4. Further connection of this spinon Fermi surface U(1) spin liquid withYbMgGaO4 and the future directions are discussed. Finally, our results may apply to other spin-orbit-coupled triangular lattice spin liquid candidates, and more broadly, our general approach canbe well extended to spin-orbit-coupled spin liquid candidate materials.

    I. INTRODUCTION

    The interplay between strong spin-orbit coupling(SOC) and strong electron correlation has attracted asignificant attention in recent years1. At the mean time,the abundance of strongly correlated materials with 5dand 4f electrons, such as iridates and rare-earth mate-rials1,2, brings a fertile arena to explore various emer-gent and exotic phases that arise from such an inter-play3–32. The recently discovered quantum spin liquid(QSL) candidate YbMgGaO4

    33, where the rare-earth Ybatoms form a perfect triangular lattice, is an ideal sys-tem that involves strong spin-orbital entanglement in thestrong Mott insulating regime of the Yb electrons34–41.

    In YbMgGaO4, the thirteen 4f electrons of the Yb3+

    ions are well localized and form a spin-orbit-entangledtotal moment J with J = 7/234,35. The eight-fold de-generacy of the J = 7/2 moment is further split by theD3d crystal electric fields. The resulting ground stateKramers doublet of the Yb3+ ion, whose two-fold de-generacy is protected by the time-reversal symmetry,is well separated from the excited doublets and is re-sponsible for the low-temperature magnetic propertiesof YbMgGaO4. No signature of time-reversal symmetrybreaking is observed for YbMgGaO4 down to the lowestmeasured temperature36–38. Applying the recent theoret-ical result on spin-orbit-coupled Mott insulators42, two ofus and collaborators have proposed YbMgGaO4 to be thefirst QSL candidate in the spin-orbit-coupled Mott in-sulator with odd electron fillings34–36,39. More broadly,YbMgGaO4 represents a new class of rare-earth materi-als where the strong spin-orbit entanglement of the localmoments meets with the geometrical frustration of thetriangular lattice such that exotic quantum phases maybe stabilized.

    Apart from the absence of magnetic ordering, the heatcapacity was found to be Cv ∝ T 0.7 at low tempera-tures33,34,37,43, and is close to the well-known T 2/3 heatcapacity44–46. The latter was the one obtained within arandom phase approximation for the spinon-gauge cou-pling in a spinon Fermi surface U(1) QSL44–46. Moresubstantially, the broad continuum36,37 of the magneticexcitation with a clear dispersion for the upper excitationedge agrees reasonably with the particle-hole continuumof the spinon Fermi surface36. However, due to the scat-tering with the phonon degrees of freedom, the thermaltransport measurement in YbMgGaO4 was unable to ex-tract the intrinsic magnetic contribution to the thermalconductivity43. Partly motivated by the spin liquid be-haviors in YbMgGaO4 and more broadly by the familiesof rare-earth magnets with identical structures, in thispaper, we carry out a systematic projective symmetrygroup (PSG) analysis for a triangular lattice Mott insu-lator with spin-orbital-entangled local moments. Unlikethe cases for the spin-only moments in the pioneeringwork by X.-G. Wen47, the space group symmetry opera-

    FIG. 1. (a) The intralayer symmetries of the R3̄m space groupfor YbMgGaO4

    35. (b) The same lattice symmetry group witha different complete set of elementary transformations. HereS6 ≡ C−13 I. The bold arrow is the axis for the C2 rotation(see Appendix).

    arX

    iv:1

    612.

    0344

    7v4

    [co

    nd-m

    at.s

    tr-e

    l] 2

    3 A

    ug 2

    017

  • 2

    tion, in particular, the rotation, transforms both the po-sition and the orientation of the Yb local moments35,39.We find that, among the eight U(1) QSL states, thespinon mean-field state that was introduced in Ref. 36and labeled as the U1A00 state in our PSG classification,contains a large spinon Fermi surface and gives a largespinon scattering density of states that is consistent withthe inelastic neutron scattering (INS) results.

    The following part of the paper is organized as follows.In Sec. II, we describe the space group symmetry and thethe multiplication rules for the symmetry transformation.In Sec. III, we introduce the fermionic spinon construc-tion and the fermionic spinon mean-field Hamiltonian. InSec. IV, we explain the scheme for the projective sym-metry group classification when the spin-orbit couplingis present. In Sec. V, we explain the relationship betweenthe spinon band structure and the projective symmetrygroup of the spinon mean-field states. In Sec. VI, wefocus on the U1A00 state and study the spectroscopicproperties of this state. Finally in Sec. VII, we discussthe experimental relevance and remark on the thermaltransport result and the competing scenarios and pro-posals. The details of the calculation are presented inthe Appendices.

    II. SPACE GROUP SYMMETRY

    It was pointed out that the intralayer symmetries in-volves two translations, T1 and T2, one two-fold rota-tion, C2, one three-fold rotation, C3, and one spatial in-version I (see Fig. 1(a))35,39. Here we use a differentcomplete set of elementary transformations for the spacegroup symmetries that involve two translations, T1 andT2, one two-fold rotation, C2, and one more operation,S6 (see the definition in Fig. 1(b)). It is ready to con-firm I = S36 , C3 = S

    26 with the definition S6 ≡ C−13 I. The

    multiplication rules of this symmetry group is given as

    T−11 T2T1T−12 = T

    −11 T

    −12 T1T2 = 1, (1)

    C−12 T1C2T−12 = C

    −12 T2C2T

    −11 = 1, (2)

    S−16 T1S6T2 = S−16 T2S6T

    −12 T

    −11 = 1, (3)

    (C2)2 = (S6)

    6 = (S6C2)2 = 1. (4)

    Due to the presence of time reversal inYbMgGaO4

    34,36–38, we further supplement the symmetrygroup with the time reversal T such that O−1T OT = 1and T 2 = 1, where O is a lattice symmetry operation.

    III. FERMIONIC PARTON CONSTRUCTION

    To describe the U(1) QSL that we propose forYbMgGaO4, we introduce the fermionic spinon opera-tor frα(α =↑, ↓) that carries spin-1/2, and express theYb local moment as

    Sr =1

    2

    ∑α,β

    f†rασαβfrβ , (5)

    U(1) QSL WT1r WT2r W

    C2r W

    S6r

    U1A00 I2×2 I2×2 I2×2 I2×2

    U1A10 I2×2 I2×2 iσy I2×2

    U1A01 I2×2 I2×2 I2×2 iσy

    U1A11 I2×2 I2×2 iσy iσy

    TABLE I. List of the gauge transformations for the four U1APSGs. For the time reversal, all PSGs here have W Tr = I2×2.The last two letters in the labels of the U(1) QSLs are extraquantum numbers in the PSG classification48.

    where σ = (σx, σy, σz) is a vector of Pauli matrices. Wefurther impose a constraint

    ∑α f†rαfrα = 1 on each site

    to project back to the physical Hilbert space of the spins.The choice of fermionic spinons allows a local SU(2)gauge freedom47.

    As a direct consequence of the spin-orbital entangle-ment, the spinon mean-field Hamiltonian for the U(1)QSL should generically involve both spin-preserving andspin-flipping hoppings, and has the following form

    HMF = −∑(rr′)

    ∑αβ

    [trr′,αβf

    †rαfr′β + h.c.

    ], (6)

    where trr′,αβ is the spin-dependent hopping. The choiceof the mean-field ansatz in Eq. (6) breaks the local SU(2)gauge freedom down to U(1). Here, to get a more com-pact form for Eq. (6), we follow Ref. 49 and introduce theextended Nambu spinor representation for the spinons

    such that Ψr = (fr↑, f†r↓, fr↓,−f

    †r↑)

    T and

    HMF = −1

    2

    ∑(r,r′)

    [Ψ†rurr′Ψr′ + h.c.

    ], (7)

    where urr′ is a hopping matrix that is related to trr′,αβ .With the extended Nambu spinor, the spin operator Srand the generatorGr for the SU(2) gauge transformationare given by47,50–53

    Sr =1

    4Ψ†r(σ ⊗ I2×2)Ψr, (8)

    Gr =1

    4Ψ†r(I2×2 ⊗ σ)Ψr, (9)

    where I2×2 is a 2× 2 identity matrix. Under the symme-try operation O, Ψr transforms as

    Ψr → UOGOO(r)ΨO(r) = GOO(r)UOΨO(r), (10)

    where GOO(r) is the local gauge transformation that cor-responds to the symmetry operation O, and we add aspin rotation UO because the spin components are trans-formed when O involves a rotation. In Eq. (10), thegauge transformation and the spin rotation are commu-tative54 simply because [Sµr , G

    νr] = 0. Moreover, from

    Eq. (9), the gauge transformation GOr is block diagonalwith GOr = I2×2 ⊗WOr , where WOr is a 2× 2 matrix (seeAppendix).

  • 3

    IV. PROJECTIVE SYMMETRY GROUPCLASSIFICATION

    For the spinon mean-field Hamiltonian in Eq. (6), thelattice symmetries are realized projectively and form theprojective symmetry group (PSG). To respect the lat-tice symmetry transformation O, the mean-field ansatzshould satisfy

    urr′ = GO†O(r)U

    †OuO(r)O(r′)UOG

    OO(r′). (11)

    The ansatz itself is invariant under the so-called invariantgauge group (IGG) with urr′ = G1†r urr′G1r′ . The IGGcan be regarded as a set of gauge transformations thatcorrespond to the identity transformation. For an U(1)QSL, IGG = U(1).

    A general group relation O1O2O3O4 = 1 for the latticesymmetry turns into the following group relation for thePSG

    UO1GO1r UO2G

    O2O2O3O4(r)UO3G

    O3O3O4(r)UO4G

    O4O4(r)

    = UO1UO2UO3UO4GO1r G

    O2O2O3O4(r)G

    O3O3O4(r)G

    O4O4(r)(12)

    ∈ IGG, (13)

    where we used the fact that the gauge transformationcommutes with the spin rotation. As the series of rota-tions O1O2O3O4 either rotate the spinons by 0 or 2π,

    UO1UO2UO3UO4 = ±I4×4, (14)

    where I4×4 is a 4× 4 identity matrix. Since{±I4×4} ⊂ IGG, then

    GO1r GO2O2O3O4(r)G

    O3O3O4(r)G

    O4O4(r) ∈ IGG. (15)

    This immediately indicates that, to classify the PSGs fora spin-orbit-coupled Mott insulator, we only need to fo-cus on the gauge part, first find the gauge transformationwith the same procedures as those for the conventionalMott insulators with spin-only moments47, and then ac-count for the spin rotation.

    For the mean-field ansatz in HMF, we choose the“canonical gauge” for the IGG with

    IGG = {I2×2 ⊗ eiφσz

    |φ ∈ [0, 2π)}. (16)

    Under the canonical gauge, the gauge transformation as-sociated with the symmetry operation O takes the formof

    GOr = I2×2 ⊗WOr≡ I2×2 ⊗

    [(iσx)nOeiφO[r]σ

    z], (17)

    where nO = 0, 1. For translations, one can always choosea gauge such that

    WT1r = (iσx)n1 , (18)

    WT2r = (iσx)n2eiφ2[x,y]σ

    z

    (19)

    with n1, n2 = 0, 1 and φ2[0, y] = 0. The group rela-tion in Eq. (3) further demands n1 = n2 = 0. Thus the

    FIG. 2. (a,b,c) The mean-field spinon bands along the high-symmetry momentum lines (see (d)) of the U1A00, U1A01and U1A11 states, where t1, t

    ′1 and t2 are hoppings in their

    spinon mean-field Hamiltonians (see Appendix). The Diraccones are highlighted in dashed circles. The dashed line refersto the Fermi level. (d) The Brioullin zone of the triangularlattice.

    group relation in Eq. (1) gives WT1r = 1,WT2r = e

    ixφ1σz

    ,where φ1 is the flux through each unit cell of the triangu-lar lattice and takes the value of 0 or π (see Appendix).The PSGs with φ1 = 0 (π) are labeled by U1A (U1B).Among the sixteen algebraic PSGs that we find, eight un-physical solutions have T 2 = 1 for the spinons and givevanishing spinon hoppings everywhere. In Tab. I and theAppendix, we list the remaining eight PSGs that haveT 2 = −1 consistent with the fact that fermionic spinonsare Kramers doublets (see Appendix).

    V. MEAN-FIELD STATES

    Here we obtain the spinon mean-field Hamiltonianfrom Tab. I and explain why the U1A00 state standsout as the candidate ground state for YbMgGaO4. Westart with the U1A states. Among the four U1A states,the U1A10 state gives a vanishing mean-field Hamilto-nian for the spinon hoppings between the first and thesecond neighbors, the remaining ones except the U1A00state all have symmetry protected band touchings at thespinon Fermi level (see Fig. 2). To illustrate the idea55,we consider the U1A01 state where the spinon Hamilto-

    nian has the form HU1A01MF =∑

    k hαβ(k)f†kαfkβ in the

    momentum space and h(k) is a 2× 2 matrix with

    h(k) = d0(k)I2×2 +

    3∑µ=1

    dµ(k)σµ. (20)

    For this band structure there are nondegenerate bandtouchings at Γ, M and K points that are protected bythe PSG of the U1A01 state. Under the operation S6,

  • 4

    the PSG demands that spinons to transform as

    fk↑ → −e−iπ/3f†−S−16 k,↓

    , (21)

    fk↓ → eiπ/3f†−S−16 k,↑

    . (22)

    Applying S6 three times and keeping HMF invariant, werequire

    h(k) = −[σyh(k)σy]T (23)

    which forces d0(k) = 0. The time reversal sym-metry (T = iσy ⊗ I2×2K) further requires thatdµ(k) = −dµ(−k). Thus we have symmetry pro-tected band touchings with h(k) = 0 at the time reversalinvariant momenta Γ and M. The K points are invariantunder C2 and S6 because the spinon partile-hole trans-formation is involved for S6 (see Appendix). Using thosetwo symmetries, we further establish the band touchingat the K points. Likewise, for the U1A11 state, thePSG demands the band touchings at Γ and M points.Because there are only two spinon bands for the U1Astates, these band touchings generically occur at thespinon Fermi level.

    Due to the Dirac band touchings at the Fermi level, thelow-energy dynamic spin structure factor, that measuresthe spinon particle-hole continuum, is concentrated at afew discrete momenta that correspond to the intra-Dirac-cone and the inter-Dirac-cone scatterings36. Clearly, thisis inconsistent with the recent INS result that observes abroad continuum covering a rather large portion of theBrillouin zone36,37.

    For the U1B states, the spinons experience a π back-ground flux in each unit cell. The direct consequence ofthe π background flux is that the U1B states support anenhanced periodicty of the dynamic spin structure in theBrillouin zone47,56,57. Such an enhanced periodicity isabsent in the INS result36,37. In particular, unlike whatone would expect for an enhanced periodicity, the spec-tral intensity at the Γ point is drastically different fromthe one at the M point in the existing experiments36,37.

    The above analysis leads to the conclusion that theU1A00 state is the most promising candidate U(1) QSLfor YbMgGaO4, and this conclusion is independent fromany microscopic model. The spinon mean field Hamilto-nian, allowed by the U1A00 PSG, is remarkably simpleand is given as58

    HU1A00MF = −t1∑〈rr′〉,α

    f†rαfrα − t2∑

    〈〈rr′〉〉,α

    f†rαfrα, (24)

    where the spinon hopping is isotropic for the first andthe second neighbors. This mean-field state only has asingle band that is 1/2-filled, so it has a large spinonFermi surface. From HU1A00MF , we construct the mean-field ground state by filling the spinon Fermi sea,

    |ΨU1A00MF 〉 =∏�k

  • 5

    FIG. 3. (a) S(q, ω) along the high-symmetry momentumlines from HU1A00MF with t2 = 0.2t1. The spinon bandwidthB = 9.6t1. (b) The RPA corrected SRPA(q, ω) along the highsymmetry momentum lines. We have set the parameters inthe spin model to be J±/Jzz = 0.915, J±±/Jzz = 0.35, andJz±/Jzz = 0.2. The ratio Jzz/t1 is obtained from Refs. 34and 36 and fixed to be 1.0 for concreteness.

    in the momentum space. We find in Fig. 3(b) that, thelow-energy spectral weight at M is slightly enhanced, afeature observed in Refs. 36 and 37. From our choiceof the parameters, it is plausible that this peak resultsfrom the proximity to a phase with a stripe-like magneticorder35,36,39.

    VII. DISCUSSION

    We have demonstrated that the spinon Fermi surfaceU(1) QSL gives a consistent explanation of the INS re-sult in YbMgGaO4. Moreover, the anisotropic spin in-teraction, slightly enhances the spectral weight at the Mpoints. The U(1) gauge fluctuation in the spinon Fermisurface U(1) QSL44,45 was suggested to be the cause forthe sublinear temperature dependence of the heat capac-ity in YbMgGaO4

    35,36,39,46.In YbMgGaO4, the coupling between the Yb moments

    is relatively weak34. It is feasible to fully polarize the spinwith experimentally accessible magnetic fields35,37,39,61

    and to study the evolution of the magnetic properties un-der the magnetic field. Recently, two of us have predictedthe spectral weight shift of the INS for YbMgGaO4 undera weak magnetic field41, and the predicted spectral cross-ing at the Γ point and the dispersion of the spinon con-tinuum have actually been confirmed in the recent INSmeasurement62. Numerically, it is useful to perform nu-merical calculation with fixed J± and Jzz that are closeto the ones for YbMgGaO4, and obtain the phase dia-gram of our spin model by varying J±± and Jz±

    35,39,63.More care needs to be paid to the disordered region of themean-field phase diagram35 where quantum fluctuationis found to be strong35. The “2kF” oscillation in the spincorrelation would be the strong indication of the spinonFermi surface. Noteworthily recent DMRG works64,65

    have actually provided some useful information aboutthe ground states of the system, in particular, Ref. 65suggested the scenario of exchange disorders. Certainamount of exchange disorder may be created by the crys-tal electric field disorder that stems from the Mg/Ga

    mixing in the non-magnetic layers37,61, but recent polar-ized neutron scattering measurement did not find strongexchange disorder59. Regardless of the possibilities ofexchange disorders, the spin quantum number fraction-alization, that is one of the key properties of the QSLs,could survive even with weak disorders. The approachand results in our present work are phenomenologicallybased and are independent of the microscopic mechanismfor the possible QSL ground state in YbMgGaO4.

    Ref. 43 claimed the absence of the magnetic thermalconductivity in YbMgGaO4 by extrapolating the low-temperature thermal conductitivity data in the zero mag-netic field. Here, we provide an alternative understand-ing for this thermal transport result. The hint lies inthe field dependence of the thermal conductivity. It wasfound that, when strong magnetic fields are applied toYbMgGaO4, the thermal conductivity κxx/T at 0.2K isincreased compared with the one at zero field43. If one ig-nores the disorder effect and assumes the zero-field ther-mal conductivity is a simple addition of the magneticcontribution and the phonon contribution with

    κxx = κspin,xx + κphonon,xx, (29)

    the strong magnetic field almost polarizes the spins com-pletely and creates a spin gap for the magnon excitation,hence suppress the magnetic contribution. The high-field thermal conductivity would be purely given by thephonon contribution, and we would expect a decreasingof the thermal conductivity in the strong field comparedto the zero field result. This is clearly inconsistent withthe experimental result. Therefore, the zero-field ther-mal conductivity is not a simple addition of the magneticcontribution and the phonon contribution, i.e.,

    κxx 6= κspin,xx + κphonon,xx. (30)

    This also strongly suggests the presence rather than theabsence of magnetic excitations in the thermal conductiv-ity result at zero magnetic field. If there is no magneticexcitation in the system at low temperatures, the low-temperature thermal conductivity at zero field shouldjust be the phonon contribution, and we would expectthe zero-field thermal conductivity to be the same as theone in the strong field limit, (although the intermediatefield regime could be different). This is again inconsis-tent with the experiments. This means that the magneticexcitation certainly does not have a large gap and couldjust be gapless as we propose from the spinon Fermi sur-face state. In fact, the gapless nature of the magneticexcitation is consistent with the power-law heat capac-ity results in YbMgGaO4. What suppresses κxx couldarise from the mutual scattering between the magneticexcitations and the the phonons. In fact, similar field de-pendence of thermal conductivity κxx has been observedin other rare-earth systems such as Tb2Ti2O7

    66–68 andPr2Zr2O7

    69. It was suggested there67–69 that the spin-phonon scattering is the cause. The Yb local moment,that is a spin-orbit-entangled object, involves the orbital

  • 6

    degree of freedom. The orbital degree of freedom is sen-sitive to the ion position, and thus couples to the phononstrongly. This is probably the microscopic origin for thestrong coupling between the magnetic moments and thephonons in the rare-earth magnets. This is quite dif-ferent from the organic spin liquid candidates and theherbertsmithite kagome system where the orbital degreeof freedom does not seem to be involved70–73.

    If the ground state of YbMgGaO4 is a QSL with thespinon Fermi surface, the field-driven transition from theQSL ground state to the fully polarized state is neces-sarily a unconventional transition beyond the traditionalLandau’s paradigm and has not been studied in the pre-vious spin liquid candidates70–73. The smooth growth ofthe magnetization with varying external fields indicatesa continuous transition34. Since we propose YbMgGaO4to be a spinon Fermi surface U(1) QSL and gapless, thetransition would be associated with the openning of thespin gap at the critical field. The continuous nature ofthe transition suggests the spin gap to open in a contin-uous manner. Moreover, the spinon confinement wouldbe concomitant with the spin gap that suppresses thespinon density of states and allows the instanton eventsof the U(1) gauge field to proliferate. Therefore, it mightbe interest to identify the critical field and obtain thecritical properties of the field-driven transition. Thermo-dynamic, spectroscopic, and thermal transport measure-ments with finer field variation would be helpful.

    Finally, several families of rare-earth triangular latticemagnets have been discovered recently35,39,74–79. Theirproperties have not been studied carefully. Our generalclassification results and the prediction of the spectro-scopic properties would apply to the QSL candidates thatmay emerge in these families of materials. It is certainlyexciting if one finds the new QSL candidates in thesefamilies behave like YbMgGaO4

    35.

    VIII. ACKNOWLEDGEMENTS

    We thank one anonymous referee for the suggestion forimprovement to this paper, and Zhu-Xi Luo for point-ing out some typos. G.C. acknowledges the discussionwith Xuefeng Sun from USTC and Yuji Matsuda aboutthermal transports in rare-earth magnets, and the dis-cussion with Professor Sasha Chernyshev about the re-lated matters. This work is supported by the Min-istry of Science and Technology of China with the GrantNo.2016YFA0301001 (G.C.), the Start-Up Funds of OSU(Y.M.L.) and Fudan University (G.C.), the National Sci-ence Foundation under Grant No. NSF PHY-1125915(Y.M.L and G.C.), the Thousand-Youth-Talent Program(G.C.) of China, and the first-class university construc-tion program of Fudan University.

    Appendix A: The coordinate System and spacegroup symmetry

    Following our convention in Fig. 1 in the main text, wechoose the coordinate system of the triangular lattice tobe

    a1 = (1, 0), (A1)

    a2 = (−1

    2,

    √3

    2). (A2)

    We label the triangular lattice sites by r = xa1 +y a2.Restricted to the triangular layer, the space group con-tains two translations T1 along the a1 direction, T2 alongthe a2 direction, a counterclockwise three-fold rotationC3 around the lattice site, a two-fold rotation C2 arounda1 + a2, and the inversion I at the lattice site. Theiractions on the lattice indices are

    T1 : (x, y)→ (x+ 1, y), (A3)T2 : (x, y)→ (x, y + 1), (A4)C3 : (x, y)→ (−y, x− y), (A5)C2 : (x, y)→ (y, x), (A6)I : (x, y)→ (−x,−y). (A7)

    In the formulation introduced in the main text, weconsider an equivalent set of generators, {T1, T2, C2, S6},where the operation S6 is defined as S6 ≡ C−13 I and actson the lattice indices as

    S6 : (x, y)→ (x− y, x). (A8)

    It is evident that these two sets of generators are equiva-lent, since we merely redefine the symmetry rather thanintroducing any new symmetry.

    The multiplication rule of this symmetry group is givenin the main text. For the convenience of the presentationbelow, we also list these rules here,

    T−11 T2T1T−12 = T

    −11 T

    −12 T1T2 = 1, (A9)

    C−12 T1C2T−12 = C

    −12 T2C2T

    −11 = 1, (A10)

    S−16 T1C6T2 = S−16 T2C6T

    −12 T

    −11 = 1, (A11)

    (C2)2 = (C6)

    6 = (S6C2)2 = 1. (A12)

    Including the time reversal symmetry, we further have

    T−11 T T1T = T−12 T T2T = 1, (A13)

    C−12 T C2T = S−16 T S6T = 1, (A14)T 2 = 1. (A15)

    Appendix B: Projective symmetry groupclassification

    As we describe in the main text, we consider the U(1)QSL. The spinon mean-field Hamiltonian has the follow-ing form

    HMF = −∑(rr′)

    ∑αβ

    [trr′,αβf

    †rαfr′β + h.c.

    ], (B1)

  • 7

    U(1) QSL WT1r WT2r W

    C2r W

    S6r

    U1A00 I2×2 I2×2 I2×2 I2×2

    U1A10 I2×2 I2×2 iσy I2×2

    U1A01 I2×2 I2×2 I2×2 iσy

    U1A11 I2×2 I2×2 iσy iσy

    U1B00 I2×2 (−1)xI2×2 (−1)xyI2×2 (−1)xy−y(y−1)

    2 I2×2

    U1B10 I2×2 (−1)xI2×2 iσy(−1)xy (−1)xy−y(y−1)

    2 I2×2

    U1B01 I2×2 (−1)xI2×2 (−1)xyI2×2 iσy(−1)xy−y(y−1)

    2

    U1B11 I2×2 (−1)xI2×2 iσy(−1)xy iσy(−1)xy−y(y−1)

    2

    TABLE II. List of the gauge transformations for the sym-metry operations of the eight U(1) PSGs, where (x, y) is thecoordinate in the oblique coordinate system. For time rever-sal symmetry, all PSGs have the same gauge transformationW Tr = I2×2.

    where trr′,αβ is the spin-dependent hopping. Withthe extended Nambu spinor representation49 Ψr =

    (fr↑, f†r↓, fr↓,−f

    †r↑)

    T , HMF has a more compact form

    HMF = −1

    2

    ∑(r,r′)

    [Ψ†rurr′Ψr′ + h.c.

    ], (B2)

    where urr′ is a hopping matrix that is related to trr′,αβ ,

    urr′ =

    trr′,↑↑ 0 trr′,↑↓ 0

    0 −t∗rr′,↓↓ 0 t∗rr′,↓↑trr′,↓↑ 0 trr′,↓↓ 0

    0 t∗rr′,↑↓ 0 −t∗rr′,↑↑

    . (B3)

    1. Spatial symmetry

    First of all, the gauge transformation and spin rotationare commutative. So in the PSG classification, we onlyneed to focus on the gauge part of the PSG transforma-tion. In the canonical gauge IGG = {I2×2 ⊗ eiφσ

    z |φ ∈[0, 2π)}, the gauge transformation associated with a givensymmetry operation O takes the form

    GOr = I2×2 ⊗WOr ≡ I2×2 ⊗[(iσx)nOeiφO[r]σ

    z], (B4)

    where nO = 0, 1. For the symmetry multiplication ruleO1O2O3O4 = 1 where Oi is an unitary transformation,the corresponding PSG relation becomes

    GO1r GO2O2O3O4(r)G

    O3O3O4(r)G

    O4O4(r) ∈ IGG (B5)

    or equivalently,

    WO1r WO2O2O3O4(r)W

    O3O3O4(r)W

    O4O4(r)

    ∈ {eiφσz

    |φ ∈ [0, 2π)}. (B6)

    We start with T1 and T2, where

    WT1r = (iσx)nT1 , (B7)

    WT2r = (iσx)nT2 eiφT2 [r]σ

    z

    . (B8)

    Through Eq. (A10) that connects T1 and T2, one imme-diately has nT1 = nT2 . From Eq. (A11) where the to-tal number of T1 and T2 is odd, one immediately hasnT1 = nT2 = 0. So we have

    WT1r = 1, (B9)

    WT2r = eiφT2 [x,y]σ

    z

    . (B10)

    Using Eq. (A9), we have

    [WT1T1]−1[WT2T2][W

    T1T1][WT2T2]

    −1

    = T−11 (WT1)−1WT2T2W

    T1T1T−12 W

    −1T2

    ∈ {eiφσz

    |φ ∈ [0, 2π)}, (B11)

    which leads to the result

    φT2 [x+ 1, y]− φT2 [x, y] ≡ φ1 (B12)

    with φ1 to be determined. Since it is always possibleto choose a gauge such that φT2 [0, y] = 0, then we haveφT2 [x, y] = φ1x.

    Similarly, T−11 T−12 T1T2 = 1 leads to

    φT2 [x+ 1, y + 1]− φT2 [x, y + 1] = φ2. (B13)

    It is ready to find φ2 = φ1.We continue to find WS6r and W

    C2r . For the operation

    S6 with WS6r = (iσ

    x)nS6 eiφS6 [x,y]σz

    , Eq. (A11) leads to

    −φS6 [T1(r)] + φS6 [r] = −φ1y + φ3, (B14)−φS6 [T2(r)] + φS6 [r] = φ4 − φ1x+ φ1y, (B15)

    for nS6 = 0, and

    −φS6 [T1(r)] + φS6 [r] = −φ1y + φ3 (B16)−φS6 [T2(r)] + φS6 [r] = φ4 + φ1x+ φ1y. (B17)

    for nS6 = 1. So we obtain

    when nS6 = 0,

    φS6 [r] = φ1xy − φ3x− φ4y −φ1y(y − 1)

    2(B18)

    when nS6 = 1,

    φS6 [r] = φ1xy − φ3x− φ4y −φ1y(y − 1)

    2. (B19)

    For nS6 = 1, we further require φ1 = 0, π. S66 = 1 is

    automatically satisfied with the above relations for bothnS6 = 0 and nS6 = 1.

    For WC2r with WC2r = (iσ

    x)nC2 eiφC2 [x,y]σz

    , we need toconsider two separate cases with nc2 = 0, 1, respectively.If nC2 = 0, Eq. (A10) leads to

    −φT2 [C−12 T1(r)]− φC2 [T1(r)] + φC2 [r] = φ5, (B20)−φC2 [T2(r)] + φT2 [T2(r)] + φC2 [r] = φ6. (B21)

  • 8

    So we obtain φC2 [x, y] = −φ5x−φ6y−xyφ1 and φ1 = 0, πfor nC2 = 0. Similary, for nC2 = 1, we obtain φC2 [x, y] =−φ5x− φ6y − xyφ1.

    Using C22 = 1, we further have φ6 = −φ5 for nC2 = 0,and φ6 = φ6 for nC2 = 1. So we arrive at the result

    nC2 = 0, φC2 [x, y] = −φ5(x− y)− xyφ1, (B22)nC2 = 1, φC2 [x, y] = −φ5(x+ y)− xyφ1. (B23)

    Here, to simplify the above expression, we choose a puregauge tranformation W̃ ar = e

    ixσzφ5 . Under the puregauge transformation, the gauge part of the PSG trans-forms as

    WOr → W̃ arWOr W̃a†O−1(r). (B24)

    Clearly W̃ ar only modifies WT1 and WT2 by an overall

    phase shift, but WC2r becomes

    WC2r = (iσx)nC2 e−ixyφ1σ

    z

    (B25)

    for both nC2 = 0, 1, except that we require φ1 = 0, π fornC2 = 0.

    For the relation (S6C2)2 = 1, we need to consider the

    four cases with nS6 = 0, 1 and nC2 = 0, 1.For nS6 = nC2 = 0, we have φ1 = π, and (S6C2)

    2 = 1gives φ3 + 2φ4 = 0. We then introduce a pure gaugetransformation W̃ br ,

    W̃ br = e−i(x+y)φ4σz . (B26)

    After applying W̃ br , we have

    φC2 = −xyφ1, (B27)

    φS6 = xyφ1 − φ1y(y − 1)

    2(B28)

    with φ1 = 0, π.For nS6 = 0 and nC2 = 1, we obtain φ3 = 0. We

    introduce a pure gauge transformation W̃ cr ,

    W̃ cr = e−i(x−y)φ4σz . (B29)

    After applying W̃ br , we have

    φC2 = −xyφ1, (B30)

    φS6 = xyφ1 − φ1y(y − 1)

    2. (B31)

    For nS6 = 1 and nC2 = 0, we obtain φ3 = 0. We apply

    a pure gauge transformation W̃ br and obtain

    φC2 = −xyφ1, (B32)

    φS6 = xyφ1 − φ1y(y − 1)

    2. (B33)

    For nS6 = 1 and nC2 = 1, we obtain φ3 + 2φ4 = 0. We

    apply a pure gauge transformation W̃ cr and obtain

    φC2 = −xyφ1, (B34)

    φS6 = xyφ1 − φ1y(y − 1)

    2. (B35)

    In summary, we have

    WT1r = 1, WT2r = e

    iφ1x. (B36)

    and

    WC2r = (iσx)nC2 e−iφ1xyσ

    z

    , (B37)

    WS6r = (iσx)nS6 eiφ1[xy−

    y(y−1)2 ]σ

    z

    , (B38)

    where φ1 = 0, π for nC2 = 0 or nS6 = 1.

    2. Time reversal symmetry

    Because time reversal is an antiunitary symmetry, theproduct O−1T −1OT becomes

    (WOr )†[(W Tr )

    †WOr WTO−1(r)]

    ∗ (B39)

    for the PSGs, where W T is the gauge transformationassociated with the time reversal. We here redefine

    W Tr = W̄Tr (iσ

    y), (B40)

    so that

    O−1T −1OT → (WOr )†(W̄ Tr )†WOr W̄ TO−1(r). (B41)

    W̄ Tr has the general form W̄Tr = (iσ

    x)nT eiφT [r]σz

    .We start with nT = 0. The relation in Eq. (A13) leads

    to

    φT [x, y]− φT [x− 1, y] = −φ7, (B42)φT [x, y + 1]− φT [x, y] = −φ8, (B43)

    so we have φT [x, y] = −φ7x − φ8y. Applying this resultto Eq. (A14), we have

    −φC2 [y, x]− φT [y, x] + φC2 [y, x]+φT [x, y] = φ9,

    −φS6 [x, y]− φT [x, y] + φS6 [x, y]+φT [y,−x+ y] = φ10, (B44)

    for nC2 = nS6 = 0. The above equations giveφ7 = φ8 = 0, so we have W̄

    Tr = 1. Other cases can be

    obtained likewise. We find that for both nT = 0 andnT = 1, there is φT [x, y] = 0 and φ1 = 0, π. So we have

    W̄ Tr = 1, iσy, (B45)

    where we have used a global and uniform rotation eiπ4 σ

    z

    to rotate σx to the basis of σy.Including the time reversal, there are 16 PSG solutions.

    But for W̄ Tr = 1, the mean-field ansatz is found to vanisheverythere. This makes sense as these PSGs have T 2 = 1for the fermionic spinons that are expected to Kramersdoublets. So only 8 of them with T 2 = −1 for the spinonssurvive. Replacing eiφ1σ

    z

    with ±1, we present the PSGsolutions in the table of the main text.

  • 9

    Appendix C: Spinon band structures and mean-fieldHamiltonians

    As we establish in the previous section and the maintext, there are four U1A PSGs and four U1B PSGs. Inthe main text, we have argued that the experimental re-suls in YbMgGaO4 is against the U1B states. So herewe focus on the U1A states. From the U1A PSGs, it isstraight to obtain the spinon transformations. We listthe results in Tab. III.

    1. Spinon band structures

    Using Tab. III, we obtain the spinon mean-field Hamil-tonian. In particular, the U1A10 state gives vanishingspinon hoppings on the first and second neighbors, andthe U1A01 state gives an isotropic spinon hopping onboth first and second neighbors. The U1A10 state, aswe described in the main text, has symmetry protectedband touchings at the Γ, M and K points. The U1A11state has symmetry protected band touchings at the Γand M points.

    For the U1A10 state, the spinon mean-field Hamilto-nian has the form

    HU1A01MF =∑k

    hαβ(k)f†kαfkβ , (C1)

    where hαβ(k) is given by

    h(k) = d0(k)I2×2 +

    3∑µ=1

    dµ(k)σµ. (C2)

    In the main text, we have used (S6)3 and T to show

    d0(k) = 0 and the band touchings at Γ and M. To accountfor the band touching at the K point, we need to use S6and C2. Under S6,

    S6HS−16 =∑k

    [ei2π3 h(−S−16 (k))↑↓f

    †k↑fk↓ + h.c.

    ]= H, (C3)

    where h(k)↑↓ = dx(k)− idy(k). Since K is invariant un-der S6,

    dx(K)− idy(K) = ei2π3 [dx(K)− idy(K)], (C4)

    hence dx(K) = dy(K) = 0.The C2 symmetry constraints the dz term, we have

    C2HC−12 =

    ∑k

    dz(C−12 (k))f

    †k↓fk↓ − dz(C

    −12 (k))f

    †k↑fk↑

    = H. (C5)

    Since K is also invariant under C2, we obtain dz(K) =−dz(K). Hence dz(K) = 0. We conclude that h(K) = 0and there exists a band touching at K.

    For the U1A11 state, T and S6 are implemented in thesame way as the U1A01 state, and we arrive at the sameconclusion that there are band touchings at the Γ and Mpoints. At the K point, however, the band structure isgenerally gapped due to a nonzero dz.

    2. Spinon mean-field Hamiltonians

    The U1A00 state has the isotropic spinon hoppings onfirst and second neighboring bonds, and the mean-fieldHamiltonain HU1A00MF has already been given in the maintext. This states gives a large spinon Fermi surface inthe Brioullin zone. The spinon mean-field states of theU1A01 state and the U1A11 state are given by

    HU1A01MF =∑x,y

    t1

    [− if†(x+1,y),↑f(x,y),↓ − if

    †(x+1,y),↓f(x,y),↑ − e

    − iπ6 f†(x,y+1),↑f(x,y),↓

    +eiπ6 f†(x,y+1),↓f(x,y),↑ − e

    iπ6 f†(x+1,y+1),↑f(x,y),↓ + e

    − iπ6 f†(x+1,y+1),↓f(x,y),↑ + h.c.]

    +t2

    [ei2π3 f†(x+1,y−1),↑f(x,y),↓ + e

    iπ3 f†(x+1,y−1),↓f(x,y),↑ + f

    †(x+1,y+2),↑f(x,y),↓

    −f†(x+1,y+2),↓f(x,y),↑ + eiπ3 f†(x+2,y+1),↑f(x,y),↓ + e

    i2π3 f†(x+2,y+1),↓f(x,y),↑ + h.c.

    ], (C6)

  • 10

    TABLE III. The transformation for the spinons under four U1A PSGs that are labeled by U1AnC2nS6 .

    U(1) PSGs T1 T2 C2 S6

    U1A00f(x,y),↑ → f(x+1,y),↑f(x,y),↓ → f(x+1,y),↓

    f(x,y),↑ → f(x,y+1),↑f(x,y),↓ → f(x,y+1),↓

    f(x,y),↑ → eiπ6 f(y,x),↓

    f(x,y),↓ → ei5π6 f(y,x),↑

    f(x,y),↑ → e−iπ3 f(x−y,x),↑

    f(x,y),↓ → e+iπ3 f(x−y,x),↓

    U1A10f(x,y),↑ → f(x+1,y),↑f(x,y),↓ → f(x+1,y),↓

    f(x,y),↑ → f(x,y+1),↑f(x,y),↓ → f(x,y+1),↓

    f(x,y),↑ → eiπ6 f†(y,x),↑

    f(x,y),↓ → e−iπ6 f†(y,x),↓

    f(x,y),↑ → e−iπ3 f(x−y,x),↑

    f(x,y),↓ → e+iπ3 f(x−y,x),↓

    U1A01f(x,y),↑ → f(x+1,y),↑f(x,y),↓ → f(x+1,y),↓

    f(x,y),↑ → f(x,y+1),↑f(x,y),↓ → f(x,y+1),↓

    f(x,y),↑ → eiπ6 f(y,x),↓

    f(x,y),↓ → ei5π6 f(y,x),↑

    f(x,y),↑ → −e−iπ3 f†(x−y,x),↓

    f(x,y),↓ → e+iπ3 f†(x−y,x),↑

    U1A11f(x,y),↑ → f(x+1,y),↑f(x,y),↓ → f(x+1,y),↓

    f(x,y),↑ → f(x,y+1),↑f(x,y),↓ → f(x,y+1),↓

    f(x,y),↑ → eiπ6 f†(y,x),↑

    f(x,y),↓ → e−iπ6 f†(y,x),↓

    f(x,y),↑ → −e−iπ3 f†(x−y,x),↓

    f(x,y),↓ → e+iπ3 f†(x−y,x),↑

    and

    HU1A11MF =∑x,y

    t1

    [if†(x+1,y),↑f(x,y),↑ − if

    †(x+1,y),↓f(x,y),↓ + if

    †(x,y+1),↑f(x,y),↑

    −if†(x,y+1),↓f(x,y),↓ − if†(x+1,y+1),↑f(x,y),↑ + if

    †(x+1,y+1),↓f(x,y),↓ + h.c.

    ]+t′1

    [− f†(x+1,y),↑f(x,y),↓ + f

    †(x+1,y),↓f(x,y),↑ + e

    iπ3 f†(x,y+1),↑f(x,y),↓

    +ei2π3 f†(x,y+1),↓f(x,y),↑ + e

    i2π3 f†(x+1,y+1),↑f(x,y),↓ + e

    iπ3 f†(x+1,y+1),↓f(x,y),↑ + h.c.

    ]+t2

    [eiπ6 f†(x+1,y−1),↑f(x,y),↓ + e

    i5π6 f†(x+1,y−1),↓f(x,y),↑ − if

    †(x+1,y+2),↑f(x,y),↓

    −if†(x+1,y+2),↓f(x,y),↑ + ei5π6 f†(x−2,y−1),↑f(x,y),↓ + e

    iπ6 f†(x−2,y−1),↓f(x,y),↑ + h.c.

    ], (C7)

    where in both Hamiltonians t1, t′1 denote the first neigh-

    bor hoppings and t2 denotes the second neighbor hop-ping.

    The band structures for specific choices of the hoppingparameters are plotted in the main text. Clearly, weobserve the band touchings at the Γ, M and K points forthe U1A01 state, and band touchings at the Γ and Mpoints for the U1A11 state.

    Appendix D: The U1A00 state and thespectroscopic results

    1. Free spinon mean-field theory

    The spinon mean-field Hamiltonian of the U1A00 stateis

    HU1A00MF = −t1∑〈rr′〉,α

    f†rαfrα − t2∑

    〈〈rr′〉〉,α

    f†rαfrα, (D1)

    from which we compute the dynamic spin structure factorfor different choices t2/t1. The dynamic spin structure

    factor is given by

    S(q, ω) = 1N

    ∑r,r′

    eiq·(r−r′)

    ∫dt e−iωt

    〈ΨU1A00MF |S−r (t)S+r′(0)|ΨU1A00MF 〉

    =∑n

    δ(ω − ξnq) |〈n|S+q |ΨU1A00MF 〉|2, (D2)

    where N is the total number of spins, the summationis over all mean-field states with the spinon particle-holeexcitation, ξnq is the energy of the n-th excited state withthe momentum q. The results are depicted in Fig. 4(a-e)and are consistent with the inelastic neutron scatteringresults36,37. All the results so far are independent fromany microscopic spin interaction.

  • 11

    FIG. 4. (a-e) Dynamic spin structure factor for the free spinon theory of the U1A00 state with different values of t2/t1. (f-h)The evolution of SRPA(q, ω) as a function of J±±. In all subfigures, the energy transfer is normalized against the correspondingbandwidth B. The parameter α is defined as Jzz/t1.

    2. Variational calculation and random phaseapproximation

    Here we consider the microscopic spin Hamiltonianthat was introduced in Refs. 34 and 35,

    Hspin =∑〈rr′〉

    JzzSzrS

    zr′ + J±(S

    +r S−r′ + S

    −r S

    +r′)

    +J±±(γrr′S+r S

    +r′ + γ

    ∗rr′S

    −r S−r′)

    − i2Jz±[(γ

    ∗rr′S

    +r − γrr′S−r )Szr′

    +Szr(γ∗rr′S

    +r′ − γrr′S

    −r′)], (D3)

    where γrr′ = 1, ei2π/3, e−i2π/3 for rr′ along the a1,a2

    and a3 bonds, respectively. Here, a3 = −a1 − a2. Itwas suggested and demonstrated that the anisotropicJ±± and Jz± interactions compete with the XXZ part ofthe Hamiltonian and may lead to disordered state34,35,39.Our calculation does show the enhancement of quantumfluctuation in certain regions of the phase diagram35.Here we comment about the choices of the exchange cou-plings in the main text and in the following calculation.The Jzz and J± couplings can be determined by the

    Curie-Weiss temperature measurement on a single crys-tal sample. The complication comes from the subtractionof the Van Vleck susceptibility. Due to the Ga3+/Mg2+

    exchange disorder in the non-magnetic layers, althoughthese ions do not directly enter the Yb exchange path,it may modify the local crystal field environment of theYb3+ ion and thus lead to some complication and varia-tion of the Van Vleck susceptibility. As a result, the veryprecise determination of the Jzz and J± couplings can bean issue. That may explain some differences of the Jzzand J± couplings that were obtained

    34–37,39. Partly forthe same reason, the results on J±± and Jz± may alsobe affected. However, quantum spin liquid, if it exists asthe ground state of our generic model, is expected to bea phase that covers a finite region of the phase diagram.Therefore, the very precise value of the couplings maynot be quite necessary from this point of view. There-fore, we here rely on our previous results of the quantumfluctuation for the mean-field phase diagram that indi-cates strong fluctations in certain parameter regimes. Wechoose the exchange parameters from these disordered re-gions.

    For this spin Hamiltonian, the mean-field variationalenergy is given as

    Evar = 〈ΨU1A00MF |Hspin|ΨU1A00MF 〉 =1

    L2

    ∑q

    〈ΨU1A00MF |Jzz(q)SzqSz−q + 2J±(q)S+q S−−q|ΨU1A00MF 〉

    =1

    L2

    ∑q

    [Jzz(q)

    ∑n

    ∣∣〈n|Szq |ΨU1A00MF 〉∣∣2 + 2J±(q)∑n

    ∣∣〈n|S+q |ΨU1A00MF 〉∣∣2]

    =1

    L4

    ∑q

    Jzz(q)4

    ∑n,k

    ∣∣∣〈n|f†k+q,↑fk,↑ − f†k+q,↓fk,↓|ΨU1A00MF 〉∣∣∣2 + 2J±(q)∑n,k

    ∣∣∣〈n|f†k+q,↑fk,↓|ΨU1A00MF 〉∣∣∣2 , (D4)

  • 12

    where we have omitted J±± and Jz± because they do notconserve spin, therefore their contribution to Evar is zero.This is an artifact of the free spinon theory of HU1A00MFthat only includes isotropic spinon hoppings for the firsttwo neighbors.

    Due to the isotropic spinon hoppings, HU1A00MF does notexplicitly reflect the absence of spin-rotational symmetrythat is brought by the J±± and Jz± interactions. To in-corporate the J±± and Jz± interactions, as we describein the main text, we followed the phenomenological treat-ment for the “t-J” model in the context of cuprate super-conductors60 and consider H = HU1A00MF + H

    ′spin, where

    H ′spin are the J±± and Jz± interactions. In the parton

    construction, H ′spin is treated as the spinon interactionsand thus introduces the spin rotational symmetry break-ing. With a random phase approximation for the inter-action H ′spin, we obtain the dynamic spin susceptibility

    60

    χRPA(q, ω) =[1− χ0(q, ω)J (q)

    ]−1χ0(q, ω), (D5)

    where χ0 is the free-spinon susceptibility, and J (q) is thespin exchange matrix from H ′spin,

    J (q) =2(uq − vq)J±± −2

    √3wqJ±± −

    √3wqJz±

    −2√

    3wqJ±± 2(−uq + vq)J±± (uq − vq) Jz±−√

    3wqJz± (uq − vq) Jz± 0

    (D6)with uq = cos(q · a1), vq = 12 (cos(q · a2) + cos(q · a3)),and wq =

    12 (cos(q · a2)− cos(q · a3)). The renormalized

    SRPA(q, ω) can be read off from χRPA via SRPA(q, ω) =− 1π Im

    [χRPA(q, ω)

    ]+−and is plotted in Fig. 3(b) in the

    main text.

    The very precise values of J±± and Jz± cannot be de-termined from the existing data-rich neutron scatteringexperiment in a strong field normal to the triangularplane. This is partly due to the experimental resolu-tion, and is also due to the fact that the linear spin wavespectrum for the field normal to the plane is indepen-dent of Jz± and is not quite sensitive to J±±

    35,39. InFig. 3(b) of the main text, instead, we choose (J±±, Jz±)to fall into the disordered region of the phase diagram inRef. 35 where the quantum fluctuations are expected tobe strong35.

    Appendix E: The U1B states

    In this section we use PSG to determine the free spinonmean-field Hamiltonian for the U1B states to the first andsecond spinon hoppings. In Fig. 5, we further presenttheir spectroscopic features for comparison. Like thenotation for U1As, the U1B states are also labeled byU1BnC2nS6 .

    1. The U1B00 state

    For the π-flux states, the dynamic spin structure fac-tor has an enhanced periodicity due to anticommutativelattice translations. One direct consequence of the pe-riodicity is that Γ and M become equivalent, and theV-shaped upper excitation edge in Ref. 36 cannot be re-produced for the U1B states.

    We choose the spinon basis in the momentum space

    fk,I = (fA,k,↑, fB,k,↑, fA,k,↓, fB,k,↓)T , where A and B de-

    note the two inequivalent sites in each unit cell due to theπ flux.

    The Hamiltonian is written in terms of the Dirac ma-trices Γa and their anticommutators

    Γab = [Γa,Γb]/(2i). (E1)

    The representation is chosen to be Γ(1,2,3,4,5) = (σx ⊗1, σz ⊗ 1, σy ⊗ τx, σy ⊗ τy, σy ⊗ τz). Γa and Γab is oddunder time reversal except when a = 4 or b = 4. TheHamiltonian is thus

    h(k) =

    5∑a=1

    da(k)Γa +

    5∑a

  • 13

    FIG. 5. Dynamic spin structure factor for six free spinon mean-field states other than U1A00. Note the U1A10 Hamiltonian isidentically zero for the first and second neighbor hoppings. None of them is consistent with the spinon Fermi surface picture.In all subfigures, the energy transfer is normalized against the corresponding bandwidth B.

    3. The U1B10 state

    d3(k) = −√

    3t1 sin[(kx −

    √3ky)/2

    ],

    d4(k) =√

    3t1 cos[(kx +

    √3ky)/2

    ],

    d23(k) = −t1 sin[(kx −

    √3ky)/2

    ],

    d24(k) = −t1 cos[(kx +

    √3ky)/2

    ],

    d25(k) = 2t1 sin kx. (E5)

    4. The U1B11 state

    d3(k) = −√

    3t2 sin[(3kx +

    √3ky)/2

    ],

    d4(k) = −√

    3t2 cos[(3kx −

    √3ky)/2

    ],

    d23(k) = −t2 sin[(3kx +

    √3ky)/2

    ],

    d24(k) = t2 cos[(3kx −

    √3ky)/2

    ],

    d25(k) = −2t2 sin(√

    3ky),

    d34(k) = 2t1 cos(kx),

    d35(k) = −2t1 sin[(kx +

    √3ky)/2

    ],

    d45(k) = −2t1 cos[(kx −

    √3ky)/2

    ]. (E6)

  • 14

    [email protected] William Witczak-Krempa, Gang Chen, Yong Baek Kim,

    and Leon Balents, “Correlated Quantum Phenomena inthe Strong Spin-Orbit Regime,” Annual Review of Con-densed Matter Physics 5, 57–82 (2014).

    2 Jeffrey G. Rau, Eric Kin-Ho Lee, and Hae-Young Kee,“Spin-Orbit Physics Giving Rise to Novel Phases in Cor-related Systems: Iridates and Related Materials,” AnnualReview of Condensed Matter Physics 7, 195–221 (2016).

    3 G. Jackeli and G. Khaliullin, “Mott Insulators in theStrong Spin-Orbit Coupling Limit: From Heisenberg to aQuantum Compass and Kitaev Models,” Phys. Rev. Lett.102, 017205 (2009).

    4 Gang Chen and Leon Balents, “Spin-orbit effects inNa4Ir3O8: A hyper-kagome lattice antiferromagnet,”Phys. Rev. B 78, 094403 (2008).

    5 Ji ř́ı Chaloupka, George Jackeli, and Giniyat Khaliullin,“Kitaev-Heisenberg Model on a Honeycomb Lattice: Pos-sible Exotic Phases in Iridium Oxides A2IrO3,” Phys. Rev.Lett. 105, 027204 (2010).

    6 Dmytro Pesin and Leon Balents, “Mott physics and bandtopology in materials with strong spin–orbit interaction,”Nature Physics 6, 376–381 (2010).

    7 Shigeki Onoda and Yoichi Tanaka, “Quantum melting ofspin ice: Emergent cooperative quadrupole and chirality,”Phys. Rev. Lett. 105, 047201 (2010).

    8 Lucile Savary and Leon Balents, “Coulombic quantum liq-uids in spin-1/2 pyrochlores,” Phys. Rev. Lett. 108, 037202(2012).

    9 Kate A. Ross, Lucile Savary, Bruce D. Gaulin, and LeonBalents, “Quantum Excitations in Quantum Spin Ice,”Phys. Rev. X 1, 021002 (2011).

    10 Yi-Ping Huang, Gang Chen, and Michael Hermele,“Quantum Spin Ices and Topological Phases from Dipolar-Octupolar Doublets on the Pyrochlore Lattice,” Phys. Rev.Lett. 112, 167203 (2014).

    11 Gang Chen and Leon Balents, “Spin-orbit coupling in d2

    ordered double perovskites,” Phys. Rev. B 84, 094420(2011).

    12 Hamid R. Molavian, Michel J. P. Gingras, and BenjaminCanals, “Dynamically Induced Frustration as a Route toa Quantum Spin Ice State in Tb2Ti2O7 via Virtual Crys-tal Field Excitations and Quantum Many-Body Effects,”Phys. Rev. Lett. 98, 157204 (2007).

    13 Gang Chen and Yong Baek Kim, “Anomalous enhance-ment of the Wilson ratio in a quantum spin liquid: Thecase of Na4Ir3O8,” Phys. Rev. B 87, 165120 (2013).

    14 R. Applegate, N. R. Hayre, R. R. P. Singh, T. Lin, A. G. R.Day, and M. J. P. Gingras, “Vindication of Yb2Ti2O7 asa Model Exchange Quantum Spin Ice,” Phys. Rev. Lett.109, 097205 (2012).

    15 K. A. Ross, J. P. C. Ruff, C. P. Adams, J. S. Gard-ner, H. A. Dabkowska, Y. Qiu, J. R. D. Copley, andB. D. Gaulin, “Two-Dimensional Kagome Correlations andField Induced Order in the Ferromagnetic XY PyrochloreYb2Ti2O7,” Phys. Rev. Lett. 103, 227202 (2009).

    16 Gang Chen and Michael Hermele, “Magnetic orders andtopological phases from f -d exchange in pyrochlore iri-dates,” Phys. Rev. B 86, 235129 (2012).

    17 Lucile Savary, Xiaoqun Wang, Hae-Young Kee, Yong BaekKim, Yue Yu, and Gang Chen, “Quantum spin ice on

    the breathing pyrochlore lattice,” Phys. Rev. B 94, 075146(2016).

    18 SungBin Lee, Eric Kin-Ho Lee, Arun Paramekanti, andYong Baek Kim, “Order-by-disorder and magnetic field re-sponse in the Heisenberg-Kitaev model on a hyperhoney-comb lattice,” Phys. Rev. B 89, 014424 (2014).

    19 SungBin Lee, Shigeki Onoda, and Leon Balents, “Genericquantum spin ice,” Phys. Rev. B 86, 104412 (2012).

    20 Eric Kin-Ho Lee, Robert Schaffer, Subhro Bhattacharjee,and Yong Baek Kim, “Heisenberg-kitaev model on the hy-perhoneycomb lattice,” Phys. Rev. B 89, 045117 (2014).

    21 Shigeki Onoda and Yoichi Tanaka, “Quantum fluctua-tions in the effective pseudospin- 1

    2model for magnetic py-

    rochlore oxides,” Phys. Rev. B 83, 094411 (2011).22 Lieh-Jeng Chang, Shigeki Onoda, Yixi Su, Ying-Jer Kao,

    Ku-Ding Tsuei, Yukio Yasui, Kazuhisa Kakurai, andMartin Richard Lees, “Higgs transition from a magneticCoulomb liquid to a ferromagnet in Yb2Ti2O7,” NatureCommunications 3, 992 (2012).

    23 J. S. Gardner, S. R. Dunsiger, B. D. Gaulin, M. J. P. Gin-gras, J. E. Greedan, R. F. Kiefl, M. D. Lumsden, W. A.MacFarlane, N. P. Raju, J. E. Sonier, I. Swainson, andZ. Tun, “Cooperative Paramagnetism in the GeometricallyFrustrated Pyrochlore Antiferromagnet Tb2Ti2O7,” Phys.Rev. Lett. 82, 1012–1015 (1999).

    24 Fei-Ye Li, Yao-Dong Li, Yue Yu, Arun Paramekanti, andGang Chen, “Kitaev materials beyond iridates: order byquantum disorder and weyl magnons in rare-earth doubleperovskites,” Phys. Rev. B 95, 085132 (2017).

    25 Gang Chen, ““Magnetic monopole” condensation of thepyrochlore ice U(1) quantum spin liquid: Application toPr2Ir2O7 and Yb2Ti2O7,” Phys. Rev. B 94, 205107 (2016).

    26 M J P Gingras and P A McClarty, “Quantum spin ice: asearch for gapless quantum spin liquids in pyrochlore mag-nets,” Reports on Progress in Physics 77, 056501 (2014).

    27 Yao-Dong Li and Gang Chen, “Symmetry enriched U(1)topological orders for dipole-octupole doublets on a py-rochlore lattice,” Phys. Rev. B 95, 041106 (2017).

    28 E. Lhotel, S. Petit, S. Guitteny, O. Florea,M. Ciomaga Hatnean, C. Colin, E. Ressouche, M. R. Lees,and G. Balakrishnan, “Fluctuations and All-In All-OutOrdering in Dipole-Octupole Nd2Zr2O7,” Phys. Rev. Lett.115, 197202 (2015).

    29 Owen Benton, Olga Sikora, and Nic Shannon, “Seeingthe light: Experimental signatures of emergent electromag-netism in a quantum spin ice,” Phys. Rev. B 86, 075154(2012).

    30 Anson W. C. Wong, Zhihao Hao, and Michel J. P. Gingras,“Ground state phase diagram of generic xy pyrochloremagnets with quantum fluctuations,” Phys. Rev. B 88,144402 (2013).

    31 Gang Chen, Rodrigo Pereira, and Leon Balents, “Exoticphases induced by strong spin-orbit coupling in ordereddouble perovskites,” Phys. Rev. B 82, 174440 (2010).

    32 S. H. Curnoe, “Structural distortion and the spin liquidstate in Tb2Ti2O7,” Phys. Rev. B 78, 094418 (2008).

    33 Yuesheng Li, Haijun Liao, Zhen Zhang, Shiyan Li, FengJin, Langsheng Ling, Lei Zhang, Youming Zou, Li Pi,Zhaorong Yang, Junfeng Wang, Zhonghua Wu, and Qing-ming Zhang, “Gapless quantum spin liquid ground statein the two-dimensional spin-1/2 triangular antiferromag-

    mailto:[email protected]://dx.doi.org/ 10.1146/annurev-conmatphys-020911-125138http://dx.doi.org/ 10.1146/annurev-conmatphys-020911-125138http://dx.doi.org/ 10.1103/PhysRevLett.102.017205http://dx.doi.org/ 10.1103/PhysRevLett.102.017205http://dx.doi.org/10.1103/PhysRevB.78.094403http://dx.doi.org/ 10.1103/PhysRevLett.105.027204http://dx.doi.org/ 10.1103/PhysRevLett.105.027204http://dx.doi.org/10.1103/PhysRevLett.105.047201http://dx.doi.org/ 10.1103/PhysRevLett.108.037202http://dx.doi.org/ 10.1103/PhysRevLett.108.037202http://dx.doi.org/10.1103/PhysRevX.1.021002http://dx.doi.org/10.1103/PhysRevLett.112.167203http://dx.doi.org/10.1103/PhysRevLett.112.167203http://dx.doi.org/ 10.1103/PhysRevB.84.094420http://dx.doi.org/ 10.1103/PhysRevB.84.094420http://dx.doi.org/ 10.1103/PhysRevLett.98.157204http://dx.doi.org/ 10.1103/PhysRevB.87.165120http://dx.doi.org/10.1103/PhysRevLett.109.097205http://dx.doi.org/10.1103/PhysRevLett.109.097205http://dx.doi.org/ 10.1103/PhysRevLett.103.227202http://dx.doi.org/10.1103/PhysRevB.86.235129http://dx.doi.org/10.1103/PhysRevB.94.075146http://dx.doi.org/10.1103/PhysRevB.94.075146http://dx.doi.org/10.1103/PhysRevB.89.014424http://dx.doi.org/10.1103/PhysRevB.86.104412http://dx.doi.org/ 10.1103/PhysRevB.89.045117http://dx.doi.org/10.1103/PhysRevB.83.094411http://dx.doi.org/10.1038/ncomms1989http://dx.doi.org/10.1038/ncomms1989http://dx.doi.org/ 10.1103/PhysRevLett.82.1012http://dx.doi.org/ 10.1103/PhysRevLett.82.1012http://dx.doi.org/10.1103/PhysRevB.95.085132http://dx.doi.org/ 10.1103/PhysRevB.94.205107http://stacks.iop.org/0034-4885/77/i=5/a=056501http://dx.doi.org/10.1103/PhysRevB.95.041106http://dx.doi.org/ 10.1103/PhysRevLett.115.197202http://dx.doi.org/ 10.1103/PhysRevLett.115.197202http://dx.doi.org/10.1103/PhysRevB.86.075154http://dx.doi.org/10.1103/PhysRevB.86.075154http://dx.doi.org/10.1103/PhysRevB.88.144402http://dx.doi.org/10.1103/PhysRevB.88.144402http://dx.doi.org/ 10.1103/PhysRevB.82.174440http://dx.doi.org/ 10.1103/PhysRevB.78.094418

  • 15

    net YbMgGaO4,” Scientific Reports 5, 16419 (2015).34 Yuesheng Li, Gang Chen, Wei Tong, Li Pi, Juanjuan Liu,

    Zhaorong Yang, Xiaoqun Wang, and Qingming Zhang,“Rare-Earth Triangular Lattice Spin Liquid: A Single-Crystal Study of YbMgGaO4,” Phys. Rev. Lett. 115,167203 (2015).

    35 Yao-Dong Li, Xiaoqun Wang, and Gang Chen,“Anisotropic spin model of strong spin-orbit-coupled trian-gular antiferromagnets,” Phys. Rev. B 94, 035107 (2016).

    36 Yao Shen, Yao-Dong Li, Hongliang Wo, Yuesheng Li,Shoudong Shen, Bingying Pan, Qisi Wang, H. C. Walker,P. Steffens, M Boehm, Yiqing Hao, D. L. Quintero-Castro,L. W. Harriger, Lijie Hao, Siqin Meng, Qingming Zhang,Gang Chen, and Jun Zhao, “Spinon Fermi surface in a tri-angular lattice quantum spin liquid YbMgGaO4,” Nature540, 559–562 (2016).

    37 Joseph A. M. Paddison, Zhiling Dun, Georg Ehlers, Yao-hua Liu, Matthew B. Stone, Haidong Zhou, and MartinMourigal, “Continuous excitations of the triangular-latticequantum spin liquid YbMgGaO4,” Nature Physics, arXivpreprint 1607.03231 (2016).

    38 Yuesheng Li, Devashibhai Adroja, Pabitra K. Biswas, Pe-ter J. Baker, Qian Zhang, Juanjuan Liu, Alexander A.Tsirlin, Philipp Gegenwart, and Qingming Zhang, “MuonSpin Relaxation Evidence for the U(1) Quantum Spin-Liquid Ground State in the Triangular AntiferromagnetYbMgGaO4,” Phys. Rev. Lett. 117, 097201 (2016).

    39 Yao-Dong Li, Yao Shen, Yuesheng Li, Jun Zhao, andGang Chen, “The effect of spin-orbit coupling on theeffective-spin correlation in YbMgGaO4,” arXiv preprint1608.06445 (2016).

    40 Yao-Dong Li, Xiaoqun Wang, and Gang Chen, “Hiddenmultipolar orders of dipole-octupole doublets on a trian-gular lattice,” Phys. Rev. B 94, 201114 (2016).

    41 Yao-Dong Li and Gang Chen, “Detecting spin fractional-ization in a spinon fermi surface spin liquid,” Phys. Rev.B 96, 075105 (2017).

    42 Haruki Watanabea, Hoi Chun Po, Ashvin Vishwanath,and Michael Zaletel, “Filling constraints for spin-orbit cou-pled insulators in symmorphic and nonsymmorphic crys-tals,” PNAS 112, 14551–14556 (2015).

    43 Y. Xu, J. Zhang, Y. S. Li, Y. J. Yu, X. C. Hong,Q. M. Zhang, and S. Y. Li, “Absence of Magnetic Ther-mal Conductivity in the Quantum Spin-Liquid CandidateYbMgGaO4,” Phys. Rev. Lett. 117, 267202 (2016).

    44 Olexei I. Motrunich, “Variational study of triangular lat-tice spin-1/2 model with ring exchanges and spin liquidstate in κ-(ET)2Cu2(CN)3,” Phys. Rev. B 72, 045105(2005).

    45 Sung-Sik Lee and Patrick A. Lee, “U(1) Gauge Theory ofthe Hubbard Model: Spin Liquid States and Possible Ap-plication to κ-(BEDT-TTF)2Cu2(CN)3,” Phys. Rev. Lett.95, 036403 (2005).

    46 Patrick A. Lee and Naoto Nagaosa, “Gauge theory of thenormal state of high-Tc superconductors,” Phys. Rev. B46, 5621–5639 (1992).

    47 Xiao-Gang Wen, “Quantum orders and symmetric spin liq-uids,” Phys. Rev. B 65, 165113 (2002).

    48 See the Supplementary information.49 Johannes Reuther, Shu-Ping Lee, and Jason Alicea, “Clas-

    sification of spin liquids on the square lattice with strongspin-orbit coupling,” Phys. Rev. B 90, 174417 (2014).

    50 Gang Chen, Andrew Essin, and Michael Hermele, “Majo-rana spin liquids and projective realization of SU(2) spin

    symmetry,” Phys. Rev. B 85, 094418 (2012).51 Michael Hermele, “SU(2) gauge theory of the Hubbard

    model and application to the honeycomb lattice,” Phys.Rev. B 76, 035125 (2007).

    52 Samuel Bieri, Claire Lhuillier, and Laura Messio, “Projec-tive symmetry group classification of chiral spin liquids,”Phys. Rev. B 93, 094437 (2016).

    53 Yi-Zhuang You, Itamar Kimchi, and Ashvin Vishwanath,“Doping a spin-orbit Mott insulator: Topological super-conductivity from the Kitaev-Heisenberg model and pos-sible application to (Na2/Li2)IrO3,” Phys. Rev. B 86,085145 (2012).

    54 Robert Schaffer, Subhro Bhattacharjee, and Yong BaekKim, “Spin-orbital liquids in non-Kramers magnets on thekagome lattice,” Phys. Rev. B 88, 174405 (2013).

    55 Yuan-Ming Lu, “Symmetry protected gapless Z2 spin liq-uids,” arXiv preprint 1606.05652 (2016).

    56 Xiao-Gang Wen, “Quantum order: a quantum entangle-ment of many particles,” Physics Letters A 300, 175 – 181(2002).

    57 Andrew M. Essin and Michael Hermele, “Spectroscopicsignatures of crystal momentum fractionalization,” Phys.Rev. B 90, 121102 (2014).

    58 In the previous work36, only the nearest-neighbor spinonhopping is included.

    59 Sandor Toth, Katharina Rolfs, Rolfs, Andrew R. Wildes,and Christian Ruegg, “Strong exchange anisotropy inYbMgGaO4 from polarized neutron diffraction,” arXivpreprint arXiv:1705.05699 (2017).

    60 Jan Brinckmann and Patrick A. Lee, “Slave Boson Ap-proach to Neutron Scattering in YBa2Cu3O6+y Supercon-ductors,” Phys. Rev. Lett. 82, 2915–2918 (1999).

    61 Yuesheng Li, Devashibhai Adroja, Robert I. Bewley, DavidVoneshen, Alexander A. Tsirlin, Philipp Gegenwart, andQingming Zhang, “Crystalline Electric-Field Randomnessin the Triangular Lattice Spin-Liquid YbMgGaO4,” Phys.Rev. Lett. 118, 107202 (2017).

    62 Yao Shen, Yao-Dong Li, H. C. Walker, P. Steffens,M. Boehm, Xiaowen Zhang, Shoudong Shen, HongliangWo, Gang Chen, and Jun Zhao, “Fractionalized exci-tations in the partially magnetized spin liquid candidateYbMgGaO4,” arXiv preprint arXiv:1708.06655 (2017).

    63 Changle Liu, Rong Yu, and Xiaoqun Wang, “Semiclassicalground-state phase diagram and multi-q phase of a spin-orbit-coupled model on triangular lattice,” Phys. Rev. B94, 174424 (2016).

    64 Qiang Luo, Shijie Hu, Bin Xi, Jize Zhao, and Xiao-qun Wang, “Ground-state phase diagram of an anisotropicspin- 1

    2model on the triangular lattice,” Phys. Rev. B 95,

    165110 (2017).65 Z. Zhu, P.A. Maksimov, S.R. White, and A.L. Cherny-

    shev, “Disorder-induced Mimicry of a Spin Liquid inYbMgGaO4,” arXiv preprint arXiv:1703.02971 (2017).

    66 M. Hirschberger, J. W. Krizan, R. J. Cava, and N. P.Ong, “Large thermal hall conductivity of neutral spin ex-citations in a frustrated quantum magnet,” Science 348,106–109 (2015).

    67 S. J. Li, Z. Y. Zhao, C. Fan, B. Tong, F. B. Zhang, J. Shi,J. C. Wu, X. G. Liu, H. D. Zhou, X. Zhao, and X. F.Sun, “Low-temperature thermal conductivity of Dy2Ti2O7and Yb2Ti2O7 single crystals,” Phys. Rev. B 92, 094408(2015).

    68 Q. J. Li, Z. Y. Zhao, C. Fan, F. B. Zhang, H. D. Zhou,X. Zhao, and X. F. Sun, “Phonon-glass-like behavior of

    http://dx.doi.org/ 10.1038/srep16419http://dx.doi.org/ 10.1103/PhysRevLett.115.167203http://dx.doi.org/ 10.1103/PhysRevLett.115.167203http://dx.doi.org/10.1103/PhysRevB.94.035107http://dx.doi.org/10.1038/nature20614http://dx.doi.org/10.1038/nature20614https://arxiv.org/abs/1607.03231https://arxiv.org/abs/1607.03231http://dx.doi.org/ 10.1103/PhysRevLett.117.097201http://arxiv.org/abs/1608.06445http://arxiv.org/abs/1608.06445http://dx.doi.org/10.1103/PhysRevB.94.201114http://dx.doi.org/10.1103/PhysRevB.96.075105http://dx.doi.org/10.1103/PhysRevB.96.075105http://dx.doi.org/10.1073/pnas.1514665112http://dx.doi.org/ 10.1103/PhysRevLett.117.267202http://dx.doi.org/10.1103/PhysRevB.72.045105http://dx.doi.org/10.1103/PhysRevB.72.045105http://dx.doi.org/10.1103/PhysRevLett.95.036403http://dx.doi.org/10.1103/PhysRevLett.95.036403http://dx.doi.org/10.1103/PhysRevB.46.5621http://dx.doi.org/10.1103/PhysRevB.46.5621http://dx.doi.org/ 10.1103/PhysRevB.65.165113http://dx.doi.org/10.1103/PhysRevB.90.174417http://dx.doi.org/ 10.1103/PhysRevB.85.094418http://dx.doi.org/10.1103/PhysRevB.76.035125http://dx.doi.org/10.1103/PhysRevB.76.035125http://dx.doi.org/ 10.1103/PhysRevB.93.094437http://dx.doi.org/ 10.1103/PhysRevB.86.085145http://dx.doi.org/ 10.1103/PhysRevB.86.085145http://dx.doi.org/10.1103/PhysRevB.88.174405https://arxiv.org/abs/1606.05652http://dx.doi.org/ http://dx.doi.org/10.1016/S0375-9601(02)00808-3http://dx.doi.org/ http://dx.doi.org/10.1016/S0375-9601(02)00808-3http://dx.doi.org/10.1103/PhysRevB.90.121102http://dx.doi.org/10.1103/PhysRevB.90.121102http://dx.doi.org/10.1103/PhysRevLett.82.2915http://dx.doi.org/ 10.1103/PhysRevLett.118.107202http://dx.doi.org/ 10.1103/PhysRevLett.118.107202http://dx.doi.org/10.1103/PhysRevB.94.174424http://dx.doi.org/10.1103/PhysRevB.94.174424http://dx.doi.org/ 10.1103/PhysRevB.95.165110http://dx.doi.org/ 10.1103/PhysRevB.95.165110http://dx.doi.org/10.1126/science.1257340http://dx.doi.org/10.1126/science.1257340http://dx.doi.org/10.1103/PhysRevB.92.094408http://dx.doi.org/10.1103/PhysRevB.92.094408

  • 16

    magnetic origin in single-crystal Tb2Ti2O7,” Phys. Rev. B87, 214408 (2013).

    69 Yuji Matsuda, (2017), Talk at workshop on topologicalmaterials, Kyoto University.

    70 Tian-Heng Han, Joel S Helton, Shaoyan Chu, Daniel GNocera, Jose A Rodriguez-Rivera, Collin Broholm, andYoung S Lee, “Fractionalized excitations in the spin-liquidstate of a kagome-lattice antiferromagnet,” Nature 492,406–410 (2012).

    71 Y. Shimizu, K. Miyagawa, K. Kanoda, M. Maesato, andG. Saito, “Spin liquid state in an organic mott insulatorwith a triangular lattice,” Phys. Rev. Lett. 91, 107001(2003).

    72 T. Itou, A. Oyamada, S. Maegawa, M. Tamura, andR. Kato, “Quantum spin liquid in the spin-12 triangularantiferromagnet EtMe3Sb[Pd(dmit)2]2,” Phys. Rev. B 77,104413 (2008).

    73 Satoshi Yamashita, Yasuhiro Nakazawa, Masaharu Oguni,Yugo Oshima, Hiroyuki Nojiri, Yasuhiro Shimizu, KazuyaMiyagawa, and Kazushi Kanoda, “Thermodynamic prop-erties of a spin-1/2 spin-liquid state in a kappa-type organicsalt,” Nature Physics 4, 459–462 (2008).

    74 Y. Q. Liu, S. J. Zhang, J. L. Lv, S. K. Su, T. Dong, GangChen, and N. L. Wang, “Revealing a triangular lattice

    ising antiferromagnet in a single-crystal CeCd3As3,” arXivpreprint 1612.03720 (2016).

    75 Shohei Higuchi, Yuki Noshima, Naoki Shirakawa, MasamiTsubota, and Jiro Kitagawa, “Optical, transport and mag-netic properties of new compound CeCd3P3,” MaterialsResearch Express 3, 056101 (2016).

    76 Andr T. Nientiedt and Wolfgang Jeitschko, “The Seriesof Rare Earth Zinc Phosphides RZn3P3 (R=Y, La−Nd,Sm, Gd−Er) and the Corresponding Cadmium CompoundPrCd3P3,” Journal of Solid State Chemistry 146, 478 –483 (1999).

    77 A. Yamada, N. Hara, K. Matsubayashi, K. Munakata,C. Ganguli, A. Ochiai, T. Matsumoto, and Y. Uwatoko,“Effect of pressure on the electrical resistivity of CeZn3P3,”J. Phys.: Conf. Ser. 215, 012031 (2010).

    78 Stanislav S. Stoyko and Arthur Mar, “Ternary Rare-EarthArsenides REZn3As3 (RE = La−Nd, Sm) and RECd3As3(RE = La−Pr),” Inorganic Chemistry 50, 11152–11161(2011).

    79 M. B. Sanders, F. A. Cevallos, and R. J. Cava, “Mag-netism in the KBaRE(BO3)2 (RE= Sm, Eu, Gd, Tb, Dy,Ho, Er, Tm, Yb, Lu) series: materials with a triangularrare earth lattice,” arXiv preprint 1611.08548 (2016).

    http://dx.doi.org/ 10.1103/PhysRevB.87.214408http://dx.doi.org/ 10.1103/PhysRevB.87.214408http://dx.doi.org/10.1103/PhysRevLett.91.107001http://dx.doi.org/10.1103/PhysRevLett.91.107001http://dx.doi.org/10.1103/PhysRevB.77.104413http://dx.doi.org/10.1103/PhysRevB.77.104413http://dx.doi.org/ 10.1038/nphys942http://stacks.iop.org/2053-1591/3/i=5/a=056101http://stacks.iop.org/2053-1591/3/i=5/a=056101http://dx.doi.org/http://dx.doi.org/10.1006/jssc.1999.8396http://dx.doi.org/http://dx.doi.org/10.1006/jssc.1999.8396http://dx.doi.org/10.1088/1742-6596/215/1/012031http://dx.doi.org/10.1021/ic201708xhttp://dx.doi.org/10.1021/ic201708xhttps://arxiv.org/abs/1611.08548

    The spinon Fermi surface U(1) spin liquid in a spin-orbit-coupled triangular lattice Mott insulator YbMgGaO4AbstractI IntroductionII Space group symmetryIII Fermionic parton constructionIV Projective symmetry group classificationV Mean-field statesVI Spectroscopic propertiesVII DiscussionVIII AcknowledgementsA The coordinate System and space group symmetryB Projective symmetry group classification1 Spatial symmetry2 Time reversal symmetry

    C Spinon band structures and mean-field Hamiltonians1 Spinon band structures2 Spinon mean-field Hamiltonians

    D The U1A00 state and the spectroscopic results1 Free spinon mean-field theory2 Variational calculation and random phase approximation

    E The U1B states1 The U1B00 state2 The U1B01 state3 The U1B10 state4 The U1B11 state

    References