1
Evaluation An Empirical Investigation The Role of Over-Parametrization in Generalization of Neural Networks Behnam Neyshabur NYU Zhiyuan Li Princeton Nathan Srebro TTI-Chicago Yann LeCun NYU Srinadh Bhojanapalli Google Empirical observation: A generalization bound that Current complexity measures with over-parametrization L with over-parametrization? J CIFAR-10 CIFAR-10 Over-parametrization helps generalization unit capacity unit impact Generalization Bound Why is this important and useful? One of the key mysteries behind success of deep learning Understanding the real cause of generalization can be used to design better models and optimization algorithms CIFAR-10 SVHN MNIST Matching Lower Bound Code available at https://github.com/bneyshabur/over-parametrization R S (F W ) R S (F W 0 )= P h j =1 j β j kXk F m ! . ü Matches both terms in the upper bound. ü First lower bound that is higher than Lipschitz of the network. ü Improves by a factor of over Bartlett et al. 2017. ü gap between linear vs non-linear networks. R S (F W ) R S (F W 0 )= P h j =1 j ku 0 j Xk 2 m ! . Distance: Euclidean distancefrominitialization. Comparing generalization measures: Behavior of different norms with over-parametrization: Comparing capacity bounds: # Reference Measure (1) Harvey et al. 2017 ˜ O(dh) (2) Bartlett and Mendelson 2002 ˜ O kUk 1,1 kVk 1,1 (3) Neyshabur et al. 2015 ˜ O ( kUk F kVk F ) (4) Bartlett et al. 2017 ˜ O kUk 2 kV - V 0 k 1,2 + kU - U 0 k 1,2 kVk 2 (5) Neyshabur et al. 2018 ˜ O kUk 2 kV - V 0 k F + p h kU - U 0 k F kVk 2 (6) ours ˜ O kU 0 k 2 kVk F + U - U 0 F kVk F + p h W = (V, U) | V 2 R ch , U 2 R hd , kv i k i , u i - u 0 i 2 β i Rademacher complexity bound: For any , w.p. > 1- δ over the training set: Generalization bound: U 0 is the initialization/ any fixed matrix. Rademacher complexity: Margin ramp loss: Properties of learned two layer ReLU networks: d: input dim c: #classes h: #hidden units m: #samples : margin R S (` γ F W ) 2 p 2c +2 γ m h X j =1 j β j kXk F + u 0 j X 2 L 0 (f ) ˆ L γ (f )+ ˜ O 0 @ p ckVk F ( U - U 0 F + U 0 2 ) q 1 m P m i=1 kx i k 2 2 γ p m + r h m 1 A .

The Role of Over-Parametrization in Generalization of ... · Evaluation An Empirical Investigation The Role of Over-Parametrization in Generalization of Neural Networks Behnam Neyshabur

  • Upload
    others

  • View
    5

  • Download
    0

Embed Size (px)

Citation preview

Page 1: The Role of Over-Parametrization in Generalization of ... · Evaluation An Empirical Investigation The Role of Over-Parametrization in Generalization of Neural Networks Behnam Neyshabur

Evaluation

AnEmpiricalInvestigation

TheRoleofOver-Parametrization inGeneralization ofNeuralNetworks

Behnam NeyshaburNYU

Zhiyuan LiPrinceton

Nathan SrebroTTI-Chicago

Yann LeCunNYU

Srinadh BhojanapalliGoogle

Empiricalobservation:

Ageneralizationboundthat

Currentcomplexitymeasures withover-parametrizationL

withover-parametrization?J

CIFAR-10 CIFAR-10

Over-parametrizationhelps generalization

unitcapacityunitimpact

GeneralizationBound

Whyisthisimportantanduseful?• Oneofthekeymysteries behindsuccessofdeeplearning• Understandingtherealcauseofgeneralizationcanbeusedto

designbettermodelsandoptimizationalgorithms

CIFAR-10 SVHN MNIST

MatchingLowerBound

Codeavailableathttps://github.com/bneyshabur/over-parametrization

RS(FW) � RS(FW0) = ⌦

Phj=1 ↵j�jkXkF

m

!.

<latexit sha1_base64="hEwNhq302TrpMUHf789N9MBh9aA=">AAACf3icdVFNa9wwEJXdr3T7tU2O6UF0KdlcjB0KbQ+B0EDorWnTzS5EWyPLY1uJJLvSOLA4vvY/9tpzf0TtzULSDR3QzOO9NzA8JZWSDsPwl+ffu//g4aONx4MnT589fzF8uXnqytoKmIhSlXaWcAdKGpigRAWzygLXiYJpcnHY69NLsE6W5hsuKphrnhuZScGxo+LhTya4+hr3/WTc96O46ce03aUshx/0f/pOZ6D77LOGnFOmIMMxZZnlomGu1nFzvh+13wvKuKoKHp+zBLAfV+wSxIxdxUdto1vKrMwL3A3i4SgMwmXRuyBagRFZ1XE8/M3SUtQaDArFnTuLwgrnDbcohYJ2wGoHFRcXPIezDhquwc2bZV4tfdMxKc1K2z2DdMne3mi4dm6hk86pORZuXevJG00aoeoUmgJ4Cra9IUyJy5TXrsHs/byRpqoRjLg+JqsVxZL2H0RTaUGgWtBBF0q0HsFdcLoXRGEQfXk7Ovi4imeDbJPXZEwi8o4ckE/kmEyIIH+8TW/be+V7/o4f+OG11fdWO1vkn/I//AU9C8Ee</latexit><latexit sha1_base64="hEwNhq302TrpMUHf789N9MBh9aA=">AAACf3icdVFNa9wwEJXdr3T7tU2O6UF0KdlcjB0KbQ+B0EDorWnTzS5EWyPLY1uJJLvSOLA4vvY/9tpzf0TtzULSDR3QzOO9NzA8JZWSDsPwl+ffu//g4aONx4MnT589fzF8uXnqytoKmIhSlXaWcAdKGpigRAWzygLXiYJpcnHY69NLsE6W5hsuKphrnhuZScGxo+LhTya4+hr3/WTc96O46ce03aUshx/0f/pOZ6D77LOGnFOmIMMxZZnlomGu1nFzvh+13wvKuKoKHp+zBLAfV+wSxIxdxUdto1vKrMwL3A3i4SgMwmXRuyBagRFZ1XE8/M3SUtQaDArFnTuLwgrnDbcohYJ2wGoHFRcXPIezDhquwc2bZV4tfdMxKc1K2z2DdMne3mi4dm6hk86pORZuXevJG00aoeoUmgJ4Cra9IUyJy5TXrsHs/byRpqoRjLg+JqsVxZL2H0RTaUGgWtBBF0q0HsFdcLoXRGEQfXk7Ovi4imeDbJPXZEwi8o4ckE/kmEyIIH+8TW/be+V7/o4f+OG11fdWO1vkn/I//AU9C8Ee</latexit><latexit sha1_base64="hEwNhq302TrpMUHf789N9MBh9aA=">AAACf3icdVFNa9wwEJXdr3T7tU2O6UF0KdlcjB0KbQ+B0EDorWnTzS5EWyPLY1uJJLvSOLA4vvY/9tpzf0TtzULSDR3QzOO9NzA8JZWSDsPwl+ffu//g4aONx4MnT589fzF8uXnqytoKmIhSlXaWcAdKGpigRAWzygLXiYJpcnHY69NLsE6W5hsuKphrnhuZScGxo+LhTya4+hr3/WTc96O46ce03aUshx/0f/pOZ6D77LOGnFOmIMMxZZnlomGu1nFzvh+13wvKuKoKHp+zBLAfV+wSxIxdxUdto1vKrMwL3A3i4SgMwmXRuyBagRFZ1XE8/M3SUtQaDArFnTuLwgrnDbcohYJ2wGoHFRcXPIezDhquwc2bZV4tfdMxKc1K2z2DdMne3mi4dm6hk86pORZuXevJG00aoeoUmgJ4Cra9IUyJy5TXrsHs/byRpqoRjLg+JqsVxZL2H0RTaUGgWtBBF0q0HsFdcLoXRGEQfXk7Ovi4imeDbJPXZEwi8o4ckE/kmEyIIH+8TW/be+V7/o4f+OG11fdWO1vkn/I//AU9C8Ee</latexit><latexit sha1_base64="hEwNhq302TrpMUHf789N9MBh9aA=">AAACf3icdVFNa9wwEJXdr3T7tU2O6UF0KdlcjB0KbQ+B0EDorWnTzS5EWyPLY1uJJLvSOLA4vvY/9tpzf0TtzULSDR3QzOO9NzA8JZWSDsPwl+ffu//g4aONx4MnT589fzF8uXnqytoKmIhSlXaWcAdKGpigRAWzygLXiYJpcnHY69NLsE6W5hsuKphrnhuZScGxo+LhTya4+hr3/WTc96O46ce03aUshx/0f/pOZ6D77LOGnFOmIMMxZZnlomGu1nFzvh+13wvKuKoKHp+zBLAfV+wSxIxdxUdto1vKrMwL3A3i4SgMwmXRuyBagRFZ1XE8/M3SUtQaDArFnTuLwgrnDbcohYJ2wGoHFRcXPIezDhquwc2bZV4tfdMxKc1K2z2DdMne3mi4dm6hk86pORZuXevJG00aoeoUmgJ4Cra9IUyJy5TXrsHs/byRpqoRjLg+JqsVxZL2H0RTaUGgWtBBF0q0HsFdcLoXRGEQfXk7Ovi4imeDbJPXZEwi8o4ckE/kmEyIIH+8TW/be+V7/o4f+OG11fdWO1vkn/I//AU9C8Ee</latexit>

W 0 =

⇢(V,U) | V 2 R1⇥h,U 2 Rh⇥d, kvjk ↵j ,

��uj � u0j

��2 �j , kU�U0k2 max

j2h�j

<latexit sha1_base64="99luRpsM/BzbEIS8JcHydmZFy6c=">AAACxHicZVFdb9MwFHXC11a+CjzyYlEhhgRVUiHBC9IE0sTjQKSdNLfBcW4ad7YTbKeiyrI/yC/gdb8EJw3a6K5k++icc6+vr5NScGOD4I/n37p95+69vf3B/QcPHz0ePnk6NUWlGUSsEIU+SagBwRVEllsBJ6UGKhMBs+Tsc6vP1qANL9R3uylhLulS8Ywzah0VD38TRsXs1UciILOkxgdkDWz6pt2j15hInuKOIVxh8m1Rh8RyCQbnzdaD/wl5L6ROUIWWdSuv41XjKv/EhIoyp/HqmlbFq7fduQicK57g1kgSsJ3tvKveGaJFQM7jybaOpL/ietVemje4dxPNl7klTTwcBeOgC3wThD0YoT6O4+ElSQtWSVCWCWrMaRiUdl5TbTkT0AxIZaCk7Iwu4dRBRd0D53U39Aa/dEyKs0K7pSzu2OsZNZXGbGTinJLa3OxqLXmlccVElUKdA01BN1eEKmz3VTvd2OzDvOaqrCwotm0mqwS2BW5/GadcA7NigwduKOHuCG6C6WQcBuPw67vR4ad+PHvoOXqBDlCI3qND9AUdowgxL/Rm3g+P+ke+8I1fba2+1+c8Q/+Ff/EXmcHdKw==</latexit><latexit sha1_base64="99luRpsM/BzbEIS8JcHydmZFy6c=">AAACxHicZVFdb9MwFHXC11a+CjzyYlEhhgRVUiHBC9IE0sTjQKSdNLfBcW4ad7YTbKeiyrI/yC/gdb8EJw3a6K5k++icc6+vr5NScGOD4I/n37p95+69vf3B/QcPHz0ePnk6NUWlGUSsEIU+SagBwRVEllsBJ6UGKhMBs+Tsc6vP1qANL9R3uylhLulS8Ywzah0VD38TRsXs1UciILOkxgdkDWz6pt2j15hInuKOIVxh8m1Rh8RyCQbnzdaD/wl5L6ROUIWWdSuv41XjKv/EhIoyp/HqmlbFq7fduQicK57g1kgSsJ3tvKveGaJFQM7jybaOpL/ietVemje4dxPNl7klTTwcBeOgC3wThD0YoT6O4+ElSQtWSVCWCWrMaRiUdl5TbTkT0AxIZaCk7Iwu4dRBRd0D53U39Aa/dEyKs0K7pSzu2OsZNZXGbGTinJLa3OxqLXmlccVElUKdA01BN1eEKmz3VTvd2OzDvOaqrCwotm0mqwS2BW5/GadcA7NigwduKOHuCG6C6WQcBuPw67vR4ad+PHvoOXqBDlCI3qND9AUdowgxL/Rm3g+P+ke+8I1fba2+1+c8Q/+Ff/EXmcHdKw==</latexit><latexit sha1_base64="99luRpsM/BzbEIS8JcHydmZFy6c=">AAACxHicZVFdb9MwFHXC11a+CjzyYlEhhgRVUiHBC9IE0sTjQKSdNLfBcW4ad7YTbKeiyrI/yC/gdb8EJw3a6K5k++icc6+vr5NScGOD4I/n37p95+69vf3B/QcPHz0ePnk6NUWlGUSsEIU+SagBwRVEllsBJ6UGKhMBs+Tsc6vP1qANL9R3uylhLulS8Ywzah0VD38TRsXs1UciILOkxgdkDWz6pt2j15hInuKOIVxh8m1Rh8RyCQbnzdaD/wl5L6ROUIWWdSuv41XjKv/EhIoyp/HqmlbFq7fduQicK57g1kgSsJ3tvKveGaJFQM7jybaOpL/ietVemje4dxPNl7klTTwcBeOgC3wThD0YoT6O4+ElSQtWSVCWCWrMaRiUdl5TbTkT0AxIZaCk7Iwu4dRBRd0D53U39Aa/dEyKs0K7pSzu2OsZNZXGbGTinJLa3OxqLXmlccVElUKdA01BN1eEKmz3VTvd2OzDvOaqrCwotm0mqwS2BW5/GadcA7NigwduKOHuCG6C6WQcBuPw67vR4ad+PHvoOXqBDlCI3qND9AUdowgxL/Rm3g+P+ke+8I1fba2+1+c8Q/+Ff/EXmcHdKw==</latexit><latexit sha1_base64="99luRpsM/BzbEIS8JcHydmZFy6c=">AAACxHicZVFdb9MwFHXC11a+CjzyYlEhhgRVUiHBC9IE0sTjQKSdNLfBcW4ad7YTbKeiyrI/yC/gdb8EJw3a6K5k++icc6+vr5NScGOD4I/n37p95+69vf3B/QcPHz0ePnk6NUWlGUSsEIU+SagBwRVEllsBJ6UGKhMBs+Tsc6vP1qANL9R3uylhLulS8Ywzah0VD38TRsXs1UciILOkxgdkDWz6pt2j15hInuKOIVxh8m1Rh8RyCQbnzdaD/wl5L6ROUIWWdSuv41XjKv/EhIoyp/HqmlbFq7fduQicK57g1kgSsJ3tvKveGaJFQM7jybaOpL/ietVemje4dxPNl7klTTwcBeOgC3wThD0YoT6O4+ElSQtWSVCWCWrMaRiUdl5TbTkT0AxIZaCk7Iwu4dRBRd0D53U39Aa/dEyKs0K7pSzu2OsZNZXGbGTinJLa3OxqLXmlccVElUKdA01BN1eEKmz3VTvd2OzDvOaqrCwotm0mqwS2BW5/GadcA7NigwduKOHuCG6C6WQcBuPw67vR4ad+PHvoOXqBDlCI3qND9AUdowgxL/Rm3g+P+ke+8I1fba2+1+c8Q/+Ff/EXmcHdKw==</latexit>

Consider

ü Matchesbothterms intheupperbound.

ü First lowerboundthatishigherthanLipschitzofthenetwork.

ü Improvesbyafactorof ℎ� overBartlettetal.2017.

ü ℎ� gapbetweenlinearvsnon-linearnetworks.

S = {xi}mi=1 ⇢ Rd<latexit sha1_base64="9XlcTsKZ+5Zans3yqSIscOgADW4=">AAACH3icZZDLSsNAFIYnXmu9RV26GSyKq5KIoJtC0Y3LeukFmjRMJift0MkkZCbFEvIGvoSv4Fb37sRttz6J6QWq7Q8DP/9/Dpz53IgzqQxjpK2srq1vbBa2its7u3v7+sFhQ4ZJTKFOQx7GLZdI4ExAXTHFoRXFQAKXQ9Pt34775gBiyULxpIYR2AHpCuYzSlQeOfqZRQl/rFipNQD67DArc1JWMbNOgC2ZuBIUth46nqOXjLIxEV425syU0Ew1R/+xvJAmAQhFOZGybRqRslMSK0Y5ZEUrkRAR2iddaOdWkACknU7+k+HTPPGwH8b5EwpP0r8bKQmkHAZuPhkQ1ZOL3Ticd0xQnniQ9oB4EGfzQIRqQmHhGuVf2ykTUaJA0OkxfsKxCvEYIPZYDFTxIS7mUMxFBMumcVE2jbJ5f1mq3szwFNAxOkHnyERXqIruUA3VEUUv6A29ow/tVfvUvrTv6eiKNts5Qv+kjX4B8MajNQ==</latexit><latexit sha1_base64="9XlcTsKZ+5Zans3yqSIscOgADW4=">AAACH3icZZDLSsNAFIYnXmu9RV26GSyKq5KIoJtC0Y3LeukFmjRMJift0MkkZCbFEvIGvoSv4Fb37sRttz6J6QWq7Q8DP/9/Dpz53IgzqQxjpK2srq1vbBa2its7u3v7+sFhQ4ZJTKFOQx7GLZdI4ExAXTHFoRXFQAKXQ9Pt34775gBiyULxpIYR2AHpCuYzSlQeOfqZRQl/rFipNQD67DArc1JWMbNOgC2ZuBIUth46nqOXjLIxEV425syU0Ew1R/+xvJAmAQhFOZGybRqRslMSK0Y5ZEUrkRAR2iddaOdWkACknU7+k+HTPPGwH8b5EwpP0r8bKQmkHAZuPhkQ1ZOL3Ticd0xQnniQ9oB4EGfzQIRqQmHhGuVf2ykTUaJA0OkxfsKxCvEYIPZYDFTxIS7mUMxFBMumcVE2jbJ5f1mq3szwFNAxOkHnyERXqIruUA3VEUUv6A29ow/tVfvUvrTv6eiKNts5Qv+kjX4B8MajNQ==</latexit><latexit sha1_base64="9XlcTsKZ+5Zans3yqSIscOgADW4=">AAACH3icZZDLSsNAFIYnXmu9RV26GSyKq5KIoJtC0Y3LeukFmjRMJift0MkkZCbFEvIGvoSv4Fb37sRttz6J6QWq7Q8DP/9/Dpz53IgzqQxjpK2srq1vbBa2its7u3v7+sFhQ4ZJTKFOQx7GLZdI4ExAXTHFoRXFQAKXQ9Pt34775gBiyULxpIYR2AHpCuYzSlQeOfqZRQl/rFipNQD67DArc1JWMbNOgC2ZuBIUth46nqOXjLIxEV425syU0Ew1R/+xvJAmAQhFOZGybRqRslMSK0Y5ZEUrkRAR2iddaOdWkACknU7+k+HTPPGwH8b5EwpP0r8bKQmkHAZuPhkQ1ZOL3Ticd0xQnniQ9oB4EGfzQIRqQmHhGuVf2ykTUaJA0OkxfsKxCvEYIPZYDFTxIS7mUMxFBMumcVE2jbJ5f1mq3szwFNAxOkHnyERXqIruUA3VEUUv6A29ow/tVfvUvrTv6eiKNts5Qv+kjX4B8MajNQ==</latexit><latexit sha1_base64="Qlfzn0O7wPopyBABbLiZvrygJh0=">AAAB6XicZZDLSsNAGIX/1FuNVevazWARXJXEjS4FNy4r2Au0oUwmf9qhcwkzE7GEvoBb9+7EZ3Lrk5imBbU9MHD4zgycOXEmuHVB8OXVdnb39g/qh/5Rwz8+OW02elbnhmGXaaHNIKYWBVfYddwJHGQGqYwF9uPZ/TLvP6OxXKsnN88wknSieMoZdSXqjJutoB1UItsmXJsWrDVufo8SzXKJyjFBrR2GQeaighrHmcCFP8otZpTN6ASHpVVUoo2KquaCXJYkIak25VGOVPTvi4JKa+cyLm9K6qZ2M1vC34wrJvIEiynSBM3iFyjtqs9ttHHpbVRwleUOFVuVSXNBnCbLXUjCDTIn5sQvNwk3F9g2vet2GLTDxwDqcA4XcAUh3MAdPEAHusAggVd48168d+9jtV3NW494Bv/kff4AN96Q9Q==</latexit><latexit sha1_base64="6KJl0mTuqNqg3hw316lPJZf8jzw=">AAACFHicZZDNSsNAFIVv6l+tVatbN4NFcVUSN7opCG5c1p/+QNOGyeS2HTqZhMykWELewJfwFdzq3p24deuTmKYFtT0wcDjnXrjzuaHgSpvml1FYW9/Y3Cpul3bKu3v7lYNySwVxxLDJAhFEHZcqFFxiU3MtsBNGSH1XYNsdX8/69gQjxQP5oKch9nw6lHzAGdVZ5FRObUbFfd1O7AmyR4fbqZPwupX2fWKr2FWoiX3X95xK1ayZuciqsRamCgs1nMq37QUs9lFqJqhSXcsMdS+hkeZMYFqyY4UhZWM6xG5mJfVR9ZL8Pyk5yRKPDIIoe1KTPP27kVBfqanvZpM+1SO13M3C345LJmIPkxFSD6P0N5CBziksXaMHl72EyzDWKNn8mEEsiA7IDCDxeIRMiykpZVCsZQSrpnVes8yadWtCEY7gGM7Aggu4ghtoQBMYPMELvMKb8Wy8Gx9zfAVjwfEQ/sn4/AHvQKGr</latexit><latexit sha1_base64="6KJl0mTuqNqg3hw316lPJZf8jzw=">AAACFHicZZDNSsNAFIVv6l+tVatbN4NFcVUSN7opCG5c1p/+QNOGyeS2HTqZhMykWELewJfwFdzq3p24deuTmKYFtT0wcDjnXrjzuaHgSpvml1FYW9/Y3Cpul3bKu3v7lYNySwVxxLDJAhFEHZcqFFxiU3MtsBNGSH1XYNsdX8/69gQjxQP5oKch9nw6lHzAGdVZ5FRObUbFfd1O7AmyR4fbqZPwupX2fWKr2FWoiX3X95xK1ayZuciqsRamCgs1nMq37QUs9lFqJqhSXcsMdS+hkeZMYFqyY4UhZWM6xG5mJfVR9ZL8Pyk5yRKPDIIoe1KTPP27kVBfqanvZpM+1SO13M3C345LJmIPkxFSD6P0N5CBziksXaMHl72EyzDWKNn8mEEsiA7IDCDxeIRMiykpZVCsZQSrpnVes8yadWtCEY7gGM7Aggu4ghtoQBMYPMELvMKb8Wy8Gx9zfAVjwfEQ/sn4/AHvQKGr</latexit><latexit sha1_base64="TdaUNydNFNkLCcFnsREA8HNAHUo=">AAACH3icZZDLSsNAFIYn9VbrrerSzWBRXJXEjW4KRTcu66UXaNIwmZy0QyeTkJkUS8gb+BK+glvduxO33fokpmmh2v4w8PP/58CZzwk5k0rXJ1phbX1jc6u4XdrZ3ds/KB8etWQQRxSaNOBB1HGIBM4ENBVTHDphBMR3OLSd4e20b48gkiwQT2ocguWTvmAeo0RlkV0+NynhjzUzMUdAn21mpnbCakba87EpY0eCwuZDz7XLFb2q58KrxpibCpqrYZd/TDegsQ9CUU6k7Bp6qKyERIpRDmnJjCWEhA5JH7qZFcQHaSX5f1J8liUu9oIoe0LhPP27kRBfyrHvZJM+UQO53E3DRccE5bELyQCIC1G6CESgcgpL1yjv2kqYCGMFgs6O8WKOVYCnALHLIqCKj3Epg2IsI1g1rcuqoVeNe71Sv5njKaITdIoukIGuUB3doQZqIope0Bt6Rx/aq/apfWnfs9GCNt85Rv+kTX4B74ajMQ==</latexit><latexit sha1_base64="9XlcTsKZ+5Zans3yqSIscOgADW4=">AAACH3icZZDLSsNAFIYnXmu9RV26GSyKq5KIoJtC0Y3LeukFmjRMJift0MkkZCbFEvIGvoSv4Fb37sRttz6J6QWq7Q8DP/9/Dpz53IgzqQxjpK2srq1vbBa2its7u3v7+sFhQ4ZJTKFOQx7GLZdI4ExAXTHFoRXFQAKXQ9Pt34775gBiyULxpIYR2AHpCuYzSlQeOfqZRQl/rFipNQD67DArc1JWMbNOgC2ZuBIUth46nqOXjLIxEV425syU0Ew1R/+xvJAmAQhFOZGybRqRslMSK0Y5ZEUrkRAR2iddaOdWkACknU7+k+HTPPGwH8b5EwpP0r8bKQmkHAZuPhkQ1ZOL3Ticd0xQnniQ9oB4EGfzQIRqQmHhGuVf2ykTUaJA0OkxfsKxCvEYIPZYDFTxIS7mUMxFBMumcVE2jbJ5f1mq3szwFNAxOkHnyERXqIruUA3VEUUv6A29ow/tVfvUvrTv6eiKNts5Qv+kjX4B8MajNQ==</latexit><latexit sha1_base64="9XlcTsKZ+5Zans3yqSIscOgADW4=">AAACH3icZZDLSsNAFIYnXmu9RV26GSyKq5KIoJtC0Y3LeukFmjRMJift0MkkZCbFEvIGvoSv4Fb37sRttz6J6QWq7Q8DP/9/Dpz53IgzqQxjpK2srq1vbBa2its7u3v7+sFhQ4ZJTKFOQx7GLZdI4ExAXTHFoRXFQAKXQ9Pt34775gBiyULxpIYR2AHpCuYzSlQeOfqZRQl/rFipNQD67DArc1JWMbNOgC2ZuBIUth46nqOXjLIxEV425syU0Ew1R/+xvJAmAQhFOZGybRqRslMSK0Y5ZEUrkRAR2iddaOdWkACknU7+k+HTPPGwH8b5EwpP0r8bKQmkHAZuPhkQ1ZOL3Ticd0xQnniQ9oB4EGfzQIRqQmHhGuVf2ykTUaJA0OkxfsKxCvEYIPZYDFTxIS7mUMxFBMumcVE2jbJ5f1mq3szwFNAxOkHnyERXqIruUA3VEUUv6A29ow/tVfvUvrTv6eiKNts5Qv+kjX4B8MajNQ==</latexit><latexit sha1_base64="9XlcTsKZ+5Zans3yqSIscOgADW4=">AAACH3icZZDLSsNAFIYnXmu9RV26GSyKq5KIoJtC0Y3LeukFmjRMJift0MkkZCbFEvIGvoSv4Fb37sRttz6J6QWq7Q8DP/9/Dpz53IgzqQxjpK2srq1vbBa2its7u3v7+sFhQ4ZJTKFOQx7GLZdI4ExAXTHFoRXFQAKXQ9Pt34775gBiyULxpIYR2AHpCuYzSlQeOfqZRQl/rFipNQD67DArc1JWMbNOgC2ZuBIUth46nqOXjLIxEV425syU0Ew1R/+xvJAmAQhFOZGybRqRslMSK0Y5ZEUrkRAR2iddaOdWkACknU7+k+HTPPGwH8b5EwpP0r8bKQmkHAZuPhkQ1ZOL3Ticd0xQnniQ9oB4EGfzQIRqQmHhGuVf2ykTUaJA0OkxfsKxCvEYIPZYDFTxIS7mUMxFBMumcVE2jbJ5f1mq3szwFNAxOkHnyERXqIruUA3VEUUv6A29ow/tVfvUvrTv6eiKNts5Qv+kjX4B8MajNQ==</latexit><latexit sha1_base64="9XlcTsKZ+5Zans3yqSIscOgADW4=">AAACH3icZZDLSsNAFIYnXmu9RV26GSyKq5KIoJtC0Y3LeukFmjRMJift0MkkZCbFEvIGvoSv4Fb37sRttz6J6QWq7Q8DP/9/Dpz53IgzqQxjpK2srq1vbBa2its7u3v7+sFhQ4ZJTKFOQx7GLZdI4ExAXTHFoRXFQAKXQ9Pt34775gBiyULxpIYR2AHpCuYzSlQeOfqZRQl/rFipNQD67DArc1JWMbNOgC2ZuBIUth46nqOXjLIxEV425syU0Ew1R/+xvJAmAQhFOZGybRqRslMSK0Y5ZEUrkRAR2iddaOdWkACknU7+k+HTPPGwH8b5EwpP0r8bKQmkHAZuPhkQ1ZOL3Ticd0xQnniQ9oB4EGfzQIRqQmHhGuVf2ykTUaJA0OkxfsKxCvEYIPZYDFTxIS7mUMxFBMumcVE2jbJ5f1mq3szwFNAxOkHnyERXqIruUA3VEUUv6A29ow/tVfvUvrTv6eiKNts5Qv+kjX4B8MajNQ==</latexit><latexit sha1_base64="9XlcTsKZ+5Zans3yqSIscOgADW4=">AAACH3icZZDLSsNAFIYnXmu9RV26GSyKq5KIoJtC0Y3LeukFmjRMJift0MkkZCbFEvIGvoSv4Fb37sRttz6J6QWq7Q8DP/9/Dpz53IgzqQxjpK2srq1vbBa2its7u3v7+sFhQ4ZJTKFOQx7GLZdI4ExAXTHFoRXFQAKXQ9Pt34775gBiyULxpIYR2AHpCuYzSlQeOfqZRQl/rFipNQD67DArc1JWMbNOgC2ZuBIUth46nqOXjLIxEV425syU0Ew1R/+xvJAmAQhFOZGybRqRslMSK0Y5ZEUrkRAR2iddaOdWkACknU7+k+HTPPGwH8b5EwpP0r8bKQmkHAZuPhkQ1ZOL3Ticd0xQnniQ9oB4EGfzQIRqQmHhGuVf2ykTUaJA0OkxfsKxCvEYIPZYDFTxIS7mUMxFBMumcVE2jbJ5f1mq3szwFNAxOkHnyERXqIruUA3VEUUv6A29ow/tVfvUvrTv6eiKNts5Qv+kjX4B8MajNQ==</latexit><latexit sha1_base64="9XlcTsKZ+5Zans3yqSIscOgADW4=">AAACH3icZZDLSsNAFIYnXmu9RV26GSyKq5KIoJtC0Y3LeukFmjRMJift0MkkZCbFEvIGvoSv4Fb37sRttz6J6QWq7Q8DP/9/Dpz53IgzqQxjpK2srq1vbBa2its7u3v7+sFhQ4ZJTKFOQx7GLZdI4ExAXTHFoRXFQAKXQ9Pt34775gBiyULxpIYR2AHpCuYzSlQeOfqZRQl/rFipNQD67DArc1JWMbNOgC2ZuBIUth46nqOXjLIxEV425syU0Ew1R/+xvJAmAQhFOZGybRqRslMSK0Y5ZEUrkRAR2iddaOdWkACknU7+k+HTPPGwH8b5EwpP0r8bKQmkHAZuPhkQ1ZOL3Ticd0xQnniQ9oB4EGfzQIRqQmHhGuVf2ykTUaJA0OkxfsKxCvEYIPZYDFTxIS7mUMxFBMumcVE2jbJ5f1mq3szwFNAxOkHnyERXqIruUA3VEUUv6A29ow/tVfvUvrTv6eiKNts5Qv+kjX4B8MajNQ==</latexit>

Forany thereexists s.t.d = h m, {↵j ,�j}hj=1 ⇢ R+,U0 2 Rh⇥d<latexit sha1_base64="uBwciCXErTePMUW+8zV6xhOfqXg=">AAACR3icZVBNS8NAEN3U7/pV9ehlsQiCpSQi6EUQvXgSFauFbg2bzbRZ3WxidiKUkH/ln/An6FG9exOPprWgtg+WffveDDP7vFhJg7b9bJUmJqemZ2bnyvMLi0vLlZXVKxOliYCGiFSUND1uQEkNDZSooBknwENPwbV3d9z3rx8gMTLSl9iLoR3yrpYdKTgWkls59Q8CpuCehjXKMsZVHHD3tsY8wOJmuZvdHjj5TUCZST0DSNnFzXaNPYBouDaTuv/OAoYyBEP93K1U7bo9AB0nzpBUyRBnbuWV+ZFIQ9AoFDem5dgxtjOeoBQK8jJLDcRc3PEutAqqeTGnnQ3+ndPNQvFpJ0qKo5EO1L8dGQ+N6YVeURlyDMyo1xd/PamFSn3IAuA+JPmvoCMcpDWyDXb225nUcYqgxc8ynVRRjGg/aOrLBASqHi0XoTijEYyTq526Y9ed893q4dEwnlmyTjbIFnHIHjkkJ+SMNIggj+SFvJF368n6sD6tr5/SkjXsWSP/ULK+AcHDsZY=</latexit><latexit sha1_base64="uBwciCXErTePMUW+8zV6xhOfqXg=">AAACR3icZVBNS8NAEN3U7/pV9ehlsQiCpSQi6EUQvXgSFauFbg2bzbRZ3WxidiKUkH/ln/An6FG9exOPprWgtg+WffveDDP7vFhJg7b9bJUmJqemZ2bnyvMLi0vLlZXVKxOliYCGiFSUND1uQEkNDZSooBknwENPwbV3d9z3rx8gMTLSl9iLoR3yrpYdKTgWkls59Q8CpuCehjXKMsZVHHD3tsY8wOJmuZvdHjj5TUCZST0DSNnFzXaNPYBouDaTuv/OAoYyBEP93K1U7bo9AB0nzpBUyRBnbuWV+ZFIQ9AoFDem5dgxtjOeoBQK8jJLDcRc3PEutAqqeTGnnQ3+ndPNQvFpJ0qKo5EO1L8dGQ+N6YVeURlyDMyo1xd/PamFSn3IAuA+JPmvoCMcpDWyDXb225nUcYqgxc8ynVRRjGg/aOrLBASqHi0XoTijEYyTq526Y9ed893q4dEwnlmyTjbIFnHIHjkkJ+SMNIggj+SFvJF368n6sD6tr5/SkjXsWSP/ULK+AcHDsZY=</latexit><latexit sha1_base64="uBwciCXErTePMUW+8zV6xhOfqXg=">AAACR3icZVBNS8NAEN3U7/pV9ehlsQiCpSQi6EUQvXgSFauFbg2bzbRZ3WxidiKUkH/ln/An6FG9exOPprWgtg+WffveDDP7vFhJg7b9bJUmJqemZ2bnyvMLi0vLlZXVKxOliYCGiFSUND1uQEkNDZSooBknwENPwbV3d9z3rx8gMTLSl9iLoR3yrpYdKTgWkls59Q8CpuCehjXKMsZVHHD3tsY8wOJmuZvdHjj5TUCZST0DSNnFzXaNPYBouDaTuv/OAoYyBEP93K1U7bo9AB0nzpBUyRBnbuWV+ZFIQ9AoFDem5dgxtjOeoBQK8jJLDcRc3PEutAqqeTGnnQ3+ndPNQvFpJ0qKo5EO1L8dGQ+N6YVeURlyDMyo1xd/PamFSn3IAuA+JPmvoCMcpDWyDXb225nUcYqgxc8ynVRRjGg/aOrLBASqHi0XoTijEYyTq526Y9ed893q4dEwnlmyTjbIFnHIHjkkJ+SMNIggj+SFvJF368n6sD6tr5/SkjXsWSP/ULK+AcHDsZY=</latexit><latexit sha1_base64="uBwciCXErTePMUW+8zV6xhOfqXg=">AAACR3icZVBNS8NAEN3U7/pV9ehlsQiCpSQi6EUQvXgSFauFbg2bzbRZ3WxidiKUkH/ln/An6FG9exOPprWgtg+WffveDDP7vFhJg7b9bJUmJqemZ2bnyvMLi0vLlZXVKxOliYCGiFSUND1uQEkNDZSooBknwENPwbV3d9z3rx8gMTLSl9iLoR3yrpYdKTgWkls59Q8CpuCehjXKMsZVHHD3tsY8wOJmuZvdHjj5TUCZST0DSNnFzXaNPYBouDaTuv/OAoYyBEP93K1U7bo9AB0nzpBUyRBnbuWV+ZFIQ9AoFDem5dgxtjOeoBQK8jJLDcRc3PEutAqqeTGnnQ3+ndPNQvFpJ0qKo5EO1L8dGQ+N6YVeURlyDMyo1xd/PamFSn3IAuA+JPmvoCMcpDWyDXb225nUcYqgxc8ynVRRjGg/aOrLBASqHi0XoTijEYyTq526Y9ed893q4dEwnlmyTjbIFnHIHjkkJ+SMNIggj+SFvJF368n6sD6tr5/SkjXsWSP/ULK+AcHDsZY=</latexit>

RS(FW) � RS(FW0) = ⌦

Phj=1 ↵jku0

jXk2m

!.

<latexit sha1_base64="bUITX933HvHBw0dr60+n4biLt5E=">AAACgXicdVFda9swFJXdfXTZV7rtbWyIhbHkJdhhsMEolA3G3tZ9pAlUqZDla1upJLuSXAiq3/cX97ofsN8wOw10S9kF3Xs451y4HCWVFNZF0c8g3Llx89bt3Tu9u/fuP3jY33t0ZMvacJjyUpZmnjALUmiYOuEkzCsDTCUSZsnph06fnYOxotTf3aqChWK5FpngzLUU7f8gnMmvtOvfhl3/SH03Zs0IkxzO8P/0V60B75PPCnKGiYTMDTHJDOOe2FpRv9yPm5MCEyargtEluSDnwGu6PIk6MCcXdNJ41WBiRF640Zj2B9E4Whe+DuINGKBNHdL+L5KWvFagHZfM2uM4qtzCM+MEl9D0SG2hYvyU5XDcQs0U2IVfJ9bgly2T4qw07dMOr9m/NzxT1q5U0joVc4Xd1jryShOayzoFXwBLwTRXhC7dOueta1z2duGFrmoHml8ek9USuxJ3X4RTYYA7ucK9NpR4O4Lr4GgyjqNx/OX14OD9Jp5d9BS9QEMUozfoAH1Ch2iKOPodPAmeBc/DnXAURuHk0hoGm53H6J8K3/0BzN/Bww==</latexit><latexit sha1_base64="bUITX933HvHBw0dr60+n4biLt5E=">AAACgXicdVFda9swFJXdfXTZV7rtbWyIhbHkJdhhsMEolA3G3tZ9pAlUqZDla1upJLuSXAiq3/cX97ofsN8wOw10S9kF3Xs451y4HCWVFNZF0c8g3Llx89bt3Tu9u/fuP3jY33t0ZMvacJjyUpZmnjALUmiYOuEkzCsDTCUSZsnph06fnYOxotTf3aqChWK5FpngzLUU7f8gnMmvtOvfhl3/SH03Zs0IkxzO8P/0V60B75PPCnKGiYTMDTHJDOOe2FpRv9yPm5MCEyargtEluSDnwGu6PIk6MCcXdNJ41WBiRF640Zj2B9E4Whe+DuINGKBNHdL+L5KWvFagHZfM2uM4qtzCM+MEl9D0SG2hYvyU5XDcQs0U2IVfJ9bgly2T4qw07dMOr9m/NzxT1q5U0joVc4Xd1jryShOayzoFXwBLwTRXhC7dOueta1z2duGFrmoHml8ek9USuxJ3X4RTYYA7ucK9NpR4O4Lr4GgyjqNx/OX14OD9Jp5d9BS9QEMUozfoAH1Ch2iKOPodPAmeBc/DnXAURuHk0hoGm53H6J8K3/0BzN/Bww==</latexit><latexit sha1_base64="bUITX933HvHBw0dr60+n4biLt5E=">AAACgXicdVFda9swFJXdfXTZV7rtbWyIhbHkJdhhsMEolA3G3tZ9pAlUqZDla1upJLuSXAiq3/cX97ofsN8wOw10S9kF3Xs451y4HCWVFNZF0c8g3Llx89bt3Tu9u/fuP3jY33t0ZMvacJjyUpZmnjALUmiYOuEkzCsDTCUSZsnph06fnYOxotTf3aqChWK5FpngzLUU7f8gnMmvtOvfhl3/SH03Zs0IkxzO8P/0V60B75PPCnKGiYTMDTHJDOOe2FpRv9yPm5MCEyargtEluSDnwGu6PIk6MCcXdNJ41WBiRF640Zj2B9E4Whe+DuINGKBNHdL+L5KWvFagHZfM2uM4qtzCM+MEl9D0SG2hYvyU5XDcQs0U2IVfJ9bgly2T4qw07dMOr9m/NzxT1q5U0joVc4Xd1jryShOayzoFXwBLwTRXhC7dOueta1z2duGFrmoHml8ek9USuxJ3X4RTYYA7ucK9NpR4O4Lr4GgyjqNx/OX14OD9Jp5d9BS9QEMUozfoAH1Ch2iKOPodPAmeBc/DnXAURuHk0hoGm53H6J8K3/0BzN/Bww==</latexit><latexit sha1_base64="bUITX933HvHBw0dr60+n4biLt5E=">AAACgXicdVFda9swFJXdfXTZV7rtbWyIhbHkJdhhsMEolA3G3tZ9pAlUqZDla1upJLuSXAiq3/cX97ofsN8wOw10S9kF3Xs451y4HCWVFNZF0c8g3Llx89bt3Tu9u/fuP3jY33t0ZMvacJjyUpZmnjALUmiYOuEkzCsDTCUSZsnph06fnYOxotTf3aqChWK5FpngzLUU7f8gnMmvtOvfhl3/SH03Zs0IkxzO8P/0V60B75PPCnKGiYTMDTHJDOOe2FpRv9yPm5MCEyargtEluSDnwGu6PIk6MCcXdNJ41WBiRF640Zj2B9E4Whe+DuINGKBNHdL+L5KWvFagHZfM2uM4qtzCM+MEl9D0SG2hYvyU5XDcQs0U2IVfJ9bgly2T4qw07dMOr9m/NzxT1q5U0joVc4Xd1jryShOayzoFXwBLwTRXhC7dOueta1z2duGFrmoHml8ek9USuxJ3X4RTYYA7ucK9NpR4O4Lr4GgyjqNx/OX14OD9Jp5d9BS9QEMUozfoAH1Ch2iKOPodPAmeBc/DnXAURuHk0hoGm53H6J8K3/0BzN/Bww==</latexit>

s.t.S = {xi}mi=1 ⇢ Rd<latexit sha1_base64="9XlcTsKZ+5Zans3yqSIscOgADW4=">AAACH3icZZDLSsNAFIYnXmu9RV26GSyKq5KIoJtC0Y3LeukFmjRMJift0MkkZCbFEvIGvoSv4Fb37sRttz6J6QWq7Q8DP/9/Dpz53IgzqQxjpK2srq1vbBa2its7u3v7+sFhQ4ZJTKFOQx7GLZdI4ExAXTHFoRXFQAKXQ9Pt34775gBiyULxpIYR2AHpCuYzSlQeOfqZRQl/rFipNQD67DArc1JWMbNOgC2ZuBIUth46nqOXjLIxEV425syU0Ew1R/+xvJAmAQhFOZGybRqRslMSK0Y5ZEUrkRAR2iddaOdWkACknU7+k+HTPPGwH8b5EwpP0r8bKQmkHAZuPhkQ1ZOL3Ticd0xQnniQ9oB4EGfzQIRqQmHhGuVf2ykTUaJA0OkxfsKxCvEYIPZYDFTxIS7mUMxFBMumcVE2jbJ5f1mq3szwFNAxOkHnyERXqIruUA3VEUUv6A29ow/tVfvUvrTv6eiKNts5Qv+kjX4B8MajNQ==</latexit><latexit sha1_base64="9XlcTsKZ+5Zans3yqSIscOgADW4=">AAACH3icZZDLSsNAFIYnXmu9RV26GSyKq5KIoJtC0Y3LeukFmjRMJift0MkkZCbFEvIGvoSv4Fb37sRttz6J6QWq7Q8DP/9/Dpz53IgzqQxjpK2srq1vbBa2its7u3v7+sFhQ4ZJTKFOQx7GLZdI4ExAXTHFoRXFQAKXQ9Pt34775gBiyULxpIYR2AHpCuYzSlQeOfqZRQl/rFipNQD67DArc1JWMbNOgC2ZuBIUth46nqOXjLIxEV425syU0Ew1R/+xvJAmAQhFOZGybRqRslMSK0Y5ZEUrkRAR2iddaOdWkACknU7+k+HTPPGwH8b5EwpP0r8bKQmkHAZuPhkQ1ZOL3Ticd0xQnniQ9oB4EGfzQIRqQmHhGuVf2ykTUaJA0OkxfsKxCvEYIPZYDFTxIS7mUMxFBMumcVE2jbJ5f1mq3szwFNAxOkHnyERXqIruUA3VEUUv6A29ow/tVfvUvrTv6eiKNts5Qv+kjX4B8MajNQ==</latexit><latexit sha1_base64="9XlcTsKZ+5Zans3yqSIscOgADW4=">AAACH3icZZDLSsNAFIYnXmu9RV26GSyKq5KIoJtC0Y3LeukFmjRMJift0MkkZCbFEvIGvoSv4Fb37sRttz6J6QWq7Q8DP/9/Dpz53IgzqQxjpK2srq1vbBa2its7u3v7+sFhQ4ZJTKFOQx7GLZdI4ExAXTHFoRXFQAKXQ9Pt34775gBiyULxpIYR2AHpCuYzSlQeOfqZRQl/rFipNQD67DArc1JWMbNOgC2ZuBIUth46nqOXjLIxEV425syU0Ew1R/+xvJAmAQhFOZGybRqRslMSK0Y5ZEUrkRAR2iddaOdWkACknU7+k+HTPPGwH8b5EwpP0r8bKQmkHAZuPhkQ1ZOL3Ticd0xQnniQ9oB4EGfzQIRqQmHhGuVf2ykTUaJA0OkxfsKxCvEYIPZYDFTxIS7mUMxFBMumcVE2jbJ5f1mq3szwFNAxOkHnyERXqIruUA3VEUUv6A29ow/tVfvUvrTv6eiKNts5Qv+kjX4B8MajNQ==</latexit><latexit sha1_base64="Qlfzn0O7wPopyBABbLiZvrygJh0=">AAAB6XicZZDLSsNAGIX/1FuNVevazWARXJXEjS4FNy4r2Au0oUwmf9qhcwkzE7GEvoBb9+7EZ3Lrk5imBbU9MHD4zgycOXEmuHVB8OXVdnb39g/qh/5Rwz8+OW02elbnhmGXaaHNIKYWBVfYddwJHGQGqYwF9uPZ/TLvP6OxXKsnN88wknSieMoZdSXqjJutoB1UItsmXJsWrDVufo8SzXKJyjFBrR2GQeaighrHmcCFP8otZpTN6ASHpVVUoo2KquaCXJYkIak25VGOVPTvi4JKa+cyLm9K6qZ2M1vC34wrJvIEiynSBM3iFyjtqs9ttHHpbVRwleUOFVuVSXNBnCbLXUjCDTIn5sQvNwk3F9g2vet2GLTDxwDqcA4XcAUh3MAdPEAHusAggVd48168d+9jtV3NW494Bv/kff4AN96Q9Q==</latexit><latexit sha1_base64="6KJl0mTuqNqg3hw316lPJZf8jzw=">AAACFHicZZDNSsNAFIVv6l+tVatbN4NFcVUSN7opCG5c1p/+QNOGyeS2HTqZhMykWELewJfwFdzq3p24deuTmKYFtT0wcDjnXrjzuaHgSpvml1FYW9/Y3Cpul3bKu3v7lYNySwVxxLDJAhFEHZcqFFxiU3MtsBNGSH1XYNsdX8/69gQjxQP5oKch9nw6lHzAGdVZ5FRObUbFfd1O7AmyR4fbqZPwupX2fWKr2FWoiX3X95xK1ayZuciqsRamCgs1nMq37QUs9lFqJqhSXcsMdS+hkeZMYFqyY4UhZWM6xG5mJfVR9ZL8Pyk5yRKPDIIoe1KTPP27kVBfqanvZpM+1SO13M3C345LJmIPkxFSD6P0N5CBziksXaMHl72EyzDWKNn8mEEsiA7IDCDxeIRMiykpZVCsZQSrpnVes8yadWtCEY7gGM7Aggu4ghtoQBMYPMELvMKb8Wy8Gx9zfAVjwfEQ/sn4/AHvQKGr</latexit><latexit sha1_base64="6KJl0mTuqNqg3hw316lPJZf8jzw=">AAACFHicZZDNSsNAFIVv6l+tVatbN4NFcVUSN7opCG5c1p/+QNOGyeS2HTqZhMykWELewJfwFdzq3p24deuTmKYFtT0wcDjnXrjzuaHgSpvml1FYW9/Y3Cpul3bKu3v7lYNySwVxxLDJAhFEHZcqFFxiU3MtsBNGSH1XYNsdX8/69gQjxQP5oKch9nw6lHzAGdVZ5FRObUbFfd1O7AmyR4fbqZPwupX2fWKr2FWoiX3X95xK1ayZuciqsRamCgs1nMq37QUs9lFqJqhSXcsMdS+hkeZMYFqyY4UhZWM6xG5mJfVR9ZL8Pyk5yRKPDIIoe1KTPP27kVBfqanvZpM+1SO13M3C345LJmIPkxFSD6P0N5CBziksXaMHl72EyzDWKNn8mEEsiA7IDCDxeIRMiykpZVCsZQSrpnVes8yadWtCEY7gGM7Aggu4ghtoQBMYPMELvMKb8Wy8Gx9zfAVjwfEQ/sn4/AHvQKGr</latexit><latexit sha1_base64="TdaUNydNFNkLCcFnsREA8HNAHUo=">AAACH3icZZDLSsNAFIYn9VbrrerSzWBRXJXEjW4KRTcu66UXaNIwmZy0QyeTkJkUS8gb+BK+glvduxO33fokpmmh2v4w8PP/58CZzwk5k0rXJ1phbX1jc6u4XdrZ3ds/KB8etWQQRxSaNOBB1HGIBM4ENBVTHDphBMR3OLSd4e20b48gkiwQT2ocguWTvmAeo0RlkV0+NynhjzUzMUdAn21mpnbCakba87EpY0eCwuZDz7XLFb2q58KrxpibCpqrYZd/TDegsQ9CUU6k7Bp6qKyERIpRDmnJjCWEhA5JH7qZFcQHaSX5f1J8liUu9oIoe0LhPP27kRBfyrHvZJM+UQO53E3DRccE5bELyQCIC1G6CESgcgpL1yjv2kqYCGMFgs6O8WKOVYCnALHLIqCKj3Epg2IsI1g1rcuqoVeNe71Sv5njKaITdIoukIGuUB3doQZqIope0Bt6Rx/aq/apfWnfs9GCNt85Rv+kTX4B74ajMQ==</latexit><latexit sha1_base64="9XlcTsKZ+5Zans3yqSIscOgADW4=">AAACH3icZZDLSsNAFIYnXmu9RV26GSyKq5KIoJtC0Y3LeukFmjRMJift0MkkZCbFEvIGvoSv4Fb37sRttz6J6QWq7Q8DP/9/Dpz53IgzqQxjpK2srq1vbBa2its7u3v7+sFhQ4ZJTKFOQx7GLZdI4ExAXTHFoRXFQAKXQ9Pt34775gBiyULxpIYR2AHpCuYzSlQeOfqZRQl/rFipNQD67DArc1JWMbNOgC2ZuBIUth46nqOXjLIxEV425syU0Ew1R/+xvJAmAQhFOZGybRqRslMSK0Y5ZEUrkRAR2iddaOdWkACknU7+k+HTPPGwH8b5EwpP0r8bKQmkHAZuPhkQ1ZOL3Ticd0xQnniQ9oB4EGfzQIRqQmHhGuVf2ykTUaJA0OkxfsKxCvEYIPZYDFTxIS7mUMxFBMumcVE2jbJ5f1mq3szwFNAxOkHnyERXqIruUA3VEUUv6A29ow/tVfvUvrTv6eiKNts5Qv+kjX4B8MajNQ==</latexit><latexit sha1_base64="9XlcTsKZ+5Zans3yqSIscOgADW4=">AAACH3icZZDLSsNAFIYnXmu9RV26GSyKq5KIoJtC0Y3LeukFmjRMJift0MkkZCbFEvIGvoSv4Fb37sRttz6J6QWq7Q8DP/9/Dpz53IgzqQxjpK2srq1vbBa2its7u3v7+sFhQ4ZJTKFOQx7GLZdI4ExAXTHFoRXFQAKXQ9Pt34775gBiyULxpIYR2AHpCuYzSlQeOfqZRQl/rFipNQD67DArc1JWMbNOgC2ZuBIUth46nqOXjLIxEV425syU0Ew1R/+xvJAmAQhFOZGybRqRslMSK0Y5ZEUrkRAR2iddaOdWkACknU7+k+HTPPGwH8b5EwpP0r8bKQmkHAZuPhkQ1ZOL3Ticd0xQnniQ9oB4EGfzQIRqQmHhGuVf2ykTUaJA0OkxfsKxCvEYIPZYDFTxIS7mUMxFBMumcVE2jbJ5f1mq3szwFNAxOkHnyERXqIruUA3VEUUv6A29ow/tVfvUvrTv6eiKNts5Qv+kjX4B8MajNQ==</latexit><latexit sha1_base64="9XlcTsKZ+5Zans3yqSIscOgADW4=">AAACH3icZZDLSsNAFIYnXmu9RV26GSyKq5KIoJtC0Y3LeukFmjRMJift0MkkZCbFEvIGvoSv4Fb37sRttz6J6QWq7Q8DP/9/Dpz53IgzqQxjpK2srq1vbBa2its7u3v7+sFhQ4ZJTKFOQx7GLZdI4ExAXTHFoRXFQAKXQ9Pt34775gBiyULxpIYR2AHpCuYzSlQeOfqZRQl/rFipNQD67DArc1JWMbNOgC2ZuBIUth46nqOXjLIxEV425syU0Ew1R/+xvJAmAQhFOZGybRqRslMSK0Y5ZEUrkRAR2iddaOdWkACknU7+k+HTPPGwH8b5EwpP0r8bKQmkHAZuPhkQ1ZOL3Ticd0xQnniQ9oB4EGfzQIRqQmHhGuVf2ykTUaJA0OkxfsKxCvEYIPZYDFTxIS7mUMxFBMumcVE2jbJ5f1mq3szwFNAxOkHnyERXqIruUA3VEUUv6A29ow/tVfvUvrTv6eiKNts5Qv+kjX4B8MajNQ==</latexit><latexit sha1_base64="9XlcTsKZ+5Zans3yqSIscOgADW4=">AAACH3icZZDLSsNAFIYnXmu9RV26GSyKq5KIoJtC0Y3LeukFmjRMJift0MkkZCbFEvIGvoSv4Fb37sRttz6J6QWq7Q8DP/9/Dpz53IgzqQxjpK2srq1vbBa2its7u3v7+sFhQ4ZJTKFOQx7GLZdI4ExAXTHFoRXFQAKXQ9Pt34775gBiyULxpIYR2AHpCuYzSlQeOfqZRQl/rFipNQD67DArc1JWMbNOgC2ZuBIUth46nqOXjLIxEV425syU0Ew1R/+xvJAmAQhFOZGybRqRslMSK0Y5ZEUrkRAR2iddaOdWkACknU7+k+HTPPGwH8b5EwpP0r8bKQmkHAZuPhkQ1ZOL3Ticd0xQnniQ9oB4EGfzQIRqQmHhGuVf2ykTUaJA0OkxfsKxCvEYIPZYDFTxIS7mUMxFBMumcVE2jbJ5f1mq3szwFNAxOkHnyERXqIruUA3VEUUv6A29ow/tVfvUvrTv6eiKNts5Qv+kjX4B8MajNQ==</latexit><latexit sha1_base64="9XlcTsKZ+5Zans3yqSIscOgADW4=">AAACH3icZZDLSsNAFIYnXmu9RV26GSyKq5KIoJtC0Y3LeukFmjRMJift0MkkZCbFEvIGvoSv4Fb37sRttz6J6QWq7Q8DP/9/Dpz53IgzqQxjpK2srq1vbBa2its7u3v7+sFhQ4ZJTKFOQx7GLZdI4ExAXTHFoRXFQAKXQ9Pt34775gBiyULxpIYR2AHpCuYzSlQeOfqZRQl/rFipNQD67DArc1JWMbNOgC2ZuBIUth46nqOXjLIxEV425syU0Ew1R/+xvJAmAQhFOZGybRqRslMSK0Y5ZEUrkRAR2iddaOdWkACknU7+k+HTPPGwH8b5EwpP0r8bKQmkHAZuPhkQ1ZOL3Ticd0xQnniQ9oB4EGfzQIRqQmHhGuVf2ykTUaJA0OkxfsKxCvEYIPZYDFTxIS7mUMxFBMumcVE2jbJ5f1mq3szwFNAxOkHnyERXqIruUA3VEUUv6A29ow/tVfvUvrTv6eiKNts5Qv+kjX4B8MajNQ==</latexit><latexit sha1_base64="9XlcTsKZ+5Zans3yqSIscOgADW4=">AAACH3icZZDLSsNAFIYnXmu9RV26GSyKq5KIoJtC0Y3LeukFmjRMJift0MkkZCbFEvIGvoSv4Fb37sRttz6J6QWq7Q8DP/9/Dpz53IgzqQxjpK2srq1vbBa2its7u3v7+sFhQ4ZJTKFOQx7GLZdI4ExAXTHFoRXFQAKXQ9Pt34775gBiyULxpIYR2AHpCuYzSlQeOfqZRQl/rFipNQD67DArc1JWMbNOgC2ZuBIUth46nqOXjLIxEV425syU0Ew1R/+xvJAmAQhFOZGybRqRslMSK0Y5ZEUrkRAR2iddaOdWkACknU7+k+HTPPGwH8b5EwpP0r8bKQmkHAZuPhkQ1ZOL3Ticd0xQnniQ9oB4EGfzQIRqQmHhGuVf2ykTUaJA0OkxfsKxCvEYIPZYDFTxIS7mUMxFBMumcVE2jbJ5f1mq3szwFNAxOkHnyERXqIruUA3VEUUv6A29ow/tVfvUvrTv6eiKNts5Qv+kjX4B8MajNQ==</latexit>

thereexistsd = h m, {↵j ,�j}hj=1 ⇢ R+,U0 = 0<latexit sha1_base64="EPJVCzeYSzIp4tpGVVKKwDAYWVY=">AAACN3icZVBNS8NAEN34bf2qevSyWATBUhIR9FIRvXisYlVwa9hspmbrZhOzE6GE/Bv/hH/Bq148CR7Eq//A9APU+mCYx3szMPO8WEmDtv1qjY1PTE5Nz8yW5uYXFpfKyyvnJkoTAU0RqSi59LgBJTU0UaKCyzgBHnoKLrzbo55/cQ+JkZE+w24MrZDfaNmWgmMhueV9vx4wBXc0rFKWMa7igLudKvMAi85yN+vUnfw6oMykngGk7PR6q8ruQTRdu2675Ypds/ug/4kzJBUyRMMtvzM/EmkIGoXixlw5doytjCcohYK8xFIDMRe3/AauCqp5CKaV9f/M6Uah+LQdJUVppH3190bGQ2O6oVdMhhwDM+r1xB9PaqFSH7IAuA9J/iPoCPvpjFyD7b1WJnWcImgxOKadKooR7QVLfZmAQNWlpSIUZzSC/+R8u+bYNedkp3JwOIxnhqyRdbJJHLJLDsgxaZAmEeSBPJFn8mI9Wm/Wh/U5GB2zhjur5A+sr2/exquh</latexit><latexit sha1_base64="EPJVCzeYSzIp4tpGVVKKwDAYWVY=">AAACN3icZVBNS8NAEN34bf2qevSyWATBUhIR9FIRvXisYlVwa9hspmbrZhOzE6GE/Bv/hH/Bq148CR7Eq//A9APU+mCYx3szMPO8WEmDtv1qjY1PTE5Nz8yW5uYXFpfKyyvnJkoTAU0RqSi59LgBJTU0UaKCyzgBHnoKLrzbo55/cQ+JkZE+w24MrZDfaNmWgmMhueV9vx4wBXc0rFKWMa7igLudKvMAi85yN+vUnfw6oMykngGk7PR6q8ruQTRdu2675Ypds/ug/4kzJBUyRMMtvzM/EmkIGoXixlw5doytjCcohYK8xFIDMRe3/AauCqp5CKaV9f/M6Uah+LQdJUVppH3190bGQ2O6oVdMhhwDM+r1xB9PaqFSH7IAuA9J/iPoCPvpjFyD7b1WJnWcImgxOKadKooR7QVLfZmAQNWlpSIUZzSC/+R8u+bYNedkp3JwOIxnhqyRdbJJHLJLDsgxaZAmEeSBPJFn8mI9Wm/Wh/U5GB2zhjur5A+sr2/exquh</latexit><latexit sha1_base64="EPJVCzeYSzIp4tpGVVKKwDAYWVY=">AAACN3icZVBNS8NAEN34bf2qevSyWATBUhIR9FIRvXisYlVwa9hspmbrZhOzE6GE/Bv/hH/Bq148CR7Eq//A9APU+mCYx3szMPO8WEmDtv1qjY1PTE5Nz8yW5uYXFpfKyyvnJkoTAU0RqSi59LgBJTU0UaKCyzgBHnoKLrzbo55/cQ+JkZE+w24MrZDfaNmWgmMhueV9vx4wBXc0rFKWMa7igLudKvMAi85yN+vUnfw6oMykngGk7PR6q8ruQTRdu2675Ypds/ug/4kzJBUyRMMtvzM/EmkIGoXixlw5doytjCcohYK8xFIDMRe3/AauCqp5CKaV9f/M6Uah+LQdJUVppH3190bGQ2O6oVdMhhwDM+r1xB9PaqFSH7IAuA9J/iPoCPvpjFyD7b1WJnWcImgxOKadKooR7QVLfZmAQNWlpSIUZzSC/+R8u+bYNedkp3JwOIxnhqyRdbJJHLJLDsgxaZAmEeSBPJFn8mI9Wm/Wh/U5GB2zhjur5A+sr2/exquh</latexit><latexit sha1_base64="Qlfzn0O7wPopyBABbLiZvrygJh0=">AAAB6XicZZDLSsNAGIX/1FuNVevazWARXJXEjS4FNy4r2Au0oUwmf9qhcwkzE7GEvoBb9+7EZ3Lrk5imBbU9MHD4zgycOXEmuHVB8OXVdnb39g/qh/5Rwz8+OW02elbnhmGXaaHNIKYWBVfYddwJHGQGqYwF9uPZ/TLvP6OxXKsnN88wknSieMoZdSXqjJutoB1UItsmXJsWrDVufo8SzXKJyjFBrR2GQeaighrHmcCFP8otZpTN6ASHpVVUoo2KquaCXJYkIak25VGOVPTvi4JKa+cyLm9K6qZ2M1vC34wrJvIEiynSBM3iFyjtqs9ttHHpbVRwleUOFVuVSXNBnCbLXUjCDTIn5sQvNwk3F9g2vet2GLTDxwDqcA4XcAUh3MAdPEAHusAggVd48168d+9jtV3NW494Bv/kff4AN96Q9Q==</latexit><latexit sha1_base64="QlGmrOoRFERJNhT2ZVAiHDmDGDQ=">AAACLHicZZDNSuRAFIVv/Lf9a926KRRBsGkqbnTTg+DGpYqtgtWGSuW2Ka1UMqkboQl5G19iXmG2MxtXgguZx5h026C2B4o6nFMXbn1hZrQjzp+9qemZ2bn5hcXG0vLK6lpzffnSpUWusKtSk+bXoXRotMUuaTJ4neUok9DgVfhwPOyvHjF3OrUXNMiwl8g7q/taSaqjoPkj6sTC4E+WtJgohTRZLIP7lgiR6ltUQXnf8avbmAlXhA6JifPbvZZ4RNUNeIcHzW3e5iOx78Yfm20Y6zRovoooVUWClpSRzt34PKNeKXPSymDVEIXDTKoHeYc3tbUyQdcrR/+s2E6dRKyf5vWxxEbp54lSJs4NkrB+mUiK3WQ3DD86bZUpIixjlBHm1UdgUxrRmdiG+oe9UtusILTqfZl+YRilbAiWRTpHRWbAGjUUfxLBd3O53/Z52z/jsACbsAW74MMBHMEJnEIXFDzBb/gDf71f3ov39o5vyhtz3IAv8v79B4Tsqf8=</latexit><latexit sha1_base64="QlGmrOoRFERJNhT2ZVAiHDmDGDQ=">AAACLHicZZDNSuRAFIVv/Lf9a926KRRBsGkqbnTTg+DGpYqtgtWGSuW2Ka1UMqkboQl5G19iXmG2MxtXgguZx5h026C2B4o6nFMXbn1hZrQjzp+9qemZ2bn5hcXG0vLK6lpzffnSpUWusKtSk+bXoXRotMUuaTJ4neUok9DgVfhwPOyvHjF3OrUXNMiwl8g7q/taSaqjoPkj6sTC4E+WtJgohTRZLIP7lgiR6ltUQXnf8avbmAlXhA6JifPbvZZ4RNUNeIcHzW3e5iOx78Yfm20Y6zRovoooVUWClpSRzt34PKNeKXPSymDVEIXDTKoHeYc3tbUyQdcrR/+s2E6dRKyf5vWxxEbp54lSJs4NkrB+mUiK3WQ3DD86bZUpIixjlBHm1UdgUxrRmdiG+oe9UtusILTqfZl+YRilbAiWRTpHRWbAGjUUfxLBd3O53/Z52z/jsACbsAW74MMBHMEJnEIXFDzBb/gDf71f3ov39o5vyhtz3IAv8v79B4Tsqf8=</latexit><latexit sha1_base64="sgY5Mo7yLszbw3HI3ZsBWqe4b8g=">AAACN3icZVBNS8NAEN34bf2qevSyWATBUjZe9FIRvXhUsSq4NWw2U7O62cTsRCgh/8Y/4V/wqhdPggfx6j8wrQW1Phjm8d4MzDw/0coiYy/OyOjY+MTk1HRlZnZufqG6uHRq4yyV0JKxjtNzX1jQykALFWo4T1IQka/hzL/Z7/lnd5BaFZsT7CbQjsSVUR0lBZaSV90JmiHXcEujOuU5FzoJhXdd5z5g2Xnh5ddNt7gMKbeZbwEpP77cqPM7kC2PNZlXrbEG64P+J+6A1MgAh171jQexzCIwKLWw9sJlCbZzkaKSGooKzywkQt6IK7goqRER2Hbe/7Oga6US0E6clmWQ9tXfG7mIrO1GfjkZCQztsNcTfzxlpM4CyEMQAaTFj2Bi7KczdA12ttu5MkmGYOT3MZ1MU4xpL1gaqBQk6i6tlKG4wxH8J6ebDZc13CNW290bxDNFVsgqWScu2SK75IAckhaR5J48kify7Dw4r8678/E9OuIMdpbJHzifX92Gq50=</latexit><latexit sha1_base64="EPJVCzeYSzIp4tpGVVKKwDAYWVY=">AAACN3icZVBNS8NAEN34bf2qevSyWATBUhIR9FIRvXisYlVwa9hspmbrZhOzE6GE/Bv/hH/Bq148CR7Eq//A9APU+mCYx3szMPO8WEmDtv1qjY1PTE5Nz8yW5uYXFpfKyyvnJkoTAU0RqSi59LgBJTU0UaKCyzgBHnoKLrzbo55/cQ+JkZE+w24MrZDfaNmWgmMhueV9vx4wBXc0rFKWMa7igLudKvMAi85yN+vUnfw6oMykngGk7PR6q8ruQTRdu2675Ypds/ug/4kzJBUyRMMtvzM/EmkIGoXixlw5doytjCcohYK8xFIDMRe3/AauCqp5CKaV9f/M6Uah+LQdJUVppH3190bGQ2O6oVdMhhwDM+r1xB9PaqFSH7IAuA9J/iPoCPvpjFyD7b1WJnWcImgxOKadKooR7QVLfZmAQNWlpSIUZzSC/+R8u+bYNedkp3JwOIxnhqyRdbJJHLJLDsgxaZAmEeSBPJFn8mI9Wm/Wh/U5GB2zhjur5A+sr2/exquh</latexit><latexit sha1_base64="EPJVCzeYSzIp4tpGVVKKwDAYWVY=">AAACN3icZVBNS8NAEN34bf2qevSyWATBUhIR9FIRvXisYlVwa9hspmbrZhOzE6GE/Bv/hH/Bq148CR7Eq//A9APU+mCYx3szMPO8WEmDtv1qjY1PTE5Nz8yW5uYXFpfKyyvnJkoTAU0RqSi59LgBJTU0UaKCyzgBHnoKLrzbo55/cQ+JkZE+w24MrZDfaNmWgmMhueV9vx4wBXc0rFKWMa7igLudKvMAi85yN+vUnfw6oMykngGk7PR6q8ruQTRdu2675Ypds/ug/4kzJBUyRMMtvzM/EmkIGoXixlw5doytjCcohYK8xFIDMRe3/AauCqp5CKaV9f/M6Uah+LQdJUVppH3190bGQ2O6oVdMhhwDM+r1xB9PaqFSH7IAuA9J/iPoCPvpjFyD7b1WJnWcImgxOKadKooR7QVLfZmAQNWlpSIUZzSC/+R8u+bYNedkp3JwOIxnhqyRdbJJHLJLDsgxaZAmEeSBPJFn8mI9Wm/Wh/U5GB2zhjur5A+sr2/exquh</latexit><latexit sha1_base64="EPJVCzeYSzIp4tpGVVKKwDAYWVY=">AAACN3icZVBNS8NAEN34bf2qevSyWATBUhIR9FIRvXisYlVwa9hspmbrZhOzE6GE/Bv/hH/Bq148CR7Eq//A9APU+mCYx3szMPO8WEmDtv1qjY1PTE5Nz8yW5uYXFpfKyyvnJkoTAU0RqSi59LgBJTU0UaKCyzgBHnoKLrzbo55/cQ+JkZE+w24MrZDfaNmWgmMhueV9vx4wBXc0rFKWMa7igLudKvMAi85yN+vUnfw6oMykngGk7PR6q8ruQTRdu2675Ypds/ug/4kzJBUyRMMtvzM/EmkIGoXixlw5doytjCcohYK8xFIDMRe3/AauCqp5CKaV9f/M6Uah+LQdJUVppH3190bGQ2O6oVdMhhwDM+r1xB9PaqFSH7IAuA9J/iPoCPvpjFyD7b1WJnWcImgxOKadKooR7QVLfZmAQNWlpSIUZzSC/+R8u+bYNedkp3JwOIxnhqyRdbJJHLJLDsgxaZAmEeSBPJFn8mI9Wm/Wh/U5GB2zhjur5A+sr2/exquh</latexit><latexit sha1_base64="EPJVCzeYSzIp4tpGVVKKwDAYWVY=">AAACN3icZVBNS8NAEN34bf2qevSyWATBUhIR9FIRvXisYlVwa9hspmbrZhOzE6GE/Bv/hH/Bq148CR7Eq//A9APU+mCYx3szMPO8WEmDtv1qjY1PTE5Nz8yW5uYXFpfKyyvnJkoTAU0RqSi59LgBJTU0UaKCyzgBHnoKLrzbo55/cQ+JkZE+w24MrZDfaNmWgmMhueV9vx4wBXc0rFKWMa7igLudKvMAi85yN+vUnfw6oMykngGk7PR6q8ruQTRdu2675Ypds/ug/4kzJBUyRMMtvzM/EmkIGoXixlw5doytjCcohYK8xFIDMRe3/AauCqp5CKaV9f/M6Uah+LQdJUVppH3190bGQ2O6oVdMhhwDM+r1xB9PaqFSH7IAuA9J/iPoCPvpjFyD7b1WJnWcImgxOKadKooR7QVLfZmAQNWlpSIUZzSC/+R8u+bYNedkp3JwOIxnhqyRdbJJHLJLDsgxaZAmEeSBPJFn8mI9Wm/Wh/U5GB2zhjur5A+sr2/exquh</latexit><latexit sha1_base64="EPJVCzeYSzIp4tpGVVKKwDAYWVY=">AAACN3icZVBNS8NAEN34bf2qevSyWATBUhIR9FIRvXisYlVwa9hspmbrZhOzE6GE/Bv/hH/Bq148CR7Eq//A9APU+mCYx3szMPO8WEmDtv1qjY1PTE5Nz8yW5uYXFpfKyyvnJkoTAU0RqSi59LgBJTU0UaKCyzgBHnoKLrzbo55/cQ+JkZE+w24MrZDfaNmWgmMhueV9vx4wBXc0rFKWMa7igLudKvMAi85yN+vUnfw6oMykngGk7PR6q8ruQTRdu2675Ypds/ug/4kzJBUyRMMtvzM/EmkIGoXixlw5doytjCcohYK8xFIDMRe3/AauCqp5CKaV9f/M6Uah+LQdJUVppH3190bGQ2O6oVdMhhwDM+r1xB9PaqFSH7IAuA9J/iPoCPvpjFyD7b1WJnWcImgxOKadKooR7QVLfZmAQNWlpSIUZzSC/+R8u+bYNedkp3JwOIxnhqyRdbJJHLJLDsgxaZAmEeSBPJFn8mI9Wm/Wh/U5GB2zhjur5A+sr2/exquh</latexit><latexit sha1_base64="EPJVCzeYSzIp4tpGVVKKwDAYWVY=">AAACN3icZVBNS8NAEN34bf2qevSyWATBUhIR9FIRvXisYlVwa9hspmbrZhOzE6GE/Bv/hH/Bq148CR7Eq//A9APU+mCYx3szMPO8WEmDtv1qjY1PTE5Nz8yW5uYXFpfKyyvnJkoTAU0RqSi59LgBJTU0UaKCyzgBHnoKLrzbo55/cQ+JkZE+w24MrZDfaNmWgmMhueV9vx4wBXc0rFKWMa7igLudKvMAi85yN+vUnfw6oMykngGk7PR6q8ruQTRdu2675Ypds/ug/4kzJBUyRMMtvzM/EmkIGoXixlw5doytjCcohYK8xFIDMRe3/AauCqp5CKaV9f/M6Uah+LQdJUVppH3190bGQ2O6oVdMhhwDM+r1xB9PaqFSH7IAuA9J/iPoCPvpjFyD7b1WJnWcImgxOKadKooR7QVLfZmAQNWlpSIUZzSC/+R8u+bYNedkp3JwOIxnhqyRdbJJHLJLDsgxaZAmEeSBPJFn8mI9Wm/Wh/U5GB2zhjur5A+sr2/exquh</latexit>

Forany

Distance:Euclideandistancefrominitialization.

Comparinggeneralizationmeasures:

Behaviorofdifferentnormswithover-parametrization:

Comparingcapacitybounds:

# Reference Measure

(1) Harvey et al. 2017 O(dh)

(2) Bartlett and Mendelson 2002 O

⇣kUk1,1 kVk1,1

(3) Neyshabur et al. 2015 O�kUkF kVkF

(4) Bartlett et al. 2017 O

⇣kUk2 kV �V0k1,2 + kU�U0k1,2 kVk2

(5) Neyshabur et al. 2018 O

⇣kUk2 kV �V0kF +

ph kU�U0kF kVk2

(6) ours O

⇣kU0k2 kVkF +

��U�U0��FkVkF +

ph

<latexit sha1_base64="zpxXtO4LSS2jBCXKSDYBAqzaZPM=">AAAFyHicjVRLb9NAEHZpAk14tXDksiKlStUS2aalVU9VkaIKCSiItJW6IaztSWy6XpvddVHk+sJf5MSB/8LYSamT9MFKlmfnm8c3szvrxDxQ2jR/z92Zr1Tv3luo1e8/ePjo8eLSk0MVJdKFjhvxSB47TAEPBHR0oDkcxxJY6HA4ck7f5PjRGUgVROKzHsbQDdlABP3AZRpVvaX5X9SBQSBSzdAlo5w5wFPt8B03CmMmAxWJrF6jKmSc53/AFGKgfYoOmF5BnG7EOjcZxXFBaJC4r10GTjiT2Yl2uum5e87xc89zgxr1c9qF1CAr5BP0QYJwAeV3wFQigVA6adi0VhHdZ/IMhgQ0YbxFbNPaQuVySnXAPUg/ZE3PX82WZ53t3HmPSaxBo6/wMI/wgGORGMW0J6MQyqGvm4SKSIYpPQO3k/VSGoi+Hq4TKysBh1OADAa+vpLCq5zCexgqH/siSyVsrpDbk7enkrZLqWZzbUyUW+rW7WXapJyIvCx+PRMpWOt2RtbKcGcEdy7hCY72Te3YvK4d2/9xFvY1HNvIj6rvUqd+VnK4ZNmeZngTxdc5RZw3VTAiN1HKY9vTJ7Q2Q+GLeeVRllhfy4fiff03UsXmYuAuEJzi3mLDbJnFIrOCNRYaxngd9Bb/UC9ykxBDuZwpdWKZse6meHMCF+PVaYJzztxTNoATFAULQXXT4vXJyAvUeKQfSfyEJoW27JGyUKlh6KBlyLSvprFceRV2kuj+djcNRJxofBVGifoJJzoi+VNGvECCq/mwTgub1AfmFZ0YbUWki0cuq2M/rOnqZ4VDu2WZLeuj3djdG3dmwXhmPDeahmVsGbvGvnFgdAy3slP5Wgkq36pvq3H1R3U4Mr0zN/Z5akys6s+/DxrU1Q==</latexit><latexit sha1_base64="zpxXtO4LSS2jBCXKSDYBAqzaZPM=">AAAFyHicjVRLb9NAEHZpAk14tXDksiKlStUS2aalVU9VkaIKCSiItJW6IaztSWy6XpvddVHk+sJf5MSB/8LYSamT9MFKlmfnm8c3szvrxDxQ2jR/z92Zr1Tv3luo1e8/ePjo8eLSk0MVJdKFjhvxSB47TAEPBHR0oDkcxxJY6HA4ck7f5PjRGUgVROKzHsbQDdlABP3AZRpVvaX5X9SBQSBSzdAlo5w5wFPt8B03CmMmAxWJrF6jKmSc53/AFGKgfYoOmF5BnG7EOjcZxXFBaJC4r10GTjiT2Yl2uum5e87xc89zgxr1c9qF1CAr5BP0QYJwAeV3wFQigVA6adi0VhHdZ/IMhgQ0YbxFbNPaQuVySnXAPUg/ZE3PX82WZ53t3HmPSaxBo6/wMI/wgGORGMW0J6MQyqGvm4SKSIYpPQO3k/VSGoi+Hq4TKysBh1OADAa+vpLCq5zCexgqH/siSyVsrpDbk7enkrZLqWZzbUyUW+rW7WXapJyIvCx+PRMpWOt2RtbKcGcEdy7hCY72Te3YvK4d2/9xFvY1HNvIj6rvUqd+VnK4ZNmeZngTxdc5RZw3VTAiN1HKY9vTJ7Q2Q+GLeeVRllhfy4fiff03UsXmYuAuEJzi3mLDbJnFIrOCNRYaxngd9Bb/UC9ykxBDuZwpdWKZse6meHMCF+PVaYJzztxTNoATFAULQXXT4vXJyAvUeKQfSfyEJoW27JGyUKlh6KBlyLSvprFceRV2kuj+djcNRJxofBVGifoJJzoi+VNGvECCq/mwTgub1AfmFZ0YbUWki0cuq2M/rOnqZ4VDu2WZLeuj3djdG3dmwXhmPDeahmVsGbvGvnFgdAy3slP5Wgkq36pvq3H1R3U4Mr0zN/Z5akys6s+/DxrU1Q==</latexit><latexit sha1_base64="zpxXtO4LSS2jBCXKSDYBAqzaZPM=">AAAFyHicjVRLb9NAEHZpAk14tXDksiKlStUS2aalVU9VkaIKCSiItJW6IaztSWy6XpvddVHk+sJf5MSB/8LYSamT9MFKlmfnm8c3szvrxDxQ2jR/z92Zr1Tv3luo1e8/ePjo8eLSk0MVJdKFjhvxSB47TAEPBHR0oDkcxxJY6HA4ck7f5PjRGUgVROKzHsbQDdlABP3AZRpVvaX5X9SBQSBSzdAlo5w5wFPt8B03CmMmAxWJrF6jKmSc53/AFGKgfYoOmF5BnG7EOjcZxXFBaJC4r10GTjiT2Yl2uum5e87xc89zgxr1c9qF1CAr5BP0QYJwAeV3wFQigVA6adi0VhHdZ/IMhgQ0YbxFbNPaQuVySnXAPUg/ZE3PX82WZ53t3HmPSaxBo6/wMI/wgGORGMW0J6MQyqGvm4SKSIYpPQO3k/VSGoi+Hq4TKysBh1OADAa+vpLCq5zCexgqH/siSyVsrpDbk7enkrZLqWZzbUyUW+rW7WXapJyIvCx+PRMpWOt2RtbKcGcEdy7hCY72Te3YvK4d2/9xFvY1HNvIj6rvUqd+VnK4ZNmeZngTxdc5RZw3VTAiN1HKY9vTJ7Q2Q+GLeeVRllhfy4fiff03UsXmYuAuEJzi3mLDbJnFIrOCNRYaxngd9Bb/UC9ykxBDuZwpdWKZse6meHMCF+PVaYJzztxTNoATFAULQXXT4vXJyAvUeKQfSfyEJoW27JGyUKlh6KBlyLSvprFceRV2kuj+djcNRJxofBVGifoJJzoi+VNGvECCq/mwTgub1AfmFZ0YbUWki0cuq2M/rOnqZ4VDu2WZLeuj3djdG3dmwXhmPDeahmVsGbvGvnFgdAy3slP5Wgkq36pvq3H1R3U4Mr0zN/Z5akys6s+/DxrU1Q==</latexit><latexit sha1_base64="zpxXtO4LSS2jBCXKSDYBAqzaZPM=">AAAFyHicjVRLb9NAEHZpAk14tXDksiKlStUS2aalVU9VkaIKCSiItJW6IaztSWy6XpvddVHk+sJf5MSB/8LYSamT9MFKlmfnm8c3szvrxDxQ2jR/z92Zr1Tv3luo1e8/ePjo8eLSk0MVJdKFjhvxSB47TAEPBHR0oDkcxxJY6HA4ck7f5PjRGUgVROKzHsbQDdlABP3AZRpVvaX5X9SBQSBSzdAlo5w5wFPt8B03CmMmAxWJrF6jKmSc53/AFGKgfYoOmF5BnG7EOjcZxXFBaJC4r10GTjiT2Yl2uum5e87xc89zgxr1c9qF1CAr5BP0QYJwAeV3wFQigVA6adi0VhHdZ/IMhgQ0YbxFbNPaQuVySnXAPUg/ZE3PX82WZ53t3HmPSaxBo6/wMI/wgGORGMW0J6MQyqGvm4SKSIYpPQO3k/VSGoi+Hq4TKysBh1OADAa+vpLCq5zCexgqH/siSyVsrpDbk7enkrZLqWZzbUyUW+rW7WXapJyIvCx+PRMpWOt2RtbKcGcEdy7hCY72Te3YvK4d2/9xFvY1HNvIj6rvUqd+VnK4ZNmeZngTxdc5RZw3VTAiN1HKY9vTJ7Q2Q+GLeeVRllhfy4fiff03UsXmYuAuEJzi3mLDbJnFIrOCNRYaxngd9Bb/UC9ykxBDuZwpdWKZse6meHMCF+PVaYJzztxTNoATFAULQXXT4vXJyAvUeKQfSfyEJoW27JGyUKlh6KBlyLSvprFceRV2kuj+djcNRJxofBVGifoJJzoi+VNGvECCq/mwTgub1AfmFZ0YbUWki0cuq2M/rOnqZ4VDu2WZLeuj3djdG3dmwXhmPDeahmVsGbvGvnFgdAy3slP5Wgkq36pvq3H1R3U4Mr0zN/Z5akys6s+/DxrU1Q==</latexit>

W =�(V,U) | V 2 Rc⇥h,U 2 Rh⇥d, kvik ↵i,

��ui � u0i

��2 �i

<latexit sha1_base64="tysZu14Qz2zOKHzwGx3h8pfI5P8=">AAACq3icbVFba9swFJa9W5fdsu1xDMTCoIM22GWwvgzK+rKHDbpLkrIoNbJ8HIvKsisdB4Lw79pv2cP+y2QnHV27A5LO+b7vcC5KayUtRtGvILx1+87dezv3Bw8ePnr8ZPj02dRWjREwEZWqzGnKLSipYYISFZzWBniZKpil58cdP1uBsbLS33Fdw6LkSy1zKTh6KBn+ZIKr2XumIEfm6C5bgZjudffkDWWlzGiPMKkp+3rmBENZgqVFu9HQS6LYElm751jfljOQtUxXpnSddJXI1le5oIyruuA++itMVQNXlE0i9/v3LPKq5IB2aSwF7JKYkcsCWZsMR9E46o3edOKtMyJbO0mGv1lWiaYEjUJxa+dxVOPCcYNSKGgHrLFQc3HOlzD3ruZ+moXrG2zpa49kNK+MPxppj17NcLy0dl2mXllyLOx1rgP/x80bzA8XTuq6QdBiUyhvFMWKdr9FM2lAoFoPWK9xBfAMTHsZ6gr7f2wHfh/x9elvOtODcRyN4y9vR0cftpvZIS/IK7JLYvKOHJGP5IRMiAheBsfBp+BzuB9+C3+EbCMNg23Oc/KPhfAHBvTVuQ==</latexit><latexit sha1_base64="tysZu14Qz2zOKHzwGx3h8pfI5P8=">AAACq3icbVFba9swFJa9W5fdsu1xDMTCoIM22GWwvgzK+rKHDbpLkrIoNbJ8HIvKsisdB4Lw79pv2cP+y2QnHV27A5LO+b7vcC5KayUtRtGvILx1+87dezv3Bw8ePnr8ZPj02dRWjREwEZWqzGnKLSipYYISFZzWBniZKpil58cdP1uBsbLS33Fdw6LkSy1zKTh6KBn+ZIKr2XumIEfm6C5bgZjudffkDWWlzGiPMKkp+3rmBENZgqVFu9HQS6LYElm751jfljOQtUxXpnSddJXI1le5oIyruuA++itMVQNXlE0i9/v3LPKq5IB2aSwF7JKYkcsCWZsMR9E46o3edOKtMyJbO0mGv1lWiaYEjUJxa+dxVOPCcYNSKGgHrLFQc3HOlzD3ruZ+moXrG2zpa49kNK+MPxppj17NcLy0dl2mXllyLOx1rgP/x80bzA8XTuq6QdBiUyhvFMWKdr9FM2lAoFoPWK9xBfAMTHsZ6gr7f2wHfh/x9elvOtODcRyN4y9vR0cftpvZIS/IK7JLYvKOHJGP5IRMiAheBsfBp+BzuB9+C3+EbCMNg23Oc/KPhfAHBvTVuQ==</latexit><latexit sha1_base64="tysZu14Qz2zOKHzwGx3h8pfI5P8=">AAACq3icbVFba9swFJa9W5fdsu1xDMTCoIM22GWwvgzK+rKHDbpLkrIoNbJ8HIvKsisdB4Lw79pv2cP+y2QnHV27A5LO+b7vcC5KayUtRtGvILx1+87dezv3Bw8ePnr8ZPj02dRWjREwEZWqzGnKLSipYYISFZzWBniZKpil58cdP1uBsbLS33Fdw6LkSy1zKTh6KBn+ZIKr2XumIEfm6C5bgZjudffkDWWlzGiPMKkp+3rmBENZgqVFu9HQS6LYElm751jfljOQtUxXpnSddJXI1le5oIyruuA++itMVQNXlE0i9/v3LPKq5IB2aSwF7JKYkcsCWZsMR9E46o3edOKtMyJbO0mGv1lWiaYEjUJxa+dxVOPCcYNSKGgHrLFQc3HOlzD3ruZ+moXrG2zpa49kNK+MPxppj17NcLy0dl2mXllyLOx1rgP/x80bzA8XTuq6QdBiUyhvFMWKdr9FM2lAoFoPWK9xBfAMTHsZ6gr7f2wHfh/x9elvOtODcRyN4y9vR0cftpvZIS/IK7JLYvKOHJGP5IRMiAheBsfBp+BzuB9+C3+EbCMNg23Oc/KPhfAHBvTVuQ==</latexit><latexit sha1_base64="DTvkFUA7XI7FgwDpHjTga/sjJ6I=">AAAB5HicbVBNS8NAEJ3Urxqr1rOXYBE8lcSLHgUvHivYD2hL2Wwm7dLNbtidCCXkD3j15E38Vx78L6ZpBW19MPD2vRl25oWpFJZ8/9Op7ezu7R/UD92jhnt8ctps9KzODMcu11KbQcgsSqGwS4IkDlKDLAkl9sP5/dLvP6OxQqsnWqQ4TthUiVhwRqXUmTRbftuv4G2TYE1asMak+TWKNM8SVMQls3YY+CmNc2ZIcImFO8ospozP2RSHJVUsQTvOqzUL77JUIi/WpixFXqX+nshZYu0iCcvOhNHMbnpL8T9vmFF8O86FSjNCxVcfxZn0SHvLm71IGOQkF+6o6slnyCI0xc9TaarSKNwyjmDz+G3Su24Hfjt49KEO53ABVxDADdzBA3SgCxwieIFXxzpvzvsqtpqzzu8M/sD5+Abkdo9U</latexit><latexit sha1_base64="Abdeos5yOlIZOgmDpaUl0QhHRtc=">AAACoHicbVHbatwwEJXdS9Jt2mz6WgqioZBCsth5aV8KhbzkIYX0sruhq42R5fFaRJYdabywCH9XvqUP/ZfK3k1Jkw5ImjnnDHNRWitpMYp+BeGjx0+ebm0/GzzfefFyd7i3M7FVYwSMRaUqc5FyC0pqGKNEBRe1AV6mCqbp1UnHT5dgrKz0D1zVMC/5QstcCo4eSoY3THA1/cQU5MgcPWBLEJPD7h6/p6yUGe0RJjVl3y6dYChLsLRo1xp6SxQbImsPHevbcgaylunKlK6TLhPZ+irXlHFVF9xHf4WpauCOsknkUf9eRl6VHNMujaWAXRIzclEga5PhfjSKeqMPnXjj7JONnSfD3yyrRFOCRqG4tbM4qnHuuEEpFLQD1lioubjiC5h5V3M/zdz1Dbb0nUcymlfGH420R+9mOF5auypTryw5FvY+14H/42YN5h/nTuq6QdBiXShvFMWKdr9FM2lAoFoNWK9xBfAMTHsb6gr7f2wHfh/x/ekfOpPjURyN4q8R2SavyVtyQGLygXwmp+ScjIkI3gQnwVnwJTwKv4c/15sLg80KX5F/LGR/AL+k1KI=</latexit><latexit sha1_base64="Abdeos5yOlIZOgmDpaUl0QhHRtc=">AAACoHicbVHbatwwEJXdS9Jt2mz6WgqioZBCsth5aV8KhbzkIYX0sruhq42R5fFaRJYdabywCH9XvqUP/ZfK3k1Jkw5ImjnnDHNRWitpMYp+BeGjx0+ebm0/GzzfefFyd7i3M7FVYwSMRaUqc5FyC0pqGKNEBRe1AV6mCqbp1UnHT5dgrKz0D1zVMC/5QstcCo4eSoY3THA1/cQU5MgcPWBLEJPD7h6/p6yUGe0RJjVl3y6dYChLsLRo1xp6SxQbImsPHevbcgaylunKlK6TLhPZ+irXlHFVF9xHf4WpauCOsknkUf9eRl6VHNMujaWAXRIzclEga5PhfjSKeqMPnXjj7JONnSfD3yyrRFOCRqG4tbM4qnHuuEEpFLQD1lioubjiC5h5V3M/zdz1Dbb0nUcymlfGH420R+9mOF5auypTryw5FvY+14H/42YN5h/nTuq6QdBiXShvFMWKdr9FM2lAoFoNWK9xBfAMTHsb6gr7f2wHfh/x/ekfOpPjURyN4q8R2SavyVtyQGLygXwmp+ScjIkI3gQnwVnwJTwKv4c/15sLg80KX5F/LGR/AL+k1KI=</latexit><latexit sha1_base64="PY2MLwRVODmxjJPdaJ+b7xz3kAg=">AAACq3icbVFbb9MwFHbCbZTLOnhESBYV0pC2KtkLvEya2AsPII1L24m6ixznpLHmOME+qVRZ+V38Fh74Lzhph8bGkWyf833f0bk4rZW0GEW/gvDO3Xv3H+w8HDx6/OTp7nDv2dRWjREwEZWqzHnKLSipYYISFZzXBniZKpill6cdP1uBsbLS33Bdw6LkSy1zKTh6KBn+ZIKr2TFTkCNzdJ+tQEwPunvyhrJSZrRHmNSUfblwgqEswdKi3WjoFVFsiaw9cKxvyxnIWqYrU7pOukpk66v8oIyruuA++itMVQPXlE0iD/v3IvKq5Ih2aSwF7JKYkcsCWZsMR9E46o3eduKtMyJbO0uGv1lWiaYEjUJxa+dxVOPCcYNSKGgHrLFQc3HJlzD3ruZ+moXrG2zpa49kNK+MPxppj17PcLy0dl2mXllyLOxNrgP/x80bzN8tnNR1g6DFplDeKIoV7X6LZtKAQLUesF7jCuAZmPYq1BX2/9gO/D7im9PfdqZH4zgax5+j0cn77WZ2yAvyiuyTmLwlJ+QDOSMTIoKXwWnwMfgUHoZfw+8h20jDYJvznPxjIfwBBbTVtQ==</latexit><latexit sha1_base64="tysZu14Qz2zOKHzwGx3h8pfI5P8=">AAACq3icbVFba9swFJa9W5fdsu1xDMTCoIM22GWwvgzK+rKHDbpLkrIoNbJ8HIvKsisdB4Lw79pv2cP+y2QnHV27A5LO+b7vcC5KayUtRtGvILx1+87dezv3Bw8ePnr8ZPj02dRWjREwEZWqzGnKLSipYYISFZzWBniZKpil58cdP1uBsbLS33Fdw6LkSy1zKTh6KBn+ZIKr2XumIEfm6C5bgZjudffkDWWlzGiPMKkp+3rmBENZgqVFu9HQS6LYElm751jfljOQtUxXpnSddJXI1le5oIyruuA++itMVQNXlE0i9/v3LPKq5IB2aSwF7JKYkcsCWZsMR9E46o3edOKtMyJbO0mGv1lWiaYEjUJxa+dxVOPCcYNSKGgHrLFQc3HOlzD3ruZ+moXrG2zpa49kNK+MPxppj17NcLy0dl2mXllyLOx1rgP/x80bzA8XTuq6QdBiUyhvFMWKdr9FM2lAoFoPWK9xBfAMTHsZ6gr7f2wHfh/x9elvOtODcRyN4y9vR0cftpvZIS/IK7JLYvKOHJGP5IRMiAheBsfBp+BzuB9+C3+EbCMNg23Oc/KPhfAHBvTVuQ==</latexit><latexit sha1_base64="tysZu14Qz2zOKHzwGx3h8pfI5P8=">AAACq3icbVFba9swFJa9W5fdsu1xDMTCoIM22GWwvgzK+rKHDbpLkrIoNbJ8HIvKsisdB4Lw79pv2cP+y2QnHV27A5LO+b7vcC5KayUtRtGvILx1+87dezv3Bw8ePnr8ZPj02dRWjREwEZWqzGnKLSipYYISFZzWBniZKpil58cdP1uBsbLS33Fdw6LkSy1zKTh6KBn+ZIKr2XumIEfm6C5bgZjudffkDWWlzGiPMKkp+3rmBENZgqVFu9HQS6LYElm751jfljOQtUxXpnSddJXI1le5oIyruuA++itMVQNXlE0i9/v3LPKq5IB2aSwF7JKYkcsCWZsMR9E46o3edOKtMyJbO0mGv1lWiaYEjUJxa+dxVOPCcYNSKGgHrLFQc3HOlzD3ruZ+moXrG2zpa49kNK+MPxppj17NcLy0dl2mXllyLOx1rgP/x80bzA8XTuq6QdBiUyhvFMWKdr9FM2lAoFoPWK9xBfAMTHsZ6gr7f2wHfh/x9elvOtODcRyN4y9vR0cftpvZIS/IK7JLYvKOHJGP5IRMiAheBsfBp+BzuB9+C3+EbCMNg23Oc/KPhfAHBvTVuQ==</latexit><latexit sha1_base64="tysZu14Qz2zOKHzwGx3h8pfI5P8=">AAACq3icbVFba9swFJa9W5fdsu1xDMTCoIM22GWwvgzK+rKHDbpLkrIoNbJ8HIvKsisdB4Lw79pv2cP+y2QnHV27A5LO+b7vcC5KayUtRtGvILx1+87dezv3Bw8ePnr8ZPj02dRWjREwEZWqzGnKLSipYYISFZzWBniZKpil58cdP1uBsbLS33Fdw6LkSy1zKTh6KBn+ZIKr2XumIEfm6C5bgZjudffkDWWlzGiPMKkp+3rmBENZgqVFu9HQS6LYElm751jfljOQtUxXpnSddJXI1le5oIyruuA++itMVQNXlE0i9/v3LPKq5IB2aSwF7JKYkcsCWZsMR9E46o3edOKtMyJbO0mGv1lWiaYEjUJxa+dxVOPCcYNSKGgHrLFQc3HOlzD3ruZ+moXrG2zpa49kNK+MPxppj17NcLy0dl2mXllyLOx1rgP/x80bzA8XTuq6QdBiUyhvFMWKdr9FM2lAoFoPWK9xBfAMTHsZ6gr7f2wHfh/x9elvOtODcRyN4y9vR0cftpvZIS/IK7JLYvKOHJGP5IRMiAheBsfBp+BzuB9+C3+EbCMNg23Oc/KPhfAHBvTVuQ==</latexit><latexit sha1_base64="tysZu14Qz2zOKHzwGx3h8pfI5P8=">AAACq3icbVFba9swFJa9W5fdsu1xDMTCoIM22GWwvgzK+rKHDbpLkrIoNbJ8HIvKsisdB4Lw79pv2cP+y2QnHV27A5LO+b7vcC5KayUtRtGvILx1+87dezv3Bw8ePnr8ZPj02dRWjREwEZWqzGnKLSipYYISFZzWBniZKpil58cdP1uBsbLS33Fdw6LkSy1zKTh6KBn+ZIKr2XumIEfm6C5bgZjudffkDWWlzGiPMKkp+3rmBENZgqVFu9HQS6LYElm751jfljOQtUxXpnSddJXI1le5oIyruuA++itMVQNXlE0i9/v3LPKq5IB2aSwF7JKYkcsCWZsMR9E46o3edOKtMyJbO0mGv1lWiaYEjUJxa+dxVOPCcYNSKGgHrLFQc3HOlzD3ruZ+moXrG2zpa49kNK+MPxppj17NcLy0dl2mXllyLOx1rgP/x80bzA8XTuq6QdBiUyhvFMWKdr9FM2lAoFoPWK9xBfAMTHsZ6gr7f2wHfh/x9elvOtODcRyN4y9vR0cftpvZIS/IK7JLYvKOHJGP5IRMiAheBsfBp+BzuB9+C3+EbCMNg23Oc/KPhfAHBvTVuQ==</latexit><latexit sha1_base64="tysZu14Qz2zOKHzwGx3h8pfI5P8=">AAACq3icbVFba9swFJa9W5fdsu1xDMTCoIM22GWwvgzK+rKHDbpLkrIoNbJ8HIvKsisdB4Lw79pv2cP+y2QnHV27A5LO+b7vcC5KayUtRtGvILx1+87dezv3Bw8ePnr8ZPj02dRWjREwEZWqzGnKLSipYYISFZzWBniZKpil58cdP1uBsbLS33Fdw6LkSy1zKTh6KBn+ZIKr2XumIEfm6C5bgZjudffkDWWlzGiPMKkp+3rmBENZgqVFu9HQS6LYElm751jfljOQtUxXpnSddJXI1le5oIyruuA++itMVQNXlE0i9/v3LPKq5IB2aSwF7JKYkcsCWZsMR9E46o3edOKtMyJbO0mGv1lWiaYEjUJxa+dxVOPCcYNSKGgHrLFQc3HOlzD3ruZ+moXrG2zpa49kNK+MPxppj17NcLy0dl2mXllyLOx1rgP/x80bzA8XTuq6QdBiUyhvFMWKdr9FM2lAoFoPWK9xBfAMTHsZ6gr7f2wHfh/x9elvOtODcRyN4y9vR0cftpvZIS/IK7JLYvKOHJGP5IRMiAheBsfBp+BzuB9+C3+EbCMNg23Oc/KPhfAHBvTVuQ==</latexit><latexit sha1_base64="tysZu14Qz2zOKHzwGx3h8pfI5P8=">AAACq3icbVFba9swFJa9W5fdsu1xDMTCoIM22GWwvgzK+rKHDbpLkrIoNbJ8HIvKsisdB4Lw79pv2cP+y2QnHV27A5LO+b7vcC5KayUtRtGvILx1+87dezv3Bw8ePnr8ZPj02dRWjREwEZWqzGnKLSipYYISFZzWBniZKpil58cdP1uBsbLS33Fdw6LkSy1zKTh6KBn+ZIKr2XumIEfm6C5bgZjudffkDWWlzGiPMKkp+3rmBENZgqVFu9HQS6LYElm751jfljOQtUxXpnSddJXI1le5oIyruuA++itMVQNXlE0i9/v3LPKq5IB2aSwF7JKYkcsCWZsMR9E46o3edOKtMyJbO0mGv1lWiaYEjUJxa+dxVOPCcYNSKGgHrLFQc3HOlzD3ruZ+moXrG2zpa49kNK+MPxppj17NcLy0dl2mXllyLOx1rgP/x80bzA8XTuq6QdBiUyhvFMWKdr9FM2lAoFoPWK9xBfAMTHsZ6gr7f2wHfh/x9elvOtODcRyN4y9vR0cftpvZIS/IK7JLYvKOHJGP5IRMiAheBsfBp+BzuB9+C3+EbCMNg23Oc/KPhfAHBvTVuQ==</latexit>

Rademacher complexitybound:

Forany𝑚,w.p.>1- δ overthetrainingset:

Generalizationbound:

• U0 istheinitialization/anyfixedmatrix.

• Rademacher complexity:

• Marginramploss:

PropertiesoflearnedtwolayerReLU networks:

d:inputdimc:#classes h:#hiddenunits m:#samples 𝛾:margin

RS(`� � FW) 2p2c+ 2

�m

hX

j=1

↵j

⇣�j kXkF +

��u0jX

��2

<latexit sha1_base64="HF6EgMRqtbHZ9Bvk1K/GkxiDMLw=">AAACs3icbZFdb9MwFIad8DXKxzq45MZQgTpNqpIKxG6QJpAGlx3QtVLdRY5z0rqzncx2JlWWfxm/hEv+CUkWNNg4kq1X73OOfXROWgpubBT9DMI7d+/df7DzsPfo8ZOnu/29Z6emqDSDKStEoecpNSC4gqnlVsC81EBlKmCWnn9q+OwStOGF+m63JSwlXSmec0ZtbSX9H4RR8TVp7m9DAkIkZEWlpJgwrhlu/OOWzvbxGyLgApNcU+bGxFxo68bMH4y962qkJ6aSidt8iP3ZGjvS9uc0ZJ5QUa5psvG4fiS3wz8sFRV4koJtGFGFlo5cApv75Bgf4GujSjZnUUfGRPPV2u4n/UE0itrAt0XciQHqYpL0f5GsYJUEZZmgxiziqLRLR7XlTIDvkcpASdk5XcGilopKMEvXNurx69rJcF7o+iiLW/fvCkelMVuZ1pmS2rW5yRrzf2xR2fxw6bgqKwuKXX2UVwLbAjcbwxnXwKzY4h7hiokqA7cGmoH214YqbLtQX88kvjmB2+J0PIqjUXzydnD0sZvODnqBXqEhitF7dIS+oAmaIha8DD4Hk+AkfBcuwjTMrlLDoKt5jv6JUP4G+CjYVg==</latexit><latexit sha1_base64="HF6EgMRqtbHZ9Bvk1K/GkxiDMLw=">AAACs3icbZFdb9MwFIad8DXKxzq45MZQgTpNqpIKxG6QJpAGlx3QtVLdRY5z0rqzncx2JlWWfxm/hEv+CUkWNNg4kq1X73OOfXROWgpubBT9DMI7d+/df7DzsPfo8ZOnu/29Z6emqDSDKStEoecpNSC4gqnlVsC81EBlKmCWnn9q+OwStOGF+m63JSwlXSmec0ZtbSX9H4RR8TVp7m9DAkIkZEWlpJgwrhlu/OOWzvbxGyLgApNcU+bGxFxo68bMH4y962qkJ6aSidt8iP3ZGjvS9uc0ZJ5QUa5psvG4fiS3wz8sFRV4koJtGFGFlo5cApv75Bgf4GujSjZnUUfGRPPV2u4n/UE0itrAt0XciQHqYpL0f5GsYJUEZZmgxiziqLRLR7XlTIDvkcpASdk5XcGilopKMEvXNurx69rJcF7o+iiLW/fvCkelMVuZ1pmS2rW5yRrzf2xR2fxw6bgqKwuKXX2UVwLbAjcbwxnXwKzY4h7hiokqA7cGmoH214YqbLtQX88kvjmB2+J0PIqjUXzydnD0sZvODnqBXqEhitF7dIS+oAmaIha8DD4Hk+AkfBcuwjTMrlLDoKt5jv6JUP4G+CjYVg==</latexit><latexit sha1_base64="HF6EgMRqtbHZ9Bvk1K/GkxiDMLw=">AAACs3icbZFdb9MwFIad8DXKxzq45MZQgTpNqpIKxG6QJpAGlx3QtVLdRY5z0rqzncx2JlWWfxm/hEv+CUkWNNg4kq1X73OOfXROWgpubBT9DMI7d+/df7DzsPfo8ZOnu/29Z6emqDSDKStEoecpNSC4gqnlVsC81EBlKmCWnn9q+OwStOGF+m63JSwlXSmec0ZtbSX9H4RR8TVp7m9DAkIkZEWlpJgwrhlu/OOWzvbxGyLgApNcU+bGxFxo68bMH4y962qkJ6aSidt8iP3ZGjvS9uc0ZJ5QUa5psvG4fiS3wz8sFRV4koJtGFGFlo5cApv75Bgf4GujSjZnUUfGRPPV2u4n/UE0itrAt0XciQHqYpL0f5GsYJUEZZmgxiziqLRLR7XlTIDvkcpASdk5XcGilopKMEvXNurx69rJcF7o+iiLW/fvCkelMVuZ1pmS2rW5yRrzf2xR2fxw6bgqKwuKXX2UVwLbAjcbwxnXwKzY4h7hiokqA7cGmoH214YqbLtQX88kvjmB2+J0PIqjUXzydnD0sZvODnqBXqEhitF7dIS+oAmaIha8DD4Hk+AkfBcuwjTMrlLDoKt5jv6JUP4G+CjYVg==</latexit><latexit sha1_base64="HF6EgMRqtbHZ9Bvk1K/GkxiDMLw=">AAACs3icbZFdb9MwFIad8DXKxzq45MZQgTpNqpIKxG6QJpAGlx3QtVLdRY5z0rqzncx2JlWWfxm/hEv+CUkWNNg4kq1X73OOfXROWgpubBT9DMI7d+/df7DzsPfo8ZOnu/29Z6emqDSDKStEoecpNSC4gqnlVsC81EBlKmCWnn9q+OwStOGF+m63JSwlXSmec0ZtbSX9H4RR8TVp7m9DAkIkZEWlpJgwrhlu/OOWzvbxGyLgApNcU+bGxFxo68bMH4y962qkJ6aSidt8iP3ZGjvS9uc0ZJ5QUa5psvG4fiS3wz8sFRV4koJtGFGFlo5cApv75Bgf4GujSjZnUUfGRPPV2u4n/UE0itrAt0XciQHqYpL0f5GsYJUEZZmgxiziqLRLR7XlTIDvkcpASdk5XcGilopKMEvXNurx69rJcF7o+iiLW/fvCkelMVuZ1pmS2rW5yRrzf2xR2fxw6bgqKwuKXX2UVwLbAjcbwxnXwKzY4h7hiokqA7cGmoH214YqbLtQX88kvjmB2+J0PIqjUXzydnD0sZvODnqBXqEhitF7dIS+oAmaIha8DD4Hk+AkfBcuwjTMrlLDoKt5jv6JUP4G+CjYVg==</latexit>

L0(f) L�(f) + O

0

@pckVkF

���U�U0��F+

��U0��2

�q1m

Pmi=1 kxik22

�pm

+

rh

m

1

A .

<latexit sha1_base64="9menaas4xc16xwIr/F9PhcO9MdI=">AAAC5HicbVLbihMxGM6Mp9162KqX3gSL0KVYZoqgN8KisHix4Aq2u9C0QyaTacMmmdkks1hCHkDwysVbn8lbH0YwcxDrrj9k+PkO+ZI/k5acaRNFP4Pwxs1bt+/s7Pbu3rv/YK//8NFMF5UidEoKXqjTFGvKmaRTwwynp6WiWKScnqRnb2v+5IIqzQr50WxKuhB4JVnOCDYeSvq/jhIbuWG+DxGn5xCtsTlK0AoLgWtwBJFhPKPvPZubIcoVJhbpc2UscRY1+VbRzCFZKGHRBSUzlxy6Vv5HkPKKbimmz5vvMnKJPXRutMXU2AQptlqbfQjboDY0dlY4pCuRWPY6dksB/9o+JczblhMH/Zmas3dW4ZHd0fY263ob1yWMk/4gGkdNwetN3DUD0NVx0v+BsoJUgkpDONZ6HkelWVisDCOcuh6qNC0xOcMrOvetxILqhW2m4OAzj2QwL5Rf0sAG3XZYLLTeiNQrBTZrfZWrwf9x88rkrxaWybIyVJI2KK84NAWs3xxmTFFi+Ab2EJOEVxm1sjDNH+D8COKrF77ezCbjOBrHH14MDt50w9gBT8BTMAQxeAkOwDtwDKaABCT4HHwNLsM8/BJeht9aaRh0nsfgnwq//wZo6+5m</latexit><latexit sha1_base64="9menaas4xc16xwIr/F9PhcO9MdI=">AAAC5HicbVLbihMxGM6Mp9162KqX3gSL0KVYZoqgN8KisHix4Aq2u9C0QyaTacMmmdkks1hCHkDwysVbn8lbH0YwcxDrrj9k+PkO+ZI/k5acaRNFP4Pwxs1bt+/s7Pbu3rv/YK//8NFMF5UidEoKXqjTFGvKmaRTwwynp6WiWKScnqRnb2v+5IIqzQr50WxKuhB4JVnOCDYeSvq/jhIbuWG+DxGn5xCtsTlK0AoLgWtwBJFhPKPvPZubIcoVJhbpc2UscRY1+VbRzCFZKGHRBSUzlxy6Vv5HkPKKbimmz5vvMnKJPXRutMXU2AQptlqbfQjboDY0dlY4pCuRWPY6dksB/9o+JczblhMH/Zmas3dW4ZHd0fY263ob1yWMk/4gGkdNwetN3DUD0NVx0v+BsoJUgkpDONZ6HkelWVisDCOcuh6qNC0xOcMrOvetxILqhW2m4OAzj2QwL5Rf0sAG3XZYLLTeiNQrBTZrfZWrwf9x88rkrxaWybIyVJI2KK84NAWs3xxmTFFi+Ab2EJOEVxm1sjDNH+D8COKrF77ezCbjOBrHH14MDt50w9gBT8BTMAQxeAkOwDtwDKaABCT4HHwNLsM8/BJeht9aaRh0nsfgnwq//wZo6+5m</latexit><latexit sha1_base64="9menaas4xc16xwIr/F9PhcO9MdI=">AAAC5HicbVLbihMxGM6Mp9162KqX3gSL0KVYZoqgN8KisHix4Aq2u9C0QyaTacMmmdkks1hCHkDwysVbn8lbH0YwcxDrrj9k+PkO+ZI/k5acaRNFP4Pwxs1bt+/s7Pbu3rv/YK//8NFMF5UidEoKXqjTFGvKmaRTwwynp6WiWKScnqRnb2v+5IIqzQr50WxKuhB4JVnOCDYeSvq/jhIbuWG+DxGn5xCtsTlK0AoLgWtwBJFhPKPvPZubIcoVJhbpc2UscRY1+VbRzCFZKGHRBSUzlxy6Vv5HkPKKbimmz5vvMnKJPXRutMXU2AQptlqbfQjboDY0dlY4pCuRWPY6dksB/9o+JczblhMH/Zmas3dW4ZHd0fY263ob1yWMk/4gGkdNwetN3DUD0NVx0v+BsoJUgkpDONZ6HkelWVisDCOcuh6qNC0xOcMrOvetxILqhW2m4OAzj2QwL5Rf0sAG3XZYLLTeiNQrBTZrfZWrwf9x88rkrxaWybIyVJI2KK84NAWs3xxmTFFi+Ab2EJOEVxm1sjDNH+D8COKrF77ezCbjOBrHH14MDt50w9gBT8BTMAQxeAkOwDtwDKaABCT4HHwNLsM8/BJeht9aaRh0nsfgnwq//wZo6+5m</latexit><latexit sha1_base64="9menaas4xc16xwIr/F9PhcO9MdI=">AAAC5HicbVLbihMxGM6Mp9162KqX3gSL0KVYZoqgN8KisHix4Aq2u9C0QyaTacMmmdkks1hCHkDwysVbn8lbH0YwcxDrrj9k+PkO+ZI/k5acaRNFP4Pwxs1bt+/s7Pbu3rv/YK//8NFMF5UidEoKXqjTFGvKmaRTwwynp6WiWKScnqRnb2v+5IIqzQr50WxKuhB4JVnOCDYeSvq/jhIbuWG+DxGn5xCtsTlK0AoLgWtwBJFhPKPvPZubIcoVJhbpc2UscRY1+VbRzCFZKGHRBSUzlxy6Vv5HkPKKbimmz5vvMnKJPXRutMXU2AQptlqbfQjboDY0dlY4pCuRWPY6dksB/9o+JczblhMH/Zmas3dW4ZHd0fY263ob1yWMk/4gGkdNwetN3DUD0NVx0v+BsoJUgkpDONZ6HkelWVisDCOcuh6qNC0xOcMrOvetxILqhW2m4OAzj2QwL5Rf0sAG3XZYLLTeiNQrBTZrfZWrwf9x88rkrxaWybIyVJI2KK84NAWs3xxmTFFi+Ab2EJOEVxm1sjDNH+D8COKrF77ezCbjOBrHH14MDt50w9gBT8BTMAQxeAkOwDtwDKaABCT4HHwNLsM8/BJeht9aaRh0nsfgnwq//wZo6+5m</latexit>