ON EQUIVALENCE OF SOME BASIC PRINCIPLES IN VARIATIONALANALYSISJONATHAN M. BORWEIN1, BORIS S. MORDUKHOVICH2 and YONGHENG SHAO3Abstract. The primary goal of this paper is to study relationships between certain basicprinciples of variational analysis and its applications to nonsmooth calculus and optimiza-tion. Considering a broad class of Banach spaces admitting smooth renorms with respect tosome bornology, we establish an equivalence between useful versions of a smooth variationalprinciple for lower semicontinuous functions, an extremal principle for nonconvex sets, andan enhanced fuzzy sum rule formulated in terms of viscosity normals and subgradients withcontrolled ranks. Further re�nements of the equivalence result are obtained in the case ofa Fr�echet di�erentiable renorm. Based on the new enhanced sum rule, we provide a simpli-�ed proof for the re�ned sequential description of approximate normals and subgradients insmooth spaces.1991 Mathematical Subject Classi�cation. Primary 49J52; Secondary 58C20, 46B20.Key words and phrases. Nonsmooth analysis, smooth Banach spaces, variational andextremal principles, generalized di�erentiation, fuzzy calculus, viscosity normals and subd-i�erentials.1Department of Mathematics and Statistics, Simon Fraser University, Burnaby, BC V5A 1S6, Canada(jborwein@cecm.sfu.ca). Research was partly supported by NSERC and by the Shrum Endowment at SimonFraser University.2Department of Mathematics, Wayne State University, Detroit, MI 48202, USA (boris@math.wayne.edu).Research was partly supported by NSF under grants DMS-9404128 and DMS-9704751 and by an NSERCForeign Researcher Award.3Department of Mathematics and Statistics, Simon Fraser University, Burnaby, BC V5A 1S6, Canada(shao@cecm.sfu.ca). 1

1. INTRODUCTIONThe paper concerns with some basic principles of variational analysis in Banach spaces.Variational analysis has been recognized as a fruitful area in mathematics that, on onehand, focuses on the study of optimization and related problems and, on the other hand,applies optimization, perturbation, and approximation ideas to the analysis of a broad rangeof problems which may not be of a variational nature. We refer the reader to the bookof Rockafellar and Wets [34] for a systematic exposition of variational analysis in �nitedimensions.In this paper we deal with a large class of Banach spaces admitting a smooth renorm ofany kind. For such spaces, fundamental tools of variational analysis and its applications arerelated to so-called smooth variational principles. Roughly speaking, the latter mean thatfor a given bounded below lower semicontinuous function there exists a smooth function,small in a certain sense, such that the sum of these two functions attains a local minimumat some point. The �rst smooth variational principle for general Banach spaces with smoothrenorms was obtained by Borwein and Preiss [4]. In [5, 8, 11, 12, 15, 23, 32], one can �ndrecent extensions and variants of this principle with further discussions and applications.Another fruitful approach to nonsmooth calculus and optimization consists of using so-called extremal principles that can be viewed as extremal extensions of the classical sepa-ration theorem to nonconvex sets. This approach goes back to the beginning of dual-spacemethods in �nite-dimensional variational analysis and directly provides necessary optimalityconditions for nonsmooth constrained problems in terms of nonconvex normal cones andsubdi�erentials; see [25{27]. In in�nite dimensions, a version of the extremal principle interms of Fr�echet "-normals was �rst established by Kruger and Mordukhovich [22] for Ba-nach spaces with Fr�echet di�erentiable renorms. It was recently proved in [29] that thisversion is equivalent to the one in terms of Fr�echet normals, holds in any Asplund space,and completely characterizes the Asplund property of a Banach space. Various variants andapplications of the extremal principle can be found in [3, 5, 19, 21, 26{31, 34] and theirreferences.The third basic principle of variational analysis we address in this paper is related tosubdi�erential fuzzy sum rules. Results in this vein were �rst obtained by Io�e for Dini andFr�echet "-subdi�erentials in corresponding smoothable spaces and their exact counterpartsin �nite dimensions; see [16, 17]. Note that we focus here not on a \full" fuzzy formulafor subdi�erentials of sums but on its special case concerning minimum points of a sum oftwo functions. In contrast to full fuzzy calculus, the latter result (sometimes called \zero"or \basic" fuzzy sum rule) is now known for most useful subdi�erentials in correspondingBanach spaces; see [2, 5, 10, 14, 19, 20, 24, 31] for recent results, further references, andapplications.The main intention of this paper is to establish an equivalence between the mentionedbasic principles in variational analysis under appropriate conditions. In has been alreadyproved by Mordukhovich and Shao [29] that the extremal principle, formulated in terms2

of Fr�echet normals, is equivalent to the fuzzy sum rule for Fr�echet subdi�erentials in thecase of Asplund spaces. Now we are going to verify that these rules actually transpire to beequivalent to a variant of the smooth variational principle in spaces with Fr�echet di�erentiablerenorms.To obtain the equivalence result in more general settings, we need to employ appro-priate constructions of normals and subgradients in non-Fr�echet-smooth spaces. A naturalgeneralization of the Fr�echet subdi�erential to smoothable spaces with respect to a weakerbornology � is the concept of viscosity �-subdi�erentials [6, 11] that goes back to the theoryof viscosity solutions of Hamilton-Jacobi equations [9]. However, it turns out that viscosity�-subdi�erentials and corresponding normal cones are too large to �t our purposes. A propernarrowing of these constructions was introduced by Borwein and Io�e [2] as �-normals and�-subdi�erentials of controlled rank (in the sense of viscosity) to provide re�ned representa-tions for the approximate G-normal cone and G-subdi�erential [18] in spaces with �-smoothrenorms. In what follows we use slight modi�cations of the subdi�erential constructionsin [2] to establish an equivalence between the smooth variational principle and appropriateversions of the extremal principle and fuzzy sum rule in smoothable Banach spaces.In this way we obtain new enhanced versions of the extremal principle and fuzzy sumrule in �-smooth spaces that are of independent interest for bornologies weaker than theFr�echet one. As an application of the enhanced fuzzy sum rule, we provide a new proof ofthe main representation results in [2] that is a substantial simpli�cation and clari�cation ofthe original one.The paper is organized as follows. In Section 2 we study the constructions of �-normalsand �-subdi�erentials with controlled ranks. After providing the requisite preliminary mate-rial, we establish relationships between these constructions based on a new �-smooth versionof the implicit function theorem. Section 3 is devoted to the main equivalence results andrelated discussions. In the concluding Section 4 we present some applications.Our notation is standard. For any Banach space X we denote its norm by k � k and thedual space by X? with the canonical pairing h�; �i. As usual, B and B? stand for the unitclosed balls in the space and dual space in question. The symbol Br(x) denotes the closedball with center x and radius r; cl? signi�es the weak-star topological closure in X?. In thispaper we consider various multifunctions � fromX into the dual spaceX?. For such objects,the expression lim supx!�x �(x)always connotes the sequential Kuratowski-Painlev�e upper limit with respect to the normtopology in X and the weak-star topology in X?, i.e.,lim supx!�x �(x) := fx? 2 X?j 9 sequences xk ! �x and x?k w?! x?with x?k 2 �(xk) for all k = 1; 2 : : :g:If ' : X ! �R := [�1;1] is an extended-real-valued function, then, as usual,dom ' := fx 2 X with j'(x)j <1g; epi ' := f(x; �) 2 X �Rj � � '(x)g:3

In this case, lim sup'(x) and lim inf '(x) denote the upper and lower limits of such (scalar)functions in the classical sense. Depending on context, the symbols x '! �x and x ! �x mean,respectively, that x! �x with '(x)! '(�x) and x! �x with x 2 .2. NORMALS AND SUBDIFFERENTIALS OF CONTROLLED RANKLet X be a real Banach space. Recall that a bornology � on X is a family of bounded andcentrally symmetric subsets of X whose union is X, which is closed under multiplication bypositive scalars and is directed upwards (i.e., the union of any two members of � is containedin some member of �). The most important bornologies are those formed by all boundedsets (the Fr�echet bornology), weak compact sets (the weak Hadamard bornology), compactsets (the Hadamard bornology), and �nite sets (the Gateaux bornology); see [4, 12, 32] formore details and references.Let us denote by X?� the dual space X? of X endowed with the topology of uniformconvergence on �-sets. It is well known that the latter convergence agrees with the normconvergence in X? when � is the Fr�echet bornology, and with the weak-star convergencein X? when � is the Gateaux bornology. In the cases of Hadamard and weak Hadamardbornologies,X?� corresponds to X? endowed with the bounded weak-star and Mackey topolo-gies, respectively.Every bornology generates a certain concept of di�erentiability. Given an extended-real-valued function ' on X is said to be �-di�erentiable at x 2 dom ' with �-derivativer�'(x) 2 X? provided thatt�1('(x+ tv) � '(x)� thr�'(x); vi) ! 0as t ! 0 uniformly in v 2 V for every V 2 �. We say that ' is �-smooth around x if it is�-di�erentiable at each point of a neighborhood of x and r�' : X ! X?� is continuous onthis neighborhood. Clearly the �-smoothness of ' implies that r�' : X ! X? is continuousaround x with respect to the norm topology on X and the weak-star topology on X?. When� = F is the Fr�echet bornology, one has more: rF' : X ! X? is norm-to-norm continuousaround x. This is an essential speci�c characteristic of the Fr�echet case used in the sequel.Now we de�ne the main constructions of generalized di�erentiation considered in thispaper.2.1. De�nition. (i) Let be a nonempty subset of the Banach space X. Given x 2 cl ,we say that x? is a �-normal of rank � > 0 to at x if there exist a neighborhood U of x anda �-smooth function g : U ! R such that g is Lipschitz continuous on U with modulus �,g(u) � 0 for all u 2 \U , g(x) = 0, and x? = r�g(x). The collection of all these �-normalsof controlled rank is denoted by N�� (x; ).(ii) Let ' be an extended-real-valued function on X. Its (viscosity) �-subdi�erentialof rank � at x 2 dom ' is the set @��'(x) of all x? 2 X? with the following properties:4

there exist a neighborhood U of x and a �-smooth function g : U ! R such that g is Lips-chitz continuous on U with modulus �, r�g(x) = x?, and '�g attains a local minimum at x.Both notions in De�nition 2.1 are slight modi�cations of the corresponding de�nitionsproposed in [2] where the concept of �-smoothness does not require that r�g : X ! X?�is continuous around x. The given continuity requirement seems to be important for someapplications including those in this paper.One can see that the union @�'(x) := [�>0 @��'(x) (2.1)reduces to the viscosity �-subdi�erential of ' at x de�ned in [6] as a modi�cation of the corre-sponding de�nition in [11] where g is not required to be Lipschitzian. The latter requirementis essential for bornologies weaker than the Fr�echet bornology since the �-smoothness of gmay not imply its Lipschitz continuity. This never happens in the case of � = F due tothe required norm-to-norm continuity of rFg : X ! X? that automatically ensures theLipschitz continuity of g. Actually the usage of controlled ranks in the Fr�echet case does notprovide any advantages in comparison with the full viscosity subdi�erential (2.1); see belowProposition 3.6 and related discussions. On the contrary, the controlled rank narrowing ofthe viscosity �-subdi�erential is a crucial tool of our analysis in the general non-Fr�echet case.In this paper we need to consider �-subdi�erentials of controlled rank for lower semicon-tinuous (l.s.c.) functions. It follows from De�nition 2.1(ii) that we may always assume thatg(x) = '(x). Also one can easily see that@��'(x; y) = (@��'1(x); @��'2(y)) for '(x; y) = '1(x) + '2(y): (2.2)As it was observed in [2], for the function g in De�nition 2.1(i) one hasg(u) � � dist(u; ) for all u 2 Uwhere dist(�; ) signi�es the distance function associated with . The latter inequalityallows us to derive the relationshipN�� (x; ) = @��f� dist(x; )g = �@1� dist(x; ) for all x 2 (2.3)used in the sequel. On the other hand, one always has the relationshipN�� (x; ) = @���(x; ) for all x 2 (2.4)involving the indicator function �(�; ) of which equals 0 if x 2 and 1 otherwise. Notethat the union N�(x; ) := [�>0N�� (x; ) (2.5)5

is a cone, called the �-normal cone to at x, which relates to the �-subdi�erential (2.1) asin (2.4).It easy follows from (2.2) and (2.4) thatN�� ((x1; x2); 1 �2) = N�� (x1; 1) �N�� (x2; 2) for any x1 2 1 � X1 and x2 2 2 � X2:For the proof of the main result in Section 3 we need more subtle relationships between�-normals and �-subdi�erentials of controlled rank that involve epigraphs of l.s.c. functions.We establish them below in Proposition 2.3 based on the following �-smooth version of theimplicit function theorem that is certainly of some independent interest.2.2. Theorem. Assume that X is a Banach space and X�R is equipped with a norm whichrestricts to the original norm on the subspace X. Suppose further that W is a neighborhoodof (x0; �0) 2 X �R and f : W ! R is �-smooth with the �-derivative r�f = (r�xf;r��f)and Lipschitz continuous with modulus . Let f(x0; �0) = 0 and r��f(x0; �0) 6= 0. Then forany � > there exist a neighborhood U of x0, a positive number �, and a unique �-smoothand Lipschitz continuous function g : U ! [�0 � �; �0 + �] with modulus �=jr��f(x0; �0)j,such that f(x; g(x)) = 0 for all x 2 U andr�g(x) = �(r��f(x; g(x)))�1r�xf(x; g(x)) for all x 2 U: (2.6)Proof. Let '(x; �) := �� �0 � f(x; �)r��f(x0; �0) for all (x; �) 2W:Due to the assumptions made, ' is Lipschitz continuous on W satisfying'(x0; �0) = 0 and r��'(x0; �0) = 0: (2.7)Since f :W ! R is �-smooth, r�f : W ! X?��R is continuous that implies the continuityof the partial derivative r��f on W . Using the second equality in (2.7), one can �nd aneighborhood U1 of x0 and a number � > 0 such that U1 � [�0 � �; �0 + �] �W andjr��'(x; �)j < l for all x 2 U1 and j� � �0j � �where 0 < l < 1� ( =�) is a constant. Now let us construct a sequence fgng of functions onU asg1(x) := �0 + '(x; �0) and gn(x) := �0 + '(x; gn�1(x)) for all x 2 U and n = 1; 2; : : : :(2.8)First we justify by induction that there is a neighborhood U � U1 of x0 such thatjgn(x)� �0j < � for all x 2 U and n = 1; 2; : : : : (2.9)Using the �rst equality in (2.7) and continuity of ' at (x0; �0), we �nd a neighborhoodU � U1 of x0 with j'(x; �0)j < (1 � l)� on U . This gives jg1(x) � �0j < � for all x 2 U .6

Now assuming that jgn�1(x)� �0j < � for all x 2 U and employing the classical mean valuetheorem, one hasjgn(x)� �0j = j'(x; gn�1(x))j � j'(x; gn�1(x))� '(x; �0)j + j'(x; �0)j= jr��'(x; �1)(gn�1(x)� �0)j + j'(x; �0)j (�1 lies between �0 and gn�1(x))< l� + (1� l)� = �:In this way we get (2.9) for all n by induction.Next let us show that fgng converges uniformly in U . Again using the mean valuetheorem, we getjgn+1(x) � gn(x)j = j'(x; gn(x))� '(x; gn�1(x))j= jr��'(x; �2)(gn(x)� gn�1(x))j (�2 lies between gn�1(x) and gn(x))< ljgn(x) � gn�1(x))j for all x 2 U and n = 1; 2; : : : :Therefore, one hasjgn+1(x) � gn(x)j < ljgn(x) � gn�1(x))j < � � � < lnjg1(x) � �j< lnj'(x; �0)j < ln(1� )� for all x 2 U and n = 1; 2; : : :that yields the uniform convergence of fgng in U . Setting g(x) := limn!1 gn(x) and passingto the limit in (2.8), we conclude that g(x) = �0 + '(x; g(x)) for all x 2 U . It follows from(2.9) that �0 � � � g(x) � �0 + � and, moreover, g(x0) = �0 since gn(x0) = �0 for alln = 1; 2; : : :.Now we verify that g is Lipschitz continuous on U with modulus �=jr��'(x0; �0)j. Tothis end we observe that the function ' de�ned above is Lipschitz continuous in x withmodulus =jr��'(x0; �0)j on U . Picking any x1; x2 2 U and employing again the meanvalue theorem, one hasjg(x1)� g(x2)j � j'(x1; g(x1))� '(x2; g(x1))j + j'(x2; g(x1))� '(x2; g(x2))j� ( =jr��'(x0; �0)j)kx1 � x2k+ jr��'(x2; �3)j � jg(x1)� g(x2)j(�3 lies between g(x1) and g(x2))< ( =jr��'(x0; �0)j)kx1 � x2k+ ljg(x1)� g(x2)j:This gives jg(x1) � g(x2)j � (�=jr��'(x0; �0)j)kx1 � x2k and justi�es the desired Lipschitzproperty.Next let us show that � = g(x) is a unique solution to the equation � = �0 + '(x; �) asx 2 U . Arguing by contradiction, we assume that there exist two solutions g and h on Usatisfying �0 � � � g(x) � �0 + � and �0 � � � h(x) � �0 + �:7

Then one hasjg(x) � h(x)j = j[�0 + '(x; g(x))] � [�0 + '(x; h(x))]j = j'(x; g(x)) � '(x; h(x))j� jr��'(x; �4)j � jg(x) � h(x)j � ljg(x) � h(x)j(�4 lies between g(x) and h(x))that is a contradiction unless g(x) = h(x) for all x 2 U .Finally we prove that g is �-smooth on U and (2.6) holds. Taking arbitrary � > 0,(x; �) 2 W , and V 2 � with 0 2 V � X and using the �-di�erentiability of f on W , we �nd� > 0 such thatjf(x + h; � + �)� f(x; �) �r�x'(x; �)h �r��'(x; �)�j � �k(h; �)k (2.10)for any h 2 V and k(h; �)k � �. Let us substitute � = g(x) and � = g(x + h) � g(x) into(2.10) and remember that f(x + h; g(x + h)) = f(x; g(x)) = 0 whenever x 2 U and h issu�ciently small. Then (2.10) yieldsjr��f(x; �)j � �����g(x+ h)� g(x) + r�xf(x; g(x))hr��f(x; g(x)) �����= jr�xf(x; g(x))h +r��f(x; g(x))[g(x + h)� g(x)]j= jf(x+ h; g(x + h)) � f(x; g(x)) +r��f(x; g(x))[g(x + h) � g(x)]j� �k(h; jg(x + h) � g(x)j)k � �k(h; (�=jr��f(x0; �0)j)khk)kfor all x 2 U and small h 2 V . The latter implies that g is �-di�erentiable at each pointx 2 U and (2.6) holds. The continuity of r�g : U ! X?� follows directly from (2.6) due tothe �-smoothness of f . This is all we need to prove in the theorem. 2Nowwe are ready to establish important relationships between �-normals and �-subdi�erentialsof controlled rank. Part (ii) of the following proposition provides a substantial impact to theproof of the main theorem in the next section.2.3. Proposition. Let X be a Banach space and let ' : X ! (�1;1] be l.s.c. aroundx 2 dom '. Then one has:(i) @��'(x) � fx? 2 X?j (x?;�1) 2 N � ((x; '(x)); epi ')g where := maxf�; 1g andX �R is equipped with the norm k(x; r)k := kxk+ jrj.(ii) fx? 2 X?j (x?;�1) 2 N � ((x; '(x)); epi ')g � @��'(x) for any � > , whenever X�Ris equipped with a norm which restricts to the original norm on the subspace X.Proof. (i) Let x? 2 @��'(x) for any given � > 0. According to De�nition 2.1(ii) we �nd aneighborhood U of x and a �-smooth function g : U ! R such that g is Lipschitz continuouson U with modulus �, g(x) = '(x), x? = r�g(x), and '(u)� g(u) attains a local minimumat x. Considering the functionm(u; �) := g(u) � � for u 2 U and � 2 R;8

one can easily observe that m is Lipschitz continuous around (x; '(x)) with modulus =f�; 1g. Moreover, m is �-smooth around (x; '(x)) and one has (x?;�1) = r�m(x; '(x)).Furthermore, it is evident that m(u; �) � 0 for all (u; �) 2 (epi ') \ (U � R). Due toDe�nition 2.1(i) we arrive at the inclusion (x?;�1) 2 N � ((x; '(x)); epi ') which completesthe proof of part (i) of the proposition.To prove (ii), we assume that (x?;�1) 2 N � ((x; '(x)); epi ') for any given > 0. Byde�nition there are a neighborhood W of (x; '(x)) and a �-smooth function f : W ! Rsuch that f is Lipschitz continuous on W with modulus and one has the properties:(a) f(u; �) � 0 if (u; �) 2 W \ epi f ;(b) f(x; '(x)) = 0;(c) x? = r�xf(x; '(x)) and �1 = r��f(x; '(x)).Now taking any � > and employing Theorem 2.2, we �nd a neighborhood U of x, apositive number �, and a �-smooth function g : U ! ['(x) � �; '(x) + �] such that g isLipschitz continuous on U with modulus �, g(x) = '(x), x? = r�g(x), and f(u; g(u)) = 0for all u 2 U .Since r��f(x; '(x)) = �1 and f is �-smooth around (x; '(x)), we may assume thatf(u; �) is strictly decreasing in � near '(x) for each u 2 U . This yieldsg(u) = supf�j f(u; �) > 0g 8u 2 U: (2.11)It follows from condition (a) that '(u) � � if f(u; �) > 0. The latter implies g(u) � '(u)for all u 2 U which is the last step to justify x? 2 @��'(x). This completes the proof of theproposition. 2 3. MAIN RESULTSIn this section we establish the main results of the paper providing an equivalence betweenappropriate versions of the smooth variational principle, the extremal principle, and the fuzzysum rule in Banach spaces with bornologically smooth renorms. Given a bornology � onthe Banach space X, we say that X is �-smooth if it admits an equivalent norm whichis �-di�erentiable away from the origin. It is well known that many spaces important forapplications to optimization and variational analysis happen to be �-smooth with respect tosome bornology. In particular, every re exive space is Fr�echet-smooth and every separablespace is Hadamard-smooth (equivalently Gateaux-smooth). Considering further �-smoothBanach spaces, we always deal with �-di�erentiable norms on them.To formulate the principal equivalence result below, we need to recall the concept of setextremality introduced in [22]. Given closed sets 1 and 2 with a common point �x 2 1\2in a Banach space X, we say that �x is a locally extremal point of the set system f1;2gif for any " > 0 there are b 2 X with kbk < " and a neighborhood U of �x such that(1 + b) \ 2 \ U = ;. 9

It is clear that any boundary point �x of a closed set is a locally extremal point of thepair f; �xg. There are also close connections between extremality and separability of set sys-tems. Furthermore, the given geometric concept of extremality covers conventional notionsof optimal solutions to general constrained problems in both scalar and vector optimization;see [3, 21, 22, 26{29] for various examples, discussions, and more references. By \extremalprinciple" we mean necessary conditions for local extremal points of set systems which canbe treated as generalized Euler equations in an abstract geometric setting and provide aproper extremal analog of the separation theorem for nonconvex sets. In the main result asfollows we present a variant of the extremal principle in �-smooth spaces which is provedto be equivalent to appropriate versions of both smooth variational principle and enhancedfuzzy sum rule.3.1. Theorem. The following principles hold in any �-smooth spaces X and are equivalentto each other:(i) (smooth variational principle) Let f : X ! (�1;1] be a l.s.c. function bounded frombelow. Suppose that " > 0 and �x 2 X are given satisfyingf(�x) < infX f + ": (3.1)Then for any > 0 one can �nd x0 2 X and a function g : X ! (�1;1], which is�-smooth and Lipschitz continuous around x0 with modulus � = "= , such that(a) kx0 � �xk < ;(b) f(x0) < infX f + ";(c) f + g attains a local minimum at x0.(ii) (extremal principle) Let �x 2 1 \2 � X be a locally extremal point of the closed setsystem f1;2g. Then for any positive numbers ", �, and � there exist xi 2 i \ B"(�x) andx?i 2 N�� (xi; i) + "B? with � = (� + �)=2, i = 1; 2, such thatkx?1k+ kx?2k = � and x?1 + x?2 = 0: (3.2)(iii) (enhanced fuzzy sum rule) Let 'i : X ! �R; i = 1; 2, be l.s.c. functions �nite at �x.Suppose that '1 is Lipschitz continuous around �x with modulus l and that '1 + '2 attainsa local minimum at �x. Then for any " > 0, � > 0, and � > l there exist xi 2 B�(�x) withj'i(xi)� 'i(�x)j � �; i = 1; 2, such that0 2 @��'1(x1) + @��'2(x2) + "B?: (3.3)Proof. The smooth variational principle in (i) follows from the variational principle ofBorwein and Preiss [4] in any �-smooth space. Our goal is to show that (i) is equivalent toeach of the other principles (ii) and (iii) with the same type of bornology. Let us prove thetheorem following the scheme (i)=)(ii)=)(iii)=)(i).10

First we show that the smooth variational principle in (i) implies the extremal principlein (ii). Let �x be a locally extremal point of the set system f1;2g. According to thede�nition, for any positive numbers �, �, and " given in (ii) we take 0 < "0 < minf"; �g=2and select b 2 X such that�kbk < 2("0)2 and (1 + b) \ 2 \ U = ;for some neighborhood U of �x. On localizing, one may always assume that U = X. Thusconsidering a �-smooth norm k � k on X and setting the function'(u; v) := (�=2)ku� v + bk; (3.4)we conclude that '(u; v) > 0 for any u 2 1 and v 2 2 with '(�x; �x) < ("0)2.Let us endow the Cartesian productX�X with the norm k(u; v)k := (kuk2+kvk2) 12 . Thisnorm is obviously di�erentiable away from the origin with respect to the product bornologyon X �X generated by �. Using the same symbol � for this product bornology, we see thatthe Banach space X �X is �-smooth. Since the smooth variational principle of (i) holds inany bornologically smooth space, we can apply it to the extended-real-valued functionf(u; v) := '(u; v) + �((u; v); 1 �2)on X � X involving the indicator function �(�; 1 � 2) of the set 1 � 2. Taking intoaccount that f(�x; �x) < infX�X f + ("0)2and employing (i) with the localization constant = "0, we �nd a point (x1; x2) 2 1 � 2,a convex neighborhood W of (x1; x2), and a �-smooth function � : W ! R such thatkxi � �xk < "0, � is Lipschitz continuous on W with modulus "0 < �=2, and f +� attains alocal minimum at (x1; x2). Due to "0 < "=2 one getskr��(x1; x2)k � "0 < "=2 and kxi � �xk < "=2 for i = 1; 2: (3.5)Let us further observe that since the norm function (3.4) never vanishes on 1 � 2, itis �-di�erentiable at the point (x1; x2) and its neighborhood W and, moreover,r�'(x1; x2) = (�=2)(~x?;�~x?) for ~x? 2 X? with k~x?k = 1: (3.6)Taking into account that ' is convex on W , we conclude that ' is �-smooth around (x1; x2).Therefore, the sum function ' + � is �-smooth around (x1; x2) and Lipschitz continuouson W with modulus � = (� + �)=2 generated by modulus �=2 for ' in (3.4) and modulus"0 < �=2 for � from the above. Moreover, the function('(u; v) + �(u; v)) + �((u; v); 1 �2)11

attains a local minimum at (x1; x2). Without loss of generality one can always assume that'(x1; x2)+�(x1; x2) = 0. By virtue of De�nition 2.1(ii) and relationships (2.4) and (2.2) wehave r�(�'��)(x1; x2) 2 N�� ((x1; x2); 1 �2) (3.7)= N�� (x1; 1)�N�� (x2; 2):Letting x?1 := �(�=2)~x? and x?2 := (�=2)~x?, we obtain from (3.5){(3.7) thatx?i 2 N�� (xi; i) + "B?; i = 1; 2:This gives (3.2) and ends the proof of (i) ) (ii).Next we are going to prove that (ii)=)(iii). Given numbers l and � in (iii), we pick0 < a < 1 with a� > l and denote p := a�� l. Now let us de�ne a norm on X �R byk(x; r)k := (kxk2 + p�2r2) 12 : (3.8)Clearly this norm satis�es the norm assumption in Proposition 2.3(ii). Moreover, the dualspace (X�R)? can be identi�ed with X?�R by the pairing h(x?; r?); (x; r)i := hx?; xi+r? �rthat generates the norm k(x?; r?)k = (kx?k2 + p2(r?)2) 12 (3.9)on (X �R)? essential in the sequel.Fix numbers " > 0 and � > 0 in (iii) and suppose, without loss of generality, that '1 isLipschitz continuous on B�(�x) with modulus l. For the remainder we choose a number � > 0satisfying � < min f 38(a�+ 1) ; 1� a2a(1 + �) + 2 ; "p8a�(p+ 8a�) ; �; �pg: (3.10)Assume for simplicity that �x = 0 is a local minimizer for '1+'2 with '1(0) = '2(0) = 0.We de�ne closed sets 1 and 2 as1 := epi '1 and 2 := f(x; �) 2 X �Rj '2(x) � ��g:Then one can easily check that (0; 0) is a locally extremal point of the set system f1;2gin the space X �R. Now we are going to apply the extremal principle in (ii) to this system.First we observe that if k � k is a �-smooth norm on X, then (3.8) provides a smooth normon X �R with respect to the bornology consisting of all sets V � [�s; s] where V 2 � ands > 0. Since the extremal principle of (ii) holds in an arbitrary bornologically smooth space,we apply it to the given system with � chosed in (3.10), � = 1, and � = 2�. This gives us(xi; �i) 2 i, x?i 2 X?, and �i 2 R such thatk(xi; �i)k � � for i = 1; 2; (3.11)12

1=2� � � k(x?1; �1)k � 1=2 + �; 1=2� � � k(x?2; �2)k � 1=2 + �; (3.12)(x?1;��1) 2 N 12+�� ((x1; �1)); 1); (�x?2; �2) 2 N 12+�� ((x2; �2)); 2); (3.13)and k(x?1;��1) + (�x?2; �2)k � �: (3.14)It easily follows from (3.13) and the structure of i that �i � 0 for i = 1; 2. Our goalis to prove that actually �i > 0 and, moreover, �1 = '1(x1) and �2 = �'2(x2). To furnishthis, we employ De�nition 2.1(i) in (3.13) and �nd a neighborhood U of (x1; �1) and a �-smooth function g : U ! R such that g(x1; �1) = 0, g is Lipschitz continuous on U withmodulus 12 + �, g(x; �) � 0 for all (x; �) 2 1 \ U , and r�g(x1; �1) = (x?1;��1). The latterimplies that for any � > 0 and (h; l) 2 X �R with khk = 1 there exists t0 > 0 such thatg(x1 + th; �1 + tl)t � h(x?1;��1); (h; l)i > �� (3.15)whenever 0 < t < t0, where l is the given Lipschitz modulus of '1 in (iii). Since'1(x1 + th) � lt + �1;one has g(x1 + th; �1 + tl) � 0 whenever t is su�ciently small. This fact and (3.13) yieldhx?1; hi < l�1 + � for all h 2 X with khk = 1that gives hx?1; hi � l�1 for all such h since � > 0 is arbitrary. In this way we arrive atkx?1k � l�1: (3.16)Now using (3.9), (3.16), and the de�nition of p above, one hask(x?1; �1)k � l�1 + p�1= l�1 + (a�� l)�1 = a��1:The latter implies the estimates�1 � (12 � �) 1a� > 18a� > 0 (3.17)due to the �rst inequality in (3.12) with � < 3=8 in (3.10). Moreover, from (3.14), (3.17),and the choice of � < 3=8(a�+ 1) in (3.10) we get the following estimates for �2:�2 � �1 � � � (12 � �) 1a� � � > 18a� > 0: (3.18)Now we are able to show that �1 = '1(x1) and �2 = �'2(x2). Let us do it only forthe �rst case since the other one is symmetric. Assuming that �1 > '1(x1) and picking any� > 0, one can �nd, similarly to (3.15), a positive number t1 < (�1 � '1(x1))=2 such thatg(x1; �1 � t)t � h(x?1;��1); (0;�1)i > �� (3.19)13

whenever 0 < t < t1. Taking into account that '1(x1) < �1 � t, we get g(x1; �1 � t) � 0if t is su�ciently small. Then (3.19) implies �1 < � that contradicts (3.17). Therefore, theinclusions in (3.13) can be written as follows(x?1=�1;�1) 2 N ( 12+�)=�1� ((x1; '1(x1)); epi '1); (3.20)(�x?2=�2;�1) 2 N ( 12+�)=�2� ((x2; '2(x2)); epi '2) (3.21)due to the structure of the sets i; i = 1; 2, and the de�nition of �-normals with controlledrank. Denote by := max f(12 + �)=�1; (12 + �)=�2g (3.22)and observe that 1�i � 1(12 � �) 1a� � � for i = 1; 2by virtue of (3.17) and (3.18). Therefore, one has the estimates � (1 + 2�)a�1� 2�(1 + a�) < � (3.23)since � < (1 � a)=(2a(1 + �) + 2) in (3.10). Now empoying Proposition 2.3(ii), we deducefrom (3.20){(3.23) that~x?1 := x?1=�1 2 @��'1(x1) and ~x?2 := �x?2=�2 2 @��'2(x2): (3.24)To establish (3.3), we need to estimate k~x?1 + ~x?2k. One immediately hask~x?1 + ~x?2k = kx?1�1 � x?2�2 k � kx?1 � x?2k�1 + j�2 � �1j�1�2 � kx?2k: (3.25)To proceed, let us �rst observe that kx?2k < 1 due to the last inequality in (3.12) and � < 3=8in (3.10). Furthermore, it follows from (3.14) and the form of dual norm (3.9) thatkx?1 � x?2k � � and j�2 � �1j � �=p: (3.26)Now using (3.17), (3.18), (3.25), (3.26), and the estimate� < "p8a�(p+ 8a�)in (3.10), we conclude that k~x?1 + ~x?2k < ". This justi�es the fuzzy sum rule inclusion (3.3).To �nish the proof of (ii)=)(iii), it remains to show thatkxi � �xk � � and j'i(xi)� 'i(�x)j � � for i = 1; 2 (3.27)14

where � > 0 is given in (iii), �x = 0, and '1(0) = '2(0) = 0. Using the norm de�nition in(3.8) and the choice of � < minf�; �=pg in (3.10), we deduce (3.27) directly from (3.11) withj�ij = j'i(xi)j for i = 1; 2.Finally let us prove the implication (iii)=)(i) in Theorem 3.1. Given " and �x satisfying(3.1), one always has � 2 (0; ") such thatf(�x) < infX f + ("� �): (3.28)Now we employ Ekeland's variational principle [13] in (3.28) with the localization constant given in (i). According to [13] there exists x1 2 X satisfying the propertiesf(x1) < infX f + ("� �); kx1 � �xk < ; and (3.29)f(x1) < f(x) + (" � �) kx� x1k for all x 2 X with x 6= x1: (3.30)Observe that the function '(x) := (" � �) kx� x1kis Lipschitz continuous on X with modulusl := "� � < " (3.31)and the sum f + ' attains its minimum at x1. Employing the enhanced sum rule (iii) with0 < � < minf � kx1 � �xk; �g; (3.32)we �nd x0 2 X such thatkx0 � x1k � �; jf(x0)� f(x1j � �; and (3.33)@��f(x0) 6= ; with � := "= (3.34)since � > l in (3.31). It follows from (3.29), (3.33), and the choice of � in (3.32) thatkx0 � �xk � kx1 � �xk+ � < andf(x0) � f(x1) + � < infX f + ";i.e., one gets both properties (a) and (b) in (i). Furthermore, using (3.34) and applyingDe�nition 2.1(ii) to �x? 2 @��f(x0), we have a function g : X ! (�1;1], which is �-smooth and Lipschitz continuous around x0 with modulus � = "= , such that (c) holds.15

This justi�es the implication (iii)=)(i) and completes the proof of the theorem. 23.2. Remark. Theorem 3.1 establishes that the three basic principles under considerationare equivalent in the class of all bornologically smooth Banach spaces. Given a Banach spaceX smooth with respect to the underlying bornology �, we actually prove the following:(a) the smooth variational principle (i) in X �X equipped with the product bornologyimplies the extremal principle (ii) in X;(b) the extremal principle in X � R equipped with the product bornology implies theenhanced fuzzy sum rule (iii) in X;(c) the exhanced fuzzy sum rule (iii) in X implies the smooth variational principle (i) inthe same space X.Since X �R is a subspace of X2, we can conclude that the three principles in Theorem 3.1are equivalent being considered in product spaces X2n for all n = 1; 2; : : :.3.3. Remark. One can observe that the proof of (i)=)(ii) in Theorem 3.1 does not useproperty (b) in (i). On the other hand, we verify the implications (ii)=)(iii)=)(i) includingthis property. Therefore, the smooth variational principle (i) with property (b) is equivalentto (i) without this property.3.4. Remark. The proof of Theorem 3.1 given above allows us to obtain an analog ofthe equivalence result with a convex �-smooth and Lipschitz continuous function g(�) inthe variational principle (i). To have the corresponding extremal principle and enhancedfuzzy sum rule equivalent to such a convex smooth variational principle, we need to modifyDe�nition 2.1 of �-normals and �-subdi�erentials of controlled rank. One can observe fromthe proof of Theorem 3.1 that all we need for this purpose is to require supporting functionsin both parts of De�nition 2.1 to be concave.Indeed, if we assume the convex smooth variational principle in (i), then the function'+� in the proof of (i)=)(ii) is convex, and one has the required version of the extremalprinciple (ii). To prove (ii)=)(iii) for the modi�cations made, it su�ces to check that inProposition 2.3(ii) concave supporting functions for normals generate concave supportingfunctions for subdi�erentials according to De�nition 2.1. The latter follows directly from theproof of Proposition 2.3(ii) due to the explicit representation (2.11). Finally the proof of theconvex/concave modi�cation in (iii)=)(i) is obviously contained in the proof of Theorem3.1.Note that the Borwein-Preiss smooth variational principle ensures the ful�llment of (i)in Theorem 3.1 with a convex function g(�) in any �-smooth Banach space. Therefore, bothextremal principle (ii) and enhanced fuzzy sum rule (iii) hold in this setting with concavesupporting functions in De�nition 2.1.Now let us consider the case of the Fr�echet bornology � = F and obtain further re�ne-ments of the equivalence results in this case. First we observe the following relationshipbetween viscosity subdi�erentials of controlled rank in De�nition 2.1(ii) and the (full) F -16

subdi�erential (2.1) for locally Lipschitz functions.3.5. Proposition. Let X be a F-smooth space and let ' : X ! �R be Lipschitz continuousaround �x with modulus l. Then one has@�F'(�x) = @F'(�x) for all � > l: (3.35)Proof. The inclusion \�" in (3.35) follows directly from (2.1) for any viscosity �-subdi�erentials of l.s.c. functions with any � > 0. Let us justify the opposite inclusionbased on our assumptions that � = F and ' is Lipschitz continuous with modulus l < �.It is well known that kx?k � l for any x? 2 @F'(�x). Since x? is a viscosity subgradi-ent of ' at �x in the Fr�echet bornology, there exist a neighborhood U of �x and a Fr�echet-di�erentiable function g : U ! R such that rFg(�x) = x?, rFg : X ! X? is norm-to-normcontinuous around �x, and ' � g attains a local minimum at �x. Taking into account thatkrF g(�x)k � l < � and rFg(�) is norm-to-norm continuous, one gets krF g(x)k < � insome neighborhood of �x. Now employing the classical mean value theorem (again due to thenorm-to-norm continuity of rF g(�)), we conclude that g is Lipschitz continuous around �xwith modulus �. This yields x? 2 @�F'(�x) and ends the proof of the proposition. 2Next we establish that in the case of F-smooth spaces the enhanced fuzzy sum rule(iii) in Theorem 3.1 is equivalent to the conventional fuzzy sum rule in terms of full F -subdi�erentials (2.1) without using controlled ranks.3.6. Proposition. Let X be an F-smooth space and let 'i : X ! �R; i = 1; 2, be l.s.c.functions �nite at �x. Suppose that '1 is Lipschitz continuous around �x with modulus l andthat '1 + '2 attains a local minimum at �x. Then for any " > 0 and � > 0 there existxi 2 B�(�x) with j'i(xi)� 'i(�x)j � �; i = 1; 2, such that0 2 @F'1(x1) + @F'2(x2) + "B?: (3.36)Proof. Obviously the enhanced fuzzy sum rule (3.3) implies the one in (3.36), and thisis true for any bornology �. Let us prove the opposite implication which is speci�c for theFr�echet case. Due to Proposition 3.5 the �rst terms of the sums in (3.3) and (3.36) areidentical for su�ciently small � since � > l. We are going to show that (3.36) implies (3.3)despite the di�erence between @F'2 and @�F'2 for general l.s.c. functions.Indeed, let (3.36) hold, i.e., given positive numbers " and � we �nd xi 2 B�(�x) withj'i(xi)� 'i(�x)j � � and x?i 2 @F'i(xi); i = 1; 2, such that kx?1 + x?2k � ". The latter yieldskx?2k � kx?1k+ kx?2 + x?1k � kx?1k+ ": (3.37)Taking an arbitrary � > l, we suppose, without loss of generality, that � and " are su�cientlysmall to satisfy our requirements in what follows. Thus '1 is locally Lipschitzian around x1with modulus l and, therefore, kx?1k � l. This implies x?1 2 @F'�1(x1) due to Proposition 3.5and also kx?2k < � due to (3.37). Based on the latter fact and arguing similarly to the proof17

of Proposition 3.5, we conclude that x?2 2 @�F'2(x2). Thus we arrive at (3.3) and end theproof of the proposition. 2Combining results in Theorem 3.1 and Proposition 3.6 with an equivalence result in [29]for the Fr�echet case, we are able to establish the equivalence between the smooth variationalprinciple formulated above and versions of the extremal principle and the fuzzy sum ruleinvolving the full viscosity constructions (2.1) and (2.5) with � = F .3.7. Theorem. In the case of F-smooth spaces, the smooth variational principle in Theorem3.1 is equivalent to the fuzzy sum rule in Proposition 3.6 as well as the following F -extremalprinciple:For any locally extremal point �x 2 1\2 of the closed set system f1;2g in a F -smoothspace and for any " > 0 there exist xi 2 i \ B"(�x) and x?i 2 NF (xi; i) + "B? such thatkx?1k+ kx?2k = 1 and x?1 + x?2 = 0:Proof. The equivalence between the smooth variational principle and the fuzzy sum rule(3.36) follows from (i)=)(iii) in Theorem 3.1 and Proposition 3.6. On the other hand, ithas been proved in [29] that the F -extremal principle is always equivalent to the fuzzy sumrule (3.36). Note that in the case of F -smooth spaces under consideration the de�nitions ofthe Fr�echet subdi�erential and normal cone used in [29] are equivalent to those in (2.1) and(2.5) for � = F ; see [12]. This completes the proof of the theorem. 23.8. Remark. The results in Propositions 3.5, 3.6 and Theorem 3.7 are based on propertiesof the Fr�echet bornology and do not seem to be true in the case of weaker bornologies. Themost essential property of the Fr�echet bornology we use is that for � = F the topology inX?� agrees with the norm topology in X? and, therefore, the continuity of r�g : X ! X?�required in De�nition 2.1 reduces to the norm-to-norm continuity. The latter fact, whichdoes not hold for weaker bornologies �, is crucial in the proofs of the mentioned results.3.9. Remark. It follows from the proofs given above that analogs of the results in 3.5{3.7 hold if one imposes stronger (than Fr�echet) di�erentiability properties on supportingfunctions in the de�nitions of viscosity normals/subdi�erentials and the smooth variationalprinciple. Indeed, we can assume that these supporting functions are simultaneously s-H�oldersmooth and X has a power modulus of smoothness ts+1 for some s 2 (0; 1], in particular, su-perre exive (see [4, 12]). The case of s = 1 corresponds to proximal normals and subgradientsin Hilbert spaces [7, 33].3.10. Remark. As we mentioned before, the equivalence between the extremal principleand the fuzzy sum rule has been established by Mordukhovich and Shao [29] in the case ofthe Fr�echet bornology. This result is extended by Zhu [35] to general bornological spaceswho also expands the list of equivalent results including a (non-Lipschitz) �-local fuzzy sumrule under an appropriate quali�cation condition, a �-nonlocal fuzzy sum rule, and a �-multidirectional mean value inequality formulated in terms of the full viscosity constructions18

(2.1) and (2.5); see [35] for more details and references. The proofs of all these equivalentresults are based on the smooth variational principle not being equivalent to the latter. Aswe showed above, the usage of a proper control rank narrowing allows us to establish such anequivalence in general bornologically smooth spaces. At the same time, one does not needa control rank narrowing in the case of the Fr�echet bornology as well as stronger s-H�oldersmoothness concepts. 4. SOME APPLICATIONSIn the concluding section of the paper we consider some applications of the main resultsobtained in Section 3. First we present a corollary of the extremal principle in Theorem 3.1that provides a re�ned nonconvex generalization of the Bishop-Phelps density theorem [32]in term of viscosity �-normals of controlled rank. Then we show that the enhanced fuzzysum rule allows us to give a simpli�ed proof and clari�cation of the main representationresults in Borwein and Io�e [2]. Sharper representations, recoverd results in Mordukhovichand Shao [30], are obtained in the case of Fr�echet-smooth spaces.4.1. Theorem. Let X be a �-smooth Banach space, be a nonempty closed subset of X,and � > 0. Then the set of (�; �)-proper points, de�ned byx 2 with N�� (x; ) 6= f0g; (4.1)is dense in the boundary of .Proof. Let �x be a boundary point of the set . Then it is a locally extremal point of thesystem f1;2g where 1 := and 2 := f�xg. Let us apply to this system the extremalprinciple (ii) in Theorem 3.1. Given � > 0, we take positive numbers ", �, and � such that" < �=2 and � = � = �:According to Theorem 3.1(ii) there exist x�" 2 and x?1�"; x?2�" 2 X? satisfyingx�" 2 B"(�x); x?1�" 2 N�� (x�"; ) + "B?; (4.2)kx?1�"k+ kx?2�"k = �; and x?1�" + x?2�" = 0: (4.3)>From the second inclusion in (4.2) we �nd x?�" 2 N�� (x�"; ) such that x?�" = x?1�"+ "e withkek � 1. Since kx?1�"k = �=2 due to (4.3) and " < �=2 by our assumption, one haskx?�"k � kx?1�"k � " = �=2� " > 0:This yields that x�" 2 B"(�x) belongs to the set (4.1) whenever " is su�ciently small.19

To �nish the proof of the theorem, we need to conclude that x�" is a boundary point ofthe closed set . The latter follows from the fact that N�� (x; ) = f0g for any interior pointx of . Indeed, if x? 2 N�� (x; ) with x 2 int ;then the corresponding �-smooth function g(�) in De�nition 2.1(i) attains a local minimumon at an interior point x. Therefore, x? = r�g(x) = 0 due to the Fermat stationaryprinciple which works in this setting. 2Next let us consider an application of the enhanced fuzzy sum rule to prove re�nedrepresentations of nonconvex limiting normal and subdi�erential constructions in smoothBanach spaces. First we recall the notions of A- and G-subdi�erentials developed by Io�ein general Banach spaces; see [18] and its references. If X is a smooth space (admittinga Gateaux-smooth renorm), then the A-subdi�erential of a l.s.c. function ' : X ! �R at�x 2 dom ' can be de�ned as follows@A'(�x) := Limsupx '!�x; "#0@�" '(x) (4.4)where \Limsup" means the topological Kuratowski-Painlev�e upper limit (in the norm topol-ogy of X and the weak-star topology of X?) of the Dini subdi�erential constructions@�" '(x) := fx? 2 X?j hx?; vi � d'(x; v) + "kvk 8v 2 Xg withd'(x; v) := lim infu!v; t#0 t�1('(x+ tu) � '(x)):Then the G-normal cone of � X at �x 2 cl and the G-subdi�erential of ' at �x 2 dom 'are de�ned, respectively, byNG(�x; ) := [�>0�@Adist(�x; ) and (4.5)@G'(�x) := fx? 2 X?j (x?;�1) 2 NG((�x; '(�x)); epi ')g (4.6)through the A-subdi�erential (4.4) of the (Lipschitz continuous) distance function. Notethat constructions (4.5) and (4.6) were introduced in [18] as nuclei of the G-normal coneand the G-subdi�erential therein which were obtained from (4.5) and (4.6) by using theweak-star topological closure. The terminology used herein was �rst adopted in [2]. In �nitedimensions, constructions (4.4){(4.6) reduce to the limiting normal cone and subdi�erentialde�ned in Mordukhovich [25].We also consider the sequential A-subdi�erential@�A'(�x) := lim sup'x!�x; "#0 @�" '(x) (4.7)20

that may be strictly smaller than its topological counterpart (4.4) even for locally Lipschitzfunctions. Relationships between weak-star topological and sequential limits are studied inBorwein and Fitzpartick [1] where one can �nd the following important result.4.2. Proposition. Let X be a Banach space and let fSkg be a sequence of bounded subsetsof X? such that Sk+1 � Sk for all k = 1; 2; : : :. If the unit ball of X? is weak-star sequentiallycompact, then one has 1\k=1 cl?Sk = cl?f limk!1 x?kj x?k 2 Sk for all kgwhere the topological closure cl? and the sequential limit are taken in the weak-star topologyof X?.Based on the enhanced fuzzy sum rule (3.3), now we can prove a re�ned sequentialrepresentation of the (topological) A-subdi�erential (4.4) for Lipschitz continuous functionsthrough their viscosity subdi�erentials of controlled rank.4.3. Theorem. Let X be a �-smooth Banach space and let ' : X ! �X be Lipschitzcontinuous around �x with modulus l. Then for any � > l one has@A'(�x) = cl?flim supx!�x @��'(x)g: (4.8)Proof. It follows directly from the de�nitions that @��'(x) � @�" '(x) for any x 2 dom ',� > 0, and " � 0. This gives the inclusion \�" in (4.8). To prove the opposite inclusion, we�rst observe that the A-subdi�erentials (4.4) and (4.7) can be represented as@A'(�x) = 1\k=1 cl?Sk and @�A'(�x) = f limk!1 x?kj x?k 2 Sk for all kgwhere Sk := Sf@�1=k'(x) with kx� �xk � 1=kg. Obviously Sk+1 � Sk for all k = 1; 2; : : : and,since ' is Lipschitz continuous around �x, the sets Sk are bounded inX? when k is su�cientlylarge. Now taking into account that for any smooth space X the unit ball in X? is weak-starsequentially compact [32], we conclude from Proposition 4.2 that@A'(�x) = cl?(@�A'(�x)):Therefore, representation (4.8) follows from the inclusion@�A'(�x) � cl?flim supx!�x @��'(x)g: (4.9)To establish (4.9), we are going to verify that@�A'(�x) � lim supx '!�x @��'(�x) + V ? (4.10)21

for any weak-star neighborhood V ? of the origin in X?. To this end we observe that for anysuch V ? there exist a �nite dimensional subspace L � X and a number > 0 satisfyingL? + 2 B? � V ? where L? := fx? 2 X?j hx?; xi = 0 8x 2 Lg: (4.11)Now picking any x? 2 @�A'(�x) and using de�nition (4.7), one gets sequences xk ! �x; "k # 0,and x?k w?! x? with x?k 2 @�"k'(xk) for all k. We take k to be su�ciently large to ensure that0 < "k < . It follows from the de�nition of the Dini "-subdi�erential that the functionfk(x) := '(x)� hx?k; x� xki+ kx� xkk+ �(x � xk;L) (4.12)attains a local minimun at xk for each k. Applying the enhanced fuzzy sum rule (iii) ofTheorem 3.1 to the sum of two functions in (4.12) with'1(x) := '(x) and '2(x) := �hx?k; x� xki+ kx� xkk+ �(x� xk;L);we �nd x1k 2 B1=k(xk) and x2k 2 B1=k(xk) \ (xk + L) such that0 2 @��'(x1k) + @��'2(x2k) + B? (4.13)for any � > l and large k. Note that '2 is convex and its �rst two summands are continuous.Thus one can employ the classicalMoreau-Rockafellar theorem to compute the subdi�erentialof convex analysis for '2 and conclude that@��'2(x2k) � �x?k + B? + L?: (4.14)Now combining (4.13) with (4.11) and (4.14), we getx?k 2 @��'(x1k) + V ?: (4.15)It follows directly from De�nition 2.1(ii) that the sets @��'(x1k) are uniformly bounded by �for all k su�cient large. Using again the weak-star sequential compactness of bounded setsin X? for smooth spaces X and passing to the limit in (4.15) as k !1, we �nally arrive at(4.10) and complete the proof of the theorem. 2Theorem 4.3 implies the main representation results in Borwein and Io�e [2] for the G-normal cone (4.5) and the G-subdi�erential (4.6) in smooth Banach spaces.4.4. Corollary. Let X be a �-smooth Banach space. Then the following hold:(i) For any closed set � X and any �x 2 one hasNG(�x; ) = [�>0 cl?flim supx !�x N�� (x; )g: (4.16)(ii) For any function ' : X ! �R l.s.c. around �x 2 dom ' one has@G'(�x) = [�>0 cl?flim supx '!�x @��'(x)g: (4.17)22

Proof. To prove (i), we just observe thatlim supx !�x N�� (x; ) = lim supx !�x @��f� dist(x; )g = lim supx !�x �@1� dist(x; ) (4.18)due to (2.3). Therefore, representation (4.16) follows directly from (4.5), (4.8), and (4.18).Representation (4.17) follows from (4.16) due to (4.6) and Proposition 2.3. 24.5. Remark. One can easily observe from Remark 3.4 and the proofs given above thatthe supporting functions g(�) for �-normals and subdi�erentials of controlled rank in repre-sentations (4.8), (4.16), and (4.17) can always be chosen to be concave.Now let us con�ne our treatment to the case of Fr�echet-smooth spaces and recall thenotions of limiting normal cone and subdi�erential introduced by Kruger and Mordukhovich[22] as extensions of the corresponding �nite dimensional constructions in [25]. Based on(Fr�echet) "-normals and subdi�erentialsN"(x; ) := fx? 2 X?j lim supu !x hx?; u� xiku� xk � "g; (4.19)@"'(x) := fx? 2 X?j lim infu!x '(u)� '(x)� hx?; u� xiku� xk � �"g; " � 0; (4.20)we de�ne these sequential limiting constructions as followsN(�x; ) := lim supx!�x; "#0 N"(x; ); (4.21)@'(�x) := lim sup'x!�x; "#0 @"'(x): (4.22)A recent comprehensive study of the limiting constructions (4.21) and (4.22) in the frame-work of Asplund spaces (including spaces with Fr�echet renorms) has been conducted in [30]where the reader can �nd detailed discussions and further references. Our purpose here is toshow how one can easily prove results of [30] related to the description of (4.21) and (4.22)through viscosity normals and subdi�erentials of controlled rank.4.6. Theorem. Let X be an F-smooth Banach space. Then the following hold:(i) For any closed set � X and any �x 2 one hasN(�x; ) = [�>0[lim supx !�x N�F (x; )]: (4.23)(ii) For any function ' : X ! �R l.s.c. around �x 2 dom ' one has@'(�x) = [�>0[lim supx '!�x @�F'(x)]: (4.24)23

Proof. First we consider the case when ' is a function Lipschitz continuous around �xwith modulus l. Let us show that in this case the limiting subdi�erential (4.22) admits therepresentation @'(�x) = lim supx!�x @�F'(x) 8� > l: (4.25)Indeed, due to Proposition 3.5 the F -subdi�erential @�F'(x) of controlled rank coincides withthe full viscosity F -subdi�erential (2.1) for any � > l. Therefore, (4.25) reduces to@'(�x) = lim supx!�x @F'(x):It is shown in [12] that @F' agrees with de�nition (4.20) as " = 0 for any F -smooth space.To �nish the proof of (4.25), it su�ces to recall that the original limiting construction (4.22)is equivalent to the one with " = 0 when X is an F -smooth (even Asplund) space; cf. [19,30].Based on (4.25), we establish representation (4.23) by using the relationshipN(�x; ) := [�>0�@dist(�x; )between the limiting constructions (4.21) and (4.22).It remains to justify representation (4.24) for general l.s.c. functions. This follows from(4.23) due to the formula@'(�x) = fx? 2 X?j (x?;�1) 2 N((�x; '(�x)); epi ')gand its counterpart for the Fr�echet constructions (4.19) and (4.20) with " = 0. 2One can observe that the only di�erence between the limiting constructions (4.21), (4.22)and the corresponding G-constructions (4.5), (4.6) in the case of F -smooth spaces is the ab-sence of the weak-star topological closure in representations (4.23) and (4.24). We refer thereader to Section 9 of [30] for more discussions in this direction.REFERENCES1. J. M. Borwein and S. Fitzpatrick, Weak-star sequential compactness and bornologicallimit derivatives, J. Convex Anal., 2 (1995), 59{68.2. J. M. Borwein and A. D. Io�e, Proximal analysis in smooth spaces, Set-Valued Analysis4 (1996), 1{24.3. J. M. Borwein and A. Jofr�e. A nonconvex separation property in Banach spaces andapplications, preprint. 24

4. J. M. Borwein and D. Preiss, A smooth variational principle with applications tosubdi�erentiability and di�erentiability of convex functions, Trans. Amer. Math. Soc.303 (1987), 517{527.5. J. M. Borwein, J. S. Treiman, and Q. J. Zhu, Partially smooth variational principlesand applications, preprint.6. J. M. Borwein and Q. J. Zhu, Viscosity solutions and viscosity subderivatives in smoothBanach spaces with applications to metric regularity, SIAM J. Control Optim 34(1996), 1568{1591.7. F. H. Clarke, Methods of Dynamic and Nonsmooth Optimization, CBMS-NSF Re-gional Conference Series in Applied Mathematics, vol. 57, Soc. Indust. Appl. Math.,Philadelphia, Pa., 1989.8. F. H. Clarke, Yu. S. Ledyaev, and P. R. Wolenski, Proximal analysis and minimizationprinciples, J. Math. Anal. Appl. 196 (1995), 722{735.9. M. G. Crandall and P.-L. Lions, Viscosity solutions of Hamilton-Jacobi equations,Trans. Amer. Math. Soc. 277 (1983), 1{42.10. R. Deville and E. M. El Haddad, The subdi�erential of the sum of two functions inBanach spaces I: First order case, Convex Analysis (to appear).11. R. Deville, G. Godefroy, and V. Zizler, A smooth variational principle with applicationsto Hamilton-Jacobi equations in in�nite dimensions, J. Func. Anal. 111 (1993), 192{212.12. R. Deville, G. Godefroy, and V. Zizler, Smoothness and Renorming in Banach Spaces,Pitman Monographs and Surveys in Pure and Applied Math., vol. 64, Longman, 1993.13. I. Ekeland, On the variational principle, J. Math. Anal. Appl. 47 (1974), 325{358.14. M. Fabian, Subdi�erentiallity and trustworthiness in the light of a new variationalprinciple of Borwein and Preiss, Acta Univ. Carolinae 30 (1989), 51{56.15. M. Fabian, P. H�ajer, and J. Vanderwer�, On smooth variational principles in Banachspaces, J. Math. Anal. Appl. 197 (1996), 153{172.16. A. D. Io�e, On subdi�erentiability spaces, Ann. New York Acad. Sci. 410 (1983),107{119.17. A. D. Io�e, Calculus of Dini subdi�erentials and contingent derivatives of set-valuedmaps, Nonlinear Anal. 8 (1984), 517{539.18. A. D. Io�e, Approximate subdi�erential and applications. III: The metric theory,Mathematika 36 (1989), 1{38. 25

19. A. D. Io�e, Proximal analysis and approximate subdi�erentials, J. London Math. Soc.41 (1990), 175{192.20. A. D. Io�e and R. T. Rockafellar, The Euler and Weierstrass conditions for nonsmoothvariational problems, Calculus of Variations and PDEs 4 (1996), 59{87.21. A. Y. Kruger, Generalized di�erentials of nonsmooth functions and necessary condi-tions for an extremum, Siberian Math. J. 26 (1985), 370{379.22. A. Y. Kruger and B. S. Mordukhovich, Extremal points and the Euler equation innonsmooth optimization, Dokl. Akad. Nauk BSSR 24 (1980), 684{687.23. Y. Li and S. Shi, A generalization of Ekeland's "-variational principle and of itsBorwein-Preiss' smooth variant, J. Math. Anal. Appl. (to appear)24. P. D. Loewen, A mean value theorem for Fr�echet subgradients, Nonlinear Anal. 23(1994), 1365{1381.25. B. S. Mordukhovich, Maximum principle in problems of time optimal control withnonsmooth constraints, J. Appl. Math. Mech. 40 (1976), 960{969.26. B. S. Mordukhovich, Metric approximations and necessary optimality conditions forgeneral classes of nonsmooth extremal problems, Soviet Math. Dokl. 22 (1980), 526{530.27. B. S. Mordukhovich, Approximate Methods in Problems of Optimization and Control,Nauka, Moscow, 1988.28. B. S. Mordukhovich, Generalized di�erential calculus for nonsmooth and set-valuedmappings, J. Math. Anal. Appl. 183 (1994), 250{288.29. B. S. Mordukhovich and Y. Shao, Extremal characterizations of Asplund spaces, Proc.Amer. Math. Soc. 124 (1996), 197{20530. B. S. Mordukhovich and Y. Shao, Nonsmooth sequential analysis in Asplund spaces,Trans. Amer. Math. Soc. 348 (1996), 1235{1280.31. B. S. Mordukhovich and Y. Shao, Fuzzy calculus for coderivatives of multifunctions,Nonlinear Anal. 29 (1997), 1{22.32. R. R. Phelps, Convex functions, Monotone Operators and Di�erentiability, 2nd edition,Lecture Notes in Mathematics, vol. 1364, Springer, 1993.33. R. T. Rockafellar, Proximal subgradients, marginal values, and augmented Lagrangiansin nonconvex optimization, Math. Oper. Res. 6 (1981), 424{436.34. R. T. Rockafellar and R. J-B Wets, Variational Analysis, Springer, 1997.26

35. Q. J. Zhu, The equivalence of several basic theorems for subdi�erentials, preprint.

27