Theory of Learnable

8/3/2019 Theory of Learnable

1/9

RESEARCH C ONTRIBUTIONS

Ar t i f i c ia lI n t e l li g e n c e a n dLanguage Pr oces s ingD a v i d W a l t zEdi tor

A Theory of the Learna ble

L, G. VALIANT

A B S T R A C T : H u m a n s a p p e a r t o b e a b le to le ar n n e wc o n c e p t s w i t h o u t n e e d i n g t o b e p r o g r a m m e d e x p l i c i t l y i na n y c o n v e n t i o n a l s e n s e . I n t h i s p a p e r w e r e g a rd l e a r n i n g a st h e p h e n o m e n o n o f k n o w l e d g e a c q u i s i t i o n i n t h e a b s e n c e o fe x p l i c it p r o g r a m m i n g . W e g i v e a p r e c i se m e t h o d o l o g y f o rs t u d y i n g t h i s p h e n o m e n o n f r o m a c o m p u t a t i o n a l v i e w p o i n t .I t cons i s t s o f choos ing an appr opr ia te in for mat ion g a ther ingmechani s m, the l ear n ing pr o toco l , and exp lor ing the c las s o fconcep t s tha t can be l ear ned us ing i t i n a r eas onable( p o l yn o m i a l ) n u m b e r o f st e p s. A l t h o u g h i n h e r e n t a l g o r i t h m i ccomplex i t y appear s to s e t s er ious l imi t s to the r ange o fconcep t s tha t can be l ear ned , w e s how tha t ther e ar e s omei m p o r t a n t n o n t r i v i a l c l a s s e s o f p r o p o s i ti o n a l c o n c e p t s t h a tcan be l ear ned in a r ea l i s t i c s ens e .1. INTRODUCTIONComputability theory became possible once precisemodels became available for modeling the c ommon-place phenomenon of mechanical calculation. The the-ory that evolved has been used to explain hu man expe-rience and to suggest how artificial computing devicesshould be built. It is also worth studying for its ownsake.The commonplace phenomenon of learning surelymerits similar attention. The problem is to discovergood models that are interesting to study for their ownsake and that promise to be relevant both to explaininghuman experience and to building devices that canlearn. The models should also shed light on the limitsof what can be learned, just as computability does onwhat can be computed.In this paper we shall say that a program for perform-ing a task has been acquired by l ear n ing if it has beenacquired by any means other than explicit program-ming. Among human skills some clearly appear to haveThis research was supported in part by National Science Foundationgrant MCS-83-02385. A prelim inary version of this paper appeared inthe proceedings of the 16th ACM Symposium on Theory of Computing,Washington, D.C., 1984, 436-445.1984ACMO001-0782/84/1100-1134 75

a genetically preprogrammed element, whereas someothers consist of executing an explicit sequence of in-structions that has been memorized. There remains alarge area of skill acquisition where no such explicitprogramming is identifiable. It is this area that we de-scribe here as learning. The recognition of familiar ob-jects, such as tables, provides such examples. Theseskills often have the additional property that, althoughwe have learned them, we find it difficult to articulatewhat algorithm we are really using. In these cases itwould be especially significant if machines could bemade to acquire the m by learning.This paper is concerned with precise computationalmodels of the learning phenom enon. We shall restrictourselves to skills that consist of recognizing whet her ac o n c e p t (or predicate) is true or not for given data. Weshall say that a concept Q has been learned if a pro-gram for recognizing it has been deduced (i.e., by somemethod other than the acquisition from the outside ofthe explicit program).The main contribution of this paper is that it showsthat it is possible to design l e a r n i n g m a c h i n e s that haveall three of the following properties:1. The machines can provab ly learn whole classes ofconcepts. Furthermore, these classes can be charac-terized.2. The classes of concepts are appropriate and nontri-vial for general-purpose knowledge.3. The computational process by which the machinesdeduce the desired programs requires a feasible

(i.e., polynomial) nu mber of steps.A learning machine consists of a l ear n ing pr o toco l to-gether with a d e d u c t i o n p r o c e d u r e . The former specifiesthe manner in which information is obtained from theoutside. The latter is the mechanism by which a correctrecognition algorithm for the concept to be learned is

deduced. At the broadest level, the suggested m e t h o d o l -ogy for studying learning is the following: Define a plau-sible learning protocol, and investigate the class of con-

1134 C om mun ica t ions f t he AC M N ovember 1 9 8 4 Volume 2 7 N umb er 11


2/9

Research Con tr ibut ions

cepts for which recognition programs can be deducedin polynomial time using the protocol.The specific protocols considered in this paper allowfor two kinds of information supply. First the learnerhas access to a supply of typical data that positivelyexemplify the concept. More precisely, it is assumedthat these positive examples have a probabilistic distri-bution determined arbitrarily by nature. A call of aroutine EXAMPLES produces one such positive exam-ple. The relative probability of producing different ex-amples is determined by the distribution. The secondsource of information that may be available is a routineORACLE. In its most basic version, w hen presentedwith data, it will tell the learner whether or not thedata positively exemplify the concept.

The major remaining design choice that has to bemade is that of knowledge representation. Since ourdeclared aim is to represent general knowledge, itseems almost unavoidable that we use some kind oflogic rather than, for example, formal grammars or geo-metrical constructs. In this paper we shall representconcepts as Boolean functions of a set of propositionalvariables. The recognition algorithms that we attemptto deduce will be therefore Boolean circuits or expres-sions.The adequacy of the propositional calculus for repre-senting knowledge in practical learning systems isclearly a crucial and controversial question. Few wouldargue that this much power is not necessary. The ques-tion is whether it is enough. There are several argu-ments suggesting that such a system would, at least, bea very good start. First, when one examines the mostfamous examples of systems that embody prepro-grammed knowledge, namely, expert systems such asDENDRAL and MYC!N, essentially no logical notationbeyond the propositional calculus is used. Surely itwould be overambitious to try to base learning systemson representations that are more powerful than thosethat have been successfully managed in programmedsystems. Second, the results in this paper can be nega-tively interpreted to suggest that the class of learnableconcepts even within the propositional calculus is se-verely circumscribed. This suggests that the search forextensions to the propositional calculus that have sub-stantially larger learnable classes may be a difficultone.The positive conclusions of this paper are that thereare specific classes of concepts that are learnable inpolynomial time using learning protocols of the kinddescribed. These classes can all be characterized bydefining the class of programs that recognize them. Ineach case the programs are special kinds of Booleanexpressions. The three classes are (l) conjunctive nor-mal form expressions with a bounded nu mber of liter-als in each clause, (2) monotone disjunctive normalform expressions, and (3) arbitrary expressions inwhich each variable occurs just once. In the first ofthese, no calls of the oracle are necessary. In the last,no access to typical examples is necessary but the ora-cles need to be more powerful than the one describedabove.

The deduction procedure will in each case output anexpression that with high likelihood closely approxi-mates the expression to be learned. Such an approxi-mate expression never says yes when it should not butmay say no on a small fraction of the probability spaceof positive examples. This fraction can be made arbi-trarily small by increasing the runtime of the deductionprocedure. Perhaps the main technical discovery con-tained in the paper is that with this probabilistic notionof learning highly convergent l earning is possible forwhole classes of Boolean functions. This appears to dis-tinguish this approach from more traditional oneswhere learning is seen as a process of "inducing" somegeneral rule from information that is insufficient for areliable deduction to be made. The power of this proba-bi!istic viewpoint is illustrated, for example, by the factthat an arbitrary set of polynomially many positive ex-amples cannot be relied on to determine expressionsconsisting even of only a single monomial in any relia-ble way. A more detailed description of this methodo-logical problem can be found in [9].There is another aspect of our formulation that isworth emphasizing. In a learning system that is aboutto learn a new concept, there may be an enormousnumber of propositional variables available. These maybe primitive inputs, the values of preprogrammed con-cepts, or the values of concepts that have been lcarnedpreviously. We want the complexity of learning thenew concept to be related only to the number of vari-ables that may be set in natural examples of it, and noton the cardinality of the universe of available variables.Hence, the questions asked of ORACLE and the valuesgiven by EXAMPLES will not be t ruth assignments toall the variables. In the archetypal case, they will spec-ify an assignment to a subset of the variables that is stillsufficient to guarantee the truth of the function.

Whether the classes of learnable Boolean conceptscan be extended significantly beyond the three classesgiven is an interesting question. There is circumstantialevidence from cryptography, however, that the wholeclass of functions computable by polynomial size cir-cuits is not learnable. Consider a cryptographic schemethat encodes messages by evaluating the function Ekwhere k specifies the key. Suppose that this scheme isimmune to chosen plaintext attack in the sense that,even if the values of Ek are known for polynomiallymany dynam ically chosen inputs, it is computationallyinfeasible to deduce an algorithm for Ek or for an ap-proximation to it. This is equivalent to saying, however,that Ek is not learnable. The conjectured existence ofgood cryptographic functions that are easy to computetherefore implies that some easy-to-compute functionsare not learnable.If the class of learnable concepts is as severely lim-ited as suggested by our results, then it would followthat the only way of teaching more complicated con-cepts is to build them up from such simple ones. Thus,a good teacher would have to identify, name, and se-quence these intermediate concepts in the mann er of aprogrammer. The results of l e a r n a b i l i t y t h e o r y wouldthen indicate the maximum granularity of the single

N o v e m b e r 1 9 8 4 V o l u m e 2 7 N u m b e r 1 1 C o m m u n i ca t io n s o f t h e A C M 1135


3/9

Research Contr ibut ions

c o n c e p t s t h a t c a n b e a c q u i r e d w i t h o u t p r o g r a m m i n g .I n s u m m a r y , t h i s p a p e r a t t e m p t s t o e x p l o r e t h e l im i t s

o f w h a t i s l e a r n a b l e a s a l lo w e d b y a l g o r i t h m i c c o m p l e x -i t y. T h e r e s u l t s a r e d i s t i n g u i s h a b l e f r o m t h e d i v e r s eb o d y o f p r e v i o u s w o r k o n l e a r n i n g b e c a u s e t h e y a t -t e m p t t o r e c o n c i l e t h e t h r e e p r o p e r t i e s ( ( 1 ) - ( 3 ) ) m e n -t i o n e d e a r l i e r . C l o s e s t i n r i g o r t o o u r a p p r o a c h i s t h ei n d u c t i v e i n f e r e n c e l i t e r a t u r e ( s e e A n g l u i n a n d S m i t h[ 1] f o r a s u r v e y ) t h a t d e a l s w i t h i n d u c i n g s u c h t h i n g s a sr e c u r s i v e f u n c t i o n s o r f o r m a l g r a m m a r s ( b u t n o t B o o -l e a n f u n c t io n s ) f r o m e x a m p l e s . T h e r e i s a l a r ge b o d y o fw o r k o n p a t t e r n r e c o g n i t i o n a n d c l a s s if i c a ti o n , u s i n gs t a t i s t i c a l a n d o t h e r t o o l s ( e .g ., [4 ]) , b u t t h e q u e s t i o n o fg e n e r a l k n o w l e d g e r e p r e s e n t a t i o n i s n o t a d d r e s s e dt h e r e d i r e c t l y . L e a r ni n g , i n v a r i o u s l e s s f o r m a l s e n s e s ,h a s b e e n s t u d i e d w i d e l y a s a b r a n c h o f a r t i f i c ia l i n t e l l i -g e n c e . A s u r v e y a n d b i b l i o g r a p h y c a n b e f o u n d i n[ 2, 7] . I n t h e i r t e r m i n o l o g y t h e s u b j e c t o f t h i s p a p e r i sc o n c e p t l e a r n in g .

I n p r e v i o u s w o r k o n l e a r n i n g , m u c h e m p h a s i s i sp l a c e d o n t h e d i v e r s it y o f m e t h o d s t h a t h u m a n s a p p e a rt o b e u s in g . T h e s e i n c l u d e l e a r n i n g b y e x a m p l e , l e a r n -i n g b y a n a l o g y , a n d l e a r n i n g b y b e i n g t o ld . T h i s p a p e re m p h a s i z e s t h e p o l a r i t y b e t w e e n l e a r n i n g b y b e i n g p ro -g r a m m e d a n d l e a r n i n g i n t h e a b s e n c e o f a n y p r o g r a m -m i n g e l e m e n t . M a n y o f t h e c o n v e n t i o n a l l y a c c e p t e dm o d e s o f l e a r n i n g c a n b e r e g a r d e d a s b e i n g h y b r i d s o ft h e s e t w o e x t r e m e s .2. A L E A R N I N G P R O T O C O LF O R B O O L E A N F U N C T I O N SW e c o n s i d e r t B o o l e a n v a r i a b l e s pl . . . . . p t , e a c h o fw h i c h c a n t a k e t h e v a l u e 1 o r 0 t o i n d i c a t e w h e t h e r t h ea s s o c i a t e d p r o p o s i t i o n i s t r u e o r f a l s e . T h e r e i s n o a s -s u m p t i o n a b o u t t h e i n d e p e n d e n c e o f t h e v a r i a b l e s . I n -d e e d , t h e y m a y b e f u n c t i o n s o f e a c h o t h e r .

A v e c t o r i s a n a s s i g n m e n t t o e a c h o f t h e t v a r i a b l e s o fa v a l u e f r o m { 0, 1 , *} . T h e s y m b o l * d e n o t e s t h a t av a r i a b l e i s u n d e t e r m i n e d . A v e c t o r i s t o t a l i f e v e r y v a r i -a b l e i s d e t e r m i n e d ( i . e . , i s a s s i g n e d a v a l u e f r o m { 0 , 1 } ) .F o r e x a m p l e , t h e a s s i g n m e n t p l = p 3 = 1 , p 4 = 0 , a n dp 2 = * i s a v e c t o r t h a t i s n o t t o t a l .

A B o o l e a n f u n c t i o n F is a m a p p i n g f r o m t h e s e t o f 2t o t a l v e c t o r s t o { 0 , 1 }. A B o o l e a n f u n c t i o n F b e c o m e s ac o n c e p t F i f i t s d o m a i n i s e x t e n d e d t o t h e s e t o f a l lv e c t o r s a s f o l l o w s : F o r a v e c t o r v , F ( v ) = 1 i f a n d o n l y i fF ( w ) = 1 f o r a l l t o t a l v e c t o r s w t h a t a g r e e w i t h v o n a l lv a r i a b l e s f o r w h i c h v i s d e t e r m i n e d . T h e p u r p o s e o ft h i s e x t e n s i o n i s t h a t i t p e r m i t s u s n o t t o m e n t i o n t h ev a r i a b l e s o n w h i c h F d o e s n o t d e p e n d .

G i v e n a c o n c e p t F , w e c o n s i d e r a n a r b i t r a r y p r o b a b i l -i t y d i s t r i b u t i o n D o v e r t h e s e t o f a l l v e c t o rs v s u c h t h a tF (v ) = 1 . I n o t h e r w o r d s , f o r e a c h v s u c h t h a t F (v ) = 1, i ti s t h e c a s e t h a t D(v} _> O. A l s o ~ D ( v ) = 1 w h e n s u m m a -t i o n i s o v e r t h i s s e t o f v e c t o r s . T h e r e a r e n o o t h e r a s -s u m e d r e s t r i c t i o n s o n D, w h i c h i s i n t e n d e d t o d e s c r i b et h e r e l a ti v e f r e q u e n c y w i t h w h i c h t h e p o s i ti v e e x a m -p l e s o f F o c c u r i n n a t u r e .

W h a t a r e r e a s o n a b l e l e a r n i n g p r o t o c o l s t o c o n s i d e r ?F i r s t w e m u s t a v o i d g i v i n g t h e t e a c h e r t o o m u c h p o w e r ,n a m e l y , t h e p o w e r t o c o m m u n i c a t e a p r o g ra m i n s t r u c -

t i o n b y i n s t r u c t i o n . F o r e x a m p l e , i f a p r e m e d i t a t e d s e -q u e n c e o f v e c t o r s w i t h r e p e t i t i o n s c o u l d be c o m m u n i -c a t e d , th e n t h i s c o u l d b e u s e d t o e n c o d e t h e d e s c r i p t i o no f t h e p r o g r a m e v e n i f j u s t t w o s u c h v e c t o r s w e r e u s e df o r b i n a r y n o t a t i o n . S e c o n d w e m u s t a v o i d g i v i n g t h et e a c h e r w h a t i s e v i d e n t l y t o o l i t t l e p o w e r . I n p a r t i c u l a r ,t h e p r o to c o l m u s t p r o v i d e s o m e t y p i c a l e x a m p l e s o fv e c t o r s f o r w h i c h F is t r u e , f o r o t h e r w i s e , i f F is t r u e f o rj u s t o n e v e c t o r t h a t i s t o t a l, o n l y a n e x p o n e n t i a l s e a r c ho r m o r e p o w e r f u l o r a c l e s w o u l d b e a b l e t o f in d i t.

S u c h c o n s i d e r a t i o n s l e d u s to c o n s i d e r t h e f o l l o w i n gl e a r n i n g p r o t o c o l a s a r e a s o n a b l e o n e . I t g i v e s t h el e a r n e r a c c e s s t o t h e f o l l o w i n g t w o r o u t i n e s :1. E X A M P L E S : T h i s h a s n o i n p u t . I t g iv e s a s o u t p u t a

v e c t o r v s u c h t h a t F (v ) = 1 . F o r e a c h s u c h v , t h ep r o b a b i l i t y t h a t v i s o u t p u t o n a n y s i n g l e c al l i sD ( v ) .

2 . O R A C L E ( ): O n v e c t o r v , a s i n p u t i t o u t p u t s 1 o r 0a c c o r d in g t o w h e t h e r F (v ) = 1 o r 0 .

T h e f i rs t o f t h e s e g i v e s r a n d o m l y c h o s e n p o s i t i v e e x -a m p l e s o f t h e c o n c e p t b e i n g l e a r n e d . T h e s e c o n d p r o -v i d e s a t e s t f o r w h e t h e r a v e c t o r , w h i c h t h e d e d u c t i o np r o c e d u r e c o n s i d e r s t o b e c r i ti c a l , i s a n e x a m p l e o f t h ec o n c e p t o r n o t . In a r e a l s y s t e m , t h e o r a c l e m a y b e ah u m a n e x p e r t , a d a t a b a s e o f p a s t o b s e r v a t io n s , s o m ed e d u c t i o n s y s t e m , o r a c o m b i n a t i o n o f t h e s e .

F i n a l l y w e o b s e r v e t h a t o u r p a r t i c u l a r c h o i c e o fl e a r n i n g p r o t o c o l w a s s t r o n g l y i n f l u e n c e d b y t h e e a r l i e rd e c i s i o n t o d e a l w i t h c o n c e p t s r a t h e r t h a n r a w B o o l e a nf u n c t i on s . I f w e w e r e t o d e a l w i t h t h e l a t t e r, t h e n m a n yo t h e r a l t e r n a t i v e s a r e e q u a l l y n a t u r a l . F o r e x a m p l e ,g i v e n a v e c t o r v , i n s t e a d o f as k i n g , as w e d o , w h e t h e ra l l c o m p l e t i o n s o f i t m a k e t h e f u n c t i o n t r u e , w e c o u l da s k w h e t h e r t h e r e e x i st a n y c o m p l e t i o n s t ha t m a k e i tt r u e . T h i s s ug g e s ts n a t u r a l a l t e r n a t i v e s e m a n t i c s f o rE X A M P L E S a n d O R A C L E . I n S e c t io n 7 w e s h a l l u s es u c h m o r e e l a b o r a t e o r a c l e s.3 . L E A R N A B I L I T YW e s h a l l c o n s i d e r v a r i o u s c l a s s e s o f p r o g r a m s h a v i n gp l . . . . . p t a s i n p u t s a n d s h o w t h a t i n e a c h c a s e a n yp r o g r a m i n t h e c l a s s c a n b e d e d u c e d , w i t h o n l y s m a l lp r o b a b i l i t y o f e r r o r , i n p o l y n o m i a l t i m e u s i n g t h e p r o t o -c o l p r e v i o u s l y d e s c r i b e d . W e a s s u m e t h a t p r o g r a m s c a nt a k e t h e v a l u e s 1 , 0 , a n d u n d e t e r m i n e d .

M o r e p r e c i s e l y w e s a y t h a t a c l a s s X o f p r o g r a m s i sl e a r n a b l e w i t h r e s p e c t t o a g iv e n l e a r n i n g p r o t o c o l i f a n do n l y i f t h e r e e x i s t s a n a l g o r i t h m A { t he d e d u c t i o n p r o -c e d ur e } i n v o k i n g t h e p r o t o c o l w i t h t h e f o l l o w i n g p r o p -e r t i e s :1 . T h e a l g o r i t h m r u n s i n t i m e p o l y n o m i a l i n a n a d -

j u s t a b l e p a r a m e t e r h , i n t h e v a r i o u s p a r a m e t e r s t h a tq u a n t i f y t h e s i z e o f t h e p r o g r a m t o be l e a r n e d , a n di n t h e n u m b e r o f v a r i a b l e s t.

2 . F o r a ll p r o g r a m s f E X a n d a l l d i s t r i b u t i o n s D o v e rv e c t o r s v o n w h i c h f o u t p u t s 1 , t h e a l g o r i t h m w i l ld e d u c e w i t h p r o b a b i l i t y a t le a s t ( 1 - h -1 } a p r o g r a mg ~ X t h a t n e v e r o u t p u t s o n e w h e n i t s h o u l d n o tb u t o u t p u t s o n e a l m o s t a l w a y s w h e n i t s h o u l d . I n

1 1 3 6 C om mu ni c a t i ons f t he A C M N ov e mbe r 1 9 8 4 V ol ume 2 7 N umbe r 1 1


4/9

Research Contributions

par t i c u la r , { i) fo r a ll vec to r s v , g ( v ) = i imp l ie sf ( v ) = 1 , and ( ii } the s um o f D ( v ) o v e r a l l v s u c h t h a tf ( v ) = 1 , bu t g ( v ) ~ 1 i s a t mos t h -1 .

I n o u r d e f in i t i o n w e h a v e d i s a l lo w e d p o s i ti v e a n -s w e r s f r o m g to b e w r o n g , o n l y b e c a u s e t h e p a r t i c u l a rc l a s s es X i n t h i s p a p e r d o n o t r e q u i r e i t. T h i s w e c a l lo n e - s i d e d e r r o r l ea r n in g , i n o t h e r c i r c u m s t a n c e s i t m a yb e e f f i c a c i o u s t o a l l o w t w o - s i d e d e r r o r s . I n t h a t m o r eg e n e r a l c a s e , w e w o u l d n e e d t o p u t a p r o b a b i l i t y d i s t r i -b u t i o n D o n t h e s e t o f v e c t o r s s u c h t h a t f ( v} ~ 1 . C o n d i -t ion ( i) i n (2} i s then r ep lace d by ~ D{v} ove r a l l v suc ht h a t f ( v ) ~ 1, bu t g { v ) = 1 i s a t mo s t h -1 .A s e c o n d a s p e c t o f o u r d e f i n i t i o n i s t h a t t h e p a r a m e -t e r h is u s e d i n t w o i n d e p e n d e n t p r o b a b i li s ti c b o u n d s .T h i s s i mp l i f i e s t h e s t a t e m e n t o f t h e r e s u l t s . I t w o u l d b em o r e p r e c i se , h o w e v e r , t o w o r k e x p l i c it l y w i t h t w o i n -d e p e n d e n t p a r a m e t e r s h i a n d h 2 .

T h i r d w e s h o u l d n o t e t h a t p r o g r a m s t h a t c o m p u t ec o n c e p t s s h o u l d b e d i s t in g u i s h e d f r o m t h o s e t h a t c o m -p u t e m e r e l y t h e B o o l e a n f u n c t i o n . T h i s d i s t i n c t i o n h a sl i tt l e c o n s e q u e n c e f o r t h e t h r e e c l a s s e s o f e x p r e s s i o n st h a t w e c o n s i d e r , s i n c e i n t h e s e c a s e s p r o g r a m s f o r t h ec o n c e p t s f o l l o w d i r e c t l y f r o m t h e s p e c i f i c a t i o n o f t h ee x p r e s s i o n s . F o r t h e s a k e o f g e n e r a l i t y , o u r d e f i n i t i o n sd o a l l ow t h e v a l u e o f a p r o g r a m t o b e u n d e f i n e d , s i n c ea n o n t o t a l v e c t o r w i l l n o t , i n g e n e r a l , d e t e r mi n e t h ev a l u e o f a n e x p r e s s i o n a s 0 o r 1 .I t w o u l d b e i n t e r e s t i n g t o o b t a i n n e g a t i v e r e s u l t s ,o t h e r t h a n t h e i n f o r m a l c r y p t o g r a p h i c e v i d e n c e m e n -t i o n e d i n t h e i n t r o d u c t i o n , i n d i c a t i n g t h a t t h e c l a s s o fu n r e s t r i c t e d B o o l e a n c i r c u i t s i s n o t l e a r n a b l e . A r e c e n tr e s u l t [ 6 ] i n t h is d i r e c t i o n d o e s e s t a b l is h t h i s, u n d e r t h eh y p o t h e s i s t h a t i n t e g e r f a c t o r i z a t i o n i s i n t r a c t a b l e .T h e i r r e s u l t a p p l i e s t o t h e l e a r n i n g p r o t o c o l c o n s i s t i n go f E X A M P L E S a n d O R A C L E , w h e n t h e l a t t er i s re -s t r i c t e d t o t o t a l v e c t o r s . I t m a y a l s o b e i n t e r e s t i n g t o g oi n t h e o p p o si t e d i r e c t io n a n d d e t e r m i n e , f o r e x a m p l e ,w h e t h e r t h e h y p o t h e s i s t h a t P e q u a l s N P w o u l d i m p l yt h a t a l l i n t e r e s t i n g c l a s s e s a r e l e a r n a b l e .4 . A C O M B I N A T O R I A L B O U N DT h e p r o b a b i l i s t i c a n a l y s i s n e e d e d f o r o u r l a te r r e s u l t sc a n b e a b s t r a c t e d i n t o a s i ng l e l e m m a . T h e p r o o f o f t h el e m m a i s in d e p e n d e n t o f t h e r e s t o f t h e p a p e r .

W e d e f i n e t h e f u n c t i o n L ( h ,s ) f o r a n y r e a l n u m b e r hg r e a t e r t h a n o n e a n d a n y p o s i t i v e i n t e g e r S a s fo l l o w s :L e t L ( h , S ) b e t h e s ma l l e s t i n t e g e r s u c h t h a t i n L ( h , S )i n d e p e n d e n t B e r n o u l l i t r i a l s e a c h w i t h p r o b a b i l i t y a tl e a s t h -1 o f s u c c e s s , t h e p r o b a b i l i t y o f h a v i n g f e w e rt h a n S s u c c e s s e s i s l e s s t h a n h - 1.T h e r e l e v a n c e o f t h e f u n c t i o n L ( h , S ) t o l e a r n i n g c a nb e i ll u s t r a t e d a s f o l l o w s : C o n s i d e r a n u r n U c o n t a i n i n ga v e r y l a r ge n u m b e r o f m a r b l e s p o s s i b ly o f m a n y d i ff e r-e n t t y p es . S u p p o s e w e w i s h t o " l e a r n " w h a t k i n d s o fm a r b l e s U c o n t a i n s b y t a k in g a s m a l l r a n d o m s a m p l e X .I f a l l t h e m a r b l e s i n U a r e o f d i s t i n c t t y p e s , t h e n i t isc l e a r l y i m p o s s i b l e t o d e d u c e m u c h i n f o r m a t i o n a b o u tt h e m . S u p p o s e , h o w e v e r , t h a t w e k n o w t h a t t h e r e a r eo n l y S t y p e s o f m a r b l e s a n d t h a t w e w i s h t o c h o o s e X s ot h a t i t c o n t a i n s r e p r e s e n t a t i v e s o f a l l b u t 1 p e r c e n t o f

t h e m a r b l e s i n U . W h a t e v e r t h e r e s p e c t i v e f r e q u e n c i e so f t h e S t y p e s m a y b e , t h e d e f i n i ti o n o f L ( h , S } i mp l i e st h a t i f i X [ = L ( 1 0 0 , S ) t h e n w i t h p r o b a b i l i t y g r e a t e r t h a n9 9 p e r c e n t w e w i l l h a v e s u c c e e d e d . T o se e w h y t h i si mp l i c a t i o n h o l d s , s u p p o s e t h a t a f t e r L ( 1 0 0 , S } s u c c e s s i v ec h o i c e s m a r b l e t y p e s r e p r e s e n t i n g b e t w e e n t h e m a tl e as t 1 p e r c e n t o f th e u r n r e m a i n u n c h o s e n . I n t h a t c a s ew e h a v e ma d e L [ 1 0 0 , S ) B e r n o u l l i t r i a l s e a c h w i t h p r o b -a b i l i t y a t l e a s t 1 p e r c e n t o f s u c c e s s 0 . e . , o f c h o o s i n g ap r e v i o u s l y u n c h o s e n t yp e } a n d h a v e s u c c e e d e d f e w e rt h a n S ti m e s ( s i n ce s o m e t y p e r e m a i n s u n c h o s e n } . N o t et h a t w h a t c o n s t i t u t e s a s u c c e s s h e r e w i l l d e p e n d o np r e v i o u s c h o i c e s . T h e p r o b a b i l i ty o f s u c c e ss , h o w e v e r ,w i l l a l w a y s b e a t l e as t 1 p e r c e n t, i n d e p e n d e n t o f p r e -v i o u s c h o i c e s , a n d t h i s is s u f f i c i e n t f o r o u r a r g u m e n t .

T h e f o l l o w i n g s i m p l e u p p e r b o u n d h o l d s f o r t h ew h o l e r a n g e o f v a l u e s o f S a n d h a n d s h o w s t h a t L ( h , S )i s e s s e n t i a l l y l i n e a r b o t h i n h a n d i n S .

P R O P O S I T I O N : F o r a l l i n t eg er s S >_ 1 a n d a l l r ea lh > l .

L(h,S) _< 2h(S + log~h).P r o o f : W e u s e t h e fo l l o w i n g t h r e e w e l l - k n o w n i n e -q u a l i t i e s o f w h i c h t h e l a s t i s d u e t o C h e r n o f f ( se e [ 6 ]page 18 ) .1 . F o r a l l x > 0 , (1 + x -l } x < e .2. F o r a l l x > 0 , ( 1 - x - 1 } x < e - I.3 . I n m i n d e p e n d e n t t ri a l s, e a c h w i t h p r o b a b i l i t y a t

l e a s t p o f s u c c e s s , t h e p r o b a b i l i t y t h a t t h e r e a r e a tm o s t k s uc c e s s , w h e r e k < m p , i s a t mo s t

m -

T h e f i r s t f a c t o r i n t h e e x p r e s s i o n i n ( a ) a b o v e c a n b er e w r i t t e n a s 1 m p - - k~ ((m-k)/(mp-k}}{mp-k}F/U s i n g ( 2) w i t h x = ( m - k ) / ( m p - k } , w e c a n u p p e rb o u n d t h e p r o d u c t b y

e -m p + k(m p / k ) k .S u b s t i t u t i n g m = 2 h ( S + log~h), p = h -1 , an d k = S andu s i n g l o g a r i th m s t o t h e b a s e e g iv e s t h e b o u n d

e-2S-21ogh+S. [2(1 + (tog h)/S)] (s/lg h)logh.R e w r i t i n g t h i s u s i n g ( 1} w i t h x = ( l o g ~ h } / S g ives

e-S-21ogh. 2 s. elOgh _< (2/e)S . e-log h < h-1.A s a n i l l u s t ra t i o n o f a n a p p l i c a t i o n , s u p p o s e t h a t w e

h a r e a c c e s s to E X A M P L E S , a s o u r c e o f n a t u r a l p o s i ti v ee x a m p l e s o f v e c to r s f o r c o n c e p t F . S u p p o s e t h a t w ew a n t t o f i n d a n a p p r o x i m a t i o n t o th e s u b s e t P o f v a ri -a b l es t h a t a r e d e t e r m i n e d i n a t l e as t o n e n a t u r a l e x a m -p l e o f F . C o n s i d e r t h e f o l l o w i n g p r o c e d u r e : P i c k t h e f i r stL ( h , I P I} v e c t o r s p r o v i d e d b y E X A M P L E S , a n d l e t P ' b et h e u n i o n o f t h e s e t s o f v a r i a b le s d e t e r m i n e d b y t h e s ev e c t o r s . T h e d e f i n i t i o n t h e n i mp l i e s t h a t w i t h p r o b a b i l -i t y a t l e a st 1 - h - 1 t h e s e t P ' w i l l h a v e t h e f o l l o w i n g

N o vem b er 1984 Vo lu m e 27 N u m b er 11 C o m m u n ica tio ns o f th e AC M 1137


5/9


p r o p e r t y : I f a v e c t o r i s c h o s e n w i t h p r o b a b i l i t y d i s t r i b u -t i o n D , t h e n t h e p r o b a b i l i t y t h a t i t d e t e r mi n e s a v a r i -a b l e in P - P ' i s l e s s t h a n h - 1 . T h e a r g u m e n t u s e da b o v e f o r t h e u r n mo d e l e s t a b l i s h e s t h i s a s fo l l o w s : If P 'd o e s n o t h a v e t h e c l a i m e d p r o p e r t y , t h e n t h e f o l l o w i n gh a s h a p p e n e d : L ( h , [P [ ) t r i a l s hav e bee n m ad e ( i .e . , c a l l so f E X A M P L E S ) e a c h w i t h p r o b a b i l i t y g r e a t e r t h a n h - 1o f s u c c e s s ( i .e ., f i n d i n g a v e c t o r t h a t d e t e r m i n e s a v a r i -a b l e i n P - P ' , b u t t h e r e h a v e b e e n f e w e r t h a n [ P ]s u c c e s s e s ( i .e ., d i s c o v e r i e s o f n e w a d d i t i o n s t o P ' ) . T h ep r o b a b i l i t y o f s u c h a h a p p e n i n g i s l e ss t h a n h - 1 b y t h ed e f i n i t i o n o f L.

T h e a b o v e a p p l i c a t i o n s h o w s t h a t t h e s e t o f v a r i a b le st h a t a re d e t e r m i n e d i n n a t u r a l e x a m p l e s o f F c a n b ea p p r o x i m a t e d b y a p r o c e d u r e w h o s e r u n t i m e i s i n d e -p e n d e n t o f t , t h e t o t a l n u m b e r o f v a r i a b l e s . I t f o l l o w st h a t i n t h e d e f i n i t i o n O f l e a r n a b i l it y t h e d e p e n d e n c e o nt i n p a r t ( 1} c a n b e r e l a x e d t o a d e p e n d e n c e o n t h en u m b e r o f v a r i a b le s t h a t a r e e v e r d e f i n e d i n p o s it i v ee x a m p l e s .5. BOU NDED CNF EXPRESSIONSA c o n j u n c t i v e n o r m a l f o r m ( C N F ) e x p r e s s i o n i s a n y p r o d -u c t c~c2 Cr o f c l a u s e s w h e r e e a c h c l a u s e c i i s a sumq~ + q2 + + qi , of l it e ra l s . A l i t e r a l q i s e i t h e r av a r i a b l e p o r t h e n e g a t i o n p o f a v a r i a b l e . F o r e x a m p l e ,(p~ + #2)(p~ + f i2 + pa) is a CNF expression. For ap o s i t i v e i n t e g e r k , a k - C N F e x p r e s s i o n i s a C N F e x p r e s -s i o n w h e r e e a c h c l a u s e i s t h e s u m o f a t mo s t k l i te r a l s.

F o r C N F e x p r e s s io n s i n g e n e r a l, w e d o n o t k n o w o fa n y p o l y n o m i a l t i m e d e d u c t i o n p r o c e d u r e w i t h r e s p e c tt o o u r l e a r n i n g p r o t o c o l. T h e q u e s t i o n o f w h e t h e r o n ee x i s t s is t a n t a l i z i n g b e c a u s e o f i t s a p p a r e n t s i mp l i c i t y .

I n th i s s e c t i o n w e s h a l l p r o v e t h a t f o r a n y f i x e d i n t e -g e r k t h e c l a s s o f k - C N F e x p r e s s i o n s i s le a r n a b l e . I n f a c ti t i s l e a r n a b l e e a s i l y i n t h e s e n s e t h a t c a l l s o f E X A M -P L E S s uf f ic e a n d n o c a ll s O f O R A C L E a r e r e q u i r e d .I n th i s a n d s u b s e q u e n t s e c t io n s , w e s h a l l u s e t h e r e -l a t i o n ~ o n c o n c e p t s a s f o l l o w s : F ~ G me a n s t h a t f o ra l l v e c t o r s v w h e n e v e r F ( v ) = 1 i t is a ls o t h e c a s e t h a tG(v} = i . ( N .B . T h i s i s e q u i v a l e n t t o t h e r e l a t i o n F ~ Gw h e n F , G a r e r e g a r d e d a s f u n c t io n s . ) F o r b r e v i t y w es h a ll o f t e n d e n o t e b o t h e x p r e s s i o n s a n d e v e n v e c t o r s b yt h e c o n c e p t s t h e y r e p r e s e n t . T h u s t h e c o n c e p t r e p r e -s e n t e d b y v e c t o r w i s t r u e f o r e x a c t l y t h o s e v e c t o r s t h a ta g r e e w i t h w o n a l l v a ri a b l e s o n w h i c h w i s d e t e r m i n e d .T h e c o n c e p t r e p r e s e n t e d b y e x p r e s s i o n f i s o b t a i n e d b yc o n s i d e r i n g t h e B o o l e a n f u n c t i o n c o m p u t e d b y f a n de x t e n d i n g i ts d o m a i n t o b e c o m e a c o n c e p t . I n t h is s e c -t i o n w e r e g a r d a c l a u s e c l t o b e a s p e c i a l k i n d o f e x p r e s -s io n . I n S e c t i o n s 7 a n d 8 , w e c o n s i d e r m o n o m i a l s m ,s i mp l e p r o d u c t s o f li t e ra l s , a s s p e c i a l f o r m s o f e x p r e s -s i o n s . I n a l l c a s e s s t a t e m e n t s o f t h e f o r m f ~ g , v ~ ci ,v ~ m , o r m ~ f c a n b e u n d e r s t o o d b y i n t e r p r e t in g t h et w o s i d e s a s c o n c e p t s o r f u n c t i o n s i n t h e o b v i o u s w a y .T H E O R E M A : F o r a n y p o s i t i v e i n t e g e r k, t h e c l a s s o fk - C N F e x p r e ss i o n s i s le a r n a b l e v ia a n a l g o r i t h m A t h a t u s e sL = L ( h , (2t) kl) c a ll s o f E X A M P L E S a n d n o c a l ls o f O R A -C L E , w h e r e t i s t h e n u m b e r o f v a ri a b l e s.P r o o f : T h e a l g o r i t h m i s i n i ti a l iz e d w i t h f o r m u l a g a s

t h e p r o d u c t o f a l l p o s s ib l e c l a u s e s o f u p t o k l i t e ra l sf r o m {p l , p l , p 2 ,/ ~2 . . . . . p t , # t} . C l e a r l y t h e n u m b e r o fw a y s o f c h o o s i n g a c l a u s e o f e x a c t l y k li t e r a ls i s a t mo s t(2t}k, a n d h e n c e t h e n u m b e r o f w a y s o f c h o o s i n g ac lause wi th up to k l i t e ra l s i s l e s s than 2 t + (2 t) 2 + + (2t) k < {2t}k +l . T h i s b o u n d s t h e i n i ti a l n u m b e r o fc l a u s e s i n g .

T h e a l g o r i t h m t h e n c a l l s E X A M P L E S L t i m e s t o g i v ep o s i ti v e e x a m p l e s o f t h e c o n c e p t r e p r e s e n t e d b y k - C N Ff o r m u l a f. F o r e a c h v e c t o r v s o o b t a i n e d , i t d e l e t e s a l l o ft h e r e m a i n i n g c l a u s e s i n g t h a t d o n o t c o n t a i n a l it e r alt h a t i s d e t e r m i n e d t o b e t r u e i n v (i .e ., i n t h e c l a u s e sd e l e t e d , e a c h l i t e r a l i s e i t h e r n o t d e t e r mi n e d o r i s n e -g a t e d i n v }. M o r e p r e c i s e l y t h e f o l l o w i n g is r e p e a t e d Lt i me s :

begin v : = E X A M P L E Sf o r e a c h c i i n g d e l e t e ci if v ~ cl .

e n dT h e c l a i m i s t h a t w i t h h i g h p r o b a b i l i t y t h e v a l u e o f ga f te r t h e L t h e x e c u t i o n o f t h is b l o c k w i l l b e t h e d e s i r e da p p r o x i m a t i o n t o th e f o r m u l a f t h a t i s b e i n g l e a r n e d .I n i t i a l l y t h e s e t o f v e c t o r s I v [ v ~ g } i s c l e a r l y e m p t y .W e f i r st c l a i m t h a t a s t h e a l g o r i t h m p r o c e e d s t h e s e t[ v [v ~ g } w i l l a l w a y s b e a s u b s e t o f { v I v ~ f } . T op r o v e t h i s i t i s c l e a r l y s u f f i c i e n t t o p r o v e t h e s a mes t a t e m e n t w h e n b o t h s e t s a r e r e s t r i c te d t o o n l y t o ta lv e c t o r s . L e t B b e t h e p r o d u c t o f a l l t h e c l a u s e s c c o n -t a i n i n g u p t o k l i t e ra l s w i t h t h e p r o p e r t y t h a t " V v i fv ~ f t h e n v ~ c . " I t i s c l e a r t h a t i n t h e c o u r s e o f t h ea l g o r i t h m n o c l a u s e i n B c a n e v e r b e d e l e t ed . H e n c e i ti s ce r t a in ly t rue tha t {v [ v ~ g} C { v [ v ~ B } . T o e s t a b -l is h t h e c l a i m , i t r e m a i n s o n l y t o p r o v e t h a t B c o m p u t e st h e s a me B o o l e a n f u n c t i o n a s f . ( In fa c t B w i l l b e t h ema x i m a l k - C N F f o r m u l a e q u i v a l e n t t o f .) i t i s e a s y t os e e t h a t f ~ B , s i n c e b y t h e d e f i n i t i o n o f B , f o r e v e r y cin B i t i s the case tha t " fo r a l l v i f v ~ f the n v ~ c ." Tov e r i f y t h e c o n v e r s e , t h a t B ~ f , w e f i r st n o t e t h a t , b yd e f i n i t i o n , f i s a k - C N F f o r m u l a . I f s o m e c l a u s e i n f d i dn o t o c c u r i n B, t h e n t h i s c l a u se c ' w o u l d h a v e t h ep r o p e r t y th a t " 3 v s u c h t h a t v ~ f b u t v ~ c ' . " B u t t h i si s i mp o s s i b l e s i n c e i f c ' i s a c l a u s e o f f a n d i f v ~-~ c't h e n v ~ f . W e c o n c l u d e t h a t e v e r y k - C N F re p r e s e n t a -t i o n o f t h e f u n c t i o n t h a t f r e p r e s e n t s c o n s i s t s o f a s e t o fc l a u s e s t h a t i s a s u b s e t o f t h e c l a u s e s i n B . H e n c e B ~ f .

L e t X = ~ D ( v ) w i t h s u m m a t i o n o v e r a ll (n o t n e c e s -s a r i l y t o ta l ) v e c t o r s v s u c h t h a t v ~ f b u t v ~ g . T h i sq u a n t i t y i s d e f i n e d f o r e a c h i n t e r m e d i a t e v a l u e o f g i nt h e c o u r s e o f t h e a l g o r i t h m a n d , a s i s e a s i l y v e r i f ie d , i ti s m o n o t o n e d e c r e a s i n g w i t h t i m e . N o w c l a u s e s w i l l b er e m o v e d f r o m g w h e n e v e r E X A M P L E S o u t p u t s a v e c to rv s u c h t h a t v ~ g . T h e p r o b a b i l i t y o f t h i s h a p p e n i n g a ta n y m o m e n t i s e x a c t l y t h e c u r r e n t v a l u e o f X . A l s o , t h ep r o c e s s o f r u n n i n g t h e a l g o r i t h m t o c o m p l e t i o n c a nh a v e o n e o f ju s t t w o p o s s ib l e o u t c o m e s : ( 1) A t s o m ep o i n t X b e c o m e s l e s s t h a n h - I , i n w h i c h c a s e t h e gf o u n d w i l l a p p r o x i m a t e t o f e x a c t l y a s re q u i r e d b y t h ed e f i n i t i o n o f l e a r n a b i l i t y . ( 2) T h e v a l u e o f X n e v e r g o e sb e l o w h - 1, a n d h e n c e g is n o t a n a c c e p t a b l e a p p r o x i m a -t i o n . T h e p r o b a b i l i t y o f t h e l a t t e r e v e n t u a l i t y i s , h o w -

1138 C o m m u n ica t io n s o f th e AC M N o vem b er 1984 Vo lu m e 27 N u m b er 11


6/9


e v e r , a t m o s t h - 1 s i n c e i t c o r r e s p o n d s t o t h e s i t u a t i o n o fp e r f o r m i n g L ( h , ( 2 t) k +l ) B e r n o u l l i e x p e r i m e n t s ( i .e . , c a l l so f E X A M P L E S ) e a c h w i t h p r o b a b i l i t y g r e a t e r t h a n h - 1o f s u c c e s s ( i .e ., f i n d i n g a v s u c h t h a t v ~ g ) a n d o b t a i n -i n g f e w e r t h a n ( 2 t) k +l s u c c e s s e s ( a s u c c e s s b e i n g m a n i -f e s t ed b y t h e r e m o v a l o f a t l e a s t o n e c l a u s ef rom g ) . [ ]

I n c o n c l u s i o n w e o b s e r v e t h a t a C N F e x p r e s s i o n gi m m e d i a t e l y y i e l d s a p r o g r a m f o r c o m p u t i n g t h e a s s o c i-a t e d c o n c e p t . G i v e n a v e c t o r v , w e s u b s t i t u t e t h e d e t e r -m i n e d t r u t h v a l u e s i n g . T h e c o n c e p t w i l l b e t r u e f o r vi f a n d o n l y i f a l l th e c l a u s e s a r e m a d e t r u e b y t h es u b s t i t u t i o n .6 . D N F E X P R E S S I O N SA d i s j u n c t i v e n o r m a l fo r m ( D N F ) e x p r e s s i o n i s a n y s u mm l + m 2 + + m r o f m o n o m i a l s w h e r e e a c h m o n o m -i a l m i i s a p r o d u c t o f l i t e r a ls . F o r e x a m p l e , p l/ ~ 2 + p l p 3 p 4i s a D N F e x p r e s s i o n . S u c h e x p r e s s i o n s a p p e a r p a r t i c u -l a r l y e a s y fo r h u m a n s t o c o m p r e h e n d . H e n c e w e e x p e c tt h a t a n y p r a c t i c a l l e a r n i n g s y s te m w o u l d h a v e t o a l l o wf o r t h e m .

A n e x p r e s s i o n i s m o n o t o n e i f n o v a r i a b l e i s n e g a t e d i ni t. W e s h a l l sh o w t h a t f o r m o n o t o n e D N F e x p r e s s i o n st h e r e e x i s t s a s i m p l e d e d u c t i o n p r o c e d u r e t h a t u s e sb o t h E X A M P L E S a n d O R A C L E . F o r u n r e s t r i c t e d D N F as i m i l a r r e s u l t c a n b e p r o v e d w i t h r e s p e c t t o a d if f e r e n ts i z e m e a s u r e . I n t h e u n r e s t r i c t e d c a s e , t h e r e i s t h e a d d i -t i o n a l d if f i c u l ty t h a t w e c a n g u a r a n t e e t o d e d u c e a p r o -g r a m o n l y f o r t h e f u n c t i o n a n d n o t f o r t h e c o n c e p t . T h i sd i f f ic u l t y d o e s n o t a r i s e i n t h e m o n o t o n e c a s e w h e r eg i v e n an e x p r e s s i o n w e c a n a l w a y s c o m p u t e t h e v a l u eo f t h e a s s o c i a t e d c o n c e p t . f o r a v e c t o r v b y m a k i n g t h es u b s t i t u t i o n a n d a s k i n g w h e t h e r t h e r e s u l t i n g e x p r e s -s i o n i s i d e n t i c a l l y t r u e .

A m o n o m i a l m i s a p r i m e i m p l i c a n t o f a f u n c t i o n F ( oro f a n e x p r e s s i o n r e p r e s e n t i n g t h e f u n c t io n ) i f m ~ Fa n d i f m ' ~,~ F fo r a n y m ' o b t a i n e d b y d e l e t i n g o n el i t e r a l f r o m m . A D N F e x p r e s s i o n i s p r i m e i f i t c o n s i s t so f t h e s u m o f p r i m e i m p l i c a n t s n o n e o f w h i c h i s r e d u n -d a n t i n t h e s e n s e o f b e i n g i m p l i e d b y t h e s u m o f t h eo t h e rs . T h e r e i s a lw a y s a u n i q u e p r i m e D N F e x p r e s s i o ni n t h e m o n o t o n e c a s e, b u t n o t i n g e n e r a l . W e t h e r e f o r ed e f i n e t h e d e g r e e o f a D N F e x p r e s s i o n t o b e t h e l a r g e s tn u m b e r o f m o n o m i a l s t h a t a p r i m e D N F e x p r e s s i o ne q u i v a l e n t t o i t c a n h a v e . T h e u n i q u e p r i m e D N Fe x p r e s s i o n i n th e m o n o t o n e c a s e is s i m p l y t h e s u m o fa l l t h e p r i m e i m p l i c a n t s .T H E O R E M B : T h e c l a s s o f m o n o t o n e D N F e x p r e s s i o n s i sl e a r n a b l e v i a a n a l g o r i t h m B t h a t u s e s L = L ( h , d ) c a l l s o fE X A M P L E S a n d d t c a ll s o f O R A C L E , w h e r e d i s th e d e g r eeo f t h e D N F e x p r e s s io n f t o b e l e a r n e d a n d t t h e n u m b e r o fv a r i a b l e s .P r o o f : T h e a l g o r i t h m i s i n i t i a l i z e d w i t h f o r m u l a gi d e n t i c a l l y e q u a l t o t h e c o n s t a n t z e r o . T h e a l g o r i t h mt h e n c a l l s E X A M P L E S L t i m e s t o p r o d u c e p o s i t i v e e x -a m p l e s o f f . E a c h t i m e a v e c t o r v i s p r o d u c e d s u c h t h a tv ~ g , a n e w m o n o m i a l m i s a d d e d t o g . T h e m o n o m i a lm i s t h e p r o d u c t o f t h o s e l i t e r a l s d e t e r m i n e d i n v t h a ta r e e s s e n t i a l t o m a k e v ~,~ f . M o r e p r e c i s e l y , t h e l o o p

t h a t i s e x e c u t e d L t i m e s i s th e f o l lo w i n g :b e g i n v : = E X A M P L E S

i f v ~ g t h e nb e g i n f o r i : = 1 t o t d o

i f p i i s d e t e r m i n e d i n v t h e nb e g i n s e t ~ e q u a l t o v b u t w i t h pi := *;

i f O R A C L E (W ) = 1 t h e n v : =e n ds e t m e q u a l t o t h e p r o d u c t o f a l l l i t e r a l s q

s u c h t h a t v ~ q ;g : = g + m

en de n d

T h e t e s t v ~ g a m o u n t s t o a s k i n g w h e t h e r n o n e o ft h e m o n o m i a l s o f g a re m a d e t r u e b y t h e v a l u e s d e t e r -m i n e d t o b e t ru e i n v . E v e r y t i m e E X A M P L E S p r o d u c e sa v a l u e v s u c h t h a t v ~ g , t h e i n n e r l o o p o f t h e a l g o -r i t h m w i l l f i n d a p r i m e i m p l i c a n t m t o a d d t o g . E a c hs u c h m i s d i f f e r e n t f ro m a n y p r e v i o u s l y a d d e d ( s i n c et h e c o n t r a r y w o u l d h a v e i m p l i e d t h a t v ~ g ). I t f o l lo w st h a t s u c h a v w i l l b e f o u n d a t m o s t d ti m e s , a n d h e n c eO R A C L E w i l l h e c a l l e d a t m o s t d t t i m e s .

L e t X = ~ D ( w ) w i t h s u m m a t i o n o v e r a l l ( n o t n e c e s -s a r i l y to t a l ) v e c t o r s w s u c h t h a t w ~ f b u t w ~ g . T h i sq u a n t i t y i s d e f i n e d f o r e a c h i n t e r m e d i a t e v a l u e o f g i nt h e c o u r s e o f t h e a l g o r i t h m , i s i n i t i a l l y u n i t y , a n d d e -c r e as e s m o n o t o n i c a l ly w i t h t i m e . N o w a m o n o m i a l w i l lb e a d d e d t o g e a c h t i m e E X A M P L E S o u t p u t s a v e c t o r vs u c h t h a t v ~-~ g . T h e p r o b a b i l i t y o f t h i s o c c u r r i n g a ta n y c a l l o f E X A M P L E S i s e x a c t l y X . T h e p r o c e s s o fr u n n i n g t h e a l g o r i t h m t o c o m p l e t i o n c a n h a v e j u s t t w op o s s i b l e o u t c o m e s : (1 ) A t s o m e t i m e X h a s b e c o m e l e s st h a n h - 1, i n w h i c h c a s e t h e f i n a l e x p r e s s i o n g f o u n dw i l l a p p r o x i m a t e t o f a s r e q u i r e d b y t h e d e f i n i t i o n o fl e a r n a b i l i t y ; a n d ( 2 ) t h e v a l u e o f X n e v e r g o e s b e l o wh - 1, a n d h e n c e g i s n o t a n a c c e p t a b l e a p p r o x i m a t i o n .T h e p r o b a b i l i t y o f t h i s s e c o n d e v e n t u a l i t y i s, h o w e v e r ,a t m o s t h - 1 , s i n c e i t c o r r e s p o n d s t o t h e s i t u a t i o n o fp e r f o r m i n g L ( h , d ) B e r n o u l l i e x p e r i m e n t s ( i. e. , c a l l s o fE X A M P L E S} e a c h w i t h p r o b a b i l i t y g r e a t e r t h a n h - 1 o fs u c c e s s ( i . e ., o f f i n d i n g a v s u c h t h a t v ~ f b u t v ~ g )a n d o b t a i n i n g f e w e r t h a n d s u c c e s s e s ( e a c h m a n i f e s t e db y t h e a d d i t i o n o f a n e w m o n o m i a l ) . [ ]

F o r u n r e s t r i c t e d D N F e x p r e s s i o n s , s e v e r a l p r o b l e m sa r i s e . T h e m a i n s o u r c e o f d i f f i c u l t y i s t h e f a c t t h a t t h ep r o b l e m o f d e t e r m i n i n g w h e t h e r a n o n t o ta l v e c t o r i m -p l i e s t h e f u n c t i o n s p e c i f i e d b y a D N F f o r m u l a i s N P -h a r d . T h i s i s s i m p l y b e c a u s e t h e p r o b l e m o f d e t e r m i n -i n g w h e t h e r t h e n o w h e r e d e t e r m i n e d v e c t or i m p l i est h e f u n c t i o n i s t h e t a u t o l o g y q u e s t i o n o f C o o k [ 3] . A ni m m e d i a t e c o n s e q u e n c e o f t h i s is t h a t i t i s u n r e a s o n a -b l e t o a s s u m e t h a t t h e p r o g r a m b e i n g l e a r n e d c o m p u t e sc o n c e p t s r a t h e r t h a n f u n c t i o n s . A n o t h e r c o n s e q u e n c ei s t h a t i n a n y a l g o r i t h m a k i n t o A l g o r i t h m B t h e t e s tv ~ g m a y n o t b e f e a s i b l e i f v is n o t t o t a l . A l g o r i t h m Bi s s u f f i c i e n t , h o w e v e r , t o e s t a b l i s h t h e f o l l o w i n g :T H E O R E M B ' : S u p p o s e t h e n o t i o n o f l e a r n a b i l i t y i s r e -s t r ic t e d t o d i s t r ib u t i o n s D s u c h t h a t D ( v ) = 0 w h e n e v e r v i sn o t t o t a l . T h e n t h e c l a s s o f D N F e x p r e s s i o n s i s l e a r n a b l e , i n

N o vem b er 198 4 Vo lu m e 27 N u m b er 11 C o m m u n ica t io ns o f th e AC M 1139


7/9


t h e s e n s e t h a t p r o g r a m s f o r t h e f u n c t i o n s ( b u t n o t n e c e s s a r -i l y the concep t s ) can be deduced in L (h ,d ) ca l l s o f EX AM -P L E S a n d d t c a l l s o f O R A C L E w h e r e d i s t h e d e g r e e o f t h eDN F expr es s ion to be l ear ned .

T h e r e a d e r s h o u l d n o t e t h a t h e r e t h e r e i s t h e a d d i -t i o n a l p h i l o s o p h i c a l d i f f i c u lt y t h a t a v a i l a b i l i t y o f t h ee x p r e s s i o n d o e s n o t i n i t s e l f i m p l y a p o l y n o m i a l t i m ea l g o r i t h m f o r O R A C L E . O n t h e o t h e r h a n d , t h e t h e o r e md o e s s ay t h a t i f a n a g e n t h a s s o m e , m a y b e a d h o c , b l a c kb o x fo r O R A C L E w h o s e w o r k i n g s a r e u n k n o w n t o h i mo r h e r, i t c a n b e u s e d t o t e a c h s o m e o n e e l s e a D N Fe x p r e s s i o n t h a t a p p r o x i m a t e s t h e f u n c t i o n .

F i n a l l y , i t m a y b e w o r t h e m p h a s i z i n g t h a t t h e m o n o -t o n e c a se i s n o n p r o b l e m a t i c a n d t h e d e d u c t i o n p r o c e -d u r e f o r i t v e r y s i m p l e m i n d e d . I t m a y b e i n t e r e s t i n g t op u r s u e m o r e s o p h i s t i c a t e d d e d u c t i o n p r o c e d u r e s f o r i ti f c o n t e x t s c a n b e f o u n d i n w h i c h t h e y c a n b e p r o v e da d v a n t a g e o u s . T h e q u e s t i o n a s t o w h e t h e r m o n o t o n eD N F e x p r e s s i o n s c a n b e l e a r n e d f r o m E X A M P L E S a l o n ei s o p e n . A p o s i t i v e a n s w e r w o u l d b e e s p e c i a l l y s ig n i f i-c a n t . A p a r t i a l r e s u l t i n t h is d i r e c t i o n c a n b e o b t a i n e db y d u a l i z i n g T h e o r e m A : F o r a n y k t h e c l a s s o f D N Fe x p r e s s i o n s h a v i n g a b o u n d k o n t h e l e n g t h o f e a c hd i s j u n c t c a n b e l e a r n e d i n p o l y n o m i a l t im e f r o m n e g a -t i v e e x a m p l e s a l o n e ( i . e. , f r o m a s o u r c e o f v e c t o r s vs u c h t h a t / 3 ( v ) # 0 ) .7 . a -E X P RE S S I O NSW e h a v e a l r e a d y s e e n a c l a s s o f e x p r e s s i o n s , n a m e l yt h e k - C N F e x p r e s s io n s , t h a t c a n b e l e a r n e d f r o m p o s i -t i v e e x a m p l e s a l o n e . H e r e w e s h a l l c o n s i d e r t h e o t h e re x t r e m e , a c l as s th a t c a n b e l e a r n e d u s i n g o r a c l e sa l o n e .

D e d u c i n g e x p r e s s i o n s o n w h i c h t h e r e a r e l e s s s e v e r es t r u c t u r a l r e s t r i c t i o n s t h a n D N F o r C N F a p p e a r s m u c hm o r e d i f f i c u l t . T h e a i m o f th i s s e c t i o n i s t o g iv e a p a r a -d i g m a t i c e x a m p l e o f h o w f a r o n e c a n g o i n t h a t d i r e c -t i o n i f o n e i s w i l l i n g t o p a y t h e p r i c e o f o r a c l e s t h a t a r em o r e s o p h i s t i c a t e d t h a n t h e o n e p r e v i o u s l y d e s c r i b e d .

A g e n e r a l expr es s ion o v e r v a r i a b l e s e t { p l . . . . . pt} isd e f i n e d r e c u r s i v e l y a s f o ll o w s :1 . F o r eac h i (1 _< i _< t ) "p l" an d "Pi" a r e e x p r e s s i o n s ;2 . i ff ~ . . . . . f r a r e e x p r e s s i o n s , t h e n " ( f l + f2 + . - " +

fr)" i s a n e x p r e s s i o n ( c a l l e d a p l u s e x p r e s s i o n ) ;3 . i f f l . . . . . f i a r e e x p r e s s i o n s , t h e n " ( f l x f 2 x . . . x

f r) " i s a n e x p r e s s i o n ( c a l l e d a t i m e s e x p r e s s i o n ) .A a-expr es s ion i s a n e x p r e s s i o n i n w h i c h e a c h v a r i -

a b l e a p p e a r s a t m o s t o n c e . W e c a n a s s u m e t h a t a r -e x p r e s s i o n i s m o n o t o n e , s i n c e w e c a n a l w a y s r e l a b e ln e g a t e d v a r i a b l e s w i t h n e w n a m e s t h a t d e n o t e t h e i rn e g a t i o n . I n t h e r e c u r s i v e d e f i n i t i o n o f a n y j u-e x p r e s s i o n , t h e r e a r e c l e a r l y at m o s t 2 t - 1 i n t e r m e d i -a t e e x p r e s s i o n s o f w h i c h a t m o s t t a r e o f t y p e (1 ) a n d a tm o s t t - 1 o f t y p e s ( 2) o r ( 3) . W i t h o u t l o s s o f g e n e r a l i t y ,w e s h a l l a s s u m e t h a t i n t h e d e f i n i t io n o f a t - e x p r e s s i o nr u l e s ( 2) a n d ( 3) a l t e r n a t e . W e s h a l l r e g a r d t w o / z -e x p r e s s i o n s a s i d e n t i c a l i f t h e i r d e f i n i t i o n s c a n b e m a d ef o r m a l l y th e s a m e b y r e o r d e r i n g s u m s a n d p r o d u c t s a n dr e l a b e l i n g a s n e c e s s a r y .

F o r l e a r n i n g a - e x p r e s s i o n s , w e s h a l l e m p l o y m o r ep o w e r f u l o r a c l e s . T h e b o u n d a r y b e t w e e n r e a s o n a b l ea n d u n r e a s o n a b l e o r a c l e s d o e s n o t a p p e a r s h a r p . W em a k e n o c l a i m s a b o u t t h e r e a s o n a b l e n e s s o f t h e s e n e wo r a c l e s e x c e p t t h a t t h e y m a y s e r v e a s v e h i c l e s f o r u n -d e r s t a n d i n g l e a r n a b i l i t y .

T h e d e f i n i t i o n s r e f e r t o t h e B o o l e a n f u n c t i o n F ( n o tr e g a r d e d a s a c o n c e p t h e r e ) . T h e o r a c l e o f S e c t i o n 2 w i l lb e r e n a m e d N - O R A C L E , s i n c e i t i s o n e o f n e c e s s i t y : N -O R A C L E ( v ) = 1 i f a n d o n l y i f fo r a l l to t a l v e c t o r s ws u c h t h a t w ~ v i t i s t h e c a s e t h a t F(w) = 1 . T h e d u a l o ft h i s w o u l d b e a pos s ib i l i t y or ac le : P - O R A C L E ( v ) = 1 i fa n d o n l y i f t h e r e e x i s t s a t o t a l v e c t o r w s u c h t h a t w ~ va n d F(w} = 1 . F o r b r e v i t y w e s h a l l a l s o d e f i n e t h e p r i m ei m p l i c a n t o r a c l e : P l ( v ) = 1 i f a n d o n l y i f N - O R A C L E ( v ) =1 , b u t N - O R A C L E ( w ) = 0 f o r a n y w o b t a i n e d f r o m v b ym a k i n g o n e v a r i a b le d e t e r m i n e d i n v u n d e t e r m i n e d .

F i n a l l y w e d e f i n e t w o f u r t h e r o r a c l e s, o n e s o f r e l e v a n tpos s ib i l i t y RP a n d o f r e l ev a n t a c c o m p a n i m e n t R A . F o r c o n -v e n i e n c e w e d e f i n e t h e f i r st b y h o w i t b e h a v e s o n v e c -t o r s r e p r e s e n t e d a s m o n o m i a l s : R P ( m ) = 1 i f f f r o m s o m em o n o m i a l m ' r a m ' i s a p r i m e i m p l i c a n t o f f . F o r s e t sV , W o f v a r i a b l e s , w e d e f i n e R A ( V , W ) = 1 if f e v e r yp r i m e i m p l i c a n t o f f t h a t c o n t a i n s a v a r i a b l e f r o m V a l soc o n t a i n s a v a r i a b l e f r o m W .T H E O R E M C : Th e class of in-express ions in learnab le viaa deduct ion pr ocedur e C tha t us es O( t3) c a l ls o f N - O R A C L ER P a n d R A a l t o g et h e r , w h e r e t i s t h e n u m b e r o f v a r ia b l e s( a n d n o ca l ls o f E X A M P L E S ) . T h e p r o c e d u r e a l w a y s d e d u c e sexac t l y the cor r ec t expr es s ion .P r o o f : L e t ~ b e t h e m o n o m i a l t h a t i s t h e p r o d u c t o f n ol i te r a l s . T h e a l g o r i t h m w i l l f i rs t c o m p u t e RP{pj) f o r e a c hp j t o d e t e r m i n e w h i c h o f t h e v a r i a b l e s o c c u r i n t h ep r i m e i m p l i c a n t s o f th e f u n c t i o n t h a t t h e e x p r e s s i o n f t ob e l e a r n e d r e p r e s e n t s. L e t g l . . . . . g r b e d i s t i n c t s i n g l ev a r i a b l e e x p r e s s i o n s , o n e f o r e a c h p j fo r w h i c h RP(p/) =1 . N o t e t h a t t h e s e a r e e x a c t l y t h e v a r i a b l e s t h a t o c c u ri n f, s i n c e f i s m o n o t o n e .

W i t h e a c h gi w e a s s o ci a t e t w o m o n o m i a l s m i a n d r h i .T h e f o r m e r w i l l b e t h e s i n g l e v a r i a b l e p j t h a t gi r e p r e -s e n ts . T h e l a t t e r i s d e f i n e d a s a n y m o n o m i a l rh h a v i n gn o v a r i a b l e i n c o m m o n w i t h m ~ s u c h t h a t m ~ rh i s ap r i m e i m p l i c a n t o f f . T h e a l g o r i t h m w i l l c o n s t r u c t e a c hs u c h r hi a s t h e f i n a l v a l u e o f m i n t h e f o l l o w i n g p r o c e -d u r e : S e t m = ~ ; w h i l e P I ( m m i ) = 0 f i n d a p ~ n o t i n m i n is u c h t h a t R P ( p k m m i ) = 1 a n d s e t m : = p k m . H e n c e i n a tm o s t t 2 c a l l s o f P I ( i .e . , t 3 c a l l s o f N - O R A C L E ) a n d t 2c a l l s o f R P v a l u e s f o r e v e r y rh i w i l l b e f o u n d .

O n c e i n i t i a l i z e d t h e a l g o r i t h m p r o c e e d s b y a l t e r -n a t e l y e x e c u t i n g a p l u s - p h a s e a n d a t i m e s - p h a s e . A t t h es t a r t o f e a c h p h a s e , w e h a v e a s e t o f e x p r e s s i o n s g l . . . . .gr w h e r e e a c h gi i s a s s o c i a t e d w i t h t w o m o n o m i a l s m ia n d r hi ( h a v i n g n o v a r i a b l e s i n c o m m o n ) w h e r e m i is ap r i m e i m p l i c a n t o f g i a n d m ir hi i s a p r i m e i m p l i c a n t o f f .W e d i s t i n g u i s h t h e g s as p l u s o r t i m e s e x p r e s s io n s a c c o r d -i n g t o w h e t h e r t h e o u t e r m o s t c o n s t r u c t i o n r u l e w a sa d d i t i o n o r m u l t i p l i c a t i o n . A p l u s - p h a s e w i l l fi r st c o m -p u t e a n e q u i v a l e n c e r e l a t i o n S o n t h e s u b s e t o f {gi} t h a ta r e t i m e s e x p r e s s i o n s . F o r e a c h e q u i v a l e n c e c l a s s Gs u c h t h a t n o gi E G a l r e a d y o c cu r s a s a s u m m a n d i n a

1140 C om mun ica t ions f t he AC M N ovember 1 9 8 4 Volume 2 7 N um ber 1 1


8/9


s u m , w e c o n s t r u c t a n e w e x p r e s s i o n t h a t i s t h e s u m o ft h e m e m b e r s o f G a n d c a l l t h i s s u m g k w h e r e k i s ap r e v i o u s l y u n u s e d i n d e x . I f s o m e m e m b e r s o f G a l r e a d yo c c u r i n a s u m, s a y g j ( N .B . t h e y a r e n e v e r d i s t r i b u t e di n m o r e t h a n o n e s u m ) , th e n w e m o d i f y th e s u me x p r e s s io n & t o eq u a l t h e s u m o f e v e r y e x p r e s s i o n i n G .A t i me s - p h a s e i s e x a c t l y a n a l o g o u s e x c e p t t h a t i t c o m-p u t e s a d i f f e r e n t e q u i v a l e n c e r e l a t i o n T , n o w o n p l u se x p r e s s i o n s , a n d w i l l f o r m n e w , o r e x t e n d o l d , t i me se x p r e s s i o n s . I n t h e a b o v e c o n t e x t , s i n g l e v a r i a b l ee x p r e s s i o n s w i l l b e r e g a r d e d a s b o t h t i me s a n d p l u se x p r e s s i o n s . A l s o , i t i s i mm a t e r i a l w h i c h o f t h e t w ok i n d s o f p h a s e i s u s e d t o s t a r t t h e a l g o r i t h m .

T h e i n t e n t i o n o f th e a l g o r i t h m i s b e s t e x p r e s s e d b yt h e c l a i m t o f o l l o w , w h i c h w i l l b e v e r i f i e d l a t e r . S u p -p o s e t h a t t h e e x p r e s s i o n s o c c u r r i n g i n t h e d e f i n i t io n o f fa re f~ . . . . . f q (w here q _< 2 t - 1}. W e sha l l s a y tha t g _< fii f f t h e s e t o f p r i me i mp l i c a n t s o f g i s a s u b s e t o f t h e s e to f p r i m e i m p l i c a n t s o f f i w h e r e ( 1) i f f i i s a p lus exp res -s i o n t h e n f i = fi a n d ( 2) i f f i is a t i me s e x p r e s s i o n t h e n f~i s t h e p r o d u c t o f s o m e s u b s e t o f t h e mu l t i p l i c a n d s o ff ~.C l a i m 1 : A f t e r e v e r y p h a s e o f t h e a l g o r i th m f o r e v e r yg~ t h a t h a s b e e n c o n s t r u c t e d , t h e r e i s e x a c t l y o n ee x p r e s s i o n f i s u c h t h a t ( 1) gi -< fi and (2 ) fi i s o f th e s a m ek ind ( i .e . , p lu s o r t imes ) a s gi .

T h e p r o c e d u r e b u i l d s u p t h e r o o t e d t r e e o f t h ee x p r e s s i o n a s r o o t e d s u b t r e e s s t a r t i n g f r o m t h e l e a v e s .O n e e v i d e n t d i f f i c u l t y i s t h a t t h e r e i s n o a p r i o r i k n o w l -e d g e a v a i la b l e a b o u t t h e s h a p e o f t h e t r e e . H e n c e i n t h eg r a f ti n g p r o c e s s a s u b t r e e m a y b e c o m e a t t a c h e d t o a n -o t h e r a t j u s t a b o u t a n y n o d e .

W h e n e v e r a p l u s o r t i m e s e x p r e s s i o n gi i s c r e a t e d o ren la rged , i t s a s soc ia ted m i a n d t hi a r e u p d a t e d a s f o l-lows . I f gi i s a s u m, t h e n w e l e t m i = m j and rhi = ~ j fo ra n y & t h a t i s a s u m m a n d i n g~ . I fg ~ i s a p r o d u c t , t h e n m~w i l l b e t h e p r o d u c t o f al l th e mj s t h a t c o r r e s p o n d t om u l t i p l i c a n d s i n g i. F i n a l l y r h i w i l l b e g e n e r a t e d a s t h ef i n a l v a l u e o f m i n t h e f o l l o w i n g p r o c e d u r e : S e t m : = ~ ;w h i l e P l ( m m i ) = 0 f ind a pk n o t i n m in i s u c h t h a tR P ( p k m m i ) = 1 a n d s e t m : = p k m . S i n c e s u c h a n r hi h a s t ob e f o u n d a t mo s t t t i me s i n t h e o v e r a l l a l g o r i t h m, t h eto ta l co s t i s a t m os t t 2 ca l l s o f P I ( i .e . , t 3 cal ls of N -O R A C L E ) a n d t 2 c a l ls o f R P .

I n o r d e r t o c o mp l e t e t h e d e s c r i p t i o n o f t h e a l g o r i t h m,i t r e ma i n s o n l y to d e f i n e t h e e q u i v a l e n c e r e l a t i o n s Sa n d T .Def in i t ion: giS& i f a n d o n l y i f1. PI(rnirhj) = PI(m jrhi) = 1 and2. mi , ~ j c o n t a i n d i s j o i n t s e t s o f v a r ia b l e s , a s d o mj , m i.

F o r d e f i n i n g T w e s h a l l d e n o t e b y V i t h e s e t o f v a r i -a b l e s t h a t o c c u r i n t h e e x p r e s s i o n g i.Def in i t ion: giTgj i f a n d o n l y i f RA( V i , V j ) = RA( V # V i )

F i r s t w e s h a l l v e r i f y t h a t S a n d T a r e i n d e e d e q u i v a -l e n c e r e l a t i o n s u n d e r t h e a s s u m p t i o n t h a t C l a i m 1h o l d s a t t h e b e g i n n i n g o f e a c h p h a s e . C l e a r l y S a n d T

a r e d e f in e d t o b e b o t h r e f le x i v e a n d s y m m e t r i c . T ov e r i f y t h a t T is t r a n s it i v e , s u p p o s e t h a t giTgj a n d & T g k .T h e f o r m e r i m p l i es t h a t e v e r y p r i m e i m p l i c a n t o f f c o n -t a i n i n g s o m e v a r i a b l e f r o m gi a l so c o n t a i n s s o m e v a r i -a b l e f r o m &. T h e l a t t e r i m p l i e s t h a t e v e r y p r i m e i m p l i -c a n t o f f c o n t a i n i n g s o m e v a r i a b l e f r o m & a ls o c o n t a i n sa v a r i a b l e f r o m gk . H e n c e giTgk f o l l o w s . I n o r d e r t o v e r -i f y t h e t r a n s i t i v i t y o f S , w e s h a l l m a k e a mo r e g e n e r a lo b s e r v a t i o n a b o u t a n y s u b e x p r e s s i o n o f f .C l a i m 2 : I f m i , m j a r e p r i m e i mp l i c a n t s o f d i s t i n c tt i m e s s u b e x p r e s s i o n s f i , f j o f f a n d i f f o r s o m e m , w i t h n ov a r i a b le i n c o m m o n w i t h e i t h e r m i o r mj , b o t h mira a n dm jm a r e p r i m e i m p l i c a n t s o f f , t h e n fi, fj m u s t o c c u r a ss u m m a n d s i n t h e s a m e p l u s e x p r e s s i o n o f f .Proof : M o s t e a s i ly v e r i f i e d b y r e p r e s e n t i n g e x p r e s s i o nf a s a d i r ec ted g raph (e .g . , [8 ] ) wi th edges l abe led by theB o o l e a n v a r i a b l e s . T h e s e t s o f l a b e l s a l o n g d i r e c t e dp a t h s b e t w e e n a d i s t i n g u i s h e d s o u r c e n o d e a n d a d i s -t i n g u i s h e d s i n k n o d e c o r r e s p o n d e x a c t l y t o p r i m e i m -p l i c a n t s i n t h e c a s e o f # e x p r e s s i o n s w h e r e a l l e d g e sw i l l h a v e d i f f e r e n t l a b e l s .

N o w t o v e r i f y t h a t S i s t r a n s i t i v e , s u p p o s e t h a t g~S&a n d gjSgk w h e r e , b y C l a i m 1 , gi -< fi, & -< f# a n d gk ~ fkf o r a p p r o p r i a t e t i me s e x p r e s s i o n s f i , f j , fk. T h e n i t f o l -l o w s t h a t m in i , m j rh j , a n d mk rh j a r e a l l p r i m e i m p l i c a n t so f f w h e r e m i , m # m k h a v e n o v a r ia b l e i n c o m m o n w i t hrh j. I t fo l lows f rom Cla im 2 tha t f i , f j, and fk a re add end si n t h e s a m e s u m e x p r e s si o n . H e n c e giSgk fo l lows .C l a i m 3 : S u p p o s e t h a t a ft e r s o m e p h a s e o f t h e a lg o -r i t h m C l a i m 1 h o l d s a n d t i me s e x p r e s s i o n s g~, g j h a v eb e e n f o r m e d w h e r e gi -< fi, & -< ~, a n d f i , f j a r e t i me se x p r e s s i o n s . T h e n gi a n d & w i ll b e p l a c e d i n t h e s a m es u m e x p r e s s i o n a t t h e n e x t p l u s - p h a s e i f a n d o n l y i f f i =fi, fj = )~, and ffi, fj a r e a d d e n d s i n t h e s a m e s u m e x p r e s -s ion in f .Proof : ( ~ ) I f gi -< fi = fi , &


9/9


gi, g] are placed in the same times expression at anysubsequent phase, the n f i , ~ are in the same product in

f .P r o o f : (1) If the cond itio ns of (1) above hold, the n g i T &will be discovered at the next times-phase and theclaim follows. (2) I f f i , J ~ are not in the sa me product in f,then f contains some prime implicant containing vari-ables from one of gi , & and not from the other. Henceg i T g j will never hold.P r o o f o f C l a i m 1: By induction on the numb er ofphases on Claims 1, 3, and 4 simultan eously .

We define the d e p t h of a formula fi to be the maxi-mum numb er of alternations of sum an d product re-quired in its definition.C l a i m 5 : I f f i is an expression of depth k in f, the n afterk phases a g i identical to fi will have been constructedby the algorithm.P r o o f : By induction on Claims 3 and 4.

To conclude the proof of the theorem, it remains toanalyze the runtime. This is dominated by the cost ofcomputing m i and rhi, for whic h we have a lready ac-counted, plus the cost of computing the equivalencerelations S and T. We first note that fewer than 2texpression nam es g i are used in the course of the algo-rithm. When an expression is grafted into another at apoint deep in the latter, then the semantics of all thesubexpressions above will change. By Claims 3 and 4such grafting can occur when a g i is added to a sumexpression but not when added to a times expression.Hence the values m i and rhi do not nee d to be ch angedfor any ex pression wh en such grafting occurs. It followsthat for co mpu tin g S only 2t va lues of m~, rh~ nee d to beconside red. Henc e 0(t 2) calls of P I or 0(t a) calls of N-ORACLE suffice overall. For computing T grafting maycause a ripple effect. On each occasion the value of Vjmay have to be updated for up to t such sets, and hencet 2 calls of RA will suffice. Henc e 0(t a) calls of RA in theoverall algorithm will be enough. []8 . R E M A R K SIn this paper we h ave cons idered le arni ng as the proc-ess of deducing a program for performing a task, frominforma tion that does not provide an explicit descrip-tion of such a program. We have given precise meaningto this notion of learning and have shown that in somerestricted but nontrivial contexts it is computationallyfeasible.

Consider a world contai ning robots and elephants.Suppose that one of the robots has discovered a recog-nition algorithm for elephants that can be mea ningfullyexpressed in k-conjunctive normal form. O ur Th eoremA implies that this robot can communicate its algo-rithm to the rest of the robot population by simplyexclaiming "elephant" when ever one appears.

An imp ortan t aspect of our approach, if cast in itsgreatest generality, is that we require the recognitionalgorithms of the teacher and learner to agree on anoverwhelmin g fraction of only the natural inputs. Their

behavior on unnatu ral inputs is irrelevant, and hencedescriptions of all possible worlds are not necessary. Iffollowed to its conclusion, this idea has consider ablephilosophical implications: A learnable concept is noth-ing more than a short program that distinguishes somenatural inputs from some others. If such a concept ispassed on among a population in a distributed manner,substantial variations in meaning may arise. More im-portantly, what consensus there is will only bemeaningful for natural inputs. The behavior of an indi-vidual's program for unnatur al inputs has no relevance.Hence thought experiments a nd logical arguments in-volving unna tura l hypothetical situations may bemeaning less activities.

The second impor tant aspect of the for mulat ion isthat the notio n of oracles makes it possible to discuss awhole range of teacher-le arner intera ctions beyond themere i dentific ation of examples. This is significant inthe context of artificial intelligence where huma ns maybe willing to go to great lengths to conve y their skills tomachines but are frustrated by their ina bility to articu-late the algorithms they themselves use in the practiceof the skills. We expect that some explicit programmingdoes become essentia l for trans mitti ng skills that arebeyon d certai n limits of difficulty. The identif ication ofthese limits is a major goal of the l ine of work proposedin this paper.REFERENCES1. Angluin,D., Smith, C.H. Inductive nference:Theory and methods.Comput. Surv. 15, 3 (Sept. 1983), 237-269.2. Barr, A., and Feigenbaum,E.A.The H andbook of Artificial Intelligence.Vol. 2. WilliamKaufmann,Los Altos, Calif., 1982.3. Cook, S.A. The complexity of theorem provingprocedures. In Pro-ceedings of 3rd Annual A CM Sym posium on Theory of Computing(Shaker Heights, Ohio, May 3-5). ACM, New York, 1971, 151-158.4. Duda, R.O., and Hart, P.F.Pattern C lassification and Scene Analysis.Wiley, New York. 1973.5. ErdSs, P., and Spencer, J.Probabilistic Methods in C ombinatorics. Aca-

demic Press, New York, 1974.6. Goldreich, O., Goldwasser, S., and Micali,S. How to construct ran-dom functions. n Proceedings of 25th IEEE Symposium on Foundationsof Computer Science (Singer Island, Fla., Oct. 24-26). IEEE, New York,1984.7. Michalski,R.S.. Carbonell,J.G., and Mitchell, T.M.Machine Learning:An Artificial Intelligence Approach. Tioga PublishingCo., Palo Alto,Calif., 1983.8. Skyum,S., and Valiant, L.G. A complexity heory based on Booleanalgebra. In Proceedings of 22rid IEEE Symposium on Foundations ofComputer Science (Nashville,Tenn.,Oct. 28-30}. IEEE, New York,1981, 244-253.9. Valiant,L.G. Deductive earning.Philosophical Transactions of theRoyal Society of London (1984). To be published.CR Categories and Subject Descriptors: F.1.1 [Computation by Ab-stract Devices[: Models of Computation;F.2.2 Analys i s o f Algori thmsand Problem Complexity[: NonnumericalAlgorithmsand Problems;L2.6[Artif icial Intelligence]: Learning--concept learningGeneral Terms: AlgorithmsAdditional Key Words and Phrases: inductive nference,probabilis-tic models of learning,propositionalexpressions

Received 1/84; revised 6/84; accepted 7/84Author's Present Address: L.G. Valiant, Center for Research in Comput-ing Technology.AikenComputationLaboratory, Harvard University,Cambridge, MA 02138.Permission o copy without fee all or part of this material s grantedprovided that the copies are not made or distributed for direct commer-cial advantage, the ACM copyright notice and the title of the publicationand its date appear, and notice s given hat copying s by permissionofthe Association or ComputingMachinery. To copy otherwise, or torepublish, requires a fee and/or specific permission.

1142 C o m m u n i c a t i o n s o f t h e A C M N o v e m b e r 1 98 4 V o l u m e 2 7 N u m b e r 1 1

Documents

Theory of Learnable