View
218
Download
1
Embed Size (px)
Citation preview
Exact Learning of Boolean Functions with Queries
Lisa Hellerstein
Polytechnic University
Brooklyn, NY
AMS Short Course on Statistical Learning Theory, 2007
I. Introduction
Learning Boolean formula f
Problem: Boolean formula f hidden in a black box. Told that f is from class C of formulas.
Task: “Learn” f
f(x1, x2, x3) = x1 Λ x3
Learning representation f of Boolean function
Problem: f hidden in a black box. Told that f is from class C of representations of Boolean functions.
Task: “Learn” f
f = x1 + x2 (mod 2)
Boolean functions can represent
• Whether a person is good or bad
• Whether an email message is spam
• Whether tumor is malignant
• Whether a book is a romance novel
• etc.
How hard is it to learn target f?
Need to specify:Type of information available
What’s meant by “learning”
Learning Models
II. Learning Models
Valiant’s PAC Model (1984)
• PAC = Probably Approximately Correct• Type of info available:
– Random examples:Value of f on “random” points in its domain
• Success Criterion:– Approximate learningOutput h that is approximately functionally
equivalent to f
Query Models (this talk)
• Type of info available:– Oracles that answer questions about f
• Success Criterion:– Exact learning
Output h where h ≡ f
• Want to learn f within “polynomial” number of queries, in “polynomial” time– polynomial in n and size of f
Types of queries
Membership queries (point evaluation)Question: What is f(x)?
Answer: f(x)
Equivalence queriesQuestion: Is h ≡ f?
h is hypothesisAnswer: Yes if so, else x such that f(x) ≠ h(x) x is counterexample
Definition: A membership and equivalence query algorithm learns a class C of representations if given 1. Oracles to answer membership and equivalence queries for some f in C
2. The number n of variables of f the algorithm outputs a representation h s.t. f ≡ h
Say algorithm runs in polynomial time if running time is poly(n, size of f)
About membership and equivalence queries
• Assume queries answered perfectly• Membership queries
– Black-box interpolation– Perfect answers often not available in practice
• Equivalence queries– Can be simulated in PAC model
• Test whether f(x) = h(x) on random examples x
– Relation to mistake-bound learning in on-line model
III. Example Query Algorithm
E.g. C = Boolean monomials
Boolean monomial = conjunction of literals
f(x1, x2, x3) =
¬x1 Λ x3 = ¬x1x3
x1 ¬x2 x3
¬x2
Learning monomial f(x1, x2, x3)
1. Ask equiv. query: Is f ≡ 0? Suppose get counterexample f(1,0,1)=1
2. For each xi, determine if it appears in monomial f with negation, without negation, or not at all
Since x1=1 in counterexample, x1 appears in f without negation, or not at all. Ask membership query: What is f(0, 0, 1)?
If answer is 0, x1 without negation is in monomial f
If answer is 1, x1 does not appear in f at all
Do similarly for x2 and x3
Learning Boolean monomials
• Previous approach learns Boolean monomials in n+1 equivalence and membership queries, polynomial time
• Can also learn Boolean monomials with equivalence queries alone
• Need exponential queries (worst-case) with membership queries alone.
If monomial includes all n variables, f=1 for only one of the 2n points in its domain
IV. Four interesting representation classes
1. DNF Formulas
OR of ANDs
f = ¬x1 x2 x3 V ¬x1x4V x1x2x3
• Natural way of describing classification rule• Not known whether DNF learnable in polynomial
time with membership and equivalence queries (or in PAC model)
• Best known algorithm runs in time )loglog(O 3
2 snn
2. Boolean linear threshold functions f = 1 if x1 + x2 + x3 > 2
= 0 otherwise
Learnable in polynomial time, equivalence q’s
3. Polynomials over GF[2] (integers mod 2) f = x2 x3 + x1 x3 + x1 x2 x3
Learnable in polynomial time, memb+equiv q’s
4. Boolean decision trees
Learnable in polynomial time, memb+equiv q’s
x1
x3
01
x2
0 1
=1=0
=0
=1=0
=1
Representation and size
• Can represent every Boolean function as DNF formula, GF[2] polynomial, or decision tree– But sizes of representations can be very different– e.g. Parity function
Representation as GF[2] polynomial is small
f(x1,…,xn) = x1 + x2 + … + xn (mod 2)
Requires DNF formula of size exponential in n
Requires decision tree of size exponential in n
V. Learning with Polynomial Number of Queries
Halving AlgorithmGeneric algorithm for learning with poly number of queries
• Assume (for simplicity) know size s of target f• Keep set V of all possible f
Initially, V contains all representations in C (on n variables) of size s
• Repeat until success:– Use V to construct Majority Hypothesis h– Ask equivalence query with h– Either “yes” (success), or receive counterexample. – If the latter, update V
f6
f1
f8 f3
f5
V
Majority Hypothesis hFor each x in domain of f h (x) = 1 if majority of fi’s in V have fi(x) = 1 = 0 if majority of fi’s in V have fi(x) = 0
Counterexample to majority hypothesis eliminates at least half of fi’s in V Number of equivalence queries of Halving Algorithm is log2(Original size of V)
VI. Challenge: Learn in polynomial time
If restrict hypotheses to be in C
• May be NP-hard to learnComputational hardness of learning– Tools to prove: Complexity theory, NP-completeness
reductions, non-approximability• May require exponential number of queries to
learn Informational hardness of learning
– Tools to prove: Structural properties of C, combinatorial arguments
Example
• Suppose C is class of 2-term DNF formulas
and want to learn C with equivalence queries alone
• NP-hard to learn 2-term DNF formulas with equivalence queries alone if hypotheses must be 2-term DNF formulas
f = ¬x1 x3 V x1x2
• 2-term DNF formulas can be factored
f = ¬x1 x3 V x1x2
= (¬x1 V x2)(x3 V x1) (x3 V x2)
• Result is 2-CNF formula – AND of ORs in which each OR has at most 2
literals– Size of 2-CNF formula O(n2)– 2-CNF formulas can be learned in poly-time
with equivalence queries alone (how?)
• Learn 2-term DNF formula using algorithm for learning 2-CNF formulas.
Learning 2-CNF
2-CNF formula
f = (¬x1 V x2)(x3 V x1) (x3 V x2)Can be viewed as monomial over new variable set
{y1, y2, …,}
y1 = (¬x1 V x2)
y2 = (x1 V x2)
y3 = (x2 V ¬x3) etc. Learn 2-CNF formulas using algorithm for learning
monomials by translating between original vars and new vars
Two Useful Techniques
1.To show C learnable, find C' s.t. – C' poly-time learnable– each f in C has equivalent f' in C' of size at
most polynomially larger.– Learn C using algorithm for C’
2. Use existing algorithm with new variable set
BUT…
Even if allow hypotheses not from C, can still be hard to learn C in polynomial time If C sufficiently rich class of Boolean
circuits/formulas– Can show that C can represent cryptographic
primitives– Learning C as hard as breaking cryptographic
primitives
VII. Learning GF[2] Polynomials and Decision Trees
GF[2] Polynomials and Decision Trees
• Poly-time learnable with membership and equivalence queries using algorithm for learning Hankel matrix representations (multiplicity automata)– Useful Technique 1
• Hankel matrix representations learnable using variant of Deterministic Finite Automaton learning algorithm
Hankel Matrix H of f(x1,…,xn)
View f as function on binary strings
Rows/columns of H indexed by all binary strings.
H[x,y] = f(x◦y) if |x|+|y|=n = 0 otherwise
Hankel matrix of f(x1,x2) = x1 V x2
ε 0 1 00 01 10 11 111 ... ε 0 0 0 0 1 1 1 0 0 0 0 1 1 0 1 1 00 0 01 1 10 1 11 1 …
Learning Hankel matrices of Boolean functions
• Can represent Hankel matrix compactly– Suffices to specify particular O(r2) entries, where r is rank of
matrix– Running time of Hankel matrix algorithm polynomial in r, n
• Lemma: If f(x1,…,xn) is a GF[2] polynomial with s terms, then rank of its Hankel matrix is poly(n,s)
• Lemma: If f(x1,…,xn) is a decision tree with s nodes, then rank of its Hankel matrix is poly(n,s)
• Use Hankel matrix algorithm to learn GF[2] polynomials and decision trees
VIII. Summary
• Definition of Query Learning Models
• Halving algorithm for learning with polynomial number of equivalence queries
• Techniques for polynomial-time learning
• Examples of classes learnable in polynomial-time
• Barriers to polynomial-time learning
Selected References
• Learning Models– Valiant, L. G., A Theory of the Learnable. Communications of the
ACM, 1984– Angluin, D. Queries and concept learning. Machine Learning
2(4), 1988
• Learning Algorithms– Beimel, A., Bergadano, F., Bshouty, N. H., Kushilevitz, E., and
Varricchio, S. Learning functions represented as multiplicity automata. Journal of the ACM (3), 2000
– Maass, W. and Turan, G. On the complexity of learning with counterexamples. Proc. of the 30th IEEE Symposium on Foundations of Computer Science (FOCS), 1989
– Klivans, A. and Servedio, R. Learning DNF in Time 2^{O(n^{1/3})}. Journal of Computer and System Sciences 68(2), 2004
• Hardness of Learning– Kearns, M. J. and Valiant, L. G. Cryptographic limitations on
learning Boolean formulae and finite automata. J. ACM (1), 1994
– Angluin, D. Negative results for equivalence queries.
Machine Learning (5), 1990– Hellerstein, L., Pillaipakkamnatt, K., Raghavan, V., and Wilkins,
D. How many queries are needed to learn? J. ACM (43), 1996