1. Introduction Consistency of learning processes To explain when a learning machine that minimizes...
Preview:
Citation preview
- Slide 1
- 1
- Slide 2
- Introduction Consistency of learning processes To explain when
a learning machine that minimizes empirical risk can achieve a
small value of actual risk (to generalize) and when it can not.
Equivalently, to describe necessary and sufficient conditions for
the consistency of learning processes that minimize the empirical
risk. It is extremely important to use concepts that describe
necessary and sufficient conditions for consistency. This
guarantees that the constructed theory is general and cannot be
improved from the conceptual point of view. 2
- Slide 3
- 3
- Slide 4
- The classical definition of consistency Let Q(z, l ) be a
function that minimizes the empirical risk functional for a given
set of i.i.d. observations z1,, z l. 4
- Slide 5
- The classical definition of consistency The ERM principle is
consistent for the set of functions Q(z, ),, and for the p.d.f. F
(z) if the following two sequences converge in probability to the
same limit Equation (1) asserts that the values of achieved risks
converge to the best possible. Equation (2) asserts that one can
estimate on the basis of the values of empirical risk the minimal
possible value of the risk. 5
- Slide 6
- The classical definition of consistency 6
- Slide 7
- Goal To obtain conditions of consistency for the ERM method in
terms of general characteristics of the set of functions and the
probability measure. This is an impossible task because the
classical definition of consistency includes cases of trivial
consistency. 7
- Slide 8
- Trivial consistency 8
- Slide 9
- 9
- Slide 10
- ERM consistency In order to create a theory of consistency of
the ERM method depending only on the general properties (capacity)
of the set of functions, a consistency definition excluding trivial
consistency cases is needed. This is done by non-trivial (strict)
consistency definition. 10
- Slide 11
- Non-trivial consistency 11
- Slide 12
- 12
- Slide 13
- Key theorem of learning theory 13
- Slide 14
- Consistency of the ERM principle 14
- Slide 15
- Consistency of the ERM principle In contrast to the so-called
uniform two-sided convergence defined by the equation 15
- Slide 16
- Consistency of the ML Method 16
- Slide 17
- Consistency of the ML Method 17
- Slide 18
- 18
- Slide 19
- Empirical process 19
- Slide 20
- Consistency of an empirical process The necessary and
sufficient conditions for an empirical process to converge in
probability to zero imply that the equality holds true. 20
- Slide 21
- Law of large numbers and its generalization 21
- Slide 22
- Law of large numbers and its generalization 22
- Slide 23
- Entropy 23
- Slide 24
- Entropy of the set of indicator functions (Diversity) 24
- Slide 25
- Entropy of the set of indicator functions (Diversity) 25
- Slide 26
- Entropy of the set of indicator functions (Random entropy)
26
- Slide 27
- Entropy of the set of real functions (Diversity) 27
- Slide 28
- 28
- Slide 29
- VC Entropy of the set of real functions 29
- Slide 30
- VC Entropy of the set of real functions (Random entropy and VC
entropy) 30
- Slide 31
- Conditions for uniform two-sided convergence 31
- Slide 32
- Conditions for uniform two-sided convergence (Corollary)
32
- Slide 33
- 33
- Slide 34
- Uniform one-sided convergence Uniform two-sided convergence can
be described as which includes uniform one-sided convergence, and
it's sufficient condition for ERM consistency. But for consistency
of ERM method, the second condition on the left-hand side of (9)
can be violated. 34
- Slide 35
- Conditions for uniform one-sided convergence 35
- Slide 36
- Conditions for uniform one-sided convergence 36
- Slide 37
- 37
- Slide 38
- Models of reasoning Deductive Moving from general to
particular. The ideal approach is to obtain corollaries
(consequences) using a system of axioms and inference rules.
Guarantees that true consequences are obtained from true premises.
Inductive Moving from particular to general. Formation of general
judgments from particular assertions. Judgments obtained from
particular assertions are not always true. 38
- Slide 39
- Demarcation problem Proposed by Kant, it is a central question
of inductive theory. Demarcation problem What is the difference
between the cases with a justified inductive step and those for
which the inductive step is not justified? Is there a formal way to
distinguish between true theories and false theories? 39
- Slide 40
- Example Assume that meteorology is a true theory and astrology
is a false one. What is the formal difference between them? The
complexity of the models? The predictive ability of their models?
Their use of mathematics? The level of formality of inference? None
of the above gives a clear advantage to either of these theories.
40
- Slide 41
- Criterion for demarcation Suggested by Popper (1930), a
necessary condition for justifiability of a theory is the
feasibility of its falsification. By falsification, Popper means
the existence of a collection of particular assertions that cannot
be explained by the given theory although they fall into its
domain. If the given theory can be falsified it satisfies the
necessary conditions of a scientific theory. 41
- Slide 42
- 42
- Slide 43
- Nature of the ERM principle What happens if the condition of
one-side convergence (theorem 11) is not valid? Why is the ERM
method not consistent is this case? Answer: if uniform two-sided
convergence does not take place, then the method of minimizing the
empirical risk is non-falsifiable. 43
- Slide 44
- Complete (Popper's) non- falsifiability 44
- Slide 45
- Complete (Popper's) non- falsifiability 45
- Slide 46
- Complete (Popper's) non- falsifiability 46
- Slide 47
- Partial non-falsifiability 47
- Slide 48
- Partial non-falsifiability 48
- Slide 49
- Potential non-falsifiability (Definition) 49
- Slide 50
- Potential non-falsifiability (Graphical representation) A
potentially non-falsifiable learning machine 50
- Slide 51
- Potential non-falsifiability (Generalization) 51
- Slide 52
- Potential non-falsifiability 52
- Slide 53
- Refrences Vapnik, Vladimir,The Nature of Statistical Learning
Theory, 2000 Miguel A. Veganzones, Consistency and bounds on the
rate of convergence for ERM methods 53
- Slide 54
- 54