Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
Physical Fuctuomatics (Tohoku
University) 1
Physical Fluctuomatics
1st Review of probabilistic information processing
Kazuyuki Tanaka
Graduate School of Information Sciences
http://www.smapip.is.tohoku.ac.jp/~kazu/
Webpage:
http://www.smapip.is.tohoku.ac.jp/~kazu/PhysicalFluctuomatics/2013/
Physical Fuctuomatics (Tohoku
University) 2
Textbooks
Kazuyuki Tanaka: Introduction of Image
Processing by Probabilistic Models, Morikita
Publishing Co., Ltd., 2006 (in Japanese) .
Kazuyuki Tanaka: Mathematics of Statistical
Inference by Bayesian Network, Corona
Publishing Co., Ltd., 2009 (in Japanese).
Physical Fuctuomatics (Tohoku
University) 3
References of the present lecture
K. Tanaka: Statistical-mechanical approach to image processing (Topical Review), Journal of Physics A: Mathematical and General, vol.35, no.37, pp.R81-R150, 2002. Y. Kabashima and D. Saad: Statistical mechanics of low-density parity-check codes (Topical Review), J. Phys. A, vol.37, no.6, pp.R1-R43, 2004.
H. Nishimori: Statistical Physics of Spin Glasses and Information Processing, ---An Introduction, Oxford University Press, 2001.
M. Opper and D. Saad D (eds): Advanced Mean Field Methods --- Theory and Practice, MIT Press, 2001.
C. M. Bishop: Pattern Recognition and Machine Learning, Springer, 2006.
M. J. Wainwright and M. I. Jordan: Graphical Models, Exponential Families, and Variational Inference, now Publishing Inc, 2008.
M. Mezard, A. Montanari: Information, Physics, and Computation, Oxford University Press, 2009.
Physical Fuctuomatics (Tohoku
University) 4
Benefit of Information &
Communications Technology
Ubiquitous Computing
Ubiquitous Internet
Benefit of Information & Communications Technology
Demand for Intelligence
It cannot be satisfied only with it being only cheap
and being quick.
Physical Fuctuomatics (Tohoku
University) 5
Field of Information Processing
Information processing according to theories
Inference from propositions
Realization by progress of computational processing capacity
Information processing in real world
Diversity of reason in phenomenon
Compete data is not necessarily obtained.
It is difficult to extract and select some important
information from a lot of data.
Uncertainty caused by the gap of knowing simply and
understanding actually.
We hope to deal successfully with such uncertainty.
Information processing for numerical calculations
Definite Procedure has been given for each calculation.
Physical Fuctuomatics (Tohoku
University) 6
Computer for next generations
Required Capacity
Capability to sympathize with a user (Knowledge)
Capability to put failure and experience to account in the next chance (Learning)
How should we deal successfully with the uncertainty caused
by the gap of knowing simply and understanding actually?
Formulation of knowledge and uncertainty
Realization of information processing data with
uncertainty
Physical Fuctuomatics (Tohoku
University) 7
Computational model for information
processing in data with uncertainty
Probabilistic Inference Probabilistic model
with graphical structure
(Bayesian network) Medical diagnosis
Failure diagnosis
Risk Management
Probabilistic information processing can give us unexpected
capacity in a system constructed from many cooperating
elements with randomness.
Inference system for data with uncertainty
modeling
Node is random variable.
Arrow is conditional probability.
Mathematical expression of uncertainty
=>Probability and Statistics
Graph with cycles
Important aspect
Physical Fuctuomatics (Tohoku
University) 8
Computational Model
for Probabilistic Information Processing
Probabilistic
Information Processing Probabilistic Model
Bayes Formula
Algorithm
Monte Carlo Method
Markov Chain Monte Carlo Method
Randomized Algorithm
Genetic Algorithm
Approximate Method
Belief Propagation
Mean Field Method
Randomness and
Approximation
Physical Fuctuomatics (Tohoku
University) 9
Probabilistic Image Processing
Noise Reduction by
Probabilistic Image
Processing
K. Tanaka: J. Phys. A, vol.35, 2002.
A. S. Willsky: Proceedings of IEEE, vol.90, 2002.
173
110218100
120219202
190202192
Average
192 202 190
202 219 120
100 218 110
192 202 190
202 173 120
100 218 110
Modeling of Probabilistic Image Processing based on Conventional Filters
Markov Random Field Model Probabilistic Image
Processing
The elements of such a
digital array are called pixels.
At each point, the intensity
of light is represented as an
integer number or a real
number in the digital image
data.
Algorithm
Conventional Filter
Physical Fuctuomatics (Tohoku
University) 10
Probabilistic Image Processing
Degraded Image (Gaussian Noise) Probabilistic Image Processing
Lowpass Filter Wiener Filter Median Filter
MSE:520 MSE: 2137
MSE:860 MSE:767 MSE:1040
K. Tanaka: J. Phys. A, vol.35, 2002.
A. S. Willsky: Proceedings of IEEE, vol.90, 2002.
Physical Fuctuomatics (Tohoku
University) 11
Error Correcting Code
Y. Kabashima and D. Saad: J. Phys. A, vol.37, 2004.
High Performance Decoding Algorithm
010 000001111100000 001001011100001
0 1 0
code
010
error
decode
Parity Check Code
Turbo Code, Low Density Parity Check (LDPC) Code
majority rule Error Correcting Codes
14 January, 2010
Hokkaido University GCOE Tutorial
(Sapporo) 12
Error Correcting Codes and
Belief Propagation
)2 (mod
)2 (mod
)2 (mod
5329
6438
4217
XXXX
XXXX
XXXX
1
1
0
0
1
1
6
5
4
3
2
1
x
x
x
x
x
x
0
1
0
1
1
0
0
1
1
9
8
7
6
5
4
3
2
1
x
x
x
x
x
x
x
x
x
0
1
1
1
1
0
1
1
1
9
8
7
6
5
4
3
2
1
y
y
y
y
y
y
y
y
y
0)2 (mod 101
1)2 (mod 100
0)2 (mod 011
9
8
7
X
X
X
1 1
0
p1
1
0
0
p1
p
p
Received Word
Code Word
Binary Symmetric
Channel
14 January, 2010
Hokkaido University GCOE Tutorial
(Sapporo) 13
Error Correcting Codes and
Belief Propagation
)2 (mod
)2 (mod
)2 (mod
5329
6438
4217
XXXX
XXXX
XXXX
Fundamental Concept for Turbo Codes and LDPC Codes
Physical Fuctuomatics (Tohoku
University) 14
CDMA Multiuser Detectors in Mobile Phone Communication
Relationship between mobile phone communication
and spin glass theory
T. Tanaka, IEEE Trans. on Information Theory, vol.48, 2002
Signals of
User A
Spreading Code
Sequence
Wireless
Communication
Received Data
Decode
Spreading Code
Sequence
Probabilistic model for
decoding can be
expressed in terms of a
physical model for spin
glass phenomena
Noise
Coded Signals
of Other Users
Coded
Signals of
User A
Physical Fuctuomatics (Tohoku
University) 15
Artificial Intelligence
Bayesian Network
CA
SA RA
WA
Probabilistic inference system
Practical algorithms
by means of belief
propagation
J. Pearl: Probabilistic Reasoning in Intelligent Systems: Networks of
Plausible Inference (Morgan Kaufmann, 1988).
Physical Fuctuomatics (Tohoku
University) 16
Main Interests
Information Processing:
Data
Physics:
Material,
Natural Phenomena
System of a lot of elements with mutual relation
Common Concept between Information Sciences and Physics
Material Molecule
Materials are constructed from a lot of molecules.
Molecules have interactions of each other.
0,1 101101
110001
01001110111010
10001111100001
10000101000000
11101010111010
1010 Bit
Data
Data is constructed from many bits
A sequence is formed by deciding the arrangement of bits.
A lot of elements have mutual relation of each other Some physical concepts
in Physical models are
useful for the design of
computational models in
probabilistic
information processing.
Physical Fuctuomatics (Tohoku
University) 17
Horizon of Computation
in Probabilistic Information Processing
Compensation of expressing uncertainty
using probability and statistics
It must be calculated by taking account of both events
with high probability and events with low probability.
Computational Complexity
It is expected to break throw the computational
complexity by introducing approximation algorithms.
Physical Fuctuomatics (Tohoku
University) 18
What is an important point in
computational complexity?
How should we treat the calculation of the summation over 2N configuration?
FT, FT, FT,
21
1 2
,,,x x x
N
N
xxxf
}
}
}
;,,,
F){or Tfor(
F){or Tfor(
F){or Tfor(
;0
21
2
1
L
N
xxxfaa
x
x
x
a
N fold loops
If it takes 1 second in the case of
N=10, it takes 17 minutes in
N=20, 12 days in N=30 and 34
years in N=40.
Physical Fuctuomatics (Tohoku
University) 19
Why is a physical viewpoint effective in probabilistic
information processing?
Matrials are constructed from a lot of molecules.
(1023
molecules exist in 1 mol.)
Molecules have intermolecular forces of each other
1 2
,,, 21
x x x
N
N
xxxf
Theoretical physicists always have to
treat such multiple summation.
Development of Approximate Methods
Probabilistic information processing is also usually reduced
to multiple summations or integrations.
Application of physical approximate methods
to probabilistic information processing
Physical Fuctuomatics (Tohoku
University) 20
Academic
Circulation
Academic
Circulation
Academic Circulation between
Physics and Information Sciences
Physics Information Sciences
Understanding and prediction
of properties of materials and
natural phenomena
Extraction and processing of
information in data
Common Concept Statistical
Mechanical
Informatics
Probabilistic
Information
Processing
Statistical
Sciences
Physical Fuctuomatics (Tohoku
University) 21
Summary of the present lecture
Probabilistic information processing
Examples of probabilistic information processing
Common concept in physics and information sciences
Application of physical modeling and approximations
Future Lectures
Fundamental theory of probability and statistics
Linear model
Graphical model
.
.
.