Block Emulation and Computation in One-dimensional Cellular Automata: Breaking Behavioural Class Boundaries

Block Emulation and Computation in One-dimensionalCellular Automata: Breaking Behavioural Class

Boundaries

Jurgen Riedel1

Carl von Ossietzky University, OldenburgHector Zenil2

Karolinska Institute, Sweden

(Special thanks to Stephen Wolfram and Todd Rowland3)

Bremen, October 7, 2013

[email protected]@ki.se3Wolfram Research

Riedel & Zenil (University of Oldenburg) Emulation and Universality Bremen, October 7, 2013 1 / 33

Overview

1 Computation Universality

2 Cellular Automata

3 Rule Emulation

4 Complexity Measures

5 First Results

6 Outlook


Universality Computer

Simple Programs/ Computer

Symbolic computing systems are dynamical systems, i.e., a set ofconfigurations together with a transformation of this set. Such simplecomputers or programs compute or transform an input into somethingelse usually in the same language.The most common examples of suchsystems are:

1 Cellular automata (e.g. Conway’s Game of Life)

2 Mobile automata

3 Turing machines

4 Substitution systems

5 Post Tag systems

6 Register machines


Universality Universality

Computational Universality

Behind all general-purpose computers there is the concept ofcomputation universality. That is, one can simulate any computation inany of the other systems (formalisms). All these models arecomputationally equivalent.

Theorem (Universal Turing Machine (Turing 1937, 1938))

Any arbitrary Turing machine which, by appropriate programmingusing a finite length of input tape, can act as any other arbitraryTuring machine.


Universality Universality

Computational Universality

Theorem (Church-Turing thesis (1936))

Any real-world computation can be translated into an equivalentcomputation involving a Turing machine (general recursive functions).

Conjecture (Principle of Computational Equivalence)a Almost all processes that are not obviously simple can be viewed ascomputations of equivalent sophistication, i.e. can performcomputations up to a maximal (”universal”) level of computational power.

aWolfram, S., A New Kind of Science,pp. 5-6 and 715-846, 2002.


Universality Undecidability

Undecidability

Problem (The Halting Problem (Alonzo Church/ Alan Turing 1936))

Given the textual representation of a Turing Machine, i.e., a program, isthere a (universal) Turing Machine that can determine whether theprogram (Turing Machine) will halt on all inputs?

Turing proof (1936): A general algorithm running on a Turing machinethat solves the halting problem for all possible program-input pairsnecessarily cannot exist.In other words: The Halting Problem is undecidable and means that itis impossible, even given unlimited resources and an infinite amount oftime, that the problem can be decided by algorithmic means.Consequence for universality: If a dynamical system is computationallyuniversal then it is impossible to algorithmically compute most of itsproperties.


Cellular Automata Cellular Automata

Cellular Automata History

1 First introduced by John von Neumann (1903-1957) and StanislawUlam (1909-1984) during their common work at Los Alamos Nationallab.

2 In 1970 American mathematician John Conway developed the Gameof Life which popularized CA among the scientists. Life is a cellularautomaton operating on a two-dimensional grid of cells (sites).

3 In the 80’s Stephen Wolfram exhaustive exploration ofone-dimensional cellular automata and introduction of WolframClasses (Wolfram, 1984).

4 Main overall discovery: Extremely simple rules (with extremelysimple initial conditions) can lead to extremely complex behaviour.



Cellular Automata - Definition

1 A CA is a spatial lattice of N cells (sites), each of which is one of kstates which take values from 0..(k − 1) at time t.

2 The cell’s state s at time t+1 depends on its own state and thestates of some number of neighbouring cells at time t.

3 Each cell follows the same simple transition rule for updating itsstate.

4 The set of all possible rules is called the rule space of a CA and hasan order of kk

(2r+1)




1 CAs with k = 2 and r = 1 are called Elementary CellularAutomata (ECA). The ECA rule space has order 256.

2 Another rule space we investigate in this talk has k = 2 and r = 3/2and we call it GCA. The GCA rule space has order 65536.

3 Rule spaces are enumerated. For example the ECA rule 90 isenumerated (encoded) like:




1 For n-dimensional CAs the neighbourhood of a cell consists of thecell itself and r neighbours and time evolution:1

1Bio-Inspired Artificial Intelligence: Theories, Methods, and Technologies by DarioFloreano and Claudio Mattiussi, MIT Press


Cellular Automata Wolfram Classes

Wolfram’s Behavioural Classes (of Cellular Automata)

1 Class I: CA evolving to a homogeneous state, i.e. dominated by aunique state of its alphabet for any random initial condition.

2 Class II: CA evolving periodically, i.e. dominated by blocks of cellswhich are periodically repeated for any random initial condition.

3 Class III: CA evolving chaotically, i.e. for a long time and for anyrandom initial condition, the evolution is dominated by sets of cellswithout any defined pattern.

4 Class IV: Includes all previous cases, known as a class of complexrules, i.e. the evolution is dominated by non-trivial structuresemerging and travelling along the evolution space where uniform,periodic, or chaotic regions can coexist with these structures.



Example Wolfram class I

Rule 136 Rule 136

Figure: Periodic initial condition (left), Random initial condition (right)



Example Wolfram class II

Rule 28 Rule 28

Figure: Periodic initial condition (left), Random initial condition (right)



Example Wolfram class III

Rule 90 Rule 90

Figure: Simple initial condition (left), Random initial condition (right)



Example Wolfram class IV

Rule 110 Rule 110

Figure: Simple initial condition (left), Random initial condition (right)


Cellular Automata Rule Emulation

Universality in CAs

1 Due to the Halting Problem and the associated undecidabilitythere exists no effective algorithm that is capable of decidingwhether a CA is complex or universal.

2 Only a few discovered cellular automata have been proven to becapable of universal computation notably Wolfram’s Rule 110 andConway’s Game of Life. 1

3 ”The fourth class is probably capable of universal computation,so that properties of its infinite time behaviour are undecidable.” 2

1Cook, M. ”Universality in Elementary Cellular Automata.” Complex Systems 15,1-40, 2004; Sapin, Emmanuel and Bailleux, Olivier and Chabrier, Jean-Jacques andCollet, Pierre, ”A New Universal Cellular Automaton Discovered by EvolutionaryAlgorithms”, Springer, Berlin Heidelberg 2004

2Wolfram, S. (1984) Universality and complexity in cellular automata, Physica D 101-35



Universality in CAs

1 ”In general what one needs to do in order to prove universality isto find a procedure for setting up initial conditions in one system soas to make it emulate some general class of other systems.”1

2 If a CA is universal, then this means that with a suitable encoding ofinitial conditions its evolution must emulate the evolution of anyother CA.

3 One might be able to get evidence about universality by tryingdifferent possible encodings, and then seeing what range of otherCAs they can emulate.

4 Given for example a particular ECA one can test what other ECA itcan emulate using blocks up to a certain length.

1Wolfram, S. (1984) Universality and complexity in cellular automata, Physica D 101-35



Block emulation

Rule 22Rule 22 simulating 90

Æ Æ

Rule 90

Figure: ECA rule 20 emulates ECA rule 90 with block size 4



Block emulation

Figure: Emulation Diagram for Rule 22 emulates Rule 90


Complexity Measures Shannon information

Shannon of Information

Given are two events A and B of probability space (Ω, F, P) on thedomain Ω, where (Ω,S) is a measurable space, F, are the measurablesubsets of Ω, and P is a measure on Ω with P(Ω)=1.

The information I (A | B) which is contained in the event (message)A over B is given by:

I (A | B) = log2P(A | B)

P(A)(1)

If A = B one finds:

I (A | A) = I (A) = log2P(A | A)

P(A)= log2

1

P(A)= −log2P(A) (2)


Complexity Measures Shannon information

Properties of Information

Property 1

P(A | A) is the information which is contained in the event A.

Property 2

Is follows that the amount of information I (A) is smaller the larger theprobability of event A is.

Property 3

If the two events A and B are independent of each other we findI (A | B) = 0 and that the event B over A contains zero information andvice versa.

Property 4

It is also true that if events A and B are independent thatI (AB) = I (A) + I (B)


Complexity Measures Shannon entropy

Shannon Entropy

Define of a measure of information for as set of events (messages) Ω asa reale-valued function H(Ω) that should satisfy:

1 Continuity. H(p1...pN) is continuous.

2 Additivity. H(p1q1...pNqN) = H(P) + H(Q), for probabilitydistributione P,Q.

3 Monoticity. Information increases with N for uniform distributions:If M > N then H(Q) > H(P) for any P = ( 1

N , ...,1N ) and

Q = ( 1M , ..., 1

M ).

4 Branching. H(p1...pN) is independent of how the process is dividedinto parts.

5 Bit normalisation. The average information gained for equally likelymessages is one bit, i.e. H(12 ,

12) = 1.



Shannon entropy

Theorem (Uniqueness of Measure )

There is exactly one function that satisfies the above criteria, i.e. theShannon Entropy or Shannon Information:

H(Ω) = −N∑i=1

pi log2 pi (3)

Main properties of H(Ω):

1 H(Ω) is maximal for pi = 1N

2 H(Ω) = 0 if there exist one pi = 1 and the rest is 0.



Shannon Entropy & Boltzmann Entropy

The entropy is the average information of the source, that is, it is theweighted average of the information of each of the message (event)according to their respective probabilities:

Hp = H(Ω) =N∑i=1

pi I (pi ) = −N∑i=1

pi log pi (4)

If the events are equally probable (microcanonical ensemble) then theyeach have probability pi = 1

N and we find by multiplying with kB andchange of basis:

kBHp = kBH(Ω) = −kBN∑i=1

1

Nlog

1

N= kB logN = SB (5)

Boltzmann Entropy


Complexity Measures Entropy Rate

Entropy Rate

1 Take a bit sequence, e.g. evolution step of CA and partition intoblocks of length L.

2 Find probability distribution of block of length L.3 Calculate block entropies (Shannon entropy):

H(L) = −∑

b∈blocksPL(b) logPL(b) (6)

4 Calculate entropy rate (metric entropy):

hµ ≡ limL→∞

H(L)

L; hµ(L) = H(L)− H(L− 1) (7)

hµ(L) is the incremental increase in uncertainty in going from(L− 1)-blocks to L blocksRiedel & Zenil (University of Oldenburg) Emulation and Universality Bremen, October 7, 2013 25 / 33

Complexity Measures Excess Entropy

Excess Entropy

1 The entropy rate leads to the definition of excess entropy 1:

E ≡∞∑L=1

[hµ(L) − hµ] (8)

2 The length-L approximation to the entropy density2 overestimatesthe entropy density by an amount hµ(L)− hµ

1Grassberger, Crutchfield 19862D. P. Feldman, A brief introduction to: Information Theory, California, 1997



Example

1 Example 1 : Given a bit sequence (bit train)”...1110111011101110111011101110...”.

2 4 6 8 10 12 14

L1.0

1.5

2.0

2.5

3.0

3.5

4.0

HHLL

2 4 6 8 10 12 14L

0.2

0.4

0.6

0.8

h_Μ HLL

2 hµ = 0; the sequence is perfectly predictable.3 E = log216 = 4: four bits of information4 For any period-p sequence, hµ = 0 and E = log2p.1Crutchfield and Feldman, Chaos, 15: 25-54, 2003



Properties of Excess Entropy

1 E is the amount of apparent randomness that is eventuallyexplained by longer block sizes.

2 Large E indicates a large amount of structure.

3 Large E accounts for the apparent randomness observed whendistributions over small block sizes are considered.

4 Is a structural property of the system measures a featurecomplementary to entropy.

5 Measures memory or spatial structure.


Complexity Measures Kolmogorov–Chaitin complexity

Kolmogorov–Chaitin complexity

Given three 24-bit binary strings1

1: 0101010101010101010101012: 1001110111010111001001103: 110100110010110100101100

1 The first string can be fully described by stating that it is a 24-bitstring with a 1 in position n iff n is odd

2 The third has the property that position i is a 1 iff there are anodd number of 1’s in the binary expansion of position i

3 The second string appears not to have a similarly simpledescription

Conclusion: Find a way to recite its contents verbatim.

1Example by L. Fortnow, 2001Riedel & Zenil (University of Oldenburg) Emulation and Universality Bremen, October 7, 2013 29 / 33


Kolmogorov–Chaitin complexity

Theorem

Kolmogorov–Chaitin complexity (algorithmic complexity, algorithmicentropy)

KU(s) = min|p|;U(p) = s (9)

1 U is a universal Turing machine and p the program that, running onU, produces s.

2 To approach the algorithmic complexity of a string is throughlossless compression algorithms.

3 The result of a lossless compression algorithm applied to s is anupper bound of the Kolmogorov complexity of s.1

4 Is related to the Entropy Rate hµ

1Hector Zenil and Elena Villarreal–Zapata, Asymptotic Behaviour and Ratios ofComplexity in Cellular Automata Rule Spaces, 2013, arXiv:1304.2816v2



First Results...


Outlook What next

Things to investigate

1 ECA for larger block size.1 If we increase the block size do we emulate more rules? Find analytical

solution2 Focus on a few candidate rules for very large block sizes.

2 Relations to group theory (irreducible groups)

3 Connections to dynamic systems (attractors, escapees, cycles)

4 Social network statistics of rule networks (close friends, friends offriends, clustering coefficient)

5 Topology of emulating networks

6 Systematically identify jumpers


Outlook What next

Thank You!
