An Order-Theoretic Approach toward Entropy

Kevin H. KnuthDepartment of Physics

University at Albany

An Order-Theoretic Approach toward

Entropy

26 Oct 2007 Kevin H KnuthFacets of Entropy

In the Beginning…


Lifting Rocks

This caveman finds it easy to order these rocks in terms of how heavy they are to lift.


Diagram Representing the OrderingH

eavi

er

He uses a binary weight comparison to order his rocks

This configuration is called a chain


IsomorphismsIs

Les

s R

ipe

than

1

2

3

4

5

Is L

ess

than

or

Equ

al t

o

1

2

4

8

16

Div

ides


Antichains

These elements are incomparable under our binary ordering relation

They form a poset called an ANTICHAIN


Partitioning

A set of elements together with a binary ordering relation based onthe notion of containing


Two Posets with Integers

1

2

3

4

Is L

ess

than

or

Equ

al t

o

1

2 3

4D

ivid

es

5

6

7

8

9


The Powerset of {a, b, c}

}{b

}{a }{c

},{ ca},{ ba },{ cb

},,{ cba

,},,{},,{},,{},,{},{},{},{, cbacbcabacbaP

Is a subset of


Posets

A partially ordered set (poset) is a set of elements together with a binary ordering relation .

}{},,{ acba }{},{ aba

1312

includes

covers


Lattices

A lattice is a poset P where every pair of elements x and y has

a least upper bound called the join

a greatest lower bound called the meet

yx

}{b

}{a }{c

},{ ca},{ ba },{ cb

},,{ cba },{}{}{ cbcb

}{},{},{ bcbba Similarly

yx

The greenelements areupper boundsof the blue circledpair. The green circled element is theirleast upper bound ortheir join.


The Lattice Identities

L1. Idempotent

L2. Commutative

L3. Associative

L4. Absorption

If the meet and join follow the Consistency Relations

C1. (x is the greatest lower bound of x and y)

C2 . (y is the least upper bound of x and y)

Lattice Identities

xxxxxx ,

xyyxxyyx ,

zyxzyxzyxzyx )()(,)()(

xyxxyxx )()(

xyx

yyx

yx


Lattices are Algebras

aba

bbaba

StructuralViewpoint

OperationalViewpoint


Lattices are Algebras

aba

bbaba

StructuralViewpoint

OperationalViewpoint

aba

bbaba

Assertions, Implies

aba

bbaba

Sets, Is a subset of

aba

bbaba

),gcd(

),lcm(|

Positive Integers, Divides

aba

bbaba

),min(

),max(Integers, Is less than or equal to

Hypothesis Space(Our States of Knowledge)


Describing a State of Knowledge

These statements describe the fruit:

a = ‘It is an Apple!’

b = ‘It is a Banana!’

c = ‘It is an Citrus Fruit!’

They can also describe what one knows about the fruit.


The Hypothesis Space

a b c

The atoms are mutually exclusive and exhaustive logical statements

a = ‘It is an apple!’

b = ‘It is a banana!’

c = ‘It is a citrus fruit!’


The Boolean Hypothesis Space

a b c

The meet of any two atoms is the absurdity: a b =

We do not allow our state of knowledge to include:

‘The fruit is an apple AND a banana!’



a b c



a b c

a b

The join of any two elements represents a logical OR: a b



a b c

a b a c b c

The join of any two elements represents a logical OR: a b



a b c

a b a c b c

The final join gives us the TOP element, often called the TRUISM

= a b c“It is an Apple or a Banana or an Orange!”


The Hypothesis Space

This is a HYPOTHESIS SPACE!!!

It consists of all the statements that can be constructed from a set of mutually exclusive exhaustive statements.

The space is ordered by the ordering relation “implies”

We allow concepts like:‘The fruit is an apple OR a banana!’

while we disallow concepts like:‘The fruit is an apple AND a banana!’

a b c

a b a c b c


Superpositions of States???

This was your initial state of knowledge.

The fruit was never in this state!

This was the state of the fruit.


Two Spaces

Space of statements

describing the fruitSpace of statements describing

a state of knowledge about the fruit

Generalizing Partial Orders to Measures


Inclusion and the Zeta Function

yxif

yxifyx

0

1),(

The Zeta function encodes inclusion on the lattice.

a b c

a b a c b c



yxif

yxifyx

0

1),(

a b c

a b a c b c

1),( baa

0),( abasince

since

aba baa



The Zeta Function

yxif

yxifyx

0

1),(

a b c

a b a c b c

a b c avb avc bvc T

1 1 1 1 1 1 1 1

a 0 1 0 0 1 1 0 1

b 0 0 1 0 1 0 1 1

c 0 0 0 1 0 1 1 1

avb 0 0 0 0 1 0 0 1

avc 0 0 0 0 0 1 0 1

bvc 0 0 0 0 0 0 1 1

T 0 0 0 0 0 0 0 1



yxif

yxifyx

0

1),(


We can define its dual by flipping around the ordering relation

yxif

yxifyx

0

1),(


Degrees of Inclusion and Z

yxif

yxifyx

0

1),(

We generalize the dual of the Zeta function

yxif

yxifz

yxif

yxz

0

1

),(

to the function z


Z

yxif

yxifz

yxif

yxz

0

1

),(

The function z

Continues to encode inclusion, but has generalized the concept to degrees of inclusion.

In the lattice of logical statements ordered by implies, this function describes degrees of implication.


How do we Assign Values to z?

yxif

yxifz

yxif

yxz

0

1

),(

Are all of the values of the function z arbitrary?

Or are there constraints?

Here there be monsters…

a b c avb avc bvc T

1 0 0 0 0 0 0 0

a 1 1 0 0 ? ? 0 ?

b 1 0 1 0 ? 0 ? ?

c 1 0 0 1 0 ? ? ?

avb 1 1 1 0 1 ? ? ?

avc 1 1 0 1 ? 1 ? ?

bvc 1 0 1 1 ? ? 1 ?

T 1 1 1 1 1 1 1 1


Lattice Structure Imposes Constraints

I showed that in “general”:

Associativity leads to a Sum Rule…

Distributivity leads to a Product Rule…

Commutivity leads to Bayes Theorem…

zyxzyx )()(

)()()( zxyxzyx

xyyx

),(),(),(),( wyxzwyzwxzwyxz

),(),(),( wxyzwxzwyxz

),(

),(),(),(

wyz

wxyzwxzwyxz


Inclusion-Exclusion (The Sum Rule)

),(),(),(),( wyxzwyzwxzwyxz The Sum Rule for Lattices



)|()|()|()|( iyxpiypixpiyxp

The Sum Rule for Probability




),()()();( YXHYHXHYXI

Definition of Mutual Information





),min(),max( yxyxyx

Polya’s Min-Max Rule for Integers






This is intimately related to the Möbius function for the lattice, which is related to the Zeta function.




),min(),max( yxyxyx


Probability

yxif

xyifp

xyif

yxp

0

1

)|(

Changing notation

The MEANING of p(x|y) is made explicit via the Zeta function.

These are degrees of implication!

NOT plausibility!NOT degrees of belief!NOT frequencies of occurrences!

yxif

yxifz

yxif

yxz

0

1

),(


Statement – QuestionDuality


Richard T. Cox (1979) defined a question as the set of all possible assertions that answer it. I recast his definition to obtain new insights.

In lattice theory, such a set is called adown-set

In the figure the colored elementsbelong to

I call the set of top elements the irreducible set.

Defining a Question

cba

ba ca cb

ba c

xX

},{

)()(

cbba

cbbaBCAB


When are Questions Equal?

Two questions are equivalent when they are defined by the same set of assertions.

Consider the questions

“Is it raining?”

“Is it not raining?”

They are both answered by the set of assertions generatedby the irreducible set { “It is raining!”, “It is not raining!”} They are therefore equivalent.


Animal, Vegetable, Mineral

Space of statementsdescribing the object

Space of Statementsdescribing a state of knowledge about

the object

a v m


The Central Issue

I = “Is it an Animal, a Vegetable, or a Mineral?”

This question is answered by the following set of statements:

I = { a = “It is an animal!”, v = “It is a vegetable!”, m = “It is a mineral!” }

},,,{ mvaI


Some Questions Answer Others

Now consider the binary question

B = “Is it an animal?”

B = {a = “It is an animal!”, ~a = “It is not an animal!”}

As the defining set of I is exhaustive, mva ~

},,,,{ mvmvaB


Ordering Questions

B = “Is it an animal?”

I = “Is it an Animal, a Vegetable, or a Mineral?”

BI I answers B

B includes I

},,,{ mvaI

},,,,{ mvmvaB


Meets and Joins of Questions

With “is a subset of” as the ordering relation among questions,one can show that:

The meet of two questions is the set intersection of the set of assertions answering the question.

The join of two questions is the set union of the set of assertions answering the question.

YXYX

YXYX


Ideals

An ideal is a nonvoid subset J of a lattice A with the properties (Birkhoff 1967)I1. , where thenI2. , then

I1 is the condition for the set J to be a down-set, or equivalently a question.

I2 assures that there is a unique maximum

Therefore, as ideals are questions, I call them Ideal Questions.

JxJx Jz

Ay xy Jzx

Jy


Ideals and Ideal Questions


There are More Questions!

Questions can be formed from set unions of ideal questions.

which can be written

These questions along with the ideal questions form alattice ordered by set inclusion.

)()( mvaMAV

},,,,{},{},,,{ mvavamvava


The Lattice of Questions

The lattice of questions is the Free Distributive Lattice, Q(N) = FD(N)

No complements!


Real Questions

Real Questions are questions thatcan always be answered by a true assertion.

Real questions must include the exhaustive set of mutually exclusive assertions.

The Real Question lattice R(N)looks almost Boolean for N=3,but this is not true in general.


Real Questions

In Q(3) these are the questions R where

The bottom question is called the Central Issue and as such it results in an unambiguous answer.

RMVA

R(3)


Partition Questions

A partition question P neatly partitions the possible answers.These are the join-irreducible elements of the lattice R(N).

partition latticeR(3)


Valuations on the Question Lattice


Inquiry Calculus

Generalizing the zeta function for the question lattice gives a valuation that describes the degree to which the question Q answers the issue I

I call this quantity the relevance

The relevance follows the Sum Rule

otherwised

QIif

QIif

QId

10

0

1

)|(

),()|( QIzQId


Consistency Principle

Probability assignments in A must be consistent with relevance assignments in Q


Ideal Questions again

Only one assumption:

))|(,),|(),|(()|( 21 TTT nn xpxpxpHPId

The degree to which a partition question resolves the central issue is a function of the probabilities of the statements comprising the partition question’s irreducible set.

The goal is to determine the form of the functions Hn().

Again, the lattice properties constrain the solution!


Consequences of Our Assumption

YXiffYIdXIdYXId )|()|()|(

1. Additivity

(sum rule)

otherwiseYIdXIdYXId )|()|()|(

2. Subadditivity

(sum rule)

)|()|( )()2()1(21 nn XXXIdXXXId

3. Commutativity

(commute)

4. Expansibility

(identifying with bottom)

)|()|( 2121 nn XXXIdFXXXId


Entropy

),,,(),,,(),,,( 212121 nonnn pppHbpppHapppH

n

iiin pppppH

1221 log),,,(

NpppH no 221 log),,,(

Shannon

Hartley

Aczél (Aczél et al., 1974): The function Hm satisfying additivity, subadditivity, commutativity and expansibility is a linear combination of Shannon (1949) and Hartley (1928) entropies


Normalization Conditions

)(

log1 2

IH

Nba

Relevance is maximized when

1)|( IId

This implies that

Relevance is minimized when which gives0)|( TId

I

T

)()(log

)()1()(

12

1

IHNPHN

IHNPH

bP

ii

P

ii


Results

a

a

a a

aa

v v v

v v v

m m m

m m m

AVM VAM MAV

AVAM AVVM AMVM


Relevanceand

Information Theory


Relevance and Entropy

)|( QId

mmvvaamva pppppppppHIH 222 logloglog),,()(

),( mva ppH


Higher-Order Informations

))()(|()|()|()|( VMAAMVIdVMAIdAMVIdVMAMId

);(~)|( VMAAMVIVMAMId

This relevance is related to the mutual information.

In this way one can obtain higher-order informations.


Questions and Geometry


Join-Irreducible Elements

a

a

v

a

v

m

The join-irreducible elements are the simplexes.With the bottom, they form a Boolean Lattice.


Questions ~ Simplicial Complexes

Is it or Is it Not an Animal?

a m

v

mv


Lattice of Simplicial Complices

The lattice of questions is isomorphic to the lattice ofsimplicial complices.

These are alsoisomorphic to the lattice of hypergraphs.


Summary Inclusion can be generalized to degrees of inclusion

The lattice structure places constraints on the values of these degrees of inclusion.

On a Distributive Lattice, these constraints take the form of a Sum Rule, a Product Rule and a Bayes Theorem.

The logical statements ordered by implication gives rise to degrees of implication that follow the rules of probability theory

The lattice of questions is dual to the lattice of assertions in the sense of Birkhoff’s Representation Theorem.

The relevance of questions depends on the Shannon and Hartley entropies.

Special Thanks to:

John SkillingAriel CatichaJanos AczélPhilip GoyalSteve GullJeffrey JewellCarlos Rodriguez

for many insightful discussions and comments

Supported in part by:NASA AIST-QRS-07-0001NASA 05-AISR05-0143

Documents

An Order-Theoretic Approach toward Entropy