Of 27 12/03/2015 Boole-Shannon: Laws of Communication of Thought 1 Laws of Communication of Thought? Madhu Sudan Harvard

of 2712/03/2015 Boole-Shannon: Laws of Communication of Thought 1

Laws of Communication of Thought?

Madhu SudanHarvard

of 27

George Boole (1815-1864)

The strange math of

Typical Derivation: Axiom:

Example: Object is Good and Good Object is Good Consequence: Principle of Contradiction

“… it is impossible for any being to possess a quality and at the same time to not possess it.”

Proof: or

or does not hold

12/03/2015 Boole-Shannon: Laws of Communication of Thought 2

(page 34)

of 27

vs.

Boole’s Mathematics: Focus on tiny part of mathematical universe.


: Algebra/Calculus

: optimization

: number theory

{0,1}

ProgressIn Math

of 27

Boole’s “modest” ambition

“The design of the following treatise is to investigate the fundamental laws of those operations of the mind by which reasoning is performed; to give expression to them in the symbolical language of a Calculus, and upon this foundation to establish the science of Logic and construct its method; to make that method itself the basis of a general method for the application of the mathematical doctrine of Probabilities; and, finally, to collect from the various elements of truth brought to view in the course of these inquiries some probable intimations concerning the nature and constitution of the human mind.” [G.Boole, “On the laws of thought …” p.1]


: Algebra/Calculus

: optimization

: number theory

{0,1} Mathematics All of reasoning

of 27

Shannon (1916-2001)

“The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point. ... The system must be designed to operate for each possible selection, not just the one which will actually be chosen since this is unknown at the time of design.” [Mathematical Theory of Communication. 1948]


Engineering is the motivation! Mathematics emerges from the theory

Information (Bit), Entropy, Capacity, Rate … The Probabilistic Method!

… And captures natural phenomena

Yet …

of 27

“We can also approximate to a natural language by means of a series of simple artificial languages.”

order approx.: Given symbols, choose according to the empirical distribution of the language conditioned on the length prefix.

3-order (letter) approximation“IN NO IST LAT WHEY CRATICT FROURE BIRS GROCID PONDENOME OF DEMONSTURES OF THE REPTAGIN IS REGOACTIONA OF CRE.”

Second-order word approximation“THE HEAD AND IN FRONTAL ATTACK ON AN ENGLISH WRITER THAT THE CHARACTER OF THIS POINT IS THEREFORE ANOTHER METHOD FOR THE LETTERS THAT THE TIME OF WHO EVER TOLD THE PROBLEM FOR AN UNEXPECTED.”

“ order approx. produces plausible sequences of length ”

E.g. “Series of approx. to English”


of 27

Interim Period: Boole Shannon


?

X+ X

X

!

of 27

The broader picture

Reasoning + Communication athematics Even Shannon couldn’t resist models of

language!

Importance of the discrete world: Captures reasoning [Boole] … or does it? Maybe only captures communicable reasoning?

The Communication Lens Maybe all intellectual endeavor in society can be

captured mathematically?


of 27

Some CS Adventures


of 27

1950-today

Old “axioms” Discrete communication Discrete reasoning

New “axiom” Resource bounded computing. (P,NP …)

Many phenomena captured: Cryptography, (Pseudo-)Randomness,

Knowledge, Privacy


of 27

Pseudorandomness?

Is the sequence random? Answer 1: No … no finite sequence is random Answer 2: No ….Those are the th digits of

Why does it feel random then? Interpretation of [Blum-Micali ‘82] …

Can certainly tell after ~ steps of computations.

But could I have done it in ~ steps, or ? Or ? Randomness: In the eye (mind) of beholder


of 27

(computational in-)Distinguishability

[Shannon] “Comm. Theory of Secrecy Systems” Variables if for every test ,

[Goldwasser-Micali’82] Computational (eye-of-beholder) Indistinguishability

Can apply to your favorite Explains many more “seemingly” random things.


of 27

Knowledge?

Some sequences have lots of information but little knowledge. E.g., my email … every day! Why? What is the difference?

Can sequences have more knowledge than information? E.g., your banking password:

Your internet provider already has sufficient “information”, but hopefully not “knowledge”


of 27

Knowledge [GoldwasserMicaliRackoff’86]

Axioms of Knowledge: Should be useful … can do something with it

that you can’t without. Utility should be “verifiable”.

Theorems [GoldreichMicaliWigderson]: Can design interactions that reveal exactly what is intended and no more (“zero-knowledge”). Applications:

“Pepsi Coke!”, “Where is Waldo?” Secure MultiParty Computation


Would you pay for it?

Shouldn’t pay for random bits!

of 27

Communication of Thought


of 27

Motivation

Communication as a “network phenomenon” Intellectual Pursuit:

Start with existing knowledge. Add to the body of knowledge.

Find new facts/concepts/designs that rely on knowledge from past.

Repeat. Two questions:

What is the error-correction mechanism? How do we communicate “knowledge”

(language, dictionary, journals …)


of 27

Error-correction via proofs

Communication of Mathematical knowledge: Doesn’t matter how you arrive at the new

piece of knowledge. But must provide “self-contained proof” of the

new piece. Sound approach:

Is it slowing us down? Verification time-consuming Proofs long …


of 27

Variations on Proofs

Central part of “CS Adventures” Relates to computation [Turing, Church, Gödel]

Proofs need to be “easy” to verify. Relates to cryptography [GMR]

How can user convince system it has the right to access some resource.

Relates to optimization [Cook, Levin, Karp] Better mechanisms for proofs lead to

stronger barriers to optimization.


of 27

Variations on Proofs - 2

Interactive Proofs [GoldwasserMicaliRackoff, Babai] Use power of bidirectional communication

(with some randomness) to improve efficiency of verification

E.g. can prove “Pepsi Coke” interactively. Can prove “non-existence of short proof”

[LundFortnowKarloffNisan, Shamir] with short interaction!


of 27


Probabilistically Checkable Proofs [AroraSafra’92] Let verifier toss random coins to “sample”

proof. PCP Theorem [AroraLundMotwaniSafraSudanSzegedy’92,Hastad’97]

format to write proof (which requires “slightly” longer proofs) where the verifier reads only bits of the proof to check it, to get confidence.

State of the art versions [Dinur’06]: ; [MoshkovitzRaz’08]:


of 27


Computationally Sound (CS) Proofs [Kilian’90,Micali’94] “proofs” much shorter than classical proofs. Proofs of incorrect statements can exist, but

are very hard to find!

Incrementally Verifiable Proofs [P.Valiant ‘08] Can utilize pre-existing knowledge+proofs to

generate (short, CS) proofs of new facts

Proofs for Delegated computing, Quantum Proofs …


of 27

Do proofs solve the problem?

No. (Even in Mathematics) Proofs are never self-contained.

Always assume (even to state theorem) … Language (Unboundedly Large). Background/Context (Bounded, but still

enormous). Most effective communication relies on the fact

that context is shared (even if imperfectly).


“We give a elementary self-contained proof of *** theorem”.

But two sentences earlier: “We assume reader is familiar with [***]” (60 pages)

of 27

Unbounded Language Problem

[GoldreichJubaSudan’14] How to learn language, or even detect misunderstanding, using (only) communication? Can’t try to learn language while

“communicating” unless you have a goal for communication!

But if Goal exists, is useful, and verifiable then Goal can be achieved even with misunderstanding.


Same assumptions as in “Knowledge”. Not coincidental.

of 27

Communicating with large context

Example: Compression Sender , drawn from distribution . Needs to send to receiver, compressed. [Shannon, Huffman]: In expectation need to send only

bits. Implied context: the distribution Schemes break if context not perfectly shared!

Contrast Communication length: Context length:

Can we get rid of perfectly shared context? [JubaKalaiKhannaSudan’11] – Yes. [HaramatySudan’14] – Maybe.


of 27

More about Context

Many forms of communication rely on shared context. Natural Communication:

Language, Grammar, Common Knowledge … (typically each unbounded).

Designed Communication: Protocols, Encoding/Decoding functions

(used to be bounded, now no longer so). Are they resilient to imperfect sharing?

Natural … maybe! What are the axioms? Designed … no! What can we do?


No Certainty (ever) Trust but check Simplicity first Survival of fittest

of 27

Broader investigation of Communication

Social/Network Process: No one player makes all rules In fact may have few central players Cooperative game

What are good practices: Can we use Boolean calculus and information theoretic

principles to compare solutions. Can we recommend good solutions:

Ultimate archival format for documents in digital libraries? Let machines (learn to) talk to each other without human

intervention. Can we measure societal (intellectual) progress? Will it be

monotone?


of 27

Thank You!


Documents

Of 27 12/03/2015 Boole-Shannon: Laws of Communication of Thought 1 Laws of Communication of Thought? Madhu Sudan Harvard