Information TheoryInformation Theory

informationinformation

the non-redundant portion of any string the non-redundant portion of any string of numbersof numbers

.333333333333 . . ., holds very little .333333333333 . . ., holds very little information, since the 3s can be information, since the 3s can be expressed more succinctly as a fraction.expressed more succinctly as a fraction.

π, or 3.141592653589793 . . ., with its π, or 3.141592653589793 . . ., with its apparently endless non-patterned and apparently endless non-patterned and non-repeating sequence of numbers, non-repeating sequence of numbers, contains a very high level of information contains a very high level of information (often called (often called entropyentropy). ).

entropyentropy

Information measures the amount of Information measures the amount of non-redundant information in data. non-redundant information in data.

Entropy measures the amount of Entropy measures the amount of improbability or unpredictability in improbability or unpredictability in data. data.

Preference or aesthetic Preference or aesthetic valuevalue

High information is no better or High information is no better or worse than low information worse than low information (entropy).(entropy).

ExampleExample

language is highly redundantlanguage is highly redundant J-st tr- t- r--d th-s s-nt-nc-. J-st tr- t- r--d th-s s-nt-nc-. The meaning of a message can The meaning of a message can

remain unchanged even though remain unchanged even though parts of it are removed. parts of it are removed.

Definition Definition

Information theoryInformation theory is a branch of is a branch of communications theory that deals communications theory that deals with the amount and accuracy of with the amount and accuracy of information when transmitted from a information when transmitted from a source, through a medium, to a source, through a medium, to a destination. destination.

Claude Shannon (1949)Claude Shannon (1949)• (1) Zero-order (1) Zero-order symbolsymbol approximation as in ZZFYNN PQZF LQN, approximation as in ZZFYNN PQZF LQN,

where errors and correct symbols exist side by side;where errors and correct symbols exist side by side;

• (2) First-order (2) First-order symbolsymbol approximation (including letter frequency), approximation (including letter frequency), as in ID AHE RENI MEAT, where letter frequency is taken into as in ID AHE RENI MEAT, where letter frequency is taken into consideration as well as which letter likely follows which letter;consideration as well as which letter likely follows which letter;

• (3) Second-order (3) Second-order symbolsymbol approximation, as in RENE ID AHA MIET, approximation, as in RENE ID AHA MIET, where letter ordering two steps back is taken into consideration;where letter ordering two steps back is taken into consideration;

• (4) Third-order (4) Third-order symbolsymbol approximation as in HE ARE ID TI NEAM, approximation as in HE ARE ID TI NEAM, where letter ordering three steps back is taken into consideration;where letter ordering three steps back is taken into consideration;

• (5) First-order (5) First-order wordword approximation as in I DARE HE IN TAME, approximation as in I DARE HE IN TAME, where letters and now word existence are taken into consideration;where letters and now word existence are taken into consideration;

• (6) Second-order (6) Second-order wordword approximation as in I DARE HE NAME IT, approximation as in I DARE HE NAME IT, where letters, word existence, and probable word order are taken where letters, word existence, and probable word order are taken into consideration;into consideration;

Algorithmic information Algorithmic information theorytheory

(AIT), a branch of information (AIT), a branch of information theory, concentrates less on the theory, concentrates less on the communication accuracy of communication accuracy of information, and more on the precise information, and more on the precise amount of non-compressible amount of non-compressible information contained in a message. information contained in a message.

Gregory Chaitin Gregory Chaitin

the complexity of something to be the size of the complexity of something to be the size of the simplest theory for it, in other words, the the simplest theory for it, in other words, the size of the smallest program for calculating size of the smallest program for calculating it. This is the central idea of algorithmic it. This is the central idea of algorithmic information theory (AIT), a field of information theory (AIT), a field of theoretical computer science. Using the theoretical computer science. Using the mathematical concept of program-size mathematical concept of program-size complexity, we exhibit irreducible complexity, we exhibit irreducible mathematical facts, mathematical facts that mathematical facts, mathematical facts that cannot be demonstrated using any cannot be demonstrated using any mathematical theory simpler than they are.mathematical theory simpler than they are.

Compression Compression

An important component of AIT is An important component of AIT is that the files compressed that the files compressed mustmust be be "lossless," that is, restorable (by "lossless," that is, restorable (by reverse processing) to their precise reverse processing) to their precise original form. original form.

Musical Algorithmic Musical Algorithmic Information TheoryInformation Theory

MIDI (Musical Instrument Digital MIDI (Musical Instrument Digital Interface) presents an interesting Interface) presents an interesting example of compressionexample of compression

MIDI produces accurate results with MIDI produces accurate results with a fraction of the storage space a fraction of the storage space required.required.

MIDI also loses a significant amount MIDI also loses a significant amount of information due to its score of information due to its score representation approach representation approach

Better exampleBetter example

The information content of a musical The information content of a musical work—the part that cannot be work—the part that cannot be further reduced—consists of further reduced—consists of material that does not repeat material that does not repeat sufficiently, exactly, or in variation, sufficiently, exactly, or in variation, such that a symbol can replace it to such that a symbol can replace it to compress that passage. compress that passage.

Aesthetics againAesthetics again

low information content music is low information content music is more developed and well formed, more developed and well formed, while higher information content while higher information content music is more erratic and music is more erratic and disorganized???????disorganized???????

better music has lower information better music has lower information content?????????content?????????

Not necessarily.Not necessarily.

Examples Examples

information contents of various information contents of various segments of works by J. S. Bach segments of works by J. S. Bach (0.51), W. Mozart (0.45), Ludwig (0.51), W. Mozart (0.45), Ludwig Beethoven (0.50), Johannes Brahms Beethoven (0.50), Johannes Brahms (0.64), Anton Webern (0.88), Ernst (0.64), Anton Webern (0.88), Ernst Krenek (0.87), and David Cope Krenek (0.87), and David Cope (0.63) using a very simple (0.63) using a very simple compression scheme demonstrate compression scheme demonstrate interesting resultsinteresting results

Dynamic Musical AIT Dynamic Musical AIT (DMAIT).(DMAIT).

In music, the In music, the ear shifts from one ear shifts from one musical parameter to another.musical parameter to another.

Melody, harmony, dynamics, timbre, Melody, harmony, dynamics, timbre, rhythm, and so on, all jostle for rhythm, and so on, all jostle for attention with each succeeding at attention with each succeeding at one time or another.one time or another.

The mind’s ear tends to go where The mind’s ear tends to go where the most information occurs.the most information occurs.

How it worksHow it works

Dynamic Musical AIT systematically Dynamic Musical AIT systematically refigures Musical AIT for several musical refigures Musical AIT for several musical parameters simultaneously.parameters simultaneously.

DMAIT fingerprints help identify the DMAIT fingerprints help identify the important elements of music at different important elements of music at different times.times.

To avoid flatlining in longer works, a To avoid flatlining in longer works, a moving DMAIT aperture of several beats moving DMAIT aperture of several beats preceding the already calculated preceding the already calculated information contentavoids re-computing information contentavoids re-computing information from the very beginning of a information from the very beginning of a work. work.

Bartók's Mikrokosmos Bartók's Mikrokosmos No. 81 No. 81

A DMAIT graph of previousA DMAIT graph of previous