2003.03.10 - SLIDE 1IS246 - SPRING 2003

Lecture 13: Computational Media Theory

IS246Multimedia Information

(FILM 240, Section 4)

Prof. Marc DavisUC Berkeley SIMS

Monday and Wednesday 2:00 pm – 3:30 pmSpring 2003

http://www.sims.berkeley.edu/academics/courses/is246/s03/

2003.03.10 - SLIDE 2IS246 - SPRING 2003

Today’s Agenda

• Review of Last Time

– Theory of Computation

• Computational Media Theory

– Manovich on New Media

– Bloch on Representing Film

– Dorai and Venkatesh on Computational Media

Aesthetics

– Discussion Questions

• Action Items for Next Time

2003.03.10 - SLIDE 3IS246 - SPRING 2003

Computation in Intellectual History

• Computation as instrumentality– PCs, PDAs, embedded processors, etc.

• Computation as ideas– Modeling process– Languages for modeling process– Primitives, combination, abstraction– Parameterization– Black boxing functionality– Optimization

2003.03.10 - SLIDE 4IS246 - SPRING 2003

Algorithms and Programming

• Algorithm– A step-by-step description of a procedure to

achieve a desired result

• Programming– Primitives– Means of combination– Means of abstraction

2003.03.10 - SLIDE 5IS246 - SPRING 2003

Computation for Designing Artifacts

• Four computational ideas/techniques from Carlo Sequin– Procedural generation– Parameterization– Optimization– Evolutionary power

2003.03.10 - SLIDE 6IS246 - SPRING 2003

Today’s Agenda

• Review of Last Time

– Theory of Computation

• Computational Media Theory

– Manovich on New Media

– Bloch on Representing Film

– Dorai and Venkatesh on Computational Media

Aesthetics

– Discussion Questions

• Action Items for Next Time

2003.03.10 - SLIDE 7IS246 - SPRING 2003

What New Media Is Not Defined By

• Digitized analog media

• Media displayed on a computer

• Random access media

• Necessarily having less information than analog media

• Necessarily being able to be copied without generation loss

• Being “interactive”

2003.03.10 - SLIDE 8IS246 - SPRING 2003

Manovich on New Media

• Numerical representation– Can be described formally (mathematically)– Can be manipulated algorithmically (programmability)

• Modularity– Constructed out of substitutable components

• Automation– Automation of media creation, manipulation, and access– Low level (bits) and high level (semes) automation

• Variability– Media objects can have potentially infinite versions– Media database, separation of data and interface,

customization/personalization, branching-type interactivity, hypermedia (links), periodic updates, scalability (e.g., resolution)

• Transcoding– … Media => Data => Media …

2003.03.10 - SLIDE 9IS246 - SPRING 2003

Today’s Agenda

• Review of Last Time

– Theory of Computation

• Computational Media Theory

– Manovich on New Media

– Bloch on Representing Film

– Dorai and Venkatesh on Computational Media

Aesthetics

– Discussion Questions

• Action Items for Next Time

2003.03.10 - SLIDE 10IS246 - SPRING 2003

Bloch “From Concepts to Film Sequences”

• Describing shots– ACTION performed by the actors– SETTING– ACTORS– Direction of the LOOKs of the main actors– POSITIONs of the actors in the frame– Direction and speed of the apparent MOTION of the

main actors or objects in motion• Constructing sequences

– CHOICE– CONSTRUCTION– When APPRECIATION bad, CORRECTION– PROJECTION

2003.03.10 - SLIDE 11IS246 - SPRING 2003

Today’s Agenda

• Review of Last Time

– Theory of Computation

• Computational Media Theory

– Manovich on New Media

– Bloch on Representing Film

– Dorai and Venkatesh on Computational Media

Aesthetics

– Discussion Questions

• Action Items for Next Time

2003.03.10 - SLIDE 12IS246 - SPRING 2003

The Semantic Gap

• “However, the semantic gap between the rich meaning that users want when they query and browse media and the shallowness of the content descriptions that we can actually compute is weakening today’s automatic content-annotation systems.”

2003.03.10 - SLIDE 13IS246 - SPRING 2003

Computational Media Aesthetics

• “ […] the algorithmic study of a variety of image and aural elements in media (based on their use in film grammar). It is also the computational analysis of the principles that have emerged underlying their manipulation in the creative art of clarifying, intensifying, and interpreting an event for an audience.”

• “Our research systematically uses film grammar to inspire and underpin an automated process of analyzing, characterizing, and structuring professionally produced videos.”

2003.03.10 - SLIDE 14IS246 - SPRING 2003

CMA Challenges

• Can we dynamically detect successful aesthetic principles with accuracy and consistency using computational analysis?

• Can we build new postproduction tools based on this analysis for rapid, cost-efficient, and effective moviemaking and consistent evaluation?

• How can we use these successful audio–visual strategies for improved training and education in mass communication?

• How do we raise the quality of media annotation and improve the usability of content-based video search and retrieval systems?

2003.03.10 - SLIDE 15IS246 - SPRING 2003

Today’s Agenda

• Review of Last Time

– Theory of Computation

• Computational Media Theory

– Manovich on New Media

– Bloch on Representing Film

– Dorai and Venkatesh on Computational Media

Aesthetics

– Discussion Questions

• Action Items for Next Time

2003.03.10 - SLIDE 16IS246 - SPRING 2003

Discussion Questions

• On Manovich on New Media (Lisa Wang)– Manovich wrote, "Before, we would read a sentence of a story

or a line of a poem and think of other lines, images, memories. Now interactive media asks us to click on a highlighted sentence. In short, we are asked to follow pre-programmed, objectively existing associations. Put differently... we are asked to mistake the structure of somebody else's mind for our own." (p. 314 of the reader)

– When viewing a piece of media, each person accesses their individual repertoire of associations and connotations. Despite this subjectivity, there are many associations that are shared within a culture. Filmmakers draw upon these shared associations to tell their stories. Could a web or database of common associations and connotations be built? How could it be used in conjunction with semantic-based editing programs?

2003.03.10 - SLIDE 17IS246 - SPRING 2003

Discussion Questions

• On Manovich on New Media (Phoebe Liu)– Through out the article, the author talks about one

general tendency of new media that it favors individuality over conformity and the switch from mass standardization to individual customization. This means that the user is given many options to modify the performance of a program or a media object. In terms of designing multimedia information systems, say if we want to design a automatic editing program for some home videos, what are the important factors we will be looking for in order to achieve individual customization? 

2003.03.10 - SLIDE 18IS246 - SPRING 2003

Discussion Questions

• On Bloch on “From Concepts to Film Sequences” (David Warthen)– In describing “the program,” Gilles refers to the program’s operators

(e.g., CHOICE, CONSTRUCTION, etc.) “recognizing” and “examining” aspects of the film content in order to generate sequences.

• Assuming that these operators are working with human-generated markups of the film content, what would this markup need to look like? In particular, considering our discussion of algorithms in the last class, what level of detail would be necessary to describe the shots so that the software could reasonably generate the sequence?

• Assuming that the software itself is analyzing the content performing the “recognition” operations, what capabilities might this software require?

– Gilles’ examples at the end relate to “live” film sequences, such as documentaries and news broadcasts. However, the state-of-the-art of computer generated film or video content has advanced greatly in recent years (e.g., Monsters Inc., Final Fantasy – The Spirits Within). How might the ability to generate content using such techniques extend Bloch’s ideas and system?

– Bloch is pursuing greater automation in the generation of film. What is the possible downside if such systems go into wide deployment? (That is, what new problems might such systems create?)

2003.03.10 - SLIDE 19IS246 - SPRING 2003

Discussion Questions

• On Dorai and Venkatesh on Computational Media Aesthetics (Lily Chen)– How can we deal with the

• Ambiguous factors which are rooted in aesthetics• Factors which may come from personal bias when

we do computational media analysis?

2003.03.10 - SLIDE 20IS246 - SPRING 2003

Today’s Agenda

• Review of Last Time

– Theory of Computation

• Computational Media Theory

– Manovich on New Media

– Bloch on Representing Film

– Dorai and Venkatesh on Computational Media

Aesthetics

– Discussion Questions

• Action Items for Next Time

2003.03.10 - SLIDE 21IS246 - SPRING 2003

Readings for Next Time

• Wednesday 03/12 Guest Lecturer: Jonathan Foote from FX Pal on “Automated Media Analysis for Audio”– Required

• Jonathan Foote. An Overview of Audio Information Retrieval. Multimedia Systems, vol. 7, 1999; pp. 1-18.

• Kenichi Minami, Akihito Akutsu, Hiroshi Hamada, and Yoshinobu Tomomura. Video Handling with Music and Speech Detection. IEEE MultiMedia, vol. 5, 1998; pp. 17-25.

– Optional (On IS246 Course Web Site)• Christian Spevak and Emmanuel Favreau. Sound Spotter - A

Prototype System for Content-Based Audio Retrieval. In: Proceedings of the 5th International Conference on Digital Audio Effects (DAFx-02). Hamburg, 2002.

• George Tzanetakis, Perry Cook. Automatic Musical Genre Classification of Audio Signals. In Proc. International Symposium for Audio Information Retrieval (ISMIR 2001) Bloomington, USA, October 2001.