Web contentcommonlyincorporates userprofile and trackingdata to personalizeinformation toclients. Video post-production anddelivery systems,however, generallypromote a one-size-fits-all authoringapproach. Viper letsproducers createcomplex videoprograms that canre-edit themselves inresponse to variousaudience-relatedfactors.

Consumers of Internet-mediatedcontent increasingly expect a highdegree of responsiveness in theironline experiences. Responsive

media sense and react usefully to factors such aspresentation conditions, audience profile, directinteraction, and usage history.1 Common Web-authoring tools let designers create content thatincorporates user profile and tracking informa-tion to personalize the presentation. Some of themost advanced sites include richly interactiveanimations and other elements created with Java,Flash, and other programming environments.

Thus far this revolution in responsiveness hasleft behind producers of television and othervideo content, whose tools still largely promotethe creation of one-size-fits-all programs tailoredonly for large audience segments. However, theincreasing availability of general-purpose com-puting power and bandwidth throughout theproduction chain (from cable head-ends to con-sumer set-top boxes, for example) makes possiblea new genre of video programming that exhibitsthe same level of responsiveness we now expectin our Web browsers.

In a client-side approach to video personaliza-tion, a source (broadcast center, cable head-end,or Internet server) sends a collection of mediaobjects with metadata describing their assemblyto a playback device (set-top box, personal videorecorder, or computer), which performs the finalediting while the viewer watches the program.

This not only allows for a fine-grained adaptationof content to each viewing situation, it avoidstransmitting potentially sensitive viewer infor-mation outside the viewing environment.Possible applications include audience-specificadvertisements and news presentations, educa-tional programs that adjust in real time to a stu-dent’s attention level and other emotionalresponses, or environmentally responsive videoinstallations in public spaces.

Viper is a tool for creating responsive videoprograms. It aims not to take editing controlfrom the program creator and give it to the view-er, but to increase video producers’ and directors’control over how their stories and messages arepresented in different situations and over whatfreedoms with the content, if any, are granted tothe audience. In other words, we wish to enablethe substitution of a fast-forward button with a“tell me the same story faster” button.

Viper system and processViper aims to take a step beyond current sys-

tems, which we discuss in the “Related Work inResponsive Television” sidebar, by supportingmore demanding forms of responsiveness thatmight require complex sensing systems and theability to seamlessly alter editing during programviewing. Rather than focus on the methods forsensing or gathering the information to which aprogram might respond, our work concentrateson how to use this information to modify edit-ing in a rich and meaningful way.

Viper supports multiple video and audiotracks and provides primitives for assemblingcomplex edits and effects—for example, L-cuts(transitions in which the audio cut precedes orfollows the video cut), audio and video inserts,dissolves, and slow motion—and for finelyadjusting these elements during playback.

Perhaps most importantly, Viper addressesthe need for a common framework for develop-ing responsive video applications. Unlike someother video tools, Viper doesn’t have a predeter-mined knowledge base or domain-specific edit-ing model that controls output assembly. Rather,Viper aims to provide a common set of extensi-ble primitives with which designers can buildapplication-specific models on their own.Because the models would be based on a com-mon framework, projects could share compo-nents, video producers could learn from eachother’s systems, and component libraries couldbe packaged and distributed.

88 1070-986X/03/$17.00 © 2003 IEEE Published by the IEEE Computer Society

Viper: AFramework forResponsiveTelevision

Stefan Agamanolis and V. Michael Bove Jr.Media Lab Europe and MIT Media Laboratory

Feature Article

Figure 1 (next page) diagrams the tasks associ-ated with creating a production in Viper. Afterpreproduction planning, the program designercaptures media and uses Viper to form an anno-

tated clip database. The designer also developsthe production’s editing model. These parallelprocesses result in the clip database and editingmodel used in the presentation system.

89

Several earlier projects illustrate the issues involved in pro-ducing responsive video content.

Borrowing techniques from natural-language processing,Bloch’s system applies cinematic rules defined within “scene”objects to assemble meaningful sequences of shots,1 inspiringlater systems such as Chua and Ruan’s Video Retrieval andSequencing System (VRSS).2

Sack and Davis demonstrated another early system, IDIC,that could generate simple silent video stories, such as sciencefiction trailers, using a story planner based on a general prob-lem solver (GPS).3

Nack and Parkes developed AUTEUR, a tool that employeda hierarchical system to describe shot content and scene struc-ture that could generate a limited form of slapstick-style comi-cal video sequences.4

Mateas, Domike, and Vanouse created Terminal Time, a sys-tem that assembles a montage of archival film and videofootage to accompany a constructed narrative describing his-torical events, the focus and bias of which is dictated in part bythe audience’s answers to multiple choice questions at severalpoints during the presentation.5

Lienhart developed techniques for annotating home movieson the fly and automatically summarizing the movies in shortvideo abstracts.6

Several projects from the Interactive Cinema group at theMIT Media Lab explore automated editing, such as Brooks’Agent Stories, Murtaugh’s Contour/Dexter, Houbart’sViewpoints on Demand, and Morgenroth’s Homer. For moreinformation, see the MIT Media Lab Interactive Cinema groupWeb site at http://ic.media.mit.edu.

Some types of tightly structured content, such as newsbroadcasts, lend themselves to modeling in a way that sup-ports personalized playback. Researchers have pursued per-sonalized news video presentation based on contentmodeling since at least the 1980s.7 Kunieda and Wakita’sMovieTool generates content models through a combinationof automatic analysis and manual editing.8 The Informediaproject also experimented with summarizing news contentinto autodocumentaries, which incorporate results of userqueries.9

These works raise other important issues, such as

❙ Most of these systems can’t incorporate real-time audienceactivity or other passive feedback during a presentation andalter editing immediately, which could be extremely usefulin educational applications or interactive installations.

❙ Some systems use sophisticated annotations to guide shotsequence assembly and avoid cinematically awkward juxta-positions (match cutting). However, none of the systems con-sist of a general multitrack framework in which we can treataudio and video separately, and in which we might con-struct commonly used yet more complex kinds of edits ortransitions—such as video or audio inserts, keying and com-positing, and L-cuts.

❙ No common framework underlies the editing and annota-tion models and algorithms employed by these systems, andthus designers can’t easily share or combine components tomake new systems.

These issues greatly influenced our thinking in designing Viper.

References 1. G.R. Bloch, “From Concepts to Film Sequences,” Proc. User-Ori-

ented Content-Based Text and Image Handling (RIAO), CID, Paris,

1988, pp. 760-767.

2. T.-S. Chua and L.-Q. Ruan, “A Video Retrieval and Sequencing

System,” ACM Trans. Information Systems, vol. 13, no. 4, 1995,

pp. 373-407.

3. W. Sack and M. Davis, “IDIC: Assembling Video Sequences from

Story Plans and Content Annotations,” Proc. IEEE Int’l Conf. Multi-

media Computing and Systems, IEEE CS Press, 1994, pp. 30-36.

4. F. Nack and A. Parkes, “The Application of Video Semantics and

Theme Representation in Automated Video Editing,” Multimedia

Tools and Applications, vol. 4, no. 1, Jan. 1997, pp. 57-83.

5. M. Mateas, S. Domike, and P. Vanouse, “Terminal Time: An Ide-

ologically Biased History Machine,” Proc. AISB Symp. Artificial

Intelligence and Creative Language: Stories and Humor, Soc. for

the Study of Artificial Intelligence and the Simulation of Behav-

ior, 1999, pp. 69-75.

6. R. Lienhart, “Dynamic Video Summarization of Home Video,”

Proc. SPIE 3972: Storage and Retrieval for Media Databases, SPIE

Press, 2000, pp. 378-389.

7. V.M. Bove Jr., Personalcasting: Interactive Local Augmentation of

Television Programming, SM thesis, Media Arts and Sciences,

Mass. Inst. of Technology, 1985.

8. T. Kunieda and Y. Wakita, Package-Segment Model for Movie

Retrieval System and Adaptable Applications, Ricoh tech. report

no. 25, Ricoh Company Ltd., Nov. 1999, http://www.ricoh.

co.jp/rdc/techreport/No25/Ronbun/033039.pdf

9. H.D. Wactlar, “Informedia: Search and Summarization in the

Video Medium,” Proc. Imagina, Monaco, 2000.

Related Work on Responsive Television

Preproduction planningDuring preproduction planning, the designer

develops the program concept, which will informlater phases of the process. The program designermust answer such questions as

❙ What viewer or environmental variables willthe program respond to?

❙ How will the system sense or gather theseresponse factors?

❙ How can the system represent these factorsnumerically or textually, and as simply aspossible?

❙ What’s the video program’s overall structure?

❙ What types of editing alterations will the pro-gram exhibit?

❙ What material should we shoot or gather tocover all possible alterations?

❙ How can we produce this material mostefficiently?

This planning phase also includes other typi-cal video preproduction activities like scriptwrit-ing and storyboarding.

Capturing mediaIn a Viper production, because not every piece

of footage gathered will appear in every final ver-sion of the program, covering all possible editingvariations typically requires additional shooting.This suggests a shooting strategy that de-empha-sizes perfecting individual shots and favors cap-turing a wide variety of imagery within theconstraints of the program concept. Having aclear plan of the imagery to be captured will helpthe designer avoid redoing setups and shootingadditional scenes at a later time.

In many situations, producing the extrafootage might involve fairly simple variations ona theme or shooting setup. For example, aresponsive advertisement for a restaurant mightfeature imagery of dishes that are likely to appealto the viewer. It might possible to shoot a largenumber of dishes fairly quickly with similar cam-era and studio setups.

Although our productions have mostlyemphasized shooting new material, libraries ofpreproduced and repurposeable footage of thekind needed for a project might exist. Viperallows designers to import these libraries into theproduction if desired.

Forming a clip databaseAfter obtaining the video and audio source

material, the designer splits the material intoindividual clip objects and annotates the clipswith information about their content and func-tion in the larger scheme of the production. Theediting model will express how the playback sys-tem will choose clips from this database andassemble them into complete programs.

Designers can use Viper’s graphical interface(shown in Figure 2) to select source footage, setin and out points, name clips, and so on. Theycan also specify “fuzzy” in and out points to indi-cate flexibility in how the system may cut arounda critical portion of a clip to fit it into a certaintime slice or to alter pacing. Viper’s clip-basedapproach differs from non-shot-based descriptiontechniques such as Aguierre Smith and Daven-port’s Stratification System,2 but it does letdesigners reuse source material in multiple clipsthat might be annotated differently.

Depending on the implementation of theediting model, clip sequence can vary in differ-

90

IEEE

Mul

tiM

edia

Capturemedia

Makeclip database

Annotatedclip

database

Editingmodel

Presentationengine

Renderedvideo program

Responsevariablestates

Buildediting model

Preproductionplanning

Figure 1. Diagram of

the Viper production

process. Task areas are

in red, and data being

fed into the

presentation engine are

in blue.

ent versions of the assembled program, and itmight be difficult to predict exactly what typesof juxtapositions the model will generate. Thissuggests a clip-preparation strategy that doesn’tfocus on perfecting each clip’s in and out pointswith respect to other clips as do traditional lin-ear-editing strategies. Rather, designers shouldbuild a database of flexible clips and tailor themonly as much as necessary to support the desiredediting alterations.

Adding annotationsWhile building clip objects, designers must

add annotations that describe the clip’s contentor its intended use in the program. The graphicalinterface supports keywords and numerical scales,while other custom annotations, such as ontolo-gies or indications of relationships between clips,can be expressed in a text file.

Unlike generic and exhaustive approaches tomedia annotation, such as that employed byDavis’ Media Streams,3 Viper lets designers devel-op a concise set of annotations that are specific tothe production and likely to be needed to supportthe editing model under construction. For exam-ple, if the plan is to create an educational programthat will include more or less detail on a topic indifferent situations, the designer can rate clips byimportance and use this annotation in the edit-ing model to decide which clips to include in thefinal program. Or, when annotating home moviematerial, the designer of a responsive video fami-ly scrapbook might add keywords that list thefamily members in each clip.

Rather than develop a competing standard formedia content annotation, we wish to facilitatethe creation of production-specific annotationlibraries that designers can reuse in subsequentproductions or other editing systems. Our proto-type is just one of many possible interfaces andformats that might satisfy this approach. Futureversions of Viper could directly interact with therelevant standards for content description, suchas MPEG-74,5 and the Advanced AuthoringFormat (AAF, http://www.aafassociation.org), tolet users import annotated clip databases createdby different systems or export databases createdwith Viper in a form usable by other applications.

Building the editing modelAs they prepare the annotated clip database,

designers must also develop an editing model thatexpresses how the playback system will chooseclips from the database and combine them to cre-

ate different versions of a complete video pro-gram based on the selected response factors.

The input to the editing model is the anno-tated clip database and response factor states, andthe output is a multitrack edit decision list (EDL),which the system passes directly to the playbackengine.

We can envision the editing model in manyways. At a high level, the model expresses a storyplan similar to that of IDIC6 and several othertools (see the “Related Work on ResponsiveTelevision” sidebar). Viper editing models alsodescribe how the system will translate these storycomponents into visual representations by edit-ing and layering the material in the media data-base. How finely detailed the model is at eitherlevel is up to the program designer.

For this key task area, Viper provides a set ofprimitives, based in the Isis programming lan-guage,1 that aims to insulate the designer asmuch as possible from the details of general-pur-pose programming while remaining extensibleand flexible in how the designer create map-pings. Four classes of primitives let designersselect subsets of clips that satisfy certain proper-ties or constraints, order and arrange the clips invarious ways, and specify the details of theirtranslation into an EDL with transitions and

91

July–Septem

ber 2003

Figure 2. A screen

image of the Viper

interface for preparing

and annotating a

database of clips,

showing both keyword

and numerical scale

annotations.

effects. The “Viper Primitives” sidebar describesthese classes of primitives in more detail, and the“Viper Editing Model” sidebar (on p. 94) gives anextended example.

Viper’s chief advantage is its support for a gen-eral multitrack framework in which editing mod-els can generate output that includes thecomplex editing constructs and effects com-monly needed to create a compelling high-qual-ity video program.

Inspecting and refiningThe nature of many responsive video produc-

tions dictates that the designer can’t inspectevery version of a program the system might gen-erate before they’re presented. Instead, designersmust iteratively refine the editing model’s behav-ior and the clip database’s integrity until they’reconfident that the system will generate mean-ingful output for any given response state.

Viper’s model-building component runs inparallel with both the annotation tool and theplayback system to support an incrementaldesign process in which designers can change or

add clips or annotations and test or refine edit-ing model components as needed. Viper also letsdesigners manually set response factor states andview the system-generated EDLs in a graphicalform. To facilitate comparison among differentversions of a program, the interface has three sec-tions, each of which can display an entire EDL,as Figure 3 shows.

Presenting the resultAfter building the editing model and manu-

ally testing the program’s responsiveness, thedesigner connects the playback system to thesensors or data sources that will gather theresponse data. The designer must also decidehow the system will handle real-time changes toresponse variables, if changes are possible. Achange to the variables’ state will always causethe movie to be re-edited, and the designer canchoose how to transition the current movie intothe new version from several options, some ofwhich involve checkpoints the designer can addto the EDL to indicate appropriate transitionpoints:

92

IEEE

Mul

tiM

edia

Viper editing models are essentially computer programs writ-ten in Isis, a functional programming language tailored for mul-timedia prototyping and developed at our lab.Designers createediting models using a standard text editor and four main col-lections of primitives. They can create higher-level componentsby combining these basic primitives with general Isis syntax.

In general, a model begins by selecting appropriate mediaclips from the database and ordering them based on their anno-tations. The last step is detailing how the system should renderthese media elements with transitions and effects into a finaledit decision list (EDL). The system provides global variablesthat carry the states of the response factors for the designer touse as parameters in any part of the model.

For brevity, we list only a subset of the available primitives.More information is available elsewhere.1 See the “Viper EditingModel” sidebar for a full example of these primitives in use.

Clip selectionThis set of primitives lets a model select a subset of clips from

the database that satisfies certain properties or constraints.Essentially, this is how designers choose the story elements thatwill appear in the final program.

The identifier tag represents a keyword or other annotation,and cd represents the list of clips that will be narrowed.

❙ (select-tags tag tag … cd): Select all clips that have at least

one of the given annotations.

❙ (select-name name cd): Select all clips with the specified name.

❙ (select-val tag val cd): Select all clips in which the given anno-tation (usually a numerical scale) has the given value.

❙ (select-ineq tag eqproc val cd): Select all clips in which thespecified inequality holds for the given annotation and value.

Scoring and sortingA second class of primitives lets designers control aspects of

how the selected story elements will be ordered or arranged inthe final program.

❙ (sort-tag-increasing tag cd): Sort clips in increasing order ofthe given annotation’s value.

❙ (sort-tag-nearest tag val cd): Sort clips in order of the distanceof the given annotation from the given value.

❙ (score-tags newtag tags cd): Add a numerical annotationexpressing how many of the given annotations are attachedto each clip.

❙ (score-proc newtag sproc cd): Add an annotation based on

Viper Primitives

93

July–Septem

ber 2003

the return value of the given procedure when invoked oneach clip.

TransformationThe third class of primitives lets designers transform the

selected clips into short single-clip EDLs. Conceptually, theseprimitives represent the transformation of simple story elementsinto details of how the elements will be rendered in the finalprogram. At this point, designers can specify how tightly orloosely to trim a clip if it consists of fuzzy in and out points, andother intraclip effects.

❙ (prepare-clip clip): Make a single-clip EDL from the specified clip.

❙ (prepare-clip-time clip time): Make an EDL from the specifiedclip, trimmed to the given duration using any fuzzy in or outpoint as a guide.

❙ (prepare-clip-trim clip trimval): Make an EDL from the speci-fied clip, trimmed the given amount from 0.0 (very loose)to 1.0 (very tight).

EditingDesigners use a fourth set of primitives to assemble EDLs into

larger EDLs, aiming for a single EDL to represent the full programthat the playback engine will render. These primitives supportmultiple video and audio tracks, allowing designers to express

complex edits, such as inserts and L-cuts. Fades let designerscontrol each track’s visibility, while transition and mixing prim-itives let them mix and dissolve video or audio elements togeth-er in differing amounts to create customized effects.

❙ (video-only edl): Extract only the video elements of an EDL.

❙ (shift-start time edl): Shift the starting time of all EDL elementsby a given amount.

❙ (ramp-in ramptime edl): Place a ramp-in (dissolve) transitionat the beginning of every EDL element.

❙ (change-track tracknum edl): Change the track number of allEDL elements.

❙ (sequence-movies spacing edls): Sequence the given EDLs inorder, spaced by the given amount.

Additional primitives let designers create graphics, such astitle screens or overlays, and include them in a movie as theywould any other audio or video element.

Reference 1. S. Agamanolis, Isis, Cabbage, and Viper: New Tools and Strategies

for Designing Responsive Media, PhD dissertation, Media Arts and

Sciences, Mass. Inst. of Technology, 2001

Figure 3. A screen image of the Viper edit decision log visualizer showing versions of a campaign advertisement generated

for three viewer profiles.

❙ Interrupt the current movie immediately

❙ Interrupt the current movie at a checkpoint

❙ Wait until the current movie ends

❙ Enter the new movie at its beginning

❙ Enter the new movie at a checkpoint

❙ Enter the new movie at a particular time offset

Our prototype playback system connects theresponse data to the presentation engine via inter-faces written in Isis, and it can render the moviefull screen at full frame rate on a moderately pow-erful personal computer. The system assembleseach audio and video frame from its constituentmedia clips and objects in real time and outputsa single synchronized audio and video stream fordisplay. It also supports writing the movie to adisk file or a network connection. Viper doesn’tassume a particular type of distribution channel:

94

IEEE

Mul

tiM

edia

Viper Editing ModelThe extended editing model in Figure A assembles a short

“interview with Jane,” similar to parts of the campaign adver-tisement discussed in the main article. The interview’s contentcaters to the viewer’s primary concern (selected from a list ofchoices), and a video montage of Jane’s important daily activi-ties runs during the interview. A pacing variable (ranging from 1

to 5) meters the approximate duration of shots in the montage,and another variable (ranging from 1 to 10) governs abruptnessof transitions throughout the program. Introductory and clos-ing shots are L-cut into the beginning and end of the interview.Background music is included to suit the viewer’s taste.

Figure B1 gives a graphical example of the edit decision list(EDL) the model generates for a viewer who’s concerned about

# calculate length of dissolve transitions based on abruptness # factor(set dislen (* 0.1 (- 10 rv-abruptness)))

# find and prepare the intro clip, same for any version(set introClip (select-head (select-name “intro” clip-database)))(set introMovie (video-only (prepare-clip introClip)))(set introMovie (change-track 2 (fade-in-out dislen introMovie)))(set introDur (movie-duration introMovie))

# find and prepare the closer clip, same for any version(set closerClip (select-head (select-name “closer” clip-database)))(set closerMovie (video-only (prepare-clip closerClip)))(set closerMovie (change-track 2 (fade-in-out dislen closerMovie)))(set closerDur (movie-duration closerMovie))

# pick an interview clip that caters to viewer’s primary concern(set interviewClips (select-tags-all “interview” rv-concern

clip-database))(set interviewClip (select-head interviewClips))(set interviewMovie (prepare-clip interviewClip))(set interviewDur (movie-duration interviewMovie))

# select enough imagery clips to fill time in the middle part of # interview(set janeClips (select-tag “jane” clip-database))(set janeClips (sort-tag-decreasing “importance” janeClips))(score-length “length” rv-pacing janeClips)(set janeClips (select-first-tagsum “length” 14.0 interviewDur

janeClips))

(set janeClips (sort-tag-increasing “chronology” janeClips))

# sequence the imagery clips with dissolves between them(set janeEDLs (map (proc (clip) (prepare-clip-trim clip rv-pacing))

janeClips))(set janeEDLs (map (proc (movie) (ramp-in-out dislen (video-

only movie))) janeEDLs))(set janeMovie (sequence-movies (* -1.0 dislen) janeEDLs))(set janeMovie (change-track 2 (fade-in-out dislen janeMovie)))(set janeDur (movie-duration janeMovie))

# determine start times for each of the 4 parts(set interviewStart (- introDur 3.0))(set interviewMovie (shift-start interviewStart interviewMovie))(set janeStart (+ interviewStart (* 0.5 (- interviewDur janeDur))))(set janeMovie (shift-start janeStart janeMovie))(set closerStart (+ introDur interviewDur -6.0))(set closerMovie (shift-start closerStart closerMovie))

# combine the 4 parts into one big movie(set final-movie (combine-movies introMovie interviewMovie

janeMovie closerMovie))

# add music under it all(set runtime (movie-duration final-movie))(set musicClips (select-tags-all “music” rv-genre clip-database))(set musicClip (select-head musicClips))(set musicEDL (audio-only (prepare-clip-time musicClip runtime)))(set musicEDL (ramp-in-out dislen (set-level 0.3 musicEDL)))(set final-movie (combine-movies musicEDL final-movie))

Figure A. An editing model for a short responsive interview program. The prefix “rv” indicates response variables.

It targets scenarios in which the receiver performsthe final program assembly, but it will also sup-port assembling personalized programs at aremote server and streaming them to the receiv-er in a traditional linear fashion. Other multime-dia playback systems, such as that associated withSynchronized Multimedia Integration Language(SMIL),7 might also be amenable to deliveringViper content.

Production experienceTo date, we’ve completed three major Viper

productions and plan more. One program is amutable documentary that viewers control tosuit their personal comfort levels; another is a labinformational kiosk that lets viewers request fur-ther detail on research areas of interest; and thethird is a responsive political advertisement thatadapts to viewer profiles.

Mutable documentaryOur mutable documentary depicts MIT’s

Steer Roast Festival, which features (amongother things) live bands and mud wrestling.The presentation environment includes “sex”and “violence” knobs that viewers can set totheir personal comfort levels or vary duringplayback to explore different ways of portray-ing the event. A “length” knob controls the pre-sentation’s duration. The Viper editing modeldeveloped for the production includes clipsbased on their sex and violence ratings (asdetermined by the program’s designer), as wellas an importance rating that helps the systemomit less critical passages for shorter versions ofthe documentary.

Personalized research overviewThe interactive kiosk project presents a per-

sonalized research “open house” for visitors tothe MIT Media Laboratory. Viewers use a touch-screen to select areas of interest and an approxi-mate duration for the program. The mediadatabase contains imagery from various researchprojects and narrated descriptions that are usedpartly as voiceovers.

The editing model creates a survey of thelaboratory research, including projects that arelikely to appeal to viewers based on their selec-tions. It spends more time on projects thatclosely match the viewer’s choices and less onprojects that are only partially related. The edit-ing model embodies a story plan, constructing acoherent thread that includes any introducto-ry material about the laboratory or specificresearch groups that might be relevant to theprojects being presented.

Responsive campaign advertisementTo test Viper’s handling of a production with

a large number of response factors, and as a sideeffect of viewing the many coarsely targetedcampaign advertisements during the Fall 2000US presidential campaign, we attempted aresponsive political campaign advertisement.We wanted a presentation that could adapt tothe viewer’s personal profile and concerns to“sell” the candidate in the most appealing andeffective way for each individual. In this pro-duction, the mock candidate isn’t running forthe US presidency, but for a position on our lab-oratory’s student committee. The advertise-ment’s goal is to tailor the candidate’s message

95

July–Septem

ber 2003

cleanliness and likes classical music,with pacing set to 4 and abruptnessset to 9. Red blocks are video clipsand blue are audio. Green indicateswhen a track is enabled, with high-er tracks overruling lower tracks.

Figure B2 is an EDL generatedfor a viewer who’s concerned aboutlab space and likes Latin music, withboth pacing and abruptness set to2. The difference between shotduration and transition length (con-veyed by fades at the beginning orend of a shot element) is subtle butnoticeable.

Figure B. Graphical representations of an edit decision list for: (1) a viewer

concerned about cleanliness and fond of classical music, and (2) a viewer

concerned about lab space and fond of Latin music.

(1)

(2)

based on the viewer profile, shown in Table 1.If the playback system knows the viewer’s

identity, it can gather values for many of thesevariables by consulting online databases for per-sonal information and perhaps inferring con-cerns by checking for membership on certainmailing lists. It could estimate the attention spanvariable (an indication of how much of a rushthe viewer might be in) from the time of day andthe viewer’s schedule. For demonstration pur-poses, the viewer simply fills out a brief onlineform before watching the program.

The advertisement’s editing model empha-sizes a seamless flow between five conceptualparts of the program.

❙ Introduction. Background music begins, itsgenre governed by the viewer’s music prefer-ence. Fade in to an uplifting image of the can-didate, which is the same for any version ofthe program. The candidate, Aisling, intro-duces herself. The introduction is shorter ifthe viewer is in a rush.

❙ Build relationship with viewer. In a video insert, aseries of clips shows the candidate touring thelab, concentrating on areas on the viewer’shome floor. Presented in slow motion, the pac-ing is increased (that is, clip duration isdecreased) for shorter attention spans, andonly as many clips as needed to cover theduration of the candidate’s remarks are includ-ed. During this insert, the candidate makes astatement about the general condition of thelab, tailored to appeal to the viewer’s mindset.

❙ Address specific issues. We see the candidate inthe interview again, stating that many prob-lems at the lab need to be addressed. A secondinsert begins with a series of descriptive clipsillustrating these problems. The clips for thissegment appeal to the viewer’s concerns aboutthe lab, and pacing is again varied to cater to

the viewer’s attention span. If the viewer is inless of a rush (has a medium or long attentionspan), we hear a voiceover of the candidatemaking an extended statement about an itemon the viewer’s list of concerns.

❙ Build confidence in abilities. Back in the inter-view, the candidate states why she thinksshe’s the best person for the job. In a thirdvideo insert, we see a series of slow-motionclips of the candidate talking to and shakinghands with people who share the viewer’shome floor and job position. Again, the atten-tion span variable meters pacing.

❙ Closing. The candidate makes a brief finalstatement that caters to the viewer’s mindset.Dissolve slowly into a final image of the can-didate emerging from the laboratory lookingconfident (we use the same image for everyversion). Fade to black.

Shooting the advertisement was fairly straight-forward. We spent a single afternoon going fromfloor to floor to capture images of the candidatetouring the building, shaking hands, and point-ing out problem areas in entertaining (and oftentongue-in-cheek) vignettes. On a second day, wefilmed the mock interview in which the candidatemade all of the statements used in the program.

The clip annotations let the editing modelselect the most appropriate clips to include in eachsection of the advertisement. We classified eachimagery clip with keywords identifying the flooron which it was shot, the type of people in theshot (students, professors, and so on), the con-cerns illustrated in the clip (such as equipmentand food), and the clip’s function as it relates tothe program’s conceptual parts (walking shot,handshake shot, lab concern shot, and so on).Each clip includes fuzzy in and out points that letthe editing model adjust its duration within cer-tain bounds to alter the program’s pacing.

We annotated the interview clips based onwhere they should appear in the advertisementand the mindset and attention span they cater to.The database also includes various backgroundmusic selections, classified by genre. We tried toshoot enough material to satisfy any possible ver-sion of the program, but where holes exist, theediting model falls back on clips that satisfy fewerof the constraints for each program segment.

Without video, it’s difficult to convey a senseof the range of advertisements the system can

96

IEEE

Mul

tiM

edia

Table 1. Elements of the viewer profile used to target the responsive

advertisement.

Job position Student, faculty, administration, sponsor

Home floor First, second, third, fourth

General mindset Negative, positive

Concerns Equipment, cleanliness, safety, space, food, facilities,

classes, stress, distractions

Favorite music genre Jazz, classical, Latin, funk, techno

Attention span Short, medium, long

generate, but Figure 3 gives graphical renderingsof three versions, and the example in the “ViperEditing Model” sidebar illustrates several of theediting concepts used in the campaign advertise-ment. Video files of several variations are avail-able at http://www.media.mit.edu/~stefan/viper/.A detailed discussion of the production’s editingmodel is available elsewhere.8

Results and possible extensionsThe campaign advertisement was successful in

laboratory demonstrations. With six viewer vari-ables, the editing model could generate hundredsof versions of the program, including severalcoherent versions for the same profile, suggest-ing that repeat viewers would rarely see exactlythe same thing.

From a production viewpoint, the advertise-ment suggests that creating compelling respon-sive video content might require less work thanoriginally thought. We used fewer than 30unique keywords to annotate the clips in thedatabase to satisfy the editing model’s require-ments. The model also used Viper’s multitrackframework and facilities for visualizing and tight-ly controlling the more complex edits typicallyfound in fast-paced advertisements, withoutwhich the program wouldn’t have been as effec-tive. Moreover, the editing model supported theprogram’s ability to self-summarize—that is, toshorten itself appropriately in response to a view-er with a short attention span.

Viewers unaware of the infrastructure largelybelieved they were watching a professionallyedited program. Professional editors, however,weren’t fooled as frequently and recognized sev-eral imperfections. For example, the editingmodel placed the clips for the three montage seg-ments in random order, as we believed it ade-quate in the initial version to get the messageacross to the viewer. This approach resulted insome awkward juxtapositions—for example,when the camera moved in one direction in oneclip and the opposite direction in the next, whentwo extreme close-ups of the candidate were cuttogether, or when two otherwise unrelated shotsthat originated at the same physical locationappeared in sequence.

We could avoid these difficulties by addingadditional annotations on the relevant clips and afew more “smarts” in the editing model, in amanner similar to that of Bloch9 and Parkes.10 Wecould add information about framing and cam-era motion at the clip’s entry and exit points, for

example, and the editing model could use theseannotations to avoid orderings that result inuncomfortable transitions. We could use similarstrategies to avoid other cinematic taboos likejump cuts, 180-degree cuts, and other breaks incontinuity. If we added annotations about con-tent and action to the clips, Viper could automatethe common practice of cutting on gestures whenassembling a sequence of shots that represents acontinuous stretch of time. It’s also possible (forexample, in the campaign advertisement) to entera timed transcript of the candidate’s interviewsinto the editing model and use it to choose andorder clips to correspond to the moment-to-moment content of candidate statements.

In the end, the complexity of a video programgenerated by Viper is only as rich as the editingmodel and system of annotations the designercreates for it. If one or both of these componentsis lacking, a program might not flow as smooth-ly as desired.

Future directionsPerhaps Viper’s main achievement is the for-

mation of an open framework of primitives fordeveloping the annotation schemes and editingmodels that govern the behavior of responsivevideo programs. As producers continue to useViper, they can share and build on pieces fromprevious productions to reduce the amount ofwork required for new projects. Over time, pro-ducers might build large libraries of commonlyused components that they might even packageand market to other users.

Our experience suggests Viper is a good matchto genres and situations in which content modi-fication for different audiences is well understood,such as targeted advertising or age-appropriateediting. Viper also seems to offer new opportuni-ties to documentary filmmakers and creators ofnews or educational programming. Its usefulnessin authoring other types of material remains to beexplored.

We could extend Viper to handle live videomaterial. Assuming a production team couldannotate each live feed in real time with infor-mation about its content, Viper could choosewhich stream to show at different times, openingthe door to applications such as live sportingevents that let viewers select preferences abouthow they want the event presented: team orplayer emphasis, specific commentators, shotpacing, and so on.

Opinions differ on the usefulness of a more

97

July–Septem

ber 2003

graphical interface for developing Viper editingmodels. The availability of the general-purposeIsis syntax for use in making editing models isimportant in situations that require highly cus-tomized calculations or mappings, but a moregraphical interface might better reflect themanipulations being performed with the mediaobjects, making them accessible to a wider rangeof users. These issues suggest many clear avenuesfor further research. MM

AcknowledgmentsThanks to Aisling Kelliher for starring in the

campaign advertisement, Matthew Palmer for hishelp developing the tool and working on our firstproductions, and Glorianna Davenport for herinvaluable thoughts and suggestions throughoutthis research. This work has been supported bythe Digital Life Consortium and theBroadercasting Special Interest Group at the MITMedia Laboratory.

References1. S. Agamanolis and V.M. Bove Jr., “Multilevel Script-

ing for Responsive Multimedia,” IEEE Multimedia,

Oct.–Dec. 1997, pp. 40-50.

2. T.G. Aguierre Smith and G. Davenport, “The Strati-

fication System: A Design Environment for Random

Access Video,” Proc. 3rd Int’l Workshop Networking

and Operating System Support for Digital Audio and

Video, San Diego, 1992, pp. 250-261.

3. M. Davis, “Media Streams: An Iconic Visual

Language for Video Representation,” Readings in

Human–Computer Interaction: Toward the Year

2000, Morgan Kaufmann, 1995, pp. 854-866.

4. F. Nack and A.T. Lindsay, “Everything You Wanted

to Know about MPEG-7: Part 1,” IEEE Multimedia,

vol. 6, no. 3, July–Sept. 1999, pp. 65-77.

5. F. Nack and A.T. Lindsay, “Everything You Wanted

to Know about MPEG-7: Part 2,” IEEE Multimedia,

vol. 6, no. 4, Oct.–Dec. 1999, pp. 64-73.

6. M. Mateas, S. Domike, and P. Vanouse, “Terminal

Time: An Ideologically Biased History Machine,”

Proc. AISB Symp. Artificial Intelligence and Creative

Language: Stories and Humor, Soc. for the Study of

Artificial Intelligence and the Simulation of Behav-

ior, 1999, pp. 69-75.

7. W3C Synchronized Multimedia Activity Statement,

http://www.w3.org/AudioVideo/Activity.html.

8. S. Agamanolis, Isis, Cabbage, and Viper: New Tools

and Strategies for Designing Responsive Media, PhD

dissertation, Media Arts and Sciences, Mass. Inst. of

Technology, 2001.

9. G.R. Bloch, “From Concepts to Film Sequences,”

Proc. User-Oriented Content-Based Text and Image

Handling (RIAO), ACM Press, 1988, pp. 760-767.

10. A. Parkes, “Computer-Controlled Video for Intelli-

gent Interactive Use: A Description Methodology,”

Multimedia Interface Design in Education, A.D.H.

Edwards and S. Holland, eds., Springer-Verlag,

1992, pp. 97-116.

Stefan Agamanolis is a principal

research scientist and the director

of the Human Connectedness

research group at Media Lab

Europe, the European research

partner of the MIT Media Labora-

tory. His research interests include object-based repre-

sentations for media, interactive storytelling, responsive

environments, and automated video editing. He received

MS and PhD degrees from the MIT Media Laboratory.

V. Michael Bove Jr. is head of the

MIT Media Laboratory’s Object-

Based Media group. His research

interests include digital television

systems, video processing hard-

ware and software design, multi-

media, scene modeling, and optics. He has an SBEE, an

SM in visual studies, and a PhD in media technology, all

from the Massachusetts Institute of Technology. Bove is

on the board of editors of the Journal of the Society of

Motion Picture and Television Engineers, and is a fellow of

the Society of Photo-Optical Instrumentation Engineers.

Readers may contact the authors at {stefan,

vmb}@media.mit.edu.

For further information on this or any other computing

topic, please visit our Digital Library at http://computer.

org/publications/dlib.

98

IEEE

Mul

tiM

edia