SceneMaker:Multimodal Visualisation

of Natural Language Film Scripts

Dr. Minhua Eunice Ma

School of Computing & Intelligent SystemsFaculty of Computing & EngineeringUniversity of Ulster, Magee, Northern Ireland

hanser-e@email.ulster.ac.uk, {p.mckevitt, tf.lunney, j.condell}@ulster.ac.uk

Eva Hanser Prof. Paul Mc Kevitt Dr. Tom Lunney Dr. Joan Condell

School of Computing and MathematicsFaculty of Business, Computing and LawUniversity of Derby, England

m.ma@derby.ac.uk

PRESENTATION OUTLINE

Aims & Objectives

Related Projects

SceneMaker Design and Implementation

Relation to Other Work

Conclusion and Future Work

AIMS

• Automatically generate well-designed and affective virtual scenes from screenplays

• Realistic visualisation of emotional aspects

• Multimodal representation with 3D animation, speech, audio and cinematography

• Enhance believability of virtual actors and scene presentation

: AIMS & OBJECTIVES

Input:Screen-play SceneMaker

SystemOutput: Animation

OBJECTIVES

• Processing/inferencing emotions and semantic information within story context

• Common sense, affective and cinematic knowledge ontologies reflecting human cognitive reasoning rules

• Automatic genre recognition from text

• Design, implementation and evaluation of SceneMaker

: AIMS & OBJECTIVES

• Standardized format and language of screenplays

• Automatic annotation of formal screenplay elements (Jhala 2008)

• Semantic information on location, timing, props, actors, events, manners, dialogue and camera direction

SEMANTIC TEXT PROCESSING: RELATED PROJECTS

INT. M.I.T. HALLWAY -- NIGHT Lambeau and Tom come around a corner. His P.O.V. reveals a figure in silhouette blazing through the proof on the chalkboard. There is a mop and a bucket beside him. As Lambeau draws closer, reveal that the figure is Will, in his janitor's uniform. There is a look of intense concentration in his eyes. 

LAMBEAUExcuse me!

 WILL

Oh, I'm sorry. 

LAMBEAUWhat're you doing?

 WILL

(walking away)I'm sorry.

Screenplay Extract from ‘Good Will Hunting (1997)’

• Emotion recognition from text:keyword spotting, lexical affinity, statistical models, fuzzy logic rules, machine learning, commonsense knowledge, cognitive models

• XML-based annotations defining visual appearance of animated characters and scenes:

BEAT – Behaviour Expression Animation Toolkit (Cassell et al. 2001) MSML – Movie Script Markup Language(Van Rijsselbergen et al. 2009)

VISUAL AND EMOTIONAL SCRIPTING

<GAZE word=1 time=0.0 spec=AWAY_FROM_HEARER><GAZE word=3 time=0.517 spec=TOWARDS_HEARER><R_GESTURE_START word=3 time=0.517 spec=BEAT><EYEBROWS_START word=3 time=0.517>

: RELATED PROJECTS

• Automatic physical transformation and synchronisation of 3D models reflecting emotion

• Manner influences intensity, scale, force, fluency and timing of an action

• Multimodal annotated affective video or motion captured data (Gunes and Piccardi 2006)

MODELLING AFFECTIVE BEHAVIOUR

Personality & Emotion Engine(Su et al. 2007)

Greta (Pelachaud 2005)

: RELATED PROJECTS

• WordsEye – Scene composition(Coyne and Sproat 2001)

• ScriptViz – Screenplay visualisation(Liu and Leung 2006)

• CONFUCIUS – Action, speech & scene animation(Ma 2006)

• CAMEO – Cinematic and genre visualisation(Shim and Kang 2008)

VISUALISING 3D SCENES

WordsEye CONFUCIUSScriptViz CAMEO

: RELATED PROJECTS

• Emotional speech synthesis (Schröder 2001)

- Prosody rules

• Music recommendation systems

- Categorisation of rhythm, chords, tempo, melody, loudness and tonality

- Sad or happy music and genre membership (Cano et al. 2005)

- Associations between emotions and music (Kuo et al. 2005)

AUDIO GENERATION: RELATED PROJECTS

• Context consideration through natural language processing, common sense knowledge and reasoning methods

• Extract genre and moods from screenplays

• Influence on all elements of visualisation

• Enhance naturalism and believability

• Text-to-animation software prototype, SceneMaker

KEYOBJECTIVES: DESIGN AND IMPLEMENTATION

Animation Player

Animation Player

Script EditorScript Editor

Screen-play

Text & LanguageProcessing

Text & LanguageProcessing

ContextInterpretation

ContextInterpretation

MultimediaGenerationMultimediaGeneration}

Genre

Emotion

Action

}

ARCHITECTURE OF SCENEMAKER: DESIGN AND IMPLEMENTATION

SOFTWARE AND TOOLS: DESIGN AND IMPLEMENTATION

LVSR(2)

Lexical Visual Semantic

Representation

Script Format

Ontology

Unity(6)

3D Engine(JavaScript,XML)

MSML(5)

/SMIL

ConceptNet(3)

Common Sense Knowledge

Gate(1)

ANNIEOnto-Gazetteer

GenreOntologyRDFS/OWL

MovieOntologyRDFS/OWL

WordNet-Affect(4)

Festival(7)

Speech Synthesiser

Natural Language Processing & Script Segmentation

Context + Emotion Reasoning

Event Synchronisation

3D Rendering + Multimedia

3D Models(3D Studio Max)

MovieScript

AutomaticSound & Music

Selection

(1) http://gate.ac.uk (2) Ma 2006 (3) Liu and Singh 2004 (4) Strapparava and Valitutti 2004 (5) Van Rijsselbergen et al. 2009 (6) http://unity3d.com (7) http://www.cstr.ed.ac.uk/projects/festival

Evaluating 4 aspects of SceneMaker:

EVALUATION OF SCENEMAKER

Aspect EvaluationCorrectness of screenplay analysis & visual interpretation

Hand-animating scenes

Effectiveness of output scenes

Existing feature film scenes

Suitability for genre type Scenes of unknown scripts categorised by readers

Functionality of interface Testing with drama students and directors

: DESIGN AND IMPLEMENTATION

Text to Animation System Year Text Input: Genre Context Emotion AnimationMovie Script Reasoning (3D )

CONFUCIUS (Ma 2006) 2006 – – – – ScriptViz (Liu and Leung 2006) 2007 – – – – CAMEO (Shim and Kang 2008) 2007 – P&E Engine (Su et al. 2007) 2007 – – P&E rules Behaviour Generation System (Breitfuss et al. 2007) 2007 – (Dialogue) – – MSML (Van Rijsselbergen et al. 2009) 2009 – – – externalSceneMaker

RELATION TO OTHER WORK

CONCLUSION AND FUTURE WORK

• Automatic expressive multi-media animation of screenplays

• Focus on:– automatic reasoning about story context and emotional interpretation – based on world knowledge and context memory – emotions influencing scene compositions and event execution– scene direction refined by genre-specifics

• Analysis of script format to access semantic information

• Automatic genre specification from script

• Heightened expressiveness, naturalness and artistic quality

• Assist directors, actors, drama students, script writers

• Future work: Implementation & Testing of SceneMaker

Thank you.

QUESTIONS OR COMMENTS?