Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
MultimediaInteraction in utcc
Olarte & Rueda.
A Declarative Language for DynamicMultimedia Interaction Systems
Carlos Olarte Camilo Rueda
LIX, Ecole Polytechnique Paris.Pontificia Universidad Javeriana Cali.
IRCAM.
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Motivation
I Multimedia interactive systems aims at devising waysfor the machine to be an active partner in a collectivebehavior constructed dynamically by many actors.
I The machine must always actively adapt its behavioraccording to the information derived from the activityof the other partners. For example:
1. Interactive Scores : defines a hierarchical structurewhere loosely coupled music processes may change theirproperties in reaction to stimulus from the environment.
2. Musical improvisation : A style-learning /improvisationsystem is a reactive system where several learning andimprovising agents react to information provided by theenvironment or by other agents.
I Challenges:
1. Complex coordination patterns of multiple agents.2. Ensuring the correctness of the system.
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Motivation
I Multimedia interactive systems aims at devising waysfor the machine to be an active partner in a collectivebehavior constructed dynamically by many actors.
I The machine must always actively adapt its behavioraccording to the information derived from the activityof the other partners. For example:
1. Interactive Scores : defines a hierarchical structurewhere loosely coupled music processes may change theirproperties in reaction to stimulus from the environment.
2. Musical improvisation : A style-learning /improvisationsystem is a reactive system where several learning andimprovising agents react to information provided by theenvironment or by other agents.
I Challenges:
1. Complex coordination patterns of multiple agents.2. Ensuring the correctness of the system.
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Motivation
I Multimedia interactive systems aims at devising waysfor the machine to be an active partner in a collectivebehavior constructed dynamically by many actors.
I The machine must always actively adapt its behavioraccording to the information derived from the activityof the other partners. For example:
1. Interactive Scores : defines a hierarchical structurewhere loosely coupled music processes may change theirproperties in reaction to stimulus from the environment.
2. Musical improvisation : A style-learning /improvisationsystem is a reactive system where several learning andimprovising agents react to information provided by theenvironment or by other agents.
I Challenges:
1. Complex coordination patterns of multiple agents.2. Ensuring the correctness of the system.
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Motivation
I Multimedia interactive systems aims at devising waysfor the machine to be an active partner in a collectivebehavior constructed dynamically by many actors.
I The machine must always actively adapt its behavioraccording to the information derived from the activityof the other partners. For example:
1. Interactive Scores : defines a hierarchical structurewhere loosely coupled music processes may change theirproperties in reaction to stimulus from the environment.
2. Musical improvisation : A style-learning /improvisationsystem is a reactive system where several learning andimprovising agents react to information provided by theenvironment or by other agents.
I Challenges:
1. Complex coordination patterns of multiple agents.2. Ensuring the correctness of the system.
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Motivation
Concurrent Constraint Programming (CCP):
I A declarative model for concurrency tied to logic .
I It benefits from the large body of reasoning techniquesfrom both process calculi and logic.
I Universal timed CCP (utcc) extends CCP for mobilereactive systems :
1. Connection with logic is preserved (declarative ).2. The calculus is deterministic (elegant denotational
semantics, e.g. closure operators).3. A powerful synchronization mechanism based on
entailment of constraints (information available).
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Motivation
Concurrent Constraint Programming (CCP):
I A declarative model for concurrency tied to logic .
I It benefits from the large body of reasoning techniquesfrom both process calculi and logic.
I Universal timed CCP (utcc) extends CCP for mobilereactive systems :
1. Connection with logic is preserved (declarative ).2. The calculus is deterministic (elegant denotational
semantics, e.g. closure operators).3. A powerful synchronization mechanism based on
entailment of constraints (information available).
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Motivation
In this paper we argue for utcc as a declarative model fordynamic multimedia interaction systems:
1. Partial Information (constraints) allows for thespecification of weakly defined temporal relationsbetween agents/events.
2. Simple coordination and synchronization of agentsbased on constraints (blocking asks).
3. The expressivity of utcc allows for more flexible anddynamic systems.
4. Relying on the underlying temporal logic in utcc,non-trivial properties of the model can be verified .
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Motivation
In this paper we argue for utcc as a declarative model fordynamic multimedia interaction systems:
1. Partial Information (constraints) allows for thespecification of weakly defined temporal relationsbetween agents/events.
2. Simple coordination and synchronization of agentsbased on constraints (blocking asks).
3. The expressivity of utcc allows for more flexible anddynamic systems.
4. Relying on the underlying temporal logic in utcc,non-trivial properties of the model can be verified .
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Motivation
In this paper we argue for utcc as a declarative model fordynamic multimedia interaction systems:
1. Partial Information (constraints) allows for thespecification of weakly defined temporal relationsbetween agents/events.
2. Simple coordination and synchronization of agentsbased on constraints (blocking asks).
3. The expressivity of utcc allows for more flexible anddynamic systems.
4. Relying on the underlying temporal logic in utcc,non-trivial properties of the model can be verified .
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Motivation
In this paper we argue for utcc as a declarative model fordynamic multimedia interaction systems:
1. Partial Information (constraints) allows for thespecification of weakly defined temporal relationsbetween agents/events.
2. Simple coordination and synchronization of agentsbased on constraints (blocking asks).
3. The expressivity of utcc allows for more flexible anddynamic systems.
4. Relying on the underlying temporal logic in utcc,non-trivial properties of the model can be verified .
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Contributions (Applications)
We present two applications:
1. A model for dynamic interactive scores whereinteractive points can be defined to adapt thehierarchical structure depending on the informationinferred from the environment.
2. A model of a music improvisation system based on theFactor Oracle that scales up to situations involvingseveral players, learners and improvisers.
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Contributions (Applications)
We present two applications:
1. A model for dynamic interactive scores whereinteractive points can be defined to adapt thehierarchical structure depending on the informationinferred from the environment.
2. A model of a music improvisation system based on theFactor Oracle that scales up to situations involvingseveral players, learners and improvisers.
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Outline
The utcc calculus
Dynamic Interactive Scores
Music Improvisation Systems
Concluding Remarks
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Outline
The utcc calculus
Dynamic Interactive Scores
Music Improvisation Systems
Concluding Remarks
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Concurrent Constraint Programming
I Von Neumann’s store is replaced by a store of partialinformation represented as Constraints (e.g. x < 10).
I The type of constraints is given by a Constraint System(e.g. first-order formulae over a given signature).
I Agents monotonically add (tell ) constraints. Theysynchronize via blocking asks querying the store.
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Concurrent Constraint Programming
I Von Neumann’s store is replaced by a store of partialinformation represented as Constraints (e.g. x < 10).
I The type of constraints is given by a Constraint System(e.g. first-order formulae over a given signature).
I Agents monotonically add (tell ) constraints. Theysynchronize via blocking asks querying the store.
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Concurrent Constraint Programming
I Von Neumann’s store is replaced by a store of partialinformation represented as Constraints (e.g. x < 10).
I The type of constraints is given by a Constraint System(e.g. first-order formulae over a given signature).
I Agents monotonically add (tell ) constraints. Theysynchronize via blocking asks querying the store.
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Concurrent Constraint Programming
I Von Neumann’s store is replaced by a store of partialinformation represented as Constraints (e.g. x < 10).
I The type of constraints is given by a Constraint System(e.g. first-order formulae over a given signature).
I Agents monotonically add (tell ) constraints. Theysynchronize via blocking asks querying the store.
tell temperature > 42 ask temperature = 50 then P
ask 0<temperature<100 then Q
temperature=?
tell temperature < 70
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Concurrent Constraint Programming
I Von Neumann’s store is replaced by a store of partialinformation represented as Constraints (e.g. x < 10).
I The type of constraints is given by a Constraint System(e.g. first-order formulae over a given signature).
I Agents monotonically add (tell ) constraints. Theysynchronize via blocking asks querying the store.
ask temperature = 50 then P
ask 0<temperature<100 then Q
42 <temperature
tell temperature < 70
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Concurrent Constraint Programming
I Von Neumann’s store is replaced by a store of partialinformation represented as Constraints (e.g. x < 10).
I The type of constraints is given by a Constraint System(e.g. first-order formulae over a given signature).
I Agents monotonically add (tell ) constraints. Theysynchronize via blocking asks querying the store.
ask temperature = 50 then P
ask 0<temperature<100 then Q
42 <temperature<70
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Concurrent Constraint Programming
I Von Neumann’s store is replaced by a store of partialinformation represented as Constraints (e.g. x < 10).
I The type of constraints is given by a Constraint System(e.g. first-order formulae over a given signature).
I Agents monotonically add (tell ) constraints. Theysynchronize via blocking asks querying the store.
ask temperature = 50 then P
Q
42 <temperature<70
Remains Blocked
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Constraint Systems.
I A constraint system is a tuple 〈∑,∆〉 where
∑is a
signature and ∆ a consistent first-order theory over∑
I Constraints are first-order formulae over∑
I Entailment relation c ` d holds iff c ⇒ d is valid on∆. c ≡ d iff c ` d and d ` c
I C denotes the set of constraints modulo ≡ in 〈∑,∆〉
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
The tcc Model
1. Receives a stimulus (i.e a constraint) from theenvironment.
2. Computes a CCP process in the current time-unit andwait for stability.
3. Responds with the resulting store.
4. Executes the Residual process in the next time-unit.
* Note: Stores are not automatically transferred from atime unit to the next one.
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
The tcc Model
1. Receives a stimulus (i.e a constraint) from theenvironment.
2. Computes a CCP process in the current time-unit andwait for stability.
3. Responds with the resulting store.
4. Executes the Residual process in the next time-unit.
* Note: Stores are not automatically transferred from atime unit to the next one.
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
The tcc Model
1. Receives a stimulus (i.e a constraint) from theenvironment.
2. Computes a CCP process in the current time-unit andwait for stability.
3. Responds with the resulting store.
4. Executes the Residual process in the next time-unit.
* Note: Stores are not automatically transferred from atime unit to the next one.
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
The tcc Model
1. Receives a stimulus (i.e a constraint) from theenvironment.
2. Computes a CCP process in the current time-unit andwait for stability.
3. Responds with the resulting store.
4. Executes the Residual process in the next time-unit.
* Note: Stores are not automatically transferred from atime unit to the next one.
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
The utcc calculus
Syntax
P,Q := skip | tell(c) | (abs ~x ; c) P | P ‖ Q | (local~x ; c) P |next P | unless c next P | ! P | p(~x)
I tell(c) : adds c to the store in the current time interval.
I (abs ~x ; c) P : executes P[~t/~x ] for each ~t s.t. c[~t/~x ] canbe deduced from the current store.
I (local~x ; c) P : behaves like P but the informationabout variables in ~x is local to P
I next P : executes P in the next time unit.
I unless c next P : executes P in the next time unit if ccannot be entailed now.
I p(~x) : procedure call (derived operator).
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Operational Semantics
RT 〈tell(c), d〉 −→ 〈skip, d ∧ c〉RP
〈P, c〉 −→ 〈P′, d〉〈P ‖ Q, c〉 −→ 〈P′ ‖ Q, d〉
RL
〈P, c ∧ (∃~xd)〉 −→ 〈P′, c ′ ∧ (∃~xd)〉〈(local~x ; c) P, d〉 −→ 〈(local~x ; c ′) P′, d ∧ ∃~xc′〉
RUd ` c
〈unless c next P, d〉 −→ 〈skip, d〉RR 〈! P, d〉 −→ 〈P ‖ next ! P, d〉
RA
d ` c[~t/~x] |~t |= |~x |〈(abs ~x ; c) P, d〉 −→
˙P[~t/~x] ‖ (abs ~x ; c ∧ ~x 6= ~t ) P, d
¸RS
γ1 −→ γ2
γ′1 −→ γ′2
if γ1 ≡ γ′1 and γ2 ≡ γ′2
RO
〈P, c〉 −→∗ 〈Q, d〉 6−→
P(c,d)
====⇒ F (Q)
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
An Example
Let P be a process modeling a musician and Q animprovisation system :
Pdef= tell(play (A)) ‖ next (tell(play (G)) ‖ next tell(play (B))) . . .
Qdef= ! (abs x ; play(x)) R
When executing P ‖ Q, we observe, e.g., R[A/x ],R[G/x ],etc meaning that P and Q synchronized on the constraintplay(·): The note played by P (i.e. G ) is read by Q andthen processed by R.
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Outline
The utcc calculus
Dynamic Interactive Scores
Music Improvisation Systems
Concluding Remarks
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Interactive Scores
A B
D
R
C
Allen relations:
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Interactive Scores
A B
D
R
C Score relations:A precedes B A meets DD overlaps B R contains CR contains D C contains A
I An Interactive Score is a pair composed of temporalobjects and structural and temporal (Allen) relations .
I Each object is comprised of a start-time, a duration,and a procedure (operational meaning).
I The idea: Dynamic changes in the hierarchy. E.g., if aninteraction does not occur, the composer may move theinterval to a similar musical context.
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Dynamic Interactive Scores
A B
D
R
C
Dynamic Reconfiguration:
I Moving boxes.
I Adding/deleting intervals.
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
The Model
BoxOpdef= (abs id , d ; mkbox(id , d)) (local s) tell(box(id , d , s))
‖ (abs id ; destroy(id))(abs x , sup; in(x , id) ∧ in(id , sup))
unless play(id) next tell(in(x , sup))‖ (abs x , y ; before(x , y)) when ∃z (in(x , z) ∧ in(y , z)) do
unless play (y) next tell(bf(x , y))‖ (abs x , y ; into(x , y)) unless play (x) next tell(in(x , y))‖ (abs x , y ; out(x , y)) when in(x , y) do
unless play (x) next (abs z, in(y , z); tell(in(x , z)))
Constdef= (abs x , y ; in(x , y)) (abs dx , sx ; box(x , dx , sx ))
(abs dy , sy ; box(y , dy , sy ))tell(sy < sx ) ‖ tell(dx + sx ≤ dy + sy )
‖ (abs x , y ; bf(x , y)) (abs dx , sx ; box(x , dx , sx ))(abs dy , sy ; box(y , dy , sy )) tell(sx + dx ≤ sy )
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
The Model
BoxOpdef= (abs id , d ; mkbox(id , d)) (local s) tell(box(id , d , s))
‖ (abs id ; destroy(id))(abs x , sup; in(x , id) ∧ in(id , sup))
unless play(id) next tell(in(x , sup))‖ (abs x , y ; before(x , y)) when ∃z (in(x , z) ∧ in(y , z)) do
unless play (y) next tell(bf(x , y))‖ (abs x , y ; into(x , y)) unless play (x) next tell(in(x , y))‖ (abs x , y ; out(x , y)) when in(x , y) do
unless play (x) next (abs z, in(y , z); tell(in(x , z)))
Constdef= (abs x , y ; in(x , y)) (abs dx , sx ; box(x , dx , sx ))
(abs dy , sy ; box(y , dy , sy ))tell(sy < sx ) ‖ tell(dx + sx ≤ dy + sy )
‖ (abs x , y ; bf(x , y)) (abs dx , sx ; box(x , dx , sx ))(abs dy , sy ; box(y , dy , sy )) tell(sx + dx ≤ sy )
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
The Model(2)
Clock(t, v)def= tell(t = v) ‖ nextClock(t, v + 1)
Play(x , t)def= when t ≥ 1 do tell(play(x)) ‖
unless t ≤ 1 nextPlay(x , t − 1)
Init(t)def= (wait x ; init(x)) do
(abs dx , sx ; box(x , dx , sx ))Clock(t, 0) ‖ tell(sx = t) ‖! (wait y , dy , sy ; box(y , dy , sy ) ∧ sy ≤ t) do Play(y , dy )
Systemdef= (local t) Init(t) ‖! Constraints ‖! BoxOp ‖ UsrBoxes
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
An Example
UsrBoxesdef=
tell(mkbox(a, 22)) ‖tell(mkbox(b, 12)) ‖tell(mkbox(c, 4)) ‖tell(mkbox(d , 5))tell(mkbox(e, 2)) ‖tell(into(b, a)) ‖tell(into(c, b)) ‖tell(into(d , b)) ‖tell(into(e, d)) ‖tell(before(c, d)) ‖whenever play(b) dounless signal next
tell(out(d , b)) ‖tell(mkbox(f , 2)) ‖tell(into(f , a)) ‖tell(before(b, f ) ‖tell(before(f , d)))
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Declarative Interpretation of utcc
I Processes defined by the user may lead to inconsistentstores: E.g. placing a box that exceeds the boundariesof the container.
I The idea: Using the declarative view of utcc processesas FLTL formulae to verify the model.
Definition (utcc logic characterization)
Let [[·]] be the following map
[[skip]] = true [[tell(c)]] = c[[P ‖ Q]] = [[P]] ∧ [[Q]] [[(abs ~y ; c) P]] = ∀~y(c ⇒ [[P]])[[(local~x ; c) P]] = ∃~x(c ∧ [[P]]) [[nextP]] = ◦[[P]][[unless c nextP]] = c ∨ ◦[[P]] [[! P]] = 2[[P]]
Theorem (Logic correspondence)
Let s = c1.c2.c3... ∈ ouput(P). [[P]] ` 3d iff there existsi ≥ 1 s.t. ci ` d
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Declarative Interpretation of utcc
I Processes defined by the user may lead to inconsistentstores: E.g. placing a box that exceeds the boundariesof the container.
I The idea: Using the declarative view of utcc processesas FLTL formulae to verify the model.
Definition (utcc logic characterization)
Let [[·]] be the following map
[[skip]] = true [[tell(c)]] = c[[P ‖ Q]] = [[P]] ∧ [[Q]] [[(abs ~y ; c) P]] = ∀~y(c ⇒ [[P]])[[(local~x ; c) P]] = ∃~x(c ∧ [[P]]) [[nextP]] = ◦[[P]][[unless c nextP]] = c ∨ ◦[[P]] [[! P]] = 2[[P]]
Theorem (Logic correspondence)
Let s = c1.c2.c3... ∈ ouput(P). [[P]] ` 3d iff there existsi ≥ 1 s.t. ci ` d
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Verification of the Model
We can verify, for example,
I [[P]] ` 3∃x ,dx ,sx ,y ,dy ,sy (box(x , dx , sx) ∧ box(y , dy , sy ) ∧in(x , y) ∧ sx + dx > sy + dy ): The end time of the boxy is less than the end time of the inner box x . I.e., thebox y cannot contain x .
I [[P]] ` ∀x(∃dx ,sx (box(x , dx , sx)⇒ 3 play(x)): All themusical structures are eventually played .
I [[P]] ` 3∀x(∃dx ,sx (box(x , dx , sx)⇒ play(x)): At somepoint all the boxes are playing simultaneously.
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Outline
The utcc calculus
Dynamic Interactive Scores
Music Improvisation Systems
Concluding Remarks
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Learning a Style
I Learning a Style implies finding out what musicmaterial appears in what contexts .
I Capture music material during performanceMusic material: some sequence of music events.
I Identify contextsContext: repeated suffix of a sequence.
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Factor Oracle
I A structure for efficient representation of all substrings(factors) of a string.
I Used to represent performed music sequences and alltheir contexts .
A FO is represented as a graph:
I nodes are referred to as states Si .I Arcs are partitioned in
I forward links (Si ,Sk), k > i , labelled with elements ofthe string.
I backward links, (Si ,Sj), j < i , that bear no label.
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Factor Oracle: example
Figure: A FO automaton for s = aba
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Factor Oracle: properties
Let s = σ1, σ2, ..., σn
I States are S0, S1, ...,Sn
I All links (Si−1,Si ), 1 ≤ i < n, labelled with σi , arepresent.
I Some links (Si ,Sk), 0 ≤ i < k ≤ n, labelled with σk ,can be present.
I Some links (Si ,Sj), 0 ≤ j < i ≤ n, can be present(suffix).
For any factor p of s
I There is a unique path labelled p from S0 to some Si ,following factor links.
A suffix link (Si ,Sj) exists iff
I The longest repeated suffix of s[1..i ] is recognized in j .
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Factor Oracle: properties
Let s = σ1, σ2, ..., σn
I States are S0, S1, ...,Sn
I All links (Si−1,Si ), 1 ≤ i < n, labelled with σi , arepresent.
I Some links (Si ,Sk), 0 ≤ i < k ≤ n, labelled with σk ,can be present.
I Some links (Si ,Sj), 0 ≤ j < i ≤ n, can be present(suffix).
For any factor p of s
I There is a unique path labelled p from S0 to some Si ,following factor links.
A suffix link (Si ,Sj) exists iff
I The longest repeated suffix of s[1..i ] is recognized in j .
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Factor Oracle: properties
Let s = σ1, σ2, ..., σn
I States are S0, S1, ...,Sn
I All links (Si−1,Si ), 1 ≤ i < n, labelled with σi , arepresent.
I Some links (Si ,Sk), 0 ≤ i < k ≤ n, labelled with σk ,can be present.
I Some links (Si ,Sj), 0 ≤ j < i ≤ n, can be present(suffix).
For any factor p of s
I There is a unique path labelled p from S0 to some Si ,following factor links.
A suffix link (Si ,Sj) exists iff
I The longest repeated suffix of s[1..i ] is recognized in j .
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Factor Oracle: on-line construction
For each new input symbol σi (say σ4 = a):
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Factor Oracle: on-line construction
A new state Si is added
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Factor Oracle: on-line construction
A factor link (Si−1,Si ), labelled σi is created
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Factor Oracle: on-line construction
Starting from Si−1, suffix links are iteratively followedbackwards
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Factor Oracle: on-line construction
until a state Sj where link (Sj , Sk) labelled σi exists... or there are no suffix links to follow (i.e. Sj = 0)
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Factor Oracle: on-line construction
for each state Sm encountered in the iteration,create a factor (Sm, Si ), labelled σi
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Factor Oracle: on-line construction
finally, create a suffix link (Si ,Sj)
Factor Oracle for s = abaa
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
A utcc model for the FO
I The data structure is represented as constraints of theform edge(x , y ,N).
I Each event (input) play(Note) triggers theconstruction of the new node and arcs.
I Synchronization is done semantically , through theavailable information in the store: It scales up tosituations involving several agents.
Step1(Note)def= tell(edge(i − 1, i ,Note)) ‖ Step2(Note, i − 1)
Step2(Note,E)def= when E = 0 do
(abs k; edge(E , k,Note))(tell(edge(i , k, ?)) ‖ next tell(ready))‖ unless ∃kedge(E ,K ,Note) next
(tell(ready) ‖ tell(edge(i , 0, ?))). . .
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Outline
The utcc calculus
Dynamic Interactive Scores
Music Improvisation Systems
Concluding Remarks
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Concluding Remarks
We argued for utcc as a declarative framework for modelingand verifying dynamic multimedia interaction systems.
I Synchronization mechanism based on entailment ofconstraints leads to simpler models that scale up whenmore agents are added.
I Systems can be formally verified by using theunderlying temporal logic in utcc.
Two models of non trivial interacting systems.
I Interactive scores where the composer can dynamicallychange the structure of the score according to theinformation derived from the environment.
I A simpler model of the FO that scales up when multipleagents are considered.
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Concluding Remarks
We argued for utcc as a declarative framework for modelingand verifying dynamic multimedia interaction systems.
I Synchronization mechanism based on entailment ofconstraints leads to simpler models that scale up whenmore agents are added.
I Systems can be formally verified by using theunderlying temporal logic in utcc.
Two models of non trivial interacting systems.
I Interactive scores where the composer can dynamicallychange the structure of the score according to theinformation derived from the environment.
I A simpler model of the FO that scales up when multipleagents are considered.
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Future work
I Guarantee reliable responses in time: Assessing thebehavior of utcc processes in real-time contexts.
I Provide a more principled notion of time where theduration of each time-unit can be related to the amountof computation involved in it.
I Enrich the FO model with probabilistic traversals of thegraph.
MultimediaInteraction in utcc
Olarte & Rueda.
Motivation
Outline
The utcc calculus
DynamicInteractive Scores
MusicImprovisationSystems
ConcludingRemarks
Thank you!