A Continuous Media Data Rendering System For Visualizing Psychological Impression-Transition

A Continuous Media Data Rendering System For

Visualizing Psychological Impression-Transition

† Fujiko Yara ([email protected]) 　 ‡ Naofumi Yoshida ([email protected])

‡ Shiori Sasaki ([email protected]) † Yasushi Kiyoki ([email protected])

† Faculty of Environmental Information, Keio University‡ Graduate School of Media and Governance, Keio University

Application Example and Demonstration

• Our system enables us to render the continuous impression-transition for making user’s feeling change.

• Our system calculates the relationship between media data and impression.• And then renders set of media data continuously.• The demonstration shows an example of rendering media data.

Demonstration

STRTFINISHconfuse comfortable

Input : from “confuse” to “comfortable”

Output : Sequence of colors by emotionally continuous transition

t

( ﾟﾟ ;)( ｡｡ ;))((; ﾟﾟ )(; ｡｡ )

(*´ ｪ `*)

～～～Smell

♪♪

♪ ♪Music

Picture

Lighting

The statement of user is “confuse”. The statement of user is “comfortable”.

Demonstration

Three Main Features in our system

1. Our system enables us to render media data with continuous impression-transition to guide user’s feelings by psychological models.

2. There are a lot of variety of routes. Because of many models and routes, our system renders the appropriate set of media data in the several scenes for users.

3. Our system use the psychological impression-transition models that can be applied to render various media continuously.

Our Understanding of the Impression-Transition Models

We believe that it is necessary to visualize psychological impression-transition for rendering continuous media data emotio

nally. • In psychological impression-transition models, many elements (for exa

mple, color data, impression-words) are continuously allocated by something to define.

• CIS (Color Image Scale) model allocates color data into 2 dimensional manner, and Hevner-model allocates impression-words into 1 dimensional manner.

• We think the continuous relationships between each elements in these models show emotionally continuous transition.

CIS-model

Allocated by color gradation

Allocated by impression-words

Hevner-model

Overview• In this presentation, we show an

implementation method and effectiveness of our method .

• We demonstrate several experiments for the feasibility.

• We explain about our understanding of Impression-Transition Model and how to apply to our system.

Background Issues

Retrieval ResultsDB

Design

Rendering media data with emotionally

sequence

Our target

•Multi Media Data Base

•Meta Data Base System [KK_1993]

•Data Base Management system

•Metadata of Multimedia management system

•Media data searching method and system [IK_2005] [KK_1994]

•Metadata extraction method and system [IKNS_2003]

•Psychological Field

•Image transaction technology

•Psychological Field

•[KK_1993] Kitagawa, T. and Kiyoki, Y. :The mathematical model of meaning and its application to multidatabase systems, Proceedings of 3rd IEEE International Workshop on Research Issues on Data Engineering Interoperability in Multidatabase Systems, April 1993, 130-135•[KK_1994] Kiyoki, Y. Kitagawa, T. and Hayama, T. : A metadatabase system for semantic image search by a mathematical model of meaning, ACM SIGMOD Record, Vol.23, No.4, 1994, 34-41.•[IKNS_2003] shibashi, N. Kiyoki, Y. Nakagami, Y. and Sato, A. : An Impressionistic Metadata Extraction Method for Music Data with Multiple Note Streams, DBSJ Letters, Vol.2, No.2, October 2003, pp.61-64 .•[IK_2005] Ijichi, A. and Kiyoki, Y.:A Kansei Metadata Generation Method for Interpretation, Information Modeling and Knowledge Bases, 16, 170-182, 2005.

Retrieval technology

Rendering technology

System Overviewresult 1

・・・

Impression-Transition DB

CIS-model

Hevner-model

Metric for rendering emotionally continuous transition

Start-point-query

Goal-point-query

Start-media data　

Goal-media data

Metric for distances on vector spaces

Multi Media DBColor ＤＢ Music ＤＢSensor ＤＢ

start-query　

goal-query

terminal-impression word

starting-impression word

▲

▲

▲

▲

▲ ▲▲

▲■

■■■

■

■■■

■

▲■

starting-point query◆ ◆

terminal-point query

▲▲

starting-media data

terminal-media data

Vector Space　

Basic Method (1/2)• Calculation in Metric for distances on vector space.

■　 Impression-Words in Longman Dictionary　▲　 Impression-Words expressing Media data◆ Query words



▲

▲

▲

▲

▲ ▲

▲

▲■

■

■■

■

■■

■

■

▲■



▲

▲

starting-media data

terminal-media data

1. Creating an impression-words vector space to calculate the impression retrieval.

2. Impression-words and media data are mapped into this vector space.

3. Converting 2-queries into 2-media-data.

Basic Method (2/2)• Calculating in Metric for rendering

emotionally continuous transition.

・・・・・・・●

●● ●

●●

●●●

● ● ●

●●●●

・・・・・・

W (1)W (2) W (3)

W (n)W (n-1)

W (1)W (2)W (n)

W (n-1)W (n-2)

・・・・・・

Based Point

Based Point

・・・●● ● ●●

● ●●● ●

●● ●●● ● ●・・・

・・・

Sets of rendering output media data

CIS-model

Hevner-model

1. Using transition model representing something to meant.

2. Creating database of transition’s routes.

3. Defining many parameters to decide the route.

4. Mapping 2-media data (start-media data and terminal-media data) into transition model.

5. Converting data on transition model into output media.

Implementation Method (1/8)• Step 1 : Creating an impression-words vector space

of media data and mapping the impression-words of media contents into it.

▲

▲

▲

▲

▲

▲

▲▲

▲

▲■

■■

■

■■

■

■

■

■■

■

■

▲

■　 Impression-Words in Longman Dictionary　　　▲　 Impression-Words expressing Media data

1. We have implemented this system using the Mathematical Model of Meaning (MMM) [KK_1994].

2. The MMM search space is created by using the Longman Dictionary of Contemporary English.

3. The impression-words of media data are mapped into this impression-word vector space.

•[KK_1994] Kiyoki, Y. Kitagawa, T. and Hayama, T. : A metadatabase system for semantic image search by a mathematical model of meaning, ACM SIGMOD Record, Vol.23, No.4, 1994, 34-41.

•Longman Dictionary of Contemporary English, Longman, 1987

途中！

Implementation Method (2/8)• Step 2 : Creating database representing the

route of the impression-transition.

1. By using psychological models as impression-transition models, we create databases which express the route of the impression-transition.

2. Using the Hevner-model and the CIS-model (Color Image Scale model) as impression-transition.

途中！

Implementation Method (3/8)• Step 3 : Submitting two query words (starting-query

and terminal-query) into the impression space.

■　 Impression-Words in Longman Dictionary　▲　 Impression-Words expressing Media data● Query words

1. A user submits two query words into MMM search space created in step1.

2. Two query are not always words used in Longman Dictionary, so in next step, two query words are converted into the words used in Longman Dictionary.

terminal-impression

start-impression▲

▲

▲

▲

▲ ▲▲

▲

▲■

■■

■

■

■

■■■

■

start-query

terminal-query・・

■

▲■

途中！

Implementation Method (4/8)• Step 4 : Converting two query words as two

impression words (starting-impression and terminal-impression).

■　 Impression-Words in Longman Dictionary　▲　 Impression-Words expressing Media data● Query words

terminal-impression

start-impression▲

▲

▲

▲

▲ ▲▲

▲

▲■

■■

■

■

■

■■■

■

starting-query

terminal-query・・

■

▲■

1. The starting-query is converted into the semantically closest word (starting-impression) within the impression words included in the route representing the impression transition (Step2).

2. The terminal-query is also converted into the semantically closest word (terminal-impression).

途中！

Implementation Method (5/8)• Step 5 : Converting impression-words to media data

(starting-media data and terminal-media data)

start-impression

terminal-media data

start-media data

▲terminal-

impression

▲

▲

▲

▲

▲

▲

▲

■

■■

■

■

■

■■■

■

▲

■　 Impression-Words in Longman Dictionary　　　▲　 Impression-Words expressing Media data

途中！

Implementation Method (6/8)• Step 6 :Extracting two media data from vector

space and mapping into the impression-transition models respectively.

Hevner-model

CIS-model

1. Two media data (Starting-media data and terminal-media data) are mapped into the database based on the impression-transition models.

2. The appropriate route is chosen for connecting starting-media data and terminal-media data continuously.

途中！

Implementation Method (7/8)• Step 7 : Choosing the route on the impression-transition

model for rendering media data for visualization by color data.

CIS-model

Allocated by color gradation

Hevner-model

Allocated by impression-words

1. To realize rendering the media data continuously, the route from starting-media data to terminal-media data is chosen.

2. According to taking time, the way how to walk on the impression-transition is decided.

3. The way how to walk must be uniformed like chess.

途中！

Implementation Method (8/8)• Step 8 ： Rendering the sequence of output media

data generated in Step7.

1. We render the set of color data generated in Step7 along the selected route using the impression-transition model.

2. A personal computer is used to display the rendering of the set of color data.

途中！

Results Example

Experimental Results (1/3)[confuse → comfortable]

• These results of the set of color data show the feasibility of our continuous media data rendering system for visualizing the change of psychological impression-transition.

(start-point-query=“confuse”,　 goal-point-query=”comfortable” R=shortest (右回り ))

(start-point-query=“confuse”,　 goal-point-query=”comfortable”, R=longest (左回り ))

(start-point-query=“confuse”, goal-point-query=”comfortable” R=第４節 5(B))

Hevner-model CIS-model

Results Example

Experimental Results (2/3)[merry → calm]

• These experiments have shown the applicability of our method for user’s various requirement of impression-transition.

(start-point-query=”merry”, goal-point-query=”calm” R=shortest (右回り ))

(start-point-query=”merry”, goal-point-query=”calm” R=longest (左回り ))

(start-point-query=“merry”, goal-point-query=“calm”　 R=longest (Right and Down))

(start-point-query=“merry”, goal-point-query=“calm”　 R=longest (Down and Right))

(start-point-query=“merry”, goal-point-query=“calm” R=shortest (Left and Down))

(start-point-query=“merry”, goal-point-query=“calm” R=shortest (Down and Left))

Hevner-model

CIS-model

Results Example

Experimental Results (3/3)[simple → graceful]

• Our method has applicability for various strength of relationship between two query words, even if the two impression-words have a weak relationship in the vector space.

(start-point-query=“simple”, goal-point-

query=“graceful”)

(start-point-query=“simple” goal-point-query=“graceful”, R=longest (Right and Down))

(start-point-query=“simple” goal-point-query=“graceful”,

R=longest (Right and Down))



Hevner-model

CIS-model

Summary and Future Work• Our method makes it possible to implement visualizations

of the continuous change of impression-transition, according to impression-words expressed for starting point and terminal point.

• By the implementation and experiments using color data as output media data, we have clarified the feasibility of our method for visualizing the change of impression-transition from starting point to terminal point by using the research results of musical psychology and color psychology.

• We will design aggregate functions for color data expression in the experiments using psychological word groups by the Hevner-model.

• We will approach to the computation mechanisms of continuous transition of impression.

References• [KK_1993] Kitagawa, T. and Kiyoki, Y. :The mathematical model of meaning and its application t

o multidatabase systems, Proceedings of 3rd IEEE International Workshop on Research Issues on Data Engineering Interoperability in Multidatabase Systems, April 1993, 130-135

• [KK_1994] Kiyoki, Y. Kitagawa, T. and Hayama, T. : A metadatabase system for semantic image search by a mathematical model of meaning, ACM SIGMOD Record, Vol.23, No.4, 1994, 34-41.

• [KKH_1995] Kiyoki, Y. Kitagawa, T. and Hitomi, Y. : A fundamental framework for realizing semantic interoperability in a multidatabase environment, Journal of Integrated Computer-Aided Engineering, Vol.2, No.1, Jan.1995, 3-20.

• [AS_1994] Aiello, R. and Slobada, J.A.: Musical perceptions, Oxford University Press, 1994.• [IK_2005] Ijichi, A. and Kiyoki, Y.:A Kansei Metadata Generation Method for Interpretation, Infor

mation Modeling and Knowledge Bases, 16, 170-182, 2005.• Longman Dictionary of Contemporary English, Longman, 1987• [H_1937] Hevner, K. : The affective value of pitch and tempo in music, American Journal of psyc

hology, 49, 621-630.• [IKNS_2003] shibashi, N. Kiyoki, Y. Nakagami, Y. and Sato, A. : An Impressionistic Metadata Ex

traction Method for Music Data with Multiple Note Streams, DBSJ Letters, Vol.2, No.2, October 2003, pp.61-64 .

Thank you for your attention.

Derivation of Kansei• 印象遷移モデルは一定を約束している• 別のストーリを持つモデルは多く存在し、• 微分係数が一定でないものが多い。

・・・・・・・●

●● ●

●●

●●●

● ●●

●●●●

・・・

・・・W (1)

W (2) W (3)

W (n)W (n-1)W (1)

W (2)W (n)

W (n-1)W (n-2)

・・・・・・

Based Point

Based Point



▲

▲

▲

▲

▲ ▲▲

▲■

■■■

■

■■■

■

▲■



▲▲

starting-media data

terminal-media data

Vector Space　

c1

sober

serioussacred

dignifiedinspiring

awe

loftyspiritual

solemn

c2

dark

depressing

dolefulfrustrated

sad

melancholy

heavygloomy

tragicmournful

pathetic

c3

dreamy longingplaintive sentimental

yielding

yearningtender

c4calm leisurely

tranquilsoothingserenesatisfying

quietlyricalc5

playfullight

humorousgraceful fancifuldelicate

whimsicalsprightly

quaint

c6

merryjoyousbrighthappygaycheerful

c7

passionateimpetuous

exhilaratedexciting dramaticagitated

sensationalrestless

triumphant

soaring

c8

emphatic exaltingvigorous

robustponderousmartialmajestic

始点の印象語S:Start-query

終点の印象語t:goal-query

Hevner’s model[4,5]

CIS model[6]

Neuton’s model

○

○○

○

○○

●● ●● ●● ●●・・・・・・・・ Hevner’ modelでのルート群RHev・・・・・・・・・・・・・・・・

n

●● ● ・・・・・・・・●● CIS model でのルート群RCIS・・・・・・・・・・・・・・・・

n

○○○○○○○

●

○

○○

○

Mansel’s model

メディアデータＭＤで出力メディアデータＭＤで出力

・・・・・・

心理学的印象モデル Co群

○ ：印象語● ：ＭＤ

●● ●●●● ● ●●● ●● ●●●● ● ●●● ●● ●●

●● ● ●●● ●● ●●●● ● ●●● ●● ●●●● ● ●●● ●● ●●

c1

sober

serioussacred

dignifiedinspiring

awe

loftyspiritual

solemn

c2

dark

depressing

dolefulfrustrated

sad

melancholy

heavygloomy

tragicmournful

pathetic

c3dreamy longingplaintive sentimental

yielding

yearningtender

c4calm leisurely

tranquilsoothingserenesatisfying

quietlyricalc5

playfullight

humorousgraceful fancifuldelicate

whimsicalsprightly

quaint

c6

merryjoyousbrighthappygaycheerful

c7

passionateimpetuous

exhilaratedexciting dramaticagitated

sensationalrestless

triumphant

soaring

c8emphatic exalting

vigorous

robustponderousmartialmajestic

Documents

A Continuous Media Data Rendering System For Visualizing Psychological Impression-Transition