Visualization of Indian Classical Music

A Thesis Submitted

in Partial Fulfillment of the Requirements

for the Degree of

Master of Technology

by

Nameeta Shah

to the

Department of Computer Science & Engineering

Indian Institute of Technology, Kanpur

February 2001

i

Dr. Sanjay G. Dhande

Professor, Computer Science & Engineering & Mechanical Engineering

Indian Institute of Technology,

Kanpur.

Certificate

This is to certify that the work contained in the thesis entitled Visualization of Indian

Classical Music, by Nameeta Shah, has been carried out under my supervision and that

this work has not been submitted elsewhere for a degree.

February 2001

ii

To m y husband

K inshuk

iii

Acknowledgement This M.Tech thesis has been an enlightening research experience, which can only be

attributed to my thesis supervisor Dr. Sanjay G. Dhande. Without his encouraging

words it would have not been possible to carry out the work in the area that was

unknown to me. At certain points I felt I had come to a dead end, but then Dr. Dhande

always found a way out.

Dr. T. V. Prabhakar and Archana ma’am always listened to my problems, suggested

solutions and more than often helped me solve the problem. Their affectionate ways

will always remind me of “good things in life” .

I would like to express my gratitude towards the people at Banaras Hindu University

especially Dr. Ritvik Sanyal and Ms. Urmila Sharma who helped me with my

research in Visual−Music association in the Indian arts history.

People in the department, in the CAD−P lab and at IIT, Kanpur in general have been

very kind and co−operative. I thank Mrs. Biswas and Dr. Avinash Singh who helped

me record the Indian Classical music in MIDI format. It was a pleasant experience

taking a few lessons of piano from Mrs. Chowdhary.

My friend Charu although remotely located gave me some ideas for my thesis work.

Richa, Raju and Pooja were always there to listen to me and support me.

Finally I thank my family whose presence I always feel. It strengthens me all the time.

I am grateful to my husband Kinshuk for his annoying interference with only one aim

to make me do better work.

iv

Table of Contents

I. Certificate……………………………………………………………..i

II. Dedication…………………………………………………………….ii

III. Acknowledgement…………………………………………………...iii

IV. Table of Contents…………………………………………………….iv

V. List of Figures………………………………………………………viii

VI. List of Tables………………………………………………………...ix

VII. Abstract……………………………………………………………….x

1. Introduction ...................................................................................1

1.1 Development in Multimedia Technology........................................1

1.2 Music and Computers.....................................................................4

1.3 Computers Graphics.......................................................................5

1.4 Objectives and Goals......................................................................6

1.4.1 Associate music with graphical images:.......................................6

1.4.2 Have a grounding in Indian classical arts:....................................6

1.4.3 Study parallels in western music:.................................................6

1.4.4 Propose a model:.........................................................................6

1.4.5 Implement the model:..................................................................7

1.5 Related Work..................................................................................7

1.5.1 Fred Collopy’s Sonnet+Imager....................................................7

1.5.2 Scott Draves’s Bomb ..................................................................7

1.5.3 Sandy Cohen’s Muse X−Rayer.....................................................8

v

1.6 Organization...................................................................................8

2. The Visual – Music Association in the Indian Art History..........9

2.1 Rasa Theory....................................................................................9

2.2 Associations of Rasa, Note, Color, Shape: ...................................10

2.3 Ragadhyana..................................................................................12

2.4 Ragamala Paintings......................................................................12

2.5 Associations in Recent History.....................................................14

3. Approach......................................................................................15

3.1 Dimensions of Music: Pitch, Timbre and Rhythm & Speed..........15

3.1.1 Pitch .........................................................................................15

3.1.2 Timbre .....................................................................................17

3.1.3 Rhythm & Speed ......................................................................17

3.2 Visual Dimensions: Color, Form and Motion................................18

3.2.1 Color.........................................................................................18

3.2.2 Form.........................................................................................19

3.2.3 Motion ......................................................................................21

3.3 The Audio–Visual Analogy..........................................................23

4. Proposed Design and Implementation........................................27

4.1 Proposed Design...........................................................................27

4.2 Implementation.............................................................................28

4.3 Results..........................................................................................34

4.3.1 Raga Asavari and Todi..............................................................34

4.3.2 Jhaptaal.....................................................................................36

vi

4.3.3 Jugalbandi of Sitar and Tabla....................................................36

4.3.4 Flute and Harmonium................................................................37

4.3.5 Triads of Western Classical Music............................................37

5. Conclusion and Recommendations for Future Work................38

5.1 Technical summary.......................................................................38

5.1.1 Contributions............................................................................38

5.1.2 Limitations................................................................................39

5.2 Suggestions for further work.........................................................39

References 41

Appendix I – Glossary 43

Appendix II – Setup Diagram 48

Appendix III – MIDI 49

vii

List of Figures

Figure 1−1 Computer – Human Interfaces...................................................................2

Figure 1−2 A Simple Model of Computation..............................................................2

Figure 1−3 A simplified model of our system.............................................................3

Figure 2−1 Ragamala Painting of Pahadi Ragini.......................................................13

Figure 3−1 Figure and Ground..................................................................................19

Figure 3−2 ‘Differentiation’ or ‘Segregation’ ............................................................20

Figure 3−3 Closure...................................................................................................20

Figure 3−4 Audio−Visual Functional Relationship.....................................................25

Figure 4−1 Block Diagram of the Visualization System............................................27

Figure 4−2 Snapshot of ControlPanel........................................................................29

Figure 4−3 BrushCurve.............................................................................................33

Figure 4−4 BrushBackground...................................................................................33

Figure 4−5 BrushSpiral.............................................................................................33

Figure 4−6 BrushFlute..............................................................................................34

Figure 4−7 BrushLetter.............................................................................................34

Figure 4−8 Aroha and Avaroha of Asavari and Todi.................................................35

Figure 4−9 Jhaptaal...................................................................................................36

Figure 4−10 Sitar and Tabla......................................................................................36

Figure 4−11 Harmonium and Flute...........................................................................37

Figure 4−12 Triads of Western Music.......................................................................37

viii

List of Tables

Table 2−1 Rasa, Bhava, Color, Deity Associations in Natyashastra...........................11

Table 2−2 Note, Color, Varna Associations in Naradiya Shiksha..............................11

Table 2−3 Note, Color, Deity, Rasa Associations in Sangita Ratnakara.....................11

Table 3−1 Frequency Ratios of Sruti Values.............................................................16

Table 3−2 Action Words Resultant from Different Combinations of Space, Time and

Force.................................................................................................................22

Table 4−1 Sruti Color−Scheme..................................................................................30

Table 4−2 Equi−tempered Color−Scheme...................................................................31

Table 4−3 Features of Asavari and Todi....................................................................35

ix

Abstract

The motivation behind this research in Visualization of Indian Classical Music is an

attempt at bringing the visual and the aural modes together and to evolve a new art

form of Visual Music. Though Indian attempts in this context date back to the 14th

century, the evolving technologies of Computer Graphics and Synthesized music have

added a new dimension to the concept of visualization. Emerging visual patterns

generated by computer can draw a listener’s attention to the way music evolves.

Music can be considered as some complex information that can be visualized with

computer based information visualization concepts and technologies. Aesthetic

principles are also taken into consideration.

As the area of this thesis is the Indian Classical Music, the computational model for

Visualization is based on the principles of Indian Aesthetics. A Visualization System

that can evolve with time was designed and a working model was implemented. The

results of this system have encouraged further experimentation and discovery of this

art, using computer as a medium.

x

1

Chapter 1

Introduction

The evolution of computing paradigms has shifted drastically over the last few years.

There now exists, courtesy of the World Wide Web, a large amount of interleaving of

sensory perceptions and of computer generated content. The advancements in

visualization have led to projects like the National Tele−immersion Initiative [1] that

create environments in which users interact across an Internet2 connection as in real

life, to give a feeling of immersion by two remotely located users in a common virtual

environment.

The motivation behind this research in Visualization of Indian Classical Music is an

attempt at bringing the visual and the aural modes together and to evolve a new art

form of Visual Music. Though Indian attempts in this context date back to the 14th

century, the evolving technologies of Computer Graphics and Synthesized music have

added a new dimension to the concept of visualization [2]. Emerging visual patterns

generated by computer can draw a listener’s attention to the way music evolves.

Music can be considered as some complex information that can be visualized with

computer based information visualization concepts and technologies [3]. Aesthetic

principles are also taken into consideration. As the area of this thesis is the Indian

Classical Music, the computational model for Visualization is based on the principles

of Indian Aesthetics.

The attempt at the Visual Music is a global activity and is not confined to any

particular region. In the early 18th Century, inspired by Newton’s work on color

theory, Louis Bertrand Castel produced a light organ, clavecin oculaire. Painters,

composers, and filmmakers have carried out numerous experiments since then. [4]

1.1 Development in Multimedia Technology

Today Intelligent Human−Computer Interfaces −− interfaces that raise the computer to

the human level, rather than requiring that the human adapt to the computer, is a very

important area of research. An intelligent interface makes interaction with the

2

computer easier, more intuitive, and more flexible. The following figure (Fig.1.1) is a

sketch of an interpretation of Computer Interfaces. The subject of multimedia research

deals with the interfaces in various forms. An interface between humans and the

system or computers, is possible through the human sensory organs and sensory

perceptions. The initial studies of these interfaces have been centered on the visual

and audio interfaces, and how to optimize them. The term ‘multimedia’ generally

implies sound and graphics as medium of interfacing the user with the computer.

Figure 1−1 Computer – Human Interfaces

However, recent developments in multimedia technology have explored the touch [5],

tactile tablet [6] and ubiquitous computing [7] which aims to extend computing

beyond traditional interfaces.

The figure below (Fig.1.2) is the simplest model of computation, which remains

consistent across various areas. There is an analogue input signal that is digitized with

an analogue to digital converter. This digital information is processed to get the output

in digital format. The processed data is again converted to an analogue form.

Depending on the type of processing of the digitized input, the output changes. The

multimedia system proposed in this thesis takes music signal as input and produces a

combination of audio and visuals as an output to the user (Fig.1.3).

Input Output

Figure 1−2 A Simple Model of Computation

Computer

Human

Interfaces

Eye Graphics

Ear Voice, Music

Skin Touch

Nose Smell

Tongue Taste

A/D Converter

A/D Converter Digital

Processing

3

Figure 1−3 A simplified model of our system, where pieces of the jigsaw puzzle

are the inputs, which fit together to produce the output, which is the

Visualization of Music

Visualization of Indian Classical Music

Music Input

Individual user and his mood, time of day

Mapping algorithm and our software implementation

Elements of the Indian Classical Raga or Taal

4

1.2 Music a nd Computers

The relation between computation and music has been explored in the last 30 years or

so, since the advent of digital music and synthesizer − generated sounds artificially

produced, through the late 70’s and the 80’s (the disco generation). However, since

then, these two disciplines have come together far more frequently.

The developments centered around

• New digital formats for soundtrack recording

• The analysis, filtering and smoothing of sound using software in studios, and

also the recording of sound produced via instruments, directly onto software

based interface, through sound formats like .WAV and .MIDI.

Since the evolution of the Internet and Web−based content in the 90’s, we see a wider

array of instruments and also big range of standards for recording, playback and

indeed, real−time interaction of music and musicians. Also, the reach of these

technologies has become truly global, with the requirements being trivialized to a

computer, a browser and an Internet connection, for all wannabe Internet musicians

around the world.

We are however, exploring the relationship from a much lower level, as far as music

is concerned. We are dealing with and siphoning out the notes, the rhythms, the tempo

and the instrument, and going to the science of music.

Also, the uniqueness of this endeavor is the Indian−ness or the Indian flavor of music,

which we have adopted as our mainstay. The main and essential character of Indian

Music is its linearity of one−line movement. Due to this melodic nature, transposition

and temperament−pitch levels have become very significant and “hair−breadth”

distinction necessary. These differences are measured and indicated by assigning

them sruti values. Musically, sruti points to the interval between notes that can be just

perceived auditorily. Musically viable pitches in an octave are literally infinite but for

practical purposes twenty−two srutis have been enumerated. The other factor is the

way in which a tone is produced. It could be a straight note; but as is often the case,

highly embellished. This is done by giving it a shake, a trill, a glide or touch of

another note, a swing and so on. Raga and Tala are the melodic and rhythmic

concepts of Indian Classical Music. Raga is a seed idea that has to be grown and

5

blossomed out by the creativity of artist. The Raga has these qualities: the notes have

to be of definite nature (scale), they are to be in a certain order of tonal syntax, with

emphasis on determined ones. Besides, there have to be characteristic phrases. Tala is

a rhythmic experience so arranged that there is a feeling of ‘ coming back’ to the

origin, the arrangement becomes repetitive or cyclic; for it is only in a circle that one

returns to the beginning, in a uniform manner. Another distinguishing feature of

Indian Music is ‘ improvisation’ . No musical production is on the basis of a score. It is

entirely from memory but it is not repetitive reproduction. It is the unfoldment of a

musical idea. The ‘ time’ theory whereby a Raga is supposed to be sung at a particular

time of the day is also an interesting psychological aspect of Indian Raga music [8].

The synthesized computer music still has to go a long way in order to capture the

concept of srutis and embellishments, which are the essence of Indian Music.

1.3 Computers Graphics

Computer Graphics as a research field began in the 1960’s. People such as Charles

Csuri, Ken Knowlton and John Whitley Sr were creating “Computer assisted

graphics” as a new and unique art form.

Developments like the Sketchpad by Ivan Sutherland in 1961 were conceived as a

revolution [9]. Using an electromagnetic light pen, Sketchpad allowed you to draw

simple shapes on the computer screen, save them and recall them later. Sutherland

along with Evans are considered as pioneers of this field, and dominated the research

in the University of Utah through the next decade.

The field began with what is known as vector graphics, which essentially is mappings

of thin lines between the starting and ending point of the drawing “pen” or instrument.

The field of Raster based graphics instead deals with pixels, or very small partitions

of the viewer screen, to which various colors and shapes are mapped, and hence are

capable of producing continuous tone images and produce the whole array of colors.

The modern developments are centered more on pervasive computer graphics, with its

applications in all the fields like business, arts, engineering, and animation. The

challenges are now the generation of Real−time graphics and the research centers

6

around Tele−immersion [1], which deal with creating a virtual reality environment in

which the users from different locations can participate at the same time.

1.4 Objectives and Goals

Like any broad and open area of research, there is only a certain part that can be

accomplished in the given constraints of time and other resources. This thesis work

has the following clearly defined objectives and goals:

1.4.1 Associate music with graphical images: The main aim of this thesis is to have a visualization process for Indian classical

music. It is a well−established fact that it is easier to remember something when an

image is associated with it. Taking this fact into consideration the aim is to associate

music with graphical images so that something as complex and non−representational

as Indian Music is better understood. The association has to be meaningful.

1.4.2 Have a grounding in Indian classical arts: Since the subject matter that is being dealt with is Indian classical music, the

emphasis has been to implement a system with a strong undercurrent of the rules and

traditions of Indian classical arts. The uniqueness of Indian classical music is

explained with some detail in the description, and has been behind the selection of

most processing elements which account for the color and feel of the output visuals.

1.4.3 Study parallels in western music: In order to make allowance for further extension of the work, care has been taken to

keep abreast of parallels in western music. Also, since some work has already been

done in various universities and institutions around the world in this area, inspiration

from such work is drawn to some extent.

1.4.4 Propose a model: A model of visualizing music is proposed with the above factors in mind. The use of

the model will help to facilitate further work along these lines, and is highly flexible

and extensible.

7

1.4.5 Implement the model: A working implementation has been developed to implement the model. This

implementation has been described in greater detail in the ensuing chapters.

Moreover, the need for a platform−independent solution has been addressed to some

extent, and the implementation has been made suitably simple.

1.5 Related Work

Presently a lot of work is being done in the field of Visual−Music. Some of the

present−day artists are: Fred Collopy [10], Scott Draves [11], Sandy Cohen [3], Larry

Cuba [12], Ron Pellegrino [13], Jeff Hoekman [14], Greg Jalbert [15], Stephen

Malinowski [16], Andrew Schloss [17] and David Tristram [18]. Work of each of

them is unique in that a different aesthetic model has been implemented. Description

of all the attempts is beyond the scope of this document and so a few of them will be

stated. All the information is obtained from the web sites or is based on the personal

experience of the candidate.

1.5.1 Fred Collopy’s Sonnet+Imager [10] Sonnet+Imager is software for visual−music that runs under Max on Macintosh

computers. With Imager an environment is designed in which a variety of graphic

instruments can be built. Its architecture is such that the aesthetic choices are

decoupled from the underlying rendering engine. Imager uses three elements of form,

color and rhythm to describe the space of dynamic visuals. For each of these three

domains the choice is made about how the composers and players will control them.

The choice is based on some aesthetic principles.

Sonnet furnishes a platform on which the Imager visuals are triggered and activity is

synchronized between Imager and external MIDI music content.

1.5.2 Scott Draves’s Bomb [11] Bomb is a software system that produces visual music. It creates fluid, textured,

rhythmic, animated and generally non−representational video stream. It uses

techniques of non−linear iterated systems, like video feedback, but implemented on an

ordinary PC.

Bomb has a symbolic component: icon sequences. The icons drive the graphics, and

are drawn from a collection of scanned images. The important sequences are

geometric, basic shapes and symmetries, faces, masks, petroglyphs (symbols for

8

people), basic cultural and religious, human communication systems: letters, scripts,

and words, and technology.

Bomb has about eighty color palettes built into it. As of now, bomb makes only

minimal and rather simplistic use of the audio signal to change the visual parameters.

Keyboard commands can be used to do a variety of things like changing speed,

texture, color, rhythm, kind of sequences that are mentioned above, and many more.

The visuals are abstract and the interpretation is left to ones imagination.

1.5.3 Sandy Cohen’s Muse X−Rayer [3] Muse X−Rayer is a prototype program that shows how MIDI music may be visualized

and interactively changed by the viewer in real time. A release version, in the form of

a java applet is in development.

Bindu visualization is done using the Muse X−Rayer where loudness of each note of

music is visualized using brightness and relative size (silence is black). Position

shows pitch (treble on top), and color shows voice (bass is red). Time moves toward

the viewer in the zoom axis (the eternal present is the screen plane).

1.6 Organization

The report has been organized as follows.

• Chapter 2 goes into detail regarding the Visual − Music Association in Indian

Art History, and the author’s research on the elements of this association.

• Chapter 3 explains our approach to the generation of visuals based on music,

and details how music parameters are mapped to visual parameters.

• Chapter 4 proposes the actual system model, and also gives the details about

the implementation, and selective results.

• Chapter 5 is the concluding chapter in which we suggest the future extension

of this research and related possibilities on the horizon.

9

Chapter 2

The Visual – Music Association in the Indian Art

History

At the heart of every Indian Art is the Rasa Theory. Every artistic creation is supposed

to express a particular Rasa or a combination of Rasas. Thus it is important to

understand the Rasa theory in some detail.

2.1 Rasa Theory

The word rasa is generally translated as emotion, relish, etc. A closer study would not

warrant such a facile interpretation and hence it is often equated to ‘aesthetic

emotion’ . However, it is best to omit the connotation of sentiment completely and

translate it as ‘aesthesis’ . This distinction, though very fine, is yet a significant one.

Though rasa is not itself an excited state of mind, the emotional aspects of behavior

which go to form the substrate − though not the cause − of rasa have been analyzed in

great detail. It is not easy to find the modern Western psychological equivalents of

these various factors, which go to make an emotional state. But garbing the ancient

concepts in modern costumes makes for easier comprehension. The factors that form

the constituents of emotional action are, in Indian psychology, the following:

Vibhavas (the determinants): They are the causes of emotional responses. In modern

terminology we may call them the stimuli or releasers. Vibhavas themselves are of

two categories. Alambana − the person or object (or the idea of these), which acts as a

stimulus. Uddeepana − the situation that is the context for such behavior. Anubhavas

(the consequents): These form the responsive reactions. Obviously, with the Indian

theory of emotion being closely linked to dramaturgy, this usually refers to overt

expression. Sattvabhavas (involuntary responses): They are also the bodily signs of

emotion in which are included both external and internal responses to vibhavas.

Bhavas: By this is meant the states of mind, which are usually referred to as emotions.

Bhavas are considered to be of two kinds (1) sthayi bhavas are the ‘permanent’

10

emotions and (2) sanchari bhavas are the transient moods. The essential difference

between the two sthayi and sanchari bhavas seems to be that the former are more

lasting and common to all human beings, and the latter more fleeting and

characterized by the personal idiosyncrasy of the individual. According to the

generally accepted theories, there are nine differentia for the former and thirty−seven

for the latter. Though rasa itself is identified with emotion, in most of the current

literature, it is both emotional behavior and more; it is the awareness of the totality of

the emotional situation. It is a detached observance of such a condition of mind and

body. This is a very important statement, the significance of which cannot be

overemphasized. The experience of rasa is absolute and is known only by empathy,

that is to say, by entering into, feeling the permanent motif. Delightful or disgusting,

exalted or lowly, obscure or refined, actual or imagery, there is no subject that cannot

evoke rasa in man. While finally rasa is a contemplative state of mind, there are said

to be nine rasas corresponding to nine emotional conditions: sringara (erotic), hasya

(humorous), karuna (pathetic), raudra (furious), veera (valorous), bhayanaka (fearful),

beebhatsa (odious), adbhuta (wonderous) and santa (peaceful). Right from the very

ancient days attempts have been made to relate these specific rasas to music [19].

2.2 Associations of Rasa, Note, Color, Shape: [2]

The earliest associations of Rasa, Bhava, Color and Deity are found in Bharata’s

Natyashastra. They are tabulated as shown (Table 2.1).

Rasa Bhava Color Deity Shringar Rati Shyama (light green) Vishnu

Hasya Hasya Sita (white) Pranrath

Karuna Shoka Kapota (grey) Yama

Roudra Krodha Rakta (red) Rudra

Vira Utsaha Gaura (yellow−red) Mahendra

Bhayanaka Bhaya Krishna (black) Kala

Bibhatsa Jugupsa Nila (blue) Mahakala

Adbhuta Vismaya Pita (yellow) Brahma

11

Table 2−1 Rasa, Bhava, Color, Deity Associations in Natyashastra

The Naradiya Shiksha adds the further association of Swara with Color. (Table 2.2)

Note Color Varna

Shadja Padmapatraprabha (lotus−petal−red) Brahmin

Rishabha Shukapinjara (reddish yellow) Kshatriya

Gandhara Kanakabha (golden red) Half−Vaishya

Madhyama Kundasaprabha (white) Brahmin

Panchama Krishna (black) Brahmin

Dhaivata Pitaka (yellow) Kshatriya

Nishad Sarvavarna (multi / all colored) Half−Vaishya

Table 2−2 Note, Color, Varna Associations in Naradiya Shiksha

The associations head towards firm visualization and personification processes by the

time of Sangita Ratnakara, the musicological landmark. (Table 2.3) [2]

Note Color Deity Rasa

Shadja Rakta Vanhi Vira, Adbhuta, Roudra

Rishabha Pinjara Brahma Vira, Adbhuta, Roudra

Gandhara Swarna, Atipita Chandra Karuna

Madhyama Shubhra Vishnu Hasya, Shringara

Panchama Krishna Narada Hasya, Shringara

Dhaivata Pita Tumburu Bibhatsa, Bhayanaka

Nishad Vichitra Tumburu Karuna

Table 2−3 Note, Color, Deity, Rasa Associations in Sangita Ratnakara

Sangeethakalpadrumam goes to relate shape with note. In the chapter swaraprakarana

the author gives the week, star, dveepa, colors, clothes, ornaments, Shape and form of

the swaras. Tantric descriptions of swaras are also available in his work. A step

further is the concept of Ragadhyana, where the Ragas are given visual attributes

instead of individual swara.

12

2.3 Ragadhyana

The term “dhyana” is derived from dhyai, to meditate upon, imagine, and call to

mind. Dhyana is a mental representation of personal attributes of an image –

traditionally of a deity. Dhyana could be described as arresting the march of otherwise

evanescent, impressions received from everything selected as a stimulus−support and

the consequent stabilization of a particular impression.

Non−representational expression, such as offered by music, would need strategic and

representational applications and ragadhyanas were devised with this end in view [2].

The earliest prayer formulae in the shape of dhyanas (descriptive verses) were found

in Sangitopanisatsaroddhara. Here is a translation of one of the dhyana shlokas for

raga bhairav. (Translation by Gargi Pande – Project Associate, IIT Kanpur − 2001)

“Bhairav is white in complexion, has one head or visage and eight hands.

(He) rides a bull, dons antelope skin and puts on the features of Kalabhairav.

He is ornamented with serpent, trident, a club with human skull, rosary,

Veena, noose, fruits and lotus.”

SangitaRaj of Kumbh was the second one to have Ragadhyanas. The ‘Dhyanas’ of 34

‘Desi Ragas’ form a distinctive feature of the treatment of Raga in this text [20].

2.4 Ragamala Paintings

Till Ragadhyanas the visual−music relationship is verbal. Ragamala paintings

exemplify the Indian attempts to combine audio and visual to evolve art forms

combining music, painting, and literature or drama.

Ragamala means "a garland of musical modes" in the Sanskrit language. This

"garland of modes," or collection of melodies, is divided up into ragas and raginis.

The word raga literally means "something that colors the mind with a particular

feeling or emotion". The characters in the paintings, also called ragas (princes) and

raginis (ladies), personify the spirits of the various melodies. Each raga or ragini is

associated with a certain season and time of day, and each has its accompanying verse

of poetry [21].

The sixteenth and the seventeenth centuries saw two major works of Ragamala

paintings i.e. Kshemakarna/Meshakarna’s Ragamala (1509) and Pundarik Vitthal’s

13

Ragamala (1576). The works listed raga−ragini−putra families, provided descriptive

verses and followed up with pictures. This was followed by a number of Ragamalas

from different parts of the country [2].

Here is a verse of poetry describing Pahadi ragini from Sangitdarpana (ragamala of

mid 17th century) [22]:

“Pahadi ragini is a lady with a tambura seated on a rock. She is of golden

complexion. Pahadi is playing during the rainy season, represented by dark clouds and

golden lightning.” (Fig.2.1)

Figure 2−1 Ragamala Painting of Pahadi Ragini

It can be seen that Ragamala paintings are very metaphoric in nature. Water birds

(herons, egrets and cranes) might indicate a ragamala of the rainy season, as this is the

time when they mate. Clouds might also indicate the rainy season, or they might

14

symbolize the turbulent feelings of an anxious lover. Peacocks, because of their

impressive mating display, often represent beseeching male lovers.

2.5 Associations in Recent History

In modern times musicians have attributed moods to ragas. Melodies of khamaj type,

some say, are erotic, and those with ma−dha combinations express pathos and

lassitude.

Of course, there is not much doubt that the moods created do depend on the notes

used and their interrelations, for the state of mind aroused by a set of consonant notes

is surely different from that due to dissonant ones. But there is more to it. The

highness or lowness of a tone, the nature of melodic movement (straight or

meandering), the tempo and various other factors have their own part to play. Further,

there is the often missed but vitally important element − gamaka, for such micro−

variations and ornamentations of sounds contribute to a fullness and suppleness. Very

recently some experiments on current scientific lines were conducted by measuring

the responses to defined phrases for a few ragas. It was found that they did produce

fairly similar moods in all the listeners participating in the experiment, as instanced

below: Kafi: is very affective, humid, cool, soothing, light (not dense), deep; does not

agitate. Misra Mand: is pleasing, gay, refreshing, light, sweet, deep but does not

agitate; has no feeling of novelty. Pooriya Dhanasri: is sweet, colourful, deep, heavy,

weary, reflects stability, cloudy, sacred, has no vitality. Ragesri: is sweet, soothing,

deep, weary, dark; no novelty and is inflexible; stable and calm [19].

Thus, as we explore deeper, it becomes clear that visualizing melody is at the very

root of Indian Music.

15

Chapter 3

Approach

The visualization of music requires a movement from the audio domain to the visual

domain. The musical and visual dimensions are looked at carefully and then

correlated.

3.1 Dimensions of Music: Pitch, Timbre and Rhythm &

Speed

3.1.1 Pitch [23] [24] Vibratory motion of a mass in contact with air produces sound. A musical sound too

is a vibratory motion with some specific frequency. Frequency is a physical entity and

may be measured objectively but there is a psychological counterpart for frequency,

Pitch. In general, it is found that, as the frequency of a pure sound is raised, its pitch

goes up but the relationship is not linear. Music is created only with sounds of certain

frequencies. These sounds with certain frequencies are notes. Arrangement of notes in

time domain creates music.

The audible range is divided into 'octaves'. An octave is really a frequency range from

a frequency f1 to f2 such that f2 is twice that of f1 in terms of cycles or hertz. For

some physiological reason, the human ear is logarithmic and is sensitive to frequency

octaves. The audible frequency is then comprised of many, many octaves. Each

octave is divided into intervals. This division could be based on some mathematical

foundation. For example in Western Music, the frequencies are arranged in such a

manner that they are in a geometric series. The octave is divided into twelve parts and

the adjacent frequencies have a constant ratio i.e. twelfth root of two. By the time

thirteenth frequency is reached the ratio is doubled. This geometric arrangement of

frequencies in an octave is called an 'Equally tempered' arrangement. The frequencies

can be located based on some other non−geometric criteria, which might 'sound' even

better. Such scales in fact, exist and they are called 'Just tempered scales'.

16

In Indian Classical music Saptak corresponds to an octave. The Saptak comprises of

twenty−two tones called Shruti. The twenty−two ‘shruti scheme’ is given in Table−3.1.

Shruti Frequency ratio Frequency (Hertz)

Sa 1 240

Re1 32/31 252.8

Re2 16/15 256

Re3 10/9 266.6

Re4 9/8 270

Ga1 32/27 284.4

Ga2 6/5 288

Ga3 5/4 300

Ga4 81/64 303.7

Ma1 4/3 320

Ma2 27/20 324

Ma3 45/32 337.5

Ma4 64/45 341.3

Pa 3/2 360

Dha1 128/81 379

Dha2 8/5 384

Dha3 5/3 400

Dha4 27/16 405

Ni1 16/9 426.6

Ni2 9/5 432

Ni3 15/8 450

Ni4 31/16 465

Table 3−1 Frequency Ratios of Sruti Values

Shrutis are microtones that cluster around the seven pure notes – shuddha swaras of

Indian Classical Music. There are also additional five notes – vikrit swaras: Re komal,

Ga komal, Ma tivra, Dha komal and Ni komal. Komal implies a fall in the frequency

from the corresponding pure note while Tivra implies a rise in frequency. The

frequency table above is for the Madhya Saptak. There are other Saptaks above and

17

below the Madhya Saptak. The Mandra Saptak below the Madhya Saptak has the

same collection of twelve notes but each note has half the frequency of the note in

Madhya Saptak. For example Sa in Mandra Saptak has the frequency of 120.

Similarly there is a Tar Saptak above the Madhya Saptak where frequencies double −

Sa has the frequency of 480.

3.1.2 Timbre [23] [25] When equally loud tones of the same pitch are sounded from sarod, flute, santoor and

tuning fork, the ear instantly recognizes a difference. The same note does not sound

the same; this difference is called quality, timbre, or tone color.

A musical sound is due to the vibration of some part of a musical instrument: string,

air column, rod, reed, lips, etc. A musical sound wave when analyzed by the method

of Fourier is found to be a combination of many partial tones. In most cases these

partials are integral multiples of the lowest or fundamental tone, in which case they

are called harmonics. The generic term for upper partials is overtone, of which the

harmonic is a special case. In a few instruments some of the upper partials are

definitely not harmonics, although they are still called overtones. While the pitch of a

complex tone is principally determined by its fundamental, it is the upper partials or

overtones that practically produce tone quality.

Each instrument has a timbre of its own. Surbahar produces a very deep and resonant

sound full of gravity. Sarod has a very sparkling, liquid and rounded sound. Santoor’s

sound has a light, quick, silvery quality, like a water spring, Flute can acquire a

deeply haunting sound, sound of Shehnai has a bright pealing clarity.

3.1.3 Rhythm & Speed [24] [25] If frequency and related concepts like tone, scale and octaves form an important

ingredient in music; the other equally important element is time and related items like

speed, rhythm, meter etc. In fact, a musical piece is nothing but a source of sound

emitting sound waves as a function of time.

The first concept is 'speed'. Any song has a prescribed speed and if sung faster or

slower, may sound funny. The Western music and Hindustani music recognize

various degrees of speed or tempo (laya), all the way from very, very slow (vilambit)

to ultra fast (drut). The other concept of rhythm is probably the most fundamental

aspect of music. It basically is a repetitive sound pattern.

18

In Indian Classical Music Tala is the science and system of punctuating music with

beats. The Tala organizes rhythm into a structured recurring pattern. The beats of the

Tala do not proceed linearly but in closed loops or cycles of six, eight, ten, twelve,

fourteen, sixteen or at times even larger numbers of beats. There are also cycles

having an odd number of beats. Among these beats, some are stressed and some

unstressed. These various cycles are also denoted as different Talas, or Tals, as for

example, Ektal, Jhaptal, Tintal, Choutal, Adachoutal etc.

3.2 Visual Dimensions: Color, Form and Motion

3.2.1 Color Color is among the strongest visual stimuli that our brains receive from the outside

world. Color is complex. There are quite a few color models but here the HSV model

is used to understand it as this is the model closest to perception.

Hue is the principal way in which one color is distinguished from another. Describing

and managing hues is generally taken to be the central problem for color theory.

Indeed the very language we use to denote colors is associated primarily with their

hues. A hue can be referenced by its angle around a color wheel, for example, red at

0, yellow at 60 degrees, green at 120, blue at 240, and purple at 300. In a well−

balanced color wheel, complementary colors appear at 180 degrees opposite.

Numerous color wheels have been defined. They all share the objective of making the

relationships among hues more accessible. Because hue is a continuous space, naming

and distinguishing among hues is somewhat arbitrary. Goethe and Schopenhauer

spoke of six distinct hues, Ostwald of eight, Munsell of ten.

Saturation describes how pure a particular hue is. It is also referred to as the intensity,

strength, or chroma of a color. Reducing the saturation of a particular hue, while

maintaining its value, has the effect of adding white pigment, producing what artists

call tints.

Value is the quality that differentiates a light color from a dark one. It is also referred

to as lightness. A particular color moves toward black by a reduction in its value.

Low−valued colors are less visible than ones with higher values are. Decreasing value

while leaving saturation alone has the effect of adding black pigment, producing what

are referred to as different shades. Finally, what artists refer to as tones, can be created

19

by decreasing both saturation and value. One of the reasons that the HSV color model

is so useful is that there is a substantial literature that uses these concepts – hue, tint,

shade and tone – to describe art history and technique [26].

Color has been associated with a number of things: planets, gods, emotions, sound,

moods, seasons, major faculties of a university and so on. The colors of the aura

would mean as follows: Red is the physical phase of mentality, indication health,

vigor, friendship and love. Man’s baser qualities show themselves in deep shades of

red. Yellow is the intellectual phase of being and golden yellow it’ s highest form.

Blue is the religious or spiritual phase whose higher form is a violet tint and whose

lower form is an indigo shade. Orange is the union of mind and body, a sign of sound

wisdom and justice. Green marks the lover of nature and is indicative of sympathy,

altruism, and charity. Slate green is a telling symbol of jealousy and deceit. Violet

exposes a love of form and ceremony. Here is the union of spirit and body (blue and

red), a symbol of great idealism and sublimity [27].

In Indian Arts also color is associated with sounds, rasa etc. (See section 2.)

3.2.2 Form It may be said that form or structure is basic to art and aesthetic experiences of all

kinds. Here the visual form perception will be discussed.

Gestalt psychology has set forth the principles of form perception for the visual and

other arts in a very clear and significant way. The first principle is that of ‘ figure and

ground’ by which every perceptual experience is essentially a pattern related to a

background of other experiences or their absence (Fig.3.1).

Figure 3−1 Figure and Ground

The second principle of Gestalt psychology is that of ‘differentiation’ or

‘ segregation’ , by which the patterns of stimulation organize characteristic structures

for perception, owing to their special properties. Thus if a ring−shaped or cross−shaped

This may be seen either as a vase or as two faces. It shows alternation of ‘ figure’ and ‘ground’

20

pattern of large dots is placed among a number of small dots, or if a group of thick

lines, or of lines or dots of a certain darkness or color, is placed among an otherwise

uniform group of dots or lines, which are not distinguished by thickness, size, color or

darkness, the ring−shaped or cross−shaped dots or lines will cohere and stand out as a

segregated or differentiated figure on the uniform background (Fig.3.2).

Figure 3−2 ‘Differentiation ’ or ‘Segregation’

The third principle is that of closure, by which an incompletely represented outlines or

structure appears to be complete or tends to be completed in perception (Fig.3.3).

Figure 3−3 Closure

The fourth principle is that of the ‘good Gestalt’ . According to this principle a

stronger or more adequate pattern in perception will tend to take precedence over

weaker patterns. Very often good Gestalten are simple structures, like circular, square

or rectangular patterns. While some are apparently based on simple geometrical

organizations, others are based on familiarity, or the power of emotional expression.

Many clever illusions of perception are based on giving two or more configurations

equal weight in representation, and then alternating figures may be produced. Any

influence, such as conscious attitude or emotional response, will tend to bring out one

figure rather than the other (Fig.3.1).

Simple lines and shapes may create the form or structure. Various psychological

experiments conducted show some interesting results. In general ‘beautiful lines’

showed unity in direction and movement, continuity, absence of angles and

In this four kinds of stimuli are combined, and a rectangle of columns of dots and circles is perceived which forms the ‘ground’ for an upright cross of large dots and a diagonal cross of small crosses.

The top figure shows that a simple change in the stimulus pattern makes the IO−sided outline the principle structure perceived, but ‘closure’ still makes the diamond and rectangle apparent. The bottom figure illustrates ‘closure’ , by which incomplete shapes tend to be perceived as if complete. It looks like a triangle with part of one side missing.

21

intersections, and a periodical return of the same elements or certain symmetry.

‘Ugly’ lines were full of discontinuities, irregular changes of direction, and angles.

Horizontal lines were called ‘quiet’ , downward−sloping curves were ‘sad’ or ‘gentle’ ;

rising lines were ‘merry’ or ‘agitating’ ; and downward lines were ‘sad’ , ‘weak, or

‘ lazy’ . Other basic principles of composition depend on balance of structural parts, so

that the organization of a picture is divided, vertically or horizontally, or both

together, in terms of golden section (), or otherwise. The balance may involve large

objects in the foreground or near distance, against more distant objects [28].

All Indian Arts use the corresponding medium in such a way that certain geometrical

motifs emerge. The geometrical figures have symbolic significance. In the art of

dance, the ordinary vertical stance of man is equivalent to a straight line with the

shortest distance and least movement from one point to the other. It divides space but

does not circumscribe it: thus it is both controlled and rigid. Brahma, the deity of

totality represents this line, which can by itself be limitless. Much later in Tantric

symbolism it represents direct perception of pure consciousness. The simple erect

standing posture can assume cosmic proportions. The triangle, the first of the

rectilinear figures, is the first to define dimension and contains in itself the strongest

inner cohesion. Each arm of the triangle is connected with the others, while one is in

opposition to the other, the two are complementary. The equibalanced tension and

their equidistant position is joined together by the base. The space contains

unassailable unity in plurality. In Vedic thought the equilateral triangle standing on its

base and dominated by the vortex represents Purusa: it also represents fire. The

triangle also represents Vishnu, the one energy who permeates into the universe with

his three strides. It is seen as the manifest body or form. In kinetic terms, this is the

moment of movement, created through the tension of the limbs to represent a series of

triangles. Much later in Tantrism the triangle standing on its apex represents the

power of manifestation [29].

3.2.3 Motion Motion in space as well as time makes the visuals dynamic. It is a very important

dimension but much less studied compared to color and form. The sequences of color

and form in succession impart temporality to the visual art.

There is one area i.e. dance where the laws of movement are well studied and has a

well−defined grammar of motion. (Austrian−) Hungarian Rudolf von Laban (1879−

22

1958) an important figure in European modern dance is best known for his work with

movement in the field of dance. Laban juxtaposed space, time and force with two

polarities: space was direct or indirect, time was fast or slow, and force was light or

firm. Together with something he called "effort/shape," Laban developed a precise

and elegant system of movement and dance notation, called Labanotation.

Space Time Force Action word

Direct Slow Light Glide

Indirect Slow Light Float

Indirect Fast Light Flick

Direct Fast Light Dab

Direct Fast Firm Stab/Punch

Indirect Fast Firm Slash

Indirect Slow Firm Wring

Direct Slow Firm Pull/Push

Table 3−2 Action Words Resultant from Different Combinations of Space, Time

and Force

There are different combinations of space, time, and force, resulting in countless

movement qualities or action words (Table 3.2). Each action word has its own

corresponding internal feeling and thinking state. For example, moving in a "floating"

way (with indirect, slow and light movement), produces a relaxed, peaceful state, in

which one would behave gently, is less rigid and which may even induce a little

spaceyness. "Wringing" movement (indirect, slow and firm movement) might

encourage your brow to contort in an accompanying awareness of angst. "Stabbing"

or "punching" motions (direct, fast and firm movements) accompanies an aggressive

or angry internal state [30].

In Indian Arts, laws of ‘movement’ in space and time are very much laid. Bharata’s

Natyasashtra describes various gaits of characters for evoking different Rasas; easy

and graceful for erotic, quick and vigorous for heroic, slow for compassion and so on

[31]. The movement on the Sanskrit theatre is not free individual subjective

expression but it is the near transmutation of the human body into geometrical figures

23

capable of psychical evocation. The body assumes postures, all contained within the

space of a circle (with navel as a center and the spinal cord as the vertical median) or

a square. Bharata explores all movement from this position, whether it is the smaller

movements of the face or the movements on a larger scale of the head, torso, hips and

legs. All dance forms are a series of shapes in space; Bharatanatyam is a series of

triangles, Kathakali a square, Manipuri a spiral or an intertwined serpent and Kathak

an axis [30].

3.3 The Audio –Visual Analogy

Before the audio–visual analogy is stated it is important to understand the issues

involved in moving from one sensory domain to the other. Sound by ear and vision by

eye respectively, are not perceived in the same way. Ear is far superior to the eye as

an analyst. The ear has a fabulous power of breaking down and analyzing the

vibrations, which constitute sound, indeed its power with regard to such break−down

analysis is far superior to the most modern sophisticated electronic acoustical

scientific devices. Thus two or more tones heard together remain distinguishable, at

least to the trained ear, in spite of their integration to form a distinctive chord,

whereas two or more beams of light rays will fuse to form an intermediate color

indistinguishable in any way from a single beam of light of that color. This inability

of eye to break down the color into its constituents is also due to the fact that different

combinations of different colors of light can produce the same mental color sensation.

Consequently color harmony always presumes positional relationships in space,

although succession may be obtained by changes in temporal sequences as in music.

A further difficulty is that while two of the fundamental qualities of tones and colors

are comparable, namely pitch with hue and loudness with intensity, timbre for sounds

and saturation for colors are not comparable on physical basis [28]. Another issue is

of form, on what basis, physical or philosophical to compare form and sound.

Although research in cymatics (a branch that studies the interrelationship of

waveforms and matter) has shown that certain sounds produce certain patterns and in

tantric philosophy also the form and sounds are correlated, it still is difficult to map

the form with musical parameters. The same applies to the dimension of motion.

Besides movement from aural to visual domain it also is a movement from a temporal

domain to spatial−temporal domain. Musical notes are separated in time, and as one

24

note follows the other, both the notes leave a distinct impression on the listener.

However, consider the case when these notes are mapped to a given shape and a given

color, as an example, note "Sa" is represented as a red circle, and a "Re" follows it by

a green square. The issue in such a case is, when Sa is followed by Re what is the

impression of their corresponding visuals on the viewer? Is the red−circle distinctly

noticed, and then replaced by the green−square? Is there a moment of transition

between these visuals where the space remains a mixture of the previous and the

ensuing visual? At the end of music piece is there a complete picture of that piece or

everything vanishes with the end of the piece? The solution to these issues may drag

us into the fields of persistence of vision, theory of relativity, and principles of matter.

It can be seen that the issues at hand are not trivial and cannot be handled in one go.

Taking this fact into consideration a simple computational model for visualization of

Indian Classical Music is proposed with the use of the following audio−visual

analogy. The analogy is such that the proposed model can capture all the dimensions

and handle all the issues but it also allows handling one dimension at a time. The

model can evolve with time, with more experiments and with more understanding.

Indian Classical Music is melodic in nature compared to the harmonic nature of

Western Classical Music. Indian Music is many times compared with a flowing river.

Melodic nature would mean one note at a time. This movement of note in time

domain can be captured visually as a movement of brush on a canvas. The brush

moves with time on the canvas (i.e. space) depending on the change of note. There are

four properties of the brush.

• Kind of the brush

• Color of the brush

• Movement of the brush

• Size of the brush

This leads to a domain and image set of a mathematical function (Fig.3.4).

25

Aural Mapping Visual

Domain black box Image Set

Figure 3−4 Audio−Visual Functional Relationship

Finding the Mapping is the key to Visualization. The computational model proposed

in this thesis use the following functional dependencies.

• Brush = FBRUSH (Instrument)

The brushes could be of various types. The kind of brush to use will depend on

the instrument that is being played. There are instruments that seem to fill the

space; there are instruments that have piercing voice; there are instruments

used for rhythm; drone instruments; each instrument having its own timbre.

Depending on the timbre the brush could fill the background, produce a curve,

expand and contract in a circle, etc.

Each brush will have its own color and movement scheme.

• Brush−Color = FCOLOR (Note, Intensity, Brush)

Depending on the color−scheme of the Brush, color will be calculated using

Note and Intensity.

• Brush−Movement = FMOVEMENT (Time, Note, Brush)

The Brush−Movement at time Time will be calculated depending on the

movement−scheme of Brush using the Note.

• Brush−Size = FSIZE (Intensity, Brush)

The Brush−Size (for e.g. it could be the thickness of the curve) of Brush will

vary depending on the Intensity of the Note.

Note

Instrument

Intensity

Brush Brush−Color Brush−Movement Brush−Size

26

The definitions of these functions have to be based on the following:

Aesthetic principles of Indian Art: All Indian arts are strongly interrelated and held

together by certain fundamental principles [29].

Experimentation and feedback: After defining the functions, a series of repeated

experiments on a diverse array of music pieces yield a better insight onto the feel and

intuitiveness of these functions and mappings, and based on this feedback, changes

can be adapted.

27

Chapter 4

Proposed Design and Implementation

4.1 Proposed Design

Based on the computation model described in Section 3.3 the following Visualization

System has been designed (Fig.4.1).

Figure 4−1 Block Diagram of the Visualization System

The design is modular and each module can be developed and modified individually.

• AudioPlayer: The function of the AudioPlayer is to play the input audio file

and send parameters like notes, their corresponding intensity and the

instrument.

• Analyzer: The Analyzer analyses the input audio file. It also provides the

VisualsGenerator with additional information like the raga, style, etc.

• ControlPanel: This is the tool that allows user interaction with the core

program and has controls for the selection of the brush, color−scheme and

movement scheme for each instrument.

28

• VisualsGenerator: The VisualsGenerator consists of the various brushes,

color−scheme and movement−scheme. It takes input from all the three

modules: AudioPlayer, ControlPanel and Analyzer. Depending on the input it

then generates Visuals.

• PlaybackMonitor: The playback monitor is a canvas on which the visuals are

created.

4.2 Implementation

To implement the above−mentioned design, there are two major issues.

1. Extraction of the musical parameters from an audio file

Any audio signal needs to be analyzed and the musical parameters: note, instrument

and intensity extracted.

2. Generation of the visuals

The graphical output required needs to be realized using some software.

There are various formats like .au, .wav, .mp3, .mid, etc. for the audio files and

standards like OpenGL for computer graphics. The following were selected for the

purpose of implementing the Visualization System described in Section 4.1.

• Audio input file format – MIDI (Musical Instrument Digital Interface) file format

Any other audio file format contains the sampled audio data. To extract the musical

parameters from any of these formats would essentially mean analyzing the audio

signal using signal−processing techniques. Till date, there is no standard algorithm for

extracting all the three parameters of pitch, amplitude (would mean intensity) and

timbre simultaneously from a musical signal.

MIDI file contains only the instructions needed by a synthesizer to play the sounds.

These instructions are in the form of MIDI messages, which instruct the synthesizer

which sounds to use, which notes to play, and how loud to play each note. The

synthesizer then generates the actual sounds. The synthesizer could be a real physical

one or it could be a software program on a computer. More about MIDI messages is

given in Appendix III. The detailed information can be obtained from

http://www.midi.org.

• The language – JAVA

Java is a higher−level language that provides ease of programming and platform−

independent implementation. Moreover, it provides a very good API javax.sound.midi

29

for dealing with MIDI related operations. There also is a Graphics2D class to provide

more sophisticated control over geometry, coordinate transformations, color

management, and text layout. The detailed information on JAVA can be obtained

from http://java.sun.com

Using JAVA and MIDI the modules were implemented. Implementation of each of

the modules will be described briefly.

1. AudioPlayer: The AudioPlayer was implemented using the javax.sound.midi API.

The Sequencer interface was used for playing the audio. With classes Sequence,

MidiEvent and MidiMessage the information of note, intensity and instrument

was sent to VisualsGenerator at every fixed interval.

2. ControlPanel: javax.swing API was used to make the ControlPanel. Following is

the snapshot of the ControlPanel (Fig.4.2).

Figure 4−2 Snapshot of ControlPanel

MIDI supports sixteen channels and each channel can play a different instrument.

Thus at a given instant sixteen instruments can play simultaneously. Each channel can

be configured separately. Each channel will have its own brush, its own color−scheme

and movement−scheme.

30

3. VisualsGenerator: Graphics and its extension Graphics2D class of java.awt were

used to generate the visuals. The java.awt.Color class realized the color

generation. VisualsGenerator is essentially a collection of brushes, color−schemes

and movement−schemes. Depending on the note, intensity, brush, color−scheme

and movement−scheme supplied by the AudioPlayer as well as ControlPanel the

VisualsGenerator will create graphics. Now, the various color−schemes,

movement−schemes and brushes that were implemented will be described.

Color−schemes:

1. Sruti – Saptak is divided into 22 microtones according to the ratios given in

Table 3.1. The same ratios are used to divide the 360 degrees of hue. This

gives following hues for the seven pure notes (Table 4.1).

Note Hue(H in degrees)

Sa 0 (red)

Re 45 (ochre yellow)

Ga 90 (parrot green)

Ma 120 (green)

Pa 180 (light blue)

Dha 240 (dark blue)

Ni 315 (magenta)

Table 4−1 Sruti Color−Scheme

Higher the saptak more saturated the color. Base saturation SB is fixed for the

Madhya saptak, it decreases and increases as sound moves to mandra and tar

saptak respectively. The following equation is used to calculate the saturation

of the color.

S = SB + Spk*SConst

Where S = Saturation

SB = Base saturation

Spk = saptak no. (Madhya saptak is 1, mandra saptak is –1)

SConst = increment in saturation with increase of one saptak

31

Lightness of the color varies with the intensity or loudness of the sound. The

following equation calculates the lightness of the color.

V = A*V Const

Where V = Lightness

A = amplitude or loudness of the sound

VConst = increment in intensity with every increment in loudness

2. Equi−tempered – The saturation S and lightness V is calculated in the same

way as for the Sruti scheme. The hue H is calculated by dividing the 360

degrees into twelve equal parts and is assigned to notes in the following way

(Table 4.2).

Note Hue (H in degrees)

Sa 0 (red)

re(komal) 30

Re 60 (yellow)

ga (komal) 90

Ga 120 (green)

Ma 150 (light green)

MA (tivra) 180

Pa 210 (dark blue)

dha (komal) 240

Dha 270 (light blue)

ni (komal) 300

Ni 330 (dark pink)

Table 4−2 Equi−tempered Color−Scheme

3. One−One – This scheme allows the user total control over the color mapping.

Every note can be assigned a color individually.

Movement−schemes:

1. Linear – This scheme calculates the position of the brush on the canvas using

following equations:

32

X = Time * XConst

Y = Note * YConst

Where X = x co−ordinate of the brush

Y = y co−ordinate of the brush

Time = time elapsed from the start of the music piece

Note = the note being played at time Time

XConst = the increment set by the user in x direction

YConst = the increment set by the user in y direction

2. Spiral – The brush moves in a spiral path using the following equations:

θ = θConst * Time

X = a * cos(θ) * e b*θ

Y = a * sin(θ) * e b*θ

Where X = x co−ordinate of the brush

Y = y co−ordinate of the brush

Time = time elapsed from the start of the music piece

θConst = the increment in θ set by the user

a,b are constants

3. Circular – The circular path is calculated as follows:

θ = θConst * Time

X = a * cos(θ)

Y = a * sin(θ)

Where X = x co−ordinate of the brush

Y = y co−ordinate of the brush

Time = time elapsed from the start of the music piece

θConst = the increment in θ set by the user

a is a constant

Brushes:

1. BrushCurve – Depending on the color and movement scheme the brush will

draw a B−spline curve while moving from one point to the other (Fig.4.3).

33

Figure 4−3 BrushCurve

2. BrushBackground – This brush fills the background with the color selected by

the color−scheme (Fig.4.4).

Figure 4−4 BrushBackground

3. BrushSpiral – Each note is assigned a position on a spiral that has the

following equation:

θ = θConst * Note

X = a * cos(θ) * e b*θ

Y = a * sin(θ) * e b*θ

The spiral involves or evolves to the position of the note that is being played at

that instant (Fig.4.5).

Figure 4−5 BrushSpiral

34

4. BrushFlute – A particular shape (Fig.4.6) that was created based on the

overtones of flute is drawn at the position calculated by the movement−

scheme. The color is based on the color−scheme.

Figure 4−6 BrushFlute

5. BrushLetter – The brush writes the letter corresponding to the note at the

position calculated by movement−scheme with the color selected by color−

scheme (Fig.4.7).

Figure 4−7 BrushLetter

4. PlaybackMonitor: The PlaybackMonitor was implemented using the Canvas

class of java.awt API.

4.3 Results

4.3.1 Raga Asavari and Todi Let us look at the features of the two ragas, Asavari and Todi (Table 4.3).

35

Feature Asavari Todi

Aroha Sa Re Ma Pa ni dha ni dha Sa Sa re ga MA Pa dha Ni Sa

Avaroha Sa ni dha Pa, Ma Pa dha Ma Pa Ma

ga Ma ga Re Sa

Sa Ni dha Pa MA ga re Sa

Vadi dha dha

Samvadi ga ga

Time Mid morning Early morning

Rasa Moods depicting yearning for love,

anguish and melancholy.

Devotion

Pakad Re Ma Pa, ni dha Pa Dha Ni Sa re ga re re Sa MA re ga re Sa

Table 4−3 Features of Asavari and Todi

Both the ragas have same vaadi: dha and samvaadi: ga but yet the rasa they create are

different. The rasas associated with note dha are Shanta and Karuna and that with note

ga is Veera. The aroha and avaroha of both the ragas can be seen visually in Fig.4.8

Figure 4−8 Aroha and Avaroha of Asavari and Todi

The upper figure shows the aroha−avaroha of raga Asavari that has a lot of ups and downs and is asymmetric The bottom figure shows the aroha−avaroha of raga Todi, which has symmetry and easy movement.

36

As seen in the figure raga Todi has continuity, absence of angles and has symmetry

while the Asavari raga shows irregular changes in direction and asymmetry. Thus the

movement of Asavari gives it the mood of anguish while a different movement makes

Todi a devotional raga.

4.3.2 Jhaptaal

Figure 4−9 Jhaptaal

The figure above (Fig.4.9) shows the visual generated when the midi file of the tala

Jhaptaal is played using the circular movement of the brush. The figure is indicative

of the cyclic repetitive movement of the tala.

4.3.3 Jugalbandi of Sitar and Tabla

Figure 4−10 Sitar and Tabla

The BrushSpiral is used for the sitar and the BrushCurve shows the beats of Tabla.

37

4.3.4 Flute and Harmonium The harmonium is being played in the background and flute in the foreground.

BrushFlute shows the movement of flute against the changing colors of background

(Fig.4.11).

Figure 4−11 Harmonium and Flute

4.3.5 Triads of Western Classical Music The figure very distinctly shows the harmonic nature of Western Music unlike the

flowing melody of Indian Classical Music.

Figure 4−12 Triads of Western Music

38

Chapter 5

Conclusion and Recommendations for Future

Work

The process of Visualization of Music is a continuous effort that is difficult to

quantify. The results of this work have encouraged further experimentation and

discovery of this art, using computer as a medium. The following contributions were

made while certain limitations were encountered.

5.1 Technical summary

5.1.1 Contributions

• A computational model for Visualization of Indian Classical Music was proposed

and implemented. Although there are precedents of such work in Western Music,

(as detailed in Chapter 1, Introduction), this is a pioneering work in this area, as

per our knowledge.

• The implementation of the model is also unique, as due to the use of Java as our

language of choice, the model can be ported onto any platform, and hence, is able

to address a larger audience by virtue of being platform−independent.

• The literature on Indian Classical Arts relevant to the process of visualization was

explored and studied in some detail. The intricacies involved were realized, and

kept in mind while designing the mapping algorithm.

• The results are promising and helps one understand the theory behind Indian

Classical Music.

• With this thesis work, there is a hope that the model or the idea will act as

groundwork for the further development of a sophisticated visualization system

for Indian Classical Music.

39

5.1.2 Limitations

• MIDI does not support the representation of the subtleties of Indian Classical

Music. As MIDI supports only twelve notes it cannot represent Srutis. Even the

embellishments cannot be represented in this file format.

• Synthesizing an Indian instrument is very difficult because of its capability to

display music intonation, or the theoretically infinity pitches which it generates.

Not very many Indian Instruments are available for the use in MIDI format. As

the MIDI format supports only 12 notes, the lilt of Indian classical music has not

been tapped due to the limitation of this format.

• The music works that would capture the features of Indian Classical Music were

not available in the MIDI file format. Although some work was done to convert

Indian music into MIDI format, most of it remains confined to private collections

or personal computers of the creators, and is not available via any of the standard

resources like music libraries and the Internet.

• It was possible only to work with Instrumental Music and the visualization of

vocals was left for further work.

The limitations have to be faced with following suggestions for the continuation of

the work.

5.2 Suggestions for further work

• Creating or obtaining MIDI files for Indian Classical Music. For creating the

files a setup (Appendix II) will be required.

• A more detailed study of the principles of Indian Aesthetics, with stress on the

interleaving of audio−visual elements.

• Exploration of Tantric texts for finding out the association of sound and form

• The implementation of the proposed model worked only at the lowest level:

the parameters being pitch, timbre and loudness. The following parameters

should be considered for the next level – the higher level of abstractions:

o Embellishments – The Srutis, and the intermediate frequencies that

they are made of.

o Raga – The combination of different swaras, are the pillars of Indian

music. These Ragas make up the basis for Indian music of all kinds,

40

and each Raga is designed for a particular mood, or Rasa, and also the

season, time of day etc.

o Combination and interaction of multiple instruments – An orchestra

consisting of modest number of instruments would generate a

symphony, which would have an element of mixture of sounds and

hence visuals, which would need further exploration.

o The style or form – Indian classical music has different styles of

rendering a raga like Dhrupad, Thumri, Tappa, etc.

• Conducting psychological experiments to justify/support the visualization

model. A very immediate extension of this work can be surveying a

representative set of individuals, who would provide feedback on whether the

model accurately represents their visualization of a musical piece.

• Computers entered Western Music long before. A lot of work is being done

for the Visualization of music in the West. But the field is nascent for Indian

Classical Music. A comparative study of our work with the parallels in West

should be taken up.

41

References

[1] National Tele−Immersion Initiative, http://www.advanced.org/teleimmersion.html

[2] Essays in Indian Ethnomusicology, 19. Ragamala−paintings: A Musicology

Perspective, BHU Central library

[3] Sandy Cohen's Bindu, http://www.slip.net/~bindu/index.htm

[4] Fred Collopy, Robert M. Fuhrer, David Jameson, "Visual Music in a Visual

Programming Language", In 1999 IEEE Symposium on Visual Languages,

Tokyo, Japan, pp.111−118

[5] Wearable Computing Meets Ubiquitous computing,

http://rhodes.www.media.mit.edu/people/rhodes/Papers/wearhive.html

[6] The Talking Tactile Tablet, http://www.touchgraphics.com/ttt.htm

[7] Ubiquitous Computing, http://www.ubiq.com/hypertext/weiser/UbiHome.html

[8] B. Chaintanya Deva, Indian Music, Indian Council for Cultural Relations, 1974

[9] Sketchpad: The First Interactive Computer Graphics Ph.D. Thesis, 1963, MIT,

http://www.sun.com/960710/feature3/sketchpad.html

[10] Fred Collopy's Imager, http://RhythmicLight.com/studios/index.html

[11] Scott Drave's Bomb, http://draves.org/bomb/

[12] Larry Cuba's homepage, http://www.well.com/user/cuba/

[13] Ron Pellegrino’s homepage, http://www.microweb.com/ronpell/

[14] Jeff Hoekman's homepage, http://cm.stanford.edu/~jhoekman/welcome.html

[15] Greg Jalbert, http://www.imaja.com/Artwork.html

[16] Stephen Malinowski's music animation machine,

http://www.well.com/user/smalin/mam.html

[17] Andrew Schloss's homepage, http://www.finearts.uvic.ca/~aschloss/

[18] David Tristram's homepage, http://www.tristram.com/

[19] B. Chaintanya Deva, An Introduction to Indian Music, Publications Division,

Ministry of Information and Broadcasting, Govt. of India, Feb 1973, Ch−8 Mind

and Music, pp.66−68

[20] Maharana Kumbha, Sangitaraja − Vol I, BHU press, 1433−68

[21] http://www.bampfa.berkely.edu/exhibits/indian/u0500.html

42

[22] Ragamala Paintings in Harivallabha’s Illustrated Manuscript,

http://www.princeton.edu/~simundza/diss/ragamala.html

[23] Jess J. Josephs, The Physics of Musical Sound, Van Nostrand Company, INC.

[24] A Gentle Introduction to South Indian Classical Music by Mahadevan Ramesh,

http://www.aoe.vt.edu/~boppe/MUSIc/PRIMES/icmb.html

[25] Sir Aurobindo society, Pondicherry, Alaap−A Discovery of Indian Classical

Music, Times Music

[26] Fred Collopy, Color, Form, and Motion: Dimensions of a Musical Art of Light,

LEONARDO, Vol.33, No.5, pp.355−360, 2000

[27] Fabel Birren, Color: A Survey in Words and Pictures, University Books, INC

[28] R.W. Pickford, Psychology and Visual Aesthetics, Hutchinson educational

[29] Kapila Vatsyayan, The Square and the Circle of the Indian Arts, Roli books

international, 1983

[30] EnneaMotion: The Somatic Enneagram,

http://ideodynamic.com/enneamotion/Articles/Somatic.htm

[31] Adya Rangacharya, The Natyasastra (The English translation of the Bharat's

Natyasastra of 7−8th century), Munshiram Manoharlal publishers

43

Appendix I − Glossary

A Adachoutal: An Indian Classical Tala with 4+2+2 units.

Adhbhuta: One of the Rasas meaning Wondrous.

Alambana: The people in whom emotion is stimulated

Alap: Free extemporizations of raga.

Anubhava: Expression.

Aroha: The ascending pattern of a raga.

Asavari: An Indian Classical Raga with Sa Re Ma Pa dha in the ascent; and Sa ni dha

Pa Ma ga Re in the descent.

Avaroha: The descending pattern of a raga.

B Beebhatsa: One of the Rasas meaning Disgusting.

Bhairav: An Indian Classical Raga using Sa re Ga Ma Pa dha Ni.

Bharatanatyam: One form of Indian Classical Dance.

Bhava: Emotion.

Bhayanaka: One of the Rasas meaning Fearful.

C Choutal: An Indian Classical Tala

D Dha: Mnemonic for the note Dhaivat.

Desi raga: The regional or parochial raga which may not strictly adhere to rules.

Dhaivat: The sixth note of the seven pure notes of Indian Classical Music.

Dhrupad: A type of composition in Indian Classical music with emphasis on rhythm,

and alap preceding the song.

44

Drut : Fast tempo.

dha: Mnemonic for the note komal Dhaivat.

E Ektaal: An Indian Classical Tala with 4+4+2+2 units

G Gamaka: A tonal ornament; a grace: glide, trill, swing, etc.

Ga: Mnemonic for the note Gandhar.

Gandhar: The third of the seven pure notes of Indian Classical Music.

ga: Mnemonic for the note komal Gandhar.

H Harmonium: An Indian musical instrument. A keyboard instrument where the right

hand presses the keys and thus selects the particular reeds, the left hand moves a

bellow to blow air through them.

Hasya: One of the Rasas meaning Humorous.

J Jhaptaal: An Indian Classical Tala with 2+3+2+3 units.

K Kafi : An Indian Classical Raga with Sa Re ga Ma Pa Dha ni

Karuna : One of the Rasas meaning Compassion or Pathos.

Kathak: A form of Indian Classical dance.

Kathakali : A form of Indian Classical dance.

Khamaj : An Indian Classical Raga with Sa Ga Ma Pa Dha Ni in the ascent and Sa ni

Dha Pa Ma Ga Re in the descent.

Komal (swara): Flat (note).

45

L Laya: Tempo.

M MA : Mnemonic for the note tivra Madhyama.

Ma: Mnemonic for the note Madhyama.

Madhya (saptak): Middle (octave).

Madhyama: The fourth of the seven pure notes of Indian Classical Music.

Mandra (saptak): Lower (octave).

Manipuri : A form of Indian Classical Dance.

Misra mand: An Indian Classical Raga.

N Ni: Mnemonic for the note Nishad.

Nishad: The seventh note of the seven pure notes of Indian Classical Music.

ni: Mnemonic for the note komal Nishad.

P Pa: Mnemonic for the note Pancham.

Pahadi: An Indian Classical Ragini.

Pakad: The characteristic phrase of a raga.

Pancham: The fifth of the seven pure notes of Indian Classical Music.

Pooriya Dhanashri: An Indian Classical Raga.

R Raga: A combination of notes, ascending and descending according to certain rules,

and with a characteristic mood. The western equivalent would be the scale.

Ragesri: An Indian Classical Raga.

Ragini: The raga with feminine qualities is known as ragini.

Rasa: Aesthetic emotions.

46

Raudra: One of the Rasas meaning Terror

Re: Mnemonic for the note Rishabha.

Rishabha: The second of the seven pure notes of Indian Classical Music.

re: Mnemonic for the note komal Rishabha.

S Sa: Mnemonic for the note Shadja

Samvadi: The second most important note of a raga.

Sanchari bhava: Transient emotion.

Santa: One of the Rasas meaning Peaceful

Santoor: An Indian musical instrument where strings are not plucked but struck with

two small wooden mallets.

Saptak: Octave.

Sarod: An Indian musical instrument. It is a string instrument.

Satva bhava: Reflexive expressions.

Shadja: The first of the seven pure notes of Indian Classical Music.

Shehnai: An Indian musical instrument. It is a wind instrument similar to flute.

Sitar: An Indian musical instrument. It has a long slim wooden stem fixed to a hollow

gourd as air chamber. The cover of the air chamber is a thin sheet of wood over which

the live bridge sits. Frets on the stem constitute the fingerboard and the strings are

pressed down by the left hand fingers to produce different notes. The plucking is done

with a wire plectrum fixed on the tip of the right forefinger.

Shuddha (swara): Pure or natural (note).

Sringar: One of the Rasas meaning Erotic

Sruti : A microtonal interval.

Sthayi bhava: Foundational emotion.

Surbahar: An Indian musical instrument. This is a sort of bigger Sitar, but lower in

pitch.

Swara: ‘note’ ; the mental quality of a tonal interval.

T Tabla: An Indian percussion instrument consisting of two pieces.

47

Tala: A cyclic arrangement of time units, with defined sections.

Tappa: A compositional form with very quick phrases.

Tar (saptak): The upper (octave).

Thumri : A lyrical form of singing.

Tintaal : An Indian Classical Tala with 4+4+4+4 units

Tivra (swara): Sharp (note)

Todi: An Indian Classical Raga with Sa re ga ma Pa dha Ni.

U Uddeepana: The excitant determinant.

V Vadi: The most important note (center of gravity) of a raga; often translated as

‘dominant’

Varna: A musical phrase.

Veera: One of the rasas meaning Heroic.

Vibhava: Stimuli.

Vikrit : Variant−flat or sharp−note

Vilambit (laya): Slow (translated as delayed) (tempo);

48

Appendix II – The Setup Diagram

49

Appendix III – MIDI

The Musical Instrument Digital Interface (MIDI) enables people to use multimedia computers and electronic musical instruments to create, enjoy and learn about music. There are major three components to MIDI

1. Communications Protocol (language)

2. The Connector (hardware interface)

3. A distribution format called Standard MIDI Files.

Protocol

The MIDI protocol is an entire music description language in binary form. Each word describing an action of musical performance is assigned a specific binary code. MIDI was designed for keyboards, so many of the actions are percussion oriented. To sound a note in MIDI language you send a "Note On" message, and then assign that note a "velocity", which determines how loud it plays. Other MIDI messages include selecting which instrument to play, mixing and panning sounds, and controlling various aspects of electronic musical instruments. The table of all the messages can be found at http://www.midi.org/midi−about/resources.htm

Standard MIDI Files

When MIDI messages are stored on disks, they are commonly saved in the Standard MIDI file format, which is slightly different from native MIDI protocol, because the events are also time−stamped for playback in the proper sequence. Unlike digital audio files (.wav, .aiff, etc.) or even compact discs or cassettes, a MIDI file does not need to capture and store actual sounds. Instead, the MIDI file can be just a list of events, which describe the specific steps that a soundcard or other playback device must take to generate ceratin sounds. This way, MIDI files are very much smaller than digital audio files, and the events are also editable, allowing the music to be rearranged, edited, even composed interactively, if desired.

With the recent introduction of the Downloadable Sounds format, MIDI files can also contain standardized samples of musical instruments, sound effects, or even dialogue, which are used to recreate an exact copy of the sound intended by the composer. MIDI files with DLS are the ideal solution for composers of all kinds who want the predictable playback of digital audio, but also need the compactness and/or interactivity of Standard MIDI Files for delivering their music.

50

Connector

According to the MIDI 1.0 Specification, the only approved MIDI connector is a 5−pin DIN connector. It is certainly possible to send MIDI messages using other connectors and cables. Meanwhile, since many personal computers do not have space for a 5−pin DIN connector, many manufacturers have decided to use either a serial port or a joystick port to connect to MIDI instruments. A few MIDI instruments are actually equipped with an 8−pin "mini DIN" serial port, which makes it possible to connect those devices directly to some computers. But the only way to connect a 5−pin DIN equipped MIDI device to a computer's joystick port is via a special adapter cable, which usually must be purchased separately.