HAVE 2008 – IEEE International Workshop on

Haptic Audio Visual Environments and their Applications Ottawa – Canada, 18-19 October 2008

A Comparison of Mamdani and Sugeno Fuzzy Inference Systems for Evaluating the

Quality of Experience of Hapto-Audio-Visual Applications

Abdelwahab Hamam and Nicolas D. Georganas

Distributed & Collaborative Virtual Environments Research Laboratory

School of Information Technology and Engineering

University of Ottawa, Ottawa, ON, Canada

{ahamam, georganas}@discover.uottawa.ca

Abstract – Virtual reality applications which incorporate haptic

devices to enrich users’ sense of touch are increasing in number.

Assessing the Quality of Experience (QoE) of these applications

reflects the amount of overall satisfaction and benefit gained from

the application plus it lays the foundation for ideal user-centric

design in the future. In this paper we build on our QoE fuzzy logic

model, previously simulated and tested, by comparing results from

two different and well-established fuzzy systems: Mamdani and

Sugeno. The results analytically demonstrate the essential

differences between the two systems and the benefits of using either

one in assessing the overall QoE.

Keywords – Quality of Experience, Human-Computer Interaction,

Virtual Reality Applications, Haptics, Fuzzy Inference System,

Sugeno, Mamdani

I. INTRODUCTION

Haptic applications are increasingly gaining popularity.

They are being developed and used in different areas such as

gaming, rehabilitation, and medical simulation to name a few.

Unfortunately, there is no clear way to define the advantages

of these applications to users. Did they really undergo a

unique and enriching experience or were they just dazzled by

this new technology that allows them to touch and manipulate

virtual objects? Is there an area we should work on as

designers to make users more comfortable and happy with

their experience? Can we compare different applications

objectively and conclude that one of them is better suited

from a user’s perspective?

All these are questions that will benefit the research

community if addressed. Quality of Experience (QoE)

concerns has been addressed before but rarely for haptic and

virtual reality applications. QoE is a regarded as a subjective

issue and it is the goal of our work to objectify it eventually

based on a solid QoE model.

In this paper we discuss one of those steps that will lead us

closer to that objective. Our previous work in this area was

coming up with a taxonomy of QoE parameters and building

a fuzzy logic system for QoE evaluation. Here we compare

two different types of fuzzy inference system, Mamdani and

Sugeno and study the benefits of using one over the other

when evaluating the quality of experience.

The rest of the paper is organized as follows. First we

review our QoE Model and briefly explain its taxonomy

components. Next we present the fuzzy logic system design

that is used in evaluating and testing our QoE Model. Then

we outline the difference between Mamdani and Sugeno

fuzzy inference systems. Simulation results obtained and

analysis of both systems are discussed afterwards. Finally, we

conclude this paper and state the future work.

II. QOE MODEL AND TAXONOMY

In [1] we have proposed a taxonomy to evaluate the

Quality of Experience of Hapto-Audio-Visual applications.

The QoE model taxonomy is based on our definition of QoE

which is the Quality of Service (QoS) of the application in

addition to User Experience. The higher level QoE model is

shown in Figure 1.

Figure 1 – Higher level organization of our QoE model taxonomy

This taxonomy includes all the related parameters that are

necessary to assess and test the advantage/disadvantages of a

haptics application. The purpose of the categories is to

organize the given parameters into a comprehensible list to

the evaluator of the application. Often the evaluator chooses

to evaluate one aspect of the application rather than evaluate

every single aspect. The QoE model shown here serves this

requirement by organizing parameters into different

categories, and evaluators are free to choose whichever

parameters they want to evaluate from one or multiple

categories.

978-1-4244-2669-0/08/$25.00 ©2008 IEEE

Each category serves a different purpose. The QoS

category deals with conventional parameters that are

associated with most applications that require networking and

synchronization, although synchronization here refers to

network synchronization and media synchronization (audio,

video, and haptics).

User Experience on the other hand is divided to four sub-

categories, namely:

• Perception Measures: User-centric category that

mirrors how the user perceives the application

• Rendering Quality: Quality of the three major

modalities in virtual reality application, namely:

graphics, audio, and haptics

• Physiological Measures: Biological parameters

measured directly through users while they are

using the application

• Psychological Measures: Reflect the status of the

user through observation

The above is a brief description of the taxonomy used in

building our QoE model. Readers interested in detailed

description should refer to [1].

III. FUZZY INFERENCE SYSTEM

Within the taxonomy there are many parameters that are

fuzzy in nature. For example, it cannot be deduced that the

application is causing fatigue without level of uncertainty. To

tackle this we have decided to construct a fuzzy logic system,

flexible in the number of inputs, that would map fuzzy inputs

into a crisp value, in our case the QoE value.

Using MATLAB® [2], we have built a fuzzy logic system

to test our QoE model and apply it in evaluating a Hapto-

Audio-Visual (HAVE) environment [3]. We initially started

with the Mamdani Fuzzy Inference System (FIS) [4] but later

extended it to the Sugeno FIS [5].

Our proof of concept model is based on a five input, one

output FIS. We diversified the inputs to fall under as many

categories as possible from the higher level organization in

Figure 1. We chose the parameters to be relevant to 3D

interface virtual reality application, such as medical surgery

simulation [6]. However most of these parameters are

generic, i.e. applicable to any type of HAVE. Just the Degree

of Immersion is specific to immersive 3D interfaces, and if

changed or eliminated the model can be applied to different

types of virtual reality applications such as 2D interface

application.

A. Membership Functions

Each input we have chosen has a unique type of

membership function, depending on the property of the

parameter. The five parameters used in the model along with

their membership functions are:

1. Media Synchronization (QoS parameter): There are usually

three media modals in a HAVE application. Any miss-

synchronization between the audio, video, and haptics can

cause a drastic loss of perception of both media that are miss-

synchronized. Therefore the membership function should

have a Gaussian waveform with high decay rate.

Figure 2 - Media Synchronization membership function

2. Fatigue (Quality of perception): Research has shown that

fatigue, which is caused by muscle exhaustion, is linearly

distributed as a function of time [7]. Therefore we chose a

linear triangular membership function.

Figure 3 - Fatigue membership function

3. Haptic rendering (Rendering Quality): This can be a

trapezoidal function due to the fact that the Haptic rendering

quality remains the same until we reach a threshold (that is

usually referred to as the JND - Just Noticeable Difference)

after which the quality starts decaying [8].

Figure 4 - Haptic Rendering membership function

4. Degree of immersion (Psychological measures): Even

though the degree of immersion will cause a difference in

quality, this difference is still not quite understood [9]. We

chose a linear triangular membership function in this case as

we except immersion to be linearly distributed based on the

user.

Figure 5 - Degree of Immersion membership function

5. User Satisfaction (Quality of perception): This is a

Gaussian membership function because of the normal

distribution of human satisfaction measures. As proven by

some researchers human satisfaction increases until it reaches

a threshold point [10, 11]. The decaying rate is lower than

media synchronization since user satisfaction is not as

volatile.

Figure 6 - User Satisfaction membership function

B. Rule Selection

Rules were selected according to relationship between the

parameters chosen. Interested readers please refer to [3].

C. FIS Types

There are two well established types of FIS: Mamdani [4]

and Sugeno [5]. As shown in Figure 7, we modeled two

instances of our system using those two types of FIS to

acquire our results. Our goal is to see if the results differ and

if there is a significant advantage of using one type over the

other.

Figure 7 - Mamdani FIS (top) and Sugeno FIS (bottom)

Both systems contain the same number of inputs with the

same type of membership functions. They even contain the

same rules. They differ, however, in the output generation

process from the fuzzy inputs. The next section outlines the

differences between the Mamdani and Sugeno FIS.

IV. MAMDANI FIS VS. SUGENO FIS

The most fundamental difference between Mamdani type

FIS and Sugeno type FIS is the way the crisp output is

generated from the fuzzy inputs [12]. While Mamdani FIS

uses the technique of defuzzification of a fuzzy output,

Sugeno FIS uses weighted average to compute the crisp

output. Therefore in Sugeno FIS the defuzzification process

is bypassed. It can be noticed from Figure 7, that this is

actually noticeable from the outer view design. The Mamdani

QoE output looks like a fuzzy output. The Mamdani output is

displayed in figure 8. The output has 5 linear membership

functions: InTolerable, UnAcceptable, Average, Excellent,

and Perfect.

Once the fuzzy logic system resolves the input values

along the given rules, the output can be either fuzzy if the

HCI system designer requires a fuzzy output or it can be

crisp, as in our case, which requires an extra defuzzification

step [13].

Figure 8 – Mamdani FIS output membership output function

Since sugeno FIS uses weighted average for the output, we

divided the output into five levels and we labeled them to

correspond to Mamdani’s five output membership functions.

The five constant membership functions along with their

values are given in Table 1.

Table 1 – Sugeno FIS constant output

Perfect 1.0

Excellent 0.75

Average 0.5

UnAcceptable 0.25

UnTolerable 0

It can be noted that these are output values, and the labels

are just there to assist the design in MATLAB® [2]. Hence

critics of the Sugeno FIS argue that the expressive power and

interpretability of the Mamdani output is lost in the Sugeno

FIS since the consequents of the rules are not fuzzy [12, 14].

Table 2 summarizes the differences between the Mamdani

FIS and the Sugeno FIS [12, 13, 14, 15].

Table 2 – Comparison between Mamdani FIS and Sugeno FIS

Mamdani Sugeno

Output membership function No output membership function

Output distribution No output distribution only ‘resulting action’:

Mathematical combination of the

rule strength and the output

Crisp result obtained through defuzzification of rules’ consequent

No defuzzification: crisp result is obtained using weighted average of

the rules’ consequent

Non-continuous output surface Continuous output surface

MISO and MIMO systems Only MISO systems 1

Expressive power and interpretable

rule consequents

Loss of interpretability

Less flexibility in system design More flexibility in system design; more parameters in the output

Based on the above, there are some advantages of using

either Mamdani FIS or Sugeno FIS. The advantages of using

Mamdani FIS are:

• Expressive power

• Easy formalization and interpretability

• Reasonable results with relatively simple structure

• Intuitive and interpretable nature of the rule base.

For this reason Mamdani FIS is widely used in

particular for decision support application

• Can be used for both MISO and MIMO systems

• Output can either be fuzzy or a crisp output

The advantages of using Sugeno FIS are:

• There are algorithms which can be used to

automatically optimize the Sugeno FIS. One of

the tools that can calibrate the weights of the

Sugeno FIS output is MATLAB’s ANFIS

• Better processing time since the weighted average

replace the time consuming defuzzification

process

• Computational efficiency and accuracy

• More robust when in presence of noisy input data

such as sensor data

• Rules’ consequents can have as many parameters

per rule as input values allowing more degrees of

freedom and more flexibility in the design

• Adequate for functional analysis because of the

continuous structure of output function (same

inputs do not originate substantially different

outputs

1 MISO : Multiple Input Single Output

MIMO: Multiple Input Multiple Output

V. SIMULATION RESULTS

To test the system we ran both visual tests and command

based testing in MATLAB. The visual testing involved

running the MATLAB fuzzy logic toolbox, called ‘rule

viewer’. The rule viewer gives a visual aid on which rules are

selected and activated and their effect on the output. The

input can be given by dragging the red line over the input or

in the text box provided at the bottom (Figure 9(a)). The

command based testing eased the testing process since we

had the option to run a script like the one shown in Figure

9(b). That particular script fixes all inputs to nine except for

the first input (media synchronization) that is incremented

from one to ten. Subsequently, MATLAB will display the

results of the ten QoE values corresponding to each media

synchronization value.

Figure 9 - (a) Rule viewer of MATLAB’s fuzzy logic toolbox, (b) An

Excerpt of Matlab script

We ran the script above for both Mamdani FIS and

Sugeno FIS and got the following results:

Table 3 - Mamdani and Sugeno FISs results when varying Media

Synchronization input

Media Synch Mamdani FIS Sugeno FIS

1 0.5029 0.5178 2 0.5409 0.6074 3 0.6188 0.7030 4 0.6787 0.7417 5 0.6975 0.7492 6 0.7036 0.7521 7 0.7184 0.7631 8 0.7535 0.7941 9 0.7803 0.8294

10 0.7803 0.8434

Both Mamdani FIS and Sugeno FIS increase in value as

the value of Media Synchronization increases. Figure 10

shows the correlation of both results within that context. The

numerical correlation is 0.9831 and was calculated using the

following equation:

correlation of x and y = ∑−

−−

)1(**

))((

nsysx

yyxx (1)

where x and y are two separate series (in our case they are the

Mamdani FIS results and the Sugeno FIS results), the bar

value is the average of the series, the s value is the standard

deviation, and n is the number of elements in the series.

Mamdani FIS and Sugeno FIS Results

0.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9

0 1 2 3 4 5 6 7 8 9 10

Medai Synchronization

QoE

Mamdani FIS

Sugeno FIS

Figure 10 – Correlation between Mamdani and Sugeno QoE results when varying Media Synchronization input

Additional results where calculated similarly. Four

variables would be fixed to nine and the remaining variable

would be incremented sequentially from one to ten. The

constant value nine for the fixed variables was chosen

arbitrary. The aim is to test the results of both systems while

their environments are identical. Nevertheless the value nine

of the four fixed variables reflects the parameters of a high

quality application and our expectation was to see high QoE

results as the fifth varying variable becomes closer to ten.

The following table shows the correlation value for each of

the five cases (where in each case only one of the five inputs

was incremented while the other variables were set to nine).

Table 4 – Mamdani and Sugeno FIS correlation for various results

Incremented Variable Correlation between Mamdani and Sugeno FIS results

Media Synchronization 0.9831

Fatigue 0.9857

Haptic Rendering 0.9959

Degree of Immersion 0.9951

User Satisfaction 0.9960

The table indicates high correlation between both

inference systems for all five tests. This indicates consistency

of both systems and that the results, though might differ in

value, are going in the same direction and there are no

unusual exceptions in the results. Thus both systems are

reliable systems for evaluating QoE.

Looking closely at the results in Table 3, we note that

Sugeno FIS results are more accurate in the sense that they

generate closer values to what is expected. For example,

>> mamdani =

readfis('QOEmodelMamdan

i.fis')

mamdani =

name:

'QOEmodelMamdani'

type: 'mamdani'

andMethod: 'min'

orMethod: 'max'

defuzzMethod: 'centroid'

impMethod: 'min'

aggMethod: 'max'

input: [1x5 struct]

output: [1x1 struct]

rule: [1x13 struct]

>> for j=1:10,

qoe = evalfis ([j 9 9 9 9],

mamdani)

end

when all values are set to nine, Mamdani FIS generates

78.03% as QoE while Sugeno FIS results in 82.94%. An

application with high media synchronization, high haptic

rendering quality, low fatigue, and high degree of immersion

and user satisfaction should have a high QoE. In general

though, Sugeno FIS values are always higher than Mamdani

FIS values which could be due to the defuzzification process

of the Mamdani inference system. Another point that can be

attributed to the defuzzification process is that Sugeno FIS

results in most cases had higher standard deviation values.

This denotes that Sugeno values are more dynamic to input

changes.

Checking the boundaries of the system for minimum and

maximum input, we find that when all input values are set to

one, the lowest these inputs can reach, Mamdani FIS results

in the QoE being 21.97% and Sugeno FIS results in 21.10%.

On the other hand when the inputs are at their maximum

value, i.e. all are set to ten, QoE is 78.65% when using

Mamdani FIS and it is 87.48% when using Sugeno FIS. In

the boundary cases furthermore, Sugeno FIS establishes to be

more accurate than its Mamdani FIS counterpart.

Still, although the Sugeno FIS seems to be more accurate

we cannot rule out the consistency of Mamdani’s inference

process. Mamdani FIS has a solid defuzzification process and

that keeps the result in a consistent form. For example, in all

five cases the last value of Mamdani QoE is close to 78%, i.e.

when all inputs are set to nine except for one input which is

set to ten. Mamdani’s results can also be verbalized. In the

case when QoE is 78%, we can say instead that the QoE was

excellent. This verbalization is lost in Sugeno FIS and we can

only refer to the QoE in numbers.

During our simulation runs there were no noticeable

performance variations. In MATLAB all tests seemed to run

smoothly. Researchers, however, found that Sugeno FIS runs

faster than Mamdnai FIS [14]. In those tests the data vector

was very large in oppose to our methodology. In our

experiments we had a small data vector which was looped

around ten times only. For large data vectors performance

issue might be a concern, bur for our model this concern is

limited, unless we use bigger data vectors in the future.

VI. CONCLUSION

This paper presents a comparison between Mamdani and

Sugeno FIS models for evaluating the QoE of a multimodal

virtual environment. Both our models were built and

simulated in MATLAB and hypothetical results were

generated using identical environment for both models.

Our results show that there are advantages and

disadvantages of using either model. Sugeno FIS

demonstrates higher accuracy and more dynamical values.

Mamdani FIS on the other hand displays consistency in the

results and expressive power. Both models, nevertheless,

show high correlation value and thus reflect fairly reliable

results and can be utilized to come up with a crisp QoE value.

Their usage can be determined on whether the VR application

evaluator desires consistent values or higher accuracy values.

In our future testing and as our QoE model is growing, we

will adopt only one system. Most likely we are going to

utilize the Sugeno FIS due to its accuracy and because we can

use certain features in MATLAB such as the ANFIS tool

which optimizes the inputs.

VII. REFERENCES

[1] A. Hamam, M. Eid, A. El Saddik, and N.D. Georganas, A Quality of

Experience Model for Haptic User Interfaces, Haptic User Interfaces in

Ambient Media System, Quebec City, Canada, 2008. [2] Fuzzy Logic Toolbox for Use with MATLAB®, Math Works Inc., 2001.

[3] A. Hamam, M. Eid, A. El Saddik, and N.D. Georganas, A Fuzzy Logic

System for Evaluating Quality of Experience of Haptic-based Applications, EuroHaptics, Madrid, Spain, 2008.

[4] E.H. Mamdani and S. Assilian, An Experiment in Linguistic Synthesis

with a Fuzzy Logic Controller, International Journal of Human-Computer Studies. 51, 2 (1999), 135-147.

[5] T. Takagi and M. Sugeno, Fuzzy identification of systems and its

applications to modeling and control, IEEE Transactions on Systems, Man and Cybernetics, 15 (1985), pp. 116-132.

[6] A. Hamam, S. Nourian, N.R. El-Far, F. Malric, X. Shen, and N.D.

Georganas, A Distributed, Collaborative, and Haptic-Enabled Eye Cataract Surgery Application with a User Interface on Desktop, Stereo

Desktop, and Spatially Immersive Displays, Proc. IEEE Workshop on

Haptic Audio Visual Environments and their Applications, Ottawa, Canada, November 2006.

[7] R. Seroussi, M.H. Krag, P. Wilder, and M.H. Pope, The Design and

Use of a Microcomputerized Real-Time Muscle Fatigue Monitor Based on the Medial Frequency Shift in the Electromyographic Signal, IEEE

Transaction on Biomedical Engineering, 36, 2 (1989), pp. 284-286.

[8] M.A. Srinivasan, and C. Basdogan, Haptics in Virtual Environments: Taxonomy, Research Status and Challenges, Computers and Graphics,

Vol. 21, No. 4 (1997), pp. 393-404. [9] F. Gutiérrez, J. Pierce, V.M. Vergara, R. Coulter, L. Saland, T.P.

Caudell, T.E. Goldsmith, and D.C. Alverson, The effect of degree of

immersion upon learning performance in virtual reality simulations for medical education, Studies in Health Technology and Informatics,

125(2007), pp. 155-160.

[10] M.. Andrews, J. Cao, and J. McGowan, Measuring Human Satisfaction in Data Networks, IEEE International Conference on Computer

Communications, Barcelona, Spain, 2006.

[11] D. Gao, J. Cai, P. Bao, and Z. He, MPEG-4 Video Streaming Quality Evaluation in IEEE 802.11e WLANs, IEEE International Conference on

Image Processing, Genoa, Italy, 2005.

[12] J. Jassbi, S.H. Alavi, P. Serra, R.A. Ribeiro, Transformation of a Mamdani FIS to First Order Sugeno FIS, IEEE International Fuzzy

Systems Conference, 2007.

[13] Fuzzy Inference Systems, http://www.cs.princeton.edu/courses/archive/fall07/cos436/Knapp/fuzz

y004.htm, last accessed on July 26, 2008.

[14] J. Jassbi, P. Serra, R.A. Ribeiro, A. Donati, A Comparison of Mandani and Sugeno Inference Systems for a Space Fault Detection Application,

World Automation Congress (WAC), Budapest, Hungary, July 2008.

[15] T.J. Meitzler and E. Sohn, A Comparison of Mamdani and Sugeno Methods for Modeling Visual Perception Lab Data, Annual Meeting of

the North American Fuzzy Information Processing Society, 2005.