View
215
Download
1
Embed Size (px)
Citation preview
Measuring User Satisfactionin Virtual Environment
Maciej A. OrzechowskiDesign System and Urban Planning Group
@ TU/e
Workshop Mass Customisation 26.06.2003
Plan
• Introduction• VR System (brief description)• Belief Networks (introduction)• Results of the experiment (Benchmark)• MuseV3 in action – Live Demo
The user is asked to modify that design according to his/her needs and desires.
General Idea ofMeasuring User’s Preferences
The Virtual Environment (VE) is used to present an architectural design to a user.
Behind that visual system there is a statistical model to estimate and predict respondent’s preferences based on applied modifications.
MuseV – VR System
MuseV3 – a virtual reality (VR) application with functionality of a simple CAD system for non-designers.
Two categories of modifications:• Structural modifications (change of layout)• Textural modifications (change of visual impression)
Structural Modifications
The most important from the point of view of estimation of user’s preferences.
Change of internal and external layout
Direct impact on overall costs
Expressed in simple and direct commands: create/resize/divide space; insert openings
Textural Modifications
Secondary modifications (visual impact), mainly used to check proportions, dimensions (inserting furniture) and to decorate (applying finishes).
Not included in the preference model
No influence on costs
Belief Network
Searching for new, flexible method to access user’s preferences.
Criteria:Criteria:• Interaction with the model during the time of preferences estimation
• Possibility to find weak points (where the knowledge about preferences is the worst)
• Improve data collection by direct feedback
• Incremental learning
Short explanation of BN
What it is?• Belief network (BN) also known as a Bayesian network or probabilistic causal network• BN captures believed relations (which may be uncertain, stochastic, or imprecise) between a set of variables which are relevant to some problem (e.g. coefficients and choices).
How does it work?After the belief network is constructed, it may be applied to a particular case. For each variable you know the value of, you enter that value into its node as a finding (also known as “evidence”). Then Netica does probabilistic inference to find beliefs for all the other variables.
Incremental learning.After the beliefs are found (post priori) MuseV updates the network, so they become a’ priori for the next respondent.
BN - Model
In our proposal the network (model) is learning while a user is modifying a design!
To improve the quality of collected data and the knowledge about design attributes, the system, (based on beliefs), can post a question to user.
Experiment Types
There are in total four experiment types (FMVR, OEVR, MECA, VECA).
Two in each of two groups (VR and CA).
Each respondent had to complete two random tasks (one from each group), however each combination of tasks should be presented approximately equal number of times.
Experiment Types – cont.
VR Experiment CA Experiment
Type Free Modification Preset Options Multimedia Presentation
Verbal Description
Software(Mean of
presentation)
MuseV3 FM MuseV3 OE MuseV3 SC Web Pages
CollectionMethod
Interaction with 3D environment
Interaction with 3D environment
Questionnaire Questionnaire
Task Modification of architectural
design
Respond to pre designed options
Choice from between three
design alternatives
Choice from between three
design alternatives
Interactivity with 3D model
Restrained to design constrains
Finishes and furniture
Walk Through N/a
Feedback from the system
yes yes none none
Estimation method
Belief Network Belief Network MNL Model MNL Model
RespondentsThe truth about the respondents:
We sent 1,600 letters in total !!!!
The preparations to send those letters took 2,5 daysfor two people (Vincent and Maciek)
Within 2 weeks we received 96 positive conformations.
At the end of the experiment we end up with solid number of 64 respondents that have completed the both appointed to them tasks!
5 of the 64 respondents would not buy the house that they have designed.
4 respondents did not completed second task(as the design was not relevant to them)
2 respondents did not started the experiment for the same reason!
External validity Real Life Data – Overall (CA, BN)
G-O-F of REAL LIFE (RL) PREDICTION (Rho2 calculation based on log likelihood)Calculations based on BETAS
GOF (CA) = -0.0384GOF (BN) = 0.1128
External validity Real Life Data – BN (FMVR, OEVR)
G-O-F of REAL LIFE (RL) PREDICTION (Rho2 calculation based on log likelihood)Calculations based on BETAS
GOF (FMVR) = 0.13096GOF (OEVR) = 0.01979
External validity Real Data – BNbased on probability distribution of each option
Option Type
Lounge Ext.
First Floor Ext.
Garage Ext.
Extra Kitchen
Bedrooms Dormer Window
Real life situation 0.867 0.867 0.667 0.067 0.067 0.133
Belief Network includes both subtypes (all respondents)
0.81 0.35 0.71 0.32 0.208 0.519
Subtype: Free Modification 0.83 0.334 0.781 0.491 0.168 0.391
Subtype: Preset Options 0.81 0.394 0.633 0.156 0.244 0.666
The table illustrates ratio (percentage) of choosing certain design option.In case of real life - based on numbers of subjects buying certain option.Ri = Ni / N, where Ri – ratio for option i, Ni – number of subjects choosing option i, N – all subjects
In case of BN based on beliefs read from the network.
Summary• The majority of the respondents prefer the VR environment to the traditional.
• Respondents highly valued the freedom in modifying the architectural design.
• Due to learning and understanding the software - VR is slightly difficult.
• The traditional method was find as the most difficult (due to problems related with imagining the description of the house)
Summary• Direct observation of respondent's engagement (created designs and the time spent on the process) into the VR - indicates that people prefer to work with 3D models rather then with textual description.
• The possibility of experiencing with the not existing house reinsure users' decision, raise questions and provokes discussions.
• The numerical analyses showed that working with virtual reality helps respondents to understand the design and improve their decision consistency.