Evaluating Trust and Mobile Technologies: Lab versus Fieldstaff.bath.ac.uk/pssds/presentation_on_evaluating_mobile_technolog… · Trust and Mobile Technologies Evaluating Trust and

Evaluating Trust and Mobile Technologies: Lab versus Field

Jonathan [email protected]

April 24, 2008

1 of 23Evaluating Trust and Mobile Technologies: Lab versus Field

Contents

1 Trust and Mobile Technologies

2 Evaluating Trust and Mobile Technologies

3 Risk and Ethics

4 Conclusion

5 References


Trust and Mobile Technologies Evaluating Trust and Mobile Technologies Risk and Ethics Conclusion References



3 Risk and Ethics

4 Conclusion

5 References



Concept of Trust

• At the most general level, trust is based on the incomplete knowledge of

the intended behaviour of another person, artifact, or process [2, 4].

• The assessment of trust must be qualified within a given situation, where

the assessment may or may not generalise across dissimilar contexts or

situations [4].

• One view of trust is that it is only required in situations that are

characterized by risk [9].

• Trust will “only be required if there are things at stake and if there is the

possibility of adverse outcomes” [9, pp. 384].

• Although the concept of risk has varying definitions and connotations in

many different domains.



Concept of Trust





situations [4].









Concept of Trust





situations [4].









Concept of Trust





situations [4].









Concept of Trust





situations [4].









Trust and HCI

• With the proliferation of mobile technologies supporting transactions over

distance and replacing traditional forms of interaction, and the

ever-increasing number of first-time encounters being mediated by

technology, understanding and evaluating trust has become a key concern

for researchers within HCI [9].

• Trust is of particular concern in Human Computer Interaction (HCI) when

users are interacting with systems which require the surrender of, or

threat to, personal information; such as e-commerce services or WiFi

hotspots [1].

• There is not much research on whether this generalises from static

systems to pervasive systems accessed through mobile technologies.



Trust and HCI









hotspots [1].





Trust and HCI









hotspots [1].







3 Risk and Ethics

4 Conclusion

5 References



User Experience Evaluation

• Until recently, evaluation approaches in HCI have been concerned with

the usability of a system in terms of goals [10], for example, those

proposed by Neilsen [7].

• However, there is now a significant shift towards broadening the definition

of usability and incorporating other aspects of the user experience into

interaction design.



User Experience Evaluation

• Until recently, evaluation approaches in HCI have been concerned with

the usability of a system in terms of goals [10], for example, those

proposed by Neilsen [7].

• However, there is now a significant shift towards broadening the definition

of usability and incorporating other aspects of the user experience into

interaction design.



User Experience Evaluation Continued

• With the shift towards a more holistic view of users’ experiences with

technologies, there has been a reconsideration on the validity and

reliability of usability evaluation methods [6, 11].

• In particular, the controversial debate of lab versus field based approaches

to usability evaluation has been revived.

• Most approaches to usability evaluation are inherently based on the use of

a dedicated laboratory, however, the cases for lab and field based

approaches are still evolving [6].

























So what are the issues with evaluating trust and mobile

technologies in the lab and field?



Issues with Subjective Techniques

• Can be used in both approaches, however, is more in the lab than field.

• It is only when questions are asked that data is generated.

• Self-reported subjective measures may not correspond to the actual user

experience.

• Increased cognitive effort is also required to articulate feelings and

emotions of trustworthiness, this can contaminate the results.







experience.









experience.









experience.





Issues with Heterogeneous Environments

• The assessment of trust within the interaction with a mobile technology,

is qualified within a complex combination of contexts, such as social,

cultural or organisational.

• It is essential to:

• Understand the relationship between these different contexts, for example,

social and cultural.

• Understand the effects of the different contexts on the users experience

with the mobile technology.

























Issues with Heterogeneous Environments Continued

• This has implications for the evaluation of trust with mobile technologies,

due to the heterogeneous nature of the environments in which individuals

use mobile technologies. For example, interactions can occur in

segmented social environments.

• There may be inherent rules in an individuals behaviour, implicit or

explicit in nature, that effect the interactions with mobile technologies

within differing environments.




• This has implications for the evaluation of trust with mobile technologies,

due to the heterogeneous nature of the environments in which individuals

use mobile technologies. For example, interactions can occur in

segmented social environments.

• There may be inherent rules in an individuals behaviour, implicit or

explicit in nature, that effect the interactions with mobile technologies

within differing environments.




• Evaluation techniques using a field based approach are perhaps more

suited for the evaluation of trust, where the observed interactions

between individuals and the mobile technologies in the field occur are

more likely to resemble those applicable to real world usage.





3 Risk and Ethics

4 Conclusion

5 References



Risk and Ethics, the Milgram Studies

• The Milgram studies are a series of studies conducted by Stanley Milgram

in 1961 to 1962 [5], which investigated the willingness of participants to

obey authority figures in tasks which conflicted with their personal

conscience [10].



Influence of Power Relationships

• The Milgram studies demonstrated how power relationships can influence

others [10]. This is relevant to the evaluation of trust both in the lab and

field, as participants will leave decision making to a group or its hierarchy,

such as the instructor in the study. Essentially, to some degree

participants loose the sense of responsibility for their actions.

• The responses of a participant may even be skewed to please the

investigator.



Influence of Power Relationships

• The Milgram studies demonstrated how power relationships can influence

others [10]. This is relevant to the evaluation of trust both in the lab and

field, as participants will leave decision making to a group or its hierarchy,

such as the instructor in the study. Essentially, to some degree

participants loose the sense of responsibility for their actions.

• The responses of a participant may even be skewed to please the

investigator.



Influence of Power Relationships Continued

• This creates a situation where trust is not required, as there is nothing at

stake and no possibility of adverse outcomes for the participants [9]. In

such a situation in a study, the decisions in any of the trust or risk based

tasks participants undertake become unrepresentative of responses made

outside of the study.

• Taking this into account, it is likely that field based studies are more

suitable for evaluating trust, as they are less guided and let the users

qualify their decisions in trust or risk based tasks in situations more

applicable to the real world.



Influence of Power Relationships Continued

• This creates a situation where trust is not required, as there is nothing at

stake and no possibility of adverse outcomes for the participants [9]. In

such a situation in a study, the decisions in any of the trust or risk based

tasks participants undertake become unrepresentative of responses made

outside of the study.

• Taking this into account, it is likely that field based studies are more

suitable for evaluating trust, as they are less guided and let the users

qualify their decisions in trust or risk based tasks in situations more

applicable to the real world.



Risk and Ethics

• The Milgram studies also raised ethical issues, as the study relied on

deception [10]. As such, the evaluation of trust becomes problematic:

how can trust be evaluated accurately in an ethical way? [3].

• Applying this question to the lab versus field discussion, it could be

argued that only lab based approaches offer sufficient control for studies

involving deception. For example, debriefing participants or presenting

rewards for participation can be problematic in the field.



Risk and Ethics

• The Milgram studies also raised ethical issues, as the study relied on

deception [10]. As such, the evaluation of trust becomes problematic:

how can trust be evaluated accurately in an ethical way? [3].

• Applying this question to the lab versus field discussion, it could be

argued that only lab based approaches offer sufficient control for studies

involving deception. For example, debriefing participants or presenting

rewards for participation can be problematic in the field.





3 Risk and Ethics

4 Conclusion

5 References



Conclusion

• Exchanges that have traditionally been conducted face-to-face are

increasingly being mediated through technologies. With the proliferation

of mobile technologies, such as laptops and mobile phones, interactions

between individuals or individuals and static or pervasive services can

now, increasingly, take place in a wide variety of environments.

• In a world void of organisational or market pressures, an extensive and

empirical approach to evaluating trust in user interactions with mobile

technologies would be ideal. However, this is perhaps an unrealistic

proposition, especially in the corporate sphere where discount methods

are often preferred [8].



Conclusion

• Exchanges that have traditionally been conducted face-to-face are

increasingly being mediated through technologies. With the proliferation

of mobile technologies, such as laptops and mobile phones, interactions

between individuals or individuals and static or pervasive services can

now, increasingly, take place in a wide variety of environments.

• In a world void of organisational or market pressures, an extensive and

empirical approach to evaluating trust in user interactions with mobile

technologies would be ideal. However, this is perhaps an unrealistic

proposition, especially in the corporate sphere where discount methods

are often preferred [8].



Conclusion Continued

• The lab and field approaches should not be considered mutually exclusive,

however, research shows that the different approaches can yield different

results, which can either contradict or support each other [6].

• As the establishment of trust is qualified within a given situation [4],

studies in the field are perhaps more suited for the evaluation of trust,

where the complex combination of contexts in which the observed

interactions between individuals and the mobile technologies occur are





• The lab and field approaches should not be considered mutually exclusive,

however, research shows that the different approaches can yield different

results, which can either contradict or support each other [6].

• As the establishment of trust is qualified within a given situation [4],

studies in the field are perhaps more suited for the evaluation of trust,

where the complex combination of contexts in which the observed

interactions between individuals and the mobile technologies occur are





• It is also possible to employ facilities in the lab for the collection of high

quality data through audio or video recording [6].

• It is essential that the ethical issues involved in evaluating trust,

sometimes controversial in nature [5], are fully understood before any

evaluation study is undertaken. Lab based approaches to trust evaluation,

which afford a high degree of control, may be more suitable to studies

where there is a high perceived or actual risk in the tasks to be completed

by participants.




• It is also possible to employ facilities in the lab for the collection of high

quality data through audio or video recording [6].

• It is essential that the ethical issues involved in evaluating trust,

sometimes controversial in nature [5], are fully understood before any

evaluation study is undertaken. Lab based approaches to trust evaluation,

which afford a high degree of control, may be more suitable to studies

where there is a high perceived or actual risk in the tasks to be completed

by participants.





3 Risk and Ethics

4 Conclusion

5 References



B. Friedman, D. Hurley, D. C. Howe, H. Nissenbaum, and E. Felten.Users’ conceptions of risks and harms on the web: a comparative study.

In CHI ’02: CHI ’02 extended abstracts on Human factors in computing systems, pages 614–615. ACM,

2002.

B. Friedman, J. Peter H. Khan, and D. C. Howe.Trust online.

Commun. ACM, 43(12):34–40, 2000.

M. Jakobsson and J. Ratkiewicz.Designing ethical phishing experiments: a study of (rot13) ronl query features.

In WWW ’06: Proceedings of the 15th international conference on World Wide Web, pages 513–522, New

York, NY, USA, 2006. ACM.

R. C. Mayer, J. H. Davis, and D. F. Schoorman.An integrative model of organizational trust.

The Academy of Management Review, 20(3):709–734, 1995.

S. Milgram.Obedience to Authority.

Perennial, August 1983.

C. M. Nielsen, M. Overgaard, M. B. Pedersen, J. Stage, andS. Stenild.



It’s worth the hassle!: the added value of evaluating the usability of mobile systems in the field.

In NordiCHI ’06: Proceedings of the 4th Nordic conference on Human-computer interaction, pages

272–280, New York, NY, USA, 2006. ACM.

J. Nielsen.Usability Engineering (The Morgan Kaufmann Series in Interactive Technologies).

Morgan Kaufmann, September 1994.

D. Pinelle, C. Gutwin, and S. Greenberg.Task analysis for groupware usability evaluation: Modeling shared-workspace tasks with the mechanics of

collaboration.

ACM Trans. Comput.-Hum. Interact., 10(4):281–311, 2003.

J. Riegelsberger, M. A. Sasse, and J. D. McCarthy.The mechanics of trust: a framework for research and design.

Int. J. Hum.-Comput. Stud., 62(3):381–422, 2005.

P. J. Rogers Y., Sharp H.C.Interaction Design: Beyond Human-Computer Interaction.

John Wiley, second edition, 2007.

C. E. Wilson.Usability and user experience design: The next decade.

Technical report, Intercom, 2005.


Documents

Evaluating Trust and Mobile Technologies: Lab versus Fieldstaff.bath.ac.uk/pssds/presentation_on_evaluating_mobile_technolog… · Trust and Mobile Technologies Evaluating Trust and