Hi everyone. I’m Andrew Harder and I want to talk to you about how researchers can have a better impact when working with designers. I work for Nokia Design in London. Previously I’ve worked with them as a design researcher on pre-roadmap concepts like the 2012 range of phones; and right now I’m running a Nokia-wide UX evaluation process for products that are launching as soon as next month. Before that I worked for an agency

1

And I want to talk about the change in perspective that I had when I made this jump that I made from being agency-side at Flow delivering research, to going to Nokia and managing external researchers to deliver work that I would then use to try and be useful to designers and Nokia’s design process. Massive shift in my perspective about what good research looks like, and how research can be useful to design, when researchers THINK they’re being useful to design and they’re not and when research can totally hijack design.

I think it starts with education. As a researcher I did an HCI masters in London, and what we learned boiled down, to me, to being exposed to evaluation techniques fundamentally based on cognitive psychology or ergonomics. Techniques like user tests or cognitive walkthroughs draw directly from cognitive psychology. And the ultimate result of these kinds of evaluation projects is a big list of issues with a design. You know, it’s a list of what is wrong with the current state of things.

The designers I worked with at both Flow and Nokia have a completely different approach to their work – that is broadly speaking a seen is a focus on the generation and development of new ideasthat resonate with social need.

So as we all know this can be a great pairing – one capability focusing on what is wrong now, and one focusing on what can be done next.

But often the results of research and evaluation aren’t presented in a way that can help designers take the next step, and actually in my opinion show little respect for the role of designers and their responsibilities. This is what I want to talk to you about today – how we can work well together,trying to draw some lessons from the pitfalls that I’ve seen myself as a researcher fall into, then see researchers that I’ve managed fall into too. Reasons for this:•What kind of people go into research•The research process, and what it does to researchers•The education that we get and tools that we use,

But first I want to consider the issue of leadership in the field. So I’ve picked Don Norman to start with

2

Look at that face, who could find fault with him?

Stating the obvious, Don Norman is the guru of modern usabilityAll but created the field with Design of Everyday ThingsPersonally inspired me to join this fieldUndisputed leader in UX

3

Even Newsweek says so. Well his website told me that Newsweek said that at least.

So as well as being an HCI practitioner, he is also, you might say primarily in public an educator and an opinion leader. His very successfully evangelises our fieldLaid the basic frameworks of UX in ways that are still relevant todayNow also deliberately provokes us to think critically about what it is that we do.

Before I tell you what my beef is with his work, let me take you on an aside. Like I’ve said, this talk comes from my own experience and seeing common attitudes in researchersProblem is that I can’t share specific client examples, so I was trying to think about how I could ground this talk in some way so everyone could understand my pointThen I found some this essay of his

4

This piece he wrote.

Trapped in a Lufthansa Airline Seat – Don Norman

Perfectly encapsulates the worst of presenting research. So when I found this, I was so relieved. Being a nearly digital native, feeling some emotion at my desk made me need to express it

5

On Twitter. I was so relieved. Imagine my surprise when the next day on my way to work I saw this…

6

Stupid Twitter!

So I kind of feel like Don is in the room with me now. And before all of you go tweeting “thevagrant hates the Don” let me make one thing really really clear

I love the Don. I love him.

But he’s not perfect.

7

And articles like this drive me up the wall. It’s basically a documentation of the problems he found with this Lufthansa Airline seat. Trapped in a Lufthansa Airline Seat – Don NormanThe audio and video selections are generous even though it took me five minutes to figure out how to get the menu. There is a remarkable variety of video available, and each one seems to be playable on demand… (As far as I can tell, once started, it cannot be paused, fast forwarded or backed up, but maybe I just didn’t find those controls.)And there is even a hidden panel for computer support: regular AC power… It even has jacks labeled Internet and USB. The person across the aisle soon discovered that the internet is not yet implemented. “Real Soon Now,” is the promise, but hey, give Lufthansa credit. There is even storage space for shoes. It’s wonderful. Except it doesn’t work. We were on a ten-hour flight, so my wife and I took off our shoes…. But only one shoe would fit into a storage space, and there were only two spaces for each pair of passengers — two spaces for four shoes. Maybe German use smaller shoes than Americans? More likely, that’s all the room they had.And you know that nice, reclining bed? It doesn’t disturb the neighbors, but it certainly uses every inch of space in the compartment. Leave a pillow, blanket, or heaven forbid …your computer laptop in it on the floor and, crush, grind, scrunch. No more pillow, no more blanket, no more laptop. “Please do not leave hand luggage on the floor when using this position,” says the manual next to the heading “Sleep.” But where does the hand luggage go? Up into the overhead compartment? Pain. Between the two seats? Maybe, but the space is only about six inches wide, occupied by the two shoes we couldn’t fit into the platform. Sure the briefcase can fit on top of the shoes, but if it leans even slightly to one side, it gets crunched.

8

So all in, these are all the problems he finds with this airline seat. This is basically what the article is about, this one-sided list of problems. He tries to balance some positives by says that he’s praising Lufthansa for trying hard, but I have to say my reading of it never gets far away from sarcasm.

The problem I have with this is that it epitomises a form of writing that is considered legitimate in our field. It is somehow acceptable to write articles like this that describe a user’s experience of a product or service and just list fault after fault. I’ve used Don’s work because as a leader in our field he legitimises this as a valid outcome of a UX review. At a basic, fundamental level, by reading articles like this especially when we are learning our trade, we implicitly believe that this is an acceptable way for us to talk about a product or service. My point is that this is NOT acceptable.

This type of discourse is ugly, uninformative, its tone and presentation style drives good designers and service providers AWAY from us as UX researchers. In Australia we’d call this whinging. This is nothing more than complaining about a service. And complaining is not useful in a design environment.

And as researchers, it is far, far too easy for us to drop into complaining about something rather than having a more impactful approach.

9

These are the bad habits & practices that we get into that can make it feel like we’re complaining.

Tells everything – We want to say everything that we foundSuggests solutions – We do this to be helpful, but this is a problemIgnores – Because it wasn’t in the test, right? Users didn’t see it. Documents and formalises – Because we don’t want to forget anything, and we want everyone to remember everythingShows how smart we are – We learned a lot, and we want to be valued by our peers

Introduce the critique: Designers has a highly developed practice for address problems in their work, the design crit that is a fundamental skill taught in design schools. •Two skills, giving the crit and taking the crit, they’re both very important. But I’m only going to talk about GIVING the crit today

As researchers we need to learn more about giving critique, because that’s more often the side we’re on

10

CritiqueFillet the findings – We only talk about the VERY MOST IMPORTANT THINGSCreative Authority – Even if they don’t have it in the rest of the organisation, we need to elevate designers to the people who can respond to our researchIntent – We research, analyse and talk about the hidden ideas about what is BEHIND the prototype or system that we testInterpersonal – A critique is ONLY EVERY interpersonalMakes everyone else feel smartTalks about the design and the qualities of the design, not the endlessly complex experiences users can have.

11

So where do these differences come from? There are two basic models that explain A LOT of this.

Common model, Preece Rogers & Sharp Interaction Design book, written in an HCI faculty at Sussex University•Research plays a primary role, but the easy implication to draw is that evaluation and needs are the only thing that the design needs to follow. •No brand, no portfolio of other products, no strategy, no COMPETITION etc. •This was outside the scope of their book, but that’s part of the problem. We’re training people to ignore the most important parts

The biggest fallacy I see people reading into models like this is that research can provide all the answers•Research provides the requirements and identifies the gaps between the prototype and what users want. • This model IMPLIES that successive iterations of design exist to address and remove usability problems•Design, therefore, is an activity that introduces error into this process

12

Immediately this is a different model of design – no flow chart for a start! Problem – initial brief or problem, stimulus to first round of activity. It’s a single point so it might be as little as a sentence, but it’s certainly not a thesis. Discover is a phase that searches for ideas that respond to this brief. •We deliberately explore multiple different avenues on how to respond to the brief•What are the possible dimensions to explore this brief? •What are the underlying user needs? Define aligns the discovered user needs to the basic business objectives & project managementEnds in the problem statement, a second brief about what needs to be done. Develop – multiple design solutions are developed, iterated and evaluatedDeliver – resulting design is finalised and launched in the market. Primary focus of this model is to scope and define the exploration of ideas in the process. This view also shows the nature of design ideas as they progress through their development.

HCI is strong at the converging parts of this diagram. Loads of methods, approach, practice, publications, and so on. Lots of methods, even ethnography has been brought out in this

We have very little to say about the diverging parts of this diagram. That previous model says we have to gather requirements, in the form of needs – but it implies that researchers are in charge of this phase of the project and that needs are just out there waiting to be discovered. We can apply the same methods that we apply in the diverging sections, without a prototype – look at behaviours, identify pain points, etc. This is all useful stuff, but the real value of this divergent phase – looking at what CAN BE DONE we have little to say about. This is what, at the moment at least, I believe that designers uniquely bring to the table. Their training in generating and developing ideasPatience with ideas

13

Not everyone thinks these diverging parts of the diagram are good and that starting from a single point rather than a thesis is good.

14

The Don again. Remember, I love the man. I love him so much that I love arguing with him even more.

15

Published on Core 77 last year – you might have read it

He makes no statement of the problem beyond his problems with design competition winners, and a dogmatic view that everyone should understand cognitive science. He also gives no examples of any of these designs he finds naïve. So really, as an argument, it’s total horseshit.

But that last paragraph that I’ve snipped rings some bells – designers don’t understand the complexity of the issues and the depth of knowledge already known. Researcher’s argument, it prioritises the explicit reading-knowledge that researchers like me valueLet me tell you a bit of a secret – in ANY multinational consumer company, they are drowning in research. They have just too much research. And yet we always have to commission more to get what we need to know! In particular, it is his view that not understanding the last degree of complexity in a situation is an indefensible failure that is the most illuminating. This is the complaint of someone who passionately believes in that iterative process of research, and is deliberately marginalising the ideation work.

16

But there are appropriate questions to ask at each stage. Understand the purpose of the design activity that’s happening in the moment and take it from there. We need to know when to mix in the complexity of the real world, and when to protect the design process from all the challenges that the real world presents.

17

... But... There are some researchers who don’t agree with The Don either

18

In this paper, Usability Evaluation Considered Harmful (Some of the time)

19

We’re not trained how to deal with ambiguity in interfaces, we can criticise them in light of being perfectly mature interfaces which they aren’t. The double diamond model helps us situate them in their purpose during the design process. Radical new ideas may be immatureIf there are social or cultural issues, today’s rejection may be tomorrow’s acceptance

20

Which brings me back to my original question

What do we do?

21

So, a tangent to make this point. I’m going to tell you a story about two authors, Ernest Hemingway and Kurt Vonnegut. One of Hemingway’s definitive works is The Old Man and the Sea, which is a novella about Santiago, a very old fisherman

22

In the story, Santiago goes out to sea, much further than other fisherman go, and engages in a herculean struggle to catch an enormous marlin. He thrashes all day and all night, the fight nearly kills him, but eventually he catches the marlin only to discover that it is too big to fit into his boat. Because he can’t imagine not landing this fish, he strats it to the outside of his boat and beings rowing back to shore. The problem is that there are sharks in the water, and they begin attacking the marlin to eat it. The fisherman tries to fight them off, but he’s already exhausted by catching this fish and he’s no match for them. By the time he gets back to short, these sharks have eaten all the fish, all that’s left is a bit of skeleton, and this being Hemingway he’s scorned and ridiculed by the people who have stayed at shore, for doing all this struggle and then having nothing to show for it. .

So another author, Kurt Vonnegut took up this story and offered a meaning for it. Hemingway published The Old Man and the Sea 10 years after his previous novel, which he had struggled to finish but he considered the great work of his life – but the critics lambasted it and ridiculed it. So in Vonnegut’s version, the old man was Hemingway struggling to finish his life’s work, going out on a mammoth task that nearly kills him, catching a prize fish but having the critics as shark decimate his work before he could land it.

And I think in any large research project I’ve been in a similar situation – this might sound odd to non-researchers, but even a 10 person user test can sometimes feel like an odyssey, meeting so many new people, learning about their attitudes, behaviours, even lives, and my head is swimming with the richness of what I’ve learned and the possibilities of what could be done in response to this information. If you haven’t done user research it might sound weird, but even coming up with a list of usability bugs can be really exciting and inspirational.

23

The struggle is then to land this big catch of information and insight back to shore where clients can use it. And if I look back to my time in agency,

Still remember trying to deliver 100 page powerpoint, met with blank stares about for what designers should actually do next. And when you deliver a 100 page ppt, the implicit response is – well, all of this! Because we fall so in love with our findings, we lose perspective about it. We know all this stuff and incomprehensible that anyone could change a single feature or flow without understanding every last detail we foundAnd the problem here is that if you try to say everything, you end up having no impact whatsoever – especially in an evaluation context – presenting every usability bug = nagging. By telling them everything you’re telling them nothing. It’s never the case that designers need to know everything. Never. We have to simplify and fillet our findings to deliver, meaningful, critical stuff that will engage people.

24

The interesting thing about Vonnegut’s take on the story is that he told Hemingway’s story to a fisherman, who basically said – the guy’s an idiot. Of course sharks are going to eat your fish if you strap it to your boat. What you have to do in that situation is choose the very best bit of marlin, cut that out and throw the best overboard – leave the sharks to it. That way, at least you get a fish soup when you get back rather than people scratching their heads at you thinking, what was all that effort for? We have to fillet our research. We have to! But how? Still remember trying to deliver 100 page powerpoint, met with blank stares about for what designers should actually do next. And when you deliver a 100 page ppt, the implicit response is – well, all of this! Because we fall so in love with our findings, we lose perspective about it. We know all this stuff and incomprehensible that anyone could change a single feature or flow without understanding every last detail we foundAnd the problem here is that if you try to say everything, you end up having no impact whatsoever – especially in an evaluation context – presenting every usability bug = nagging. By telling them everything you’re telling them nothing. It’s never the case that designers need to know everything. Never. We have to simplify and fillet our findings to deliver, meaningful, critical stuff that will engage people. The interesting thing about Vonnegut’s take on the story is that he told Hemingway’s story to a fisherman, who basically said – the guy’s an idiot. Of course sharks are going to eat your fish if you strap it to your boat. What you have to do in that situation is choose the very best bit of marlin, cut that out and throw the best overboard – leave the sharks to it. That way, at least you get a fish soup when you get back rather than people scratching their heads at you thinking, what was all that effort for? We have to fillet our research. We have to! But how?

25

Empathy is a critical skill for all UX professionalsBut after a big research project, it can make us feel a monkey to its organ grinder.

26

We can get so passionately attached to our users and in love with our findings, that we can let our empathy with them rule our reactions to them. And we want our clients or partners to feel the same way that we do with them•Forgetting that a lot of the time, we need to treat them as customers rather than people •And they say a lot of things that just aren’t relevant to what we are looking for•Simply, that after meeting a lot of people we sometimes can’t see the wood for the trees•We celebrate the complexity of their users at the expense of providing actionable insight

What to do: Draft, space, reviewExecutive summariesRemember that the most important thing about personas is their segmentation of the user market, not the 10th bullet point that we put on them. There are also some ways that we can think about what our user say that helps us identify what is likely to unlock the most value

27

Japanese academic Noriaki Kano developed different categories to explain what kind of product qualities satisfy people. Threshold – Things people expect as a credibility requirement, also called hygiene factors. Take away satisfaction if they’re not present, but no matter how much of them you add or improve, satisfaction doesn’t improve. •Ryanair defines threshold qualities. Made a lot of money out of figuring out exactly what these are and delivering precisely no more than that. Airline: seat, you go in the air, arrival at a new airport – this is undeniably a flight that you’ve been on •Conference: airconditioning, basic time keeping•Amazon: payment processing, ability to find what you’re looking for

28

29

30

Performance: classic top of mind things about what satisfies people about a product or service. The more this need is met, the more satisfied you are. •Ryanair: Price; BA perform on service too. •Conference: Quality of speakers & workshops, networking opportunities, etc. •Amazon: price, width of offering, delivery time, customer service

Excitement: the hardest to grasp in the abstract, excitement qualities make you really satisfied if they’re present, but they don’t make you dissatisfied if they’re not around. •Ryanair: Well the only thing I’m excited about on a Ryanair flight is leaving it. BA, an upgrade can be an excitement factor… hard to think of an ordinary situation where you’d be disappointed to not get one•Amazon: Black Friday sales

31

32

33

So what?

Clients understand their product already

•Clients have a really good understanding of what their product qualities are and where they fit in this, even if they don’t know the Kano model per se•Agencies often “discover” threshold factors and think that it’s new and important•They also may not realise that they are really threshold factors. •Example from phones – in my first user research for Nokia, it was in the states where durability and battery life are king. Threshold factors… if you add more durability then it won’t make people more satisfied.

HCI methods lean us towards discovering dissatisfiers rather than satisfiers. • We identify and gravitate towards tangible problems and blocks – language like ‘pain points’ ‘severity’

There is a difference between experience and design•We also conflate delivery problems with design problems and pretend that they’re threshold factors. Like this

34

So this is a threshold product quality – accurate and reliable spell check.

This is clearly a bad experience, but my point is that it WAS NOT A DESIGNED EXPERIENCE. If you find the guy who designed the messaging view in the iPhone, or the keypad –that guy did not design the autocorrection. Engineers did.

We need to prioritise the design in our evaluations rather than talking about ‘experiences’. The implication is that spell check needs to go from being 90% accurate to 100% accurate – well guess what, they knew that already. And what would the impact on satisfaction be? Minimal. We can invest a lot of money and effort without gaining significantly more satisfaction. While big companies have a lot of resources, they have very scarce attention and focus – this is very precious in big companies, so rather than thinking ‘oh my client can just pass on my findings about spell check, offline mode and the manual to the right people’ we have to help them focus on the single most important things!

35

So what? 2

FOCUS ON THE RIGHT KIND OF FACTORS FOR YOUR DESIGN OBJECTIVES•Kano helps us target our research findings against the stage of design ideation we’re at•Left diamond

•Typically looking for excitement qualities•Performance qualities can be used to define the project•Threshold qualities should just be ignored

•Right diamond•Manage performance and threshold qualities

•This explains how in innovative designs, threshold qualities may not be perfect when they launch•Windows 7 phone – no way to group your applications•Palm Pre performance sucks – it is a slow phone•Should threshold qualities be perfect, of course they SHOULD•But the managed underdelivery of threshold qualities can allow teams to deliver products that excite people •Empathic, idealistic UX people would rather run a mile than suggest cutting back on threshold qualities, yet Ryanair proves you can make millions by doing this. •It feels heretical to be suggesting we use our user research to do this… but isn’t it true? In terms of researching them, this is a very useful filter to use to understand what your consumers have said and talked about. If we were researching Ryanair, then every single person would talk about how awful the service is, and can you believe the charges that they land you with, and did you hear that the staff are on commission for all those charges (which is true), etc etc. If we were doing this research for an airline that wanted to explore low-cost alternatives, we could NEVER recommend doing this stuff, cutting back services, adding fees, etc etc. And yet, there is an enormous business built out of giving users as little as possible. Michael O’Leary even publicises the fact that they’re always looking for other things to cut back on. Is their idea of charging for toilets, not giving people a kind of sling rather than seat – are these real threshold qualities or aren’t they?

UX as a field would rather run a million miles than help businesses cut back on threshold qualities, and yet there is millions and millions to be made in this.

36

The other big problem I see is when researchers making deliver their work to designers with a list of recommendations or even worse, design concepts.

At Nokia specifically, rarely does a project go by without agencies wanting to ‘add value’ as they see it, by giving us some concepts. You know, what if there was a phone that did this, you could detach this part and it would do this, it could be a customized phone for the elderly, or trend-following teenagers, or business people in a new city for the first time, whatever. We’ve learnt all this, and this is what you could do with it as a result. Sometimes this comes off as good-intentioned but toothless, but to be honest most of the time it’s seen as patronizing or naïve.

37

It’s EASY to make recommendations to fix these problems that you’ve found, but

unless it’s explicitly part of your scope. It’s not our job as researchers to define the

solution; we have to explore and define the PROBLEM, not the solution.

This happens again and again when I engage agencies – even if I explicitly

instruct them not to do it, they just can’t help it

Deliberately provocative with the headline

38

Or a slightly more scientific look at it

39

You can see why I want researchers to avoid making recommendations… because when you do you’ve just shortcut this entire process.

What about at the diverging phases? Can’t we make recommendations even then?

Don’t understand the product, intent, business, industry, competition, suppliers, etc. When you make recommendations in isolation, you’re bound to fall foul of these gaps.

A more useful model for what researchers can do is here:

40

Useful for evaluation work

Problem is what was reported – “Users don’t see that there are multiple homescreens on the Nexus One”

Problem Space is an explanation of the nature and structure of the problem, so we go up a level in the hierarchy of the problem.

41

Problem: “Users don’t see that there are multiple homescreens on the Nexus One”

42

Problem Space is an explanation of the nature and structure of the problem, so we go up a level in the hierarchy of the problem. So one definition of the problem space might be: No affordance for moving homescreen. More precisely: Lack of affordance creates a discoverability problem for moving homescreens. And again: Users who have no experience with smartphones may have a discoverability problem with recognising the system offers multiple homescreens and controlling them. And again: Feedback that the phone gives isn’t sufficientComponents of a problem space: • User experience (emotion, judgement, behaviour etc) • Design element • [Contextual factor]• Linked together in a causal chain• In a general state that explains general events not individual comments. Solution Space is the area of possible responses to the problem. This is an active area in which we ideate, it’s a brainstorming activity. And then of course, the solution is what emerges from the solution space as the best option for whatever reason, even time. If you make recommendations without exploring the problem space or the solution space, you go straight from problem to solution•Miss out better solutions than the one that first occurred to you•Reflect your imperfect knowledge of the design intent•Disempower the creative authority of designersSo a recommendation for this could be to have page indicators on the screen with numbers.

43

Which is what Samsung does

This might test well, but without further input, how do we know that this is the best solution?

If we explored the solution space more, we’d find solutions like this:

44

45

Exploring potential solutions gives us the perspective that real users don’t really care what the navigation method they use for their screens are, they just want to be able to get around their damn phone!

46

47

48

49