Pants. I put them on every weekday
I noted in yesterday’s Pants that many of the common UX Research moderated methodologies have their origins in field Anthropology and Linguistics research. So I started perusing Linguistic fieldwork manuals to see what I could learn.
First, some context. In linguistics, fieldwork is usually used by descriptive linguists to understand and document a language. One of the core techniques of linguistic fieldwork is elicitation. Elicitation is the process by which a researcher extracts knowledge (anything from concrete nouns to grammar to super abstract language) from an informant. There’s a lot of flavors of elicitation. The one most relevant to moderated UX Research is called “Stimulus-Driven Elicitation,” in which a researcher asks questions or assigns tasks in relation to an object or picture. In the case of UX, the stimulus might be an app on a phone, a website on a computer, a series of storyboards, sketches.
The warnings and guidelines in these field manuals by Vaux and Cooper and Chelliah and Reuse are as relevant to UX Research as they are to their intended subject. Check these out (italics are my points):
- “Record everything and name the recording immediately.”
- Lost recordings become anecdotes instead of evidence.
- “For the fieldworker, it is better to observe and describe … facts regardless of whether or not they fit into someone’s formal theory. ‘Paying attention’ does not mean that the fieldworker can or should try to pay attention to all [granular details] at the same time.”
- “One can only talk about objects within sight, ideally objects on the table in front of the fieldworker and consultant.”
- Users are surprisingly terrible at imagination games. Ask about what’s in front of you. More formally, produce an input, provide a task, and monitor the output.
- “as soon as something slightly abstract is pointed at, misunderstandings will occur”
- Abstractness is inversely correlated with certainty. More specifically, users have no real idea how computers or the internet works.
- “If there are too many pictures on one sheet . . . consultants will be easily confused”
- People can competently think about one thing at a time. Crowded wireframes with notation are great for documentation and terrible for getting user feedback.
- “Don’t ask leading questions . . . . You also need to be careful not to go too far in the other direction by intentionally keeping the informant in the dark about the goals of your [inquiry]”
- Examples of leading questions: “What do you think that back button does?” or “How would you save here?”
- Non-leading questions: “Please describe what you see here.” “You mentioned Component X. What do you think it would do?” “Let’s say you’re finished here. What would you do next?”
- Keeping the user too much in the dark: not explaining at all the context of what you’re testing
- Providing adequate context: Show the user a mock-up of an ad campaign or a storyboard of events that lead them to use the app. Give them the same context that someone in the world would actually have. Almost no one downloads an app they’ve never heard of and are 100% confused about why it’s there or why they downloaded it.
- “Avoid using too much [technical] terminology with and in front of the informant.”
- Users have no idea what a modal is.
- “Don’t overload the informant. If you present informants with too much data or too many tasks at once, they may be put off.”
- People can think about one thing at a time. Similarly, they can only do one thing at a time. Use an alternating input/task/output cycle rather than frontloading a bunch of questions.
- “Beware of priming effects. It is well known among [researchers] that informants can become highly unreliable in their judgments after they have been presented with too many examples of the same type of [structure].”
- You know how someone always says, “Why don’t we show the users all the variants and see which one works best?” This is the reason not to do that.
At a high level, the best practices of social science primary research are transferrable across domains. I’m certain you could browse guides on Psych study design or Anthropological field research and get similar insights.