User testing methods
Quantitative versus qualitative tools
When to Use Which User-Experience Research Methods by Christian Rohrer
User testing is a grid with 4 corners. UX Experts advise their clients to lean towards the behavioural/qualitative quadrant (upper left), because of the addictive nature of attitudinal and quantitative “number chasing”, sometimes called “vanity metrics”. It keeps product manager busy comparing scores while no real UX improvement is made.
Check out my list of customer experience tools for a totally biased selection of services.
All services aim to influence a company’s ROI, or was it KPIs? A handy comparison at the end of this page.
Chart by UXcam
Comparison of the "two sides" of UX testing
Attitudinal / quantitative observation
Behavioural / qualitative observation
The “how many” and “how much” of a problem
The “how” and “why” of a problem
What people say about a site or app is being counted
What people do in a site or app is being recorded
Indirect gathering / comparing of figures
Direct observation / recording of people
Typical methods
Attitudinal / quantitative observation
Behavioural / qualitative observation
analytics
eye-tracking
surveys / forms
heatmaps
A/B testing
session recording
click testing
focus groups
diary studies
hallway or guerilla testing
Disadvantages
Attitudinal / quantitative observation
Behavioural / qualitative observation
Fast and easy, but meaningless for real UX work
Not fast and not easy, but essential for real UX work.
Addictive “vanity metrics”. Numbers do more for product manager personal high scores than for UX
Site goals (e.g. more revenue) need to be translated into user goals (better shopping experience, better prices etc) for effective measuring
Numbers don’t explain why a UX feature is bad
UX features (e.g. an e-commerce process) need to be translated into signals and metrics for effective measuring
If a competitor has a higher number, the method doesn’t tell you why
A TSR Task Success Rate can be extracted as a figur, but only iIf a task has a clearly defined endpoint &– for example, filling out a form or buying a product – you can measure the TSR. However, you need to be clear about what goals you consider a success in a particular case before you start collecting data
Core metrics
Attitudinal / quantitative observation
Behavioural / qualitative observation
TSR Task Success Rate (the number of correctly executed tasks) or Completion Rate
- Pass/fail metric. Either users complete the task, or they don’t.
- Create tests that have pre-defined success criteria.
- Use on wireframes
CSAT Customer Satisfaction
Task Time
- Average time spent on task
- Average task completion time
- Mean time to failure (time before giving up or completing with errors)
Errors
- Create a task for the user to carry out. Record the users’s mistakes, unintended actions and slips
- Categorize the types of errors
Select the wrong link
Misinterpret content
Miss out on the intended target
An inaccurate sense of task completion
Miss out on the intended targets
Search versus navigation
Comparison of ROI and KPI
ROI
KPI
return on investment
key performance indicator
A financial indicator and quantifies how successful a project was in relation to its investment
Key figures that you can choose or define yourself which translate the success of a project – however it may be defined – into tangible figures
Example, if a company invests EUR 10,000 in UX activities to improve its online shop and then generates EUR 25,000 more revenue in the following year, this corresponds to an ROI of 150%
Relevant for almost all employees of an organisation – from call centre employees to CEOs – and can be applied to a variety of processes