User testing methods

Quantitative versus qualitative tools

User testing is a grid with 4 corners. UX Experts advise their clients to lean towards the behavioural/qualitative quadrant (upper left), because of the addictive nature of attitudinal and quantitative “number chasing”, sometimes called “vanity metrics”. It keeps product manager busy comparing scores while no real UX improvement is made. 

Check out my list of customer experience tools for a totally biased selection of services.

All services aim to influence a company’s ROI, or was it KPIs? A handy comparison at the end of this page.

Chart by UXcam

Comparison of the "two sides" of UX testing

Attitudinal / quantitative observation

Behavioural / qualitative observation

The “how many” and “how much” of a problem

The “how” and “why” of a problem

What people say about a site or app is being counted

What people do in a site or app is being recorded

Indirect gathering / comparing of figures

Direct observation / recording of people

Typical methods

Attitudinal / quantitative observation

Behavioural / qualitative observation

analytics

eye-tracking

surveys / forms

heatmaps

A/B testing

session recording

click testing

focus groups

diary studies

hallway or guerilla testing

Disadvantages

Attitudinal / quantitative observation

Behavioural / qualitative observation

Fast and easy, but meaningless for real UX work

Not fast and not easy, but essential for real UX work. 

Addictive “vanity metrics”. Numbers do more for product manager personal high scores than for UX

Site goals (e.g. more revenue) need to be translated into user goals (better shopping experience, better prices etc)  for effective measuring

Numbers don’t explain why a UX feature is bad

UX features (e.g. an e-commerce process) need to be translated into signals and metrics for effective measuring

If a competitor has a higher number, the method doesn’t tell you why

A TSR Task Success Rate can be extracted as a figur, but only iIf a task has a clearly defined endpoint &– for example, filling out a form or buying a product – you can measure the TSR. However, you need to be clear about what goals you consider a success in a particular case before you start collecting data

Core metrics

Attitudinal / quantitative observation

Behavioural / qualitative observation

TSR Task Success Rate (the number of correctly executed tasks) or Completion Rate

  • Pass/fail metric. Either users complete the task, or they don’t.
  • Create tests that have pre-defined success criteria.
  • Use on wireframes

CSAT Customer Satisfaction

Task Time

  • Average time spent on task
  • Average task completion time 
  • Mean time to failure (time before giving up or completing with errors)

 

Errors

  • Create a task for the user to carry out. Record the users’s mistakes, unintended actions and slips
  • Categorize the types of errors

Select the wrong link

Misinterpret content

Miss out on the intended target

An inaccurate sense of task completion

Miss out on the intended targets

Search versus navigation

Comparison of ROI and KPI

ROI

KPI

return on investment

key performance indicator

A financial indicator and quantifies how successful a project was in relation to its investment

Key figures that you can choose or define yourself which translate the success of a project – however it may be defined – into tangible figures

Example, if a company invests EUR 10,000 in UX activities to improve its online shop and then generates EUR 25,000 more revenue in the following year, this corresponds to an ROI of 150%

Relevant for almost all employees of an organisation – from call centre employees to CEOs – and can be applied to a variety of processes

Leave a Reply

Your email address will not be published. Required fields are marked *