AI Usability Testing With Real Users

AI usability testing should show what happens to real users in the real product, not predict behavior with synthetic personas. Users pause, reopen docs, click the same thing twice, say "wait, which key is this?", and then leave.

UserTold is built to catch those moments while they happen. Your agent gives a participant a real task, UserTold records what they do and say, then returns source-linked evidence for review before work is created.

The pain this solves

Most product teams already know some part of the product feels harder than it should. The hard part is proving which moment matters:

  • new users stop before setup is done
  • buyers compare plans but do not choose one
  • admins abandon an integration after reading docs
  • trial users reach a blank state instead of first value
  • shipped fixes feel better internally but user confusion keeps coming back

Guessing creates vague tickets. Watching the task creates source-backed work.

How to run it

Use one real task. Keep the instruction neutral. Let the participant struggle if the product causes struggle.

Recommended study shape:

  1. speak: Set expectations and explain recording.
  2. observe: Stay quiet while the participant completes the task.
  3. talk: Ask about the moments where they paused, retried, left the flow, or sounded unsure.
  4. speak: Close the session.

Example task:

{
  "id": "connect-provider",
  "mode": "observe",
  "title": "Connect provider account",
  "instruction": "Connect the provider account and continue until you reach a confirmation screen.",
  "conductor_context": "Expected flow: choose provider -> enter credentials -> test connection -> confirmation. Watch for repeated docs visits, uncertainty about key scope, and abandoned saves.",
  "max_duration_s": 420,
  "advance_when": "url:/settings/integrations/connected"
}

The key is the follow-up. UserTold preserves the stuck moment first, then asks about that actual moment instead of asking the participant to imagine what was confusing.

Why the method works

The method is simple: watch real task behavior, keep the user's words attached, and do not turn a summary into work until someone checks the source.

The deeper research basis is in Methodology, but you do not need to run a research program to use it. You need a real workflow, a clear task, and evidence that stays connected to the recording and transcript.

What good evidence looks like

Weak finding:

Setup is confusing.

Source-backed finding:

The participant opened docs twice, changed API key scope, returned to the same field, and said "I do not know which key this wants" on /settings/integrations.

The second version tells an engineer where to look, what happened, and why the work is worth reviewing.

Next step

Use Research Script Templates to start from an onboarding, activation, pricing, or feature-validation script. When several stuck moments point to the same problem, use From Interviews to Linear and GitHub Issues to move it into delivery review.