Test the Decision, Not the Dashboard: Usability Testing for Expert, Time-Pressured Interfaces

The standard usability test goes something like this. You hand someone a task ("find the maneuver log," "change the start time for program B"), you watch, and you write down whether they finished and how long it took. That's a fine test for a settings page. It's a dangerously flattering test for an operations console.

Here's the problem. On an expert tool, finding the right screen is rarely where things go wrong. Operators are good at navigation, because they do it all day. Where things go wrong is the decision they make once they're on the screen: they trust a number that's six hours stale, they miss that a fix creates a new problem somewhere else, or they spend ten minutes being thorough when they had four. An interface can pass every findability task you throw at it and still get its operators to a wrong answer on time.

So for anything time-pressured, I'd change what gets measured. The unit of the test isn't the task. It's the decision.

Two consoles where the deadline is the design

I've designed two self-initiated consoles where this is the whole point. OrbitGuard Ops is a collision-avoidance console for a small-satellite operator. The real job isn't reading probabilities. It's deciding whether to spend fuel before the window to spend it closes. Hunter Water Management is an irrigation console for one water manager running ten sites, where everything hangs off an overnight watering window, typically 21:00 to 05:00, set by local ordinance.

Both are spec projects. I haven't sat a real flight director or a municipal water manager in front of either one, and I'm not going to pretend I have. What follows is the test plan I'd write for them, and why I'd write it this way instead of the usual list of tasks.

What "decision quality" means in practice

For each scenario I'd measure three things, roughly in order of how much they matter.

Was the decision right? Not "did they reach the planner," but "did they decline the burn they should have declined?" You need a known correct answer for each scenario, agreed in advance with someone who knows the domain. In OrbitGuard, one proposed 0.34 m/s burn resolves its event but creates two more. That's a beautiful test case because the obvious answer is wrong.

Was it made in time? A correct decision after the deadline is a wrong decision. In OrbitGuard, the last moment you can commit a burn is hours before closest approach, because of command upload and ground-station passes. The deadline is drawn as a wall for exactly that reason. The test should have a wall too.

Was the confidence justified? After each decision, ask the participant how sure they are, on a simple scale. Then compare that to whether they were right. Confident and right is great. Unsure and right is fine. Confident and wrong is the thing the interface exists to prevent, and it's invisible in a test that only records completion.

Seed the trap, then see who notices

The most useful thing you can do in one of these sessions is plant a problem in the data and not mention it. A tracking network whose last update is far older than its usual cadence. A site whose schedule quietly no longer fits inside the window. An alert whose numbers don't add up.

Both consoles were designed around this kind of failure. OrbitGuard judges each network's freshness against its own expected cadence, so 46 minutes old on an 8-hour radar cadence reads as fine, while 1 hour 14 minutes on a 1-hour optical cadence doesn't. Source health sits in a permanent strip instead of popping up as an alert. Hunter Water's alerts quote the station's design flow next to its measured flow, so a fault has to show its working.

Those are design claims. A seeded scenario is how you find out whether they're true. The measurement is simple: did the participant notice the stale source, unprompted, before committing? If three out of five sail right past it, the strip isn't doing its job, however sensible it looked in review.

A ticking clock, but a fair one

Time pressure changes how people use an interface, so the test needs some. I'd give each scenario a real countdown, shorter than comfortable but not impossible. Too generous and everyone reads every panel, which tells you nothing about the moment that matters. Too tight and you're just testing reflexes.

That has a knock-on effect on think-aloud. Asking people to narrate while they work is the classic technique, and it slows them down and changes where they look. Under a clock, I'd keep the session quiet and do a retrospective walkthrough straight afterward: replay the screen recording and ask "what were you looking at here? What made you decide?" You lose a little immediacy and keep the pressure honest, which is the trade I'd make here.

When you can't get the experts

Real satellite operators don't have spare afternoons for a stranger's prototype. This is the most common reason expert tools go untested, and it's a bad reason. Some options, from best to worst:

Get two or three real experts for the decision scenarios only, and use everyone else for the rest. Even one session with someone who's done the job will flatten assumptions you didn't know you had. For the navigation and legibility questions (can people read the deadline marker, can they find the planner) adjacent experts are fine: air-traffic, utility dispatch, or network operations people already think in windows and cadences. Students and generalists come last, and their job is only to catch the obvious stuff. Whatever mix you use, write it down in the findings. "Tested with adjacent operators, not flight directors" is a perfectly respectable sentence. Leaving it out isn't.

A template for a decision-based scenario

This is the shape I'd use for every scenario in the plan. It fits on an index card, which is a good sign.

Run five or six of those per participant and you'll know a lot more than whether the menus make sense. You'll know whether the console helps someone make the call it was built for.

The short version

If people make decisions on your screen under a deadline, don't stop at "can they find it." Write scenarios with a correct answer, put a real clock on them, plant one problem you don't mention, and ask how sure people were. Then look hardest at the confident wrong answers. That's where an interface is quietly failing.

Both consoles are live as prototypes. You can open OrbitGuard Ops and Hunter Water Management and try to beat the deadline yourself. If you're building a tool where a late answer counts as a wrong one, I'd love to talk about how to test it. Get in touch.