Building arbitrr: designing an AI product people trust

arbitrr started with a conversation between a few of us who work in user-centred design and digital delivery. We came from different parts of a project build, but we kept describing the same problem.

On project after project, the knowledge a team needed already existed somewhere in the organisation. Someone had hit the same risk, solved the same problem or done the same research. The team just didn't know it existed, so they started again from scratch.

Two years later, arbitrr is live. Along the way, it taught me more about design than almost anything else I've worked on. Here's what I learned.

Don't compete, connect

Our first ideas were the obvious ones. One was a project management platform like Jira, built specifically for UCD teams, with AI built in. We explored a few others along the same lines.

The more we looked at them, the more we saw the same flaw. Every one of them was competing with tools teams already use and aren't going to give up. Nobody wants another place to keep their work.

So we turned the idea around. Instead of replacing the tools teams already rely on, arbitrr connects to them, and brings together the knowledge that's scattered across them. That one change shaped everything that followed.

Just because it can, doesn't mean it should

AI can find patterns in almost anything you give it. That's its strength, and it's also the risk.

There were several features I decided arbitrr could do, but shouldn't. Most came down to context. An AI reading tickets and messages doesn't know the conversation in the corridor, the difficult week someone was having, or why a decision was really made. It can easily misread what it sees.

That might mean a misleading finding about a team. Worse, it might mean an unfair one about a person. So arbitrr never comments on individual performance or compares people against each other. It's a tool for teams, not for surveillance.

Designing trust, not screens

With AI at the heart of the product, trust was the biggest thing I had to design for. If people don't trust what arbitrr tells them, nothing else about it matters.

That changed where I spent my time. The most design effort didn't go into the screens. It went into the filters that decide which findings people see and which get thrown away.

Every finding has to meet strict rules before anyone sees it. It must be traceable back to a real ticket, message or document. Its evidence has to quote the original words, not the AI's summary of them. And anything that doesn't meet the bar is dropped, however interesting it looks.

The result is fewer findings, but ones people can check for themselves. That's what earns trust: not a confident-sounding answer, but one you can verify.

Knowing when to stop

Designing and building something yourself sounds like freedom, and in many ways it is. But it removed something I hadn't realised I relied on: a client asking when it would be ready.

Without that pressure, I could keep postponing the release until I was completely happy with it. And as any designer will tell you, you're never completely happy with it.

Eventually I had to have a word with myself. The product was good enough to release, and the people using it would tell me far more about what needed to change than I could work out on my own.

If I did it again, I'd release it much sooner. Real users are the best design partner you'll ever have, and every week you keep a product to yourself is a week you're not learning from them.

What I bring to every project now

If arbitrr taught me one thing, it's this: be clear about what success looks like, and stick to it.

The more time you spend agreeing that at the start, the easier it is to recognise when you've achieved it. It stops a project drifting, it makes decisions easier, and it tells you when it's time to launch and learn from real people.

So every project I take on, whether it's a website, a review or a government service, starts by agreeing what success looks like. Everything else is measured against it.

You can read the full arbitrr case study, or if you've got a project of your own, tell me about it.