Organizing Teams for AI in Production
Part 8 of my series on AI in production. Part 7 told the story of one AI product. This part is about the teams and structures that make the next ten possible.
Companies tend to organise for AI in one of two ways, and both usually disappoint.
In the first, a central AI team is formed. It's full of capable people, and it builds impressive prototypes. But it sits outside the product teams, doesn't own any customer outcome, and hands over work that nobody downstream asked for or knows how to maintain. The demos are great. Adoption isn't.
In the second, every product team builds its own AI features from scratch. Each one chooses a model, builds its own retrieval, invents its own logging and skips evals because there's no time. Five teams solve the same problems five ways, and none of them solve security or reliability properly.
When I led a fintech platform with several product managers, designers and analysts, the lesson that carried over most directly to AI was this: the structure decides what gets reused, who owns outcomes, and how fast teams can move. Get it wrong and talented people spend their time rebuilding plumbing or building things nobody adopts.
Product teams own outcomes; a platform owns the plumbing
The shape that works best in my experience combines the two. Product teams own AI features and the business outcomes they're meant to move. A small platform team owns the pieces every AI feature needs, and makes them easy to use.
Shared services support each product team; product teams own their business outcomes.
The split of ownership is what makes this work:
- Product teams own the outcome. The sales assistant team is measured on win rate and time to close, not on how many prompts it shipped. It owns its eval set, its quality bar and its users' adoption.
- The platform team owns the paved road. One way to call models, one retrieval service, one logging and audit trail, shared eval tooling, standard approval patterns and the security controls from part 3. Its customers are the product teams, and it's measured on how fast they can ship safely.
- Governance sits across both. Policies on data, risk and approvals are owned by named business and compliance leaders, as in part 4, and enforced through the platform rather than reviewed project by project.
This is the same model that works for payments and API platforms. Product teams build what customers see; a platform makes the hard, shared parts reliable so nobody rebuilds them.
The roles that matter, and who owns what
An AI product team looks a lot like any good product team, with a few roles that matter more than usual and a few responsibilities that need a clear owner.
| Role | Owns |
|---|---|
| Product manager | The business outcome, the launch bar, prioritisation, and the eval set as the spec (part 2) |
| Domain expert | The final call on what a good answer looks like; reviews failures every week |
| AI or software engineers | Prompts, retrieval, tools and agent logic; making the system observable and testable |
| Data engineer | Getting the right data in, current and permitted; often the role teams discover they needed last |
| Designer | How people check, approve and correct the AI's work (part 4), which often decides adoption |
| Platform engineers | The shared services: model access, logging, security, eval tooling |
- Role
- Product manager
- Owns
- The business outcome, the launch bar, prioritisation, and the eval set as the spec (part 2)
- Role
- Domain expert
- Owns
- The final call on what a good answer looks like; reviews failures every week
- Role
- AI or software engineers
- Owns
- Prompts, retrieval, tools and agent logic; making the system observable and testable
- Role
- Data engineer
- Owns
- Getting the right data in, current and permitted; often the role teams discover they needed last
- Role
- Designer
- Owns
- How people check, approve and correct the AI's work (part 4), which often decides adoption
- Role
- Platform engineers
- Owns
- The shared services: model access, logging, security, eval tooling
Two things are easy to get wrong. The domain expert must be a real, named person with time set aside, not a colleague consulted when someone remembers. And the data role must exist from the start; most AI projects that stall are waiting on data, not on models.
What really slows AI teams down
AI teams rarely move slowly because engineers lack skill. They move slowly because of friction that the organisation can remove.
| What slows teams | What fixes it |
|---|---|
| Fear of changing anything, because nobody knows what might break | An eval set that runs on every change, so the team knows within minutes |
| Weeks waiting for access to data | Pre-approved, governed data access through the platform |
| Every team rebuilding the same plumbing | Shared services for models, retrieval, logging and approvals |
| Prompts scattered across code and documents | Prompts versioned and reviewed like code, deployed with the application |
| Long feedback loops with experts | A standing weekly review slot with the domain expert, and a simple review tool |
| Security and legal reviews starting from zero for each project | Standard controls on the platform, so reviews focus only on what's new |
- What slows teams
- Fear of changing anything, because nobody knows what might break
- What fixes it
- An eval set that runs on every change, so the team knows within minutes
- What slows teams
- Weeks waiting for access to data
- What fixes it
- Pre-approved, governed data access through the platform
- What slows teams
- Every team rebuilding the same plumbing
- What fixes it
- Shared services for models, retrieval, logging and approvals
- What slows teams
- Prompts scattered across code and documents
- What fixes it
- Prompts versioned and reviewed like code, deployed with the application
- What slows teams
- Long feedback loops with experts
- What fixes it
- A standing weekly review slot with the domain expert, and a simple review tool
- What slows teams
- Security and legal reviews starting from zero for each project
- What fixes it
- Standard controls on the platform, so reviews focus only on what's new
Measure productivity by how quickly a team can test an idea and learn from it, not by lines of code or features shipped. A useful question for any AI team: how long does it take from "we think this change will help" to "we know whether it helped"? If the answer is weeks, that's the first thing to fix.
AI coding tools help too, and they compound with everything above. They make building internal tools, review interfaces and test harnesses cheap enough that teams actually build them.
Building the team
Most companies don't need a team of research scientists to put AI into production. They need people who ship reliable software, understand the business, and are curious enough to learn how models behave.
- Upskill before you hire. Your existing engineers already know your systems, data and customers, which is the hardest part to hire for. Pair them with one or two people who have shipped AI products before, and most will be productive within weeks.
- Hire for judgement, not just knowledge. The most valuable trait on an AI team is the habit of looking at real outputs and asking what went wrong. Ask candidates to review a set of flawed AI answers and talk you through them.
- Make failure review a ritual. A weekly session where the team and the domain expert read real failures together builds shared judgement faster than any training course. Celebrate the failures found, not just the features shipped.
- Give people a career path. AI product work is new, and good people leave when they can't see where it leads. Define what growth looks like for AI-focused PMs and engineers, the same way you would for any other speciality.
When I ran OKRs and promotions for a product team, the clearest signal of a strong PM was the same as it is in AI work now: they owned the outcome, not the output, and they could explain why something worked or didn't.
Align everyone on outcomes, not activity
AI teams are easily measured on the wrong things: models evaluated, prompts written, features launched, questions answered. These are signs of activity. They say nothing about whether the business is better off.
Set goals the way you would for any product team, with one difference: the AI team and the business owner share the same goal.
- The objective belongs to the business. "Shorten the sales cycle for mid-market deals," owned jointly by the Head of Sales and the sales assistant team.
- Key results measure outcomes, such as time to close, win rate and same-day follow-ups, with quality guardrails like eval pass rate and zero customer-facing errors.
- Platform goals measure enablement, such as how long it takes a product team to launch a new AI feature safely, and how many teams use the shared services.
Shared goals change the conversations. Sales stops asking the AI team for features and starts working with it on results. The AI team stops defending its work and starts reporting against a target the business chose. When a quarter doesn't go well, both sides look at the same numbers and decide together what to change.
Structure is a product decision
It's tempting to treat AI as a technology problem and solve it by hiring the right specialists. In my experience, the companies that get AI into production consistently are the ones that treat it as an organisational problem. They decide who owns which outcome, what is shared, how fast teams can learn, and how everyone is measured.
Get those right and the technology tends to follow. Get them wrong and even the best models end up as demos.
Next, and last, in the series: measuring AI ROI honestly. Holdouts, incremental value, and why before-and-after numbers mislead.