How to Take AI Agents From Pilot to Production at Scale

Plenty of organisations have an AI agent pilot that worked well in a demo. Far fewer have hundreds or thousands of agents running reliably in the real world, and that gap is where the hard work sits. AI agents are moving beyond experimentation, but taking them from promising pilots into production at scale is proving to be much more difficult. At the same time, AI is reshaping technical teams, changing traditional career paths and forcing organisations to rethink how technology creates value.

In this episode of Tech Transformed, host Christina Stathopoulos, founder of Dare to Data, explores what that transition looks like in practice. She is joined by Italo Vietro, VP of Engineering at Parloa, and Joshua Burks, Global Managed Services Head at Thoughtworks. They cover platform foundations, workforce change and agent management at scale. Their shared message is that powerful models are only part of the answer.

From Task Execution to Orchestration

Burks opens with a clear observation he has seen, and that is AI is shifting technical teams from task execution towards architectural orchestration. For decades, he says, engineering and operations teams have spent around 75 per cent of their time and capacity clearing tickets, sifting through alerts and writing boilerplate syntax. Agentic AI and automated coding tools are making that kind of syntax generation and routine triage a commodity. So the bottleneck has moved downstream. Burks describes a new kind of team built from T-shaped pods of engineers who manage autonomous agents, validate complex system behaviour and focus on continuous modernisation. The aim is to stop merely keeping the lights on.

The Rise of the Product-Minded Architect

Vietro sees the same pattern at Parloa and has a name for the people at the centre of it, which, as he calls them, product-minded architects. Instead of fixating on the solution itself, they spend more time in the problem space.

On the positive side, he thinks this is good news for engineering. Engineers generally want to solve problems, and understanding the problem has become more important than ever. The practical challenge is how to architect many agents together so they deliver the outcomes you expect. For Vietro, this also means ownership of something. People need to feel they can deliver the solutions they want, with the right outcomes, rather than simply carrying out tasks.

What Value Arbitrage Really Means

Stathopoulos asks Burks to unpack a phrase he used early on, and his answer is one of the most useful parts of the conversation.

For roughly 40 years, he says, managed services has been a race to the bottom on a pricing model built entirely on labour arbitrage. Vendors bill for headcount, hours, utilisation and ticket throughput. The flaw is that the vendor profits from the complexity, manual effort and lingering technical debt in a client’s systems.

Technology arbitrage flips that. AI agents reduce the headcount needed to run operations or development. Burks believes the industry is now heading towards value arbitrage, where platform automation, predictable observability and agentic workflows take on the toil.

The question then changes from how many people or agents are on the account to what the organisation gets back. That means less technical debt, smaller ticket and development backlogs, and systems that are continuously modernised and ready for whatever demands come next, whether internal or external. Burks describes them as systems that never age.

Making The Jump From Pilot to Production

Stathopoulos points out the big difference between an agent working in a controlled playground and deploying hundreds or thousands of them in the real world. Vietro, whose company deploys agents at enormous scale, walks through what the transition involves. Let’s take a look at how he breaks it down:

• Start with value, not just cost

Many organisations begin with the savings: fewer people, lower costs. Vietro encourages them to look at value instead. What better service will customers receive, and what problem is the agent solving?

• Know how many agents you have

His second priority is observability, and he offers a simple test. Can you say how many agents are running in your organisation right now? If you can, you probably have a reasonable setup. If you can’t, there’s room for improvement. He adds that the question is far from trivial to answer.

Stathopoulos says she suspects many companies don’t realise how many agents are running in the background and can’t count them. Vietro agrees that this matters well beyond performance. Observability also supports security, because you need to know what agents are doing, spot when something goes wrong and steer them back on course.

• Test until you trust

The third priority is evaluations and simulations. Parloa runs millions of them across its agents to check that they do their tasks with a given level of confidence. Vietro’s benchmark is demanding. You test until you can give customers the same quality of service an agent provides as they would get from a human.

Once that confidence exists, he says, deploying agents across enterprises becomes easier, as does replacing different work streams within an organisation.

The Platform Foundations Agents Need

If pilots work in a playground, production needs infrastructure. Burks says the current moment closely echoes the early days of cloud. Back then, accounts were spun up all over the place, there were no guardrails, and everything went everywhere. Today he sees the same thing in what he calls the “agentic brawl”. His view is that enterprises need a centralised control plane. Thoughtworks has one called AgentWorks. He frames the foundations in three parts:

1. Observability and telemetry. Agents are only as good as the real-time context they receive. Unified observability across logs, metrics and traces lets AI triage agents correlate incidents and carry out root cause analysis, including for other agents across the enterprise.
2. Governed skills. Rather than letting every team write its own isolated prompts, organisations should capture expert architectural knowledge in reusable, version-controlled skills applied across the business.
3. Governance, safety and security. A robust control plane manages model execution costs, token budgets and access controls. It also acts as an LLM-as-a-judge within evaluation loops. Burks says you can’t scale agents unless security and compliance teams have full auditability over every agentic action. Without it, budgets spiral, and security loopholes appear, just as they did in the early cloud years.

Lessons Cloud Already Taught Us

Stathopoulos says all of this gives her a feeling of déjà vu, and both guests agree the cloud era holds real lessons. Burks was in the middle of those migrations. His biggest warning is the lift-and-shift trap. Enterprises moved monolithic, brittle applications straight to the cloud without refactoring them, and ended up with the same technical debt at a different cost point.

The AI version of that mistake is easy to make. If you plug generative AI into fragmented legacy architecture, chaotic data, messy ticketing queues and poor observability, you simply produce bad outcomes faster. His advice is to modernise the underlying operational engine as you adopt the technology.

Vietro says the same thing in different words: work on your fundamentals. AI will magnify whatever it’s given. Put it into an unstructured environment, and it amplifies the bad. Build solid foundations, and it amplifies the good, helping teams deliver better software.

He also stretches the comparison. Cloud was big, but he believes today’s AI shift is closer to the arrival of the web in the 2000s, and may be even bigger. His advice for navigating it is to stay sceptical and curious in equal measure, as the industry did back then.

What getting it right looks like

After a lot of talk about what can go wrong, the conversation ends on the positive side. For Vietro, successful scaling starts with model routing, which Parloa calls its model mix. Each incoming task goes to the model with the best track record for it, based on earlier simulations and evaluations. Success is a good conversation, whether by voice or chat, handled reliably and at lower cost.

He gives a vivid example. A German-language agent serving Germany would use standard Hochdeutsch. In Switzerland, the same task would land better if the agent spoke Swiss German, with the right accent and customer warmth. Stathopoulos recognises the same effect from living in Spain, where accents and dialects vary from region to region.

Burks looks at success from the organisational side. He borrows a classic rule often credited to Jack Welch: make sure your internal rate of change matches or exceeds the rate of change outside. He says Parloa is doing this well. In managed services, he says, getting it right means clients break out of the legacy maintenance trap. The relationship moves from a vendor who reports ticket SLAs and counts to a strategic partner delivering value arbitrage. Success is measured in innovation capacity, continuous estate modernisation, lower cost to serve and guaranteed business outcomes. Thoughtworks calls this continuous operations, sustaining today’s IT while evolving it towards the transformation the client wants. To learn more, visit thoughtworks.com or follow Italo Vietro and Joshua Burks on LinkedIn.

Takeaways

• AI's impact on technical teams and workflows.
• Transitioning from pilot to production for AI agents.
• Platform foundations necessary for scaling AI.
• Security and governance in AI deployment.
• Lessons from cloud migration applicable to AI.