PAteam Logo
PAteam Logo

Author name: Shruti Sharma

How To Evaluate Agentic Ai Vendors For Cx
Blog

How to Evaluate Agentic AI Vendors for CX (Beyond the Demo)

/

Four vendors on the shortlist. Four demos in eight days. Every one of them impressive, articulate, and apparently capable of resolving a customer inquiry end to end without human involvement. The CIO closes the laptop after the last session and realises they have learned almost nothing that distinguishes one from another. What they actually need to know is which of these four will still be working in March, once it has met the CRM that has been running since 2011 and the ticketing system nobody has fully documented. That gap between the demo and day one is where agentic AI programs are won or lost. Knowing how to evaluate agentic AI vendors for CX is now one of the highest-stakes procurement decisions in the enterprise, and the evaluation criteria most organisations use are borrowed from a software buying process that does not apply. You are not buying a product. You are buying a system that will make autonomous decisions on your behalf, in your environment, against your data. Here is what we actually think, based on delivering agentic AI in production: most agentic AI vendors will show you a demo that works perfectly in a clean environment. The only question that matters is what happens when it meets your actual systems on day one. Everything else in the evaluation is secondary to that. This post gives you the questions that expose the difference. Why every agentic AI demo looks the same, and why that is the problem A demo environment is a controlled environment. The data is clean, the integrations are stubbed or purpose-built, the customer intents are the ones the vendor has already trained against, and the edge cases have been quietly removed from the script. Nothing about that is dishonest. It is simply what a demo is. The problem is that your environment is the opposite of that. Your customer data lives across three systems with inconsistent formatting. Your CRM has custom fields that were added six years ago by someone who has since left. Your contact reasons include a long tail of genuinely unusual requests that do not appear in any vendor’s training set. Your ticketing system has an API with undocumented rate limits. An agentic AI system that performs beautifully in a demo and badly in your environment has not malfunctioned. It has simply encountered the reality it was never tested against. This is why the most valuable thing you can ask a vendor to do during evaluation is not a demo at all. It is a proof of concept against your data, in your environment, with your worst-case contact scenarios included deliberately. Learn how PAteam deploys agentic AI in production environments. The seven questions that separate a real agentic AI partner from a good demo These are the questions that consistently expose capability gaps during evaluation. Ask them in this order. Can you run a proof of concept against our data, in our environment, on a use case we choose? Not a sandbox. Not a curated dataset. If a vendor resists this or wants to select the use case themselves, that resistance is your answer. What happens when the AI agent cannot resolve the contact? Listen specifically for how escalation to a human is designed. The handoff is where customer experience actually breaks. A vendor who has thought hard about escalation has thought hard about production. How does the system handle a contact type it has never seen before? The long tail is where autonomous systems fail most visibly. A vendor who says the model “handles it” has not been in production long enough. What is your governance model, and who is accountable when the agent gets something wrong? This is the question most vendors are least prepared for. There must be a clear framework covering decision logging, human review thresholds, escalation authority, and remediation. If governance is a slide rather than a system, walk carefully. Show me a client deployment you still operate today, with current volume numbers. Launches are easy to point to. Sustained production volume is the actual proof. Ask for the numbers and ask how long the deployment has been running. How do you integrate with legacy systems that have no modern API? Every enterprise has at least one. The answer to this question tells you whether the vendor has done real enterprise work or only greenfield deployments. What does your post-deployment support model look like? Agentic AI systems drift. Customer behaviour changes, products change, policies change. If the vendor’s engagement ends at go-live, you are accepting all of the ongoing accuracy risk yourself. Why agentic AI implementations fail, and it is rarely the model When an agentic AI deployment underperforms, the instinct is to question the model. Wrong provider, wrong architecture, not enough training data. That is almost always the wrong diagnosis. The failures we see in the field cluster around four causes, and none of them are the model itself. Integration debt. The agent cannot reliably read or write to the systems it needs, so it either guesses or escalates everything. Containment rate stays flat and the business case collapses. Unstructured knowledge. The agent is expected to answer from a knowledge base that is out of date, contradictory, or spread across a wiki, a shared drive, and the heads of three senior agents. The agent is only ever as accurate as the knowledge it can reach. No governance framework. Nobody defined the thresholds at which a human must review a decision, nobody is auditing agent outputs, and nobody owns remediation. The first significant error becomes an executive incident rather than a logged exception. Escalation designed as an afterthought. The agent handles the easy eighty percent well and hands off the hard twenty percent badly, so the customers with the most complex problems get the worst experience. Overall CSAT drops even though containment went up. Think of it this way: deploying an agentic AI system without an integration and governance foundation is like hiring an extremely capable new employee,

Legacy Avaya Migration Challenges
Blog

Legacy Avaya Migration Challenges: Deciding How Much of the Estate to Audit

/

Two documents that disagree The runbook for a legacy contact centre platform said the estate had forty-one queues. The configuration export from the same platform, taken the same week, listed just over four hundred entries carrying routing logic — vector steps, vector directory numbers, hunt groups, announcement branches, holiday tables and the after-hours conditionals nested inside them. Neither document was wrong. The runbook described the estate as designed. The export described the estate as it had been patched, by five different administrators, over roughly a decade of Monday-morning fixes that nobody went back to document. Every serious problem in a legacy migration lives in the gap between those two documents, and the size of that gap is the single number that determines whether the project runs to plan. This is not a lift-and-shift problem or a training problem, though it shows up as both. It is a discovery problem with a price attached, and the price can be calculated before the programme starts. What is actually inside a legacy estate A legacy private branch exchange or on-premise contact centre platform accumulates configuration the way a building accumulates wiring. Some of it carries live traffic today. Some of it was built for a product that was discontinued, a campaign that ended, or an office that closed, and it survives because deleting configuration on a production switch is a risk nobody volunteers for. That distinction is the whole discovery problem, so it needs a definition tight enough to argue with. A live routing behaviour is one that at least one real interaction reached in a defined recent window — ninety days is a reasonable default, and thirteen months if the business has genuine seasonal peaks or statutory-deadline traffic. Everything else is dormant. Dormant configuration still has to be inventoried, because you cannot prove something is dormant without looking at it, but it does not have to be replicated, tested or accepted on the new platform. In the estates we have taken apart, the live share of route points typically lands somewhere between a quarter and a half. Call it a third as a planning assumption and then measure it, because the assumption is doing a lot of work: it sets the size of the target you are aiming at. Two things follow immediately, and both cut against the way migration scopes are usually written. Counting every configuration item is not the goal — replicating every configuration item certainly is not, and a scope that promises functional parity across the whole estate has promised to rebuild the dead parts. But you cannot know which parts are dead without spending money to look, and that spend is not free either. Why the documentation you already have is not the answer The artefact that would resolve this is a current business process narrative — a written description of each contact flow and its handoffs between teams, sitting alongside the diagram. Regulated organisations are expected to maintain exactly that. The FFIEC’s IT Examination Handbook notes that business process diagrams “often depict the handoff between departments that have a role in carrying out different parts of the business activity,” and that narratives “describe the entity’s business processes and related control points, often to support control reporting” (FFIEC IT Examination Handbook, Architecture, Infrastructure, and Operations III.C.3). The expectation is reasonable. What we find on site is that the narrative was accurate on the day it was signed and has not been touched since, while the switch has been edited two hundred times. Documentation ages against a system that does not stop changing, which is why discovery has to sample the live system rather than read the file. What audit coverage costs, and what missing a rule costs Here is the trade, priced. The model below is illustrative, built on planning ranges rather than a single client’s figures, and the point of publishing it is that you can substitute your own four inputs and get a defensible number rather than a vendor’s confidence. The estate has 400 configuration items carrying routing logic. Thirty-five per cent are live, so 140 live routing behaviours exist. Audit coverage, written as c, is the share of configuration items a human actually opens, traces and records. Each item examined costs 1.25 hours at the start, and marginal cost rises as coverage climbs, because the easy items are the ones with obvious names and recent edits — the last ten per cent are the undocumented overflow branches that require call-detail archaeology to interpret. Modelled as a linear rise, the average cost per item at coverage c is 1.25 × (1 + c) hours. Every live routing behaviour costs 2 hours to reproduce and accept on the new platform whether or not the audit found it, because that work is unavoidable. And every live routing behaviour first discovered in production — after cutover, by a customer hitting it — costs an additional 8 hours: triage, a change request, an out-of-hours fix, a regression test, and a conversation with whoever the misroute affected. Blended migration engineering at EUR 82 per hour. Total cost is audit hours, plus production-discovery penalty hours, plus unavoidable build hours. It has a minimum, and the minimum is not at full coverage: Coverage 0%: audit 0 h, production penalty 1,120 h, build 280 h. Total 1,400 h, EUR 114,800. Coverage 30%: audit 195 h, production penalty 784 h, build 280 h. Total 1,259 h, EUR 103,238. Coverage 50%: audit 375 h, production penalty 560 h, build 280 h. Total 1,215 h, EUR 99,630. Coverage 62%: audit 502.2 h, production penalty 425.6 h, build 280 h. Total 1,207.8 h, EUR 99,039.60 — the optimum. Coverage 80%: audit 720 h, production penalty 224 h, build 280 h. Total 1,224 h, EUR 100,368. Coverage 90%: audit 855 h, production penalty 112 h, build 280 h. Total 1,247 h, EUR 102,254. Coverage 100%: audit 1,000 h, production penalty 0 h, build 280 h. Total 1,280 h, EUR 104,960. Read the last three rows before the others.

Why Hire An Rpa Consulting Firm
Blog

Why Hire an RPA Consulting Firm Instead of Building Inhouse

/

The IT Director has just been handed a budget. Twelve months of headcount, a tooling allocation, and a board-level mandate to “stand up an internal RPA capability.” On paper, this is a win. In practice, sitting at their desk on a Thursday afternoon, they are quietly working out whether they have just inherited a problem that nobody else in the executive team wanted to solve. That feeling is the right instinct. The question of why hire an RPA consulting firm instead of building inhouse is one of the most consequential decisions a COO or CIO will make in the first phase of an automation program. The wrong answer is rarely catastrophic in month three. It is catastrophic in month eighteen, when the bots are unstable, the team is half-staffed, and the ROI case looks nothing like the slide deck that got it approved. Here is what we actually think, based on nearly a decade of enterprise delivery: building RPA inhouse sounds cheaper until you factor in the twelve months it takes to get a developer genuinely productive and the cost of the bots that break in the meantime. The hidden cost of inhouse is not the salary line. It is the time it takes to build the muscle, and what fails while you are building it. This post answers when inhouse makes sense, when it doesn’t, and what good actually looks like. What the inhouse pitch usually leaves out The inhouse business case typically lands on a CFO’s desk looking very clean. Two developers, a process analyst, a part-time architect, an annual RPA license, and a project manager. Total cost on paper looks reasonable, especially compared to a consulting proposal that runs three or four times that number on the surface. The problem is that the inhouse business case is built on the wrong assumption. The assumption is that hiring an RPA developer is the same as hiring any other developer. It is not. A genuinely productive RPA developer is not a person who knows the tool. They are a person who knows the tool, has built bots that survived three system updates, has handled production exceptions at 3am, has worked through a real PDD-to-SDD transition, and has watched at least one of their own automations fail in a way that taught them something. That person does not exist on the open market in most regions. You either hire them from another consulting firm at a significant premium, or you build them, which takes around twelve months from a strong starting point. The inhouse plan rarely accounts for the cost of the bots that get built, deployed, and broken during that twelve-month learning curve. Learn how PAteam manages production RPA at enterprise scale. The real cost of RPA managed services vs in-house team costs Most direct cost comparisons between an internal team and a managed RPA partner come out close. The line-item math looks similar. The gap is in the cost categories that never make it into the spreadsheet. There are five. Recruitment cost and time-to-hire. A senior RPA developer in most enterprise markets takes three to six months to source, interview, offer, and onboard. That is six months of program time burned before any bots are built. Multiply by every role on the plan. Productivity ramp. Even an experienced RPA developer hired into a new business takes three to four months to understand the systems, the data, the processes, and the politics well enough to build production-grade automations. A junior developer takes much longer. Attrition. RPA developers are some of the most heavily recruited technical roles in the enterprise market. The probability that the team you spent twelve months building stays intact for another twelve is low. Every replacement resets the ramp clock. Maintenance debt. Inhouse teams almost always under-resource maintenance. Build is more visible, more rewarding, and easier to plan. Maintenance is invisible until something breaks. By month eighteen, most inhouse teams are spending sixty percent of their capacity maintaining the bots they built in months one to twelve. Opportunity cost. Every month the team is ramping is a month the business is not getting the automation benefit it approved the budget for. That cost rarely appears in the inhouse case but it is real, measurable, and often the largest number of all. A managed RPA partner absorbs all five of these costs into a delivery model that is already built. The bots get built faster, by people who have done it before, with a maintenance model already running. That is the actual comparison, not the salary line. What PAteam does differently with RPA developer training This is where the delivery quality gap actually lives. Most RPA consulting firms train their developers for one to three months before putting them on client work. The math is simple. The faster the developer is billable, the better the firm’s utilization rate. The client wears the cost of that compressed training in the form of bots that break, exception handling that was never thought through, and production issues that surface in month four instead of being designed out in month one. PAteam spends six months training every RPA developer before they touch client production work. Not because we are being generous. Because we have watched, repeatedly, what happens when the training is shorter. The first ninety days cover tooling and certification. The second ninety days cover real failure modes, exception design, the operational realities of a production estate, and how to build automations that survive system updates. The result is that the developer who arrives at your business has already seen what goes wrong. They build with that experience designed in, not bolted on after the first incident. Think of it this way: hiring an undertrained RPA developer is like hiring a pilot whose flight training was rushed. The aircraft will fly in good weather. You will only find out about the gap in their training in the conditions that matter most. What to look for in an RPA implementation

Nice Cxone Certified Implementation Partner Feature Img
Blog

Why Hire a NICE CXone Certified Implementation Partner

/

Here is a claim about PAteam that is completely true and almost completely useless: our engineers hold NICE CXone certifications. It is true. It is checkable. And on its own it tells you nothing about whether your programme will land, because it is a statement about people passing exams, not about programmes reaching production. Every serious partner you shortlist will make the same claim. If a fact is shared by everyone on your list, it cannot help you choose between them. That is not an argument against certification. It is an argument about what certification is for. A credential is a floor — it removes the firms that cannot clear a minimum. What it does not do is predict outcomes, and the interesting question is what you should verify instead, and what that verification is worth. What a certification formally attests to Take the general case first, because the standards world has already thought about this carefully. ISO publishes a plain-language guide to ISO 9001, and it is unusually direct about scope. ISO’s own guide states that “ISO does not certify organizations” — certification is performed by independent certification bodies, and what those bodies audit is “the organization’s management system against ISO 9001:2015.” The guide also notes that certification is voluntary and is pursued when an organisation needs “an independent confirmation that they meet the requirements of ISO 9001:2015,” commonly in supplier approval processes and government tenders. Read that carefully. The audit is of a management system against a set of requirements. It is not an audit of any deliverable, any programme, or any customer’s outcome. A certified quality management system is evidence that the firm has documented processes and reviews them. It is not evidence that your migration will hit its date. Platform certification works the same way one level down. It attests that named individuals demonstrated defined competencies against a defined syllabus at a point in time. That is worth having — you do not want people learning the platform on your programme — but the object being certified is a person’s knowledge, not a programme’s trajectory. So the honest formulation is: certification tells you a firm has cleared a floor. Verifying delivery tells you where above that floor it sits. Those are two different tests, and only one of them is expensive. The regulation that refuses to let a credential stand in for a record Public procurement has to make this judgement thousands of times a year with money that gets audited, so it is worth seeing how it is actually written down. Under the US Federal Acquisition Regulation, before a contract is awarded the contracting officer must make an affirmative determination of responsibility. FAR 9.104-1 sets out seven general standards a prospective contractor must meet. Two of them matter here. The first is the performance record. FAR 9.104-1(c) requires that a contractor “have a satisfactory performance record” — and then adds a sentence that is easy to skim past: “A prospective contractor shall not be determined responsible or nonresponsible solely on the basis of a lack of relevant performance history, except as provided in 9.104-2.” The regulation will neither convict nor acquit on history alone. History is necessary and insufficient, both at once. The second is capability. FAR 9.104-1(e) requires the contractor to “have the necessary organization, experience, accounting and operational controls, and technical skills, or the ability to obtain them (including, as appropriate, such elements as production control procedures, property control systems, quality assurance measures, and safety programs …).” Notice what that clause is doing. Technical skills sit in a list alongside organisation, operational controls and quality assurance measures — and the clause explicitly permits “the ability to obtain them.” The skill is one item among several, and it does not even have to be resident today. A framework designed by people who award contracts for a living treats demonstrated technical competence as one of seven tests, not as the test. If the regulation that governs hundreds of billions in awards will not let a credential carry a responsibility determination, your CXone shortlist should not let it either. Why the performance record is the hard part The obvious response is: fine, check the record instead. That is the right answer and it is harder than it sounds, because performance records are systematically incomplete even in systems built and mandated to hold them. The US federal government runs a database for exactly this purpose. The Federal Awardee Performance and Integrity Information System is, in GAO’s description, “a database that provides a ‘report card’ on prospective contractors,” holding records such as whether a contract was terminated due to the fault of the contractor. Reporting into it is required by federal regulation, and reviewing it is required before awards above a dollar threshold. GAO reported in September 2024 that federal agencies entered 2,384 integrity records covering fiscal years 2019 through 2023, of which ninety percent were contract terminations. It then found the gaps: across that five-year period, “335 contract terminations and 52 administrative agreements were reported in other sources but not in FAPIIS.” At the Department of Defense, GAO found the department “had limited insight into why 166 records were not reported.” The General Services Administration cited gaps in its own guidance and limited awareness of the requirement among its personnel. For scale, GAO notes the federal government awarded contracts to about 109,000 contractors with roughly $760 billion in obligations in fiscal 2023. Sit with what that means for a buyer. A mandatory, regulated, centrally reviewed performance database is missing hundreds of terminations, and the agency responsible for the largest share of them cannot fully explain why. If that is what happens where reporting is a legal requirement, consider what the equivalent record looks like in commercial software delivery, where nobody is required to report anything and the only performance history you will see is the history your prospective partner chooses to show you. This is the actual asymmetry in vendor selection. It is not that buyers

When A Cxone Implementation Fails -Feature Img
Blog

What Happens If CXone Implementation Fails (And Why It Usually Does)

/

A prospect asked us this on a call last year, about nine months into a build with somebody else: “At what point do we stop trying to fix this and start again?” It is the right question and almost nobody asks it out loud. What gets asked instead is whether the implementation is going badly, which is unanswerable, because every implementation is going badly somewhere. The useful question is narrower and has an actual answer. There is a last month in which starting over is cheaper than carrying on. After that month, the arithmetic flips permanently, and the flip has nothing to do with how much has already been spent. This post works out where that month falls and what moves it. NICE CXone is the platform in the examples, but the structure holds for any contact-centre platform programme where configuration decisions compound. Failure does not arrive on a date The reason the question gets asked at month nine instead of month three is that platform implementations do not fail in a way that produces a notification. Consider what a public audit found when it looked at exactly this. The US Government Accountability Office examined the Internal Revenue Service’s technology modernisation programmes and reported that the agency was meeting most of its quarterly targets. Then it looked underneath the reporting. One key investment “was within schedule estimates for 2019 and 2020” while repeated changes to the overall programme plan “led to a 9-year milestone delay–from 2014 to 2023.” The quarterly reports, , “did not show this lengthy delay because they do not include programs’ historical cost and schedule goals.” Nine years of slippage, invisible in a status report that was accurate every time it was filed. Nobody falsified anything. The reporting measured performance against the current plan, and the current plan had moved. This is the mechanism that makes a CXone implementation feel fine until it obviously is not. Your weekly report says configuration is 70% complete against the plan. It does not say that the plan was rewritten in month four because the original routing design could not accommodate the third business unit, and it will not say that the workaround adopted in month four is the thing that will require rebuilding in month eleven. The same pattern shows up on the UK side, in a programme that is a much closer analogue to a contact-centre replacement. The National Audit Office reviewed National Savings and Investments — an organisation with, in its own reporting, “some 25 million customers, who have collectively invested over £240 billion” — as it replaced a twenty-five-year incumbent supplier with a multi-supplier model. The programme was rated red in the Infrastructure and Projects Authority’s annual reports for two consecutive years, red meaning “successful delivery of the project appears to be unachievable.” The NAO’s conclusion is worth reading slowly: NS&I “underestimated the scale of this challenge and overestimated its ability to deliver the Programme, despite warning signs about its lack of skills and capability and the complexity of disentangling a highly integrated IT system.” And: “The Programme was not sufficiently thought through before key decisions were made, which led to significant cost and time increases.” Those are , published November 2025. “Not sufficiently thought through before key decisions were made” is the whole failure mode in nine words. The decisions were made early. The consequences arrived late. Nothing in between reported a problem. The five things that are actually wrong Before pricing the decision, name what is being decided about. In our experience these are the design faults that surface late and cost the most to unwind, and none of them is a technology defect. Routing built around the org chart rather than the contact. Skills and queues get modelled on how the department is structured, because that is who is in the room during design. Contacts do not arrive knowing your structure, so every contact that spans two teams becomes a transfer, and transfers are invisible in average handle time because each leg gets measured separately. Integrations designed for the demo path. The connection to the customer relationship management system works for a customer record that is complete. A meaningful share of real records are not complete, and nobody profiled them before the design was signed off, so the agent desktop shows a blank panel for cases the design never contemplated. Exceptions treated as a later phase. The configuration handles the contacts that behave, and everything else is documented as “manual handling for now”. The phrase “for now” is doing enormous work in that sentence. Reporting configured to reproduce the old reports. The migration succeeds at making the new platform output the numbers the previous platform output, which means it also reproduces the previous platform’s blind spots and nobody learns anything new for eighteen months. No named owner for configuration after go-live. The partner leaves, the changes queue up with whoever is least busy, and within a year nobody can say why a particular rule exists. Each of these is cheap to change during design and expensive to change once it is load-bearing. That asymmetry is the entire basis of the calculation that follows. Pricing the window Here is the arithmetic that answers the prospect’s question. The figures are illustrative and chosen to be legible, not drawn from a client engagement; the structure is what transfers, and you should substitute your own numbers. A twelve-month programme to go-live. Build spend of €60,000 a month, covering partner fees and internal time. At some month, someone establishes that a material design decision was wrong — one of the five above. Thirty per cent of the configuration built so far sits on that decision. Two forward paths. Finish on the flawed design, then fix it live. You spend the remaining build months as planned, and after go-live you rebuild the affected configuration at 1.8 times what it cost to build the first time, because it is now in production, integrated, and being used. Add four months of

Signs Your Rpa Implementation Needs Expert Help
Blog

Signs Your RPA Implementation Needs Expert Help (Before It Collapses)

/

Start with what the failure does not look like. It does not look like an error. There is no red banner, no failed-run alert, no ticket. The scheduler says the job completed. The platform dashboard shows a run-success rate near a hundred percent, and that figure is accurate. The bot started, executed its steps, closed cleanly and logged nothing unusual. It processed four transactions. Last month it processed nine hundred. That is the shape of most RPA trouble that reaches a Chief Operating Officer’s desk — Robotic Process Automation, software that drives existing applications the way a person would, clicking through screens and filling fields. The programme is not down. It is quietly doing less than you are paying it to do, and every instrument you built to watch it is reporting green, because you instrumented the bot rather than the work. So the useful question is not “which of these seven warning signs do I have.” It is narrower and more awkward: when something goes wrong in your automation estate, who finds out first, and how long does it take? Everything expensive follows from the answer. Who tells you first: the four detection routes, cheapest to worst Sort your incidents by who raised them. The ordering is a cost ranking, and most estates discover they live in the bottom half. The process tells you. The automation itself detects that its own output is out of range — volume below a floor, a reconciliation that does not balance, a queue that should have drained and did not — and raises it. This is the cheapest route by a wide margin, because the elapsed time between failure and knowledge is close to zero and the person who receives the alert already has the context to act. A scheduled check tells you. Nobody is watching continuously, but someone or something looks at the process on a defined interval and compares what happened against what should have happened. Cost here is a function of one variable: the length of the interval. That is the whole subject of the arithmetic below. A downstream team tells you. Finance notices the month-end file is short. A team leader notices the work queue is deeper than it should be on a Tuesday. This route is expensive because the delay is unbounded — it depends on when someone happens to look — and because the failure is now a cross-team conversation with a defensive edge to it. Your customer or your regulator tells you. The most expensive route, and the one where the operational cost is the smaller part of the bill. The distinction between the first two and the last two is the distinction Google’s Site Reliability Engineering team draws between monitoring the internals of a system and testing what a user sees from outside. Their framing is worth stealing precisely: black-box monitoring “is symptom-oriented and represents active—not predicted—problems,” while white-box monitoring, which inspects logs and internal state, “allows detection of imminent problems, failures masked by retries, and so forth” (Google SRE Book, Monitoring Distributed Systems). Most RPA estates have neither. They have platform telemetry, which reports whether the robot ran — not whether the work got done. The same chapter names the four signals worth watching on any production service: latency, traffic, errors and saturation. Note that traffic — how much demand the system is actually handling — is on that list, and note that a bot processing four transactions instead of nine hundred is a traffic failure with a clean error log. If you monitor only errors, you have covered one signal in four and you have chosen the one your particular failure mode is most likely to evade. Why the estate gets more fragile the longer it succeeds The uncomfortable property of RPA is that a bot’s dependencies keep moving after the bot stops changing. A vendor patches a web application and shifts a field. Someone adds a mandatory dropdown to a form. A Citrix session renders at a different resolution. None of this is visible from inside the automation, and none of it appears in a change advisory board paper, because nobody filed a change against your bot. Google’s SRE authors named this in the context of their own tooling: automation maintained separately from the core system “suffers from ‘bit rot,’ i.e., not changing when the underlying systems change” (Google SRE Book, The Evolution of Automation at Google). They add the sharper point, which applies directly to a monthly or quarterly automation: work “that is crucial but only executed at infrequent intervals and therefore difficult to test is often particularly fragile because of the extended feedback cycle.” Read that against your own estate. Your highest-risk automations are not your busiest ones. They are the ones that run at period end, where the feedback cycle is a month or a quarter long, and where the first competent observer is the person closing the books. There is a second compounding effect, and it is organisational rather than technical. A bot built by one developer, documented in that developer’s head, becomes an asset nobody can safely modify once that developer moves on. The estate then splits into automations people will touch and automations people work around. The second category grows. What the audit evidence actually shows This is the section where most vendor writing reaches for a statistic about pilot failure rates. Those numbers circulate without denominators, so here are three findings from public audits instead, each with a stated population. Australia’s national audit office examined fourteen artificial-intelligence models the Australian Taxation Office built and deployed over roughly ten months and assessed how they were monitored in production. For one of the fourteen — seven percent — technical performance metrics had been developed. For ten of the fourteen, seventy-one percent, no monitoring metrics were developed at all. And across all fourteen, the audit office found the agency “did not provide evidence of monitoring the effect of AI models on business outcomes, such as savings to

Bot Counts Are A Vanity Metric
Blog

Bot counts are a vanity metric. Outcomes are the metric.

/

Last quarter, an automation manager walked into a leadership review with a clean slide. “312 bots live.” “64 processes automated.” “1,400 hours saved.” The room nodded. Someone even smiled. Then the operations lead asked the only question that mattered. “Great. So what changed for the customer and the team?” Silence. Not because the programme had failed, but because it had been measured like a hobby. Lots of activity, very little proof of impact. That is the trap with robotic process automation (RPA). Bot counts feel like progress because they are easy to count. Outcomes are harder, because they require you to state what problem you were solving, which workflow it lives in, what the baseline was, what changed, and what stayed messy. This is not a niche failure. In September 2024, Gartner reported that fewer than 20% of organizations had mastered the measurement of hyperautomation initiatives. Frances Karamouzis, the Distinguished VP Analyst quoted in that release, tied the problem to scope: these programmes sit inside a much larger technology roadmap, so the measurement question is rarely owned by anyone in particular. Two years on, most of the automation reviews we sit in still open with a bot count. This post is a practical way out of that. You will get the five metrics that actually tie to return on investment (ROI), what to measure when humans stay in the loop, a worked ROI model with the arithmetic filled in, and the point at which that model stops working. Why bot counts mislead smart teams Bot counts measure output from the automation team, not outcomes for the business. A single bot can be tiny, copying two fields between systems, or it can close an end-to-end case across fou2r systems. Counting both as “1 automation” is like counting “1 meeting” without caring whether it decided anything. Bot counts also hide three uncomfortable truths. Automation can move work rather than remove it. A bot speeds up step A and generates more exceptions at step B. Total handling time barely moves. The customer still waits. Automation can increase risk quietly. If a bot takes decisions without strong logging, approvals and access controls, you become faster and less audit-ready at the same time. Worth noting that the standard reference here, NIST Special Publication 800-92 on log management, was published in 2006, and a revision has been sitting in draft since October 2023. If your audit posture depends on it, read the draft rather than the twenty-year-old final. Automation can shift cost into places you are not looking. People still absorb edge cases, rework and escalation. If you do not measure that work, your ROI number is fiction. So measure the workflow, not the bot. The five metrics that actually tie to ROI ROI comes from four things: speed, quality, cost and risk. Five metrics map onto them. Each one below gets the same shape, deliberately: a definition, a formula, what to break it down by, and the failure mode it exposes. 1. Cycle time Definition. Time from request start to request completion, end to end. Not bot runtime. Formula. Completion timestamp minus request timestamp, per request type. Break it down by: request type, and by percentile. Track the median (p50) and the 90th percentile (p90), not the average. The average hides the long tail, and the long tail is what generates complaints. Failure mode it exposes. A bot finishes in nine seconds while the case takes two days, because it sits in a queue waiting for a human decision. Track only bot runtime and you will celebrate a system that is still breaching its service level agreements (SLAs). 2. Rework rate Definition. The share of cases that come back because they were done incorrectly or incompletely. Formula. Cases returned for correction divided by total cases. Break it down by: cause. Missing data, wrong routing, wrong status update. Three causes usually account for most of it. Failure mode it exposes. Rework burns the same case twice and usually crosses teams, so it rarely appears in any one team’s numbers. Reducing it is real money, unlike “hours saved.” 3. Exception rate Definition. The share of cases the automation cannot complete and hands to a human. Formula. Exceptions divided by total volume. Break it down by: workflow, reason code, system dependency, and by type, which the next section covers. Failure mode it exposes. This is the most honest metric you can track, because it measures friction rather than effort. It is also the one most often left out of the business case. 4. Success rate and run reliability Definition. How often automation runs complete as designed, and how quickly you recover when they do not. Formula. Completed runs divided by total runs. Alongside it, mean time to recover. Break it down by: failure cause. System errors, data issues and rule gaps have different owners. Failure mode it exposes. Unreliable automation creates operational drag that never appears as a cost line. Reliability also determines whether you can safely scale, so this metric governs your roadmap more than your ROI. 5. Cost per transaction Definition. Total cost to process a request class, divided by requests handled. Formula. (Human time cost + automation run cost + support and maintenance + exception handling) divided by total requests. Break it down by: before and after, for the same request class. Comparing different classes proves nothing. Failure mode it exposes. This is the one metric leadership understands without translation. It turns automation from an activity report into unit economics. What to measure when humans stay in the loop Humans do not disappear. They supervise, approve and absorb everything the rules did not anticipate. That work is where automation programmes quietly lose the savings they claim. Two things are worth knowing before you build this view. Your platform distinguishes between kinds of failure, and so should you. Microsoft’s Power Automate documentation separates business exceptions, meaning the process met a case it was not designed for, from generic exceptions, meaning something broke. These have different owners

Scroll to Top