Part 1: Introducing the CloudOps Competency Benchmark
Governed AI is the standard an agentic system has to meet before it acts inside production. The moment an AI system moves from describing what should happen to deciding and acting on live infrastructure, the risk changes. A polished demonstration is no longer enough. The system has to show what it can do, where it can do it, how much human help it needs, and what evidence supports the outcome. That is why we built the CloudOps Competency Benchmark (CCB), a vendor-neutral framework for measuring whether an AI agent has actually earned the operational authority it is being trusted with.
Key takeaways
- Governed AI means an agent’s authority is explicit, bounded, and backed by evidence, not assumed from a good demo.
- The CloudOps Competency Benchmark (CCB) is a vendor-neutral framework, built by MontyCloud with researchers at IIIT Hyderabad, for grading whether an agentic system can do CloudOps work competently and how much authority it has earned in a defined operating domain.
- CCB defines five levels of earned autonomy, each with a measurable Human Intervention Budget (HIB).
- It scores the work, not only the answer, and disqualifies six critical safety violations outright.
- The right autonomy level is the highest one a system can safely sustain for the specific work, in a specific environment, against evidence.
What is governed AI in CloudOps?
Governed AI in CloudOps is an operating model in which an AI agent’s identity, permissions, scope, and actions are explicit and verifiable, so the agent can act on cloud infrastructure only within authority it has demonstrably earned. A production agent should look less like an LLM with API access and more like a governed operational system. Identity, permissions, tenant isolation, policy enforcement, approval boundaries, rollback, audit history, and outcome verification all have to be explicit.
Autonomy, in this model, is operational authority. Governance is what decides how much of that authority an agent is allowed to hold, and evidence is what earns it.
What is the CloudOps Competency Benchmark (CCB)?
The CloudOps Competency Benchmark (CCB) is a vendor-neutral evaluation framework, developed by MontyCloud with researchers at IIIT Hyderabad, that determines whether an agentic AI system can perform CloudOps work competently and how much operational authority it has earned within a defined operating domain.
CCB is not a MontyCloud product feature. It evaluates a specific system performing specific work under specific operating conditions and authority boundaries.
To our knowledge, CCB is the first open benchmark framework designed to evaluate end-to-end CloudOps competence across the full breadth of operational work and across graduated levels of autonomy.
How did we get here, from MOYA to Beyond Task Completion?
CCB grew out of practical work and deep research. Our collaboration with Dr. Karthik Vaidhyanathan and researchers from the Software Architecture for Sustainability group at IIIT Hyderabad began with MOYA, our multi-agent orchestration framework for CloudOps.
MOYA showed how a complex operational objective could be divided across specialized agents and coordinated through a meta-orchestrator. It was presented as a full paper at the International Conference on AI Engineering (CAIN), co-located with ICSE 2025, and was named a Distinguished Paper Award candidate at CAIN 2025. We later brought that work to more than 100 students through a joint hackathon at IIIT Hyderabad.
We then worked together on Beyond Task Completion, an end-to-end assessment framework for evaluating agentic systems across the model, memory, tools, and operating environment.
That research made one point clear. An agent can complete a task while still failing in ways that matter in production. It may retrieve the wrong context, use the wrong tool, violate a policy, or reach the right answer through an invalid reasoning path.
At the same time, we were building the practical infrastructure around these systems. We developed a CloudOps MCP Server so agents could work across the tools operations teams already use. We built a CloudOps Assistant and invested in multi-agent coordination. We also developed patented approaches to cloud infrastructure management and anomaly detection.
Each of these investments expanded what an agent could do inside a customer environment. CCB addresses the next question: how much authority should that agent be trusted to hold?
Why does governed AI matter now?
Agentic systems have crossed the line from recommending to acting, and that changes the risk category. Cloud automation has acted on infrastructure for years through runbooks, policy engines, autoscaling, infrastructure as code, and event-driven remediation. That automation generally followed predefined rules and paths.
Agentic systems change the boundary. They can interpret context, choose tools, construct multi-step plans, adapt those plans as conditions change, and attempt actions that were not explicitly scripted in advance. The same objective may produce a different reasoning path, tool choice, or action sequence on another run. That variability is part of the value of agentic AI. It allows a system to respond to situations that predefined automation cannot fully anticipate.
The operational standard still applies. Model capability is only one part of production competence. The complete result depends on the context available to the system, the tools it can use, the controls around those tools, and the workflow in which it operates. That is why a general model benchmark cannot tell us whether a system is ready to perform CloudOps work. The evaluation has to be specific to the domain and the job.
What are the five levels of earned AI autonomy?
CCB defines five levels of CloudOps autonomy, each describing a different distribution of work between the system and the human operator. Each level also includes a measurable Human Intervention Budget (HIB), which records how often the system requires human help during an evaluation.
Level 1: Assistive (information retrieval). The system retrieves and presents information. The human directs the workflow. Example: Show me which storage buckets have public access enabled.
Level 2: Analytical (independent analysis). The system independently analyzes the environment, identifies patterns, and explains what it found. Example: Assess our storage security posture and identify the material gaps.
Level 3: Advisory (recommendation). The system recommends what should be done, including priorities, tradeoffs, and an actionable plan. The human decides whether to proceed. Example: Create a remediation plan that balances security risk, operational impact, and implementation effort.
Level 4: Semi-Autonomous (bounded execution). The system executes approved types of changes within a defined scope and control envelope. Example: Apply this reversible security remediation to the approved resources and verify the result.
Level 5: Autonomous (goal-driven, closed-loop operation). The system continuously observes the environment, decides what to do, takes permitted actions, checks the results, and repeats the cycle without requiring human approval at every step. Example: Maintain an approved security posture within defined policies and authority boundaries.
The level is only meaningful when it is tied to a defined operating domain. A system is never simply “Level 4.” It may demonstrate Level 4 capability for a reversible security remediation in a sandboxed AWS environment while remaining Level 2 for incident response in production. Both can be true at the same time.
The operating domain defines where the claim applies: the task, the cloud environment, the resources, the permitted actions, and the level of risk. A higher autonomy level is also not automatically better. The right level is the highest one the system can safely sustain for the work it is being asked to perform.
What does CCB measure that task-completion scores miss?
CCB scores the work, not only the answer. Too much agent evaluation comes down to a single question: did the task get done? Did the agent find the misconfiguration? Did it produce the cost report? Did it close the ticket? That matters, but it only tells part of the story. A task-completion score does not tell us what the agent inspected, whether the evidence supported its conclusion, whether it stayed inside its authority, or whether it would behave consistently on another run.
Two failure modes matter in particular.
The first is evidence fabrication. The agent makes a claim about the environment that cannot be supported by anything it retrieved.
The second is what we call hallucinatory convergence. The agent reaches the correct, environment-specific conclusion without retrieving the evidence needed to establish it. In plain English, it got the right answer without doing the work required to justify it. A simple pass-or-fail metric can record hallucinatory convergence as a success. Operationally, it is a system that happened to be right and cannot show why. A system that cannot show why it was right cannot be trusted to act when being wrong has consequences.
CCB addresses this in several ways. The framework contains more than 600 scenarios across 13 CloudOps categories and four difficulty tiers. The scenarios cover security, cost optimization, reliability, compliance, incident response, observability, change operations, and the wider work operations teams are accountable for. Leading managed services providers Rackspace and nClouds reviewed the scenario corpus and helped us ground it in operational realism and autonomy-level calibration.
CCB also separates safety gates from competency scoring. Six critical violations cause immediate disqualification: evidence fabrication, privilege escalation, unauthorized destructive writes, disabling mandatory controls, irreversible destruction without rollback, and cross-tenant data exposure. Serious violations, such as scope overreach, constraint violations, excessive human intervention, and incomplete evidence chains, trigger mandatory score penalties. Critical failures end the evaluation. Serious failures cost points.
Across 16 dimensions, CCB evaluates what the system found, the evidence it used, how it reasoned, whether it followed policy, whether it stayed within scope, and, at higher autonomy levels, whether it executed safely and achieved the intended result. The evaluation records what the system queried, which tools it used, what those tools returned, what actions it attempted, and what changed.
What did the first CCB evaluation reveal?
We evaluated one of our own agents in the initial study, across 40 AWS scenarios at Levels 1 and 2, focused on assessment and analysis rather than autonomous remediation. At Level 1, our agent scored 12.8% on Answer, 48.7% on Fidelity, and 93.3% on Output.
Those measures gave us three different signals. Fidelity showed how well the findings were supported by evidence. Output measured whether the system returned the required response structure. Answer is the one worth explaining. Our agent works through a compliance layer that reports aggregate check counts rather than individual resource identifiers. When a scenario asked it to name the exact buckets or security groups involved, it often could not produce an ID, even when it had correctly flagged the underlying violation. That is a tool-coverage limitation more than a reasoning failure, and it accounts for most of the gap in the Answer score.
A single pass-or-fail score would have obscured all of this. The capability profile gives us a more useful engineering direction: improve discovery and tool coverage while continuing to strengthen evidence discipline and response reliability.
This is how we intend to use CCB. Not as a one-time score, but as a continuous development loop. A missed resource may point to a tool-coverage gap. An incomplete evidence chain may indicate a retrieval or context problem. Inconsistent behavior may call for changes to the model, prompts, orchestration, or workflow design.
The first evaluation was deliberately limited to Levels 1 and 2 on AWS. It does not establish performance at higher autonomy levels. Those levels require sandboxed execution, rollback, state verification, and stronger safety controls, which we will evaluate separately.
How do you start governing an AI agent? Begin with one workflow.
CCB is not only for comparing vendors. It can help an MSP, a Cloud Center of Excellence, or an internal platform team decide how much authority to grant an agent. The practical way to begin is with one workflow.
- Define the work. Choose a specific task, such as cost assessment, security posture review, incident analysis, or reversible remediation.
- Define the operating boundary. State the cloud environment, resource scope, permitted tools, authorized actions, and risk level.
- Evaluate the current level. Test what the system can observe, analyze, recommend, or execute without exceeding its authority.
- Inspect the evidence. Review what the system retrieved, how it reached its conclusion, where it required human help, and whether the outcome was verified.
- Expand authority only after it is demonstrated. A successful analytical evaluation does not automatically justify execution access.
To get started, read the CCB paper, review the initial evaluation resources, or request a demo to talk through applying the framework to one of your own workflows.
What comes next
In Part 2, we will explain how the scenario corpus was constructed, how the evaluation works, and what the first study found, including where our own system performed well and where it did not.
AI systems will receive more operational authority over time. The question is whether that authority will be granted because a demonstration looked convincing or because the system proved it was ready. We believe it should be proved. Autonomy in CloudOps should be earned level by level, within a defined operating domain, against evidence, in the open.
My thanks to Rackspace and nClouds. Both teams reviewed the scenario corpus and calibrated it against real operational work, and CCB is materially better for it.
Frequently Asked Questions (FAQs)
What is governed AI?
Governed AI is an approach to deploying AI agents in which their identity, permissions, scope, and actions are explicit, bounded, and verifiable, so the agent acts only within authority it has demonstrably earned. In CloudOps, that means an agent behaves less like an LLM with API access and more like a governed operational system, with tenant isolation, policy enforcement, approval boundaries, rollback, audit history, and outcome verification all made explicit.
What is the CloudOps Competency Benchmark (CCB)?
The CloudOps Competency Benchmark is a vendor-neutral evaluation framework, developed by MontyCloud with researchers at IIIT Hyderabad, that measures whether an agentic AI system can perform CloudOps work competently and how much operational authority it has earned within a defined operating domain. It is a framework, not a MontyCloud product feature.
What are the five levels of AI autonomy in CCB?
CCB defines five levels: Level 1 Assistive (information retrieval), Level 2 Analytical (independent analysis), Level 3 Advisory (recommendation), Level 4 Semi-Autonomous (bounded execution), and Level 5 Autonomous (goal-driven, closed-loop operation). A system can sit at different levels for different tasks and environments.
What is a Human Intervention Budget (HIB)?
The Human Intervention Budget is a measurable record of how often an agent requires human help during an evaluation. It quantifies the division of work between the system and the human operator at a given autonomy level.
What is hallucinatory convergence?
Hallucinatory convergence is when an agent reaches the correct, environment-specific conclusion without retrieving the evidence needed to establish it. The answer is right, but the system cannot show why, so a simple pass-or-fail metric can wrongly record it as a success.
Is CCB a MontyCloud product?
No. CCB is an open, vendor-neutral benchmark framework. It evaluates a specific system performing specific work under specific operating conditions and authority boundaries, and it can be applied to any agentic system, including ones not built on MontyCloud.