Part 2: How domain-specific evaluation turns agent performance into defensible operational authority
In Part 1, we shared our opinionated position that autonomy in CloudOps is operational authority. A system should not receive that authority because it performs well in a demonstration or because a vendor describes it as autonomous. It should earn authority for specific work, under defined conditions, based on evidence.
This part goes inside the CloudOps Competency Benchmark, or CCB. It explains how we turned real CloudOps work into an evaluation corpus, how CCB examines the work behind an answer, and what our first Level 1 and Level 2 evaluation taught us.
There is also a larger point behind the mechanics.
A production Agentic AI system should be built as a governed operational system rather than a model with broad API access. The platform defines what the agent is permitted to do. Domain-specific evaluation helps determine what the agent has demonstrated it can do competently inside those boundaries.
Both are required. Guardrails make action possible. Measurement makes the resulting authority defensible.
Read the paper and explore the implementation
The CCB paper and the Autonomous CloudOps Evaluation Bench (ACE Bench) repository are both available.
Download the CCB paper for the complete framework, autonomy model, scenario-construction methodology, and initial evaluation. Explore the ACE Bench repository for the scenario gold labels, the CloudFormation stacks that build the evaluation environment, and the results from the first study.
The public release covers the 40 Level 1 and Level 2 scenarios used in that study. The rest of the corpus is available to researchers on request. We are sharing this so practitioners, researchers, MSPs, and agent builders can inspect the work directly and help us strengthen it.
Key takeaways
- CCB has three separate parts: the evaluation framework, the scenario corpus, and ACE Bench, the reference evaluation harness.
- The corpus contains more than 600 scenarios across 13 CloudOps categories and four difficulty tiers, developed through a five-stage process involving practitioners, multiple models, Cloud SMEs, and external MSP review.
- Every CCB scenario is an evaluation contract. It defines the work, operating conditions, available tools, expected evidence, safety boundaries, and criteria for success.
- CCB applies critical and serious safety gates before evaluating the system across the competency dimensions relevant to the autonomy level being tested.
- The first empirical study covered 40 AWS scenarios at Levels 1 and 2. It showed why finding the correct answer and producing evidence-backed work must be measured separately.
Measurement is the second half of trust
Cloud control planes are already distributed systems. They contain asynchronous operations, stale observations, retries, partial failures, and eventual consistency. They also expose explicit APIs, permission models, state transitions, and outcomes that can often be verified programmatically.
A production platform provides the operating envelope through scoped identity, least-privilege access, tenant isolation, policy enforcement, approval boundaries, tool controls, rollback, audit history, and state verification.
A benchmark does not replace those controls. It answers a different question: given those controls, how much judgment and operational authority has the complete system demonstrated it can handle? That is why evaluation belongs alongside the production architecture, as part of how authority is granted, expanded, monitored, and, when necessary, reduced.
Why domain-specific evaluation matters
General model evaluations are useful for understanding the underlying model. Product teams also need component-level tests for retrieval, memory, tool use, policy enforcement, and output quality.
Those measures do not establish whether the complete system can perform a CloudOps job.
CloudOps is a high-context, tool-mediated discipline. A security assessment may require the system to identify the right account, inspect several related services, understand customer policy, distinguish a true violation from an accepted exception, and produce evidence that an operator can verify.
A simple task-completion score can miss important differences.
An agent may name the correct resource but use evidence that does not support the finding. It may produce a plausible recommendation after inspecting only part of the environment. It may remain correct on one run and behave differently on another.
Part 1 introduced hallucinatory convergence, where a system reaches the correct environment-specific conclusion without retrieving the evidence needed to establish it. The important addition here is how CCB detects it. The evaluation captures what the system actually queried, what evidence it retrieved, which tools it used, and how those inputs relate to the final conclusion.
This is the difference between evaluating the answer and evaluating the work.
What CCB borrows from safety-critical systems
Automotive and aviation have not solved autonomy, and neither maps directly to CloudOps. They do provide useful precedents.
SAE J3016 gave the automotive industry a shared vocabulary for graduated levels of driving automation. Aviation developed rigorous assurance practices around software, operating boundaries, failure modes, human responsibility, and evidence. NIST’s ALFUS work treated autonomy as multidimensional, taking into account the complexity of the task, the complexity of the environment, and the degree of human involvement.
The useful lesson for CloudOps is that an autonomy claim becomes meaningful only when the operating conditions, division of responsibility, permitted authority, and evidence supporting the claim are explicit.
CCB applies that principle to CloudOps. A system does not receive one autonomy label that follows the product everywhere. It demonstrates a level for a defined class of work, in a defined environment, with a defined set of permissions and risks.
CCB has three parts, kept separate on purpose
CCB is an evaluation framework with three related but distinct parts.
The CCB framework defines competence. The framework defines the five autonomy levels, Operational Design Domains, Human Intervention Budgets, safety and integrity gates, scoring dimensions, and evaluation rules. It describes what must be measured before a system can make an evidence-based autonomy claim.
The scenario corpus defines what systems must prove. The corpus contains the CloudOps tasks against which systems are evaluated. Each scenario specifies the work, context, evidence requirements, operating boundaries, and expected outcome. The corpus currently contains more than 600 scenarios across 13 CloudOps categories and four difficulty tiers.
ACE Bench runs and instruments the evaluation. ACE Bench is the reference evaluation harness. It presents the scenario, captures the resulting evidence and actions, and applies the scoring workflow.
Keeping the three parts separate supports vendor neutrality. The framework does not depend on one product architecture. The corpus is intended to represent real CloudOps work rather than one vendor’s feature set. The harness provides a reference implementation without making the framework dependent on that implementation.
How real CloudOps work became an evaluation corpus
The corpus spans work performed by Cloud Centers of Excellence, MSP operations teams, cloud-focused SRE teams, security teams, and FinOps practitioners.
The 13 categories cover areas such as operational excellence, reliability and resilience, security and risk reduction, performance efficiency, cost optimization, compliance and governance, incident response, remediation and change operations, observability, infrastructure management, workload readiness, continuous improvement, and stakeholder communication.
The corpus was developed through five stages.
Stage 1: Independent seed authoring. The process began with independently authored scenarios for a representative CloudOps category and resource set. One starting point was formal cloud-provider guidance and Well-Architected practices. Another was direct operational experience delivering CloudOps at MSP scale. The authors worked independently so that the initial set would not reflect only one source or one practitioner’s assumptions.
Internal Cloud SMEs also reviewed one of the seed sets as they would review material for a customer engagement. This helped ground the scenarios in real operational expectations before they became benchmark content.
Stage 2: Reference-set consolidation. The independent seed sets and SME feedback were consolidated into one reference set. This established the standard for scenario structure, specificity, operational realism, and coverage. It also provided a pattern for expanding the corpus into additional CloudOps categories.
Stage 3: LLM-assisted expansion and decomposition. Multiple frontier model families were used to generate candidate scenarios for additional categories and resource types. The models also helped decompose broad operational requests into atomic tasks that could be scored precisely.
For example, “verify that this storage service is secure” may combine encryption, access control, logging, retention, and network exposure. If those concerns remain bundled into one scenario, partial success becomes difficult to measure. Separating them makes it possible to identify exactly what the system inspected and what it missed.
The model outputs were treated as candidate material. They did not enter the corpus without human review.
Stage 4: Cross-model reconciliation and SME review. Candidates from the different model families were compared and reconciled. Internal Cloud SMEs then reviewed them for operational realism, missing coverage, duplication, unclear boundaries, and autonomy-level assignment. Multiple review passes were used to resolve disagreements and bring the scenarios into a consistent structure.
The purpose was to expose variation across models, use it to broaden the candidate set, and then apply human judgment rather than letting several models vote on what should be included.
Stage 5: External practitioner review. Rackspace and nClouds reviewed the complete corpus for operational realism and autonomy-level calibration. Their feedback led to targeted refinements, particularly around the boundary between Level 3 advisory work and Level 4 bounded execution. They also helped clarify scenarios whose wording could be interpreted in more than one way.
This external review mattered because a benchmark for CloudOps should reflect how the work is performed in real managed environments, not only how researchers or one product team describe it.
A scenario is more than a prompt
A CCB scenario is an evaluation contract. It identifies:
- The CloudOps task
- The cloud environment and resource scope
- The autonomy level being tested
- The tools and context available to the system
- The actions the system is permitted to take
- The required output
- The resources that should be identified
- Correctly configured resources that should not be flagged
- The evidence expected from competent work
- The safety and integrity conditions that apply
- The criteria for a successful outcome
This structure makes failures easier to diagnose.
Consider a storage-security scenario asking a system to identify resources that violate a policy. The scenario defines the account and resources in scope, the APIs that may be queried, the policy being evaluated, the affected resources that should be found, the correctly configured resources that should be left alone, and the evidence required to support the finding.
If the system fails to name an affected resource, the evaluation can examine why.
The system may have had access to the required data and failed to find it. That points to investigation or reasoning. The available tool may have returned only an aggregate control result rather than resource-level details. That points to tool coverage.
The end-to-end result is still incomplete. The scenario contract helps identify which part of the system needs attention.
How CCB verifies the work behind the answer
A CCB evaluation begins by defining the Operational Design Domain, or ODD.
The ODD identifies the category of work, cloud provider, environment, resources, task complexity, risk tier, permitted tools, and authorized actions. An autonomy result is valid only inside that declared domain.
The evaluation then runs the system through an instrumented boundary based on the Agent-Cloud-Interface pattern introduced by AIOpsLab.
The interface records:
- What the system queried
- Which tools it selected
- What those tools returned
- What evidence was used
- What errors occurred
- When the system requested human input
- What actions it attempted
- What changed in the environment
The resulting record allows an evaluator to inspect the process independently from the final report.
Before competency scoring, CCB applies safety and integrity gates.
Six critical conditions cause immediate disqualification. They include evidence fabrication, privilege escalation, unauthorized destructive writes, disabling mandatory controls, irreversible destruction without a rollback path, and cross-tenant data exposure.
Six serious conditions apply mandatory penalties. They include scope overreach, constraint violations, exceeding the Human Intervention Budget, incomplete evidence chains, hallucinatory convergence, and non-convergent processing.
The distinction matters. A strong answer does not compensate for an authority violation.
After the gates, the system is evaluated across the competency dimensions that apply to the autonomy level being tested. These dimensions cover outcome quality, reasoning, evidence use, policy alignment, safety, execution, consistency, communication, and efficiency.
Not every dimension applies at every level.
At Levels 1 and 2, the system has no execution authority. The evaluation focuses on retrieval, completeness, independent analysis, evidence discipline, and response quality.
At Level 3, the system must also demonstrate that its recommendations are contextual, prioritized, actionable, and clear about tradeoffs.
At Level 4, the evidence requirements expand to include authorization, preflight checks, bounded execution, state changes, rollback readiness, and outcome verification.
At Level 5, evaluation must establish sustained closed-loop behavior. The system must continue to observe, decide, act, and verify while maintaining its goals and safety constraints as conditions change.
This is why the higher levels cannot be established through a better answer alone. More authority requires more evidence about how the system behaves.
Multi-agent systems are evaluated as one operational system
CCB evaluates a multi-agent architecture at its primary customer-facing boundary.
A buyer or operator may interact with one assistant, even though several specialized agents work behind it. The composite system is responsible for the coherence of the evidence, the actions taken, the conflicts resolved, the escalations presented, and the final outcome.
Consider a cost agent that identifies an idle compute instance as a termination candidate. A security agent may have quarantined the same instance because it contains evidence needed for an active investigation.
Both agents may be correct within their own domains. The combined system is operationally wrong if it terminates the instance.
The orchestrator has to recognize the conflict, preserve the security state, prevent the cost action, and explain the decision.
This is one reason competence has to be measured at the system boundary. Testing the two agents independently would not expose the failure created by their interaction.
The current CCB design evaluates the observable behavior of the composite system. It does not yet directly score every internal decision involved in decomposition, agent selection, or response aggregation. Deeper orchestration evaluation remains an area for future development.
What the first evaluation tested
The first empirical study was deliberately limited.
It evaluated five systems across 40 AWS scenarios at Levels 1 and 2, covering eight CloudOps categories. The study focused on read-only assessment and analysis. It did not evaluate autonomous remediation, production write access, rollback, or closed-loop operation.
Four measures were reported.
Answer measured whether the system surfaced the correct cloud resource identifiers.
Fidelity measured whether its findings were supported by the evidence and reasoning captured during the evaluation.
Safety, for applicable Level 1 scenarios, measured restraint. It examined whether the system avoided recommending action on resources that were already correctly configured.
Output measured whether the system returned a valid response in the required structure.
These measures answer different questions. Combining them into a single pass-or-fail result would make it difficult to tell whether a system failed because of incomplete discovery, weak evidence, inappropriate recommendations, or an unreliable interface.
What the first evaluation taught us
The clearest result was that Answer and Fidelity measured different qualities, and that the difference changed the ranking.
One specialized agent surfaced correct resource identifiers at nearly five times the rate our own agent did at Level 1. On Fidelity the ranking reversed. Our agent’s conclusions traced back to what its tools actually returned. The specialized agent’s higher recall was not always supported by reasoning the evaluation could verify.
A benchmark that scored answers alone would have ranked the specialized agent first, even though it offers weaker operational guarantees.
The pattern runs the other way too. A system can reason carefully over the information it has and still miss affected resources because its tools do not expose the required details.
We also evaluated one of our own agents.
At Level 1, our agent scored:
- 12.8% on Answer
- 48.7% on Fidelity
- 93.3% on Output
At Level 2, it scored:
- 24.7% on Answer
- 65.7% on Fidelity
- 92.5% on Output
The Answer result exposed a practical limitation in resource-level discovery.
Our agent worked through a compliance layer that frequently returned aggregate check results rather than the individual resource identifiers required by the Answer measure. It could identify that a control had failed without always being able to return every affected bucket, instance, or security group.
That is an end-to-end system limitation. Even when the underlying issue is tool coverage rather than reasoning, the operator still needs the exact resource to act.
Fidelity rewarded the agent for being explicit about that limit. In one scenario our agent stated directly that the compliance layer could not surface specific bucket names from aggregate check data, delivered partial findings within that constraint, and was scored accordingly. Runs that invented plausible-sounding resource identifiers not present in any retrieved data were penalized.
The evaluation gives us a clear engineering direction. We need to improve resource-level discovery and tool coverage while continuing to strengthen evidence discipline and response reliability.
This is how we intend to use CCB, as a continuous development loop rather than a one-time test at the end of development.
A missed resource may point to tool coverage. An unsupported finding may indicate retrieval or reasoning. A malformed result may expose interface reliability. An unnecessary escalation may reveal weak confidence calibration or orchestration.
A domain-specific evaluation turns those failures into a development roadmap.
What the study does not establish, and what comes next
The initial study does not validate the complete five-level framework. It covers AWS only and evaluates Levels 1 and 2. It does not establish Level 3 advisory competence, Level 4 execution safety, or Level 5 closed-loop operation.
Those levels require additional evaluation infrastructure. Level 3 needs realistic business context and tradeoff analysis. Level 4 needs resettable environments, scoped write access, policy-gated execution, rollback, and post-action state verification. Level 5 needs sustained operation over time while the environment changes. Azure and Google Cloud implementations also remain future work.
These are the more demanding parts of the benchmark, because the consequences of failure increase as the system receives more authority.
The next phase of CCB therefore extends across three dimensions: evaluating higher levels of autonomy, broadening the implementation across cloud providers and operating environments, and strengthening the assurance process around high-autonomy claims, including the role of independent domain experts.
From evaluation evidence to operational authority
The practical purpose of CCB is to support a bounded authority decision rather than to attach one score or one autonomy label to a product.
An MSP might determine that a cost-assessment workflow has demonstrated Level 2 competence in production AWS accounts. A security workflow might eventually demonstrate Level 4 competence for a defined set of reversible changes, under specific policies, in a controlled environment. An incident-response workflow might remain at Level 1 or Level 2 until the organization has established enough context, isolation, rollback capability, and operating evidence to grant it more authority.
The same system can therefore earn different levels for different categories, customers, environments, and risk conditions.
This gives MSPs and Cloud Centers of Excellence a practical way to answer questions that broad autonomy claims do not answer:
- What can the system observe?
- What can it analyze independently?
- What can it recommend?
- What can it execute?
- Where is human approval still required?
- What is explicitly outside its authority?
This is also where I believe the role of domain experts will evolve.
The current CCB framework evaluates the system’s demonstrated behavior. It does not define a separate human certification process for Levels 4 and 5. As the framework matures, I expect qualified CloudOps experts to become increasingly important as the approvers and managers of high-autonomy systems.
Once a system moves beyond Level 3 and begins changing cloud state, an automated evaluation should not be the only basis for granting production authority. A domain expert should be able to review the operating scope, evidence, failure behavior, rollback readiness, and residual risk before approving the claim.
That review would be separate from the Human Intervention Budget. The HIB measures how often the agent needs human help while performing the evaluated task. Expert review would be an assurance and governance step used to decide whether the evidence is strong enough to support a higher-autonomy claim.
The organization that owns the environment would still make the final decision about production authority.
This is not part of the current CCB specification. It is a direction we believe responsible Agentic AI practice will need to take as evaluation advances into Levels 4 and 5.
Closing
CCB was developed with Dr. Karthik Vaidhyanathan and researchers from the SA4S group at the Software Engineering Research Center at IIIT Hyderabad. Rackspace and nClouds reviewed the complete scenario corpus for operational realism and autonomy-level calibration. I am also grateful to MontyCloud’s Cloud SME, product, AI, and engineering teams, who turned the framework into an executable evaluation and used the results to improve our own system.
The model is only one component of production Agentic AI. Guardrails make action possible by defining identity, scope, policy, and rollback. Domain-specific evaluation makes authority defensible by showing how the complete system behaves while doing real work. CCB connects those responsibilities.
Autonomy should be earned within a defined operating domain, supported by evidence, and granted by the people accountable for the outcome.
Read Part 1 for the CCB introduction and the five autonomy levels. Teams interested in applying CCB to a workflow, contributing scenarios, or challenging the methodology are invited to work with us.
Frequently asked questions
- What are the three parts of CCB?
CCB is the evaluation framework. The scenario corpus defines the CloudOps work and evidence requirements used in the evaluation. ACE Bench is the reference harness that runs the scenarios, captures the system’s activity, and applies the scoring workflow.
- How were the CCB scenarios created?
The scenarios were developed through a five-stage process involving independent practitioner-authored seeds, reference-set consolidation, LLM-assisted expansion and decomposition, internal Cloud SME review, and external review by Rackspace and nClouds.
- What does the Agent-Cloud-Interface record?
The interface records the system’s tool calls, retrieved evidence, errors, human-intervention requests, attempted actions, and relevant state changes. This allows the evaluator to inspect the work behind the final answer.
- How does CCB evaluate a multi-agent system?
CCB evaluates the composite system at the primary customer-facing boundary. The system as a whole is accountable for its evidence, conflict resolution, escalations, actions, and final outcome, regardless of how many internal agents participated.
- Can CCB be used outside AWS?
The framework is designed to be cloud-provider neutral. The initial empirical implementation is on AWS. Equivalent Azure and Google Cloud implementations remain future work.
- Does passing CCB automatically grant an agent production authority?
No. CCB produces evidence about demonstrated competence within a defined operating domain. The organization that owns or manages the environment remains responsible for deciding what production authority to grant.