Standards Body · Foundation paper, public edition · Released July 17, 2026
Canonical record: https://standardsbody.ai/library/foundation-paper/dynamic-evaluation-protocols/
Standards Body is an independent research and institutional-design project. It is not currently a regulator, accreditation body, certification body, or governmental authority. This document is research; it is not an adopted standard.
Series: Foundations for Frontier AI Evaluation Infrastructure
Version: 1.0
Status: Canonical working white paper
Project: Standards Body
Primary domain: standardsbody.ai
Core line: Foundations for Frontier AI
Research basis reviewed through: July 16, 2026
Document owner: Standards Body
Review cycle: Annual review, with event-triggered revision when the evaluation landscape changes materially
This paper defines the Standards Body position on dynamic evaluation protocols for frontier artificial intelligence.
It is intended to serve as:
This paper is not itself a technical standard. It does not prescribe one universal benchmark, scoring method, capability taxonomy, or legal threshold. It defines the principles, architecture, and governance conditions that should guide the development of evaluation protocols capable of remaining meaningful as AI systems, deployment environments, and societal needs change.
Frontier AI evaluation is often organized around fixed benchmarks. A dataset is assembled, a scoring rule is defined, systems are tested, and the resulting scores are compared.
This approach has produced substantial scientific value. It has made progress more legible, enabled model comparisons, exposed weaknesses, and supported reproducible research. Static benchmarks should remain part of the evaluation ecosystem.
They should not be mistaken for a complete evaluation system.
A fixed benchmark begins to lose value when:
Dynamic evaluation protocols respond to this problem by treating evaluation as a maintained institutional process rather than a one-time dataset.
A dynamic protocol can revise its:
The protocol should not change arbitrarily. Uncontrolled change can destroy comparability, reproducibility, fairness, and trust.
The central design problem is therefore not simply how to update an evaluation. It is how to update an evaluation without losing the properties that make measurement useful.
A strong dynamic evaluation protocol should preserve five forms of continuity:
Conceptual continuity
The underlying capability or risk construct remains clearly defined, even when tasks change.
Measurement continuity
Version-to-version changes are calibrated so that score differences are not confused with capability differences.
Procedural continuity
Changes follow documented governance, validation, and approval processes.
Evidentiary continuity
Historical results, uncertainty, limitations, and protocol lineage remain accessible.
Decision continuity
Users can understand whether a revised result supports the same decision, a new decision, or no longer supports the earlier decision at all.
Standards Body therefore adopts the following core position:
Frontier AI evaluation should be designed as a versioned, evidence-producing protocol with explicit maintenance, validation, governance, security, and retirement mechanisms. A benchmark may be one component of that protocol, but it should not be treated as the protocol itself.
Dynamic evaluation is not synonymous with continuously changing test questions. It is a broader institutional discipline.
It includes deciding:
This foundation is necessary because evaluation systems will otherwise become stale while continuing to look authoritative.
The danger is not only inaccurate scores.
The deeper danger is institutional confidence built on expired evidence.
Evaluation systems should evolve at a pace appropriate to the capabilities, environments, and decisions they are intended to measure.
This does not mean that evaluation protocols should change every time a new model is released.
It means that every serious protocol should possess an explicit theory of change.
That theory should identify:
A protocol without a theory of change is static by default, even when its maintainers occasionally add questions.
The ability to update an evaluation credibly is itself a form of public infrastructure.
Creating a benchmark is a research activity.
Maintaining a protocol across years, organizations, model generations, and disputes is an institutional activity.
The latter requires:
Evaluation results are time-bound claims produced under specified conditions, not permanent properties of a model.
A score is meaningful only relative to:
Dynamic evaluation makes these dependencies visible.
This paper covers dynamic protocols for evaluating:
It addresses evaluation of models and systems, including systems composed of:
This paper does not claim that:
Evaluation and monitoring overlap but are not identical.
Evaluation produces structured evidence under defined conditions.
Monitoring observes systems, environments, incidents, or indicators over time.
Monitoring can trigger evaluation revision. Evaluation can define what monitoring should watch. A mature system connects the two without collapsing them into one activity.
Red teaming is one method within a broader evaluation protocol.
It is especially useful for:
Red teaming alone may not provide:
Dynamic protocols can use red-team findings to update formal test populations.
A benchmark is a defined set of tasks, data, environments, or procedures used to measure one or more properties of a system.
An evaluation is the structured production and interpretation of evidence about a system relative to defined questions, conditions, metrics, and decisions.
An evaluation protocol is the complete specification governing how an evaluation is designed, administered, scored, interpreted, secured, reviewed, versioned, and connected to decisions.
A protocol can include one or more benchmarks.
A dynamic evaluation protocol is a protocol with explicit mechanisms for evidence-driven revision while preserving appropriate continuity, traceability, and comparability.
A dynamic benchmark is a benchmark whose tasks, examples, environments, models, sampling rules, or scoring components change over time.
A dynamic benchmark is narrower than a dynamic evaluation protocol.
Adaptive testing selects or generates later tasks based on earlier system performance.
It can improve efficiency and measurement precision, but it creates additional challenges for reproducibility and comparability.
A rolling evaluation uses a continuously or periodically refreshed test window, often based on newly created or newly available material.
Evaluation drift occurs when a protocol's meaning, task distribution, administration, scoring, or interpretation changes over time.
Drift can be intentional, accidental, beneficial, or harmful.
Protocol expiration is the point at which results should no longer be treated as current evidence for the intended decision without re-evaluation or additional justification.
Protocol retirement is the formal withdrawal of a protocol from active use because it is invalid, obsolete, insecure, superseded, too costly, or no longer decision-relevant.
An anchor component is a stable task, item family, environment, reference population, or measurement link used to compare versions.
An evaluation regime is the broader institutional arrangement within which protocols are developed and used, including evaluators, laboratories, governments, standards organizations, security systems, and decision-makers.
Static benchmarks fail in several distinct ways. These failure modes should not be reduced to contamination alone.
A benchmark saturates when top systems cluster near its maximum score or when score differences become too small to distinguish meaningful capability differences.
Saturation creates several problems:
A saturated benchmark may still be useful for:
Saturation does not make a benchmark worthless. It changes the claims the benchmark can support.
Contamination occurs when evaluation material, close variants, solutions, or derived artifacts become part of model training or tuning.
Contamination can arise through:
The central issue is not merely whether the exact item appeared in training.
Models may learn:
LiveBench was developed partly in response to contamination concerns by using frequently updated questions derived from recent sources and objective scoring where possible. Its design illustrates one approach, not a complete solution.[^livebench]
Once a benchmark becomes important, developers have incentives to improve performance on it.
This is not inherently improper.
Optimization becomes problematic when benchmark score improves faster than the underlying capability the benchmark is intended to represent.
Indirect optimization can occur without explicit training on test items through:
A dynamic protocol should assume that important metrics will influence behavior.
A benchmark can remain statistically stable while the underlying construct changes.
For example, "coding capability" may shift from:
A protocol that measures only the earlier form may continue producing precise but incomplete evidence.
The tested system may differ from the deployed system.
Differences can include:
A model-level benchmark may not predict system-level behavior.
Dynamic protocols should define the evaluated object precisely and identify when a deployment change requires re-evaluation.
Observed capability depends partly on how capability is elicited.
Methods may improve through:
A low score may reflect poor elicitation rather than lack of underlying capability.
A high score may reflect a scaffold unavailable in the intended deployment.
The protocol should specify whether it seeks to measure:
These are different questions.
A benchmark samples from a task distribution.
Real use may shift across:
HELM emphasized broad scenario and metric coverage because evaluation based on narrow shared tasks can hide major gaps and tradeoffs.[^helm]
A single score can compress incompatible properties.
Two systems with the same aggregate score may differ in:
Dynamic protocols should resist unnecessary scalarization.
Tasks that once resembled real work can become artificial as tools, interfaces, and workflows change.
The task remains reproducible but no longer authentic.
A system may behave differently when it detects evaluation conditions.
Potential behaviors include:
Evidence that current models systematically sandbag across broad evaluation regimes remains an open research question. The protocol should nevertheless be designed so that obvious gaming opportunities are minimized and anomalous behavior is investigated.
The decision supported by an evaluation can change while the test remains the same.
A protocol created for research comparison may later be used for:
Evidence sufficient for a leaderboard may be insufficient for a high-consequence decision.
Standards Body treats the full protocol as the unit that must be validated and governed.
A protocol should specify at least the following elements.
What decision is the evaluation meant to inform?
Examples:
An evaluation without a decision question can still support exploration, but its interpretation should remain limited.
The protocol should identify:
The protocol should define the property being measured.
A strong construct definition includes:
The task universe is the broader population from which evaluation tasks are sampled or generated.
This matters because a benchmark score is meaningful only relative to the population it represents.
The protocol should explain whether tasks are:
The protocol should define:
The protocol should identify:
Relevant references may include:
METR's time-horizon work illustrates the value of relating AI task performance to the time required by skilled humans, while also showing why the task suite and methodology must be updated as systems improve.[^metr-time][^metr-update]
The protocol should document evidence concerning:
Security may cover:
The protocol should state what results do and do not mean.
The protocol should specify:
The protocol should define when it will be withdrawn or superseded.
Dynamic evaluation requires disciplined separation between stable and changeable components.
The stable core should usually include:
These can change, but changes should be treated as major revisions.
The controlled dynamic layer can include:
The operational layer can change more frequently:
Operational changes can still affect results. They should not be dismissed as merely technical.
A protocol should change enough to preserve validity, but not more than necessary.
Unnecessary change:
When a protocol changes so substantially that comparisons are no longer defensible, maintainers should state that clearly.
False continuity is more damaging than an honest break in the series.
The protocol should produce evidence that can reasonably inform an identified decision.
The measured capability or property should be defined before tasks are selected.
Every material change should be versioned and documented.
Past protocols, results, limitations, and change rationales should remain accessible unless security or legal constraints require restricted archives.
Comparisons across versions should be supported by bridging evidence rather than assumed.
No single task family should carry more interpretive weight than its validity justifies.
A strong protocol can combine:
Results should be tied to the actual system configuration.
Confidentiality should protect evaluation integrity without shielding weak methods from scrutiny.
Qualified external reviewers should be able to challenge assumptions, methods, and interpretations.
Participants should know the protocol's general scope, rules, and decision consequences, even when specific items remain held out.
Protocol design should consider how participants might optimize against the metric without improving the intended construct.
Results should not remain current indefinitely.
The protocol should specify which components can be reproduced publicly, which can be independently reproduced under controlled access, and which cannot be reproduced for justified reasons.
Definitions, units, metadata, and reporting should support cross-jurisdictional comparison.
No protocol should be treated as permanent merely because it has institutional status.
Dynamic evaluation is a family of methods.
Tasks are refreshed at fixed intervals.
Useful when:
Risk:
Revision occurs when predefined indicators are met.
Triggers may include:
The task population moves forward through time.
Useful for:
Risk:
Tasks are derived from recent real-world events, incidents, datasets, competitions, or professional work.
LiveBench uses recent sources and periodic updates as part of its contamination-limited design.[^livebench]
Risk:
Humans generate tasks that expose weaknesses in current systems.
Dynabench demonstrated an adversarial collection process in which annotators attempt to create examples that fool a target model while remaining valid for humans.[^dynabench]
Advantages:
Risks:
Tasks are generated from rules, simulations, formal systems, or parameterized templates.
Advantages:
Risks:
Domain experts periodically create, review, and retire tasks.
Useful in:
Risks:
The next task is selected based on prior performance.
Advantages:
Risks:
The model operates in an environment that changes in response to its actions.
Useful for:
Risks:
Evidence is generated from deployed performance through:
This should complement, not replace, pre-deployment evaluation.
Most serious frontier protocols will combine several methods.
A possible hybrid design might include:
A dynamic protocol is useful only if it continues to measure something meaningful.
Construct validity asks whether the protocol actually measures the claimed capability.
Threats include:
Content validity asks whether the task portfolio adequately covers the construct.
Dynamic protocols should maintain a construct map showing:
Where possible, evaluation results should be compared with meaningful external outcomes.
Examples:
The absence of a strong external criterion should be stated.
A reliable protocol produces sufficiently consistent evidence under comparable conditions.
Relevant forms include:
Reliability does not establish validity.
A protocol can measure the wrong thing consistently.
For threshold evaluations, maintainers should consider:
A protocol should examine how its use affects behavior.
Questions include:
Every major version should include a validity argument.
That argument should explain:
Dynamic protocols create a tension.
If nothing changes, validity decays.
If everything changes, historical comparison disappears.
A subset of tasks can remain stable across versions.
Anchors should be:
Public anchors can support transparency. Private anchors can support stronger leakage control. Both have tradeoffs.
Different task forms can be designed to measure the same construct at similar difficulty.
Parallel forms require validation.
Similarity in topic is not enough.
During a protocol transition, the same systems can be evaluated under old and new versions.
Bridging studies can estimate:
A stable panel of reference systems can be tested across versions.
Reference systems might include:
Closed systems can disappear or change, so they should not be the only anchors.
Where assumptions are justified, item response models can help estimate latent ability and task difficulty.
They should not be applied mechanically.
Frontier AI systems may violate assumptions common in educational testing because:
Version-linked results may be more honest as:
When comparability is weak, the report should state:
For some protocols, maintainers should preserve the ability to rerun old versions.
This may require:
A dynamic evaluation protocol should use formal version control.
A semantic structure can be adapted:
Example:
DEP-CYBER-1.4.2
This could identify:
Every material update should include:
Emergency changes may be needed after:
Emergency changes should have:
Participants should receive reasonable notice before a protocol version is retired, unless continued use would be misleading or unsafe.
The protocol should provide a lineage showing:
A need may arise from:
Output:
Define:
Output:
Review:
Output:
Specify:
Output:
Develop tasks through one or more methods:
Output:
Test:
Output:
Run with:
Output:
Reviewers assess:
Output:
Publish appropriate components:
Sensitive content may remain controlled.
Run under controlled conditions.
Record:
Report:
Track:
Choose:
Conduct bridging studies, notify users, preserve archives, and update dependent standards.
A protocol should be renewed only if it continues to justify its institutional cost and interpretive authority.
A protocol should not depend solely on maintainer discretion.
Examples:
Examples:
Examples:
Examples:
Examples:
Examples:
A periodic review should occur even without an obvious event.
Time-based review is a backstop, not a substitute for evidence-driven triggers.
A dynamic protocol must specify what level of capability it is trying to reveal.
How does the publicly available system behave under ordinary use?
How does the system perform under a common evaluator-defined configuration?
What can the developer demonstrate using its preferred configuration?
What can qualified evaluators elicit with additional effort, tools, and scaffolds?
What might the system accomplish under realistic but highly optimized conditions?
These targets can produce different results.
Repeated attempts can reveal latent capability, but they can also misrepresent normal use.
Protocols should state:
Tools can transform capability.
Examples:
The protocol should distinguish model capability from system capability when possible.
A capability evaluation may test:
These results should not be conflated.
Developer participation can improve elicitation and reduce false negatives. It can also create asymmetry.
A balanced design can include:
AISI has emphasized that evaluator access, time, and methodology affect the strength of conclusions drawn from frontier evaluations.[^aisi-lessons] Recent work on external evaluator access similarly distinguishes model access, information access, and evaluation timeframe as separate dimensions.[^access]
Agentic systems intensify the need for dynamic protocols.
Agent performance depends on:
A dynamic agent protocol should evaluate:
Longer tasks introduce:
METR's work measuring task-completion time horizons demonstrates one possible way to summarize autonomous task capability, while its later methodology updates illustrate why the suite itself must evolve as capabilities and task coverage change.[^metr-time][^metr-update]
Agent evaluations should version:
In security and control evaluations, the environment or defender may adapt to the agent.
This can increase realism but reduce reproducibility.
Some agent evaluations may create risk if they provide:
Protocol design should include safety review and containment.
Dynamic protocols should manage contamination as an ongoing process.
The threat model should cover:
Controls may include:
Detection approaches may include:
No detection method should be treated as conclusive by default.
A contamination response plan should specify:
Claims of being contamination-free should be used cautiously.
For large models trained on broad corpora, proving absence of exposure is often difficult.
"Contamination-limited" or "contamination-resistant" is generally more defensible when evidence supports it.
A protocol may report:
HELM's multi-metric approach illustrates why accuracy alone may fail to capture calibration, robustness, fairness, bias, toxicity, or efficiency.[^helm]
A dynamic protocol should report dimensions separately when aggregation would hide important tradeoffs.
Uncertainty can arise from:
Near a consequential threshold, the protocol should avoid false certainty.
Possible responses:
The protocol should define:
Every public result should include:
A result should carry:
Dynamic evaluation places substantial power in protocol maintainers.
That power must be governed.
A mature protocol may assign:
Roles can be combined in small projects, but conflicts should be documented.
No single maintainer should be able to make an undisclosed material change to a high-consequence protocol.
Relevant conflicts include:
Disclosure alone may not resolve a conflict.
Major protocol decisions should allow:
Participants should be able to challenge:
Appeal should not become a mechanism to suppress unfavorable results.
Public consultation can improve legitimacy, but sensitive protocols may require restricted technical consultation.
The process should state:
Long-term maintenance requires sustainable funding.
Funding models include:
Each creates incentives.
Dynamic protocols should disclose major funding dependencies and protect revision decisions from payer control.
A dynamic protocol can disclose:
while protecting:
A possible framework:
General methodology, governance, version, results, limitations.
Detailed task taxonomy, validation materials, selected artifacts.
Held-out tasks, administration tools, security procedures.
Highly sensitive content with strict need-to-know controls.
Some content can be released after:
Even when content is private, the process should be auditable by qualified independent parties.
The UK AI Security Institute's Inspect framework demonstrates the value of modular, open evaluation infrastructure that can support diverse tasks, agents, scorers, and model providers.[^inspect] Open tooling does not make every evaluation transparent, but it can improve portability, shared practice, and reproducibility.
Frontier AI evaluation will cross jurisdictions.
Institutions can align on:
while retaining different task sets or legal uses.
Mutual recognition may be appropriate when protocols demonstrate:
Dynamic protocols should account for:
International agreement should not require reducing rigor to the easiest shared metric.
NIST describes testing, evaluation, verification, and validation as part of operationalizing AI risk management, and its AI Resource Center provides related guidance and resources.[^nist-rmf][^nist-airc] Future Standards Body protocols should map to recognized TEVV and standards terminology while remaining explicit where frontier AI requires additional methods.
Characteristics:
Use:
Characteristics:
Use:
Characteristics:
Use:
Characteristics:
Use:
Characteristics:
Use:
The maturity level should match the consequence of the decision.
Identify:
Choose a domain with:
Define:
Create:
Pilot:
Conduct:
Publish enough for legitimacy while protecting sensitive content.
Track predefined indicators.
Use formal change control.
Assess whether the protocol improved decisions, not merely whether it produced scores.
Standards Body should begin with a bounded pilot rather than attempting to establish a universal protocol.
Dynamic Evaluation Protocol for Long-Horizon Technical Task Performance
The pilot could examine how well AI systems complete increasingly long, verifiable technical tasks under controlled tool access.
It offers:
The pilot succeeds if it demonstrates:
The evaluation protocol itself should be evaluated.
Failure: Frequent updates create the appearance of sophistication without improving validity.
Safeguard: Every material update requires an evidence-based rationale and post-change review.
Failure: Versions cannot be compared, but rankings are presented as continuous.
Safeguard: Bridging studies and explicit discontinuity statements.
Failure: A developer, government, funder, or evaluator controls revisions.
Safeguard: Distributed governance, conflicts, dissent, independent review.
Failure: Confidentiality prevents scrutiny of weak methods.
Safeguard: Controlled independent audit and public methodological disclosure.
Failure: Adversarial task generation becomes too specific to one model family.
Safeguard: Multiple reference systems, human validation, real-use sampling, diversity review.
Failure: Tasks become harder without becoming more meaningful.
Safeguard: Construct map and criterion relevance review.
Failure: Protocols and developers escalate complexity without improving decision quality.
Safeguard: Cost-benefit review and minimum necessary change.
Failure: Generated tasks and model judges introduce hidden errors.
Safeguard: Human audits, objective scoring where possible, judge validation, uncertainty.
Failure: Small expert groups become bottlenecks or sources of bias.
Safeguard: Rotation, structured rubrics, diverse panels, scalable validation.
Failure: Decisions change because thresholds or scores move, not because capability changes.
Safeguard: Threshold governance, transition rules, decision impact analysis.
Failure: Too many overlapping protocols create confusion.
Safeguard: Registry, taxonomy, interoperability review, consolidation.
Failure: The protocol becomes a target and ceases to be a useful measure.
Safeguard: Multiple methods, rotating components, held-out tasks, incident evidence, periodic construct review.
Failure: Passing the protocol is interpreted as proof of safety.
Safeguard: Explicit claims boundary, complementary evidence, result expiration.
Failure: Only the largest laboratories can participate.
Safeguard: Tiered access, shared infrastructure, subsidized evaluation, open reference tracks.
Failure: Revision is too slow for capability change.
Safeguard: Predefined triggers, emergency procedures, standing expert groups.
Failure: Stakeholders cannot build stable processes because metrics constantly change.
Safeguard: Stable core, scheduled transitions, version support windows.
This objection is valid.
Changing tasks and environments makes exact replication harder.
Response:
Residual concern:
Some dynamic, adversarial, or interactive evaluations may remain intrinsically difficult to reproduce.
Also valid.
Maintainers can influence:
Response:
Residual concern:
Institutional power cannot be eliminated. It must be constrained and contestable.
Static benchmarks often support cleaner comparison.
Response:
Residual concern:
Some research questions are better served by fixed benchmarks. Dynamic evaluation should not become an ideology.
They require ongoing staff, task development, security, infrastructure, and review.
Response:
Residual concern:
A dynamic regime can become bureaucratic. Cost should be measured explicitly.
Yes.
Response:
Residual concern:
No evaluation system is immune to optimization.
Response:
Residual concern:
Constructs such as "general intelligence" or "dangerous capability" may remain contested.
Correct.
Recent tasks can be noisy, unrepresentative, or shallow.
Response:
Recent-data evaluation should be one component, not the definition of dynamic evaluation.
Agreed.
Response:
The goal should be interoperable protocols, not one universal test.
This may sometimes be true.
Response:
OpenAI's updated Preparedness Framework explicitly links faster model improvement to the need for scalable, more frequent evaluations while retaining expert-led deep dives.[^openai-pf]
Residual concern:
Evaluation capacity may still lag. Protocols should state when evidence is incomplete rather than creating false confidence.
Response:
This foundation proposes evidence infrastructure, not automatic regulation.
The decision use should remain explicit and proportionate.
The field lacks strong evidence on several foundational questions.
How well do common equating methods work for rapidly changing AI systems?
Which contamination-detection methods are reliable across closed training pipelines?
How much measured capability variation is attributable to prompting, scaffolding, and evaluator effort?
Which evaluation results predict real-world deployment outcomes?
When does human-and-model-in-the-loop collection improve robustness, and when does it create unrealistic distributions?
Theoretical work on dynamic benchmarks highlights that the interaction between model fitting and data collection requires distinct analysis from static benchmarking.[^dynamic-theory]
When are model-based judges sufficiently reliable, and where do they create systematic bias?
How should long-horizon, stochastic, and environment-dependent performance be summarized?
Which governance structures revise protocols quickly without enabling capture?
What evidence is sufficient for one evaluator or jurisdiction to recognize another's result?
How should result expiration be determined empirically?
Do dynamic protocols improve real decisions enough to justify their cost?
Develop and test:
Compare:
Evaluate:
Develop reporting standards for:
Study:
Pilot:
Develop:
Measure whether evaluation systems themselves remain valid, efficient, and decision-useful.
Create:
Develop controls for evaluation activities that could create security, privacy, or misuse risk.
Run the same reference systems on:
Compare:
Test whether stable anchors preserve useful longitudinal measurement after substantial task refresh.
Compare adversarial tasks against naturally sampled professional tasks.
Evaluate the same system under:
Build Protocol 1.0 and 2.0, conduct a bridging study, and test whether independent analysts reach similar comparability conclusions.
Test whether users interpret results more accurately when reports include explicit freshness and expiration status.
Run a mock change-control process with:
Introduce controlled environment updates and measure how much apparent capability change comes from the model versus the environment.
A future standard for dynamic evaluation protocols could require:
Unique identifier, owner, version, status, and scope.
Definition, exclusions, intended use, and evidence basis.
Model and system configuration metadata.
Triggers, authority, review, validation, and publication.
Anchors, bridging, reference systems, and discontinuity criteria.
Contamination, security, access, logging, and incident response.
Reliability, validity, robustness, fairness, and uncertainty.
Results, limitations, configuration, uncertainty, conflicts, and expiration.
Roles, conflicts, dissent, appeals, and funding disclosure.
Conditions, process, archive, and successor mapping.
Such a standard should be developed through the future STANDARDS_DEVELOPMENT_PROCESS.md, not declared unilaterally by this paper.
Dynamic protocols often require rotating confidential content to reduce leakage.
The consequence of the decision should determine the depth, security, and review level of the dynamic protocol.
Independent reviewers challenge construct definitions, methods, revisions, and interpretations.
Dynamic protocols require qualified organizations capable of consistent administration across versions.
Dynamic protocols may begin as voluntary research practice and later support procurement, certification, insurance, or formal requirements.
Recognition should reward participation in rigorous, revisable evaluation rather than benchmark marketing.
Protocol metadata, versioning, evidence standards, and recognition should work across borders.
Standards Body adopts the following positions as the current working foundation.
Static benchmarks remain useful, but are insufficient as the sole basis for frontier AI evaluation.
The evaluation protocol, not the dataset alone, is the proper unit of governance and validation.
Dynamic evaluation means evidence-driven maintenance, not arbitrary or constant change.
Every high-consequence protocol should define what remains stable and what may change.
Historical comparability must be demonstrated, not assumed.
A major protocol change may require a new baseline rather than a forced continuous ranking.
Evaluation results should be tied to exact system configurations and dates.
Results should have expiration or re-evaluation conditions.
Dynamic protocols should combine multiple forms of evidence when one method is insufficient.
Held-out content can coexist with transparent governance.
Independent review is required when protocol results influence consequential decisions.
Protocol maintainers should be subject to conflict disclosure, dissent, and appeal mechanisms.
Evaluation difficulty should not be confused with evaluation validity.
Recent-data tasks should not be treated as automatically representative.
Model-judge and automated task-generation methods require validation.
Agent evaluation requires environment, tool, and trajectory versioning.
Passing an evaluation is not proof of safety.
Protocols should be retired when their interpretive authority exceeds their evidence.
Interoperability is preferable to forced global uniformity.
The evaluation system itself should be continuously evaluated.
A protocol should be revised when one or more of the following is true:
A protocol should not be revised merely because:
A protocol should be retired when:
A future protocol should include the following sections.
Protocol:
Current version:
Proposed version:
Proposer:
Date:
Describe the change precisely.
Identify the evidence or event requiring change.
Explain why the current protocol is insufficient.
Describe likely effects on:
Explain how the change will be tested.
Explain how old and new versions will be linked, or state why they cannot be linked.
List relevant interests.
Identify technical, domain, security, and independent reviewers.
A protocol can be reviewed against the following dimensions:
| Dimension | Core Question |
|---|---|
| Purpose | Is the decision question explicit? |
| Construct | Is the measured property clearly defined? |
| Coverage | Does the task portfolio represent the construct? |
| Integrity | Is contamination and gaming risk managed? |
| Reliability | Are results sufficiently consistent? |
| Validity | Does evidence support the intended interpretation? |
| Elicitation | Are prompting, tools, and scaffolds specified? |
| Configuration | Is the evaluated system precisely identified? |
| Comparability | Are version-to-version claims justified? |
| Security | Are sensitive components protected proportionately? |
| Transparency | Is enough disclosed for accountability? |
| Governance | Are authority and conflicts controlled? |
| Appeals | Can material errors be challenged? |
| Freshness | Is result expiration defined? |
| Interoperability | Can others understand and map the result? |
| Cost | Is the protocol proportionate to its decision value? |
| Accessibility | Can qualified smaller actors participate? |
| Retirement | Can obsolete authority be withdrawn? |
Frontier AI evaluation will fail if it is treated as a sequence of permanent leaderboards.
The systems being measured are changing.
The ways they are trained are changing.
The tools surrounding them are changing.
The environments in which they act are changing.
The risks, benefits, and decisions attached to their performance are changing.
Evaluation infrastructure must therefore be capable of change as well.
But change alone is not the objective.
A protocol that changes constantly without stable meaning is not dynamic in a useful sense. It is unstable.
The objective is disciplined adaptation.
Dynamic evaluation protocols should make it possible to improve what is measured while preserving the evidence required to understand how and why the measurement changed.
They should support progress without allowing yesterday's evidence to govern tomorrow's systems indefinitely.
They should make uncertainty visible.
They should reveal when comparisons are weak.
They should state when an evaluation has expired.
They should allow disagreement.
They should retain history.
They should be governed in proportion to the consequences of their use.
The first foundation of Standards Body is therefore not a particular benchmark.
It is the institutional capacity to keep evaluation meaningful.
[^nist-rmf]: National Institute of Standards and Technology, AI Risk Management Framework and Generative AI Profile. https://www.nist.gov/itl/ai-risk-management-framework
[^nist-airc]: National Institute of Standards and Technology, AI Resource Center, including testing, evaluation, verification, and validation resources. https://airc.nist.gov/
[^helm]: Percy Liang et al., Holistic Evaluation of Language Models, 2022, revised 2023. https://arxiv.org/abs/2211.09110
[^dynabench]: Douwe Kiela et al., Dynabench: Rethinking Benchmarking in NLP, 2021. https://arxiv.org/abs/2104.14337
[^dynamic-theory]: Ali Shirali, Rediet Abebe, and Moritz Hardt, A Theory of Dynamic Benchmarks, ICLR 2023. https://arxiv.org/abs/2210.03165
[^livebench]: Colin White et al., LiveBench: A Challenging, Contamination-Limited LLM Benchmark, 2024. https://arxiv.org/abs/2406.19314
[^dynamic-survey]: Shijie Chen et al., Recent Advances in Large Language Model Benchmarks Against Data Contamination: From Static to Dynamic Evaluation, 2025. https://arxiv.org/abs/2502.17521
[^aisi-lessons]: UK AI Security Institute, Early Lessons from Evaluating Frontier AI Systems, 2024. https://www.aisi.gov.uk/blog/early-lessons-from-evaluating-frontier-ai-systems
[^inspect]: UK AI Security Institute, Inspect AI, open evaluation framework. https://inspect.aisi.org.uk/
[^metr-time]: METR, Measuring AI Ability to Complete Long Tasks, 2025. https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/
[^metr-update]: METR, Time Horizon 1.1, 2026. https://metr.org/blog/2026-1-29-time-horizon-1-1/
[^openai-pf]: OpenAI, Preparedness Framework v2, 2025. https://openai.com/index/updating-our-preparedness-framework/
[^deepmind-fsf]: Google DeepMind, Frontier Safety Framework, updated 2025. https://deepmind.google/blog/updating-the-frontier-safety-framework/
[^extreme-risk]: Toby Shevlane et al., Model Evaluation for Extreme Risks, 2023. https://arxiv.org/abs/2305.15324
[^dangerous-capabilities]: Mary Phuong et al., Evaluating Frontier Models for Dangerous Capabilities, 2024. https://arxiv.org/abs/2403.13793
[^access]: Jacob Charnock et al., Expanding External Access to Frontier AI Models for Dangerous Capability Evaluations, 2026. https://arxiv.org/abs/2601.11916
Date: July 16, 2026
Change type: Complete replacement
Summary: Replaces the earlier outline edition with a fully developed canonical working white paper. Adds first-principles rationale, definitions, protocol architecture, taxonomy of dynamic methods, measurement validity, version comparability, lifecycle, change triggers, elicitation standards, agent evaluation, contamination controls, scoring, governance, transparency, interoperability, maturity model, implementation pathway, Standards Body pilot proposal, protocol metrics, failure analysis, objections, evidence gaps, research agenda, standards implications, templates, canonical positions, and primary-source research basis.
Status: Ready for internal review and future expert critique.