Foundation 1: Dynamic Evaluation Protocols

Document Purpose

This paper defines the Standards Body position on dynamic evaluation protocols for frontier artificial intelligence.

It is intended to serve as:

A foundational explanation of why static benchmarks are insufficient on their own
A design framework for continuously maintained evaluation systems
A bridge between evaluation science and institutional governance
A reference for future standards, working groups, research programs, and evaluator guidance
A durable source document from which shorter public articles and technical proposals can be developed

This paper is not itself a technical standard. It does not prescribe one universal benchmark, scoring method, capability taxonomy, or legal threshold. It defines the principles, architecture, and governance conditions that should guide the development of evaluation protocols capable of remaining meaningful as AI systems, deployment environments, and societal needs change.

Executive Summary

Frontier AI evaluation is often organized around fixed benchmarks. A dataset is assembled, a scoring rule is defined, systems are tested, and the resulting scores are compared.

This approach has produced substantial scientific value. It has made progress more legible, enabled model comparisons, exposed weaknesses, and supported reproducible research. Static benchmarks should remain part of the evaluation ecosystem.

They should not be mistaken for a complete evaluation system.

A fixed benchmark begins to lose value when:

Models approach or reach its ceiling
Its questions or solutions enter training corpora
Developers optimize directly or indirectly against it
Prompting and scaffolding choices dominate the measured result
The underlying capability changes faster than the benchmark
The deployment context no longer resembles the test
The benchmark measures a proxy that has drifted away from the decision it is meant to support
New failure modes appear that were not represented when the test was created
Rankings remain visible after the evidence supporting them has expired

Dynamic evaluation protocols respond to this problem by treating evaluation as a maintained institutional process rather than a one-time dataset.

A dynamic protocol can revise its:

Construct definition
Task population
Sampling method
Administration environment
Scoring system
Model configuration requirements
Human baselines
Security controls
Confidence estimates
Reporting rules
Decision thresholds
Monitoring triggers
Retirement criteria

The protocol should not change arbitrarily. Uncontrolled change can destroy comparability, reproducibility, fairness, and trust.

The central design problem is therefore not simply how to update an evaluation. It is how to update an evaluation without losing the properties that make measurement useful.

A strong dynamic evaluation protocol should preserve five forms of continuity:

Conceptual continuity
The underlying capability or risk construct remains clearly defined, even when tasks change.
Measurement continuity
Version-to-version changes are calibrated so that score differences are not confused with capability differences.
Procedural continuity
Changes follow documented governance, validation, and approval processes.
Evidentiary continuity
Historical results, uncertainty, limitations, and protocol lineage remain accessible.
Decision continuity
Users can understand whether a revised result supports the same decision, a new decision, or no longer supports the earlier decision at all.

Standards Body therefore adopts the following core position:

Frontier AI evaluation should be designed as a versioned, evidence-producing protocol with explicit maintenance, validation, governance, security, and retirement mechanisms. A benchmark may be one component of that protocol, but it should not be treated as the protocol itself.

Dynamic evaluation is not synonymous with continuously changing test questions. It is a broader institutional discipline.

It includes deciding:

What should remain stable
What should be refreshed
What evidence should trigger change
Who has authority to revise the protocol
How revised versions remain comparable
When confidentiality is justified
How developers can contest results
How uncertainty is communicated
When an evaluation should be retired
How a protocol connects to real decisions

This foundation is necessary because evaluation systems will otherwise become stale while continuing to look authoritative.

The danger is not only inaccurate scores.

The deeper danger is institutional confidence built on expired evidence.

1. Foundational Proposition

1.1 Core Thesis

Evaluation systems should evolve at a pace appropriate to the capabilities, environments, and decisions they are intended to measure.

This does not mean that evaluation protocols should change every time a new model is released.

It means that every serious protocol should possess an explicit theory of change.

That theory should identify:

Which changes in technology matter
Which changes in deployment matter
Which changes in evidence matter
Which changes in threat models matter
Which changes in social context matter
Which changes require protocol revision
Which changes do not

A protocol without a theory of change is static by default, even when its maintainers occasionally add questions.

1.2 Institutional Thesis

The ability to update an evaluation credibly is itself a form of public infrastructure.

Creating a benchmark is a research activity.

Maintaining a protocol across years, organizations, model generations, and disputes is an institutional activity.

The latter requires:

Governance
Funding
Technical stewardship
Security
Documentation
Quality assurance
Appeals
Conflict management
Archiving
International coordination
Long-term accountability

1.3 Epistemic Thesis

Evaluation results are time-bound claims produced under specified conditions, not permanent properties of a model.

A score is meaningful only relative to:

The protocol version
The system configuration
The tools available
The prompting method
The sampling process
The scoring rule
The execution environment
The date
The evaluator
The uncertainty
The intended interpretation

Dynamic evaluation makes these dependencies visible.

2. Scope and Boundaries

2.1 What This Foundation Covers

This paper covers dynamic protocols for evaluating:

General capabilities
Domain-specific capabilities
Safety-relevant capabilities
Reliability and robustness
Agentic performance
Safeguard effectiveness
Human-AI interaction
Deployment behavior
Operational performance
High-stakes capability thresholds

It addresses evaluation of models and systems, including systems composed of:

A base model
System instructions
Retrieval
External tools
Memory
Agent scaffolding
Safety filters
Monitoring
Human oversight
Deployment constraints

2.2 What This Foundation Does Not Claim

This paper does not claim that:

All static benchmarks are obsolete
Every test item should be private
Continuous automated testing can replace expert review
Dynamic protocols eliminate contamination
Dynamic evaluation proves that a system is safe
One protocol can cover every use case
Higher benchmark performance always implies greater real-world value
Protocol revision should be controlled by a single institution
Every evaluation result should determine a regulatory action

2.3 Relationship to Monitoring

Evaluation and monitoring overlap but are not identical.

Evaluation produces structured evidence under defined conditions.

Monitoring observes systems, environments, incidents, or indicators over time.

Monitoring can trigger evaluation revision. Evaluation can define what monitoring should watch. A mature system connects the two without collapsing them into one activity.

2.4 Relationship to Red Teaming

Red teaming is one method within a broader evaluation protocol.

It is especially useful for:

Discovering failure modes
Challenging assumptions
Generating adversarial tasks
Exploring system boundaries
Testing safeguards
Identifying unknown weaknesses

Red teaming alone may not provide:

Representative sampling
Stable scoring
Historical comparability
Statistical confidence
Reproducible decision thresholds

Dynamic protocols can use red-team findings to update formal test populations.

3. Canonical Definitions

3.1 Benchmark

A benchmark is a defined set of tasks, data, environments, or procedures used to measure one or more properties of a system.

3.2 Evaluation

An evaluation is the structured production and interpretation of evidence about a system relative to defined questions, conditions, metrics, and decisions.

3.3 Evaluation Protocol

An evaluation protocol is the complete specification governing how an evaluation is designed, administered, scored, interpreted, secured, reviewed, versioned, and connected to decisions.

A protocol can include one or more benchmarks.

3.4 Dynamic Evaluation Protocol

A dynamic evaluation protocol is a protocol with explicit mechanisms for evidence-driven revision while preserving appropriate continuity, traceability, and comparability.

3.5 Dynamic Benchmark

A dynamic benchmark is a benchmark whose tasks, examples, environments, models, sampling rules, or scoring components change over time.

A dynamic benchmark is narrower than a dynamic evaluation protocol.

3.6 Adaptive Testing

Adaptive testing selects or generates later tasks based on earlier system performance.

It can improve efficiency and measurement precision, but it creates additional challenges for reproducibility and comparability.

3.7 Rolling Evaluation

A rolling evaluation uses a continuously or periodically refreshed test window, often based on newly created or newly available material.

3.8 Evaluation Drift

Evaluation drift occurs when a protocol's meaning, task distribution, administration, scoring, or interpretation changes over time.

Drift can be intentional, accidental, beneficial, or harmful.

3.9 Protocol Expiration

Protocol expiration is the point at which results should no longer be treated as current evidence for the intended decision without re-evaluation or additional justification.

3.10 Protocol Retirement

Protocol retirement is the formal withdrawal of a protocol from active use because it is invalid, obsolete, insecure, superseded, too costly, or no longer decision-relevant.

3.11 Anchor Component

An anchor component is a stable task, item family, environment, reference population, or measurement link used to compare versions.

3.12 Evaluation Regime

An evaluation regime is the broader institutional arrangement within which protocols are developed and used, including evaluators, laboratories, governments, standards organizations, security systems, and decision-makers.

4. Why Static Benchmarks Lose Meaning

Static benchmarks fail in several distinct ways. These failure modes should not be reduced to contamination alone.

4.1 Saturation

A benchmark saturates when top systems cluster near its maximum score or when score differences become too small to distinguish meaningful capability differences.

Saturation creates several problems:

Rank ordering becomes unstable
Small implementation details can determine leadership
The benchmark no longer identifies frontier weaknesses
Public attention may remain high after measurement value has declined
Research incentives can shift toward marginal score gains rather than broader capability understanding

A saturated benchmark may still be useful for:

Testing smaller systems
Measuring accessibility
Regression testing
Education
Historical analysis

Saturation does not make a benchmark worthless. It changes the claims the benchmark can support.

4.2 Training-Data Contamination

Contamination occurs when evaluation material, close variants, solutions, or derived artifacts become part of model training or tuning.

Contamination can arise through:

Public benchmark release
Scraped repositories
Academic papers containing test items
Solution websites
Evaluation transcripts
Fine-tuning datasets
Preference data
Synthetic training data generated from benchmark content
Developer use of evaluation results during iteration

The central issue is not merely whether the exact item appeared in training.

Models may learn:

Task templates
Answer patterns
Rubrics
Domain-specific shortcuts
Common transformations
Benchmark-specific conventions

LiveBench was developed partly in response to contamination concerns by using frequently updated questions derived from recent sources and objective scoring where possible. Its design illustrates one approach, not a complete solution.[^livebench]

4.3 Direct and Indirect Optimization

Once a benchmark becomes important, developers have incentives to improve performance on it.

This is not inherently improper.

Optimization becomes problematic when benchmark score improves faster than the underlying capability the benchmark is intended to represent.

Indirect optimization can occur without explicit training on test items through:

Prompt engineering
Benchmark-aware instruction tuning
Selection of model checkpoints
Scaffold design
Tool configuration
Repeated internal experimentation
Public leaderboard feedback
Training on related datasets
Selection of favorable sampling parameters

A dynamic protocol should assume that important metrics will influence behavior.

4.4 Construct Drift

A benchmark can remain statistically stable while the underlying construct changes.

For example, "coding capability" may shift from:

Producing a short function
Repairing a repository
Operating tools
Managing long-running tasks
Coordinating subtasks
Handling ambiguous requirements
Verifying its own work

A protocol that measures only the earlier form may continue producing precise but incomplete evidence.

4.5 Deployment Drift

The tested system may differ from the deployed system.

Differences can include:

Model version
System prompt
Context window
Retrieval source
Tool permissions
Sampling settings
Memory
Monitoring
Rate limits
Human escalation
Safety filters
Regional configuration

A model-level benchmark may not predict system-level behavior.

Dynamic protocols should define the evaluated object precisely and identify when a deployment change requires re-evaluation.

4.6 Elicitation Drift

Observed capability depends partly on how capability is elicited.

Methods may improve through:

Better prompts
Chain-of-thought alternatives
Search
Tool use
Longer inference
Test-time computation
Multi-agent scaffolds
Verifiers
Human assistance

A low score may reflect poor elicitation rather than lack of underlying capability.

A high score may reflect a scaffold unavailable in the intended deployment.

The protocol should specify whether it seeks to measure:

Typical capability
Best demonstrated capability
Capability under realistic deployment
Capability under evaluator-provided support
Capability under developer-provided support
Capability under an adversarially optimized scaffold

These are different questions.

4.7 Distribution Shift

A benchmark samples from a task distribution.

Real use may shift across:

Languages
regions
user populations
tools
data quality
time
legal context
domain knowledge
adversarial pressure
workflow complexity

HELM emphasized broad scenario and metric coverage because evaluation based on narrow shared tasks can hide major gaps and tradeoffs.[^helm]

4.8 Metric Collapse

A single score can compress incompatible properties.

Two systems with the same aggregate score may differ in:

Reliability
Calibration
robustness
cost
speed
fairness
security
refusal behavior
tail risk

Dynamic protocols should resist unnecessary scalarization.

4.9 Task Authenticity Decay

Tasks that once resembled real work can become artificial as tools, interfaces, and workflows change.

The task remains reproducible but no longer authentic.

4.10 Evaluation Gaming and Sandbagging

A system may behave differently when it detects evaluation conditions.

Potential behaviors include:

Producing benchmark-specific answers
Avoiding suspiciously capable behavior
Exploiting scoring rules
Manipulating a judge
Timing out strategically
Appearing compliant under test conditions
Using hidden channels or tools

Evidence that current models systematically sandbag across broad evaluation regimes remains an open research question. The protocol should nevertheless be designed so that obvious gaming opportunities are minimized and anomalous behavior is investigated.

4.11 Decision Drift

The decision supported by an evaluation can change while the test remains the same.

A protocol created for research comparison may later be used for:

Procurement
insurance
deployment approval
regulation
accreditation
public claims

Evidence sufficient for a leaderboard may be insufficient for a high-consequence decision.

5. The Protocol, Not the Dataset, Is the Unit of Evaluation

Standards Body treats the full protocol as the unit that must be validated and governed.

A protocol should specify at least the following elements.

5.1 Decision Question

What decision is the evaluation meant to inform?

Examples:

Which system performs best for a defined task?
Has a capability crossed an internal threshold?
Is a safeguard effective under specified attacks?
Does a deployment remain within an approved operating envelope?
Is additional human oversight required?
Should a system be re-evaluated after an update?

An evaluation without a decision question can still support exploration, but its interpretation should remain limited.

5.2 Evaluated Object

The protocol should identify:

Model identifier
model version
system configuration
system prompt status
fine-tuning
tools
retrieval
memory
safety layers
inference settings
hardware or execution constraints
date of evaluation

5.3 Construct Definition

The protocol should define the property being measured.

A strong construct definition includes:

Positive definition
exclusions
component abilities
expected manifestations
boundary cases
relationship to real-world outcomes
plausible confounders

5.4 Task Universe

The task universe is the broader population from which evaluation tasks are sampled or generated.

This matters because a benchmark score is meaningful only relative to the population it represents.

5.5 Task Selection and Generation

The protocol should explain whether tasks are:

Fixed
sampled
generated
expert-authored
adversarially collected
event-sourced
procedurally generated
derived from real incidents
based on current information
adapted to system performance

5.6 Administration Conditions

The protocol should define:

Time limits
token limits
tools
retries
interaction format
human assistance
evaluator intervention
network access
sandboxing
logging
failure handling

5.7 Scoring and Judgment

The protocol should identify:

Objective scoring rules
human-rater criteria
model-judge use
adjudication
partial credit
uncertainty
missing data
invalid runs
aggregation
subgroup reporting

5.8 Baselines and Reference Populations

Relevant references may include:

Earlier model versions
competitor systems
human novices
human professionals
domain experts
unaided humans
tool-assisted humans
previous protocol versions

METR's time-horizon work illustrates the value of relating AI task performance to the time required by skilled humans, while also showing why the task suite and methodology must be updated as systems improve.[^metr-time][^metr-update]

5.9 Validation Evidence

The protocol should document evidence concerning:

Construct validity
content validity
criterion relevance
reliability
sensitivity
specificity
robustness
fairness
contamination
inter-rater agreement
reproducibility
decision relevance

5.10 Security Controls

Security may cover:

Held-out content
access control
logging
insider risk
evaluation environment
artifact retention
disclosure
incident response

5.11 Interpretation Rules

The protocol should state what results do and do not mean.

5.12 Change-Control Rules

The protocol should specify:

Change triggers
authority
review
validation
version numbering
transition period
historical treatment
appeals
emergency changes

5.13 Retirement Rules

The protocol should define when it will be withdrawn or superseded.

6. What Should Change and What Should Remain Stable

Dynamic evaluation requires disciplined separation between stable and changeable components.

6.1 Stable Core

The stable core should usually include:

Mission
decision question
construct definition
fundamental inclusion and exclusion rules
governance principles
evidence standards
conflict rules
minimum reporting requirements
archival requirements

These can change, but changes should be treated as major revisions.

6.2 Controlled Dynamic Layer

The controlled dynamic layer can include:

Task samples
item difficulty
adversarial cases
recent-data components
system configurations
tools
attack strategies
domain scenarios
thresholds
weights
human baselines

6.3 Operational Layer

The operational layer can change more frequently:

Software dependencies
execution infrastructure
bug fixes
logging formats
interface details
computational limits
evaluator scheduling

Operational changes can still affect results. They should not be dismissed as merely technical.

6.4 The Principle of Minimum Necessary Change

A protocol should change enough to preserve validity, but not more than necessary.

Unnecessary change:

Increases cost
weakens comparability
creates governance burden
makes disputes harder to resolve
can conceal poor historical performance
may advantage participants with closer access to maintainers

6.5 The Principle of Explicit Discontinuity

When a protocol changes so substantially that comparisons are no longer defensible, maintainers should state that clearly.

False continuity is more damaging than an honest break in the series.

7. Design Principles

7.1 Decision Relevance

The protocol should produce evidence that can reasonably inform an identified decision.

7.2 Construct Clarity

The measured capability or property should be defined before tasks are selected.

7.3 Versioned Evolution

Every material change should be versioned and documented.

7.4 Historical Traceability

Past protocols, results, limitations, and change rationales should remain accessible unless security or legal constraints require restricted archives.

7.5 Calibrated Comparability

Comparisons across versions should be supported by bridging evidence rather than assumed.

7.6 Multi-Method Evidence

No single task family should carry more interpretive weight than its validity justifies.

A strong protocol can combine:

Automated tasks
human review
adversarial testing
simulations
mechanistic evidence
operational evidence
incident evidence

7.7 Configuration Transparency

Results should be tied to the actual system configuration.

7.8 Proportional Security

Confidentiality should protect evaluation integrity without shielding weak methods from scrutiny.

7.9 Independent Challenge

Qualified external reviewers should be able to challenge assumptions, methods, and interpretations.

7.10 Fair Notice

Participants should know the protocol's general scope, rules, and decision consequences, even when specific items remain held out.

7.11 Anti-Gaming Design

Protocol design should consider how participants might optimize against the metric without improving the intended construct.

7.12 Expiration by Design

Results should not remain current indefinitely.

7.13 Reproducibility with Boundaries

The protocol should specify which components can be reproduced publicly, which can be independently reproduced under controlled access, and which cannot be reproduced for justified reasons.

7.14 International Interoperability

Definitions, units, metadata, and reporting should support cross-jurisdictional comparison.

7.15 Revisability

No protocol should be treated as permanent merely because it has institutional status.

8. Taxonomy of Dynamic Evaluation Methods

Dynamic evaluation is a family of methods.

8.1 Scheduled Refresh

Tasks are refreshed at fixed intervals.

Useful when:

Capability changes are reasonably predictable
content can be generated reliably
administrative stability matters

Risk:

Updates can become ceremonial rather than evidence-driven.

8.2 Triggered Refresh

Revision occurs when predefined indicators are met.

Triggers may include:

Performance ceiling
contamination evidence
new system modality
major deployment change
newly discovered failure mode
material incident
model capability threshold
scoring failure
legal or standards change

8.3 Rolling-Window Evaluation

The task population moves forward through time.

Useful for:

Current knowledge
recent software
evolving security environments
changing information ecosystems

Risk:

Temporal changes can be confused with capability changes.

8.4 Event-Sourced Evaluation

Tasks are derived from recent real-world events, incidents, datasets, competitions, or professional work.

LiveBench uses recent sources and periodic updates as part of its contamination-limited design.[^livebench]

Risk:

Events may not form a representative sample.

8.5 Human-and-Model-in-the-Loop Evaluation

Humans generate tasks that expose weaknesses in current systems.

Dynabench demonstrated an adversarial collection process in which annotators attempt to create examples that fool a target model while remaining valid for humans.[^dynabench]

Advantages:

Reveals current failure boundaries
adapts to model progress
can generate difficult examples
creates a feedback loop between evaluation and development

Risks:

Distribution becomes shaped by the current target model
annotator incentives can distort task quality
repeated adversarial rounds may move away from real use
comparisons across rounds require care

8.6 Procedural Generation

Tasks are generated from rules, simulations, formal systems, or parameterized templates.

Advantages:

Large task supply
contamination resistance
adjustable difficulty
reproducibility

Risks:

Generated tasks may be artificial
generators can leak
systems can learn the generator
validity depends on generator quality

8.7 Expert Renewal

Domain experts periodically create, review, and retire tasks.

Useful in:

Cybersecurity
biology
medicine
law
engineering
scientific research
high-stakes agentic work

Risks:

Expensive
slow
subject to expert disagreement
difficult to scale
vulnerable to narrow professional assumptions

8.8 Adaptive Testing

The next task is selected based on prior performance.

Advantages:

Efficient measurement
better targeting of system ability
reduced ceiling and floor effects
can estimate capability with fewer tasks

Risks:

Path dependence
reduced transparency
complex scoring
difficult replication
strategic behavior
dependence on a calibrated item bank

8.9 Dynamic Environment Evaluation

The model operates in an environment that changes in response to its actions.

Useful for:

Agents
cyber ranges
simulations
negotiations
games
scientific workflows
tool use

Risks:

High variance
environment bugs
irreproducible trajectories
hidden dependencies
expensive administration

8.10 Continuous Post-Deployment Evaluation

Evidence is generated from deployed performance through:

Sampling
shadow testing
incident analysis
challenge suites
controlled probes
user feedback
drift detection

This should complement, not replace, pre-deployment evaluation.

8.11 Hybrid Protocols

Most serious frontier protocols will combine several methods.

A possible hybrid design might include:

Stable anchor tasks
rotating held-out tasks
monthly recent-data tasks
expert challenge rounds
system-level simulations
post-deployment monitoring
annual construct review

9. Measurement Validity Under Change

A dynamic protocol is useful only if it continues to measure something meaningful.

9.1 Construct Validity

Construct validity asks whether the protocol actually measures the claimed capability.

Threats include:

Task shortcuts
language dependence
tool dependence
hidden knowledge requirements
scoring artifacts
evaluator expectations
scaffold effects
domain mismatch

9.2 Content Validity

Content validity asks whether the task portfolio adequately covers the construct.

Dynamic protocols should maintain a construct map showing:

Required subdomains
task families
difficulty
modalities
languages
contexts
failure modes
exclusions

9.3 Criterion Relevance

Where possible, evaluation results should be compared with meaningful external outcomes.

Examples:

Professional task success
incident rates
controlled deployment performance
human expert judgment
independent replication
downstream system behavior

The absence of a strong external criterion should be stated.

9.4 Reliability

A reliable protocol produces sufficiently consistent evidence under comparable conditions.

Relevant forms include:

Test-retest reliability
inter-rater reliability
alternate-form reliability
internal consistency
environment stability
run-to-run stability

Reliability does not establish validity.

A protocol can measure the wrong thing consistently.

9.5 Sensitivity and Specificity

For threshold evaluations, maintainers should consider:

Sensitivity to meaningful capability
specificity against false alarms
false positive cost
false negative cost
uncertainty near thresholds

9.6 Consequential Validity

A protocol should examine how its use affects behavior.

Questions include:

Does it distort research priorities?
Does it privilege large organizations?
Does it discourage disclosure?
Does it reward superficial optimization?
Does it create false public confidence?
Does it become a de facto rule without due process?

9.7 Validity Review

Every major version should include a validity argument.

That argument should explain:

What the protocol measures
why the evidence supports that interpretation
major limitations
alternative explanations
decisions the result can support
decisions it cannot support

10. Preserving Comparability Across Versions

Dynamic protocols create a tension.

If nothing changes, validity decays.

If everything changes, historical comparison disappears.

10.1 Anchor Tasks

A subset of tasks can remain stable across versions.

Anchors should be:

Secure enough to retain value
representative
resistant to trivial saturation
statistically useful
reviewed for contamination

Public anchors can support transparency. Private anchors can support stronger leakage control. Both have tradeoffs.

10.2 Parallel Forms

Different task forms can be designed to measure the same construct at similar difficulty.

Parallel forms require validation.

Similarity in topic is not enough.

10.3 Bridging Studies

During a protocol transition, the same systems can be evaluated under old and new versions.

Bridging studies can estimate:

Difficulty shift
score transformation
ranking stability
subgroup effects
configuration sensitivity

10.4 Reference Systems

A stable panel of reference systems can be tested across versions.

Reference systems might include:

Archived open-weight models
reproducible baselines
earlier model snapshots
human baselines
deterministic programs

Closed systems can disappear or change, so they should not be the only anchors.

10.5 Item Response Approaches

Where assumptions are justified, item response models can help estimate latent ability and task difficulty.

They should not be applied mechanically.

Frontier AI systems may violate assumptions common in educational testing because:

Systems can use tools
task performance can be highly discontinuous
model families are not exchangeable populations
prompting alters the evaluated object
memorization and reasoning are difficult to distinguish
strategic behavior may occur

10.6 Score Bands Rather Than False Precision

Version-linked results may be more honest as:

Confidence bands
capability tiers
probability estimates
threshold ranges
qualitative profiles

10.7 Explicit Breaks

When comparability is weak, the report should state:

Version series ended
new baseline established
rankings should not be compared directly
prior thresholds were withdrawn
historical scores remain archival only

10.8 Frozen Historical Reproduction

For some protocols, maintainers should preserve the ability to rerun old versions.

This may require:

Containers
dependencies
prompts
datasets
model snapshots
scoring code
environment images
documentation
security controls

11. Protocol Versioning

A dynamic evaluation protocol should use formal version control.

11.1 Recommended Version Structure

A semantic structure can be adapted:

Major version: Construct, decision use, scoring interpretation, task universe, or system boundary changes materially
Minor version: Task population, difficulty range, environment, thresholds, or important administration details change while the core construct remains
Patch version: Bug fixes, documentation corrections, non-substantive software changes, or scoring repairs that do not alter intended interpretation

Example:

DEP-CYBER-1.4.2

This could identify:

Dynamic Evaluation Protocol
Cyber domain
Major version 1
Minor version 4
Patch version 2

11.2 Required Change Record

Every material update should include:

Change identifier
date
proposer
rationale
evidence
affected components
expected score impact
validation performed
reviewers
conflicts
approval
effective date
transition rules
historical comparability statement
security classification
follow-up review date

11.3 Emergency Revision

Emergency changes may be needed after:

Leakage
scoring failure
critical security flaw
invalid task
unsafe evaluation behavior
legal restriction
material incident

Emergency changes should have:

Narrow scope
temporary status
documented authority
rapid independent review
retrospective validation
clear expiry

11.4 Deprecation Window

Participants should receive reasonable notice before a protocol version is retired, unless continued use would be misleading or unsafe.

11.5 Version Lineage

The protocol should provide a lineage showing:

predecessor
successor
fork
merged protocol
retired branch
compatible versions

12. Lifecycle of a Dynamic Evaluation Protocol

Stage 1: Need Identification

A need may arise from:

New capability
new risk
benchmark saturation
contamination
incident
policy decision
deployment change
research gap
international coordination need

Output:

Need statement
decision question
proposed owner
initial scope

Stage 2: Construct Scoping

Define:

Capability
subcapabilities
exclusions
deployment context
decision use
potential harms from mismeasurement

Output:

Construct map
scope note
stakeholder map

Stage 3: Evidence Review

Review:

Existing benchmarks
task datasets
measurement literature
domain practice
incidents
known failure modes
evaluator methods
standards

Output:

Evidence review
gap analysis
reuse decision

Stage 4: Protocol Design

Specify:

Task universe
sampling
administration
scoring
baselines
access
security
uncertainty
reporting
change rules

Output:

Draft protocol

Stage 5: Task Development

Develop tasks through one or more methods:

Expert authorship
procedural generation
recent-data sourcing
adversarial collection
simulation
incident-derived scenarios

Output:

Task bank
provenance record
validation plan

Stage 6: Technical Validation

Test:

Correctness
difficulty
scoring
ambiguity
contamination
robustness
environment stability
inter-rater agreement
subgroup behavior

Output:

Validation report

Stage 7: Pilot Evaluation

Run with:

Reference models
human baselines
multiple configurations
independent teams where possible

Output:

Pilot results
protocol revisions
unresolved issues

Stage 8: Independent Review

Reviewers assess:

Construct
methods
security
conflicts
interpretation
decision linkage

Output:

Review report
dissent
response

Stage 9: Approval and Publication

Publish appropriate components:

Protocol summary
methodology
version
governance
limitations
reporting format

Sensitive content may remain controlled.

Stage 10: Administration

Run under controlled conditions.

Record:

Configuration
logs
deviations
invalid runs
incidents
personnel
dates

Stage 11: Analysis and Reporting

Report:

Results
uncertainty
configuration
limitations
comparisons
threshold implications
anomalies
dissent

Stage 12: Performance Monitoring

Track:

Saturation
leakage
validity
cost
reliability
disputes
use in decisions
newly discovered failure modes

Stage 13: Revision Decision

Choose:

No change
patch
minor update
major update
fork
suspension
retirement

Stage 14: Transition

Conduct bridging studies, notify users, preserve archives, and update dependent standards.

Stage 15: Retirement or Renewal

A protocol should be renewed only if it continues to justify its institutional cost and interpretive authority.

13. Change Triggers

A protocol should not depend solely on maintainer discretion.

13.1 Performance Triggers

Examples:

Top systems exceed a predefined ceiling
Score variance collapses
Human baseline is exceeded in a way that invalidates earlier interpretation
Task completion becomes trivial under common tools
Failure cases become too rare to estimate

13.2 Contamination Triggers

Examples:

Test items appear in public training corpora
Model outputs reproduce solutions unusually
confidential materials leak
a benchmark-specific fine-tuning dataset emerges
task generator becomes public

13.3 Capability Triggers

Examples:

New modality
longer context
new tool use
autonomous operation
persistent memory
improved test-time computation
multi-agent systems
new deployment scale

13.4 Incident Triggers

Examples:

Real-world failure not represented in the protocol
safeguard bypass
unexpected cross-domain transfer
evaluator security incident
scoring or infrastructure error

13.5 Decision Triggers

Examples:

Protocol begins informing a higher-consequence decision
regulator adopts the result
insurer uses the metric
procurement threshold is attached
certification claim is introduced

13.6 Evidence Triggers

Examples:

New research challenges validity
independent replication fails
domain experts identify missing coverage
demographic or linguistic bias is found
evaluation-to-deployment correlation weakens

13.7 Time Triggers

A periodic review should occur even without an obvious event.

Time-based review is a backstop, not a substitute for evidence-driven triggers.

14. Elicitation, Scaffolding, and the Evaluated System

A dynamic protocol must specify what level of capability it is trying to reveal.

14.1 Five Elicitation Targets

Target A: Default Product Performance

How does the publicly available system behave under ordinary use?

Target B: Standardized Performance

How does the system perform under a common evaluator-defined configuration?

Target C: Developer-Elicited Performance

What can the developer demonstrate using its preferred configuration?

Target D: Evaluator-Optimized Performance

What can qualified evaluators elicit with additional effort, tools, and scaffolds?

Target E: Plausible Maximum Performance

What might the system accomplish under realistic but highly optimized conditions?

These targets can produce different results.

14.2 Why Best-of-N and Retry Policies Matter

Repeated attempts can reveal latent capability, but they can also misrepresent normal use.

Protocols should state:

Number of attempts
selection method
whether failures remain counted
inference budget
human selection
verifier use

14.3 Tool Access

Tools can transform capability.

Examples:

Search
code execution
browser
APIs
databases
laboratory software
communication systems

The protocol should distinguish model capability from system capability when possible.

14.4 System Prompt and Safety Layer

A capability evaluation may test:

Pre-mitigation capability
post-mitigation behavior
safeguard robustness
complete deployed system

These results should not be conflated.

14.5 Developer Assistance

Developer participation can improve elicitation and reduce false negatives. It can also create asymmetry.

A balanced design can include:

Standard evaluation track
developer-submitted track
independent optimization track
documented differences

AISI has emphasized that evaluator access, time, and methodology affect the strength of conclusions drawn from frontier evaluations.[^aisi-lessons] Recent work on external evaluator access similarly distinguishes model access, information access, and evaluation timeframe as separate dimensions.[^access]

15. Dynamic Evaluation of Agents

Agentic systems intensify the need for dynamic protocols.

15.1 Why Agent Tasks Change Quickly

Agent performance depends on:

Tools
environment
task length
interface
memory
feedback
permissions
error recovery
external services
other agents

15.2 Outcome and Process Evidence

A dynamic agent protocol should evaluate:

Final outcome
intermediate actions
policy compliance
resource use
recoverability
oversight responsiveness
deception indicators
persistence
escalation behavior
side effects

15.3 Long-Horizon Measurement

Longer tasks introduce:

Higher variance
more environment dependence
more opportunities for compounding errors
greater sensitivity to scaffolding
higher evaluation cost

METR's work measuring task-completion time horizons demonstrates one possible way to summarize autonomous task capability, while its later methodology updates illustrate why the suite itself must evolve as capabilities and task coverage change.[^metr-time][^metr-update]

15.4 Environment Versioning

Agent evaluations should version:

Operating system
software
dependencies
APIs
network access
task state
external services
simulator
permissions

15.5 Dynamic Adversaries

In security and control evaluations, the environment or defender may adapt to the agent.

This can increase realism but reduce reproducibility.

15.6 Evaluation-Induced Risk

Some agent evaluations may create risk if they provide:

Sensitive tools
live infrastructure access
harmful objectives
uncontrolled communication
persistent credentials

Protocol design should include safety review and containment.

16. Contamination and Evaluation Integrity

Dynamic protocols should manage contamination as an ongoing process.

16.1 Contamination Threat Model

The threat model should cover:

Exact test exposure
near-duplicate exposure
solution exposure
template exposure
metadata leakage
rater leakage
transcript leakage
generator leakage
developer iteration
synthetic derivative data

16.2 Prevention

Controls may include:

Held-out task banks
recent task sourcing
rotating forms
procedural generation
access logging
data minimization
contractual controls
restricted execution environments
task watermarking or canaries where appropriate
delayed release
compartmentalized evaluator access

16.3 Detection

Detection approaches may include:

Similarity analysis
suspicious response matching
canary recovery
performance discontinuity
provenance review
model behavior analysis
developer disclosure
independent investigation

No detection method should be treated as conclusive by default.

16.4 Response

A contamination response plan should specify:

Suspension
investigation
affected result withdrawal
task replacement
re-evaluation
public notice
security review
version update
accountability

16.5 Contamination-Limited, Not Contamination-Free

Claims of being contamination-free should be used cautiously.

For large models trained on broad corpora, proving absence of exposure is often difficult.

"Contamination-limited" or "contamination-resistant" is generally more defensible when evidence supports it.

17. Scoring, Uncertainty, and Reporting

17.1 Score Families

A protocol may report:

Accuracy
success rate
pass rate
calibrated probability
time-to-completion
cost
reliability
severity-weighted failure
capability tier
safeguard bypass rate
human uplift
error profile
qualitative findings

17.2 Multi-Dimensional Reporting

HELM's multi-metric approach illustrates why accuracy alone may fail to capture calibration, robustness, fairness, bias, toxicity, or efficiency.[^helm]

A dynamic protocol should report dimensions separately when aggregation would hide important tradeoffs.

17.3 Uncertainty

Uncertainty can arise from:

Task sampling
model stochasticity
rater disagreement
environment variance
configuration
scoring ambiguity
incomplete coverage
contamination
missing data

17.4 Threshold Uncertainty

Near a consequential threshold, the protocol should avoid false certainty.

Possible responses:

Additional testing
expanded task sample
independent replication
conservative classification
temporary status
decision deferral

17.5 Invalid Runs

The protocol should define:

What makes a run invalid
who decides
whether it is repeated
whether original outcomes remain visible
how invalidation affects uncertainty

17.6 Reporting Minimum

Every public result should include:

Protocol identifier
version
date
evaluated system
configuration
access conditions
evaluator
task count or scope
scoring
uncertainty
limitations
comparability statement
result expiration
conflicts
deviations

17.7 Result Expiration

A result should carry:

Review date
expiration condition
superseding version
current-status indicator

18. Governance

Dynamic evaluation places substantial power in protocol maintainers.

That power must be governed.

18.1 Core Roles

A mature protocol may assign:

Sponsor
protocol owner
construct lead
domain experts
task developers
security custodian
administrator
scorers
independent reviewers
appeals panel
archive custodian
public-interest observer

Roles can be combined in small projects, but conflicts should be documented.

18.2 Change Authority

No single maintainer should be able to make an undisclosed material change to a high-consequence protocol.

18.3 Conflict of Interest

Relevant conflicts include:

Developer funding
evaluation vendor dependence
consulting
employment history
equity
policy advocacy
competitive interest
personal relationships
intellectual commitment to a method

Disclosure alone may not resolve a conflict.

18.4 Dissent

Major protocol decisions should allow:

Minority reports
technical objections
recorded dissent
unresolved issue tracking

18.5 Appeals

Participants should be able to challenge:

Incorrect configuration
procedural deviation
scoring error
undisclosed conflict
invalid task
security breach
interpretation beyond protocol scope

Appeal should not become a mechanism to suppress unfavorable results.

18.6 Public Consultation

Public consultation can improve legitimacy, but sensitive protocols may require restricted technical consultation.

The process should state:

Who can comment
which parts are open
how comments are evaluated
why comments are accepted or rejected

18.7 Funding

Long-term maintenance requires sustainable funding.

Funding models include:

Public grants
membership
evaluation fees
philanthropy
procurement
mixed funding

Each creates incentives.

Dynamic protocols should disclose major funding dependencies and protect revision decisions from payer control.

19. Transparency and Security

19.1 Transparent Process, Protected Content

A dynamic protocol can disclose:

Purpose
construct
governance
validation method
scoring principles
version history
limitations
conflicts

while protecting:

Specific test items
exploit details
sensitive scenarios
model vulnerabilities
personal information
security architecture

19.2 Disclosure Tiers

A possible framework:

Public

General methodology, governance, version, results, limitations.

Registered Research Access

Detailed task taxonomy, validation materials, selected artifacts.

Controlled Evaluator Access

Held-out tasks, administration tools, security procedures.

Restricted Security Access

Highly sensitive content with strict need-to-know controls.

19.3 Delayed Disclosure

Some content can be released after:

Protocol retirement
task rotation
remediation
risk reduction
legal review

19.4 Auditability

Even when content is private, the process should be auditable by qualified independent parties.

19.5 Inspectable Infrastructure

The UK AI Security Institute's Inspect framework demonstrates the value of modular, open evaluation infrastructure that can support diverse tasks, agents, scorers, and model providers.[^inspect] Open tooling does not make every evaluation transparent, but it can improve portability, shared practice, and reproducibility.

20. International Interoperability

Frontier AI evaluation will cross jurisdictions.

20.1 Interoperability Does Not Require Identical Protocols

Institutions can align on:

Definitions
metadata
versioning
confidence reporting
evaluator competence
evidence categories
change-control principles
result status
security classifications

while retaining different task sets or legal uses.

20.2 Mutual Recognition

Mutual recognition may be appropriate when protocols demonstrate:

Comparable constructs
equivalent rigor
qualified evaluators
secure administration
transparent governance
compatible reporting
appeal mechanisms

20.3 Localization

Dynamic protocols should account for:

Language
law
culture
infrastructure
threat environment
professional practice
regional deployment

20.4 Avoiding Lowest-Common-Denominator Alignment

International agreement should not require reducing rigor to the easiest shared metric.

20.5 Standards Alignment

NIST describes testing, evaluation, verification, and validation as part of operationalizing AI risk management, and its AI Resource Center provides related guidance and resources.[^nist-rmf][^nist-airc] Future Standards Body protocols should map to recognized TEVV and standards terminology while remaining explicit where frontier AI requires additional methods.

21. Maturity Model

Level 0: Unmaintained Static Benchmark

Characteristics:

Fixed task set
no expiration
no formal owner
limited versioning
leaderboard-oriented
no change triggers

Use:

Exploratory research only

Level 1: Maintained Benchmark

Characteristics:

Named owner
bug fixes
periodic additions
basic documentation
results tied to versions

Use:

Research comparison

Level 2: Versioned Evaluation Protocol

Characteristics:

Construct definition
administration specification
change control
uncertainty
archival record
validity review
comparability plan

Use:

Serious research and organizational decisions

Level 3: Adaptive and Independently Reviewed Protocol

Characteristics:

Dynamic task renewal
held-out components
independent review
formal security
appeals
bridging studies
expiration
multiple evidence methods

Use:

High-stakes organizational and pre-deployment decisions

Level 4: Interoperable Decision-Linked Evaluation Regime

Characteristics:

Multiple accredited evaluators
international mapping
mutual recognition
standards integration
continuous monitoring
incident feedback
governance oversight
public accountability

Use:

Mature institutional ecosystem

The maturity level should match the consequence of the decision.

22. Implementation Pathway

Phase 1: Inventory

Identify:

Existing benchmarks
owners
users
decision uses
saturation
contamination risk
update status
dependencies

Phase 2: Select a Pilot Domain

Choose a domain with:

Clear need
tractable construct
available experts
measurable tasks
manageable security
potential reference systems

Phase 3: Build the Protocol Charter

Define:

Purpose
scope
governance
funding
access
versioning
review
output

Phase 4: Establish a Stable Core

Create:

Construct map
metadata schema
administration rules
scoring principles
reporting minimum

Phase 5: Add Dynamic Components

Pilot:

Rotating items
recent-data tasks
expert challenge tasks
procedural generation
adaptive difficulty

Phase 6: Validate

Conduct:

Reference-model testing
human baselines
independent review
contamination analysis
reliability analysis
bridging study

Phase 7: Publish Version 1

Publish enough for legitimacy while protecting sensitive content.

Phase 8: Operate and Monitor

Track predefined indicators.

Phase 9: Revise

Use formal change control.

Phase 10: Evaluate the Evaluation

Assess whether the protocol improved decisions, not merely whether it produced scores.

23. Proposed Standards Body Pilot

Standards Body should begin with a bounded pilot rather than attempting to establish a universal protocol.

23.1 Candidate Pilot

Dynamic Evaluation Protocol for Long-Horizon Technical Task Performance

The pilot could examine how well AI systems complete increasingly long, verifiable technical tasks under controlled tool access.

23.2 Why This Domain

It offers:

Clear outputs
practical relevance
measurable success
task renewal opportunities
human comparison
agentic behavior
environment versioning
manageable initial safety profile

23.3 Pilot Components

Stable Core

Construct definition
task taxonomy
administration
human-time baseline method
scoring
reporting
versioning

Dynamic Layer

New task additions
difficulty calibration
current software environments
adversarially identified failure tasks
rotating held-out suite

Review

Software engineers
evaluation scientists
security reviewers
independent researchers

23.4 Pilot Outputs

Protocol specification
task-development guide
validation report
reference results
change-control manual
public methodology
evaluator package
first annual revision report

23.5 Pilot Success Criteria

The pilot succeeds if it demonstrates:

Better discrimination than a fixed suite
defensible version transitions
manageable cost
evaluator reproducibility
useful failure analysis
clear limits
credible governance

24. Metrics for Evaluating the Protocol

The evaluation protocol itself should be evaluated.

24.1 Measurement Quality

Ceiling rate
floor rate
discrimination
reliability
uncertainty
inter-rater agreement
task validity
subgroup coverage

24.2 Integrity

Contamination incidents
unauthorized access
suspicious response patterns
invalidated results
security findings

24.3 Operational Performance

Cost
time
evaluator burden
infrastructure failure
task-development throughput
review delay

24.4 Institutional Quality

Conflict disclosures
independent reviews
appeal resolution
change-record completeness
dissent handling
funding concentration

24.5 Decision Utility

Decision-makers who use the evidence
changes made because of findings
avoided errors
correlation with later outcomes
user understanding
rate of overinterpretation

24.6 Adaptation Quality

Time from trigger to revision
bridging-study completion
comparability quality
obsolete component retirement
unresolved issue closure

25. Failure Modes and Safeguards

25.1 Change for Appearance

Failure: Frequent updates create the appearance of sophistication without improving validity.

Safeguard: Every material update requires an evidence-based rationale and post-change review.

25.2 Loss of Comparability

Failure: Versions cannot be compared, but rankings are presented as continuous.

Safeguard: Bridging studies and explicit discontinuity statements.

25.3 Maintainer Capture

Failure: A developer, government, funder, or evaluator controls revisions.

Safeguard: Distributed governance, conflicts, dissent, independent review.

25.4 Security as a Shield

Failure: Confidentiality prevents scrutiny of weak methods.

Safeguard: Controlled independent audit and public methodological disclosure.

25.5 Overfitting to Frontier Systems

Failure: Adversarial task generation becomes too specific to one model family.

Safeguard: Multiple reference systems, human validation, real-use sampling, diversity review.

25.6 Artificial Difficulty

Failure: Tasks become harder without becoming more meaningful.

Safeguard: Construct map and criterion relevance review.

25.7 Evaluation Arms Race

Failure: Protocols and developers escalate complexity without improving decision quality.

Safeguard: Cost-benefit review and minimum necessary change.

25.8 Excessive Automation

Failure: Generated tasks and model judges introduce hidden errors.

Safeguard: Human audits, objective scoring where possible, judge validation, uncertainty.

25.9 Excessive Expert Dependence

Failure: Small expert groups become bottlenecks or sources of bias.

Safeguard: Rotation, structured rubrics, diverse panels, scalable validation.

25.10 Unstable Thresholds

Failure: Decisions change because thresholds or scores move, not because capability changes.

Safeguard: Threshold governance, transition rules, decision impact analysis.

25.11 Protocol Proliferation

Failure: Too many overlapping protocols create confusion.

Safeguard: Registry, taxonomy, interoperability review, consolidation.

25.12 Goodhart Effects

Failure: The protocol becomes a target and ceases to be a useful measure.

Safeguard: Multiple methods, rotating components, held-out tasks, incident evidence, periodic construct review.

25.13 False Assurance

Failure: Passing the protocol is interpreted as proof of safety.

Safeguard: Explicit claims boundary, complementary evidence, result expiration.

25.14 Accessibility Failure

Failure: Only the largest laboratories can participate.

Safeguard: Tiered access, shared infrastructure, subsidized evaluation, open reference tracks.

25.15 Governance Latency

Failure: Revision is too slow for capability change.

Safeguard: Predefined triggers, emergency procedures, standing expert groups.

25.16 Metric Churn

Failure: Stakeholders cannot build stable processes because metrics constantly change.

Safeguard: Stable core, scheduled transitions, version support windows.

26. Serious Objections

Objection 1: Dynamic Protocols Reduce Reproducibility

This objection is valid.

Changing tasks and environments makes exact replication harder.

Response:

Preserve stable anchors
archive versions
use controlled-access replication
distinguish exact replication from conceptual replication
document transitions
report when comparison is not possible

Residual concern:

Some dynamic, adversarial, or interactive evaluations may remain intrinsically difficult to reproduce.

Objection 2: Dynamic Protocols Give Maintainers Too Much Power

Also valid.

Maintainers can influence:

What counts
what changes
who participates
what becomes public
how scores are interpreted

Response:

Governance
transparent change records
independent review
appeals
conflicts
minority reports
multiple protocol providers

Residual concern:

Institutional power cannot be eliminated. It must be constrained and contestable.

Objection 3: Static Benchmarks Are More Scientific

Static benchmarks often support cleaner comparison.

Response:

Retain static components
avoid unnecessary change
use dynamic methods only where validity decay justifies them

Residual concern:

Some research questions are better served by fixed benchmarks. Dynamic evaluation should not become an ideology.

Objection 4: Dynamic Protocols Are Too Expensive

They require ongoing staff, task development, security, infrastructure, and review.

Response:

Match protocol maturity to decision consequence
share infrastructure
reuse validated components
prioritize high-value domains
retire low-value protocols

Residual concern:

A dynamic regime can become bureaucratic. Cost should be measured explicitly.

Objection 5: Developers Will Still Optimize Against the Protocol

Yes.

Response:

Diversify evidence
protect some content
rotate tasks
monitor for gaming
use real-world and incident evidence
evaluate generalization

Residual concern:

No evaluation system is immune to optimization.

Objection 6: Dynamic Tasks Can Become Arbitrary

Response:

Maintain a construct map
require validity evidence
use expert review
test against external outcomes
preserve inclusion rules

Residual concern:

Constructs such as "general intelligence" or "dangerous capability" may remain contested.

Objection 7: Recent Data Does Not Guarantee Better Evaluation

Correct.

Recent tasks can be noisy, unrepresentative, or shallow.

Response:

Recent-data evaluation should be one component, not the definition of dynamic evaluation.

Objection 8: A Single Global Protocol Is Unrealistic

Agreed.

Response:

The goal should be interoperable protocols, not one universal test.

Objection 9: Evaluation Cannot Keep Pace With Frontier Development

This may sometimes be true.

Response:

Scalable automated screens
expert deep dives
trigger-based testing
shared infrastructure
pre-registered protocol families
evaluation during development

OpenAI's updated Preparedness Framework explicitly links faster model improvement to the need for scalable, more frequent evaluations while retaining expert-led deep dives.[^openai-pf]

Residual concern:

Evaluation capacity may still lag. Protocols should state when evidence is incomplete rather than creating false confidence.

Objection 10: Dynamic Evaluation Encourages Premature Governance

Response:

This foundation proposes evidence infrastructure, not automatic regulation.

The decision use should remain explicit and proportionate.

27. Evidence Gaps

The field lacks strong evidence on several foundational questions.

27.1 Comparability

How well do common equating methods work for rapidly changing AI systems?

27.2 Contamination

Which contamination-detection methods are reliable across closed training pipelines?

27.3 Elicitation

How much measured capability variation is attributable to prompting, scaffolding, and evaluator effort?

27.4 Predictive Validity

Which evaluation results predict real-world deployment outcomes?

27.5 Adversarial Data

When does human-and-model-in-the-loop collection improve robustness, and when does it create unrealistic distributions?

Theoretical work on dynamic benchmarks highlights that the interaction between model fitting and data collection requires distinct analysis from static benchmarking.[^dynamic-theory]

27.6 Model Judges

When are model-based judges sufficiently reliable, and where do they create systematic bias?

27.7 Agent Evaluation

How should long-horizon, stochastic, and environment-dependent performance be summarized?

27.8 Governance

Which governance structures revise protocols quickly without enabling capture?

27.9 International Recognition

What evidence is sufficient for one evaluator or jurisdiction to recognize another's result?

27.10 Expiration

How should result expiration be determined empirically?

27.11 Decision Impact

Do dynamic protocols improve real decisions enough to justify their cost?

28. Research Agenda

Priority 1: Protocol Comparability

Develop and test:

Anchor strategies
bridging designs
reference panels
score-linking methods
discontinuity criteria

Priority 2: Dynamic Task Quality

Compare:

Expert-authored
adversarial
procedural
event-sourced
model-generated
hybrid tasks

Priority 3: Contamination Resistance

Evaluate:

recent-data methods
controlled task banks
canaries
provenance
generator security
leakage detection

Priority 4: Elicitation Standards

Develop reporting standards for:

Prompting
tools
retries
compute
scaffolds
developer assistance
evaluator optimization

Priority 5: Agent Protocols

Study:

Environment versioning
trajectory scoring
task length
recovery
oversight
process evidence
dynamic adversaries

Priority 6: Evaluation Governance

Pilot:

Change boards
independent review
appeal systems
public consultation
emergency revision
multi-institution stewardship

Priority 7: Result Expiration

Develop:

Time-based rules
event-based rules
confidence decay
re-evaluation triggers
public status labels

Priority 8: Protocol Performance Metrics

Measure whether evaluation systems themselves remain valid, efficient, and decision-useful.

Priority 9: Interoperability

Create:

Metadata standards
protocol registries
crosswalks
mutual-recognition criteria
shared terminology

Priority 10: Evaluation Safety

Develop controls for evaluation activities that could create security, privacy, or misuse risk.

29. Near-Term Experiments

Experiment 1: Static versus Rolling Task Suite

Run the same reference systems on:

A fixed suite
a monthly refreshed suite
a held-out suite

Compare:

Ranking
uncertainty
contamination indicators
cost
failure discovery

Experiment 2: Anchor Design

Test whether stable anchors preserve useful longitudinal measurement after substantial task refresh.

Experiment 3: Human-and-Model-in-the-Loop Collection

Compare adversarial tasks against naturally sampled professional tasks.

Experiment 4: Elicitation Matrix

Evaluate the same system under:

Default
standardized
developer-optimized
evaluator-optimized configurations

Experiment 5: Version Transition

Build Protocol 1.0 and 2.0, conduct a bridging study, and test whether independent analysts reach similar comparability conclusions.

Experiment 6: Result Expiration Labels

Test whether users interpret results more accurately when reports include explicit freshness and expiration status.

Experiment 7: Governance Simulation

Run a mock change-control process with:

Maintainers
developer representatives
independent experts
public-interest reviewers
appeals panel

Experiment 8: Dynamic Agent Environment

Introduce controlled environment updates and measure how much apparent capability change comes from the model versus the environment.

30. Implications for Future Standards

A future standard for dynamic evaluation protocols could require:

30.1 Protocol Identification

Unique identifier, owner, version, status, and scope.

30.2 Construct Specification

Definition, exclusions, intended use, and evidence basis.

30.3 Evaluated-System Specification

Model and system configuration metadata.

30.4 Change-Control Plan

Triggers, authority, review, validation, and publication.

30.5 Comparability Plan

Anchors, bridging, reference systems, and discontinuity criteria.

30.6 Integrity Plan

Contamination, security, access, logging, and incident response.

30.7 Validation Plan

Reliability, validity, robustness, fairness, and uncertainty.

30.8 Reporting Minimum

Results, limitations, configuration, uncertainty, conflicts, and expiration.

30.9 Governance

Roles, conflicts, dissent, appeals, and funding disclosure.

30.10 Retirement

Conditions, process, archive, and successor mapping.

Such a standard should be developed through the future STANDARDS_DEVELOPMENT_PROCESS.md, not declared unilaterally by this paper.

31. Relationship to the Other Foundations

Foundation 2: Held-Out Evaluations

Dynamic protocols often require rotating confidential content to reduce leakage.

Foundation 3: High-Stakes Capability Evaluation

The consequence of the decision should determine the depth, security, and review level of the dynamic protocol.

Foundation 4: Independent Expert Review

Independent reviewers challenge construct definitions, methods, revisions, and interpretations.

Foundation 5: Third-Party Auditor Ecosystem

Dynamic protocols require qualified organizations capable of consistent administration across versions.

Foundation 6: Progressive Standards and Requirements

Dynamic protocols may begin as voluntary research practice and later support procurement, certification, insurance, or formal requirements.

Foundation 7: Incentives and Prestige

Recognition should reward participation in rigorous, revisable evaluation rather than benchmark marketing.

Foundation 8: Global Interoperability

Protocol metadata, versioning, evidence standards, and recognition should work across borders.

32. Canonical Standards Body Positions

Standards Body adopts the following positions as the current working foundation.

Static benchmarks remain useful, but are insufficient as the sole basis for frontier AI evaluation.
The evaluation protocol, not the dataset alone, is the proper unit of governance and validation.
Dynamic evaluation means evidence-driven maintenance, not arbitrary or constant change.
Every high-consequence protocol should define what remains stable and what may change.
Historical comparability must be demonstrated, not assumed.
A major protocol change may require a new baseline rather than a forced continuous ranking.
Evaluation results should be tied to exact system configurations and dates.
Results should have expiration or re-evaluation conditions.
Dynamic protocols should combine multiple forms of evidence when one method is insufficient.
Held-out content can coexist with transparent governance.
Independent review is required when protocol results influence consequential decisions.
Protocol maintainers should be subject to conflict disclosure, dissent, and appeal mechanisms.
Evaluation difficulty should not be confused with evaluation validity.
Recent-data tasks should not be treated as automatically representative.
Model-judge and automated task-generation methods require validation.
Agent evaluation requires environment, tool, and trajectory versioning.
Passing an evaluation is not proof of safety.
Protocols should be retired when their interpretive authority exceeds their evidence.
Interoperability is preferable to forced global uniformity.
The evaluation system itself should be continuously evaluated.

33. Decision Rules

A protocol should be revised when one or more of the following is true:

Its tasks no longer discriminate among relevant systems
credible contamination undermines interpretation
the evaluated system architecture changes materially
deployment conditions move outside the protocol scope
new evidence challenges construct validity
a material real-world failure is absent from coverage
scoring error affects results
the protocol begins supporting a higher-consequence decision
independent replication fails
governance or security integrity is compromised

A protocol should not be revised merely because:

a stakeholder dislikes the result
a developer requests easier tasks
maintainers want publicity
a new benchmark is fashionable
rankings become politically inconvenient
a funder prefers a different conclusion

A protocol should be retired when:

Validity cannot be restored
maintenance cost exceeds decision value
security is irreparably compromised
the construct is obsolete
a superior successor exists
comparisons are routinely misused despite safeguards
governance legitimacy fails

34. Protocol Template

A future protocol should include the following sections.

A. Identity

Name
identifier
version
owner
status
date
predecessor
successor

B. Purpose

Decision question
users
intended use
prohibited use

C. Construct

Definition
subcomponents
exclusions
theory of measurement

D. Evaluated Object

Model
system
tools
configuration
environment

E. Task Universe

Population
sampling
generation
provenance
coverage

F. Administration

Access
limits
tools
retries
human assistance
logs

G. Scoring

Metrics
rubrics
judges
aggregation
invalid runs
uncertainty

H. Validation

Reliability
validity
robustness
fairness
baselines
contamination

I. Security

Classification
access control
storage
disclosure
incident response

J. Governance

Roles
conflicts
approval
dissent
appeals

K. Versioning

Change triggers
version policy
bridging
transition
archive

L. Reporting

Required metadata
result format
limitations
expiration
comparability

M. Retirement

Criteria
process
archive
successor

35. Change Request Template

Protocol:
Current version:
Proposed version:
Proposer:
Date:

Proposed Change

Describe the change precisely.

Trigger

Identify the evidence or event requiring change.

Rationale

Explain why the current protocol is insufficient.

Components Affected

Construct
task population
administration
scoring
threshold
security
reporting
governance
other

Expected Impact

Describe likely effects on:

Scores
rankings
uncertainty
cost
security
accessibility
decisions

Validation Plan

Explain how the change will be tested.

Comparability Plan

Explain how old and new versions will be linked, or state why they cannot be linked.

Conflicts

List relevant interests.

Reviewers

Identify technical, domain, security, and independent reviewers.

Decision

Approved
approved with conditions
pilot only
deferred
rejected

Effective Date

Follow-Up Review

36. Protocol Scorecard

A protocol can be reviewed against the following dimensions:

Dimension	Core Question
Purpose	Is the decision question explicit?
Construct	Is the measured property clearly defined?
Coverage	Does the task portfolio represent the construct?
Integrity	Is contamination and gaming risk managed?
Reliability	Are results sufficiently consistent?
Validity	Does evidence support the intended interpretation?
Elicitation	Are prompting, tools, and scaffolds specified?
Configuration	Is the evaluated system precisely identified?
Comparability	Are version-to-version claims justified?
Security	Are sensitive components protected proportionately?
Transparency	Is enough disclosed for accountability?
Governance	Are authority and conflicts controlled?
Appeals	Can material errors be challenged?
Freshness	Is result expiration defined?
Interoperability	Can others understand and map the result?
Cost	Is the protocol proportionate to its decision value?
Accessibility	Can qualified smaller actors participate?
Retirement	Can obsolete authority be withdrawn?

37. Final Perspective

Frontier AI evaluation will fail if it is treated as a sequence of permanent leaderboards.

The systems being measured are changing.

The ways they are trained are changing.

The tools surrounding them are changing.

The environments in which they act are changing.

The risks, benefits, and decisions attached to their performance are changing.

Evaluation infrastructure must therefore be capable of change as well.

But change alone is not the objective.

A protocol that changes constantly without stable meaning is not dynamic in a useful sense. It is unstable.

The objective is disciplined adaptation.

Dynamic evaluation protocols should make it possible to improve what is measured while preserving the evidence required to understand how and why the measurement changed.

They should support progress without allowing yesterday's evidence to govern tomorrow's systems indefinitely.

They should make uncertainty visible.

They should reveal when comparisons are weak.

They should state when an evaluation has expired.

They should allow disagreement.

They should retain history.

They should be governed in proportion to the consequences of their use.

The first foundation of Standards Body is therefore not a particular benchmark.

It is the institutional capacity to keep evaluation meaningful.

References and Research Basis

[^nist-rmf]: National Institute of Standards and Technology, AI Risk Management Framework and Generative AI Profile. https://www.nist.gov/itl/ai-risk-management-framework

[^nist-airc]: National Institute of Standards and Technology, AI Resource Center, including testing, evaluation, verification, and validation resources. https://airc.nist.gov/

[^helm]: Percy Liang et al., Holistic Evaluation of Language Models, 2022, revised 2023. https://arxiv.org/abs/2211.09110

[^dynabench]: Douwe Kiela et al., Dynabench: Rethinking Benchmarking in NLP, 2021. https://arxiv.org/abs/2104.14337

[^dynamic-theory]: Ali Shirali, Rediet Abebe, and Moritz Hardt, A Theory of Dynamic Benchmarks, ICLR 2023. https://arxiv.org/abs/2210.03165

[^livebench]: Colin White et al., LiveBench: A Challenging, Contamination-Limited LLM Benchmark, 2024. https://arxiv.org/abs/2406.19314

[^dynamic-survey]: Shijie Chen et al., Recent Advances in Large Language Model Benchmarks Against Data Contamination: From Static to Dynamic Evaluation, 2025. https://arxiv.org/abs/2502.17521

[^aisi-lessons]: UK AI Security Institute, Early Lessons from Evaluating Frontier AI Systems, 2024. https://www.aisi.gov.uk/blog/early-lessons-from-evaluating-frontier-ai-systems

[^inspect]: UK AI Security Institute, Inspect AI, open evaluation framework. https://inspect.aisi.org.uk/

[^metr-time]: METR, Measuring AI Ability to Complete Long Tasks, 2025. https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/

[^metr-update]: METR, Time Horizon 1.1, 2026. https://metr.org/blog/2026-1-29-time-horizon-1-1/

[^openai-pf]: OpenAI, Preparedness Framework v2, 2025. https://openai.com/index/updating-our-preparedness-framework/

[^deepmind-fsf]: Google DeepMind, Frontier Safety Framework, updated 2025. https://deepmind.google/blog/updating-the-frontier-safety-framework/

[^extreme-risk]: Toby Shevlane et al., Model Evaluation for Extreme Risks, 2023. https://arxiv.org/abs/2305.15324

[^dangerous-capabilities]: Mary Phuong et al., Evaluating Frontier Models for Dangerous Capabilities, 2024. https://arxiv.org/abs/2403.13793

[^access]: Jacob Charnock et al., Expanding External Access to Frontier AI Models for Dangerous Capability Evaluations, 2026. https://arxiv.org/abs/2601.11916

Revision Record

Version 1.0

Date: July 16, 2026

Change type: Complete replacement

Summary: Replaces the earlier outline edition with a fully developed canonical working white paper. Adds first-principles rationale, definitions, protocol architecture, taxonomy of dynamic methods, measurement validity, version comparability, lifecycle, change triggers, elicitation standards, agent evaluation, contamination controls, scoring, governance, transparency, interoperability, maturity model, implementation pathway, Standards Body pilot proposal, protocol metrics, failure analysis, objections, evidence gaps, research agenda, standards implications, templates, canonical positions, and primary-source research basis.

Status: Ready for internal review and future expert critique.