Standards Body · Foundation paper, public edition · Released July 17, 2026
Canonical record: https://standardsbody.ai/library/foundation-paper/held-out-evaluations/
Standards Body is an independent research and institutional-design project. It is not currently a regulator, accreditation body, certification body, or governmental authority. This document is research; it is not an adopted standard.
Series: Foundations for Frontier AI Evaluation Infrastructure
Version: 1.0
Status: Canonical working white paper
Project: Standards Body
Primary domain: standardsbody.ai
Core line: Foundations for Frontier AI
Research basis reviewed through: July 16, 2026
Document owner: Standards Body
Review cycle: Annual review, with event-triggered revision after material leakage, compromise, methodological change, or new evidence
This paper defines the Standards Body position on held-out evaluations for frontier artificial intelligence.
It is intended to serve as:
This paper is not a universal security standard, legal rule, or certification scheme.
It does not argue that every evaluation should be secret.
It does not claim that held-out tasks automatically produce valid results.
It establishes the principles and institutional conditions required for held-out evaluation to improve measurement rather than merely hide it.
Public benchmarks have played a central role in artificial intelligence research.
They enable comparison, reproducibility, open criticism, and rapid experimentation. Public tasks often help researchers understand what systems can do, identify weaknesses, and coordinate around shared measurement.
But public visibility creates pressure on the measurement itself.
Once benchmark items, formats, answer keys, scoring rules, and common solution strategies are broadly available, they can enter:
A system may then perform well because it has encountered the evaluation, close variants, or its underlying template. Even when exact exposure cannot be proven, repeated optimization against a known benchmark can weaken the connection between benchmark performance and genuine generalization.
Held-out evaluations address this problem by reserving some evaluation material from normal development access.
The reserved material may include:
The objective is not secrecy for its own sake.
The objective is to preserve the informational value of the evaluation.
A held-out evaluation should help answer questions such as:
Held-out evaluation is not a complete solution.
A private test can still be:
Confidentiality can protect measurement integrity, but it can also shield weak methods from scrutiny.
The central institutional challenge is therefore to combine protected content with inspectable process.
Standards Body adopts the following core position:
Some frontier AI evaluations should contain held-out components when public exposure would materially reduce their validity. Those components should be governed through proportionate security, independent oversight, documented provenance, controlled access, reproducible administration, transparent methodology, explicit compromise response, and clear limits on interpretation.
Held-out evaluation should be understood as a spectrum, not a binary category.
A protocol may keep private:
Different choices create different scientific and institutional tradeoffs.
A mature held-out evaluation ecosystem should distinguish at least five questions:
The strongest held-out systems do not rely on obscurity alone.
They use defense in depth:
The purpose of this foundation is to make held-out evaluation a credible scientific and institutional practice rather than an informal collection of secret test questions.
When public exposure would materially weaken an evaluation, some content should remain unavailable during system development and should be administered under controlled conditions.
This thesis depends on four qualifications.
First, secrecy must serve a defined measurement purpose.
Second, the protected evaluation must still satisfy scientific and institutional standards.
Third, the process must be accountable even when the content is restricted.
Fourth, held-out evidence should complement other evidence rather than become the sole basis for broad claims.
A held-out evaluation is valuable only to the extent that it provides information not already available through public optimization.
The information gain may come from:
If a held-out test closely reproduces public material, its secrecy may add little.
Maintaining a credible holdout is an institutional function, not merely a data-storage function.
It requires:
The more consequential the result, the stronger the obligation to provide fair notice, consistent administration, reviewable process, and a path to challenge material error.
A developer should not need access to exact tasks in order to understand:
This paper covers held-out components used in:
Held-out evaluations may apply to:
This paper does not claim that:
Held-out evaluation is closely connected to FOUNDATION_01_DYNAMIC_EVALUATION_PROTOCOLS.md.
A static private test can become stale, leak, saturate, or lose relevance.
A strong holdout therefore requires:
Held-out systems concentrate information in a smaller number of institutions.
That makes independent review more important, not less.
Security protects:
Security should not be used to avoid methodological scrutiny.
A held-out evaluation is an evaluation in which one or more material components are intentionally unavailable to the evaluated system's developer, training process, tuning process, or ordinary users before administration.
A holdout set is a reserved collection of tasks, examples, environments, or observations not used for development or routine validation.
A private benchmark is a benchmark whose content is not generally available.
A private benchmark may or may not be well governed.
A confidential evaluation is an evaluation whose content, process, results, or combination is restricted to authorized parties.
Confidentiality can serve security, privacy, commercial, or measurement purposes.
"Secret evaluation" is not preferred Standards Body terminology because it does not identify who is authorized, what is restricted, or why.
Use more precise terms such as:
A blind evaluation restricts information from one or more participants to reduce bias or gaming.
Examples include:
A double-blind design restricts relevant information from both the evaluated party and the assessor where feasible.
In frontier AI evaluation, perfect double blinding is often difficult because system interfaces, behavior, or infrastructure can reveal identity.
An embargoed evaluation remains restricted until a defined time or event.
A retired holdout is no longer used for active decision-making.
It may be:
Evaluation contamination occurs when information about the evaluation enters the development or optimization process in a way that weakens the intended separation between training and testing.
Leakage is unauthorized or unintended disclosure of protected evaluation information.
An evaluation is compromised when its integrity, confidentiality, validity, or fairness has been materially weakened.
Chain of custody is the documented history of who created, accessed, transferred, administered, modified, or stored evaluation materials.
An access tier is a defined level of authorization governing which evaluation components a party may see or use.
A controlled evaluation environment is a technical and procedural setting designed to limit unauthorized access, data extraction, harmful action, and unlogged modification.
Attestation is evidence used to determine whether a computing environment, code package, or security state is what it claims to be.
A cryptographic commitment allows a party to commit to data or a decision before disclosure while later enabling verification that it was not altered.
A retro-holdout is a newly created private evaluation designed to approximate the construct or distribution of an older public benchmark after the original benchmark may have become contaminated.[^retro-holdout]
A public benchmark is not only a neutral test.
Once it becomes influential, it becomes part of the environment that researchers and developers optimize within.
This can improve systems.
It can also reduce the benchmark's value as independent evidence.
Exact contamination occurs when evaluation items or solutions appear in training or tuning data.
The resulting score may partly reflect recall.
A model may encounter paraphrases, transformed versions, translations, explanations, or synthetic variants.
Exact-string matching may therefore underestimate exposure.
A model may learn the recurring structure of an evaluation even when individual items are new.
Examples:
Repeated testing on the same benchmark can influence:
The evaluation then becomes part of development.
Popular benchmarks often generate:
Research on retro-holdouts has reported that performance on newly constructed private analogues can reveal gaps not visible on established public benchmarks.[^retro-holdout]
This does not prove that all public benchmark scores are inflated by the same amount.
It supports the narrower conclusion that private comparison sets can reveal measurement weaknesses that public scores alone may miss.
For closed or extremely large training pipelines, external evaluators may not know whether evaluation content was included.
A holdout created after the relevant training cutoff can reduce some forms of contamination, but it does not eliminate:
Even with clean training data, developers can optimize around a known benchmark through repeated evaluation.
The resulting performance may be real, but the benchmark becomes less independent as evidence.
When a result influences:
the cost of a misleading score increases.
Held-out components become more valuable when the consequence of false confidence is high.
A well-designed holdout can improve evidence about:
A held-out result cannot by itself prove:
A perfectly secure evaluation can measure the wrong thing.
Security preserves the test as designed.
It does not establish that the design is meaningful.
The result depends on:
Failure on a held-out task may mean:
Success under a privileged evaluator configuration may not imply the same capability is available in normal use.
Held-out evaluation should be combined with:
A fixed set remains confidential.
Advantages:
Risks:
Tasks are replaced periodically.
Advantages:
Risks:
The administrator samples from a larger protected bank.
Advantages:
Risks:
Tasks are generated from protected rules, seeds, or environments.
Advantages:
Risks:
Tasks are based on information created after the presumed training cutoff.
Advantages:
Risks:
LiveBench and related dynamic approaches illustrate this family of contamination-limited evaluation.[^livebench]
Tasks are derived from recent:
Advantages:
Risks:
Domain experts create protected tasks.
Advantages:
Risks:
The UK AI Security Institute has described structured processes for developing frontier question-answer evaluations, including expert involvement, question design, quality review, and controlled administration.[^aisi-qa]
Evaluators create or select attacks intended to expose weaknesses.
Useful for:
Risk:
The system operates in an environment with hidden:
Useful for agents.
Risk:
Tasks remain private during active use and become public later.
Advantages:
Risks:
No single party holds all sensitive components.
For example:
Advantages:
Risks:
The model is evaluated in a controlled environment without transferring model weights or test data broadly.
Advantages:
Risks:
The evaluator knows the tasks, while the developer does not.
Common and practical.
Risk:
The developer knows the broad evaluation but the model cannot access protected task metadata or external leakage channels.
Useful in agent sandboxes.
Risk:
Raters do not know which system produced outputs.
Useful for reducing brand and expectation bias.
It does not address training contamination.
A mature protocol may combine:
A credible holdout begins with a threat model.
Assets may include:
Threats may come from:
A threat model should not assume malicious developers by default.
Many failures arise from ordinary operational mistakes.
A threat actor may seek to:
Potential surfaces include:
Advanced systems may:
AISI research has examined what sandboxed agents can infer about evaluation environments and how container isolation should be assessed.[^aisi-sandbox-learning][^aisi-breakout]
A held-out system may seek:
No holdout should be described as perfectly secure.
The threat model should state residual risk and assumptions.
Restrict only what must be restricted to preserve validity, safety, privacy, or legitimate proprietary interests.
The capability, risk, or behavior being evaluated should normally be public or reviewable.
Participants should know the general domain, administration rules, and decision consequences.
A qualified party not controlled by the primary developer should be able to verify material aspects of the process.
Do not rely on one control.
Combine:
Each participant receives only the access needed for their role.
Task creation, administration, scoring, approval, and appeals should be separated where consequence warrants it.
Every task should have a documented origin and modification history.
The same protocol should be administrable by qualified evaluators under controlled conditions.
Disclosure should be sufficient for accountability without destroying the test.
Held-out results should expire, be renewed, or be retired.
The protocol should assume that some protected content will eventually leak.
Security should not unnecessarily exclude smaller qualified evaluators, researchers, or open-source communities.
No single private test should become the only accepted measure of a broad capability.
Material errors and procedural failures should be appealable.
Security investment should reflect:
Complex controls should not substitute for sound evaluation design.
A protocol should specify the minimum necessary protected surface.
Protect when exact exposure would enable memorization or targeted tuning.
Protect when answers reveal task construction or enable reverse engineering.
Protect when predictability allows developers to narrow preparation to a small subset.
Protect when access would enable generation of near-identical training data.
Protect when disclosure would:
Threshold confidentiality is controversial.
Potential rationale:
Potential harms:
Standards Body position:
Decision thresholds should normally be disclosed at least in principle. Temporary restriction may be justified in narrow cases, but hidden thresholds should not become the default for consequential decisions.
Some hidden state can test generalization.
Core administration conditions should still be documented.
Raters may be blinded to model identity to reduce bias.
Result embargo may be justified for:
Indefinite suppression requires stronger justification.
Evaluator identity may need protection in sensitive work.
Governance should still verify competence and conflicts.
Each task should include:
Tasks may come from:
Risks include:
Models can help generate tasks, but the protocol should record:
A model trained on public benchmarks may reproduce benchmark-like tasks or solutions.
Expert review should assess:
High-consequence tasks should have more than one qualified solution review where feasible.
Organize tasks into families so coverage can be evaluated.
Each administration should update:
Retire a task when:
A held-out set should represent a defined population, not merely a collection of difficult questions.
Specify whether sampling is:
Sample size should reflect:
Repeated runs may estimate:
They also increase exposure.
If different participants receive different forms, the protocol should assess:
Near a threshold:
Human comparison should specify:
Define treatment of:
NIST has emphasized the value of richer item-level analysis and statistical modeling because aggregate benchmark scores can hide uncertainty and assumptions.[^nist-statistical]
Held-out systems should retain item-level evidence under appropriate controls.
A private score is not more precise merely because the tasks are secret.
Each component should be classified by sensitivity.
Example levels:
Controls may include:
Use:
Use:
Conventional encryption does not protect data while actively processed.
Confidential computing uses hardware-based, attested trusted execution environments to protect data in use.[^ccc-technical]
Potential use cases include:
Limitations include:
Agentic evaluations may require isolated environments.
AISI has released sandboxing tools for Inspect and has emphasized that sandbox selection should match the threat model.[^aisi-sandbox]
A sandbox is not automatically secure.
Validate:
Logs should support:
Logs can themselves expose tasks or sensitive model behavior.
Task-author and evaluator devices are common weak points.
Third-party platforms should be assessed for:
A secure system should recover from:
Secure deletion may be appropriate for:
Permanent destruction should not eliminate necessary audit records.
Available:
Available:
Available:
Requirements:
For:
Additional requirements:
Limited personnel with authority over:
Developers may receive:
They should not receive active items unless the protocol explicitly requires collaborative testing.
Integration failures sometimes require developer visibility.
Use:
Avoid exposing active holdouts merely to solve ordinary integration problems.
Structured external access can improve critique and innovation.
Recent work on external access distinguishes dimensions such as model access, information access, and evaluation time as separate determinants of evaluator effectiveness.[^external-access]
Record:
Each event should include:
Evaluation packages should have:
Confirm:
Confirm:
Preserve:
Do not overwrite original evidence.
Before seeing results, record:
The exact tasks may remain private.
Confirm the evaluated system identity through available methods.
Document and, where possible, lock:
Use non-active tasks to test integration.
Apply:
NIST guidance on evaluation cheating emphasizes transcript review as a way to detect cheating, integration problems, tool issues, and task failures.[^nist-cheating]
Transcript review should combine:
Use the pre-specified method.
Any post-hoc scoring change should be recorded and justified.
Ambiguous cases may require:
Before public release, the developer may be given a limited period to identify:
The developer should not receive veto authority over unfavorable findings.
Publish according to the disclosure plan.
Possible outputs include:
Hidden scoring rules can reduce gaming but weaken fairness.
Preferred approach:
Model judges may reduce cost.
They require validation for:
Human scoring should document:
Include:
Use bands or ranges when point estimates imply unjustified certainty.
A public report should include:
A controlled annex may include:
State what the result cannot establish.
Useful but not always compatible with a live holdout.
A qualified independent evaluator reruns the protocol under controlled access.
Researchers recreate the method using different tasks.
An auditor verifies:
without receiving unrestricted task access.
Tasks are released after retirement.
Potential tools include:
These can show that materials were not changed after commitment.
They do not establish:
Several institutions can independently verify parts of the process.
A controlled replication package can include:
The choice is not between total secrecy and total openness.
A protocol can protect test content while disclosing:
Why the holdout exists and what principles govern it.
How tasks are created, sampled, scored, and validated.
Who administers, reviews, approves, and hears appeals.
What evidence supports the result.
The exact tasks, solutions, and attack methods.
Content transparency may be delayed or restricted.
For every restricted category, record:
When detailed disclosure is unsafe, publish:
A protocol fails transparency when outsiders cannot distinguish:
Participants should receive consistent:
Large laboratories may have:
The protocol should report developer assistance and resource differences.
Open communities may lack one legal entity or centralized developer.
Possible approaches:
Open-source representation should not be symbolic.
Experts should have meaningful participation in:
Security requirements can unintentionally create a closed market.
Options:
International participation may be limited by:
These constraints should be explicit.
Task design should distinguish the intended construct from irrelevant accessibility barriers.
Different systems may require different integration.
The reasons should be documented.
Developers may need to provide:
Dependence can arise through:
Cover:
Developers should be able to present evidence concerning:
Evaluator support should solve integration problems without teaching active test content.
OpenAI has described structured external testing and subject-matter-expert probing as complements to internal Preparedness Framework evaluations.[^openai-external]
Google DeepMind's early-warning proposal similarly envisioned structured access for external safety researchers and auditors.[^deepmind-warning]
Anthropic's 2026 Responsible Scaling Policy revisions formalized external review under specified governance conditions.[^anthropic-rsp]
These examples reflect a broader movement toward combining developer evaluation with external scrutiny. They do not yet establish a common external-access standard.
The institution controlling the holdout should not possess unreviewable authority over:
A mature system may include:
For high-consequence evaluations:
Disclose:
Material conflicts may require recusal.
Allow:
Appeals should be:
Funding sources should be disclosed.
No payer should obtain:
Oversight can include:
Review governance after:
Indicators include:
Possible methods:
Determine:
Possible statuses:
Options:
A public notice should state enough to prevent continued misuse of invalid results.
Every material compromise should generate:
FAILURE_DATABASE.mdVERSION_HISTORY.mdDetermine why public evaluation is insufficient.
Output:
Identify:
Output:
Define:
Output:
Define:
Output:
Create:
Output:
Assess:
Output:
Implement:
Output:
Run on:
Output:
Review:
Output:
Authorize a version for defined use.
Conduct controlled evaluation.
Issue public and restricted outputs.
Track:
Add, replace, or recalibrate tasks.
Withdraw active authority and archive appropriately.
Held-out cyber evaluations may protect:
Requirements may include:
AISI's Inspect Cyber work provides standardized infrastructure for agentic cyber evaluations.[^inspect-cyber]
Held-out biological evaluation may involve:
Task design should avoid creating harmful operational guidance unnecessarily.
NIST work on AI-assisted protein design has used safe biological proxies as part of a TEVV approach, illustrating how evaluation design can reduce experimental risk while investigating relevant capability.[^nist-protein]
Held-out agent evaluation may protect:
Security should account for active model attempts to inspect or escape the environment.
Protecting attack methods can preserve test value and reduce misuse.
AISI's principles for evaluating misuse safeguards emphasize threat modeling, representative attacks, and careful interpretation.[^aisi-safeguards]
Tasks may involve human subjects, private conversations, or experimental conditions.
Additional requirements:
Evaluation may require simulated or digital-twin environments rather than live systems.
Held-out tasks should distinguish:
Evaluator sends tasks to a controlled model endpoint.
Benefits:
Risks:
Model is deployed in evaluator infrastructure.
Benefits:
Risks:
Developer and evaluator operate within a controlled facility.
Benefits:
Risks:
Protected computation can reduce the need for either party to expose all assets.
Attestation can help verify the environment.[^ccc-attestation]
Limitations must be assessed case by case.
Commit to:
before administration or disclosure.
Potentially allows computation across parties without revealing all inputs.
Current practicality depends on workload and implementation.
Potentially enables computation on encrypted data.
It may be too costly or limited for many frontier model evaluations.
Tasks remain with custodians while systems or outputs move through controlled interfaces.
Authorized personnel work in a monitored environment without unrestricted export.
Useful for highly sensitive content, but operationally expensive and not immune to insider risk.
Standards Body should specify desired security properties rather than prematurely mandating one technology.
Evaluation materials may cross:
Institutions should align on:
One institution may recognize another's result when there is confidence in:
Different national institutes may hold different task subsets.
Advantages:
Risks:
Retired tasks can support international research without exposing active holdouts.
Interoperability should not require one global task bank controlled by one institution.
International AI safety and security institutes have increasingly collaborated on evaluation practices and research. Such cooperation can support shared methods while preserving national authority.
Characteristics:
Use:
Characteristics:
Use:
Characteristics:
Use:
Characteristics:
Use:
Characteristics:
Use:
Start with a specific question.
Example:
"Does this system demonstrate a defined level of long-horizon cyber capability under controlled tool access?"
Explain why public tests are insufficient.
Define subcapabilities and exclusions.
Assign roles, conflicts, access, and appeals.
Identify leakage and active-agent risks.
Use multiple task sources.
Test tasks with experts, reference models, and human baselines.
Implement access, logging, packaging, and incident response.
Run on non-consequential reference systems first.
Review science and security separately.
Publish methodology and access rules.
Track every use.
Replace tasks based on exposure and evidence.
Evaluate whether the holdout remains worth its cost and authority.
Held-Out Evaluation Protocol for Long-Horizon Technical Agent Tasks
This would complement the pilot proposed in Foundation 1.
Measure whether AI agents can complete unfamiliar, verifiable technical tasks without prior access to the active task set.
Common evaluator-defined scaffold.
Developer proposes configuration without task access.
Evaluator attempts stronger elicitation.
Failure: The test is confidential but poorly designed.
Safeguard: Independent validity review and public construct explanation.
Failure: One person controls the full bank and scoring.
Safeguard: Separation of duties and access logging.
Failure: The same private set is reused until it becomes stale or informally known.
Safeguard: Exposure limits, rotation, expiration.
Failure: Maintainers dismiss criticism by citing confidentiality.
Safeguard: Controlled independent audit and public methodological disclosure.
Failure: Access dependence gives the developer control over findings.
Safeguard: Contractual independence, diversified access, no result veto.
Failure: A private evaluator benefits from making its test indispensable.
Safeguard: Multiple evaluators, protocol registry, interoperability, review.
Failure: Participants receive materially different tests.
Safeguard: Calibration, bridging, reference systems, uncertainty.
Failure: Active content enters public or training systems.
Safeguard: Compromise plan, rotation, investigation, result withdrawal.
Failure: Protected task generator becomes a training target.
Safeguard: Generator versioning, defense in depth, alternative generators.
Failure: Decision thresholds move after results are observed.
Safeguard: Pre-commitment, governance, change records.
Failure: Passing a private test is described as proof of safety.
Safeguard: Claims boundary and complementary evidence.
Failure: The evaluator misses capability because of weak configuration.
Safeguard: Multiple elicitation tracks and developer input without task disclosure.
Failure: The model detects evaluation and changes behavior.
Safeguard: varied contexts, environment analysis, monitoring, operational evidence.
Failure: Authorized person leaks or modifies content.
Safeguard: least privilege, logging, dual control, monitoring, culture.
Failure: Security and task renewal become unsustainable.
Safeguard: proportionality, shared infrastructure, prioritization.
Failure: Only large corporations can access evaluation.
Safeguard: supervised access, subsidies, shared facilities, representative governance.
Failure: The evaluation itself creates misuse or safety risk.
Safeguard: domain review, safe proxies, sandboxing, restricted disclosure.
Failure: Old tasks are released in ways that expose active methods or sensitive content.
Safeguard: retirement review and staged disclosure.
Open access supports replication and criticism.
This objection is strong.
Response:
Residual concern:
Some scientific scrutiny will remain weaker while active tasks are private.
They can become arbitrary instruments of power.
Response:
Residual concern:
Information asymmetry cannot be eliminated.
Often true.
Response:
Design for rotation, compromise detection, and recovery.
A holdout should be renewable infrastructure, not a permanent vault.
Response:
Use public shadow tasks, retired examples, synthetic tasks, and supervised troubleshooting.
Residual concern:
Some active-task exposure may occasionally be necessary. It should be recorded and those items reconsidered.
Response:
Use formal access policy, exposure records, evaluator rotation, and independent governance.
Residual concern:
People close to the evaluation ecosystem may still learn its style.
Correct.
Response:
Use provenance, post-cutoff sourcing, expert creation, procedural generation, and contamination analysis.
Residual concern:
Proving complete non-exposure is difficult.
Response:
Require complementary operational and contextual evidence.
Response:
Use them selectively where public exposure materially weakens high-value decisions.
Often true for complex workloads.
Response:
Specify security properties, pilot technologies, and avoid dependence on one mechanism.
Yes.
A holdout does not prevent legitimate domain improvement.
Its purpose is to test generalization beyond exact known content, not to stop learning.
Response:
Public reporting minimums, independent verification, and result status.
Response:
Report the difference and create standardized community participation paths.
How much do different forms of contamination inflate different benchmark results?
Which detection methods work without access to training data?
When do newly created analogues measure the same construct as older public tests?
How many administrations can a task tolerate before its value declines?
Can systems reliably detect controlled evaluation contexts and alter behavior?
Which institutional models produce credible replication without broad disclosure?
When do trusted execution environments and related methods provide practical assurance for frontier evaluation?
How can smaller evaluators and open communities participate without weakening security?
What procedural protections are required when held-out results influence deployment or legal status?
Do held-out results predict deployment behavior better than public benchmarks?
How do funding and access relationships affect conclusions?
What evidence should support mutual recognition of confidential results?
Compare:
Study:
Develop methods for testing whether a new private set is genuinely comparable to an older public benchmark.
Model task value as a function of:
Pilot:
Compare:
Develop methods to detect:
Study access models for:
Pilot:
Measure whether held-out evidence improves actual deployment and policy decisions.
Create matched public and private task forms.
Compare:
Administer a task bank repeatedly and measure whether performance changes after controlled exposure.
Construct a private analogue of a known benchmark and test construct equivalence.
Compare public and protected procedural generators.
Have two independent evaluators administer the same protected protocol.
Compare standard, developer-elicited, and evaluator-optimized performance without exposing active tasks.
Pilot a trusted execution environment for a limited evaluation workload.
Simulate a task-bank leak and test response time, result-status changes, and replacement.
Create a supervised evaluation path for an open-weight model community.
Release retired tasks and assess whether the disclosure improves scientific scrutiny without harming active evaluation.
A future held-out evaluation standard could require:
Explain why restriction is necessary.
Identify exactly what is held out.
Define assets, threats, controls, and residual risk.
Specify ownership, roles, conflicts, review, and appeals.
Maintain task and solution history.
Define tiers, authorization, review, and revocation.
Standardize configuration, logging, deviations, and scoring.
Demonstrate correctness, reliability, coverage, and fairness.
Provide controlled replication or equivalent assurance.
Publish methodology, results, limitations, and restriction rationale.
Define detection, investigation, status, remediation, and notice.
Define exposure limits, renewal, expiration, and archival treatment.
Such a standard should be developed through the future STANDARDS_DEVELOPMENT_PROCESS.md.
A holdout must evolve as tasks leak, saturate, and lose relevance.
The higher the potential consequence, the stronger the case for protected, independently administered evidence.
Restricted content requires qualified external review to maintain accountability.
Held-out evaluation depends on trustworthy organizations capable of secure administration.
Held-out protocols may evolve from research practice into procurement, certification, or regulatory evidence.
Participation should reward genuine scrutiny, not private-test mystique.
Institutions need compatible metadata, access, security, and recognition systems.
Standards Body adopts the following working positions.
Public benchmarks remain essential to open science.
Some frontier evaluations require held-out components because public exposure can weaken validity.
Held-out evaluation is a spectrum, not a binary category.
Every restriction should have a defined purpose, scope, owner, duration, and review path.
Confidentiality does not establish scientific validity.
Transparent governance can coexist with protected content.
The construct and broad methodology should normally be disclosed.
Exact tasks, solutions, attack libraries, or environment details may remain protected when disclosure would materially reduce validity or create harm.
Held-out results should not be the sole basis for broad safety claims.
High-consequence held-out evaluations require independent review.
Task provenance and chain of custody are core evaluation evidence.
Access should follow least privilege and separation of duties.
Developers should receive fair notice without active-task disclosure.
Developers should be able to challenge material errors but should not control publication.
Open-source and smaller actors require meaningful participation pathways.
A private evaluator is not automatically independent.
Result uncertainty should be reported even when the task set is confidential.
Held-out tasks should rotate, expire, or retire.
Compromise should change result status transparently.
Security controls should be proportionate and auditable.
Cryptographic or confidential-computing methods can support assurance but cannot replace governance.
Retired-task release should be considered when safe and useful.
International interoperability should favor shared requirements over one global task bank.
Passing a held-out evaluation is not proof of safety.
The holdout system itself should be regularly evaluated.
A held-out component is justified when:
A held-out component is not justified merely because:
A task should be rotated when:
A result should be suspended when:
A protocol should be retired when:
Applicant:
Organization:
Role:
Requested protocol:
Requested access tier:
Purpose:
Duration:
Describe relevant technical and domain expertise.
Describe identity, device, storage, execution, logging, and incident controls.
Disclose relevant financial, professional, competitive, and intellectual interests.
Explain how protected materials and outputs will be handled.
Describe intended outputs and review process.
Identify all third parties.
Disclose relevant security or integrity incidents.
Incident identifier:
Date discovered:
Reporter:
Protocol:
Affected version:
| Dimension | Core Question |
|---|---|
| Purpose | Is the measurement reason for holding content out explicit? |
| Construct | Is the evaluated capability clearly defined? |
| Protected surface | Is it clear what is restricted and why? |
| Provenance | Can each task and solution be traced? |
| Coverage | Does the bank represent the intended task universe? |
| Validity | Does evidence support the intended interpretation? |
| Reliability | Are results sufficiently consistent? |
| Integrity | Are contamination and manipulation risks addressed? |
| Security | Are controls proportionate to the threat model? |
| Access | Is authorization role-based and reviewable? |
| Chain of custody | Is every material access and change recorded? |
| Administration | Can qualified evaluators run the protocol consistently? |
| Scoring | Are rules, judges, and uncertainty defensible? |
| Transparency | Is enough public for accountability? |
| Reproducibility | Can independent parties verify the process? |
| Fairness | Are participants treated consistently? |
| Independence | Is evaluator judgment protected from interested parties? |
| Appeals | Can material error be challenged? |
| Compromise response | Can the system detect, suspend, recover, and disclose? |
| Renewal | Are tasks rotated before authority decays? |
| Retirement | Can obsolete or compromised evidence be withdrawn? |
| Accessibility | Can qualified smaller and open actors participate? |
| Interoperability | Can other institutions understand and recognize the result? |
| Decision utility | Does the holdout improve a real decision? |
A benchmark loses independence when the people building the system know exactly how success will be measured.
That does not make public evaluation useless.
It means public evaluation answers a different question.
A public benchmark can show how well a system performs on a shared, inspectable target.
A held-out evaluation can provide additional evidence about how the system behaves when the exact target has not become part of development.
Both matter.
The future evaluation ecosystem should preserve the openness that allows science to progress while protecting enough unseen evidence to test genuine generalization, resilience, and threshold capability.
That balance is difficult.
Too little protection produces contaminated or gameable measurement.
Too much secrecy produces unaccountable authority.
The answer is not a permanent secret benchmark.
The answer is a governed holdout system.
Such a system should know:
The second foundation of Standards Body is therefore not secrecy.
It is protected evidence under accountable control.
[^nist-tevv]: National Institute of Standards and Technology, AI Test, Evaluation, Validation and Verification. https://www.nist.gov/ai-test-evaluation-validation-and-verification-tevv
[^nist-rmf]: National Institute of Standards and Technology, AI Risk Management Framework. https://www.nist.gov/itl/ai-risk-management-framework
[^nist-statistical]: National Institute of Standards and Technology, Expanding the AI Evaluation Toolbox with Statistical Models, 2026. https://www.nist.gov/news-events/news/2026/02/new-report-expanding-ai-evaluation-toolbox-statistical-models
[^nist-cheating]: National Institute of Standards and Technology, Practices for Detecting and Preventing Evaluation Cheating, 2025. https://www.nist.gov/caisi/cheating-ai-agent-evaluations/4-practices-detecting-and-preventing-evaluation-cheating
[^nist-protein]: National Institute of Standards and Technology, Experimental Evaluation of AI-Driven Protein Design Risks Using Safe Biological Proxies, 2025. https://www.nist.gov/publications/experimental-evaluation-ai-driven-protein-design-risks-using-safe-biological-proxies
[^aisi-lessons]: UK AI Security Institute, Early Lessons from Evaluating Frontier AI Systems, 2024. https://www.aisi.gov.uk/blog/early-lessons-from-evaluating-frontier-ai-systems
[^aisi-qa]: UK AI Security Institute, Early Insights from Developing Question-Answer Evaluations for Frontier AI, 2024. https://www.aisi.gov.uk/blog/early-insights-from-developing-question-answer-evaluations-for-frontier-ai
[^aisi-sandbox]: UK AI Security Institute, The Inspect Sandboxing Toolkit: Scalable and Secure AI Agent Evaluations, 2025. https://www.aisi.gov.uk/blog/the-inspect-sandboxing-toolkit-scalable-and-secure-ai-agent-evaluations
[^aisi-sandbox-learning]: UK AI Security Institute, What Can Sandboxed AI Agents Learn About Their Evaluation Environments?, 2026. https://www.aisi.gov.uk/blog/what-can-sandboxed-ai-agents-learn-about-their-evaluation-environments
[^aisi-breakout]: UK AI Security Institute, Can AI Agents Escape Their Sandboxes?, 2026. https://www.aisi.gov.uk/blog/can-ai-agents-escape-their-sandboxes-a-benchmark-for-safely-measuring-container-breakout-capabilities
[^aisi-safeguards]: UK AI Security Institute, Principles for Safeguard Evaluation, 2025. https://www.aisi.gov.uk/blog/principles-for-safeguard-evaluation
[^inspect]: UK AI Security Institute, Inspect AI, evaluation framework. https://inspect.aisi.org.uk/
[^inspect-cyber]: UK AI Security Institute, A New Standard for Agentic Cyber Evaluations, 2025. https://www.aisi.gov.uk/blog/inspect-cyber
[^openai-pf]: OpenAI, Preparedness Framework v2, 2025. https://cdn.openai.com/pdf/18a02b5d-6b67-4cec-ab64-68cdfbddebcd/preparedness-framework-v2.pdf
[^openai-external]: OpenAI, Strengthening Our Safety Ecosystem with External Testing, 2025. https://openai.com/index/strengthening-safety-with-external-testing/
[^deepmind-fsf]: Google DeepMind, Frontier Safety Framework, updated 2025. https://deepmind.google/blog/updating-the-frontier-safety-framework/
[^deepmind-warning]: Google DeepMind, An Early Warning System for Novel AI Risks, 2023. https://deepmind.google/blog/an-early-warning-system-for-novel-ai-risks/
[^anthropic-rsp]: Anthropic, Responsible Scaling Policy, Version 3.2, 2026. https://www.anthropic.com/responsible-scaling-policy
[^extreme-risk]: Toby Shevlane et al., Model Evaluation for Extreme Risks, 2023. https://arxiv.org/abs/2305.15324
[^dangerous-capabilities]: Mary Phuong et al., Evaluating Frontier Models for Dangerous Capabilities, 2024. https://arxiv.org/abs/2403.13793
[^external-access]: Jacob Charnock et al., Expanding External Access to Frontier AI Models for Dangerous Capability Evaluations, 2026. https://arxiv.org/abs/2601.11916
[^contamination-survey]: Cheng Xu et al., Benchmark Data Contamination of Large Language Models: A Survey, 2024. https://arxiv.org/abs/2406.04244
[^contamination-trust]: Yihong Dong et al., Data Contamination and Trustworthy Evaluation for Large Language Models, 2024. https://arxiv.org/abs/2402.15938
[^retro-holdout]: Jack Haimes et al., Benchmark Inflation: Revealing LLM Performance Gaps Using Retro-Holdouts, 2024. https://arxiv.org/abs/2410.09247
[^livebench]: Colin White et al., LiveBench: A Challenging, Contamination-Limited LLM Benchmark, 2024. https://arxiv.org/abs/2406.19314
[^c2leva]: Yanyang Li et al., C2LEVA: Toward Comprehensive and Contamination-Free Language Model Evaluation, 2024. https://arxiv.org/abs/2412.04947
[^clean-eval]: Wenhong Zhu et al., CLEAN-EVAL: Clean Evaluation on Contaminated Large Language Models, 2023. https://arxiv.org/abs/2311.09154
[^ccc-technical]: Confidential Computing Consortium, A Technical Analysis of Confidential Computing. https://confidentialcomputing.io/wp-content/uploads/sites/10/2023/03/CCC-A-Technical-Analysis-of-Confidential-Computing-v1.3_unlocked.pdf
[^ccc-attestation]: Confidential Computing Consortium, Why Is Attestation Required for Confidential Computing?, 2023. https://confidentialcomputing.io/2023/04/06/why-is-attestation-required-for-confidential-computing/
Date: July 16, 2026
Change type: Complete replacement
Summary: Replaces the earlier outline edition with a fully developed canonical working white paper. Adds first-principles rationale, definitions, design taxonomy, contamination analysis, holdout threat modeling, task provenance, statistical design, security architecture, access tiers, chain of custody, administration, reporting, controlled reproducibility, transparency, fairness, developer-evaluator relationships, governance, compromise response, lifecycle, domain-specific requirements, protected-compute options, international interoperability, maturity model, implementation pathway, a Standards Body pilot, metrics, failure analysis, objections, evidence gaps, research agenda, standards implications, operational templates, scorecard, and primary-source research basis.
Status: Ready for internal review and future expert critique.