Standards Body · Foundation paper, public edition · Released July 17, 2026
Canonical record: https://standardsbody.ai/library/foundation-paper/third-party-auditor-ecosystem/
Standards Body is an independent research and institutional-design project. It is not currently a regulator, accreditation body, certification body, or governmental authority. This document is research; it is not an adopted standard.
Series: Foundations for Frontier AI Evaluation Infrastructure
Version: 1.0
Status: Canonical working white paper
Project: Standards Body
Primary domain: standardsbody.ai
Core line: Foundations for Frontier AI
Research basis reviewed through: July 16, 2026
Document owner: Standards Body
Review cycle: Annual review, with event-triggered revision after material changes in accreditation practice, frontier AI auditing, evaluator markets, international recognition, or relevant standards
This paper defines the Standards Body position on the development of a trustworthy third-party evaluator and auditor ecosystem for frontier artificial intelligence.
It is intended to serve as:
This paper is not itself an accreditation standard.
It does not authorize Standards Body to accredit, certify, inspect, or regulate any organization.
It does not claim that the existing conformity-assessment system can be copied directly into frontier AI.
It identifies the institutional architecture that could allow third-party AI evaluation to become more competent, consistent, legible, competitive, secure, and internationally trusted.
Independent expert review can improve frontier AI evaluation.
It does not automatically scale.
A small number of respected researchers may be able to evaluate one model, review one protocol, or advise one developer. That approach becomes insufficient when many organizations, governments, insurers, purchasers, standards bodies, and members of the public need evidence about:
Scaling this work requires more than additional experts.
It requires an ecosystem.
A trustworthy third-party auditor ecosystem should make it possible to answer:
These questions are not solved by branding an organization an "AI auditor."
The term can refer to radically different activities.
One organization may run capability benchmarks.
Another may inspect governance records.
Another may test cybersecurity controls.
Another may review compliance with a voluntary frontier safety framework.
Another may issue a certification mark.
Another may accredit the organization issuing the certificate.
If these roles are not distinguished, the public may incorrectly assume that:
A serious ecosystem should therefore separate at least six functions:
Standards and scheme development
Defines requirements, methods, decision rules, and governance.
Testing and evaluation
Produces technical evidence about a model or system.
Inspection and audit
Examines systems, records, processes, controls, and claims against defined criteria.
Validation and verification
Assesses whether claims, methods, statements, or evidence meet specified requirements.
Certification or attestation
Issues a formal statement that defined requirements have been fulfilled within a stated scope.
Accreditation
Provides independent recognition that a conformity-assessment body is competent and impartial to perform specified activities.
These functions can interact.
They should not collapse into one institution by default.
The ecosystem also needs multiple evaluator types.
No single organization is likely to possess world-class competence across:
Specialization should therefore be expected.
A mature system may include:
Diversity can increase resilience and innovation.
It can also create fragmentation, inconsistent quality, duplicated burden, evaluator shopping, and public confusion.
Standards Body adopts the following core position:
Frontier AI evaluation should develop into a plural, accountable, and internationally interoperable third-party assurance ecosystem in which evaluator competence, impartiality, scope, methods, security, and performance are independently assessed and continuously monitored.
The ecosystem should be designed around several principles:
Existing conformity-assessment infrastructure provides important lessons.
ISO and IEC standards distinguish requirements for:
The International Accreditation Forum and International Laboratory Accreditation Cooperation have built multilateral recognition arrangements intended to support international acceptance of accredited results.[^iaf-mla][^ilac-mra]
These systems demonstrate how competence, impartiality, consistent operation, peer evaluation, scopes of accreditation, surveillance, and recognition can support trust.
Frontier AI differs in important ways:
The goal is therefore not to copy an existing laboratory or certification regime mechanically.
The goal is to adapt its strongest institutional principles to a field whose science and objects are changing.
A credible third-party ecosystem should eventually allow a decision-maker to distinguish:
The fifth foundation of Standards Body is the infrastructure that makes those distinctions real.
Independent evaluation will not become reliable at scale unless the institutions conducting it are themselves evaluated, monitored, and held accountable.
No single evaluator should be expected to provide every form of frontier AI assurance. A plural ecosystem with clear roles and interoperable requirements is more resilient than a permanent monopoly.
Recognition should be scope-specific and based on demonstrated competence, impartiality, consistent operation, security, and performance, not reputation alone.
Competition can improve evaluator innovation and capacity, but unmanaged commercial incentives can produce client capture, evaluator shopping, weak methods, and assurance theater.
Frontier AI assurance should move toward cross-border recognition of competent evaluators and credible results without requiring one global evaluator or one universal protocol.
Evaluator competence and accreditation should expire, narrow, or change when methods, systems, personnel, or evidence change materially.
The evaluator ecosystem should be treated as part of critical public-interest infrastructure, even when many participants are private organizations.
This paper covers the ecosystem of organizations and processes that may provide:
This paper does not establish:
FOUNDATION_04_INDEPENDENT_EXPERT_REVIEW.md defines the principles of meaningful independent review.
Foundation 5 asks how those principles can be institutionalized across many organizations and repeated engagements.
This paper provides the philosophical and ecosystem foundation.
EVALUATOR_ACCREDITATION_FRAMEWORK.md should later define detailed operational requirements.
An evaluator can only assess conformity when the criteria are sufficiently defined.
Where no accepted standard exists, the activity may be:
It should not be marketed as certification against a standard that does not exist.
Assessment performed by the organization responsible for the system or claim.
Examples:
Assessment performed by a party with a user, purchaser, contractual, or direct stakeholder interest.
Examples:
Assessment performed by a body sufficiently independent of the provider and immediate user interests to support an impartial judgment.
Determination of one or more characteristics of an object according to a specified procedure.
In AI, testing may include:
Structured production and interpretation of evidence about a model, system, process, or claim.
Examination of a product, process, service, installation, or design and determination of conformity with specified or professional requirements.
ISO/IEC 17020 specifies competence, impartiality, and consistent-operation requirements for inspection bodies.[^iso-17020]
A systematic, independent, and documented process for obtaining and evaluating evidence against defined criteria.
In frontier AI, audit can cover:
Confirmation that specified requirements are adequate for an intended future use or result.
Confirmation that specified requirements have been fulfilled based on objective evidence.
Third-party attestation related to products, processes, services, persons, or management systems.
Certification is stronger and more formal than general review language.
Issue of a statement based on a decision following review that fulfillment of specified requirements has been demonstrated.
Demonstration that specified requirements relating to a product, process, system, person, or body are fulfilled.
An organization performing conformity-assessment activities.
Commonly abbreviated as CAB.
Independent recognition that a conformity-assessment body is competent and impartial to perform specified activities.
ISO/IEC 17011 specifies requirements for accreditation bodies assessing and accrediting conformity-assessment bodies.[^iso-17011]
An authoritative body that performs accreditation.
A system of rules, procedures, criteria, governance, and responsibilities for a defined conformity-assessment activity.
The organization responsible for developing and maintaining a conformity-assessment scheme.
The specific activities, methods, domains, systems, and limits for which an evaluator has been recognized as competent.
Ongoing monitoring of an accredited or recognized body after initial assessment.
Periodic comprehensive review to determine whether recognition should continue.
Evaluation of participant performance against pre-established criteria through comparison exercises.
Organization, performance, and evaluation of measurements or tests on the same or similar objects by two or more laboratories.
Material sufficiently homogeneous and stable with respect to specified properties for use in measurement, validation, quality control, or proficiency testing.
In AI, analogous reference assets may include:
Presence of objectivity and management of conflicts so that assessment judgments are not improperly influenced.
Selecting among evaluators based on the likelihood of receiving a favorable result rather than competence or fit.
An evaluator making claims beyond its recognized competence or approved activity.
A defined degree of review depth, access, rigor, continuity, and confidence.
Acceptance by one body or jurisdiction of results issued under another recognized system.
Assessment of an accreditation or recognition body by peers to determine whether it meets shared requirements.
Ongoing or repeated evidence collection, assessment, and review rather than a one-time engagement.
The number of:
will exceed the capacity of a small expert community.
Cyber evaluation and biological evaluation require different expertise, infrastructure, security, and methods.
Reliance on a few evaluator organizations can create:
Multiple qualified evaluators can challenge and reproduce results.
National institutes cannot perform every assessment worldwide.
Purchasers, governments, insurers, and developers need to know what evaluator claims mean.
Different evaluators can develop:
A plural ecosystem can continue operating if one evaluator:
Trust becomes stronger when competence and impartiality are verified through shared institutions.
An ecosystem can also create:
The ecosystem must be designed rather than assumed.
A mature ecosystem should distinguish its major functions.
Defines requirements or methods.
Risks:
Defines how conformity assessment is performed against requirements.
Responsibilities may include:
Produces technical evidence.
Relevant analogies come from ISO/IEC 17025, which sets requirements for competence, impartiality, and consistent operation of testing and calibration laboratories.[^iso-17025]
Examines systems, processes, or deployments against specified or professional criteria.
Assesses records, processes, controls, and compliance.
Issues formal attestations under a certification scheme.
ISO/IEC 17065 covers bodies certifying products, processes, and services.[^iso-17065]
ISO/IEC 17021-1 covers bodies auditing and certifying management systems.[^iso-17021]
Evaluates claims or statements according to defined criteria.
Assesses evaluator competence and impartiality.
Runs exercises that compare evaluator performance.
Maintains:
May:
Provides system access, technical information, internal evidence, and remediation.
Defines assurance needs and relies on results.
May use assurance evidence for underwriting, controls, or exclusions.
May define legal requirements, recognize schemes, inspect bodies, or use results.
Can contribute:
Maintains verified information about:
Reviews disputed decisions independently.
An organization that:
has concentrated incompatible powers.
A certification body should not unilaterally write requirements to fit its clients.
Consulting can create self-review.
Potential controls:
An organization should not accredit itself.
A certification body may use test evidence from laboratories.
It should preserve independent certification decision authority.
A scheme owner may operate evaluation activities, but risks should be disclosed and governed.
Government may act as:
Role conflicts should be explicit.
Early ecosystems may require combined roles.
Combined roles should be treated as temporary or controlled rather than invisible.
Every recognized body should disclose which functions it performs and how conflicts are controlled.
Focus:
Focus:
Focus:
Focus:
Research on third-party compliance review has proposed models ranging from minimalist to comprehensive assessment of company adherence to frontier safety frameworks.[^compliance-reviews]
Focus:
Focus:
Focus:
Focus:
Focus:
Focus:
Focus:
Combines several roles under controlled governance.
Evaluator recognition should be based on demonstrated competence.
Includes:
Includes:
The evaluator should demonstrate competence in the exact methods used.
Examples:
An evaluator competent in language-model benchmarking is not automatically competent in:
Competence may differ across:
Sensitive work may require:
Evaluators should understand the decision their evidence supports.
Can include:
Require:
Scope should narrow when:
"Accredited AI evaluator" is too broad to be meaningful.
A scope may specify:
Evaluation of autonomous cyber capabilities in tool-using language-model systems using controlled challenge environments under Protocol DEP-CYBER versions 1.x.
This does not authorize:
A flexible scope can allow validated method updates within controlled boundaries.
Risk:
Control:
Require:
May apply to:
Registries should display current and historical scopes clearly.
A third-party evaluator should operate a documented quality system.
Define:
Control:
Retain:
Maintain:
Require:
Control:
Manage:
Define response when:
Address root cause rather than only the observed error.
Evaluator should audit its own quality system.
Internal audit is not a substitute for accreditation surveillance.
Leadership should review:
Use evidence to revise methods and systems.
Use published methods where fit for purpose.
Document modifications and their effects.
Require full validation.
Validate:
Assess:
Revalidate after material changes.
Maintain public or controlled information about:
Test whether methods work across evaluators and infrastructure.
Proficiency testing asks whether different evaluators can produce credible results on common or comparable cases.
Without comparison:
All evaluators assess the same frozen or reproducible system.
All evaluators use a protected task package.
Evaluators review the same evidence and produce findings.
Evaluators audit a controlled fictional company.
Different evaluators receive equivalent forms.
One evaluator repeats another's work.
Compare:
Proficiency exercises must not leak active tasks or sensitive methods.
Possible response:
Aggregate proficiency results can improve market trust.
Individual results may need controlled disclosure.
The provider should be competent and impartial.
Reference models and tasks can become obsolete.
Use dynamic and versioned exercises.
Every evaluator should identify risks arising from:
A body dependent on one developer may be formally separate but practically constrained.
Should be prohibited.
An evaluator should not certify a system it designed or remediated without robust separation and scheme permission.
Can reduce familiarity and capture.
The person authorizing a certification or final attestation should not simply be the person who performed the evaluation.
A body may use an independent committee to review:
Disclose material:
Disclosure does not resolve every conflict.
Use the multidimensional independence profile established in Foundation 4.
Developer or purchaser pays per engagement.
Strengths:
Risks:
Provides ongoing assurance.
Strengths:
Risks:
Strengths:
Risks:
Strengths:
Risks:
Strengths:
Risks:
Can support infrastructure.
Risk:
Insurer pays or requires evaluation.
Risk:
Diversification can reduce dependence.
Evaluators may seek:
Revenue should support rigorous work without making favorable outcomes commercially necessary.
Purchasers should consider:
Evaluation quality can be undermined by procurement focused only on price.
Standing contracts can speed access.
Risk:
Purchasers may rotate qualified evaluators.
High-consequence work can use:
Developers should not have unlimited ability to choose the most favorable evaluator for mandatory or public-interest claims.
A scheme can assign evaluators based on:
Disclose selection logic for material public claims.
Evaluator performs specified procedures and reports factual findings.
No broad assurance conclusion.
Provides moderate confidence based on narrower evidence.
Provides higher confidence based on deeper evidence.
It is not absolute assurance.
Measures defined capability under specified conditions.
Tests controls against a threat model.
Assesses adherence to a framework or policy.
Examines technical and organizational controls.
Determines conformity with a defined certification scheme.
Uses ongoing access, monitoring, repeated tests, and incident evidence.
Recent frontier AI auditing research proposes staged assurance levels ranging from time-bounded audits to continuous, deception-resilient verification.[^frontier-auditing]
Standards Body should study such frameworks without adopting any one level system automatically.
Evaluator quality depends on:
Developers may control which evaluators can participate.
This can affect market independence.
A shared access agreement can define:
Possible tiers:
Qualified evaluators should not need to rebuild every integration from zero.
Shared facilities can support smaller bodies.
If a developer denies adequate access, reports should state the resulting limitation.
An evaluator should not lose future access solely because of an evidence-based unfavorable result.
OpenAI's 2026 playbook emphasizes that trustworthy third-party evaluations should report the system tested, tool access, harness, elicitation methods, available resources, and validity checks.[^openai-playbook]
Evaluators may handle:
Require:
An evaluator may create:
Evaluator recognition should specify the sensitivity level it can handle.
A serious breach may trigger:
Security controls should preserve auditability.
Developers should also protect evaluator tasks and methods.
Centralized secure facilities create efficiency and concentration risk.
Accreditation provides independent recognition of evaluator competence and impartiality within scope.
It does not endorse every result.
A credible accreditation body should demonstrate:
ISO/IEC 17011 provides a general model for accreditation-body competence and impartiality.[^iso-17011]
May include:
Decision should be independent from the assessment team.
Registry should show:
Use:
Comprehensive reassessment after a defined period.
Trigger after:
Temporary restriction pending correction.
Remove accreditation after serious or unresolved failure.
Evaluator can challenge accreditation decisions.
Multiple accreditation bodies may exist.
They themselves require peer recognition and oversight.
AI methods and systems change quickly.
Accreditation assessors observe an evaluator performing work.
Can review:
May be justified for high-risk scopes.
Track:
Surveillance frequency should increase when:
Continuous data can support more responsive oversight.
It should not become opaque algorithmic governance.
Personnel should be able to report serious issues safely.
Processes should resist harassment and strategic delay.
Material sanctions should be reflected in registries.
Define jurisdiction and cooperation.
Evaluator error can contribute to:
Assurance reports should not imply certainty.
Evaluator organizations may need:
Excessive liability can eliminate small evaluators.
Insufficient liability can weaken accountability.
A future ecosystem should define reasonable professional practice.
Third-party review should not transfer all responsibility away from the developer.
Accreditation confirms competence within scope, not correctness of every engagement.
Misleading use of evaluator reports should create consequences.
Enable users to verify:
Can reduce fraudulent certification claims.
Where appropriate, link to:
Marks can communicate assurance.
They can also oversimplify.
Specify:
A broad safety mark is likely to overclaim unless tightly scoped.
Preferred claims should identify:
Registry operator should be neutral and secure.
A small number of evaluators may control:
Support through:
Open evaluation tools can reduce entry costs.
Open tooling does not eliminate the need for quality or security.
Market structure should be monitored where assurance becomes required.
Clients should be able to change evaluators without losing all institutional evidence.
Critical decisions may benefit from more than one evaluator.
Occurs when organizations seek favorable conclusions.
A client may terminate after adverse preliminary findings and seek another evaluator.
Require disclosure of:
Differences can arise from:
Use:
Different conclusions can be legitimate.
The ecosystem should explain why they differ.
Without recognition, organizations may face repeated assessment in every market.
The IAF Multilateral Recognition Arrangement aims to support mutual recognition and acceptance of accredited certification and validation or verification statements among signatories.[^iaf-mla]
The ILAC Mutual Recognition Arrangement supports acceptance of accredited testing, calibration, medical testing, inspection, proficiency-testing, and reference-material activities.[^ilac-mra]
Accreditation bodies participating in recognition arrangements are evaluated by peers.
A future AI recognition system may cover:
Require:
Jurisdictions may impose additional requirements.
Recognition of technical competence does not force identical legal decisions.
IAF and ILAC have been moving toward a unified global accreditation cooperation structure, reflecting the long-term value of coordinated recognition infrastructure.[^ilac-home]
Begin with:
Can develop before legal mandates.
Government may recognize:
Government can require qualified evaluation for purchases.
Regulators may use third-party evidence while retaining authority.
Risks:
A government institute may evaluate and regulate.
Separation of functions may be needed.
Public funding can build:
NIST's TEVV work supports evaluation tasks, testbeds, tools, datasets, guidelines, and standards development as part of a broader AI measurement ecosystem.[^nist-tevv]
Insurers may require:
Purchasers may require evaluation before deployment.
Contracts can define:
Financial actors may rely on assurance reports.
Private requirements can spread before formal regulation.
Shared evaluator standards can reduce duplicated burden.
Identify a defined evaluation or audit scope requiring recognition.
Establish:
Confirm that the accreditation body understands the scope.
Evaluator submits:
Assess system readiness.
Select qualified assessors and witness activities.
Review governance, records, quality, security, and impartiality.
Observe real or controlled evaluator work.
Review comparison performance.
Evaluator corrects identified issues.
Separate decision authority determines scope and status.
Publish recognition.
Monitor continued competence.
Extend, reduce, suspend, or withdraw.
Conduct periodic comprehensive review.
Accreditation body participates in peer evaluation and mutual recognition.
Characteristics:
Characteristics:
Characteristics:
Characteristics:
Characteristics:
Publish common definitions for:
Create a voluntary registry with:
Registry entry is not accreditation.
Develop baseline requirements for:
Begin with bounded scopes such as:
Run inter-evaluator comparisons.
Create transparent rules for selected assurance activities.
Partner with experienced accreditation institutions or build a peer-reviewed pilot process.
Monitor recognized evaluators.
Publish verified scopes and status.
Run bilateral recognition and shared proficiency exercises.
Test reliance on evaluator evidence.
Use complaints, incidents, and performance data to revise the system.
Frontier AI Evaluator Qualification and Proficiency Pilot
Test whether multiple evaluator organizations can competently and consistently administer a defined Standards Body protocol.
Administration and reporting of the Standards Body held-out autonomous cyber capability protocol under controlled conditions.
Invite:
Participants submit:
Each evaluates:
Assess:
Independent assessors observe each evaluator.
Participants correct issues and repeat selected tasks.
The pilot succeeds if it:
Failure: Organizations call themselves AI auditors without demonstrated scope-specific competence.
Safeguard: Public scope, competence evidence, accreditation, proficiency testing.
Failure: Recognition is treated as a marketing badge rather than continuous oversight.
Safeguard: Surveillance, witnessed assessments, sanctions, public status.
Failure: Cyber competence is used to claim biology or governance competence.
Safeguard: Granular scopes and claim enforcement.
Failure: Evaluators depend on favorable results for revenue.
Safeguard: Payment separation, disclosure, pooled funding, rotation.
Failure: Clients seek favorable assessors.
Safeguard: Assignment, registry, prior-engagement disclosure, scheme rules.
Failure: Evaluator certifies its own design or remediation.
Safeguard: Separation, prohibition, independent decision.
Failure: Incumbents write requirements that entrench their practices.
Safeguard: balanced governance, public consultation, small-actor participation.
Failure: One accreditation body controls entry and methods.
Safeguard: peer recognition, multiple bodies, appeals, public oversight.
Failure: Results cannot be compared.
Safeguard: core metadata, proficiency testing, method registries.
Failure: Advisory review is marketed as proof of safety.
Safeguard: terminology rules, registry, mark enforcement.
Failure: Recognition continues after methods and personnel change.
Safeguard: event-triggered surveillance, expiration, scope reduction.
Failure: Evaluator leaks models, tasks, incidents, or harmful methods.
Safeguard: security-scoped competence and sanctions.
Failure: Accreditation excludes smaller innovative evaluators.
Safeguard: modular scopes, grants, shared infrastructure, supervised pathways.
Failure: Price competition reduces depth.
Safeguard: minimum scheme requirements and procurement quality criteria.
Failure: Evaluators create unnecessarily elaborate processes to justify fees.
Safeguard: proportionality and decision-utility metrics.
Failure: Users cannot distinguish testing, audit, certification, and accreditation.
Safeguard: canonical terminology and precise claims.
Failure: Same system is repeatedly evaluated under incompatible regimes.
Safeguard: mutual recognition and crosswalks.
Failure: Government delegates policy responsibility to private evaluators.
Safeguard: retain accountable public decision authority.
Failure: Certification is used to disclaim developer responsibility.
Safeguard: explicit responsibility allocation.
Failure: Unfavorable evaluators lose access.
Safeguard: access agreements, public reporting, multiple access channels.
This objection is partly correct.
Response:
Residual concern:
Premature formalization can freeze weak methods.
It can.
Response:
Residual concern:
Some burden is unavoidable.
Competition can also produce:
Independent oversight remains necessary.
Government offers authority and resources.
It may lack:
A mixed ecosystem is stronger.
Commercial incentives create risk but do not make impartial work impossible.
Existing assurance fields use structural controls, accreditation, surveillance, liability, and recognition.
Residual concern:
Client dependence remains material.
Yes, if scope and claim are broad.
Response:
Residual concern:
Public interpretation may still oversimplify.
Some cannot.
Response:
Full global recognition may be slow.
Bilateral and domain-specific recognition can begin earlier.
This is a central risk.
Response:
Response:
Residual concern:
Some systems may change too frequently for meaningful static certification.
Correct.
Recognition requires strong peer evaluation and scope control.
They may be too slow for some frontier changes.
The ecosystem should retain:
without abandoning rigor.
Which competence indicators predict high-quality frontier evaluations?
How should inter-evaluator comparison work for stochastic, dynamic systems?
How granular should scopes be?
Which ongoing data best predicts evaluator failure?
What degree of concentration is efficient versus dangerous?
Which business models preserve independence and capacity?
What standard of care is appropriate?
Which AI claims are mature enough for certification?
How should ongoing assessment work for frequently updated systems?
Which AI assurance activities can be mutually recognized first?
How can small evaluators securely access frontier systems?
Which controls work without eliminating legitimate choice?
Who can assess frontier AI evaluators competently?
Does accreditation improve outcomes enough to justify cost?
Create a clear map of testing, inspection, audit, certification, validation, verification, and accreditation.
Develop granular scope templates for frontier AI activities.
Define role, team, and organizational competence.
Pilot common model, task, incident, and audit exercises.
Create requirements for dynamic and held-out methods.
Adapt laboratory and conformity-assessment quality systems to AI.
Measure client dependence, consulting conflicts, and capture.
Study competition, concentration, entry, and pricing.
Build shared facilities and portable access agreements.
Develop event-triggered and continuous accreditation.
Design verified, machine-readable scope and status records.
Study professional responsibility and insurance.
Pilot peer evaluation and mutual recognition.
Identify claims mature enough for formal attestation.
Measure whether third-party assurance improves deployment, safeguards, and public trust.
Have multiple institutions classify evaluator scopes and compare consistency.
Run a shared proficiency test across evaluator types.
Provide a synthetic frontier safety framework and evidence package for review.
Observe evaluators performing the same protocol.
Test a standardized financial independence metric.
Compare client choice, rotation, and random assignment.
Pilot access for small evaluators.
Create machine-readable evaluator and scope records.
Track method, personnel, complaints, security, and performance changes.
Have evaluators in two jurisdictions repeat an assessment.
Test whether users understand narrow certification claims.
Simulate a major evaluator security incident and registry response.
A future evaluator accreditation standard could require:
Defined responsibility and authority.
Risk analysis, conflict controls, and independence profile.
Personnel, team, organizational, domain, method, and security competence.
Precise activities, domains, methods, systems, and limits.
Documents, records, internal audit, management review, corrective action.
Selection, validation, versioning, and uncertainty.
Secure compute, software, environments, logging, and external providers.
Contract review, planning, administration, scoring, review, reporting.
Separation of evaluation and final attestation where applicable.
Required metadata, scope, confidence, limitation, and expiry.
Independent and documented process.
Sensitivity-scoped controls and incident response.
Participation and response to poor performance.
Periodic and event-triggered monitoring.
Clear rules and public status.
Peer evaluation and compatible scopes.
These requirements should be developed in EVALUATOR_ACCREDITATION_FRAMEWORK.md through the future STANDARDS_DEVELOPMENT_PROCESS.md.
Evaluators must maintain competence as protocols change.
Third-party bodies need secure access, custody, administration, and compromise response.
Risk domain and consequence determine evaluator scope, rigor, and assurance level.
Foundation 4 defines meaningful review. Foundation 5 scales and institutionalizes it.
Third-party evaluation can move from voluntary review to procurement, certification, insurance, and formal requirements.
Market recognition should reward competence and integrity rather than marketing.
Accreditation, proficiency, registries, and mutual recognition support cross-border acceptance.
Standards Body adopts the following working positions.
Third-party AI assurance should be treated as an ecosystem, not a single service category.
Testing, evaluation, inspection, audit, certification, and accreditation should not be used interchangeably.
Accreditation recognizes competence within scope. It does not guarantee every result.
No evaluator should claim competence across all AI domains by default.
Evaluator scopes should identify activity, domain, method, system type, assurance level, and limits.
Recognition should depend on demonstrated competence, impartiality, consistent operation, security, and performance.
Initial approval should be followed by surveillance, proficiency testing, and reassessment.
The bodies accrediting evaluators should themselves be competent, impartial, and peer reviewed.
Scheme owners, evaluators, certification bodies, and accreditation bodies should be separated where incompatible conflicts exist.
Developer-funded evaluation can be legitimate only with strong impartiality controls and disclosure.
Result-dependent compensation should be prohibited.
Evaluator client concentration should be disclosed and monitored.
Evaluator shopping and opinion shopping should be controlled.
A third-party review should not be marketed as certification unless a valid certification scheme exists.
Broad "safe AI" certification claims should be avoided.
Marks and certificates should identify system, version, scope, scheme, and expiry.
High-consequence evaluation requires secure access and evidence preservation.
Smaller evaluators and open communities should have credible entry pathways.
Open tools and shared facilities can reduce barriers but do not replace quality assurance.
Proficiency testing and cross-evaluator replication are core infrastructure.
Evaluator security incidents should affect status and scope.
Public registries should make legitimate scope and current status verifiable.
International mutual recognition should begin with narrow, comparable scopes.
Technical recognition should not force identical policy decisions across jurisdictions.
Government, private, academic, nonprofit, and open evaluators can coexist.
No evaluator or accreditation body should become an unaccountable permanent monopoly.
Certification should not transfer responsibility away from developers or deployers.
Evaluator evidence should expire after material system, method, or personnel changes.
The assurance ecosystem should be monitored for capture, concentration, burden, and real-world effectiveness.
Formalization should proceed carefully enough to avoid freezing immature evaluation science.
An evaluator should be recognized only when:
Recognition should be limited or suspended when:
A certification claim should not be permitted when:
Mutual recognition should not be granted when:
Evaluator:
Accreditation body:
Scope identifier:
Status:
Effective date:
Expiry or reassessment date:
Evaluator:
Scope:
Assessment date:
Assessors:
Complaint identifier:
Complainant:
Evaluator:
Engagement or certificate:
Date:
| Dimension | Core Question |
|---|---|
| Role clarity | Are testing, audit, certification, and accreditation distinguished? |
| Scheme quality | Are requirements and decision rules defined? |
| Competence | Does the evaluator demonstrate scope-specific ability? |
| Scope | Are claims limited to approved activities? |
| Impartiality | Are ownership, funding, consulting, and client risks controlled? |
| Quality system | Can the body operate consistently and correct errors? |
| Method validity | Are evaluation methods fit for purpose and versioned? |
| Security | Can sensitive models, tasks, and evidence be protected? |
| Access | Does the evaluator receive enough access for its claims? |
| Proficiency | Has performance been compared with peers? |
| Surveillance | Is competence monitored after initial recognition? |
| Decision independence | Is final attestation separated from incompatible roles? |
| Complaints | Can stakeholders raise and resolve material concerns? |
| Appeals | Can recognition and certification decisions be challenged? |
| Sanctions | Can scope be reduced, suspended, or withdrawn? |
| Registry | Can users verify status and scope? |
| Mark control | Are certification claims precise and enforceable? |
| Market health | Is there competition without a race to the bottom? |
| Small-actor access | Can qualified smaller evaluators participate? |
| International recognition | Can credible results cross borders? |
| Liability | Are responsibility and professional consequences appropriate? |
| Dynamic adaptation | Can scopes and methods evolve with AI systems? |
| Public interest | Does the ecosystem serve more than client marketing? |
| Decision utility | Does assurance improve real-world decisions? |
Independent evaluation cannot become durable public infrastructure if every engagement begins from zero.
A one-off review may produce insight.
An ecosystem produces continuity.
It can preserve methods.
It can train people.
It can compare performance.
It can discipline claims.
It can investigate failure.
It can recognize competence.
It can withdraw recognition.
It can allow one jurisdiction to trust evidence produced in another.
But an ecosystem can also become a market for reassurance.
Commercial pressure can reward favorable reports.
Accreditation can become a badge.
Standards can entrench incumbents.
Security requirements can exclude smaller experts.
Certification can be mistaken for proof of safety.
A few organizations can accumulate excessive power over access, evaluation, and public legitimacy.
The design must therefore begin with clear distinctions.
Testing is not certification.
Review is not accreditation.
Accreditation is not proof that every result is correct.
A certificate is not a universal statement about a system.
Third party does not automatically mean independent.
Prestige does not automatically mean competent.
Formal process does not automatically mean useful.
The purpose of the ecosystem is not to produce more labels.
It is to produce more trustworthy evidence at scale.
That requires institutions capable of evaluating the evaluators.
It requires narrow scopes.
It requires quality systems.
It requires proficiency.
It requires surveillance.
It requires complaints and appeals.
It requires consequences for failure.
It requires pathways for new entrants.
It requires international recognition without global monopoly.
It requires the willingness to say that some forms of AI assurance are not yet mature enough for certification.
The fifth foundation of Standards Body is therefore accountable evaluation capacity.
The future of frontier AI assurance will depend not only on which systems are tested, but on whether the institutions conducting the tests deserve to be trusted.
[^iso-17011]: International Organization for Standardization, ISO/IEC 17011:2017, Conformity assessment, Requirements for accreditation bodies accrediting conformity assessment bodies. https://www.iso.org/standard/67198.html
[^iso-17025]: International Organization for Standardization, ISO/IEC 17025, Testing and calibration laboratories. https://www.iso.org/ISO-IEC-17025-testing-and-calibration-laboratories.html
[^iso-17020]: International Organization for Standardization, ISO/IEC 17020:2026, Conformity assessment, Requirements for the operation of various types of bodies performing inspection. https://www.iso.org/standard/17020
[^iso-17065]: International Organization for Standardization, ISO/IEC 17065:2012, Conformity assessment, Requirements for bodies certifying products, processes and services. https://www.iso.org/standard/46568.html
[^iso-17021]: International Organization for Standardization, ISO/IEC 17021-1:2015, Conformity assessment, Requirements for bodies providing audit and certification of management systems. https://www.iso.org/standard/61651.html
[^iso-casco-bodies]: International Organization for Standardization, CASCO Toolbox: Bodies. https://casco.iso.org/bodies.html
[^iso-casco-examples]: International Organization for Standardization, CASCO Toolbox: Examples. https://casco.iso.org/examples.html
[^iso-recognition]: International Organization for Standardization, CASCO Toolbox: Recognition of conformity-assessment bodies. https://casco.iso.org/recognition-of-cabs.html
[^iso-building-trust]: International Organization for Standardization, Building Trust: The Conformity Assessment Toolbox. https://www.iso.org/iso/casco_building-trust.pdf
[^iso-certification]: International Organization for Standardization, Certification. https://www.iso.org/certification.html
[^iaf-mla]: International Accreditation Forum, Purpose of the IAF Multilateral Recognition Arrangement. https://iaf.nu/en/mla-purpose/
[^iaf-about-mla]: International Accreditation Forum, About the IAF MLA. https://iaf.nu/en/about-iaf-mla/
[^iaf-peer]: International Accreditation Forum, IAF ML 4:2025, Policies and Procedures for the MLA. https://iaf.nu/iaf_system/uploads/documents/IAF_ML_4_Issue_9_04072025.pdf
[^ilac-mra]: International Laboratory Accreditation Cooperation, ILAC MRA and Signatories. https://ilac.org/ilac-mra-and-signatories/
[^ilac-home]: International Laboratory Accreditation Cooperation, Global Accreditation Cooperation. https://ilac.org/
[^ilac-facts]: International Laboratory Accreditation Cooperation, Facts and Figures. https://ilac.org/about-ilac/facts-and-figures/
[^nist-tevv]: National Institute of Standards and Technology, AI Test, Evaluation, Validation and Verification. https://www.nist.gov/ai-test-evaluation-validation-and-verification-tevv
[^nist-rmf]: National Institute of Standards and Technology, Artificial Intelligence Risk Management Framework 1.0, 2023. https://nvlpubs.nist.gov/nistpubs/ai/nist.ai.100-1.pdf
[^nist-zero-draft]: National Institute of Standards and Technology, Outline: Proposed Zero Draft for a Standard on AI Testing, Evaluation, Verification, and Validation, 2025. https://www.nist.gov/document/outline-proposed-zero-draft-standard-ai-testing-evaluation-verification-and-validation
[^openai-playbook]: OpenAI, A Shared Playbook for Trustworthy Third-Party Evaluations, May 29, 2026. https://openai.com/index/trustworthy-third-party-evaluations-foundations/
[^openai-external]: OpenAI, Strengthening Our Safety Ecosystem with External Testing, November 19, 2025. https://openai.com/index/strengthening-safety-with-external-testing/
[^frontier-auditing]: Miles Brundage et al., Frontier AI Auditing: Toward Rigorous Third-Party Assessment of Safety and Security Practices at Leading AI Companies, 2026. https://arxiv.org/abs/2601.11699
[^compliance-reviews]: Aidan Homewood et al., Third-Party Compliance Reviews for Frontier AI Safety Frameworks, 2025. https://arxiv.org/abs/2505.01643
[^framework-evaluation]: Lily Stelling et al., Evaluating AI Companies' Frontier Safety Frameworks: Methodology and Results, 2025. https://arxiv.org/abs/2512.01166
[^external-access]: Jacob Charnock et al., Expanding External Access to Frontier AI Models for Dangerous Capability Evaluations, 2026. https://arxiv.org/abs/2601.11916
[^aisi-lessons]: UK AI Security Institute, Early Lessons from Evaluating Frontier AI Systems, 2024. https://www.aisi.gov.uk/blog/early-lessons-from-evaluating-frontier-ai-systems
[^aisi-research]: UK AI Security Institute, Research Agenda. https://www.aisi.gov.uk/research-agenda
[^inspect]: UK AI Security Institute, Inspect AI. https://inspect.aisi.org.uk/
Date: July 16, 2026
Change type: Complete foundational edition
Summary: Establishes the fully developed canonical working white paper for Foundation 5. Defines the third-party auditor ecosystem, core assurance roles, testing, inspection, audit, certification, accreditation, evaluator types, competence, scope, quality systems, method validation, proficiency testing, impartiality, business models, selection, access, security, accreditation architecture, surveillance, complaints, liability, registries, market concentration, evaluator shopping, international recognition, government and insurance relationships, maturity model, implementation pathway, Standards Body evaluator pilot, metrics, failure analysis, objections, evidence gaps, research agenda, standards implications, operational templates, scorecard, and primary-source research basis.
Status: Ready for internal review and future expert critique.