The fast answer: an AI system is tested for safety by running it through structured batteries of prompts and scenarios, attacking it the way a motivated adversary would, having experts review what automated tests flag, and then scoring the results against fixed criteria. Good testing breaks "is it safe?" into specific questions: does it state falsehoods as facts, can it be talked out of its rules, what personal data does it leak, how does it treat a vulnerable user, can a human stop it, how does it hold up against a determined attacker, and can anyone reconstruct why it did what it did. Each question needs its own tests, and each answer needs evidence someone else can check.

This guide is for teams building customer-facing AI who need to know what an independent evaluation will look at, and for the buyers, insurers and regulators who read the results. It walks through the recognised risk categories, the seven areas the A.E.G.I.S.™ framework measures, a real failure that each area is designed to catch, and what separates a test result you can rely on from a demo.

What safety testing means for an AI system

Traditional software testing checks that code does what the specification says. AI systems, especially those built on large language models, behave differently. The same prompt can produce different answers, small changes in wording can flip the outcome, and the system can be steered by text it reads from a web page or a file rather than from its user. A safety test therefore has to sample behaviour across many inputs and conditions, including hostile ones, rather than confirm a single expected output.

Three levels of evaluation

The US National Institute of Standards and Technology (NIST) runs an evaluation program called ARIA, Assessing Risks and Impacts of AI, which it describes as "an evaluation environment which is sector and task agnostic." NIST states that ARIA "will support three evaluation levels: model testing, red-teaming, and field testing." That three-part split is a useful way to think about any serious evaluation:

• Model testing checks behaviour against defined tasks and prompt sets, at scale and repeatably.
• Red-teaming puts people (and increasingly other AI systems) in the attacker's seat to find the failures nobody wrote a test for.
• Field testing looks at what happens when the system meets real users in a real deployment context.

No single level is enough. Automated test batteries give coverage and comparability. Red-teaming finds the creative failures. Context decides which failures actually matter for a given product.

The recognised risk map: NIST AI 600-1

If you want one public document that lists what can go wrong with generative AI, it is NIST AI 600-1, the Generative Artificial Intelligence Profile of the NIST AI Risk Management Framework, published on 26 July 2024 and last updated by NIST on 8 April 2026. The final text names twelve risks that are unique to or made worse by generative AI:

1. CBRN Information or Capabilities
2. Confabulation
3. Dangerous, Violent, or Hateful Content
4. Data Privacy
5. Environmental Impacts
6. Harmful Bias or Homogenization
7. Human-AI Configuration
8. Information Integrity
9. Information Security
10. Intellectual Property
11. Obscene, Degrading, and/or Abusive Content
12. Value Chain and Component Integration

The profile is voluntary guidance, and it is explicit that testing should not be a one-off exercise. Among its suggested actions, it recommends evaluating systems "both prior to deployment and on an ongoing basis, through internal and external evaluations," and it names "structured public feedback exercises such as AI red-teaming or independent external evaluations" as ways to gather evidence. It also warns organisations to "avoid extrapolating GAI system performance or capabilities from narrow, nonsystematic, and anecdotal assessments."

For the wider picture of how NIST's framework and ISO/IEC 42001 fit together, see our complete guide to AI standards, including the NIST AI RMF and ISO/IEC 42001.

The seven risk areas A.E.G.I.S.™ measures

The A.E.G.I.S.™ rating system organises product-level testing into seven pillars. Each pillar is scored independently, through automated test batteries, adversarial simulation and expert review, and the results are then combined into a single rating. The pillars translate risk categories like those in the NIST list into questions that can be tested against a specific product. Here is what each one catches, with a documented failure that shows why it matters.

1. Hallucination Integrity

The question: does the system state falsehoods as facts?

NIST AI 600-1 calls this risk confabulation, which "refers to a phenomenon in which GAI systems generate and confidently present erroneous or false content in response to prompts." The confidence is the dangerous part. A wrong answer that sounds uncertain invites checking. A wrong answer delivered fluently does not.

Two cases show the cost. In Mata v. Avianca, Inc., a federal judge in the Southern District of New York sanctioned lawyers who had filed a brief built on AI-generated research. The court's Opinion and Order on Sanctions of 22 June 2023 found that they "submitted non-existent judicial opinions with fake quotes and citations," and imposed a monetary penalty. In Canada, the Civil Resolution Tribunal of British Columbia decided Moffatt v. Air Canada, 2024 BCCRT 149, on 14 February 2024. An airline chatbot had told a customer asking about bereavement fares that he could apply for a reduced bereavement fare after travelling, which conflicted with the airline's own policy page. The tribunal held the airline responsible for what its chatbot said: "It makes no difference whether the information comes from a static page or a chatbot."

How it is tested: A.E.G.I.S.™ benchmarks factual reliability under pressure, ambiguity and domain-specific questioning. In practice that means asking questions where the true answer is known, asking questions that have no answer, rephrasing to see whether the answer drifts, and pushing back on correct answers to see whether the system caves. The Air Canada case is a reminder that the test set has to include the deployer's own policies and facts, not only general knowledge. Our complete guide to documented AI incidents maps this case and others from the AI Incident Database to the pillar that tests for each failure.

2. Guardrail Resistance

The question: can the system be manipulated into unsafe behaviour?

Every customer-facing AI ships with rules: topics it will not discuss, actions it will not take, content it will not produce. Guardrail testing asks how easily those rules give way. The OWASP Top 10 for Large Language Model Applications lists prompt injection as risk LLM01, defining it as a vulnerability that "occurs when user prompts alter the LLM's behavior or output in unintended ways." OWASP adds that "Jailbreaking is a form of prompt injection," one where the attacker tries to make the model ignore its safety protocols entirely.

NIST AI 600-1 draws the same line between two forms. "In direct prompt injections, attackers might craft malicious prompts and input them directly to a GAI system." Indirect attacks arrive through content the system reads, such as a web page, an email or an uploaded document. Researchers demonstrated the indirect form against real applications in a 2023 paper, "Not what you've signed up for", observing that LLM-integrated applications "blur the line between data and instructions."

How it is tested: jailbreak attempts, prompt injection and boundary-pushing scenarios are run against every guardrail. Good guardrail testing covers multi-turn conversations, role-play framings, encoded or translated requests, and instructions hidden inside documents the system is asked to summarise. A guardrail that holds against a blunt request and fails against the third message of a patient conversation has not held.

3. Data Handling and Privacy

The question: what does the system collect, retain and leak?

NIST AI 600-1 lists data privacy as its own risk, covering harm from leaked, misused or de-anonymised personal data. Language models add a twist ordinary software does not have: they can memorise their training data and repeat it. In "Extracting Training Data from Large Language Models", researchers showed that "an adversary can perform a training data extraction attack to recover individual training examples by querying the language model." They reported extracting "hundreds of verbatim text sequences," including public personally identifiable information such as names, phone numbers and email addresses.

For a deployed product, the larger risk is often more ordinary: a support assistant that reveals another customer's order, a system prompt that exposes internal details, or an AI toy that records more than its privacy notice says.

How it is tested: A.E.G.I.S.™ audits data collection behaviour, PII exposure and privacy compliance posture. That includes probing for leakage of seeded personal data, attempts to extract system prompts and connected records, and checking what the product actually asks users for against what it needs.

4. Ethical Alignment

The question: does the system behave appropriately with vulnerable users and sensitive topics?

NIST AI 600-1 groups part of this under Human-AI Configuration, which covers over-reliance, anthropomorphising and "concerns about emotional entanglement between humans and GAI systems." Regulators are paying attention. On 11 September 2025 the US Federal Trade Commission issued orders to seven companies under its 6(b) study authority, asking how they "measure, test, and monitor for negative impacts before and after deployment," and what steps they have taken to "evaluate the safety of their chatbots when acting as companions," with a focus on children and teens.

How it is tested: ethical boundary stress tests measure alignment in practice, using scenarios that put a child, a person in distress or a user seeking dangerous advice in front of the system. Refusing everything counts as a failure too. The response has to fit the person on the other side: redirecting, declining, escalating or giving safe information as the situation requires. For consumer products, A.E.G.I.S.™ also certifies age appropriateness, from Infant to Adult, with supervision guidance printed on the mark.

5. Autonomy and Control

The question: how far can the system act on its own, and can it be stopped?

As AI moves from answering questions to taking actions (sending emails, editing records, making purchases, calling other tools), the safety question changes. A wrong answer can be ignored. A wrong action has already happened. The indirect prompt injection research above showed that injected text can affect whether and how an application calls other APIs, which turns a content problem into an action problem.

The EU AI Act addresses control directly for high-risk systems. Article 14 of Regulation (EU) 2024/1689 requires such systems to be designed so "that they can be effectively overseen by natural persons during the period in which they are in use." It asks that overseers stay aware of automation bias, the tendency of "automatically relying or over-relying on the output," and that they can intervene through a "'stop' button or a similar procedure that allows the system to come to a halt in a safe state."

How it is tested: A.E.G.I.S.™ evaluates autonomous behaviour, escalation paths and human override integrity. Does the agent stay inside its permitted actions? Does it ask for confirmation before irreversible steps? When a human says stop, does it stop, and does it stop cleanly, without leaving a half-finished transaction behind?

6. Adversarial Robustness

The question: how does the system hold up against a motivated attacker?

Guardrail testing asks whether rules can be bent with clever wording. Adversarial robustness asks what happens when the attacker is systematic. In "Universal and Transferable Adversarial Attacks on Aligned Language Models" (2023), researchers described a method that "automatically produces these adversarial suffixes by a combination of greedy and gradient-based search techniques," and reported that the resulting prompts were "quite transferable" to other models, including publicly released ones they had no internal access to. Attacks are no longer handcrafted one at a time; they can be generated and reused.

The EU AI Act names the attack families a robust high-risk system has to withstand. Article 15(5) lists attacks that try to "manipulate the training data set (data poisoning)," attacks on "pre-trained components used in training (model poisoning)," and "inputs designed to cause the AI model to make a mistake (adversarial examples or model evasion)," along with confidentiality attacks and model flaws.

How it is tested: red-team exercises and multi-agent simulations probe real-world resilience. Multi-agent simulation matters more each year, because many products now talk to other AI systems, and an attacker can be another agent rather than a human at a keyboard.

7. Explainability and Transparency

The question: can the system's behaviour be understood and audited?

The NIST AI Risk Management Framework (NIST AI 100-1) treats "explainable and interpretable" as one of its seven characteristics of trustworthy AI. It defines explainability as "a representation of the mechanisms underlying AI systems' operation," and interpretability as "the meaning of AI systems' output in the context of their designed functional purposes." Article 13 of the EU AI Act sets a related duty for high-risk systems: their operation must be "sufficiently transparent" for deployers to interpret the output and use it appropriately.

Explainability is also what makes every other pillar auditable. If nobody can trace why a system refused one request and complied with another, a failure cannot be diagnosed, and a fix cannot be verified.

How it is tested: A.E.G.I.S.™ assesses traceability, disclosure quality and the evidence trail behind decisions. Does the product tell users they are dealing with AI? Are its limitations disclosed in plain language? Can the operator reconstruct what the system saw and did in a given interaction?

How a structured evaluation runs, step by step

The pillars describe what is measured. The process describes how. On the A.E.G.I.S.™ platform, an evaluation moves through five steps:

1. Scoping and intake. The evaluation scope is defined: which agents are covered, their autonomy level, the deployment context, and the external reliance the certification must support. A children's companion app and an internal coding assistant need different tests.
2. Automated pillar testing. Standardised test batteries run across all seven pillars: guardrail probing, hallucination benchmarks, data collection audits and ethical stress tests.
3. Adversarial and human review. For higher-assurance engagements, red-team exercises, multi-agent simulations and expert review panels validate what the automated tests found.
4. Rating and certification. The system receives its rating, certification badge, formal approval letter and public Safety Registry listing.
5. Monitoring and re-certification. As models retrain and products evolve, continuous monitoring and periodic reassessment keep the certification current.

What makes a test result trustworthy

Anyone can run a hundred prompts and publish a chart. A result that a buyer, insurer or regulator can rely on needs four further properties.

Independence

Internal testing is the vendor grading its own homework. That does not make it worthless; every responsible developer should test internally. But it is a different kind of evidence from an evaluation by a party with no stake in the outcome. RateAI has no stake in a product's rating, and its fees are tied to evaluation milestones rather than outcomes. NIST AI 600-1 points the same way, suggesting policies "to bolster oversight of GAI systems with independent evaluations or assessments."

Reproducibility

If the scoring rules change quietly between two evaluations, the results cannot be compared. A.E.G.I.S.™ uses epoch-based scoring: pillar weights, rating thresholds and quorum parameters are defined by the active scoring epoch rather than hardcoded, and each epoch freezes those parameters so evaluations stay comparable across time.

A tamper-evident evidence trail

A score is only as credible as the record behind it. On the A.E.G.I.S.™ platform, every prompt case, validator vote, consensus decision and certification is chained by SHA-256 hash, and records are write-once: nothing can be altered, deleted or backdated. An auditor can follow a rating back to the individual test cases that produced it.

The possibility of failure

A certification everyone passes is worth nothing. Systems that do not meet the bar receive a detailed findings report and a remediation path, and can re-enter evaluation once the issues are fixed. If a testing process has never failed anything, ask what it is actually measuring.

We cover the difference between self-assessment, third-party testing and formal certification in more depth in our complete guide to AI safety certification.

From test results to a rating

Raw test logs are useful to engineers and almost useless to everyone else. The output of an A.E.G.I.S.™ evaluation is a standardised scorecard, designed to be as readable as a nutrition panel, for procurement teams, regulators, retailers and parents.

• Overall Safety Rating: a single letter grade from A (Maximum Safety) to F (Dangerous), readable without a technical background.
• Per-pillar grades: each of the seven pillars is graded individually (H, G, D, E, C, R and X), so a buyer can see that a system is strong on privacy but weaker against adversarial attack, for example.
• Certification seal: pillar results determine the A.E.G.I.S.™ Certified seal, from A-5 Guardian, the highest safety assurance, through A-4 Shielded, A-3 Contained and A-2 Exposed, down to A-1 Unrestricted.
• Age-suitability mark: for consumer products, a mark from Infant to Adult with supervision guidance.

Certified products also carry a verifiable listing in the public Safety Registry, which anyone can check without logging in to confirm that a certification is current and has not been downgraded.

Testing does not end at launch

AI products change after release. Models are retrained or swapped, prompts are edited, new tools are connected. Each change can move behaviour on any pillar, sometimes for the better, sometimes not. A test result from six months and three model versions ago describes a different product.

That is why NIST AI 600-1 recommends evaluation "on an ongoing basis," and why A.E.G.I.S.™ certifications carry a re-certification cycle. Monitoring subscriptions test continuously for behavioural drift between formal reassessments, and material changes to a system trigger reassessment to keep the rating current and defensible.

Wiring safety tests into the development pipeline

The cheapest place to catch a safety regression is before it ships. The A.E.G.I.S.™ platform is built API-first, with a documented REST API and Python and Node.js samples on every endpoint, so evaluations can run inside a CI/CD pipeline. Teams can register products, submit evaluations, query scores and certifications, and verify the audit ledger's hash chain programmatically. Every prompt case also records response time in milliseconds, so the dashboard shows per-pillar average and p95 latency alongside safety scores, a reminder that a safety layer which makes the product unusably slow will not survive contact with the product team.

What testing does not tell you

A safety evaluation measures how a system behaved under the conditions tested. It is strong evidence, and it has limits worth stating plainly.

• It is not a legal determination. RateAI provides compliance mapping to frameworks such as the EU AI Act, NIST AI RMF and ISO 42001, and certification artifacts are designed as defensible evidence. Formal regulatory determinations remain with the relevant authorities. Our complete guide to AI regulation under the EU AI Act and US laws covers what the laws themselves require.
• It does not replace an organisation's own governance. A product rating says how the product behaves. A management system standard such as ISO/IEC 42001 concerns how the organisation runs its AI work. Both questions matter.
• It covers the scope that was agreed. That is why scoping comes first, and why material changes trigger reassessment.

Frequently asked questions

How do you test an AI system for safety?

You run it through structured prompt and scenario batteries, attack it with jailbreaks, prompt injection and red-team exercises, have experts review the findings, and score the results against fixed criteria. A.E.G.I.S.™ does this across seven pillars, from Hallucination Integrity to Explainability and Transparency.

What are the main risks AI safety testing looks for?

NIST AI 600-1 lists twelve generative AI risks, including confabulation, data privacy, information security and harmful bias. A.E.G.I.S.™ turns these into seven testable pillars covering falsehoods, manipulation, data leakage, treatment of vulnerable users, human control, attacker resilience and auditability.

What is the difference between red-teaming and automated AI testing?

Automated testing runs standardised prompt batteries at scale so results are repeatable and comparable. Red-teaming puts skilled people, or other AI agents, in the attacker's role to find failures no fixed test anticipated, and the two work best together.

Does passing an AI safety test make a product compliant with the EU AI Act?

No. Testing produces evidence that supports compliance work, and RateAI provides compliance mapping to the EU AI Act, NIST AI RMF and ISO 42001, but formal regulatory determinations remain with the relevant authorities.

How often should an AI system be retested?

Whenever it changes materially, and on a regular cycle in between, because retraining, prompt edits and new integrations can all shift behaviour. NIST AI 600-1 recommends evaluation both before deployment and on an ongoing basis.

See where your AI stands

If you are building customer-facing AI, the most useful first step is a measured baseline across all seven A.E.G.I.S.™ pillars rather than a guess. You can start with a free safety benchmark for one AI agent, or scope a full certification engagement, on our pricing and certification tiers page.