The Assurance Gap

Testing, certification, defect reproduction, and root cause analysis rest on one assumption: run it again, get the same answer.

AI systems break that. The same input can produce different output, so a defect may not reproduce, a fix may not verify, and an investigation may find nothing (see PR-06). Certification assumes a product behaves after the certificate as it did before (see PR-07).

AI made production cheaper and assurance harder at once. We ship fastest into the regime where we can least tell whether a product is safe.

Discussion prompts

• Which of your assurance practices assume you can reproduce a defect on demand?
• How would you investigate an incident in a system that behaves differently each run?
• What evidence would convince you an AI product is safe, when you cannot test it the same way twice?