AI SOP Template: Quality Assurance Testing

QA is where most teams discover whether their AI development workflow actually shipped quality or just shipped speed. AI cannot replace exploratory testing or human judgment about user experience, but it can generate test plans, draft cases, run regressions, and triage failures faster than any human. This SOP is the version I run with engineering teams that want to integrate AI into QA without losing the safety net.

Why Quality Assurance Needs a Documented AI SOP

Most teams have an unwritten QA process: developers write some tests, a few people manually click around before release, and bugs found in production go into a backlog labeled "tech debt." That works at 5 engineers. It collapses at 20.

A documented SOP forces clear ownership, defined coverage targets, and consistent quality gates. AI accelerates the work that previously made the SOP unsustainable, especially around test generation, regression management, and bug triage.

The Full SOP Template

This SOP assumes a software product with a test pyramid: unit, integration, end-to-end, plus manual exploratory. Adapt to your stack.

Phase 1: Test Planning (per feature, before code is written)

1. PM and engineer write the feature ticket with clear acceptance criteria.
2. Engineer runs the Test Plan Generation Prompt in Claude or ChatGPT:
   • "Given this feature spec, generate a test plan covering: happy path, error states, edge cases, security considerations, accessibility, and cross-browser/device concerns. Output as a checklist organized by test layer (unit, integration, e2e, manual)."
3. Engineer reviews, removes irrelevant cases, adds context-specific cases AI missed.
4. Test plan is attached to the ticket and reviewed during ticket sign-off.
5. Risk classification: low, medium, high. High-risk gets stricter review and mandatory manual exploratory pass.

Phase 2: Test Authoring (alongside implementation)

1. Engineer writes unit tests as code is written. AI assists via Cursor, Claude Code, or Copilot.
2. Use the Unit Test Generation Prompt for non-trivial functions:
   • "Generate Jest tests for this function. Cover: each branch, edge inputs (empty, null, max), error cases, and one property-based test if applicable. Do not test implementation details — test behavior."
3. Integration tests are added when crossing service or module boundaries.
4. End-to-end tests via Playwright or Cypress for any critical user journey. Use AI to generate the first draft from a written test description, then refine manually.
5. All test code is reviewed in the same PR as the feature code. Same standards.

Phase 3: Continuous Test Execution

1. Every PR runs the full unit and integration suite via CI (GitHub Actions, CircleCI, or Buildkite).
2. End-to-end suite runs on every PR to main, plus nightly against a staging environment.
3. AI-powered test triage runs on every failure:
   • "Given this test failure log and the recent code changes, classify the failure as: real bug, flaky test, environment issue, or test out of date. Provide reasoning and a recommended action."
4. Real bugs auto-create tickets in Linear or Jira with the AI's analysis attached. Flaky tests get tagged in a flake-tracker dashboard.
5. Flake rate over 5 percent triggers a quality SLO breach alert.

Phase 4: Pre-Release Verification (per release)

1. Release manager runs the Release Risk Assessment Prompt:
   • "Given the changelog and the diff scope, identify: highest-risk changes, areas needing manual exploratory testing, regression risks, and any change that should be feature-flagged or rolled out gradually."
2. QA or designated tester executes the manual exploratory pass on flagged areas.
3. AI generates a release notes draft from the changelog. Human edits before publication.
4. Release goes out behind feature flags where applicable, monitored for 1 hour minimum before flag flip.
5. If anything looks off, the rollback runbook executes immediately. No "let me investigate" — rollback first, debug later.

Phase 5: Production Monitoring and Bug Triage

1. Sentry, Datadog, or your APM catches runtime errors.
2. A scheduled job runs the Error Cluster Analysis Prompt every morning:
   • "Cluster these errors from the last 24 hours by likely root cause. For each cluster, suggest: probable affected component, severity, and a starting investigation step."
3. On-call engineer reviews the AI clustering, opens tickets for real issues, dismisses noise.
4. Severity 1 issues auto-page. Everything else flows into the regular bug triage meeting.

Phase 6: Test Maintenance (weekly)

1. Review the flake-tracker dashboard. Any test flaking more than 3 times in a week is either fixed or quarantined.
2. AI runs a Coverage Gap Analysis monthly:
   • "Given the current test suite and the production error log from the last 30 days, identify the top 5 production issues that should have been caught by tests but were not. Suggest specific test cases that would have caught each."
3. Action items flow into the next sprint.

Tools You'll Use (Verified May 2026)

• Unit testing: Jest, Vitest, Pytest, JUnit — whatever matches your language.
• End-to-end: Playwright is the current default for web. Cypress is a fine alternative.
• AI-augmented end-to-end and self-healing platforms: Testim (AI element identification, claims up to 70 percent flake reduction, $450/user/mo starting), mabl (low-code GUI for browser, API, mobile, around $499/mo), Functionize (NLP plain-English authoring), Applitools (visual AI for UI consistency), Reflect.run (no-code with SmartBear HaloAI for natural-language test steps), BrowserStack Low-Code (AI self-healing claims a 40 percent reduction in build failures from UI churn).
• AI test generation and triage: Claude or GPT-class via Cursor, Claude Code, or a custom CI integration.
• Visual regression: Percy, Chromatic, Argos, or Applitools for UI changes.
• Performance testing: k6 or Artillery, with AI summarizing results into a release-blocking report.
• Error monitoring: Sentry, Datadog, or Better Stack.
• Bug tracking: Linear or Jira, with AI integration for triage.
• Framework reference: ISTQB Certified Tester AI Testing (CT-AI) v2.0 — covers ML model testing, GenAI/LLM testing, ISO/IEC 25059 AI quality characteristics, and lifecycle-based testing for input data, model, and ML development. Recommended baseline for any QA lead overseeing an AI-augmented test program.

Sample Prompts You Can Steal

Test Plan Generation: "Feature spec: [paste]. Risk level: [low/medium/high]. Generate a test plan structured as: 1) Happy path scenarios (3 to 5), 2) Error states (auth failure, network failure, validation failure, etc.), 3) Edge cases (empty data, max data, concurrent users, race conditions), 4) Security (auth bypass, injection, IDOR), 5) Accessibility (keyboard nav, screen reader, color contrast), 6) Cross-browser/device. For each, specify: test layer (unit/integration/e2e/manual), priority (P0/P1/P2), and a 1-line expected behavior."

Failure Triage: "Test that failed: [name]. Failure log: [paste]. Recent commits to relevant files: [paste]. Output JSON: classification (real_bug/flaky/env_issue/test_outdated), confidence (high/medium/low), reasoning (2-3 sentences), recommended_action (fix_code/fix_test/quarantine/ignore), affected_areas (list of components)."

Release Notes Draft: "Given this changelog, write release notes structured as: 1) New features (user-facing language, no jargon), 2) Improvements, 3) Bug fixes (only user-impacting ones), 4) Breaking changes. Skip internal refactors and dev-only changes. Tone: professional, concise, no marketing fluff."

Bug Reproduction Steps: "Given this bug report and stack trace, generate: 1) Likely reproduction steps numbered, 2) Required test data or environment state, 3) Expected vs actual behavior, 4) Suggested first investigation steps. Mark anything you are uncertain about with a question mark."

Roles and Responsibilities

• Engineers: own unit and integration tests for their code. Author tests, fix flakes, no exceptions.
• QA Lead or Senior Engineer: owns the e2e suite, the SOP, and the prompt library.
• Release Manager (rotating): owns the pre-release verification and the rollback decision.
• On-Call Engineer (rotating): owns morning error triage and severity 1 response.
• PM: owns the acceptance criteria quality. Bad criteria, bad tests.
• AI Steward: validates that AI-generated tests are testing behavior, not implementation. Audits monthly.

Common Pitfalls

1. AI generates tests, nobody reviews them. Bad tests pass forever, give false confidence, then everything breaks during a refactor. Review every AI-generated test like you would review a junior engineer's PR.
2. Coverage as the only metric. 95 percent coverage with shallow assertions is worse than 70 percent coverage with sharp ones. Track mutation score or escaped bug rate, not just coverage.
3. Skipping manual exploratory. Automation cannot find usability issues, weird interaction states, or anything you did not think to write a test for. Reserve human time for the things only humans can find.
4. Letting flake become normal. Flaky tests train the team to ignore failures. The first ignored failure leads to the first ignored real bug. Aggressive flake elimination is non-negotiable.
5. AI as the gate, not the assistant. AI can recommend "ship" or "block." Humans decide. Always.

Governance and Data Handling

• Test data is synthetic or anonymized. No production PII in test environments, ever.
• E2E tests against staging never use real customer credentials. Service accounts only.
• AI prompts that include code or stack traces are run through tools with appropriate data agreements. Use enterprise contracts.
• Test artifacts (screenshots, videos, logs) follow the same retention and access rules as production logs.
• Security test results are never shared in public channels. Dedicated private channel with explicit access list.

Measuring Whether the SOP Is Working

Track these per release and trend monthly:

• Escaped bug rate (bugs found in production within 7 days of release)
• Flake rate (percentage of test failures that are flake, target under 3 percent)
• Mean time to detect (MTTD) for production issues
• Mean time to repair (MTTR) once detected
• Test maintenance hours per week (a leading indicator of test debt)
• Release confidence score (subjective, surveyed from engineers post-release)

Healthy: escaped bugs trending down, flake under 3 percent, release confidence high. Trouble: any of those moving the wrong direction for two consecutive months.

FAQ

Quality assurance is where AI workflow promises get tested against reality. Done well, you ship faster with fewer escaped bugs and a happier team. Done poorly, you ship more bugs more confidently and the trust collapses. The SOP is the difference. Run it, measure it, and let the trend lines, not the hype, tell you whether to lean further in.