Contact Us
Contact Us
Software Testing

Testing AI-Generated Code: Why Your QA Process Needs an Upgrade

AI assistants write code faster than teams can review it. The test coverage, review rules and security checks your QA process needs to keep up.

JRC Software Team
7 min read
Testing AI-Generated Code: Why Your QA Process Needs an Upgrade

A pull request arrives late on a Thursday: 1,900 lines across 37 files, mostly written by an assistant, with tests included and every check green. The reviewer has twenty minutes. They skim it, it looks reasonable, they approve. Three weeks later a customer can read another customer's invoices because one endpoint checked the login but not the owner of the record.

That's the gap AI coding assistants open up. Pull requests get bigger and arrive faster, and review time doesn't grow to match. Testing AI generated code isn't harder in principle, but some old assumptions stop holding. The author may not understand every line. The tests may come from the same tool that wrote the bug. And security holes hide in code that compiles and passes the demo.

Why testing AI generated code needs its own rules

Adoption isn't the question any more. Google's summary of the 2025 DORA report, published on 23 September 2025, puts AI adoption among software professionals at 90%, with a median of two hours a day spent working with AI. Only 24% said they trust AI output a great deal or a lot, while 30% trust it a little or not at all. The report links AI adoption to higher delivery throughput, but notes the ongoing challenge of making sure software works as intended before it reaches users.

The security evidence is sharper. Veracode's GenAI code security research, released on 30 July 2025, tested more than 100 large language models on 80 coding tasks. When a secure and an insecure method were both available, the models chose the insecure one 45% of the time. They failed to secure code against cross-site scripting in 86% of relevant cases and against log injection in 88%, and larger models didn't do significantly better than smaller ones. It's vendor research, so read it as a strong signal, not a universal rate.

A Cloud Security Alliance research note dated 31 March 2026 pulls together several studies of AI-built apps. It cites a December 2025 Tenzai study of 15 apps built with five AI coding tools, where every app lacked CSRF protection and had no security headers, and an October 2025 Escape.tech scan of over 1,400 vibe-coded production apps, where 58% had at least one critical vulnerability. The note itself says the most alarming figures tend to come from ungoverned environments, and 15 apps is a small sample. We cover the Supabase and API key failures in our vibe coding security guide.

Five ways AI-written code goes wrong

In our reviews, defects in AI-generated code fall into recognisable patterns:

  • Plausible but wrong edge cases. The happy path works. Empty lists, time zones, rounding, concurrent updates and partial failures often don't.
  • Tests that mirror the code. When the same assistant writes the function and its tests, the tests confirm what the code does, not what it should do. Coverage looks great and the bug survives.
  • Missing authorisation. Endpoints check the user is logged in but not that they own the record. That's the invoice bug above.
  • Inconsistent patterns. Each generated module handles errors, logging and validation a little differently, which makes the next change riskier.
  • Oversized changes. One prompt can touch dozens of files, far more than a reviewer can hold in their head at once.

Coverage that actually means something

Line coverage is weak evidence for AI-written code, because generated tests can execute every line without asserting anything useful. What works better:

  1. Write acceptance examples before generating code. A person writes the expected behaviour, edge cases included, as plain examples. Tests come from those, not from reading the implementation.
  2. Keep test authorship independent. If AI drafts the tests, prompt it from the spec, not the code it just wrote, and have a person review every assertion.
  3. Probe the edges on purpose. Property-based testing tools such as Hypothesis for Python or fast-check for JavaScript generate hundreds of inputs and shrink failures to the smallest case.
  4. Check that tests can fail. Mutation testing (Stryker for JavaScript and .NET, PIT for Java) changes your code and checks the suite notices. A surviving mutant is a test that wouldn't catch a regression.
  5. Turn every escaped bug into a test. DORA's test automation guidance recommends adding a unit test whenever a later stage finds a bug, so it's caught earlier next time.

If your suite is already slow or flaky, sort that first. Our test automation strategy guide covers what to automate and what to cut.

Review rules for AI-assisted pull requests

Review is where AI code quality is won or lost. These rules have held up in the teams we support:

  1. Keep pull requests small. One concern per change, even if the assistant could do more in one go. A soft limit of around 400 changed lines is a reasonable starting point.
  2. The author explains every line. If they can't, it's not ready for review.
  3. Label AI-assisted changes. Reviewers then know to check intent and edge cases rather than style.
  4. Two reviewers for sensitive areas. Authentication, payments, permissions and personal data. A CODEOWNERS file on those paths makes it automatic.
  5. No merge without a test that fails first. New behaviour needs a test that would have failed before the change.
  6. Let the pipeline enforce it. Required status checks and branch protection on main take the temptation to wave things through off the table.

We go further into automated gates and rollback in preparing your platform for AI coding agents.

Security gates on every change

DORA's guidance on pervasive security recommends building security tests into the CI pipeline and giving developers preapproved libraries and toolchains. For AI-written code we treat these as the minimum:

  • Static analysis rules for injection, cross-site scripting and unsafe deserialisation.
  • Dependency scanning, plus a check that every new package actually exists and is maintained. Assistants sometimes suggest package names that don't exist, and someone may have registered one since.
  • Secret scanning on every commit and in build output.
  • Automated tests for authorisation rules, not just authentication.
  • A check for security headers and CSRF protection on deployed web apps.

LLM application testing is a separate job

There's a second problem: testing features that call a language model at run time, like chat assistants, summarisers and agents. The output isn't deterministic, so ordinary pass or fail assertions don't cover it.

The OWASP Top 10 for LLM Applications 2025 is a good checklist. It includes prompt injection, sensitive information disclosure, improper output handling, excessive agency and unbounded consumption. So LLM application testing should cover:

  • An evaluation set of real user prompts with expected properties, rerun whenever the model, prompt or retrieval data changes.
  • Adversarial prompts that try to override instructions or extract data.
  • Validation of model output before it reaches a database, a shell or another user.
  • Limits on what an agent can do, with tests proving those limits hold.
  • Cost and rate limit tests, so one user can't run up an unbounded bill.

How a JRC Software QA review works

Our quality engineering and automation team starts with a sample of your recently merged pull requests. In week one we score them on test quality, review depth, security findings and escaped defects, and compare AI-assisted changes with the rest. Usually two or three gaps stand out, and they're rarely the ones teams expect.

From that we set practical rules: acceptance examples before generation, independent tests, PR size limits, and pipeline gates for security and coverage. For products with LLM features, we build an evaluation set and adversarial tests that run on every model or prompt change. We pair with your developers so the process sticks after we leave.

Banning AI assistants isn't the answer. The fix is in review and the pipeline, not the editor.

If AI is writing more code than your reviewers can read, book a QA review and we'll show you where the gaps are.

Frequently asked questions

Can we use AI to write our tests?

Yes, with care. AI is good at drafting test scaffolding and boundary cases, but tests generated from the code they check often confirm the bug rather than catch it. Generate tests from written acceptance examples or specs, have a person review every assertion, and confirm each test fails when the behaviour is broken.

Is AI-generated code less secure than human code?

The evidence points to real risk. Veracode's 2025 research found models chose an insecure method in 45% of tested cases, and scans of AI-built apps have found missing security headers and exposed secrets. Results vary with tooling and governance, so treat AI-written code as untrusted until automated security checks and review have passed.

What is LLM application testing?

LLM application testing checks features that call a language model at run time. Because output varies, it uses evaluation sets of realistic prompts, adversarial prompts for injection and data leakage, validation of model output before it's used, and limits on agent actions and cost. The OWASP Top 10 for LLM Applications is a good starting checklist.

How do we measure AI code quality?

Compare AI-assisted changes with the rest of your codebase on the same measures: escaped defects, security findings per change, review comments, revert rate and how often tests catch regressions. Line coverage alone is misleading for generated code, so add mutation testing or manual fault injection to check your tests would actually fail.

Software Testing

Book a QA review

Tell us where you're stuck and we'll come back with practical next steps.

Get in touch

Sources