Contact Us
Contact Us
Software Testing

Test Automation Strategy: What to Automate First (and What Never to)

A four-condition rule for test automation: what to automate first, what to leave manual, and how to keep the suite fast enough that people trust it.

JRC Software Team
7 min read
Test Automation Strategy: What to Automate First (and What Never to)

"Our end-to-end suite takes 70 minutes and fails about one run in three. Should we buy a better tool?" No. We get asked some version of this a lot, and the tool is rarely the issue.

Nobody sets out to build a slow, flaky suite. Teams start automating because releases feel risky, then keep adding browser scripts until the run takes an hour, fails at random, and gets rerun until it goes green. At that point automation costs more than it saves, and a green build means nothing.

What's missing is a test automation strategy: a short, written rule for what gets automated first, at which level, and what stays manual on purpose. Below is the rule we use, the priority order it produces, and the checks we'd tell you never to automate.

Fast feedback is now the baseline

DORA's guidance on continuous integration says teams do better when developers merge into trunk at least daily, with automated test runs capped at about 10 minutes. A 70-minute suite can't support that rhythm, so people stop waiting for it.

DORA's page on test automation is blunt about quality over quantity. When tests pass, you should be confident the software is releasable. When they fail, it should be a real defect. Flaky tests aren't tolerated, and DORA would rather you prune a big unreliable suite down to a small one the team trusts.

AI coding assistants add pressure from the other side: more change per sprint, all of which needs a fast signal. We cover that in testing AI-generated code.

How suites turn into ice-cream cones

Martin Fowler's Test Pyramid (2012) describes the shape struggling suites take: lots of tests through the UI and very few unit tests underneath, an inverted pyramid often called an ice-cream cone. UI tests are slow, brittle when the interface changes, and more prone to non-determinism, so the suite gets slower and less trusted as it grows.

Ham Vocke's Practical Test Pyramid (2018) turns that into working rules. Lots of small, fast unit tests, some coarser integration tests, very few end-to-end tests. Push each test as far down the pyramid as it'll go, and delete higher-level tests that duplicate what lower levels already prove.

Flakiness is the hidden cost. Fowler's Eradicating Non-Determinism in Tests (2011) explains how a few randomly failing tests teach a team to ignore red builds, and then real failures get ignored too. The usual causes are shared state between tests, bare sleeps around asynchronous code, live remote services, calls to the system clock, and resource leaks. In practice that's a Thread.sleep(2000) or cy.wait(2000) that passes on a laptop and fails on a busy CI agent. Wait on a condition instead, and inject a clock you control.

A test automation strategy in four conditions

Automate a check first when it meets all four:

  1. It runs often. It'd be repeated on every change or every release, not once a quarter.
  2. It protects something expensive to break. Money, data, security, compliance or a core user journey.
  3. Its expected result is stable. The outcome is deterministic and the feature isn't being redesigned next sprint.
  4. It can run at the lowest level that proves it. If a unit or API test can prove the behaviour, don't write a browser test.

Fails condition 3? Not ready yet. Stabilise the design or keep it manual. Fails condition 4? Rewrite it lower down rather than automating through the UI.

For a typical web or mobile product, that rule gives an order like this:

PriorityWhat to automateLevelWhy it comes here
1Business rules: pricing, eligibility, permissions, calculationsUnitFast, deterministic, and where costly logic bugs live
2Core API behaviour and contracts between servicesAPI and contractProves integration without a browser
3Deployment smoke checksPipelineCatches broken builds and configuration in minutes
4A handful of critical journeys: sign-up, login, checkoutEnd-to-endThe few flows that must never break
5A regression test for every production bugLowest level that reproduces itStops the same defect coming back
6Cheap non-functional checks: dependency scanning, accessibility linting, a performance smoke testPipelineLow cost, early warning

On item 5, Fowler's advice is worth making team policy: when a high-level test exposes a bug, reproduce it with a unit test before fixing it. DORA says the same. Every late bug then strengthens the fast layer.

For item 6, a short load test in the pipeline is usually enough until launch. We cover the fuller version in performance testing before launch.

Checks we'd leave to people

Some checks cost more to automate than they ever return:

  • Exploratory testing. DORA keeps exploratory and usability testing running throughout delivery because they find what nobody thought to script.
  • Usability and visual judgement. Whether a flow is confusing or a layout looks off is a human call. Visual diff tools can flag a change, but someone still decides if it matters.
  • Features still changing every sprint. Automating a moving target means rewriting tests weekly. Wait until it settles.
  • Record and playback scripts. Fowler calls record and playback tools almost always a bad idea for automation, because they resist change.
  • Direct tests against third-party systems you don't control. Use a test double for speed and determinism, and add a contract test against the real service so the double doesn't drift, as Vocke recommends.
  • One-off checks. A data migration you verify once is a script, not a regression test. Keep it out of the permanent suite.

Measuring automation testing ROI without made-up percentages

Automation testing ROI is often sold with invented numbers. Use ones your team already has:

  1. Suite duration against DORA's 10-minute guide.
  2. Flaky test count: tests in quarantine, or rerun until they pass.
  3. Escaped defects: production bugs a test should have caught.
  4. Manual regression hours per release, before and after.
  5. Maintenance hours: time spent fixing tests rather than product code.

The sum is simple. Manual hours saved per release, times releases, minus the hours spent building and maintaining the suite. Most teams underestimate that last term. A big end-to-end suite can lose money even while it catches real bugs, because maintenance and reruns cost more than the manual effort it replaced.

Four habits that keep a suite trusted

  • Quarantine, then fix or delete. Fowler suggests moving flaky tests out of the main pipeline and capping the quarantine, either by count or by time, so it doesn't become a graveyard.
  • Developers own the tests. DORA warns that when a separate group owns automation, suites stay broken and developers write code that's hard to test. Testers pair with developers and lead exploratory work.
  • Red builds stop the line. DORA says fixing a broken build takes priority over other work, and a change that can't be fixed in a few minutes gets reverted.
  • Prune every quarter. Delete duplicated and low-value tests.

What we do in a QA strategy engagement

Our quality engineering and automation work starts with your current suite, not a tool choice. In week one we pull pipeline history to get real run times and rerun rates, list the tests that fail most without a code change, find duplicated coverage across levels, and map your highest-risk features. Then we agree a priority list with you using the four conditions above.

After that we work inside your team. Our engineers pair with your developers to move slow UI tests down to unit, API and contract level, add deployment smoke checks, and set pipeline gates that block real failures without crying wolf. You end up with a faster suite your developers own, a quarantine policy, and a one-page rule for what gets automated next.

Switching frameworks won't fix flakiness. The causes travel with you.

If your suite is slow, flaky or routinely ignored, book a QA strategy session and we'll help you decide what to automate first, and what to stop automating.

Frequently asked questions

What should we automate first?

Automate checks that run on every change, protect something costly to break, have a stable expected result, and can run at the lowest test level. In practice that's unit tests on business rules, API and contract tests on core services, deployment smoke checks, and a small number of end-to-end tests for critical journeys such as sign-up, login and checkout.

How many end-to-end tests should we have?

As few as you can while still covering the journeys that must never break. End-to-end tests are the slowest and most brittle layer, so the test pyramid keeps them to a handful of critical paths. If an end-to-end test checks logic a unit or API test could prove, move it down and delete the high-level version.

How do we calculate automation testing ROI?

Use your own data, not industry percentages. Add up the manual regression hours saved per release, multiply by your release frequency, then subtract the hours spent writing, fixing and rerunning automated tests. Track escaped defects and suite duration alongside it. If maintenance and reruns cost more than the manual effort replaced, the suite needs pruning.

Should testers or developers write automated tests?

Developers should be the main authors and maintainers, according to DORA's guidance, with testers working alongside them. Testers bring test design skill, lead exploratory and usability testing, and help curate the suite. When a separate team owns all the automation, tests tend to stay broken and the code becomes harder to test.

Software Testing

Book a QA strategy session

Tell us where you're stuck and we'll come back with practical next steps.

Get in touch

Sources