Flaky Tests: Staleness, Retries, and CI

A flaky test is one whose result depends on timing rather than on the code. Selenium suites attract them because everything is asynchronous and nothing waits by default. This lesson is the five causes worth knowing and what to do about each.

Cause 1: stale elements

with webdriver.Chrome(options=opts) as driver:
    driver.get("https://demo.playwright.dev/todomvc")
    box = driver.find_element(By.CLASS_NAME, "new-todo")
    box.send_keys("buy milk", Keys.ENTER)

    rows = driver.find_elements(By.CSS_SELECTOR, ".todo-list li")
    box.send_keys("walk the dog", Keys.ENTER)          # React re-renders the list
    print(rows[0].text)

selenium.common.exceptions.StaleElementReferenceException: Message: stale element
reference: stale element not found in the current frame
  (Session info: chrome=133.0.6943.16)

The handle pointed at a DOM node React threw away. Three fixes, in order of preference.

Re-find at the point of use. Store the locator, not the element:

ITEMS = (By.CSS_SELECTOR, ".todo-list li")

    box.send_keys("walk the dog", Keys.ENTER)
    print(driver.find_elements(*ITEMS)[0].text)

buy milk

Ignore staleness in the wait, so a mid-poll re-render is retried rather than fatal:

wait = WebDriverWait(driver, 10,
                     ignored_exceptions=(StaleElementReferenceException,
                                         NoSuchElementException))
print(wait.until(lambda d: d.find_element(By.XPATH, "//label[text()='buy milk']")).text)

buy milk

Retry the operation when even that is not enough — a click that triggers the re-render that invalidates the element you clicked:

from functools import wraps

def retry_on_stale(attempts=3, delay=0.2):
    def decorate(fn):
        @wraps(fn)
        def wrapper(*args, **kwargs):
            for attempt in range(1, attempts + 1):
                try:
                    return fn(*args, **kwargs)
                except StaleElementReferenceException:
                    if attempt == attempts:
                        raise
                    print(f"  stale on attempt {attempt}, retrying")
                    time.sleep(delay)
        return wrapper
    return decorate


@retry_on_stale()
def complete(driver, title):
    row = driver.find_element(By.XPATH, f"//label[text()='{title}']/ancestor::li")
    row.find_element(By.CSS_SELECTOR, ".toggle").click()

complete(driver, "buy milk")
print("completed")

  stale on attempt 1, retrying
completed

The retry re-runs the whole function, including the lookup — retrying only the click would reuse the same dead handle.

Cause 2: waiting for the wrong thing

    driver.find_element(By.LINK_TEXT, "Active").click()
    print(driver.find_elements(By.CSS_SELECTOR, ".todo-list li")[0].text)

buy milk

That is the old list — the click returned before the filter applied, and the assertion read the pre-filter DOM. It passes about half the time.

Wait for the change, not for an element that was already there:

    old_rows = driver.find_elements(By.CSS_SELECTOR, ".todo-list li")
    driver.find_element(By.LINK_TEXT, "Active").click()
    wait.until(EC.staleness_of(old_rows[0]))
    print([e.text for e in driver.find_elements(By.CSS_SELECTOR, ".todo-list li label")])

['walk the dog']

EC.staleness_of is the underused condition here: it waits for the old DOM to be discarded, which is the only reliable signal that a re-render happened. EC.url_contains and waiting for a specific expected value work too.

Cause 3: animations

    driver.find_element(By.ID, "open-modal").click()
    driver.find_element(By.ID, "confirm").click()

selenium.common.exceptions.ElementClickInterceptedException: Message: element click
intercepted: Element <button id="confirm">...</button> is not clickable at point (640, 400).
Other element would receive the click: <div class="modal-backdrop fade">...</div>

The modal is mid-fade — present, not yet in place. element_to_be_clickable is not always enough, because an element can be visible and enabled while still moving.

    wait.until(EC.element_to_be_clickable((By.ID, "confirm"))).click()

If that still flakes, wait for the animation to finish:

def animation_done(locator):
    def predicate(driver):
        el = driver.find_element(*locator)
        first = el.rect
        time.sleep(0.1)
        return first == el.rect and el.is_displayed()
    return predicate

    wait.until(animation_done((By.ID, "confirm")))
    driver.find_element(By.ID, "confirm").click()

confirmed

Or remove the variable entirely in the test environment:

driver.execute_script("""
  const style = document.createElement('style');
  style.textContent = '*, *::before, *::after { transition: none !important;
                        animation: none !important; }';
  document.head.appendChild(style);
""")

animations disabled for this page

Worth doing suite-wide. Animations are never what the test is about, and they are a large share of intermittent click failures.

Cause 4: tests that share state

def test_a_adds_todo(driver):
    page = TodoPage(driver).open().add("buy milk")
    assert page.titles == ["buy milk"]

def test_b_counts_todos(driver):
    page = TodoPage(driver).open()
    assert page.titles == []          # passes alone, fails after test_a

test_flaky.py::test_a_adds_todo PASSED                          [ 50%]
test_flaky.py::test_b_counts_todos FAILED                       [100%]

E   AssertionError: assert ['buy milk'] == []

TodoMVC persists to localStorage, so a shared driver carries state between tests. Run them in a different order and the failure moves — the classic signature of shared state.

A fresh driver per test fixes it, at the cost of startup time. Cheaper, if the driver is shared:

@pytest.fixture(autouse=True)
def clean_state(driver):
    yield
    driver.execute_script("window.localStorage.clear(); window.sessionStorage.clear();")
    driver.delete_all_cookies()

test_flaky.py::test_a_adds_todo PASSED                          [ 50%]
test_flaky.py::test_b_counts_todos PASSED                       [100%]

========================== 2 passed in 4.12s ==========================

Prove independence rather than assuming it:

pytest -p no:randomly test_flaky.py   # fixed order
pytest test_flaky.py --random-order   # shuffled, via pytest-random-order

Using --random-order-bucket=module
Using --random-order-seed=784201

test_flaky.py::test_b_counts_todos PASSED                       [ 50%]
test_flaky.py::test_a_adds_todo PASSED                          [100%]

Running the suite shuffled in CI turns “order-dependent” from a mystery into a failing build with a reproducible seed.

Cause 5: the environment

def test_shows_todays_date(driver):
    driver.get(APP)
    assert driver.find_element(By.ID, "today").text == date.today().isoformat()

Passes in London, fails on a CI runner in UTC-8 for four hours each evening. The same class of bug covers locale (£25.50 vs 25,50 €), viewport (a headless default of 800×600 hides elements behind a hamburger menu), and machine speed.

Pin all of it:

options = webdriver.ChromeOptions()
options.add_argument("--headless=new")
options.add_argument("--window-size=1440,900")
options.add_argument("--lang=en-GB")
options.add_experimental_option("prefs", {"intl.accept_languages": "en-GB,en"})

driver = webdriver.Chrome(options=options)
driver.execute_cdp_cmd("Emulation.setTimezoneOverride", {"timezoneId": "Europe/London"})
driver.execute_cdp_cmd("Emulation.setLocaleOverride", {"locale": "en-GB"})

timezone and locale pinned; viewport 1440x900

A test that depends on the environment should say so explicitly rather than inheriting it.

Capture evidence on failure

A CI-only failure is unfixable without artefacts. Fifteen lines gets you all of them:

# conftest.py
import pytest, pathlib, datetime

@pytest.hookimpl(hookwrapper=True, tryfirst=True)
def pytest_runtest_makereport(item, call):
    outcome = yield
    report = outcome.get_result()
    item.stash.setdefault("reports", {})[report.when] = report


@pytest.fixture
def driver(request):
    drv = webdriver.Chrome(options=opts)
    yield drv
    failed = any(r.failed for r in request.node.stash.get("reports", {}).values())
    if failed:
        out = pathlib.Path("artifacts") / request.node.name
        out.mkdir(parents=True, exist_ok=True)
        drv.save_screenshot(str(out / "screenshot.png"))
        (out / "page.html").write_text(drv.page_source, encoding="utf-8")
        (out / "url.txt").write_text(drv.current_url)
        try:
            logs = drv.get_log("browser")
            (out / "console.log").write_text("\n".join(f"{e['level']}: {e['message']}" for e in logs))
        except Exception:
            pass
        print(f"\nartifacts written to {out}")
    drv.quit()

test_todos.py::test_completes_a_todo FAILED

artifacts written to artifacts/test_completes_a_todo

E   selenium.common.exceptions.TimeoutException: Message: text never became
    '0 items left' in ('class name', 'todo-count')

cat artifacts/test_completes_a_todo/console.log

SEVERE: https://demo.playwright.dev/todomvc/main.js 412:19 Uncaught TypeError:
Cannot read properties of undefined (reading 'id')

The console log answers it — the application threw, so the counter never updated. The test was not flaky; the app was broken, and without the artefact this would have been filed as a wait problem and “fixed” with a longer timeout.

Upload the directory as a CI artifact and every failure comes with its own evidence.

Retries, and what they are for

pip install pytest-rerunfailures
pytest --reruns 2 --reruns-delay 1 -v

test_todos.py::test_adds_a_todo PASSED                          [ 33%]
test_todos.py::test_completes_a_todo RERUN                      [ 66%]
test_todos.py::test_completes_a_todo RERUN                      [ 66%]
test_todos.py::test_completes_a_todo PASSED                     [ 66%]
test_todos.py::test_filters_to_active PASSED                    [100%]

==================== 3 passed, 2 rerun in 12.44s =====================

Green, and test_completes_a_todo needed two attempts. That RERUN line is the value — treat it as a bug report, not a result.

Track the rate:

# conftest.py
def pytest_terminal_summary(terminalreporter):
    rerun = len(terminalreporter.stats.get("rerun", []))
    passed = len(terminalreporter.stats.get("passed", []))
    if rerun:
        print(f"\nFLAKE RATE: {rerun} rerun(s) across {passed} passing tests "
              f"({rerun / max(passed, 1):.1%})")

FLAKE RATE: 2 rerun(s) across 3 passing tests (66.7%)

Chart that number per run. A suite whose flake rate is climbing is degrading even while every build is green, and the day it crosses the retry count you get a “sudden” outbreak of failures that has actually been developing for months.

Quarantine rather than delete a test you cannot fix today:

@pytest.mark.flaky_quarantine
def test_intermittent_checkout(driver):
    ...

pytest -m "not flaky_quarantine"          # the blocking suite
pytest -m flaky_quarantine --reruns 3     # reported, not blocking

1 passed, 0 failed  (blocking suite)
1 failed             (quarantine — visible, not blocking the deploy)

Quarantine with an owner and a date. Without those it is deletion with extra steps.

Practice

selenium.common.exceptions.StaleElementReferenceException: Message: stale element
reference

E   AssertionError: assert ['buy milk', 'walk the dog'] == ['walk the dog']

Using --random-order-seed=784201
test_b_counts_todos PASSED
test_a_adds_todo PASSED

SEVERE: main.js 412:19 Uncaught TypeError: Cannot read properties of undefined

Next: pytest integration — fixtures, parametrisation, parallel runs and reports.