Fast, Deterministic Pytest Suites

A suite people wait for is a suite people stop running. Two things make that happen: it is slow, and it fails for reasons that have nothing to do with the change.

Find out where the time goes

$ pytest --durations=8
========================= test session starts =========================
collected 312 items

................................................. [100%]

========================= slowest 8 durations ==========================
12.41s call     tests/test_import.py::test_imports_10k_rows
9.88s call      tests/test_reports.py::test_monthly_rollup
6.02s setup     tests/test_api.py::test_checkout
5.97s setup     tests/test_api.py::test_refund
5.94s setup     tests/test_api.py::test_cancel
0.31s call      tests/test_orders.py::test_bulk_discount
0.04s call      tests/test_shipping.py::test_rates
0.01s call      tests/test_orders.py::test_total
(304 durations < 0.005s hidden.  Use -vv to show these durations.)

========================= 312 passed in 47.31s =========================

Read the setup lines first. Three tests spending six seconds each in setup is one fixture being rebuilt per test — almost certainly a container or a schema that could be session-scoped:

# before: 6s per test
@pytest.fixture
def api_client():
    container = start_postgres()
    apply_migrations(container.dsn)
    yield Client(container.dsn)
    container.stop()

# after: 6s per session, plus a cheap reset
@pytest.fixture(scope="session")
def database():
    container = start_postgres()
    apply_migrations(container.dsn)
    yield container
    container.stop()


@pytest.fixture
def api_client(database):
    truncate_all_tables(database.dsn)
    return Client(database.dsn)

$ pytest --durations=3
6.11s setup     tests/test_api.py::test_checkout
0.09s setup     tests/test_api.py::test_refund
0.08s setup     tests/test_api.py::test_cancel

========================= 312 passed in 29.44s =========================

Eighteen seconds became six. The per-test reset is what keeps the tests independent — widen a fixture’s scope and you owe the suite a cheap way to undo whatever a test did to it.

Run it in parallel

pip install pytest-xdist
pytest -n auto

========================= test session starts =========================
platform linux -- Python 3.11.9, pytest-8.3.4, pluggy-1.5.0
plugins: cov-6.0.0, xdist-3.6.1
created: 8/8 workers
8 workers [312 items]

................................................. [100%]

========================= 312 passed in 8.12s =========================

-n auto uses one worker per CPU. Each worker is a separate process with its own imports and its own session fixtures — which is exactly where suites fall over:

$ pytest -n 4
[gw2] FAILED tests/test_reports.py::test_monthly_rollup
============================== FAILURES ===============================
>       assert report_path.read_text().startswith("month,total")
E       FileNotFoundError: [Errno 2] No such file or directory: '/tmp/report.csv'

Two tests were sharing /tmp/report.csv and passing only because they ran in order. The parallel run did not break them; it revealed that they were already broken. The fix is tmp_path, not -n 0.

When tests genuinely must share a resource, keep them on one worker:

@pytest.mark.xdist_group("payments-sandbox")
def test_capture():
    ...


@pytest.mark.xdist_group("payments-sandbox")
def test_refund():
    ...

pytest -n 4 --dist loadgroup

8 workers [312 items]
.................................................  [100%]
312 passed in 8.44s

--dist loadfile is the blunter version — everything in one file goes to the same worker.

For per-worker resources, xdist provides a worker_id fixture:

@pytest.fixture(scope="session")
def database(worker_id):
    name = "test_db" if worker_id == "master" else f"test_db_{worker_id}"
    dsn = create_database(name)
    yield dsn
    drop_database(name)

$ pytest -n 4 -q
....................................             [100%]
312 passed in 9.02s

Four workers, four databases, no collisions. worker_id is "master" when xdist is not active, so the same fixture works serially.

Expose order dependence on purpose

# test_cart.py
CART = []


def test_add_item():
    CART.append("A1")
    assert len(CART) == 1


def test_cart_starts_empty():
    assert CART == []

$ pytest -q
.F                                                                [100%]
>       assert CART == []
E       assert ['A1'] == []
1 failed, 1 passed in 0.02s

Obvious here, invisible when the two tests are in different files a thousand lines apart. pytest-randomly shuffles the order every run so the failure surfaces early:

pip install pytest-randomly
pytest

========================= test session starts =========================
plugins: randomly-3.15.0
Using --randomly-seed=1739284410

collected 312 items
................F................................ [ 42%]

The seed is printed so the failure is reproducible:

$ pytest -p randomly --randomly-seed=1739284410

Random ordering also reseeds random and faker before each test, which catches tests that quietly depend on a fixed sequence of “random” values. Turn it off with -p no:randomly when you are bisecting something else.

Rerunning and the cache

pytest remembers the last run in .pytest_cache:

pytest --lf          # last failed, only
pytest --ff          # failed first, then the rest
pytest --sw          # stepwise: stop at the first failure, resume there next time
pytest --cache-clear

$ pytest -q
.....FF...F                                                       [100%]
3 failed, 8 passed in 12.44s

$ pytest --lf -q
FFF                                                               [100%]
3 failed, 8 deselected in 0.31s

Twelve seconds became a third of a second. --sw is the one for a big refactor: it stops at the first failure, and the next run picks up from that test instead of replaying everything that already passed.

Flaky tests

pip install pytest-rerunfailures
pytest --reruns 2 --reruns-delay 1

$ pytest --reruns 2 -q
.R.R.                                                             [100%]
=================== short test summary info ===================
RERUN tests/test_api.py::test_checkout - requests.exceptions.ReadTimeout
5 passed, 2 rerun in 14.02s

Green, and worse than red. The R marks a test that failed and passed on a retry — which is what a race condition looks like from the outside, and what a genuine bug in production looks like too.

Use reruns as a temporary patch with an issue number attached, scoped to the tests that need it:

@pytest.mark.flaky(reruns=3, reruns_delay=2)
def test_eventual_consistency_of_search_index():
    ...

Then fix the cause. Most flakes are one of four things: a sleep standing in for a wait condition, a shared fixture two tests mutate, a test asserting on the ordering of a set or dict, or a real race in the code being tested. Only the last one is hard.

pytest-timeout stops a hang from eating a CI job:

pytest --timeout=60

tests/test_api.py::test_checkout FAILED
E       Failed: Timeout >60.0s

Wiring it into CI

[tool.pytest.ini_options]
addopts = "-ra --strict-markers --strict-config"
testpaths = ["tests"]
xfail_strict = true

# .github/workflows/test.yml
- run: pip install -e ".[test]"
- run: pytest -n auto --junitxml=results.xml --cov=shop --cov-report=xml
- uses: actions/upload-artifact@v4
  if: always()
  with:
    name: test-results
    path: results.xml

$ pytest -n auto --junitxml=results.xml -q
8 workers [312 items]
.................................................  [100%]

- generated xml file: /home/runner/work/shop/results.xml -
312 passed in 8.12s

if: always() matters — the upload step is only useful on the runs that failed.

The exit codes are what the pipeline actually reads:

Code 5 is the one to guard against. A mistyped path collects nothing:

$ pytest test/ -q
ERROR: file or directory not found: test/
$ echo $?
4

…and a testpaths pointing at an empty directory is quieter still:

$ pytest -q
no tests ran in 0.01s
$ echo $?
5

A pipeline step that only checks for a non-zero exit treats neither as success, but a || true or a script that greps for “failed” will. Assert the count you expect if the suite is critical:

pytest --collect-only -q | tail -1

312 tests collected in 0.44s

A working configuration

[tool.pytest.ini_options]
minversion = "8.0"
testpaths = ["tests"]
addopts = "-ra --strict-markers --strict-config --durations=10"
markers = [
    "slow: takes more than a second",
    "integration: needs a live service",
]
xfail_strict = true
filterwarnings = ["error::DeprecationWarning"]

[tool.coverage.run]
branch = true
source = ["shop"]

[tool.coverage.report]
show_missing = true
skip_covered = true
fail_under = 85

$ pytest -n auto
========================= test session starts =========================
configfile: pyproject.toml
testpaths: tests
plugins: cov-6.0.0, randomly-3.15.0, xdist-3.6.1
8 workers [312 items]
................................................. [100%]

========================= slowest 10 durations ========================
6.11s setup     tests/test_api.py::test_checkout
...
---------- coverage: platform linux, python 3.11.9-final-0 -----------
TOTAL                    842     71    92%

========================= 312 passed in 9.14s =========================

Local runs and CI runs now differ in exactly one flag, which is the property that stops “works on my machine” arguments before they start.

Practice

=========================== slowest 10 durations ===========================
6.02s setup     tests/test_api.py::test_checkout
5.97s setup     tests/test_api.py::test_refund
0.31s call      tests/test_orders.py::test_bulk_discount

[gw2] FAILED tests/test_reports.py::test_monthly_rollup
E       FileNotFoundError: '/tmp/report.csv'

Using --randomly-seed=1739284410 ... 312 passed
Using --randomly-seed=884120391  ... 311 passed, 1 failed
Using --randomly-seed=2049117760 ... 312 passed

no tests ran in 0.01s
$ echo $?
5

That completes the pytest track: assertions, fixtures, parametrization, marks, configuration, mocking, side effects, coverage, plugins, and the operational side. Next in the testing tier is Playwright, where the same fixture and marker ideas drive a browser.