CI: keep browser traces for failures only, by re-running the failed tests with tracing on

tiagoagueda commented

2026-10-04 20:43:12 +00:00

Owner

Measured from Forgejo's task API and job logs, runs 561–710 (24 Sep – 4 Oct 2026): wall time per push about 52–55 min, up from about 15; the browser suite grew from 149 to 645 tests in five days and runs serially (29 min); the runner takes two jobs at a time, in the order the workflow file lists them, so the browser job starts 19–22 min after the push; and Forgejo cancels the older push run when a newer push lands, which stopped 18 of 56 browser jobs before they gave a verdict.

The browser job runs with --tracing retain-on-failure --screenshot only-on-failure. In pytest-playwright 0.9 those options do their work on every test: each context records a trace with screenshots, DOM snapshots and sources, and each page is photographed when its context closes; on a pass both files are written and then deleted (pytest_playwright.py:679-713). That is paid 645 times to keep the files of the rare failure.

Decision (user, 2026-10-04): diagnose, never retry

The main run records nothing.
When it fails, a following step re-runs only the failed tests, alone and serially, with tracing and screenshots on: pytest -m e2e --browser chromium --lf --lfnf=none -p no:xdist --tracing on --screenshot on --output browser-artifacts. Its artifact is uploaded as before (upload-artifact@v3).
The job stays red whatever the re-run does. A re-run that passes says the failure was a flake or depends on test order, and the trace of the re-run is there to look at; it does not turn the job green.
The step runs only when the test step itself failed (steps.e2e.outcome == 'failure'), not when an earlier step did, and repeats -m e2e (the default -m deselects every browser test).

Done when

A passing run writes no trace or screenshot, and a failing run uploads the trace of the failed tests' re-run.

Measured from Forgejo's task API and job logs, runs 561–710 (24 Sep – 4 Oct 2026): wall time per push about 52–55 min, up from about 15; the browser suite grew from 149 to 645 tests in five days and runs serially (29 min); the runner takes **two jobs at a time, in the order the workflow file lists them**, so the browser job starts 19–22 min after the push; and Forgejo cancels the older push run when a newer push lands, which stopped 18 of 56 browser jobs before they gave a verdict. The browser job runs with `--tracing retain-on-failure --screenshot only-on-failure`. In pytest-playwright 0.9 those options do their work on **every** test: each context records a trace with screenshots, DOM snapshots and sources, and each page is photographed when its context closes; on a pass both files are written and then deleted (`pytest_playwright.py:679-713`). That is paid 645 times to keep the files of the rare failure. ## Decision (user, 2026-10-04): diagnose, never retry - The main run records nothing. - When it fails, a following step re-runs only the failed tests, alone and serially, with tracing and screenshots on: `pytest -m e2e --browser chromium --lf --lfnf=none -p no:xdist --tracing on --screenshot on --output browser-artifacts`. Its artifact is uploaded as before (`upload-artifact@v3`). - The job stays red whatever the re-run does. A re-run that passes says the failure was a flake or depends on test order, and the trace of the re-run is there to look at; it does not turn the job green. - The step runs only when the test step itself failed (`steps.e2e.outcome == 'failure'`), not when an earlier step did, and repeats `-m e2e` (the default `-m` deselects every browser test). ## Done when - A passing run writes no trace or screenshot, and a failing run uploads the trace of the failed tests' re-run.

tiagoagueda added the

enhancement

label

2026-10-04 20:43:12 +00:00

tiagoagueda referenced this issue from a commit

2026-10-04 21:23:09 +00:00

Trace browser failures by running them again alone, not every test

tiagoagueda closed this issue

2026-10-04 21:23:09 +00:00

tiagoagueda referenced this issue from a commit

2026-10-04 22:21:24 +00:00

Run the browser tests on two processes, the walks of every page first