CI: start the browser job first, check the tree once, and bound every job

tiagoagueda commented

2026-10-04 20:43:11 +00:00

Owner

Measured from Forgejo's task API and job logs, runs 561–710 (24 Sep – 4 Oct 2026): wall time per push about 52–55 min, up from about 15; the browser suite grew from 149 to 645 tests in five days and runs serially (29 min); the runner takes two jobs at a time, in the order the workflow file lists them, so the browser job starts 19–22 min after the push; and Forgejo cancels the older push run when a newer push lands, which stopped 18 of 56 browser jobs before they gave a verdict.

What is slow

ci.yml lists the jobs test (three legs), postgres, security, styles, browser. With two slots that puts the longest job last: in run 706 it started at 15:11, nineteen minutes after the push, and finished at 15:44.
Every unit leg repeats about 2.5 minutes of checks that do not depend on the interpreter: ruff check and format, makemigrations --check, messages.py extract --check, check, stats --check, and check --deploy + collectstatic with the production settings.
No job has a time limit of its own, and pytest has no faulthandler_timeout, so a hung test shows nothing until the runner's limit ends the job.
Nothing in the log says which tests take the time.

Proposal

browser first in the file, so it takes a slot at t=0; then the unit job, a new checks job, postgres, security. No needs: between them, which would serialise them.
checks runs the interpreter-independent steps once, and takes over the styles job's steps (both ask whether the committed tree is current).
The unit job keeps install, messages.py compile (test_select_options.py:373 reads a compiled catalogue), the ESCO download and pytest.
timeout-minutes on each job; faulthandler_timeout = 120 in [tool.pytest.ini_options]; --durations=25 on both pytest commands.

Done when

The browser job starts when the run starts, the tree checks run once per push, and both pytest logs end with their 25 slowest tests.

Measured from Forgejo's task API and job logs, runs 561–710 (24 Sep – 4 Oct 2026): wall time per push about 52–55 min, up from about 15; the browser suite grew from 149 to 645 tests in five days and runs serially (29 min); the runner takes **two jobs at a time, in the order the workflow file lists them**, so the browser job starts 19–22 min after the push; and Forgejo cancels the older push run when a newer push lands, which stopped 18 of 56 browser jobs before they gave a verdict. ## What is slow - `ci.yml` lists the jobs test (three legs), postgres, security, styles, browser. With two slots that puts the longest job last: in run 706 it started at 15:11, nineteen minutes after the push, and finished at 15:44. - Every unit leg repeats about 2.5 minutes of checks that do not depend on the interpreter: ruff check and format, `makemigrations --check`, `messages.py extract --check`, `check`, `stats --check`, and `check --deploy` + `collectstatic` with the production settings. - No job has a time limit of its own, and pytest has no `faulthandler_timeout`, so a hung test shows nothing until the runner's limit ends the job. - Nothing in the log says which tests take the time. ## Proposal - `browser` first in the file, so it takes a slot at t=0; then the unit job, a new `checks` job, `postgres`, `security`. No `needs:` between them, which would serialise them. - `checks` runs the interpreter-independent steps once, and takes over the `styles` job's steps (both ask whether the committed tree is current). - The unit job keeps install, `messages.py compile` (`test_select_options.py:373` reads a compiled catalogue), the ESCO download and pytest. - `timeout-minutes` on each job; `faulthandler_timeout = 120` in `[tool.pytest.ini_options]`; `--durations=25` on both pytest commands. ## Done when - The browser job starts when the run starts, the tree checks run once per push, and both pytest logs end with their 25 slowest tests.

tiagoagueda added the

enhancement

label

2026-10-04 20:43:11 +00:00

tiagoagueda referenced this issue from a commit

2026-10-04 21:23:08 +00:00

Start the browser job first, check the tree once, and bound every job

tiagoagueda closed this issue

2026-10-04 21:23:08 +00:00

tiagoagueda commented

2026-10-04 21:23:55 +00:00

Author

Owner

Probed on a throwaway branch ci-probe (runs 715 and 716, deleted afterwards). The runner is forgejo-runner v6.4.0 against Forgejo 15:

a job-level if: that is false fails its job ("Early termination"), with or without needs:;
needs: works when the dependent job's if is true, across runners (ubuntu-latest → docker);
job-level timeout-minutes is ignored (a 1-minute job ran 2 m 30 s); step-level timeout-minutes works ("context deadline exceeded" at 1 m 0 s);
the server expands a matrix without any github context: fromJSON(github.ref == … && '[…]' || '[…]'), by ref, ref_name or event_name, always took the last branch.

Upgrading the runner may fix the first and third; that is a host change.

So the time limits landed on the test steps, not the jobs (e0dc97fd1).

Probed on a throwaway branch `ci-probe` (runs 715 and 716, deleted afterwards). The runner is forgejo-runner **v6.4.0** against Forgejo 15: - a job-level `if:` that is false **fails** its job ("Early termination"), with or without `needs:`; - `needs:` works when the dependent job's `if` is true, across runners (ubuntu-latest → docker); - job-level `timeout-minutes` is **ignored** (a 1-minute job ran 2 m 30 s); step-level `timeout-minutes` works ("context deadline exceeded" at 1 m 0 s); - the server expands a matrix **without any `github` context**: `fromJSON(github.ref == … && '[…]' || '[…]')`, by `ref`, `ref_name` or `event_name`, always took the last branch. Upgrading the runner may fix the first and third; that is a host change. So the time limits landed on the test steps, not the jobs (`e0dc97fd1`).

Rows
Columns

CI: start the browser job first, check the tree once, and bound every job #712

What is slow

Proposal

Done when