CI: run the unit suite on Python 3.14 per push, and on every supported Python weekly, on release branches and by hand

tiagoagueda commented

2026-10-04 20:43:11 +00:00

Owner

Measured from Forgejo's task API and job logs, runs 561–710 (24 Sep – 4 Oct 2026): wall time per push about 52–55 min, up from about 15; the browser suite grew from 149 to 645 tests in five days and runs serially (29 min); the runner takes two jobs at a time, in the order the workflow file lists them, so the browser job starts 19–22 min after the push; and Forgejo cancels the older push run when a newer push lands, which stopped 18 of 56 browser jobs before they gave a verdict.

Each unit leg takes about 12 minutes of one of the two slots. In the measured window the three legs failed only together, and only on real faults (catalogue checks, test_avatars, test_referrals): no failure was specific to an interpreter.

Decision (user, 2026-10-04)

A push to main and a pull request run the unit suite on 3.14 only, which is also the leg that measures coverage.
3.12 and 3.13 join on the weekly schedule, on release branches ([0-9]*) and on a manual run (workflow_dispatch, a new trigger).
The release recipe in CONTRIBUTING.md gains a step: run CI on main by hand, which runs every Python, and wait for it to pass before tagging.

The cost accepted: something that breaks only on 3.12 or 3.13 can sit on main until the next weekly run or release.

Done when

A push runs one unit leg; the schedule, a release branch and a manual run run three.

Measured from Forgejo's task API and job logs, runs 561–710 (24 Sep – 4 Oct 2026): wall time per push about 52–55 min, up from about 15; the browser suite grew from 149 to 645 tests in five days and runs serially (29 min); the runner takes **two jobs at a time, in the order the workflow file lists them**, so the browser job starts 19–22 min after the push; and Forgejo cancels the older push run when a newer push lands, which stopped 18 of 56 browser jobs before they gave a verdict. Each unit leg takes about 12 minutes of one of the two slots. In the measured window the three legs failed only together, and only on real faults (catalogue checks, `test_avatars`, `test_referrals`): no failure was specific to an interpreter. ## Decision (user, 2026-10-04) - A push to `main` and a pull request run the unit suite on **3.14 only**, which is also the leg that measures coverage. - 3.12 and 3.13 join on the weekly schedule, on release branches (`[0-9]*`) and on a manual run (`workflow_dispatch`, a new trigger). - The release recipe in `CONTRIBUTING.md` gains a step: run CI on `main` by hand, which runs every Python, and wait for it to pass before tagging. The cost accepted: something that breaks only on 3.12 or 3.13 can sit on `main` until the next weekly run or release. ## Done when - A push runs one unit leg; the schedule, a release branch and a manual run run three.

tiagoagueda added the

enhancement

label

2026-10-04 20:43:11 +00:00

tiagoagueda referenced this issue from a commit

2026-10-04 21:23:09 +00:00

Test the newest Python on every push, and the others weekly and before a release

tiagoagueda closed this issue

2026-10-04 21:23:09 +00:00

tiagoagueda commented

2026-10-04 21:23:55 +00:00

Author

Owner

Probed on a throwaway branch ci-probe (runs 715 and 716, deleted afterwards). The runner is forgejo-runner v6.4.0 against Forgejo 15:

a job-level if: that is false fails its job ("Early termination"), with or without needs:;
needs: works when the dependent job's if is true, across runners (ubuntu-latest → docker);
job-level timeout-minutes is ignored (a 1-minute job ran 2 m 30 s); step-level timeout-minutes works ("context deadline exceeded" at 1 m 0 s);
the server expands a matrix without any github context: fromJSON(github.ref == … && '[…]' || '[…]'), by ref, ref_name or event_name, always took the last branch.

Upgrading the runner may fix the first and third; that is a host change.

So the older Pythons run in a workflow of their own, every-python.yml (weekly, release branches, by hand), holding ci.yml's job word for word, which a test keeps equal; the release gate asks for a passed leg on every classifier Python from either workflow (6f8e46018).

Probed on a throwaway branch `ci-probe` (runs 715 and 716, deleted afterwards). The runner is forgejo-runner **v6.4.0** against Forgejo 15: - a job-level `if:` that is false **fails** its job ("Early termination"), with or without `needs:`; - `needs:` works when the dependent job's `if` is true, across runners (ubuntu-latest → docker); - job-level `timeout-minutes` is **ignored** (a 1-minute job ran 2 m 30 s); step-level `timeout-minutes` works ("context deadline exceeded" at 1 m 0 s); - the server expands a matrix **without any `github` context**: `fromJSON(github.ref == … && '[…]' || '[…]')`, by `ref`, `ref_name` or `event_name`, always took the last branch. Upgrading the runner may fix the first and third; that is a host change. So the older Pythons run in a workflow of their own, `every-python.yml` (weekly, release branches, by hand), holding `ci.yml`'s job word for word, which a test keeps equal; the release gate asks for a passed leg on every classifier Python from either workflow (`6f8e46018`).

tiagoagueda referenced this issue from Postulo/postulo-apprise

2026-10-20 11:04:47 +00:00

Test on Python 3.12 and 3.13 as well, weekly and on release branches #28

tiagoagueda referenced this issue

2026-10-22 01:06:16 +00:00

CI: make a push report sooner and cost less, without new hardware #1033

Rows
Columns

CI: run the unit suite on Python 3.14 per push, and on every supported Python weekly, on release branches and by hand #713

Decision (user, 2026-10-04)

Done when