Postulo/postulo

Fork 0

CI and operations tooling: 5,800 tests run serially three times, unpinned tools, releases not gated on CI, no config checks or request ids #233

New issue

Closed

opened 2026-09-15 21:23:29 +00:00 by tiagoagueda · 2 comments

tiagoagueda commented

2026-09-15 21:23:29 +00:00

Owner

Found in the 2026-09-15 code audit.

CI speed

5,802 tests run serially with no pytest-xdist (pyproject.toml:56-64), three times, each with --cov (.forgejo/workflows/ci.yml:42,89).
Every matrix leg's uv sync --locked also installs the e2e group (Playwright, about 136 MB), because default-groups includes it.
There is no cache for uv or Playwright.
Measured locally: database setup (82 migrations) takes about 8.5 s per session. tests/security/test_secret_key.py spawns a fresh interpreter per case, at 0.7–2.2 s each. The plugins page tests take 3–9 s each (see the query-performance issue (#231)).

Proposal:

pytest-xdist -n auto, after auditing module-level caches such as registry._cache.
Coverage on one leg only, with a fail_under (none exists today).
uv sync --no-group e2e in the test job, plus uv and Playwright caches.
Batch the secret-key probes into one subprocess.

Supply chain

uv is installed from https://astral.sh/uv/install.sh, always the latest version, in four places (ci.yml:62,124,227, release.yml:43).
uvx zizmor and uv run --with pip-audit take whatever is newest.
The images copy from ghcr.io/astral-sh/uv:0.12, a floating tag.
actions/checkout@v4 and upload-artifact@v3/v4 are mutable tags.
This contrasts with the lock discipline everywhere else; pre-commit pins uv-pre-commit 0.12.5.

Proposal:

Install uv from the versioned installer (or the pinned binary with a checksum), matching pre-commit.
Pin zizmor and pip-audit.
Pin actions by SHA, or record the Forgejo resolution rationale in zizmor.yml.
Base images stay unpinned by digest, as Dockerfile:151-162 argues.

Releases

release.yml publishes the sdist and wheel on any v* tag without checking the commit's CI, and image.yml pushes :X.Y, :X.Y.Z and :latest for any tag typed into the dispatch box. scripts/release_tools.py check validates only the version and the changelog.

Proposal: have release_tools.py check query the Forgejo commit-status API for the tagged SHA (success on test* and browser), and call it from image.yml too.

Dependency updates

CONTRIBUTING.md says "once a month… uv lock --upgrade", and nothing automates it; the weekly pip-audit reports vulnerabilities, not staleness.

Proposal: a self-hosted Renovate run (it supports Forgejo and uv.lock) grouped monthly, or a scheduled workflow that upgrades the lock and opens a PR when tests pass.

Configuration checks and logs

No Django system checks are registered anywhere. POSTULO_EMAIL_SECURITY, POSTULO_EMAIL_AUTH, POSTULO_PDF_BACKEND and the POSTULO_*_RATE strings are read raw, so check --deploy in the entrypoint cannot catch a typo; it surfaces at the first send, render or throttle.
The console log uses the simple format, and JSON goes only to the rotating file. No request id exists, so a gunicorn access line cannot be matched to an application log line or a scheduler run.

Proposal:

A postulo.core.checks module checking enum membership, rate syntax and the POSTULO_PUBLIC_URL scheme at Error level.
POSTULO_LOG_FORMAT=json|simple for the console.
Middleware that honours or creates X-Request-ID, adds it to log records and echoes it in the response.
The same id in gunicorn's access-log format.

Found in the 2026-09-15 code audit. ## CI speed - 5,802 tests run serially with no `pytest-xdist` (`pyproject.toml:56-64`), three times, each with `--cov` (`.forgejo/workflows/ci.yml:42,89`). - Every matrix leg's `uv sync --locked` also installs the `e2e` group (Playwright, about 136 MB), because `default-groups` includes it. - There is no cache for uv or Playwright. - Measured locally: database setup (82 migrations) takes about 8.5 s per session. `tests/security/test_secret_key.py` spawns a fresh interpreter per case, at 0.7–2.2 s each. The plugins page tests take 3–9 s each (see the query-performance issue (#231)). **Proposal:** - `pytest-xdist -n auto`, after auditing module-level caches such as `registry._cache`. - Coverage on one leg only, with a `fail_under` (none exists today). - `uv sync --no-group e2e` in the test job, plus uv and Playwright caches. - Batch the secret-key probes into one subprocess. ## Supply chain - uv is installed from `https://astral.sh/uv/install.sh`, always the latest version, in four places (`ci.yml:62,124,227`, `release.yml:43`). - `uvx zizmor` and `uv run --with pip-audit` take whatever is newest. - The images copy from `ghcr.io/astral-sh/uv:0.12`, a floating tag. - `actions/checkout@v4` and `upload-artifact@v3/v4` are mutable tags. - This contrasts with the lock discipline everywhere else; pre-commit pins `uv-pre-commit 0.12.5`. **Proposal:** - Install uv from the versioned installer (or the pinned binary with a checksum), matching pre-commit. - Pin zizmor and pip-audit. - Pin actions by SHA, or record the Forgejo resolution rationale in `zizmor.yml`. - Base images stay unpinned by digest, as `Dockerfile:151-162` argues. ## Releases `release.yml` publishes the sdist and wheel on any `v*` tag without checking the commit's CI, and `image.yml` pushes `:X.Y`, `:X.Y.Z` and `:latest` for any tag typed into the dispatch box. `scripts/release_tools.py check` validates only the version and the changelog. **Proposal:** have `release_tools.py check` query the Forgejo commit-status API for the tagged SHA (success on `test*` and `browser`), and call it from `image.yml` too. ## Dependency updates `CONTRIBUTING.md` says "once a month… `uv lock --upgrade`", and nothing automates it; the weekly `pip-audit` reports vulnerabilities, not staleness. **Proposal:** a self-hosted Renovate run (it supports Forgejo and `uv.lock`) grouped monthly, or a scheduled workflow that upgrades the lock and opens a PR when tests pass. ## Configuration checks and logs - No Django system checks are registered anywhere. `POSTULO_EMAIL_SECURITY`, `POSTULO_EMAIL_AUTH`, `POSTULO_PDF_BACKEND` and the `POSTULO_*_RATE` strings are read raw, so `check --deploy` in the entrypoint cannot catch a typo; it surfaces at the first send, render or throttle. - The console log uses the `simple` format, and JSON goes only to the rotating file. No request id exists, so a gunicorn access line cannot be matched to an application log line or a scheduler run. **Proposal:** - A `postulo.core.checks` module checking enum membership, rate syntax and the `POSTULO_PUBLIC_URL` scheme at `Error` level. - `POSTULO_LOG_FORMAT=json|simple` for the console. - Middleware that honours or creates `X-Request-ID`, adds it to log records and echoes it in the response. - The same id in gunicorn's access-log format.

tiagoagueda added the

enhancement

observability

labels

2026-09-15 21:23:29 +00:00

tiagoagueda added this to the 0.5.0 milestone

2026-09-15 21:33:30 +00:00

tiagoagueda commented

2026-09-16 13:48:58 +00:00

Author

Owner

From the CodeQL run of 2026-09-16 (#248, #249), one thing for this issue rather than a
separate one: nothing in CI runs CodeQL.

The first run over main at 23c742c7f produced 131 results and no true-positive security
finding, so the value is not in what it caught — it is that the guards it could not see
(http.public_only_client for #215, %r in the plugin logger, the username pattern) have
nothing watching them for regressions. A future commit that routes a fetch around
public_only_client, or swaps a %r for a %s, would light up a rule that is currently
green and unobserved.

Cost, measured on this machine: database build about 4 minutes for 354 files, the
python-security-and-quality suite a few minutes more on --threads=0. That is too slow
for every push and about right for a nightly or a pre-release job.

If it is wired up, the triage in #249 has to go with it or the job is 131 results of noise
on day one — the ...-in-Protocol and deferred-import rules alone are 97 of them. The
suite takes a filter file, so py/ineffectual-statement and py/cyclic-import can be
excluded there while #248 is open.

From the CodeQL run of 2026-09-16 (#248, #249), one thing for this issue rather than a separate one: **nothing in CI runs CodeQL.** The first run over `main` at `23c742c7f` produced 131 results and no true-positive security finding, so the value is not in what it caught — it is that the guards it could not see (`http.public_only_client` for #215, `%r` in the plugin logger, the username pattern) have nothing watching them for regressions. A future commit that routes a fetch around `public_only_client`, or swaps a `%r` for a `%s`, would light up a rule that is currently green and unobserved. Cost, measured on this machine: database build about 4 minutes for 354 files, the `python-security-and-quality` suite a few minutes more on `--threads=0`. That is too slow for every push and about right for a nightly or a pre-release job. If it is wired up, the triage in #249 has to go with it or the job is 131 results of noise on day one — the `...`-in-`Protocol` and deferred-import rules alone are 97 of them. The suite takes a filter file, so `py/ineffectual-statement` and `py/cyclic-import` can be excluded there while #248 is open.

tiagoagueda referenced this issue

2026-09-16 13:58:54 +00:00

CodeQL: a small cluster of real quality findings, and the guards it cannot see #249

tiagoagueda commented

2026-09-17 17:54:01 +00:00

Author

Owner

A template formatter belongs in this list

The tooling audit above covers Python, CI and the supply chain. The templates have none of
it: 179 files and 11,375 lines, checked for correctness by tests/test_template_lint.py
(multiline {# #}, physical sides) and formatted by hand.

djade — 1.9.0, written in Rust, formats Django template syntax to a style derived
from Django's own contribution guidelines. It reads the Django version from
pyproject.toml, runs as a pre-commit hook beside the ones already configured, and is fast
enough not to be noticed (377 templates in about 20 ms). It also carries fixers for
deprecated template tags and filters, which is the part that earns its place at the next
Django bump rather than this one.

Worth pairing with django-upgrade, the same author's codemod for Django idioms in
Python. Low value while pinned to django>=6.1,<6.2; genuinely useful at the version after
that, so it may be better added when the pin moves rather than now.

Both are development-only and neither touches what ships. Filed here rather than as their
own issue because this is where the tooling discussion already lives.

One note if djade is adopted: it reformats, so the first run will touch many templates at
once, and a reformat moves the #: source references in the catalogues without changing a
single string. Run it as its own commit, before any template work, and confirm with
git diff -U0 -- src/postulo/locale | grep '^[-+]msg' that nothing moved.

## A template formatter belongs in this list The tooling audit above covers Python, CI and the supply chain. The templates have none of it: 179 files and 11,375 lines, checked for correctness by `tests/test_template_lint.py` (multiline `{# #}`, physical sides) and formatted by hand. **`djade`** — 1.9.0, written in Rust, formats Django template syntax to a style derived from Django's own contribution guidelines. It reads the Django version from `pyproject.toml`, runs as a pre-commit hook beside the ones already configured, and is fast enough not to be noticed (377 templates in about 20 ms). It also carries fixers for deprecated template tags and filters, which is the part that earns its place at the next Django bump rather than this one. Worth pairing with **`django-upgrade`**, the same author's codemod for Django idioms in Python. Low value while pinned to `django>=6.1,<6.2`; genuinely useful at the version after that, so it may be better added when the pin moves rather than now. Both are development-only and neither touches what ships. Filed here rather than as their own issue because this is where the tooling discussion already lives. One note if `djade` is adopted: it reformats, so the first run will touch many templates at once, and a reformat moves the `#:` source references in the catalogues without changing a single string. Run it as its own commit, before any template work, and confirm with `git diff -U0 -- src/postulo/locale | grep '^[-+]msg'` that nothing moved.

tiagoagueda referenced this issue from a commit

2026-09-22 13:51:18 +00:00

Check the configuration at start-up, name the request in every log line, and gate releases on CI

tiagoagueda closed this issue

2026-09-22 13:51:18 +00:00

tiagoagueda referenced this issue from a commit

2026-09-22 16:58:48 +00:00

Run the suite on two processes in CI, not one per host core