Donate to Postulo/postulo

• https://buymeacoffee.com/tiagoagueda

---

See the documentation for more details on donation links.

0.6.0 fixes, bunch 8: tooling and tests #1230

Merged

tiagoagueda merged 10 commits from b8-tooling into main 2026-11-11 21:10:54 +00:00

Conversation 0 Commits 10 Files changed 14 +2443 -166

tiagoagueda commented 2026-11-11 20:36:50 +00:00

Owner

• Copy link
• Copy Markdown

Bunch 8 of the 0.6.0 fixes: tooling and tests. One signed commit per issue or item, then the changelog lines; merge fast-forward-only.

• Closes #1146: CONTRIBUTING.md gains "Two issues at once", the worktree recipe: the commands, what a new working tree lacks (virtual environment, compiled catalogues, .env, node_modules, the datasets), and the shared stash and one-heavy-suite-at-a-time warnings. A section only, as decided.
• Closes #1144: postulo.plugins.testing.cached_django_db_setup() keeps the migrated test database, keyed by every migration file, what they import and the versions involved, and restores it with SQLite's backup API; a second local run of one test file starts in about a second where migrating took 88 s or more. Off on CI (core CI migrates as before), on locally; POSTULO_TEST_DB_CACHE forces it either way or names the directory. Plugin repositories adopt it with their next repin.
• Closes #1169: extract finds the catalogue sets once and prunes its walks (extract --check from 83-103 s to 29-39 s locally, output byte-identical), and writes only the catalogues whose text changed.
• Closes #1145: compile parses each catalogue once and packs a .mo file's tables in one go (from 43-56 s to 27 s); all 2,352 .mo files and status.json are byte-identical to before.
• Refs #1033 (item 8, test-only, four commits): the status-report test proven on a small project (55 s to 0.1 s); the two migration reversal tests share one round trip; the plugin install tests read the environment's package metadata once (438 s to 184 s locally); the template lints run as one test per template (7,061 items to 1,832, the same 5,559 lint and template pairs checked).
• A longer wait for the candidate file's review in its browser tests: main's push run for bunch 7 (run 1904) failed on it under host load, and passed alone a minute later.

Tested locally: lint, migration and catalogue checks (the new extract wrote nothing: 0 written, 2,352 unchanged), and the full unit suite (16,337 passed, 167 skipped: fewer items now that the template lints are one test per template; 9 min 12 s, against about 14 before).

#1033 stays open: items 4 and 5 (the image) and item 9 (the migration count) remain.

🤖 Generated with Claude Code

Bunch 8 of the 0.6.0 fixes: tooling and tests. One signed commit per issue or item, then the changelog lines; merge fast-forward-only. - Closes #1146: CONTRIBUTING.md gains "Two issues at once", the worktree recipe: the commands, what a new working tree lacks (virtual environment, compiled catalogues, `.env`, `node_modules`, the datasets), and the shared stash and one-heavy-suite-at-a-time warnings. A section only, as decided. - Closes #1144: `postulo.plugins.testing.cached_django_db_setup()` keeps the migrated test database, keyed by every migration file, what they import and the versions involved, and restores it with SQLite's backup API; a second local run of one test file starts in about a second where migrating took 88 s or more. Off on CI (core CI migrates as before), on locally; `POSTULO_TEST_DB_CACHE` forces it either way or names the directory. Plugin repositories adopt it with their next repin. - Closes #1169: `extract` finds the catalogue sets once and prunes its walks (`extract --check` from 83-103 s to 29-39 s locally, output byte-identical), and writes only the catalogues whose text changed. - Closes #1145: `compile` parses each catalogue once and packs a `.mo` file's tables in one go (from 43-56 s to 27 s); all 2,352 `.mo` files and `status.json` are byte-identical to before. - Refs #1033 (item 8, test-only, four commits): the status-report test proven on a small project (55 s to 0.1 s); the two migration reversal tests share one round trip; the plugin install tests read the environment's package metadata once (438 s to 184 s locally); the template lints run as one test per template (7,061 items to 1,832, the same 5,559 lint and template pairs checked). - A longer wait for the candidate file's review in its browser tests: `main`'s push run for bunch 7 (run 1904) failed on it under host load, and passed alone a minute later. Tested locally: lint, migration and catalogue checks (the new `extract` wrote nothing: 0 written, 2,352 unchanged), and the full unit suite (16,337 passed, 167 skipped: fewer items now that the template lints are one test per template; 9 min 12 s, against about 14 before). #1033 stays open: items 4 and 5 (the image) and item 9 (the migration count) remain. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

tiagoagueda added 10 commits 2026-11-11 20:36:51 +00:00

Say how to work on a second issue beside the first ... 4a677919c2

A branch per issue is the rule (#940), and with one checkout the second
issue meant a stash and a branch switch while the first waited half an
hour for CI, leaving the compiled catalogues and the built files belonging
to the wrong branch. A git worktree avoids that, and nothing said how.

CONTRIBUTING.md gains "Two issues at once" under "How a change reaches
main": the commands, what a new working tree lacks because git does not
track it (the virtual environment, the compiled catalogues, .env,
node_modules, the downloaded datasets, and how to point at the first
checkout's copy of those), and the three things that bite: a branch lives
in one working tree, the stash is shared by all of them, and two heavy
suites at once starve each other. A section only, no script, as decided.

Closes #1146

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

Give the candidate file's review longer to appear in the browser tests ... de73a49364

main's own run for the previous merge failed on one browser test of 956:
the review of an uploaded candidate file, at 320 pixels in German, was not
on the page within the five seconds an expectation waits by default. The
call took fifteen seconds on a host whose swap was full, the page had drawn
its header and nothing below it, and the same test passed alone in the
job's own second attempt, as it had on the pull request for the same
commit. The page was slow, not wrong: its answer is the first one drawn in
the language the test has just chosen, so it reads that catalogue too.

The helper every test of this page goes through now waits twenty seconds
for the review. What the tests assert about it is unchanged.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

Keep the migrated test database between runs ... a35068c758

Every test run began by migrating an empty in-memory SQLite database: 292
migrations, about a minute before the first test of a single file on a
contributor's machine, and the same again in every plugin repository, whose
suite migrates the whole of Postulo to test a handful of functions. In memory
there was nothing for pytest-django's --reuse-db to keep.

The first run now migrates as before and copies the database into a file in
the person's own cache directory; a later run copies it back into its empty
in-memory database instead of migrating. Both copies are SQLite's own backup.
The file is named by a hash of what migrate reads: every migration file of
every installed app, the modules they import, the settings they name, the
tables of an app without migrations and the versions installed, so a change
to any of them migrates afresh. It is written under a temporary name and
renamed, checked for its own length before it is used, restored only into an
empty database, and the workers of one xdist run take turns so that one
migrates and the others restore. The six used last are kept.

The fixture is published as postulo.plugins.testing.cached_django_db_setup,
so a plugin repository uses the same one in two lines. It stands around
pytest-django's own fixture, which still does everything else.

On by default on a person's machine and off wherever CI is set, so core CI
migrates in every process exactly as it did until #1033 decides otherwise.
POSTULO_TEST_DB_CACHE switches it on or off or names the directory. Another
engine, the browser suite's file-backed database and --no-migrations are
migrated as before.

Closes #1144

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

Find the catalogue sets once per extract, prune the walks, write only what changed ... df94754acd

`extract --check` looked for the catalogue sets 29 times: once for the command and
once more inside `sources` for each of the 28 sets. Each search walked every
directory of the package, 4,955 of them, of which 4,704 are the `locale/` trees it
then discarded, and the walk for each set's sources entered them too. On the audit's
profile that was a third of the command's 110 seconds.

`cmd_extract` now finds the sets once and passes the list to `extract_all` and
`sources`, which take it as an optional second argument (a caller without one still
gets the answer, from one search). Both walks are `os.walk` with the skipped
directories (`migrations`, `static`, `locale`, `__pycache__`) and, for the sources,
the other sets removed from the list before they are entered. `core_set` is the
project's own set and no longer walks to say so. No memo: nothing to invalidate
when `use()` points the tool at another project.

`extract` also wrote all 2,352 catalogues whether or not one had changed, 73 MB
and 2,352 new modification times for git, editors and file watchers to react to.
It now compares the text it would write with the text it read to parse, the
comparison `--check` already made, writes only where they differ, and prints how
many it wrote and how many it left.

What the commands find and print is unchanged: `extract --check` prints the same
bytes before and after, and the tests compare the sets and every set's sources
with the old walks, in a temporary project and in Postulo itself. On a Windows
laptop, run back to back twice, `extract --check` took 83 and 103 seconds before
and 29 and 39 after.

Closes #1169

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

Parse each catalogue once in a compile and pack a .mo file's tables in one go ... 50d681b25b

`compile` parsed every catalogue to write its .mo file, and then, for Postulo
itself, `build_report` read and parsed all 2,352 of them again from the disk to
write `locale/status.json`: 4,704 parses for one command, and the second half was
about ten of its 27 seconds on the audit's measurement.

`build_report` now takes the sets and a `read` callable that gives the catalogue
at a path (both optional: `stats` passes neither and reads the disk as before).
`cmd_compile` passes its own, which parses a catalogue, writes its .mo and hands
the parsed catalogue over, so the report is built from what the compile parsed.
The report asks a language at a time, so one language's catalogues are held while
its row is counted. Keeping all of them until the end, the plainer way to do it,
held 450 MB where a compile held 50; this holds 53. Whatever the report did not
ask for (a plugin's catalogues, where there is no report, and a directory that is
no language offered) is compiled after it, in the order it was before.

`compile_catalogue` built the two tables of a .mo file by packing each (length,
offset) pair and adding it to the bytes written so far, which copied all of them
each time: twice per entry, 9,684 times for a full core catalogue. The pairs go
into one list, packed once, and the file is joined once.

Every .mo file is byte for byte what it was: all 2,352 of the real tree and
`status.json` hash the same from the old code and the new. On a Windows laptop,
run back to back twice, a compile took 56 and 43 seconds before and 27 and 27
after; `compile_catalogue` alone over every catalogue, 6.0 seconds before and 3.9
after.

Closes #1145

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

Prove the status report on a small project, not on every catalogue of Postulo ... 790ad5b019

`test_compile_writes_the_status_report_the_picker_reads` ran `compile` on
Postulo's own tree with the writes turned into no-ops, and then built the report
again to compare: every one of the 2,352 catalogues parsed three times over
before #1145 and twice after it, 23 seconds of each CI run for an answer that
does not depend on how many catalogues there are.

It now compiles a temporary project that calls itself Postulo, with two sets
that share a string and one language given translations, and reads what the
compile really wrote: `locale/status.json` beside the catalogues, byte for byte
the report `stats` builds from the disk, and the line that says so. Nothing is
patched, so the file, its encoding and its line endings are the tool's own.

Refs #1033

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

Go back and forward once for both migration reversal tests ... 2e3e2f5d65

Each of the two tests in tests/test_migration_reversal.py undid the
migrations from the latest down to where it starts and applied them
again in its teardown, most of the same 82 migrations each time: on CI
21.9 s + 10.1 s teardown and 12.7 s + 10.1 s. A module fixture now goes
back once to where both start and forward once after the last of them,
over every row the tests wrote, then empties the tables as a
transactional test does; the per-test flush is held back while the file
runs, since it would come before the way forward and on an old schema.

The tests keep their transactional database marker: without it a run of
this file alone has no test database set up by pytest-django, and the
fixture would have migrated whatever the connection pointed at. The
file is one xdist group, so `--dist loadgroup` makes the round trip once.

Refs #1033

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Read the environment's package metadata once in the plugin install tests ... b1d07ea6f6

tests/test_plugin_install.py and tests/test_plugin_rollback.py build no
environment and resolve nothing: a wheel is a few members written with
zipfile, the installer is replaced by unpacking it, and that takes
milliseconds. Their time went on reading the running environment again
and again: one install_wheel asks which versions Postulo runs on, which
parses the METADATA of every installed distribution, and then rebuilds
the registry, which asks every distribution for its entry points once for
each kind of plugin. Those files never change while the suite runs.

A fixture in tests/plugin_install_support.py, autouse in both files,
remembers what importlib.metadata reads from a distribution that sits in
a directory on sys.path before any test ran, keyed by the file's path,
size and modification time; a plugins directory, which the tests write
and break on purpose, is read from the disk every time. The cache lives
in the worker's memory, so nothing is shared between xdist workers or
left on disk. The import check's subprocess and every assertion run as
before; three tests pin that the plugin's own files are read each time
and that a replaced file is read again.

Refs #1033

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Run every template lint in one test per template ... 946968f659

tests/test_template_lint.py had seventeen lints, each parametrised over
every template: 5,559 of the suite's items, collected by the controller
and again by every worker, for checks that take a millisecond each. Each
lint is now a plain function registered with @template_lint, and
test_a_template_passes_every_lint runs all of them over one template.
Every lint runs whatever the one before found, and a failure names each
failing lint, where it is written and its own message with the lines;
a lint that breaks is shown with its traceback. A lint's own skip rule
still decides where it applies, and the lints not applied to a template
are named with their reason in the report's properties and under a
failure. The tests that read no single template are unchanged.

The same 5,559 (lint, template) pairs run as before, 5,537 passing and
the same 22 not applied; the file goes from 7,061 items to 1,832.

Refs #1033

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Record bunch 8 in the changelog ... f421402bd0

One line for each of the four issues the bunch closes: the kept test
database, the catalogue tool's extract and compile, and the worktree
recipe in CONTRIBUTING.md. The test-only commits for #1033's item 8 and
the longer wait in the candidate file's browser tests need none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

tiagoagueda merged commit f421402bd0 into main 2026-11-11 21:10:54 +00:00

tiagoagueda deleted branch b8-tooling 2026-11-11 21:10:54 +00:00

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

accessibility

Usable by everyone, including people with disabilities: one of the project's stated missions

  

authentication

Signing in, and proving who you are

  

breaking change

Changes behaviour or data in a way operators must act on

  

bug

Something does not do what it says it does

  

documentation

The wiki, the docs, or the words in the interface

  

enhancement

An improvement or a new capability

  

interface

The shape of the pages: what is where, and how it reads

  

internationalisation

Languages, regions, and the formats and symbols that go with them

  

observability

What an operator can see about a running instance: metrics, logs, health

  

performance

Uses fewer resources, or is more reliable, with no loss of features or accessibility

  

security

Keeping the code and the data secure: one of the project's stated missions

  

tier

1

Easy: small, self-contained, no prerequisites

  

tier

2

High-severity security: account access, personal data, the boundaries

  

tier

3

Foundations: prerequisite work that unblocks other issues

  

tier/4

Later: large features and long-horizon work

No labels

accessibility

authentication

breaking change

bug

documentation

enhancement

interface

internationalisation

observability

performance

security

tier

1

tier

2

tier

3

tier/4

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

Postulo/postulo!1230

No description provided.