Donate to Postulo/postulo

• https://buymeacoffee.com/tiagoagueda

---

See the documentation for more details on donation links.

0.6.0 fixes, bunch 5: memory and per-request queries #1227

Merged

tiagoagueda merged 13 commits from b5-memory into main 2026-11-11 00:58:41 +00:00

Conversation 0 Commits 13 Files changed 48 +3184 -298

tiagoagueda commented 2026-11-10 23:08:20 +00:00

Owner

• Copy link
• Copy Markdown

Bunch 5 of the 0.6.0 fixes: memory kept per process, and queries or work repeated on every request. One signed commit per issue, then the changelog lines and one test-fixture fix; merge fast-forward-only.

• Closes #1112: a person's plugin policy rows are read once a request, not twice per plugin (a company page: 72 queries to 53).
• Closes #1115: a signed-in page no longer asks whether the instance has accounts or the person has groups. No process memo: somebody signed in is an account, so the question is not asked for them; a visitor's page and every sign-up check still ask the database.
• Closes #1114: the dashboard's four counters are one aggregate.
• Closes #1171: primary_for answers from prefetched rows (a walk over 100 prefetched contacts: 203 queries to 3).
• Closes #1113: the metrics endpoint remembers that migrations are applied, once true.
• Closes #1133: the Server Overview remembers a complete font answer; an incomplete one is asked again.
• Closes #1109: a typed country is matched from names stored in the city lookup, not from every language's CLDR data (62.5 MB retained to none). An older lookup is rebuilt at first use; postulo.W003 warns when the stored names are out of date.
• Closes #1086: month names and language names are read without keeping every language loaded (115 MB and 66 MB retained to 0.1 and 6).
• Closes #1110: the ESCO occupations are one zip read a language at a time (15 MB to 2 MB held). The earlier file keeps working; postulo.W004 says to run fetch_esco again. CI's ESCO cache has a new key and file name.

Tested locally: lint, migration and catalogue checks, manage.py check, the full unit suite (21,341 passed, and one setup error from the worker-heartbeat fixture on Windows, fixed in the last commit), and 93 browser tests on the touched pages.

Not exercised locally: the real font probing (WeasyPrint's libraries do not load on the Windows machine this was written on) and the suite with ESCO downloaded in the new format; the unit job here does both.

🤖 Generated with Claude Code

Bunch 5 of the 0.6.0 fixes: memory kept per process, and queries or work repeated on every request. One signed commit per issue, then the changelog lines and one test-fixture fix; merge fast-forward-only. - Closes #1112: a person's plugin policy rows are read once a request, not twice per plugin (a company page: 72 queries to 53). - Closes #1115: a signed-in page no longer asks whether the instance has accounts or the person has groups. No process memo: somebody signed in is an account, so the question is not asked for them; a visitor's page and every sign-up check still ask the database. - Closes #1114: the dashboard's four counters are one aggregate. - Closes #1171: `primary_for` answers from prefetched rows (a walk over 100 prefetched contacts: 203 queries to 3). - Closes #1113: the metrics endpoint remembers that migrations are applied, once true. - Closes #1133: the Server Overview remembers a complete font answer; an incomplete one is asked again. - Closes #1109: a typed country is matched from names stored in the city lookup, not from every language's CLDR data (62.5 MB retained to none). An older lookup is rebuilt at first use; `postulo.W003` warns when the stored names are out of date. - Closes #1086: month names and language names are read without keeping every language loaded (115 MB and 66 MB retained to 0.1 and 6). - Closes #1110: the ESCO occupations are one zip read a language at a time (15 MB to 2 MB held). The earlier file keeps working; `postulo.W004` says to run `fetch_esco` again. CI's ESCO cache has a new key and file name. Tested locally: lint, migration and catalogue checks, `manage.py check`, the full unit suite (21,341 passed, and one setup error from the worker-heartbeat fixture on Windows, fixed in the last commit), and 93 browser tests on the touched pages. Not exercised locally: the real font probing (WeasyPrint's libraries do not load on the Windows machine this was written on) and the suite with ESCO downloaded in the new format; the unit job here does both. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

tiagoagueda added 11 commits 2026-11-10 23:08:20 +00:00

Read a person's plugin policy rows once a request, not twice per plugin ... 1220d44ec0

A plugin decision is memoised for the request, but each first answer read
its own rows: one query on the policy table and one on the access rules,
per plugin, and again for every parent and need it asked about. A company's
page spent 21 of its 72 queries there, and the number followed how many
plugins and modules are installed, not anything the person has.

The first decision of a request for a person now reads every policy row
that can reach them (their own and the instance's) and every access rule
for their account or their groups, in two queries, and every decision is
made from those. The rows sit in the same holder as the answers, under the
same stamp, so whatever forgets one forgets the other. The rules are as
they were: the person's row over the instance's, an account's rule over its
groups', the most permissive group on each right. Who decided comes with
the row, which was a third query for a plugin an administrator had switched.
The administration pages still ask `access_for` afresh, one plugin at a time.

A company's page: 72 queries to 53, the policy table 11 to 1 and the access
rules 10 to 1.

Because rows for plugins not yet asked about are now remembered too, more
has to make the request forget: rows deleted through a queryset or with the
account or group they name, and somebody joining or leaving a group. These
are signals, connected where the plugins app starts. Two of them were stale
answers before this change, for a plugin already asked about in the request.
Nothing is kept across requests, so another worker's write is seen by the
next request as it always was.

Closes #1112

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

Stop asking on every page whether there are accounts and groups ... 848ef7fee0

Every full page of somebody signed in made two EXISTS queries that had
nothing to do with the page: whether the instance has any account, which
opens sign-up on a fresh installation, and whether the person is in a
group, which decides one row of the account menu. The dashboard made 20
queries, two of them these; it now makes 18.

Whether there is an account is no longer asked for somebody signed in, who
is one: for them the page reads the sign-up setting alone. A visitor's page
asks the database as before, and so does everything that lets a sign-up in.
The issue chose between a process memo and the cache; this needs neither, so
nothing is remembered, nothing can go stale, and a worker has nothing to
hear from another worker's deletion.

Whether the person is in a group now comes with the profile, which the
preferences middleware reads for every request of somebody signed in: one
EXISTS inside that query instead of a query of its own. It is read again by
the next request, so joining or leaving a group shows on the next page. The
menu's row and its markup are as they were, and a page drawn without the
middleware asks as before.

Closes #1115

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

Count the dashboard's four counters in one statement ... f03658c3b5

The widget under "Where things stand" asked the database four times on
every view of the dashboard: how many applications are open, how many are
in conversation, how many have an offer, how many were sent in the last
thirty days. They are four counts over the same rows, so they are one
aggregate now, each number a count with its own condition, as the
fingerprint of the figures already counts reminders. The dashboard of a
new account goes from 20 queries to 17.

What "open" means is a condition of its own, `open_condition()`, which
`open()` filters by, `closed()` excludes by and the counter counts by, so
the three cannot drift apart. The numbers, the keys the template reads
and whose applications are counted are unchanged; the tests place every
status and both ends of the thirty days.

Closes #1114

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

Read the primary number and link from rows already prefetched ... abe1c9cf53

`phone_numbers.primary_for` and `web_links.primary_for` asked the database
every time, whether or not the holder's rows had been prefetched. A sync
that walks every contact to put a number and a profile on each card paid
two queries per contact for rows it could have been holding: 201 queries
for a hundred contacts, and 203 with the prefetch, which bought nothing.
Both now answer from the holder's prefetched rows where there are any and
query as before where there are none, so the same walk with the prefetch
is 3 queries. The names, the signatures and the answers are unchanged:
there is one primary per holder, and per kind, at most.

A holder that answers from rows it read must not go on doing so after
they change. The writers in the two modules that are handed the holder
(the one box, the rows on a page, the links taken in without a page, the
settling of a primary) let go of its prefetched rows first, as Django's
own managers do on `add` and `create`, so their own question about the
primary and every question after it is asked of the database.

Closes #1171

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

Read the migrations once to say they are applied, not at every scrape ... 336de004a9

`/metrics` reports `postulo_migrations_applied`, and to say it every scrape
built a MigrationExecutor: the graph of every migration file of every app,
and a read of the table of what is applied. That was most of the scrape,
paid every 15 to 60 seconds by an instance that has the endpoint on, to
give the answer it gave the last time.

A process that has found every migration applied now keeps that yes. New
migrations arrive only with new code, and new code is a new process, which
starts with nothing kept and reads its own migration files. Plugins bring
no migrations of their own, so installing one changes nothing here.

Only a yes is kept. A no is asked again at the next scrape, and so is a
check that raised, so a process that started before `migrate` finished
reports 1 at the first scrape after it has. The yes is set by the scrape
that finds it, in the worker that answers it, never at import, so nothing
is decided before `migrate` has had its turn.

A scrape after the first makes 35 queries where it made 37, and builds no
graph. The counts are left as they are, one statement for each table.

`metrics.forget_migrations()` drops the answer, and the suite calls it
around every test.

Closes #1113

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

Ask the fonts once what they draw, not at every visit of the Overview ... 40e96d6a4b

The Overview's Fonts row asked `fonts.renderable_scripts()` at every visit,
and every asking loaded fontconfig afresh, which reads the cache of every
font on the machine (several hundred faces in the image), and then looked
a font up for each script and style the offered languages need: up to 26
lookups to say what the last visit said.

`renderable_scripts()` now keeps its answer for the life of the process,
for the scripts and styles that were offered when it was given, so an
administrator who offers a language in another script is asked about that
one too. Two visits load fontconfig once where they loaded it twice, and
look up each font once.

Only a complete answer is kept: everything offered is drawn. Fonts are not
taken away from under a running process, so that one stands. An answer
with a script or a style missing, and a machine that could not be asked,
are asked again at the next visit, because that is the state an
administrator installs a font package to change, and a font installed a
minute ago has to show on the page without a restart. It is the rule
`pdf._chromium_found` already follows. The image draws everything it
offers, so there the page asks once in each worker.

`check_fonts` asks through the new `renderable_scripts_now()`, which loads
the fonts as they are at that moment whatever a process remembers.

`script_draws` and `family_draws`, which the tests skip by, share one font
map for the process instead of loading fontconfig at each call.

The theme fonts beside them are left as they are: `theme_assets.load`
already keeps each file it read, so the list reads nothing the second
time. A test now says so.

`fonts.forget()` drops the kept answer, and the suite calls it around
every test.

Closes #1133

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

Match a typed country from the lookup, not from every language's CLDR data ... 1c6219efeb

The first address placed with a country read CLDR's names of every country
in each of Postulo's 85 languages, and Babel keeps every locale it parses:
95 of them, 62 MB in each worker for the rest of its life, to answer one
question from a table of a few thousand names.

The names now go into the lookup database beside the cities when it is
built, which `fetch_geonames` does at provisioning, and a match is one
indexed read of it, as a city is. No process reads a locale to match a
country, and none holds the table, so what another process builds is the
next answer everywhere. The build reads one language at a time and hands
each locale back to Babel, so the process that builds holds nothing either.

A lookup from before this has no countries in it and is built again at its
first use, as one from before the lookup existed is, so the same names
narrow a city on an instance that was already provisioned. The lookup says
which CLDR and which languages its names were read from, and `manage.py
check` warns (postulo.W003) when they are not this Postulo's.

Measured in a fresh process, matching "Valencia, España": 62.5 MB held and
95 locales loaded before, nothing held and none loaded after.

Closes #1109

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

Read month names and language names without keeping every language loaded ... 166e7d89bd

Two tables were made by loading something for each of the 85 languages, and
each process then kept all of it for the rest of its life.

The month names a CSV date is read by were made by drawing a month in each
language. A language that has been drawn in stays active-able: Django's
catalogue merged with Postulo's and every plugin's, 2,717 files and 115 MB
in a worker from the first date with a word in it. Only the catalogues
Django ships translate a month, so they are read straight from their files,
in gettext's order and with Django's fallback, and nothing of them is kept.
The table is the same 843 names in the same order.

The table a typed language name is matched in was made from 86 CLDR
locales, which Babel keeps once parsed and `language_names` kept as well:
66 MB from the first CV a language was read off. Each locale is now read
and handed back to Babel, so what stays is the table, 6 MB. Naming
languages keeps the eight locales last read in instead of 256, and what a
language calls itself is kept as that word and not as its locale, so a list
of every language in one that names few of them no longer leaves hundreds
of locales behind. Every name, list and match is the one it was.

Closes #1086

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

Read the ESCO occupations a language at a time ... c21913b5ca

The classification was one JSON document, esco-<revision>.json: 436 unit
groups and 3,039 occupations, each with its name in the 28 languages ESCO
publishes. Every process parsed all of it soon after starting, because
saving a listing looks its title up, and kept it: 14.6 MB held and 29.8 MB
at the peak (tracemalloc, v1.2.1), to answer a reader in one language.

fetch_esco now writes esco-occupations-<revision>.zip, as it already wrote
the skills: one member says how the classification is built (the revision,
the languages, each unit group's major group, each occupation's unit group
and NACE divisions) and each language's names are a member of their own, in
the order of the entries. classification() holds the first, and a
language's names are read when somebody first reads in it, with the English
ones that stand in where it has none. The same use now holds 2.3 MB in
English and 3.7 MB with Portuguese, where it held 15.1 MB and 16.1 MB.
Every answer is the same, in every language.

An instance upgraded holds the old file until fetch_esco is next run. It is
still read, with the same answers and the same little held, at the cost of
parsing it once per language; the start-up check says so (postulo.W004) and
the next run of the command deletes it. The parts of the file are read at
different times now, and the command is run on instances that are serving,
so each part remembers the file it came out of: a process that finds two
that differ drops what it held and reads the new file whole, rather than
laying one revision's names against another's entries.

The unit jobs keep the new file between runs, under a new key, so a copy of
the old one is not what they test.

Closes #1110

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

Record bunch 5 in the changelog ... f6e2f41c4a

One line for each of the nine fixes before this commit: memory kept per
process (country, month and language names, the ESCO occupations) and
queries or work repeated on every request, scrape or visit.

Refs #1112, #1115, #1114, #1171, #1113, #1133, #1109, #1086, #1110

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

Write the tests' worker heartbeat in place ... 2a6e05c480

Every test starts with a fresh worker heartbeat (#1122), written through the
production code, which moves a temporary file over the old one. On Windows
that move, made some twenty thousand times in a run, is now and then refused
with "Access is denied" while something else has the file open, and one test
of a whole run errored at setup: seen in two runs on 10 November 2026.

The fixture now writes the file directly and tries again if it is refused.
Nothing reads it mid-write, which is what the move is for in production.

Refs #1122

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

tiagoagueda added 1 commit 2026-11-10 23:42:11 +00:00

Give the ESCO download half an hour ... ddcead5054

The step that downloads the ESCO classification got a ten-minute limit with
every other set-up step (#1138), and had not run since: the unit jobs kept
the files between runs. #1110 renames the file, so the cache was empty, the
download ran, took about fourteen minutes as it does these days, and was
stopped at ten: the unit job of this pull request failed with every test
unrun. The weekly check, which downloads on purpose, had the same limit and
would have failed on its first Monday.

Thirty minutes in the unit jobs and in the weekly check. The limit is there
to end a hang, not to time the service.

Refs #1110, #1138

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

tiagoagueda added 1 commit 2026-11-11 00:25:25 +00:00

Run CI again on a quieter host ... 9adfbaa9c1

Two browser tests timed out loading a page while the host's load average was
26 and its swap full; both passed when run again alone.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

tiagoagueda merged commit 9adfbaa9c1 into main 2026-11-11 00:58:41 +00:00

tiagoagueda deleted branch b5-memory 2026-11-11 00:58:42 +00:00

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

accessibility

Usable by everyone, including people with disabilities: one of the project's stated missions

  

authentication

Signing in, and proving who you are

  

breaking change

Changes behaviour or data in a way operators must act on

  

bug

Something does not do what it says it does

  

documentation

The wiki, the docs, or the words in the interface

  

enhancement

An improvement or a new capability

  

interface

The shape of the pages: what is where, and how it reads

  

internationalisation

Languages, regions, and the formats and symbols that go with them

  

observability

What an operator can see about a running instance: metrics, logs, health

  

performance

Uses fewer resources, or is more reliable, with no loss of features or accessibility

  

security

Keeping the code and the data secure: one of the project's stated missions

  

tier

1

Easy: small, self-contained, no prerequisites

  

tier

2

High-severity security: account access, personal data, the boundaries

  

tier

3

Foundations: prerequisite work that unblocks other issues

  

tier/4

Later: large features and long-horizon work

No labels

accessibility

authentication

breaking change

bug

documentation

enhancement

interface

internationalisation

observability

performance

security

tier

1

tier

2

tier

3

tier/4

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

Postulo/postulo!1227

No description provided.