Catalogue tool: `compile` parses every catalogue twice and builds each .mo by repeated byte concatenation #1145

New issue

Closed

opened 2026-10-31 03:04:20 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-31 03:04:20 +00:00

Owner

What exists today

src/postulo/core/messages_tool.py:

cmd_compile() (957-978) parses every django.po of every set and writes its .mo; then, for Postulo itself, build_report() (1351-1371) parses all 2,352 catalogues again from disk to write locale/status.json, calling po_path() twice per language and set.
compile_catalogue() (909-954) builds the GNU .mo with out += struct.pack("<2I", …) in two loops (lines 943-950). Bytes are immutable, so each += copies the whole buffer: quadratic in the entry count.

Measurement, in memory with nothing written (a script in the audit folder, work/w3_compile.py, imported the module, parsed and compiled every catalogue without writing, then ran build_report(); compile itself was not run): catalogue_sets 0.9 s; 2,352 catalogues: parse 8.0 s, compile_catalogue 3.9 s (39.3 MB of .mo); build_report 13.7 s, of which a profile puts about three quarters in parse (20.1 of 25.8 s profiled). So a compile is about 27 s and about 10 s of it is the second parse. The concatenation: 9,684 appends (the size of a fully translated core catalogue's two tables) take 25 ms with += against 2.5 ms with a list and b"".join, about 1-2 s over all well-translated catalogues. The quadratic claim holds but is the smaller half of this finding.

compile runs at ci.yml:140, 262, 555, every-python.yml:112, dependencies.yml:104, release.yml:68, in the Dockerfile and in each plugin's CI. #1033 item 6 caches the CI output on a hash of the .po files: that skips the run when nothing changed but does not speed up a run when something did (every Weblate pull request), and covers neither the image build nor the plugins. Not a duplicate.

What would change

cmd_compile keeps the catalogues it parsed, per set and language, and passes them to build_report so it does not read them again; stats keeps its own read.
compile_catalogue builds the two offset tables once: a flat list of integers and one struct.pack(f"<{2 * count}I", *flat), or array.array("I"), as CPython's own Tools/i18n/msgfmt.py does.
babel.messages.mofile.write_mo would be the library answer, but it would bring Babel in for some forty lines and the current writer leaves out drafts that fail message_problems; keep the house code and fix the loop.

What it saves or protects

small-medium: about 10 s of a 27 s compile (the status report re-parses every catalogue) and 1-2 s of byte copying; compile runs 4-6 times per CI push, in the image build and in each plugin's CI

Accessibility

none

Decisions

Split? Recommended: one PR; both changes sit in the compile path and share test_compile_writes_the_status_report_the_picker_reads.

Found by the efficiency and reliability audit of 2026-10-30 (finding W3; index postulo/postulo#1108). Checked against main at daee30b15 (2026-10-30).

## What exists today `src/postulo/core/messages_tool.py`: - `cmd_compile()` (957-978) parses every `django.po` of every set and writes its `.mo`; then, for Postulo itself, `build_report()` (1351-1371) parses all 2,352 catalogues again from disk to write `locale/status.json`, calling `po_path()` twice per language and set. - `compile_catalogue()` (909-954) builds the GNU .mo with `out += struct.pack("<2I", …)` in two loops (lines 943-950). Bytes are immutable, so each `+=` copies the whole buffer: quadratic in the entry count. Measurement, in memory with nothing written (a script in the audit folder, `work/w3_compile.py`, imported the module, parsed and compiled every catalogue without writing, then ran `build_report()`; `compile` itself was not run): `catalogue_sets` 0.9 s; 2,352 catalogues: parse 8.0 s, `compile_catalogue` 3.9 s (39.3 MB of .mo); `build_report` 13.7 s, of which a profile puts about three quarters in `parse` (20.1 of 25.8 s profiled). So a compile is about 27 s and about 10 s of it is the second parse. The concatenation: 9,684 appends (the size of a fully translated core catalogue's two tables) take 25 ms with `+=` against 2.5 ms with a list and `b"".join`, about 1-2 s over all well-translated catalogues. The quadratic claim holds but is the smaller half of this finding. `compile` runs at `ci.yml:140, 262, 555`, `every-python.yml:112`, `dependencies.yml:104`, `release.yml:68`, in the Dockerfile and in each plugin's CI. #1033 item 6 caches the CI output on a hash of the `.po` files: that skips the run when nothing changed but does not speed up a run when something did (every Weblate pull request), and covers neither the image build nor the plugins. Not a duplicate. ## What would change - `cmd_compile` keeps the catalogues it parsed, per set and language, and passes them to `build_report` so it does not read them again; `stats` keeps its own read. - `compile_catalogue` builds the two offset tables once: a flat list of integers and one `struct.pack(f"<{2 * count}I", *flat)`, or `array.array("I")`, as CPython's own `Tools/i18n/msgfmt.py` does. - `babel.messages.mofile.write_mo` would be the library answer, but it would bring Babel in for some forty lines and the current writer leaves out drafts that fail `message_problems`; keep the house code and fix the loop. ## What it saves or protects small-medium: about 10 s of a 27 s compile (the status report re-parses every catalogue) and 1-2 s of byte copying; compile runs 4-6 times per CI push, in the image build and in each plugin's CI ## Accessibility none ## Decisions - **Split?** Recommended: one PR; both changes sit in the compile path and share `test_compile_writes_the_status_report_the_picker_reads`. --- Found by the efficiency and reliability audit of 2026-10-30 (finding W3; index postulo/postulo#1108). Checked against `main` at `daee30b15` (2026-10-30).

tiagoagueda added this to the 0.6.0 milestone

2026-10-31 03:04:20 +00:00

tiagoagueda added the

performance

enhancement

labels

2026-10-31 03:04:20 +00:00

tiagoagueda referenced this issue

2026-10-31 03:04:41 +00:00

Efficiency and reliability audit of 2026-10-30: index #1108

tiagoagueda referenced this issue from a pull request that will close it,

2026-11-11 20:36:51 +00:00