Run the unit tests on three workers, each letting go of the languages it met (#1033)

tiagoagueda commented

2026-10-29 13:18:15 +00:00

Owner

A test worker grew towards 1 GB over the suite (#830), so the unit job's 2 GB container held
two workers and not three: with three (run 1658) the kernel killed two of them at once.

Measured one process through the whole suite in order, after every test file: what grew and
never came back was every language the worker had met. Django keeps each language's merged
catalogue, and below it Python's gettext keeps every .mo it opened (2,717 of them, about
110 MB, including the ones a fixture compiled into a directory since deleted); Django builds
reverse()'s tables once per active language in every resolver and namespace resolver (about
1.3 MB and 4,000 objects a language, although Postulo translates no route); and Babel keeps
every CLDR locale it loaded (about 75 MB for all of them). An instance meets a few languages;
the suite renders every page in all 86, reads month names in each and names every language in
every other, and a worker then carried them all to the end.

tests/conftest.py now drops all three after a test file that met more than four languages,
after the file's own fixtures are torn down. Production code is unchanged: there each cache
is paid once for the languages a process serves.

The other large cost was tests/test_translations.py, whose module fixture parses every
catalogue of every set: some 400 MB while the file runs, and under xdist's plain load every
worker was handed some of its tests near the end of the suite and parsed them all at once.
The file is now one xdist group, which CI runs with --dist loadgroup so one worker takes
it, and the fixture drops the references and comments nothing reads, a third of that.

Measured on Windows (the absolute numbers differ from CI's Linux; the shape does not), one
process through the suite: 1,124 MB at the end and a 1,421 MB peak before, 881 and 1,106
after, 2.9 million objects down to 2.0, and 35 minutes instead of 44. With -n 3 --dist
loadgroup the suite passed in 14 min 53 s and the workers peaked at 606, 754 and 965 MB.

ci.yml and every-python.yml run the unit step with -n 3 --dist loadgroup, with the numbers
in the comment; CONTRIBUTING says so; the marker is registered for -p no:xdist runs.

Refs #1033

Notes for review

Item 3 of #1033: the unit job on three workers. This replaces the -n 2 comment in #1083 (draft), which held the unit job back on memory.

The cause, measured per test file in one process. A worker kept every language it had met until the end of the run:
- the translation catalogues: Django's merged catalogue plus gettext._translations, 2,717 .mo files, about 110 MB;
- Django's per-language reverse() tables in every resolver, about 113 MB, though Postulo translates no route;
- Babel's CLDR locales, about 75 MB.
- Also, test_translations.py parses every catalogue (about 390 MB), and under --dist load every worker took some of it near the end. That is why run 1658 lost two workers at the same second.
The fix, test infrastructure only:
- after a test file that met more than four languages, release_languages() drops those caches;
- test_translations.py is one xdist_group, and its fixture keeps only what the tests read;
- the unit step runs -n 3 --dist loadgroup.
Before and after, one process on Windows: end 1,124 → 881 MB, peak 1,421 → 1,106 MB. On three workers locally, the whole suite passed in 14 min 53 s. Worker peaks were 606, 754 and 965 MB, staying under about 1.9 GB together.
To confirm on CI: the margin under the 2 GB cap on Linux, and the step time under loadgroup.
Production costs found on the way are filed as their own issue.

🤖 Generated with Claude Code

A test worker grew towards 1 GB over the suite (#830), so the unit job's 2 GB container held two workers and not three: with three (run 1658) the kernel killed two of them at once. Measured one process through the whole suite in order, after every test file: what grew and never came back was every language the worker had met. Django keeps each language's merged catalogue, and below it Python's gettext keeps every .mo it opened (2,717 of them, about 110 MB, including the ones a fixture compiled into a directory since deleted); Django builds reverse()'s tables once per active language in every resolver and namespace resolver (about 1.3 MB and 4,000 objects a language, although Postulo translates no route); and Babel keeps every CLDR locale it loaded (about 75 MB for all of them). An instance meets a few languages; the suite renders every page in all 86, reads month names in each and names every language in every other, and a worker then carried them all to the end. tests/conftest.py now drops all three after a test file that met more than four languages, after the file's own fixtures are torn down. Production code is unchanged: there each cache is paid once for the languages a process serves. The other large cost was tests/test_translations.py, whose module fixture parses every catalogue of every set: some 400 MB while the file runs, and under xdist's plain `load` every worker was handed some of its tests near the end of the suite and parsed them all at once. The file is now one xdist group, which CI runs with `--dist loadgroup` so one worker takes it, and the fixture drops the references and comments nothing reads, a third of that. Measured on Windows (the absolute numbers differ from CI's Linux; the shape does not), one process through the suite: 1,124 MB at the end and a 1,421 MB peak before, 881 and 1,106 after, 2.9 million objects down to 2.0, and 35 minutes instead of 44. With -n 3 --dist loadgroup the suite passed in 14 min 53 s and the workers peaked at 606, 754 and 965 MB. ci.yml and every-python.yml run the unit step with `-n 3 --dist loadgroup`, with the numbers in the comment; CONTRIBUTING says so; the marker is registered for `-p no:xdist` runs. Refs #1033 ## Notes for review **Item 3 of #1033: the unit job on three workers.** This replaces the `-n 2` comment in #1083 (draft), which held the unit job back on memory. - **The cause, measured per test file in one process.** A worker kept every language it had met until the end of the run: - the translation catalogues: Django's merged catalogue plus `gettext._translations`, 2,717 `.mo` files, about 110 MB; - Django's per-language `reverse()` tables in every resolver, about 113 MB, though Postulo translates no route; - Babel's CLDR locales, about 75 MB. - Also, `test_translations.py` parses every catalogue (about 390 MB), and under `--dist load` every worker took some of it near the end. That is why run 1658 lost two workers at the same second. - **The fix, test infrastructure only:** - after a test file that met more than four languages, `release_languages()` drops those caches; - `test_translations.py` is one `xdist_group`, and its fixture keeps only what the tests read; - the unit step runs `-n 3 --dist loadgroup`. - **Before and after**, one process on Windows: end 1,124 → 881 MB, peak 1,421 → 1,106 MB. On three workers locally, the whole suite passed in 14 min 53 s. Worker peaks were 606, 754 and 965 MB, staying under about 1.9 GB together. - **To confirm on CI:** the margin under the 2 GB cap on Linux, and the step time under `loadgroup`. - **Production costs found on the way** are filed as their own issue. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

tiagoagueda added 1 commit

2026-10-29 13:18:15 +00:00

Run the unit tests on three workers, each letting go of the languages it met (#1033 )

CI / Unit tests and coverage (Python 3.14) (pull_request) Successful in 15m16s

Details

CI / Checks: lint, migrations, catalogues, settings, built files (pull_request) Successful in 2m42s

Details

CI / Database tests on PostgreSQL (pull_request) Successful in 4m8s

Details

CI / Security audit (pull_request) Successful in 1m18s

Details

CI / Browser tests (Chromium) (pull_request) Successful in 38m12s

Details

ecbd513486

A test worker grew towards 1 GB over the suite (#830), so the unit job's 2 GB container held
two workers and not three: with three (run 1658) the kernel killed two of them at once.

Measured one process through the whole suite in order, after every test file: what grew and
never came back was every language the worker had met. Django keeps each language's merged
catalogue, and below it Python's gettext keeps every .mo it opened (2,717 of them, about
110 MB, including the ones a fixture compiled into a directory since deleted); Django builds
reverse()'s tables once per active language in every resolver and namespace resolver (about
1.3 MB and 4,000 objects a language, although Postulo translates no route); and Babel keeps
every CLDR locale it loaded (about 75 MB for all of them). An instance meets a few languages;
the suite renders every page in all 86, reads month names in each and names every language in
every other, and a worker then carried them all to the end.

tests/conftest.py now drops all three after a test file that met more than four languages,
after the file's own fixtures are torn down. Production code is unchanged: there each cache
is paid once for the languages a process serves.

The other large cost was tests/test_translations.py, whose module fixture parses every
catalogue of every set: some 400 MB while the file runs, and under xdist's plain `load` every
worker was handed some of its tests near the end of the suite and parsed them all at once.
The file is now one xdist group, which CI runs with `--dist loadgroup` so one worker takes
it, and the fixture drops the references and comments nothing reads, a third of that.

Measured on Windows (the absolute numbers differ from CI's Linux; the shape does not), one
process through the suite: 1,124 MB at the end and a 1,421 MB peak before, 881 and 1,106
after, 2.9 million objects down to 2.0, and 35 minutes instead of 44. With -n 3 --dist
loadgroup the suite passed in 14 min 53 s and the workers peaked at 606, 754 and 965 MB.

ci.yml and every-python.yml run the unit step with `-n 3 --dist loadgroup`, with the numbers
in the comment; CONTRIBUTING says so; the marker is registered for `-p no:xdist` runs.

Refs #1033

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

tiagoagueda force-pushed 1033-worker-memory from ecbd513486

CI / Unit tests and coverage (Python 3.14) (pull_request) Successful in 15m16s

Details

CI / Checks: lint, migrations, catalogues, settings, built files (pull_request) Successful in 2m42s

Details

CI / Database tests on PostgreSQL (pull_request) Successful in 4m8s

Details

CI / Security audit (pull_request) Successful in 1m18s

Details

CI / Browser tests (Chromium) (pull_request) Successful in 38m12s

Details

to 5126ee10a2