Memory: reading month names or language names loads every language into each process #1086

New issue

Closed

opened 2026-10-29 13:18:16 +00:00 by tiagoagueda · 1 comment

tiagoagueda commented

2026-10-29 13:18:16 +00:00

Owner

Found while measuring the test workers' memory for #1033. Two production paths load data for every language just to read a little of it, and each process keeps it for its whole life:

csv_import._month_names loads all 85 merged translation catalogues (about 110 MB per process) to know the month names in every language when reading a date from a CSV.
language_names._index loads every CLDR locale from Babel (about 75 MB) to name each language in every other.

On a small instance (Ragnar is a Raspberry Pi), that is close to 200 MB per gunicorn worker, held forever, for data most requests never touch.

What to do

Month names: read them from Babel's CLDR data for the languages actually needed, a handful of locales, or build the table lazily per language with an LRU bound. Not by activating every Django catalogue.
Language names: load the CLDR names for the languages offered on this instance, and look up any other on demand with a small LRU bound.
Measure: a test that importing and using each path loads at most a bounded number of locales.

Not a 0.5.0 blocker; the test side was fixed in #1033's item 3.

Found while measuring the test workers' memory for #1033. Two production paths load data for **every** language just to read a little of it, and each process keeps it for its whole life: - **`csv_import._month_names`** loads all 85 merged translation catalogues (about **110 MB** per process) to know the month names in every language when reading a date from a CSV. - **`language_names._index`** loads every CLDR locale from Babel (about **75 MB**) to name each language in every other. On a small instance (Ragnar is a Raspberry Pi), that is close to 200 MB per gunicorn worker, held forever, for data most requests never touch. ## What to do - **Month names:** read them from Babel's CLDR data for the languages actually needed, a handful of locales, or build the table lazily per language with an LRU bound. Not by activating every Django catalogue. - **Language names:** load the CLDR names for the languages offered on this instance, and look up any other on demand with a small LRU bound. - **Measure:** a test that importing and using each path loads at most a bounded number of locales. Not a 0.5.0 blocker; the test side was fixed in #1033's item 3.

tiagoagueda referenced this issue

2026-10-31 03:04:03 +00:00

Maps: matching a country name loads Babel's territory names for all 85 languages into every process #1109

tiagoagueda referenced this issue

2026-10-31 03:04:04 +00:00

Container: gunicorn loads the app separately in each of three workers and reloads the data caches every 500 requests #1111

tiagoagueda commented

2026-10-31 03:04:40 +00:00

Author

Owner

From the efficiency and reliability audit of 2026-10-30 (finding M2), what this issue does not yet say:

src/postulo/core/language_names.py:188-199 _index() (lru_cache(maxsize=1)) loops over ("en", *languages.NATIVE_NAMES) (86 locales) and, for each, all 637 offered codes, through _locale() (:97-106, lru_cache(maxsize=256)). Babel keeps each parsed locale in babel.localedata._cache for the life of the process. match() uses it; callers are core/importer.py:1537 (archive import), resume/candidate.py:1461 (CV candidate, also reached from the Europass reader), resume/importing.py:211 (CV import), resume/management/commands/match_languages.py:34, and migration resume/migrations/0015_match_spoken_languages.py:18.

Measured with tracemalloc under postulo.config.settings.test (audit/m_scripts/mem.py m2): 62.2 MB retained, 68.3 MB peak, 2.82 s, Babel cache 95 locales. For comparison, name("pt", "en") alone holds 3.7 MB with 2 locales, and listing("en") adds 0.1 MB.

#1086 already says this path loads every CLDR locale (about 75 MB) and asks for the offered languages plus an LRU bound. What #1086 does not say, worth a comment on it rather than a new issue:

which paths trigger it (CV import, Europass and archive imports, the match_languages command, migration 0015);
that it shares Babel's cache with jobs/places._translated_countries() (M1): after M1 has run, _index() adds only 7.2 MB, so bounding one without the other saves nothing;
the measured numbers above (62 MB, 2.8 s).

Nothing new to file. Add the three points above as a comment on #1086 and link it to the M1 issue.

Checked against main at daee30b15 (2026-10-30).

From the efficiency and reliability audit of 2026-10-30 (finding M2), what this issue does not yet say: `src/postulo/core/language_names.py:188-199` `_index()` (`lru_cache(maxsize=1)`) loops over `("en", *languages.NATIVE_NAMES)` (86 locales) and, for each, all 637 offered codes, through `_locale()` (`:97-106`, `lru_cache(maxsize=256)`). Babel keeps each parsed locale in `babel.localedata._cache` for the life of the process. `match()` uses it; callers are `core/importer.py:1537` (archive import), `resume/candidate.py:1461` (CV candidate, also reached from the Europass reader), `resume/importing.py:211` (CV import), `resume/management/commands/match_languages.py:34`, and migration `resume/migrations/0015_match_spoken_languages.py:18`. Measured with `tracemalloc` under `postulo.config.settings.test` (`audit/m_scripts/mem.py m2`): 62.2 MB retained, 68.3 MB peak, 2.82 s, Babel cache 95 locales. For comparison, `name("pt", "en")` alone holds 3.7 MB with 2 locales, and `listing("en")` adds 0.1 MB. #1086 already says this path loads every CLDR locale (about 75 MB) and asks for the offered languages plus an LRU bound. What #1086 does not say, worth a comment on it rather than a new issue: - which paths trigger it (CV import, Europass and archive imports, the `match_languages` command, migration 0015); - that it shares Babel's cache with `jobs/places._translated_countries()` (M1): after M1 has run, `_index()` adds only 7.2 MB, so bounding one without the other saves nothing; - the measured numbers above (62 MB, 2.8 s). Nothing new to file. Add the three points above as a comment on #1086 and link it to the M1 issue. Checked against `main` at `daee30b15` (2026-10-30).

tiagoagueda referenced this issue

2026-10-31 03:04:41 +00:00

Efficiency and reliability audit of 2026-10-30: index #1108

tiagoagueda referenced this issue from a pull request that will close it,

2026-11-10 23:08:20 +00:00

0.6.0 fixes, bunch 5: memory and per-request queries #1227

tiagoagueda referenced this issue from a commit

2026-11-10 23:08:22 +00:00

Read month names and language names without keeping every language loaded

tiagoagueda referenced this issue from a commit

2026-11-10 23:08:22 +00:00

Record bunch 5 in the changelog