Maps: matching a country name loads Babel's territory names for all 85 languages into every process #1109

New issue

Closed

opened 2026-10-31 03:04:03 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-31 03:04:03 +00:00

Owner

What exists today

src/postulo/jobs/places.py:199-232 _countries() (an lru_cache(maxsize=1)) reads geonames-countryinfo.txt and then _translated_countries() (:234-247), which calls Locale.parse(tag).territories for each of the 85 tags in core.languages.NATIVE_NAMES. Babel keeps every parsed locale in its module cache (babel.localedata._cache) for the life of the process, so all 85 locales (95 cache entries with their parents) stay loaded.

It is reached from resolve() (:382, the call at :402) whenever the typed text has a country part and the city table has a candidate. resolve() is called by place() (:345, at :358), which jobs/placing.py:43 runs on an address save when mapping is offered and the place key changed, and which jobs/views.py:375 runs; and by split_location() (jobs/models.py:50, call at :63), used at jobs/models.py:780 and by migration 0033.

Measured with tracemalloc around places._translated_countries(<every CLDR territory code>) under postulo.config.settings.test (script audit/m_scripts/mem.py m1): 56.6 MB retained, 57.1 MB peak, 5.61 s, Babel cache 95 locales, 22,334 (code, name) pairs. The countryinfo file is not in the local checkout, so the call was measured directly rather than through _countries(); on an instance that ran fetch_geonames the path above runs it as written.

What changed from the candidate: the memory is mostly the same Babel locale data that core/language_names._index() loads (#1086). Run after M1 in the same process, _index() added only 7.2 MB. So the two share one cost, and fixing #1086 alone leaves this path loading all 85 locales anyway. The memory only falls once both are fixed.

What would change

Build the country-name table once at provisioning, the way #399 moved the cities into geonames-cities1000.sqlite: manage.py fetch_geonames writes a countries(key, code) table (folded name to ISO code, from countryinfo plus the CLDR territory names in each NATIVE_NAMES language) into the lookup database, and _countries() becomes one indexed SELECT ... WHERE key = ? like _candidates(). No process then loads Babel locale data for this. When the table is missing the behaviour stays as today's missing-file case (no country filter). Babel stays the source; no new library.

A lighter alternative is to read only the instance's selected languages (languages.selected(), #347) plus English, but that still holds several MB per worker and changes which country names match; the provisioning table keeps every name and costs nothing at run time.

What it saves or protects

large: about 57 MB held per worker for its whole life and a 5.6 s first call (measured), on the first address placed with a country

Accessibility

none

Decisions

Same issue as #1086, or a separate one? Recommended: separate issue, linked to #1086 with a note that the two share Babel's cache, so the memory only falls once both land.
Rebuild the table when Babel is upgraded? Recommended: yes, fetch_geonames stamps the CLDR version (language_names.cldr_version()) into the table and the start-up check warns when it differs.

Found by the efficiency and reliability audit of 2026-10-30 (finding M1; index postulo/postulo#1108). Checked against main at daee30b15 (2026-10-30).

## What exists today `src/postulo/jobs/places.py:199-232` `_countries()` (an `lru_cache(maxsize=1)`) reads `geonames-countryinfo.txt` and then `_translated_countries()` (`:234-247`), which calls `Locale.parse(tag).territories` for each of the 85 tags in `core.languages.NATIVE_NAMES`. Babel keeps every parsed locale in its module cache (`babel.localedata._cache`) for the life of the process, so all 85 locales (95 cache entries with their parents) stay loaded. It is reached from `resolve()` (`:382`, the call at `:402`) whenever the typed text has a country part and the city table has a candidate. `resolve()` is called by `place()` (`:345`, at `:358`), which `jobs/placing.py:43` runs on an address save when mapping is offered and the place key changed, and which `jobs/views.py:375` runs; and by `split_location()` (`jobs/models.py:50`, call at `:63`), used at `jobs/models.py:780` and by migration 0033. Measured with `tracemalloc` around `places._translated_countries(<every CLDR territory code>)` under `postulo.config.settings.test` (script `audit/m_scripts/mem.py m1`): 56.6 MB retained, 57.1 MB peak, 5.61 s, Babel cache 95 locales, 22,334 (code, name) pairs. The countryinfo file is not in the local checkout, so the call was measured directly rather than through `_countries()`; on an instance that ran `fetch_geonames` the path above runs it as written. What changed from the candidate: the memory is mostly the same Babel locale data that `core/language_names._index()` loads (#1086). Run after M1 in the same process, `_index()` added only 7.2 MB. So the two share one cost, and fixing #1086 alone leaves this path loading all 85 locales anyway. The memory only falls once both are fixed. ## What would change Build the country-name table once at provisioning, the way #399 moved the cities into `geonames-cities1000.sqlite`: `manage.py fetch_geonames` writes a `countries(key, code)` table (folded name to ISO code, from countryinfo plus the CLDR territory names in each `NATIVE_NAMES` language) into the lookup database, and `_countries()` becomes one indexed `SELECT ... WHERE key = ?` like `_candidates()`. No process then loads Babel locale data for this. When the table is missing the behaviour stays as today's missing-file case (no country filter). Babel stays the source; no new library. A lighter alternative is to read only the instance's selected languages (`languages.selected()`, #347) plus English, but that still holds several MB per worker and changes which country names match; the provisioning table keeps every name and costs nothing at run time. ## What it saves or protects large: about 57 MB held per worker for its whole life and a 5.6 s first call (measured), on the first address placed with a country ## Accessibility none ## Decisions - **Same issue as #1086, or a separate one?** Recommended: separate issue, linked to #1086 with a note that the two share Babel's cache, so the memory only falls once both land. - **Rebuild the table when Babel is upgraded?** Recommended: yes, `fetch_geonames` stamps the CLDR version (`language_names.cldr_version()`) into the table and the start-up check warns when it differs. --- Found by the efficiency and reliability audit of 2026-10-30 (finding M1; index postulo/postulo#1108). Checked against `main` at `daee30b15` (2026-10-30).

tiagoagueda added this to the 0.6.0 milestone

2026-10-31 03:04:03 +00:00

tiagoagueda added the

performance

enhancement

labels

2026-10-31 03:04:03 +00:00

tiagoagueda referenced this issue

2026-10-31 03:04:40 +00:00

Efficiency and reliability audit of 2026-10-30: index #1108

tiagoagueda referenced this issue from a pull request that will close it,

2026-11-10 23:08:20 +00:00