ESCO: the whole 6.3 MB classification, in all 28 languages, is parsed and kept in every process #1110

New issue

Closed

opened 2026-10-31 03:04:04 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-31 03:04:04 +00:00

Owner

What exists today

src/postulo/jobs/esco.py:129-147 classification() (lru_cache(maxsize=1)) does json.loads(path.read_text()) of jobs/data/esco-1.2.1.json (6,362,937 bytes): 436 unit groups and 3,039 occupations, each with names in all 28 ESCO languages. _occupations_by_name() (:209) and _unit_groups_by_name() (:228) then build one folded index per reading language (lru_cache(maxsize=32)).

It is reached on every JobPosting.save() that may write the title (jobs/models.py:1762, esco.code_for(self.title)), by name_for() (jobs/models.py:1781), and by the title datalist (esco.suggestions() at applications/forms.py:196 and jobs/forms.py:1032), so in practice every worker loads it soon after start. Skills are already split: they live in esco-skills-1.2.1.zip and are read per language member.

Measured with tracemalloc under postulo.config.settings.test (audit/m_scripts/mem.py m3): classification() 14.6 MB retained, 29.2 MB peak, 0.22 s; then code_for(..., "en") +0.3 MB and code_for(..., "pt-PT") +1.0 MB. A copy pruned to English only (audit/m_scripts/esco_split.py) is 0.31 MB on disk and 2.4 MB in memory; English plus Portuguese 0.54 MB and 2.7 MB.

What would change

Have manage.py fetch_esco write the occupation and unit-group names per language, as the skills already are: the structure (isco, nace, major, revision, languages) in one member and the names in one member per language, in a zip like the skills. classification() keeps the structure only, and a per-language lru_cache loads names for the languages actually read (the reader's and English, the fallback). The public functions (code_for, name_for, unit_groups, suggestions, divisions_for) keep their signatures. The data-file check in jobs/checks.py and the licence notice follow the new layout. No library needed: json and zipfile, as the skills reader already uses.

What it saves or protects

medium: 14.6 MB retained and 29 MB peak per worker (measured); about 2.5 MB if only the languages read were loaded

Accessibility

none

Decisions

Zip like the skills, or one JSON file per language? Recommended: the zip, so the skills and the classification share one reader and one file per revision.

Found by the efficiency and reliability audit of 2026-10-30 (finding M3; index postulo/postulo#1108). Checked against main at daee30b15 (2026-10-30).

## What exists today `src/postulo/jobs/esco.py:129-147` `classification()` (`lru_cache(maxsize=1)`) does `json.loads(path.read_text())` of `jobs/data/esco-1.2.1.json` (6,362,937 bytes): 436 unit groups and 3,039 occupations, each with names in all 28 ESCO languages. `_occupations_by_name()` (`:209`) and `_unit_groups_by_name()` (`:228`) then build one folded index per reading language (`lru_cache(maxsize=32)`). It is reached on every `JobPosting.save()` that may write the title (`jobs/models.py:1762`, `esco.code_for(self.title)`), by `name_for()` (`jobs/models.py:1781`), and by the title datalist (`esco.suggestions()` at `applications/forms.py:196` and `jobs/forms.py:1032`), so in practice every worker loads it soon after start. Skills are already split: they live in `esco-skills-1.2.1.zip` and are read per language member. Measured with `tracemalloc` under `postulo.config.settings.test` (`audit/m_scripts/mem.py m3`): `classification()` 14.6 MB retained, 29.2 MB peak, 0.22 s; then `code_for(..., "en")` +0.3 MB and `code_for(..., "pt-PT")` +1.0 MB. A copy pruned to English only (`audit/m_scripts/esco_split.py`) is 0.31 MB on disk and 2.4 MB in memory; English plus Portuguese 0.54 MB and 2.7 MB. ## What would change Have `manage.py fetch_esco` write the occupation and unit-group names per language, as the skills already are: the structure (`isco`, `nace`, `major`, revision, languages) in one member and the names in one member per language, in a zip like the skills. `classification()` keeps the structure only, and a per-language `lru_cache` loads names for the languages actually read (the reader's and English, the fallback). The public functions (`code_for`, `name_for`, `unit_groups`, `suggestions`, `divisions_for`) keep their signatures. The data-file check in `jobs/checks.py` and the licence notice follow the new layout. No library needed: `json` and `zipfile`, as the skills reader already uses. ## What it saves or protects medium: 14.6 MB retained and 29 MB peak per worker (measured); about 2.5 MB if only the languages read were loaded ## Accessibility none ## Decisions - **Zip like the skills, or one JSON file per language?** Recommended: the zip, so the skills and the classification share one reader and one file per revision. --- Found by the efficiency and reliability audit of 2026-10-30 (finding M3; index postulo/postulo#1108). Checked against `main` at `daee30b15` (2026-10-30).

tiagoagueda added this to the 0.6.0 milestone

2026-10-31 03:04:04 +00:00

tiagoagueda added the

performance

enhancement

labels

2026-10-31 03:04:04 +00:00

tiagoagueda referenced this issue

2026-10-31 03:04:40 +00:00

Efficiency and reliability audit of 2026-10-30: index #1108

tiagoagueda referenced this issue from a pull request that will close it,

2026-11-10 23:08:20 +00:00