Job titles: gendered ESCO names are indexed whole, so a French, German or Portuguese title rarely gets a code #533

New issue

Closed

opened 2026-10-01 10:47:26 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-01 10:47:26 +00:00

Owner

Found on 2026-09-29 in a read-only review of postulo at 95348908d, reading jobs/esco.py and running code_for against the downloaded esco-1.2.1.json.

Checked again on 2026-10-01 against main at 0a5cad587: still present. The code is now at jobs/models.py:969-976 (JobPosting.save); jobs/esco.py and fetch_esco.py have not changed.

ESCO's preferred label for an occupation in a gendered language holds both forms in one string, separated by a slash: développeur de chaînes de blocs/développeuse de chaînes de blocs, Cloud-Ingenieur/Cloud-Ingenieurin. fetch_esco stores the label as it comes, and _occupations_by_name indexes only the whole string, so a title a person actually types, in one form, never matches an occupation. #266 promised that a title typed in the reader's language is found among the occupation names; for most of the classification in most gendered languages it is not.

Where: jobs/esco.py:200-212 (_occupations_by_name: one key per occupation, the whole label, .casefold() only); jobs/esco.py:224-245 (code_for: an exact .get(folded) and no other form); jobs/management/commands/fetch_esco.py, Command._labels (keeps the label whole); jobs/models.py:966-973 (JobPosting.save sets isco_code from it).
What happens: measured on v1.2.1, out of 3,039 occupations the labels holding a slash number 3,016 in Croatian, 3,000 in Slovak, 2,788 in German, 2,568 in Czech, 2,438 in Spanish, 2,315 in French, 2,278 in Portuguese, 1,500 in Italian, and 16 in English (waiter/waitress). code_for("développeur de logiciels", "fr"), code_for("Développeuse de logiciels", "fr") and code_for("Programador de software", "pt") all return ''; code_for("programador de software/programadora de software", "pt") returns '2512'; code_for("waiter", "en") returns ''. A listing titled in one form therefore carries no ISCO-08 code unless the title happens to equal the unit group's own (usually plural) name.
A second, smaller defect in the same function: the docstring says two occupations sharing a name in one language point at the same group, but in v1.2.1 names shared across different unit groups occur once in French, twice in German and Portuguese, three times in Italian, five times in Czech and seven times in Spanish. The dict comprehension keeps whichever came last, so such a title gets an arbitrary code, where _position_named (jobs/esco.py:419) treats the same ambiguity among skills as no match.
Why it matters: the code is what groups listings for a report and what the listing page names; for German, Czech or Slovak readers it is missing for the great majority of occupations.
Fix: when building _occupations_by_name, index each form of a label separately: split on /, index each non-empty half, and keep the whole label as well (for the few labels with other slashes). A folded name that points at more than one unit group matches nothing, as for skills, and the docstring says so. Fold both sides with _fold (NFKC, collapsed spaces) rather than strip().casefold(), so a title with a double space or a decomposed accent still matches.

Done when

tests/test_esco.py has a test with a fixture occupation labelled développeur de logiciels/développeuse de logiciels in fr that expects both code_for("Développeur de logiciels", "fr") and code_for("développeuse de logiciels", "fr") to return its code; it fails today.
The same file has a test where one French name belongs to occupations in two unit groups and code_for returns ''; it fails today.

Found on 2026-09-29 in a read-only review of postulo at 95348908d, reading `jobs/esco.py` and running `code_for` against the downloaded `esco-1.2.1.json`. Checked again on 2026-10-01 against `main` at `0a5cad587`: still present. The code is now at `jobs/models.py:969-976` (`JobPosting.save`); `jobs/esco.py` and `fetch_esco.py` have not changed. ESCO's preferred label for an occupation in a gendered language holds both forms in one string, separated by a slash: `développeur de chaînes de blocs/développeuse de chaînes de blocs`, `Cloud-Ingenieur/Cloud-Ingenieurin`. `fetch_esco` stores the label as it comes, and `_occupations_by_name` indexes only the whole string, so a title a person actually types, in one form, never matches an occupation. #266 promised that a title typed in the reader's language is found among the occupation names; for most of the classification in most gendered languages it is not. - **Where:** `jobs/esco.py:200-212` (`_occupations_by_name`: one key per occupation, the whole label, `.casefold()` only); `jobs/esco.py:224-245` (`code_for`: an exact `.get(folded)` and no other form); `jobs/management/commands/fetch_esco.py`, `Command._labels` (keeps the label whole); `jobs/models.py:966-973` (`JobPosting.save` sets `isco_code` from it). - **What happens:** measured on v1.2.1, out of 3,039 occupations the labels holding a slash number 3,016 in Croatian, 3,000 in Slovak, 2,788 in German, 2,568 in Czech, 2,438 in Spanish, 2,315 in French, 2,278 in Portuguese, 1,500 in Italian, and 16 in English (`waiter/waitress`). `code_for("développeur de logiciels", "fr")`, `code_for("Développeuse de logiciels", "fr")` and `code_for("Programador de software", "pt")` all return `''`; `code_for("programador de software/programadora de software", "pt")` returns `'2512'`; `code_for("waiter", "en")` returns `''`. A listing titled in one form therefore carries no ISCO-08 code unless the title happens to equal the unit group's own (usually plural) name. - **A second, smaller defect in the same function:** the docstring says two occupations sharing a name in one language point at the same group, but in v1.2.1 names shared across different unit groups occur once in French, twice in German and Portuguese, three times in Italian, five times in Czech and seven times in Spanish. The dict comprehension keeps whichever came last, so such a title gets an arbitrary code, where `_position_named` (`jobs/esco.py:419`) treats the same ambiguity among skills as no match. - **Why it matters:** the code is what groups listings for a report and what the listing page names; for German, Czech or Slovak readers it is missing for the great majority of occupations. - **Fix:** when building `_occupations_by_name`, index each form of a label separately: split on `/`, index each non-empty half, and keep the whole label as well (for the few labels with other slashes). A folded name that points at more than one unit group matches nothing, as for skills, and the docstring says so. Fold both sides with `_fold` (NFKC, collapsed spaces) rather than `strip().casefold()`, so a title with a double space or a decomposed accent still matches. ## Done when - `tests/test_esco.py` has a test with a fixture occupation labelled `développeur de logiciels/développeuse de logiciels` in `fr` that expects both `code_for("Développeur de logiciels", "fr")` and `code_for("développeuse de logiciels", "fr")` to return its code; it fails today. - The same file has a test where one French name belongs to occupations in two unit groups and `code_for` returns `''`; it fails today.

tiagoagueda added this to the 0.5.0 milestone

2026-10-01 10:47:26 +00:00

tiagoagueda added the

internationalisation

bug

labels

2026-10-01 10:47:26 +00:00

tiagoagueda referenced this issue from a commit

2026-10-04 10:05:01 +00:00

Index each gendered form of an ESCO occupation name

tiagoagueda closed this issue

2026-10-04 10:05:01 +00:00