Write every language code as a BCP 47 tag in its canonical form: pt-BR, not pt-br or pt_BR #337

Closed
opened 2026-10-01 06:39:30 +00:00 by tiagoagueda · 4 comments
Owner

The requirement: all language codes must follow BCP 47. A BCP 47 tag (RFC 5646) is made of subtags from the IANA registry joined by hyphens, and §2.1.1 gives the form to write one in: the language in lower case, a script in title case, a region in upper case (pt-BR, en-GB, sr-Cyrl). Tags compare without regard to case, so pt-br is a valid tag written the wrong way, and pt_BR is not a tag at all: it is a gettext locale name.

Postulo writes every code "as Django writes it" (src/postulo/core/languages.py:136): all lower case. That is what is stored, compared, exported and sent out, except in a few places that already disagree with it.

What exists today

  • The list. NATIVE_NAMES holds 69 codes, the regional ones as en-gb, fr-fr, pt-pt and pt-br. The same spellings key FLAG_COUNTRIES, PLURAL_FORMS, SOURCE, FROM_LANGUAGE in core/phones.py and locale/status.json; LANGUAGE_CODE is "en-gb".
  • Stored, in nine places, each a free CharField(max_length=10) (or a list) with no check: Profile.language and record_language; SiteSettings.default_language and offered_languages; language on CV, CoverLetter, UploadedDocument and RenderedDocument; resume.Translation.language. The pages write pt-br; nothing normalises on save.
  • Sent out in lower case: <html lang> of the interface (templates/base.html; 500.html hard-codes en-gb) and of every rendered document, which is where a PDF takes its language; lang= on options and spans; language on CVs, letters and documents in the API, a plain string in the OpenAPI description; the personal archive; Notification.language and the webhook payload; DocumentMetadata.language, handed to store plugins.
  • Already canonical, and so at odds with the rest:
    • the Word export: language_tag() in documents/docx.py writes pt-PT and zh-Hans, "BCP 47 is case-insensitive and Word is not always". It is the only mention of BCP 47 in the repository;
    • the last-resort literal "en-GB" in documents/models.py:108 and notifications/service.py:30;
    • the wiki: Webhooks shows "language": "en-GB" and Writing a plugin says "as a Postulo code (pt-PT)", where the code sends en-gb and pt-pt.
  • Taken in: the candidate file accepts ^[a-z]{2,3}(?:-[a-z0-9]{2,8}){0,2}$ after lower-casing and says a code is "like en-gb or pt-pt" (resume/candidate.py); Europass and captured pages lower-case what they read; the archive import takes what the file holds.
  • Four normalisers, none shared: translatable.normalise, patterns.tag_of and europass._locale lower-case; docx.language_tag writes the canonical form.

What a framework forces, and stays

  • Locale directories and the PO Language: header are gettext locale names (fr_FR, pt_BR), made by Django's to_locale. They are not language tags, and are derived from one in languages.locale_dir_name and messages_tool.
  • Django lower-cases what get_language() returns, so {{ LANGUAGE_CODE }} and the Content-Language header are lower case. It does not force the case of LANGUAGE_CODE or of the LANGUAGES keys: get_languages() lower-cases the keys and to_locale its input (read in Django 6.1.1's source, not run).

Proposal

  1. One place. core/languages.py gains the canonical writer (what docx.language_tag does; underscores accepted in, never written out) and a comparison that ignores case and falls back to the primary subtag (RFC 4647 §3.4). They replace the three lower-casers and the exact-match lookups (flag_country, PLURAL_FORMS.get, site.offers, status.get, NATIVE_NAMES.get, {% language_flag %}).
  2. The list in canonical form: en-GB, fr-FR, pt-PT, pt-BR, in status.json too.
  3. Stored values canonical. A migration re-cases the nine places; every write goes through the canonical writer; the columns grow, since 10 characters do not hold ca-ES-valencia (RFC 5646 §4.4.1 advises room for 35).
  4. Sent out canonical everywhere listed above, the interface's <html lang> re-cased from Django's value. The API's language is described as a BCP 47 tag.
  5. Taken in: any case, and underscores, accepted and made canonical instead of lowered: the candidate file (its pattern and its sentence), Europass, captured pages, and the archive import, which then reads older archives too. The archive's FORMAT_VERSION goes up, with its note.
  6. Codes to settle:
    • Norwegian. The list has nb; the ESCO data is keyed no (jobs/management/commands/fetch_esco.py:96), so esco._reading("nb") falls back to English and a Norwegian reader gets English occupation and skill names. Europass maps nor to no, and a page saying lang="no" matches nothing. The comparison needs to know no and nb together.
    • Serbian. The tag is sr; the catalogue is Cyrillic. sr has no Suppress-Script in the registry, so sr-Cyrl is the tag that says so.
    • Scripts. SCRIPTS names scripts in English ("Cyrillic", "Ethiopic") where BCP 47 has subtags (Cyrl, Ethi), and is_rtl reads the primary subtag only, so ku-Latn would be drawn right to left.
  7. Tests and documents that pin the lower-case form follow (tests/test_smoke.py, test_theme.py, test_translations.py, test_language_variants.py and others); docs/TRANSLATING.md mixes pt-br and pt-BR in one file.

Decisions

  • Canonical, or merely valid. Decided on 2026-10-01: always canonical. pt-br follows BCP 47 to the letter, but Postulo writes the canonical form wherever it writes a tag. It is the form the RFC gives, the one the wiki already promises and Word needs, and one form ends the mixture.
  • LANGUAGE_CODE and the LANGUAGES keys. Decided with the above: canonical. Django accepts either, and the list is then written once.
  • The Content-Language header. Decided with the above: canonical. Django writes it in lower case, so the middleware re-cases it.
  • Well-formed or valid. Decided on 2026-10-01: the list against the registry, free fields for shape. The interface list is kept by hand and held by a test to a copy of the IANA registry kept in the repository (every subtag registered, none deprecated, no script the registry suppresses). A free field, such as a document's language, is checked for shape only, so pt-AO saves. No library is added.
  • sr or sr-Cyrl. Decided on 2026-10-01: sr-Cyrl, now. The list says sr-Cyrl, the catalogue directories become sr_Cyrl, and a stored sr is migrated.

A breaking change for whatever reads these codes: API clients, webhook receivers, store plugins and the archive see pt-PT where they saw pt-pt. It goes in the changelog as one.

The other repositories

postulo/postulo-chromium#12, postulo/postulo-dav#14, postulo/postulo-imap#3, postulo/postulo-apprise#5, postulo/postulo-paperless#5 and postulo/postulo-helloworld#4 follow this one. postulo-firefox holds no language code and follows through its pin; postulo-templates holds none yet; postulo-mcp only relays the API's language and takes its OpenAPI snapshot again.

Done when

  • Outside locale directories and PO headers, Postulo writes no lower-case regional code and no underscore form: a test reads the list, the stored columns after the migration, the API, the archive, the webhook payload and a rendered document's <html lang>, and finds canonical tags.
  • pt_br and PT-br typed or imported are stored as pt-BR.
  • A Norwegian reader gets ESCO's Norwegian names.
  • docs/TRANSLATING.md and the wiki state the rule once: a code is a BCP 47 tag in its canonical form, and a locale directory's name is gettext's, derived from it.
  • New strings are in the working catalogues (en, fr-fr, pt-pt, pt-br as drafts).

Requested on 2026-10-01: all language codes must follow BCP 47.

The requirement: **all language codes must follow BCP 47.** A BCP 47 tag (RFC 5646) is made of subtags from the IANA registry joined by hyphens, and §2.1.1 gives the form to write one in: the language in lower case, a script in title case, a region in upper case (`pt-BR`, `en-GB`, `sr-Cyrl`). Tags compare without regard to case, so `pt-br` is a valid tag written the wrong way, and `pt_BR` is not a tag at all: it is a gettext locale name. Postulo writes every code "as Django writes it" (`src/postulo/core/languages.py:136`): all lower case. That is what is stored, compared, exported and sent out, except in a few places that already disagree with it. ## What exists today - **The list.** `NATIVE_NAMES` holds 69 codes, the regional ones as `en-gb`, `fr-fr`, `pt-pt` and `pt-br`. The same spellings key `FLAG_COUNTRIES`, `PLURAL_FORMS`, `SOURCE`, `FROM_LANGUAGE` in `core/phones.py` and `locale/status.json`; `LANGUAGE_CODE` is `"en-gb"`. - **Stored**, in nine places, each a free `CharField(max_length=10)` (or a list) with no check: `Profile.language` and `record_language`; `SiteSettings.default_language` and `offered_languages`; `language` on `CV`, `CoverLetter`, `UploadedDocument` and `RenderedDocument`; `resume.Translation.language`. The pages write `pt-br`; nothing normalises on save. - **Sent out in lower case:** `<html lang>` of the interface (`templates/base.html`; `500.html` hard-codes `en-gb`) and of every rendered document, which is where a PDF takes its language; `lang=` on options and spans; `language` on CVs, letters and documents in the API, a plain string in the OpenAPI description; the personal archive; `Notification.language` and the webhook payload; `DocumentMetadata.language`, handed to store plugins. - **Already canonical, and so at odds with the rest:** - the Word export: `language_tag()` in `documents/docx.py` writes `pt-PT` and `zh-Hans`, "BCP 47 is case-insensitive and Word is not always". It is the only mention of BCP 47 in the repository; - the last-resort literal `"en-GB"` in `documents/models.py:108` and `notifications/service.py:30`; - the wiki: *Webhooks* shows `"language": "en-GB"` and *Writing a plugin* says "as a Postulo code (`pt-PT`)", where the code sends `en-gb` and `pt-pt`. - **Taken in:** the candidate file accepts `^[a-z]{2,3}(?:-[a-z0-9]{2,8}){0,2}$` after lower-casing and says a code is "like en-gb or pt-pt" (`resume/candidate.py`); Europass and captured pages lower-case what they read; the archive import takes what the file holds. - **Four normalisers, none shared:** `translatable.normalise`, `patterns.tag_of` and `europass._locale` lower-case; `docx.language_tag` writes the canonical form. ## What a framework forces, and stays - **Locale directories and the PO `Language:` header** are gettext locale names (`fr_FR`, `pt_BR`), made by Django's `to_locale`. They are not language tags, and are derived from one in `languages.locale_dir_name` and `messages_tool`. - **Django lower-cases what `get_language()` returns**, so `{{ LANGUAGE_CODE }}` and the `Content-Language` header are lower case. It does not force the case of `LANGUAGE_CODE` or of the `LANGUAGES` keys: `get_languages()` lower-cases the keys and `to_locale` its input (read in Django 6.1.1's source, not run). ## Proposal 1. **One place.** `core/languages.py` gains the canonical writer (what `docx.language_tag` does; underscores accepted in, never written out) and a comparison that ignores case and falls back to the primary subtag (RFC 4647 §3.4). They replace the three lower-casers and the exact-match lookups (`flag_country`, `PLURAL_FORMS.get`, `site.offers`, `status.get`, `NATIVE_NAMES.get`, `{% language_flag %}`). 2. **The list in canonical form:** `en-GB`, `fr-FR`, `pt-PT`, `pt-BR`, in `status.json` too. 3. **Stored values canonical.** A migration re-cases the nine places; every write goes through the canonical writer; the columns grow, since 10 characters do not hold `ca-ES-valencia` (RFC 5646 §4.4.1 advises room for 35). 4. **Sent out canonical** everywhere listed above, the interface's `<html lang>` re-cased from Django's value. The API's `language` is described as a BCP 47 tag. 5. **Taken in:** any case, and underscores, accepted and made canonical instead of lowered: the candidate file (its pattern and its sentence), Europass, captured pages, and the archive import, which then reads older archives too. The archive's `FORMAT_VERSION` goes up, with its note. 6. **Codes to settle:** - **Norwegian.** The list has `nb`; the ESCO data is keyed `no` (`jobs/management/commands/fetch_esco.py:96`), so `esco._reading("nb")` falls back to English and a Norwegian reader gets English occupation and skill names. Europass maps `nor` to `no`, and a page saying `lang="no"` matches nothing. The comparison needs to know `no` and `nb` together. - **Serbian.** The tag is `sr`; the catalogue is Cyrillic. `sr` has no Suppress-Script in the registry, so `sr-Cyrl` is the tag that says so. - **Scripts.** `SCRIPTS` names scripts in English (`"Cyrillic"`, `"Ethiopic"`) where BCP 47 has subtags (`Cyrl`, `Ethi`), and `is_rtl` reads the primary subtag only, so `ku-Latn` would be drawn right to left. 7. **Tests and documents** that pin the lower-case form follow (`tests/test_smoke.py`, `test_theme.py`, `test_translations.py`, `test_language_variants.py` and others); `docs/TRANSLATING.md` mixes `pt-br` and `pt-BR` in one file. ## Decisions - **Canonical, or merely valid. Decided on 2026-10-01: always canonical.** `pt-br` follows BCP 47 to the letter, but Postulo writes the canonical form wherever it writes a tag. It is the form the RFC gives, the one the wiki already promises and Word needs, and one form ends the mixture. - **`LANGUAGE_CODE` and the `LANGUAGES` keys. Decided with the above: canonical.** Django accepts either, and the list is then written once. - **The `Content-Language` header. Decided with the above: canonical.** Django writes it in lower case, so the middleware re-cases it. - **Well-formed or valid. Decided on 2026-10-01: the list against the registry, free fields for shape.** The interface list is kept by hand and held by a test to a copy of the IANA registry kept in the repository (every subtag registered, none deprecated, no script the registry suppresses). A free field, such as a document's language, is checked for shape only, so `pt-AO` saves. No library is added. - **`sr` or `sr-Cyrl`. Decided on 2026-10-01: `sr-Cyrl`, now.** The list says `sr-Cyrl`, the catalogue directories become `sr_Cyrl`, and a stored `sr` is migrated. **A breaking change** for whatever reads these codes: API clients, webhook receivers, store plugins and the archive see `pt-PT` where they saw `pt-pt`. It goes in the changelog as one. ## The other repositories postulo/postulo-chromium#12, postulo/postulo-dav#14, postulo/postulo-imap#3, postulo/postulo-apprise#5, postulo/postulo-paperless#5 and postulo/postulo-helloworld#4 follow this one. postulo-firefox holds no language code and follows through its pin; postulo-templates holds none yet; postulo-mcp only relays the API's `language` and takes its OpenAPI snapshot again. ## Done when - Outside locale directories and PO headers, Postulo writes no lower-case regional code and no underscore form: a test reads the list, the stored columns after the migration, the API, the archive, the webhook payload and a rendered document's `<html lang>`, and finds canonical tags. - `pt_br` and `PT-br` typed or imported are stored as `pt-BR`. - A Norwegian reader gets ESCO's Norwegian names. - `docs/TRANSLATING.md` and the wiki state the rule once: a code is a BCP 47 tag in its canonical form, and a locale directory's name is gettext's, derived from it. - New strings are in the working catalogues (en, fr-fr, pt-pt, pt-br as drafts). Requested on 2026-10-01: all language codes must follow BCP 47.
Author
Owner

Decided on 2026-10-01: always go canonical.

Postulo writes a language code in its canonical BCP 47 form wherever it writes one. The body's decisions are updated to say so:

  • canonical, not merely valid, everywhere;
  • LANGUAGE_CODE and the LANGUAGES keys in canonical form;
  • the Content-Language header re-cased, since "always" leaves no place for Django's lower-case form where Postulo can change it.

Locale directory names and PO headers are not language tags and stay gettext's.

Still open: whether tags are checked against the registry or only for shape, and sr or sr-Cyrl. Canonical form does not settle the second: sr is already canonical, and the question is whether to state the script.

Decided on 2026-10-01: **always go canonical.** Postulo writes a language code in its canonical BCP 47 form wherever it writes one. The body's decisions are updated to say so: - canonical, not merely valid, everywhere; - `LANGUAGE_CODE` and the `LANGUAGES` keys in canonical form; - the `Content-Language` header re-cased, since "always" leaves no place for Django's lower-case form where Postulo can change it. Locale directory names and PO headers are not language tags and stay gettext's. Still open: whether tags are checked against the registry or only for shape, and `sr` or `sr-Cyrl`. Canonical form does not settle the second: `sr` is already canonical, and the question is whether to state the script.
Author
Owner

Final decision on translations (maintainer, 2026-10-01)

  1. Every language is scaffolded: each has its catalogue, with a slot for every string.
  2. A development build requires every string to be present in en-GB, fr-FR and pt-PT. A draft counts.
  3. A release requires every string to be present in the languages selected for that release. A draft counts.
  4. 0.4.0 requires no additional languages.
  5. 0.5.0 additionally requires fr-CA and en-CA; then, eventually, any language of the European continent still missing (Russian excepted); and, eventually, Chinese and Arabic. Those last two may move to 0.6.0 if the load on 0.5.0 becomes too great.

What this changes here

Only the spelling of what the decision names, which is this issue's subject:

  • The development set is written en-GB, fr-FR, pt-PT, and 0.5.0's variants fr-CA and en-CA: canonical tags, as decided here. Their locale directories keep the names gettext forces (fr_CA, en_CA).
  • en-CA is the first English catalogue beside the source. It needs a LANGUAGES entry and a catalogue of its own, seeded from en-GB.
  • Chinese cannot be one tag: simplified and traditional are zh-Hans and zh-Hant, which bears on the open point here about script subtags (sr versus sr-Cyrl).
**Final decision on translations (maintainer, 2026-10-01)** 1. **Every language is scaffolded**: each has its catalogue, with a slot for every string. 2. **A development build** requires every string to be present in **en-GB, fr-FR and pt-PT**. A draft counts. 3. **A release** requires every string to be present in **the languages selected for that release**. A draft counts. 4. **0.4.0** requires no additional languages. 5. **0.5.0** additionally requires **fr-CA** and **en-CA**; then, eventually, any language of the European continent still missing (Russian excepted); and, eventually, **Chinese** and **Arabic**. Those last two may move to 0.6.0 if the load on 0.5.0 becomes too great. ## What this changes here Only the spelling of what the decision names, which is this issue's subject: - The development set is written **en-GB, fr-FR, pt-PT**, and 0.5.0's variants **fr-CA** and **en-CA**: canonical tags, as decided here. Their locale directories keep the names gettext forces (`fr_CA`, `en_CA`). - **en-CA** is the first English catalogue beside the source. It needs a `LANGUAGES` entry and a catalogue of its own, seeded from en-GB. - **Chinese** cannot be one tag: simplified and traditional are `zh-Hans` and `zh-Hant`, which bears on the open point here about script subtags (`sr` versus `sr-Cyrl`).
Author
Owner

Decided on 2026-10-01, which closes the two points left open here:

  • Well-formed or valid: the list against the registry, free fields for shape. The interface list is kept by hand, and a test holds it to a copy of the IANA registry kept in the repository: every subtag registered, none deprecated, no script the registry suppresses. A free field, such as a document's language or the language of a career record, is checked for shape only, so pt-AO still saves. No library is added.
  • Serbian: sr-Cyrl, now. The list says sr-Cyrl, the catalogue directories become sr_Cyrl in every catalogue set, and a stored sr is migrated to it. A browser asking for sr, sr-RS or sr-Latn is still given this catalogue, as it is today.

Checked against the registry dated 2026-09-17: all 69 codes of the list are registered, none is deprecated, sr has no suppressed script, and nb belongs to the macrolanguage no, which is the relation the Norwegian ESCO names need.

The work is on the branch 337-bcp47-language-codes and is merged into main only on the maintainer's word.

What follows for the other repositories: each plugin's own locale/sr becomes locale/sr_Cyrl when it moves its pin, before its first extract.

Decided on 2026-10-01, which closes the two points left open here: - **Well-formed or valid: the list against the registry, free fields for shape.** The interface list is kept by hand, and a test holds it to a copy of the IANA registry kept in the repository: every subtag registered, none deprecated, no script the registry suppresses. A free field, such as a document's language or the language of a career record, is checked for shape only, so `pt-AO` still saves. No library is added. - **Serbian: `sr-Cyrl`, now.** The list says `sr-Cyrl`, the catalogue directories become `sr_Cyrl` in every catalogue set, and a stored `sr` is migrated to it. A browser asking for `sr`, `sr-RS` or `sr-Latn` is still given this catalogue, as it is today. Checked against the registry dated 2026-09-17: all 69 codes of the list are registered, none is deprecated, `sr` has no suppressed script, and `nb` belongs to the macrolanguage `no`, which is the relation the Norwegian ESCO names need. The work is on the branch `337-bcp47-language-codes` and is merged into `main` only on the maintainer's word. What follows for the other repositories: each plugin's own `locale/sr` becomes `locale/sr_Cyrl` when it moves its pin, before its first `extract`.
Author
Owner

Landed on main as a6363a397, with the wiki in postulo.wiki@9fff446.

What the repositories that follow this one need to know, since some of it differs from what the body proposed:

  • The codes are en-GB, fr-FR, pt-PT, pt-BR and sr-Cyrl. postulo-messages refuses a code spelt otherwise than the list spells it (po_path("fr-fr") raises), so a test that still names ("fr-fr", "pt-pt") fails loudly once the pin moves, instead of holding fewer catalogues than it thinks.
  • Serbian's directory is sr_Cyrl. Rename locale/sr to locale/sr_Cyrl before the first extract under the new pin, or the tool writes an empty sr_Cyrl beside the old directory.
  • Four names are on the plugin surface (postulo.plugins.api): language_tag, is_language_tag, language_match and language_matches. They are Postulo's own writer, shape check and comparison; a plugin that compares or stores a code uses these and writes no normaliser of its own.
  • The archive is format 28, not 27: #308 took 27 the same day. An archive of format 27 or older is read as it was written, and its codes are respelt on the way in.
  • The API's language is described as a BCP 47 tag and a letter's is filled now, so postulo-mcp retakes its OpenAPI snapshot.
  • Notification.language and DocumentMetadata.language are canonical tags. A receiver compares without regard to case.

What it did not do: Armenian and Georgian are still missing from languages.SCRIPTS, which is #455, and locale/status.json was re-keyed and not regenerated, which is #495.

Landed on `main` as `a6363a397`, with the wiki in `postulo.wiki@9fff446`. What the repositories that follow this one need to know, since some of it differs from what the body proposed: - **The codes are `en-GB`, `fr-FR`, `pt-PT`, `pt-BR` and `sr-Cyrl`.** `postulo-messages` refuses a code spelt otherwise than the list spells it (`po_path("fr-fr")` raises), so a test that still names `("fr-fr", "pt-pt")` fails loudly once the pin moves, instead of holding fewer catalogues than it thinks. - **Serbian's directory is `sr_Cyrl`.** Rename `locale/sr` to `locale/sr_Cyrl` before the first `extract` under the new pin, or the tool writes an empty `sr_Cyrl` beside the old directory. - **Four names are on the plugin surface** (`postulo.plugins.api`): `language_tag`, `is_language_tag`, `language_match` and `language_matches`. They are Postulo's own writer, shape check and comparison; a plugin that compares or stores a code uses these and writes no normaliser of its own. - **The archive is format 28**, not 27: #308 took 27 the same day. An archive of format 27 or older is read as it was written, and its codes are respelt on the way in. - **The API's `language` is described as a BCP 47 tag** and a letter's is filled now, so `postulo-mcp` retakes its OpenAPI snapshot. - **`Notification.language` and `DocumentMetadata.language` are canonical tags.** A receiver compares without regard to case. What it did not do: Armenian and Georgian are still missing from `languages.SCRIPTS`, which is #455, and `locale/status.json` was re-keyed and not regenerated, which is #495.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
Postulo/postulo#337
No description provided.