A company's name in each language, taken from Wikidata, with a typed name only where Wikidata has none #673

New issue

Closed

opened 2026-10-03 01:58:51 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-03 01:58:51 +00:00

Owner

What exists today

One name per company, unique per owner. Company.name is a CharField(200) with UniqueConstraint(owner, name) (src/postulo/jobs/models.py:248, :312-314). company.name is read 96 times in 54 files (the letters' context at src/postulo/documents/rendering.py:476, the report at src/postulo/applications/reports.py:454, search, the API, the archive), and it is the key the CSV importer, the duplicate notice and the merge compare (jobs/duplicates.py, jobs/merging.py, core/csv_import.py).
Wikidata is an identifier and nothing more. A company can carry a Wikidata item (#42), checked by shape (src/postulo/plugins/identifiers/schemes.py:167-177) and linked. #42 deferred “Look up” and “Fill from Wikidata”: “behind a deliberate per-request action and never automatically”. Nothing fetches from wikidata.org.
The decision this runs into. #131 left the names of other people's organisations out of the career's translations on purpose: “Universidade de Lisboa stays that on an English CV; translating it invents an employer who never existed” (src/postulo/resume/translatable.py:16-27). Two tests pin it: tests/test_career_languages.py::test_a_name_that_belongs_to_somebody_else_is_not_translatable (:72-79) and the check that the organisation has no translation box (:305-312). They pin free-text career fields, which are not companies today; #683 and #685 propose that they become companies.
Outbound requests have one door. docs/THREAT-MODEL.md:74 (rule 5): postulo.plugins.http, with public_only_client(), deadline_in, until and read_body(limit=…) (src/postulo/plugins/http.py:206, :225, :241, :342) and the user agent Postulo (+https://source.tiagoagueda.com/postulo/postulo) (src/postulo/plugins/public_addresses.py:29). #366 is the open inventory of outbound destinations; it lists no Wikidata fetch.
Checked on 2026-10-02 against the live service (one read-only GET, descriptive user agent): Special:EntityData/Q29671.json is the University of Aveiro, 49,969 bytes, 29 labels. en is “University of Aveiro”, pt “Universidade de Aveiro”, fr “université d'Aveiro” (a lower-case initial, Wikidata's style); the item has an ISNI (P213), coordinates (P625), a website (P856) and a Crunchbase id (P2088), but no official name (P1448).

Checked against main at f09b6010b.

Proposal

A record for a name in a language: CompanyName (an OwnedModel): company, language (LanguageField), the name, its source (Wikidata or typed), when fetched; one per company and language. Company.name stays the one name that identifies the company; uniqueness, duplicates, merge matching and CSV matching are unchanged.
One module that reads Wikidata, postulo.jobs.wikidata: from a Q-number the person already recorded it makes one GET https://www.wikidata.org/wiki/Special:EntityData/<Q>.json through public_only_client(), with a deadline, a body limit well above the 50 KB seen and below a megabyte, at most three redirects (a merged item redirects) and a check that the answer came from www.wikidata.org. It returns {language tag: label}.
Keep only names Postulo can use: labels in a language its registry knows, so a company does not carry twenty names nobody prints (written through postulo.core.languages, so pt-br is stored pt-BR; a regional tag falls back to its base language by languages.match). Labels in Wikidata's mul code (the default for all languages, per Wikidata's help page on default values for labels) serve only a language with no label of its own.
Treat the text as untrusted: Wikidata is edited by anyone. Normalise it, strip control and bidirectional-formatting characters, cap it at the column's length, store plain text, draw it escaped and, inline, in <bdi>.
A card “Names in other languages” on the company's page: the stored names, a Fetch from Wikidata button (when the company has a Wikidata identifier; otherwise a line saying so with a link to add one) and a typed name for a language that has none. Fetching uses the dialog and errand shape of #674, shows what it found before keeping it, and never runs on a page load, a save, a merge or a schedule.
Company.name_in(language): the stored name for the language, then its base language, then Company.name. Used where a document's language is known (letters, the report, a CV once #683 lands). Lists, tables, search keys and the API's name keep the name the person typed; the API lists the names beside it.
Search finds a company by any of its names; a merge moves the other company's names (the kept one wins a language it has); the archive and importer carry them.
Name the fetch where operators read about outbound requests: #366's inventory and Hardening → Where the server is allowed to dial, with its trigger (a button), its fixed address and its switch.

Decisions

Conflict with #131 and its tests. Postulo does not translate or let anyone type the name; another body publishes it for the same thing. Recommended: keep the two tests and TRANSLATABLE as they are here, since they pin free-text fields, and settle the career side in #683 and #685, where an organisation becomes a company and a CV can print its name in the CV's language. The docstring at translatable.py:16-27 then gets the new reason.
Where a name shows. Everywhere in the interface language, or in documents only. Recommended: documents (letters, the report, a CV via #683) and the card. Someone who typed “Aveiro Uni” and finds a French string in their lists would not recognise their own record.
Who may type a name. The list says “only then”, when Wikidata has none. Recommended: per language. A typed name is allowed where Wikidata has given none (including a company with no Wikidata item); a Wikidata name is replaced only by fetching again, or removed, which falls back to the default. Editing a fetched name would make its source meaningless.
How the item is found. From the recorded identifier, or by searching Wikidata by name. Recommended: identifier only here. A search is a second request that can match the wrong item, and #42 gave it its own later step.
Which endpoint. Special:EntityData (documented for reading one known entity), wbgetentities (can ask for chosen languages only), or SPARQL (60-second deadline, 60 seconds of processing per 60 seconds per client, 30 error queries a minute, 5 parallel queries per IP). Recommended: Special:EntityData; SPARQL finds things by characteristics, which one item does not need.
User agent and rate. Wikimedia's policy asks for <client name>/<version> (<contact information>) <library>/<version>, contact being “an email address, a website, or a wiki user”, and warns generic agents “may be blocked without notice”; on a 429 it says to stop sending. Recommended: Postulo/<version> (https://source.tiagoagueda.com/postulo/postulo) httpx/<version> through the client's headers=, and a per-person allowance like the other named rates (src/postulo/config/settings/base.py:411-452), one fetch per press, no retry, no background refresh. Whether a project address satisfies the policy for software every operator runs is not verified.
Privacy of the request. It names the organisation the person is looking at, from the instance's address: the exposure #108 refused for tiles. Recommended: acceptable, because a person presses for it, one item at a time, to a fixed public host, with an instance switch to turn it off. The identifiers plugin's “never looked up online” (src/postulo/plugins/identifiers/__init__.py:59) stays true: the lookup lives in jobs.
Core or plugin. Recommended: core, as logos are: the model it fills is core's, and a plugin cannot own a table.

Prior art

Verified on 2026-10-02 from the documents, PyPI and OSV. None of these packages has an advisory on OSV, so the maintainer's “unpatched severe advisory” exception does not apply; the reason to reject is the guard.

Wikidata's interfaces. Special:EntityData/<Q>.json is documented as the way to read an entity you already know; the data is CC0, so no attribution is owed, though the card should say where a name came from.
wikidata 0.9.0 (dahlia/wikidata; GPL-3.0-or-later, combinable with AGPL-3.0; released 2025-10-31; declares no runtime dependency). Reject: a dependency for one JSON parse; its HTTP layer is not verified here, and a client with its own requests sits outside plugins.http, which rule 5 requires.
qwikidata 0.4.2 (Apache-2.0, 2022-11-09) and SPARQLWrapper 2.0.0 (W3C, 2022-03-13, needs rdflib): reject, stale, and SPARQL is not needed.
pywikibot 11.8.0 (MIT, 2026-09-29) and wikibaseintegrator 0.12.15 (MIT, 2025-12-21): bot and write frameworks on the requests library (a transitive dependency in uv.lock; Postulo itself uses httpx). Reject: far more than a read, outside the guard.
Hand-written, unlike applications/ical.py and documents/docx.py, which exist because a library was heavier than the part of a format used: here the part used is “GET one JSON document and read labels”, over the existing guarded client. Not verified: whether a maintained library could be told to use plugins.http's client.

Builds on

Issues #42 (identifier, lookup deferred), #366 (outbound inventory), #131 (career entries per language), #337 (BCP 47 tags). #653 for the library-first rule.

Needed by

#675 (the online half), #701 (Wikidata coordinates), #683 and #685 (a CV printing a company's name in its language).

Done when

A new tests/test_company_names.py: labels reduced to the registry's languages and written through core.languages; the base-language fallback; control and bidi characters cleaned; an oversize, slow or non-JSON answer refused with a sentence; a redirect to another host refused; a typed name refused where Wikidata gave one; Company.name_in falls back as above; nothing fetched by a page load, save or merge.
tests/security/test_outbound.py (new cases, MockTransport only): the request goes through public_only_client with the Wikimedia-style agent; the isolation sweep (tests/security/test_isolation_sweep.py) and tests/test_page_coverage.py cover the new views.
tests/test_merging.py, tests/test_search.py, tests/test_api.py, tests/test_export.py: merge, search, the API's names, the round trip. New record type, so the standing checklist applies: the archive format goes up by one (src/postulo/core/export.py FORMAT_VERSION, to one more than whatever it is when the commit lands; 29 at f09b6010b; tests/test_export.py:96 follows it); the candidate format (4) does not change; the processing record (src/postulo/core/gdpr.py) concerns contacts and is unchanged; tests/test_wiki_surface.py only if the plugin surface gains a name; the CV print choice (#308) is #683's.
#366's inventory and Hardening name the fetch. New strings are in the working catalogues (en-GB, fr-FR, pt-PT as drafts) and Weblate (#349).

From the list of improvements of 2026-10-02.

From the list: “A company can have a different denomination in different languages, for example “Universidade de Aveiro” is “University of Aveiro” (en) and “Université d'Aveiro” (fr). Fetch it from Wikidata. If none is available, only then enable user to define it or fall back to the default.” A company has one name, and nothing in Postulo reads Wikidata. ## What exists today - **One name per company, unique per owner.** `Company.name` is a `CharField(200)` with `UniqueConstraint(owner, name)` (`src/postulo/jobs/models.py:248`, `:312-314`). `company.name` is read 96 times in 54 files (the letters' context at `src/postulo/documents/rendering.py:476`, the report at `src/postulo/applications/reports.py:454`, search, the API, the archive), and it is the key the CSV importer, the duplicate notice and the merge compare (`jobs/duplicates.py`, `jobs/merging.py`, `core/csv_import.py`). - **Wikidata is an identifier and nothing more.** A company can carry a Wikidata item (#42), checked by shape (`src/postulo/plugins/identifiers/schemes.py:167-177`) and linked. #42 deferred “Look up” and “Fill from Wikidata”: “behind a deliberate per-request action and never automatically”. Nothing fetches from wikidata.org. - **The decision this runs into.** #131 left the names of other people's organisations out of the career's translations on purpose: “*Universidade de Lisboa* stays that on an English CV; translating it invents an employer who never existed” (`src/postulo/resume/translatable.py:16-27`). Two tests pin it: `tests/test_career_languages.py::test_a_name_that_belongs_to_somebody_else_is_not_translatable` (`:72-79`) and the check that the organisation has no translation box (`:305-312`). They pin free-text career fields, which are not companies today; #683 and #685 propose that they become companies. - **Outbound requests have one door.** `docs/THREAT-MODEL.md:74` (rule 5): `postulo.plugins.http`, with `public_only_client()`, `deadline_in`, `until` and `read_body(limit=…)` (`src/postulo/plugins/http.py:206`, `:225`, `:241`, `:342`) and the user agent `Postulo (+https://source.tiagoagueda.com/postulo/postulo)` (`src/postulo/plugins/public_addresses.py:29`). #366 is the open inventory of outbound destinations; it lists no Wikidata fetch. - **Checked on 2026-10-02 against the live service** (one read-only GET, descriptive user agent): `Special:EntityData/Q29671.json` is the University of Aveiro, 49,969 bytes, 29 labels. `en` is “University of Aveiro”, `pt` “Universidade de Aveiro”, `fr` “université d'Aveiro” (a lower-case initial, Wikidata's style); the item has an ISNI (P213), coordinates (P625), a website (P856) and a Crunchbase id (P2088), but no official name (P1448). Checked against `main` at `f09b6010b`. ## Proposal 1. **A record for a name in a language**: `CompanyName` (an `OwnedModel`): company, `language` (`LanguageField`), the name, its source (Wikidata or typed), when fetched; one per company and language. `Company.name` stays the one name that identifies the company; uniqueness, duplicates, merge matching and CSV matching are unchanged. 2. **One module that reads Wikidata**, `postulo.jobs.wikidata`: from a Q-number the person already recorded it makes one `GET https://www.wikidata.org/wiki/Special:EntityData/<Q>.json` through `public_only_client()`, with a deadline, a body limit well above the 50 KB seen and below a megabyte, at most three redirects (a merged item redirects) and a check that the answer came from `www.wikidata.org`. It returns `{language tag: label}`. 3. **Keep only names Postulo can use**: labels in a language its registry knows, so a company does not carry twenty names nobody prints (written through `postulo.core.languages`, so `pt-br` is stored `pt-BR`; a regional tag falls back to its base language by `languages.match`). Labels in Wikidata's `mul` code (the default for all languages, per Wikidata's help page on default values for labels) serve only a language with no label of its own. 4. **Treat the text as untrusted**: Wikidata is edited by anyone. Normalise it, strip control and bidirectional-formatting characters, cap it at the column's length, store plain text, draw it escaped and, inline, in `<bdi>`. 5. **A card “Names in other languages” on the company's page**: the stored names, a *Fetch from Wikidata* button (when the company has a Wikidata identifier; otherwise a line saying so with a link to add one) and a typed name for a language that has none. Fetching uses the dialog and errand shape of #674, shows what it found before keeping it, and never runs on a page load, a save, a merge or a schedule. 6. **`Company.name_in(language)`**: the stored name for the language, then its base language, then `Company.name`. Used where a document's language is known (letters, the report, a CV once #683 lands). Lists, tables, search keys and the API's `name` keep the name the person typed; the API lists the names beside it. 7. **Search finds a company by any of its names**; a merge moves the other company's names (the kept one wins a language it has); the archive and importer carry them. 8. **Name the fetch where operators read about outbound requests**: #366's inventory and *Hardening → Where the server is allowed to dial*, with its trigger (a button), its fixed address and its switch. ## Decisions - **Conflict with #131 and its tests.** Postulo does not translate or let anyone type the name; another body publishes it for the same thing. **Recommended: keep the two tests and `TRANSLATABLE` as they are here**, since they pin free-text fields, and settle the career side in #683 and #685, where an organisation becomes a company and a CV can print its name in the CV's language. The docstring at `translatable.py:16-27` then gets the new reason. - **Where a name shows.** Everywhere in the interface language, or in documents only. **Recommended: documents** (letters, the report, a CV via #683) **and the card**. Someone who typed “Aveiro Uni” and finds a French string in their lists would not recognise their own record. - **Who may type a name.** The list says “only then”, when Wikidata has none. **Recommended: per language.** A typed name is allowed where Wikidata has given none (including a company with no Wikidata item); a Wikidata name is replaced only by fetching again, or removed, which falls back to the default. Editing a fetched name would make its source meaningless. - **How the item is found.** From the recorded identifier, or by searching Wikidata by name. **Recommended: identifier only here.** A search is a second request that can match the wrong item, and #42 gave it its own later step. - **Which endpoint.** `Special:EntityData` (documented for reading one known entity), `wbgetentities` (can ask for chosen languages only), or SPARQL (60-second deadline, 60 seconds of processing per 60 seconds per client, 30 error queries a minute, 5 parallel queries per IP). **Recommended: `Special:EntityData`**; SPARQL finds things by characteristics, which one item does not need. - **User agent and rate.** Wikimedia's policy asks for `<client name>/<version> (<contact information>) <library>/<version>`, contact being “an email address, a website, or a wiki user”, and warns generic agents “may be blocked without notice”; on a `429` it says to stop sending. **Recommended: `Postulo/<version> (https://source.tiagoagueda.com/postulo/postulo) httpx/<version>`** through the client's `headers=`, and a per-person allowance like the other named rates (`src/postulo/config/settings/base.py:411-452`), one fetch per press, no retry, no background refresh. Whether a project address satisfies the policy for software every operator runs is not verified. - **Privacy of the request.** It names the organisation the person is looking at, from the instance's address: the exposure #108 refused for tiles. **Recommended: acceptable, because a person presses for it, one item at a time, to a fixed public host, with an instance switch to turn it off.** The identifiers plugin's “never looked up online” (`src/postulo/plugins/identifiers/__init__.py:59`) stays true: the lookup lives in `jobs`. - **Core or plugin.** **Recommended: core**, as logos are: the model it fills is core's, and a plugin cannot own a table. ## Prior art Verified on 2026-10-02 from the documents, PyPI and OSV. None of these packages has an advisory on OSV, so the maintainer's “unpatched severe advisory” exception does not apply; the reason to reject is the guard. - **Wikidata's interfaces.** `Special:EntityData/<Q>.json` is documented as the way to read an entity you already know; the data is CC0, so no attribution is owed, though the card should say where a name came from. - **`wikidata` 0.9.0** (dahlia/wikidata; GPL-3.0-or-later, combinable with AGPL-3.0; released 2025-10-31; declares no runtime dependency). **Reject**: a dependency for one JSON parse; its HTTP layer is not verified here, and a client with its own requests sits outside `plugins.http`, which rule 5 requires. - **`qwikidata` 0.4.2** (Apache-2.0, 2022-11-09) and **`SPARQLWrapper` 2.0.0** (W3C, 2022-03-13, needs `rdflib`): **reject**, stale, and SPARQL is not needed. - **`pywikibot` 11.8.0** (MIT, 2026-09-29) and **`wikibaseintegrator` 0.12.15** (MIT, 2025-12-21): bot and write frameworks on the `requests` library (a transitive dependency in `uv.lock`; Postulo itself uses `httpx`). **Reject**: far more than a read, outside the guard. - **Hand-written**, unlike `applications/ical.py` and `documents/docx.py`, which exist because a library was heavier than the part of a format used: here the part used is “GET one JSON document and read `labels`”, over the existing guarded client. Not verified: whether a maintained library could be told to use `plugins.http`'s client. ## Builds on Issues #42 (identifier, lookup deferred), #366 (outbound inventory), #131 (career entries per language), #337 (BCP 47 tags). #653 for the library-first rule. ## Needed by #675 (the online half), #701 (Wikidata coordinates), #683 and #685 (a CV printing a company's name in its language). ## Done when - A new `tests/test_company_names.py`: labels reduced to the registry's languages and written through `core.languages`; the base-language fallback; control and bidi characters cleaned; an oversize, slow or non-JSON answer refused with a sentence; a redirect to another host refused; a typed name refused where Wikidata gave one; `Company.name_in` falls back as above; nothing fetched by a page load, save or merge. - `tests/security/test_outbound.py` (new cases, `MockTransport` only): the request goes through `public_only_client` with the Wikimedia-style agent; the isolation sweep (`tests/security/test_isolation_sweep.py`) and `tests/test_page_coverage.py` cover the new views. - `tests/test_merging.py`, `tests/test_search.py`, `tests/test_api.py`, `tests/test_export.py`: merge, search, the API's names, the round trip. **New record type, so the standing checklist applies:** the archive format goes up by one (`src/postulo/core/export.py` `FORMAT_VERSION`, to one more than whatever it is when the commit lands; 29 at `f09b6010b`; `tests/test_export.py:96` follows it); the candidate format (4) does not change; the processing record (`src/postulo/core/gdpr.py`) concerns contacts and is unchanged; `tests/test_wiki_surface.py` only if the plugin surface gains a name; the CV print choice (#308) is #683's. - #366's inventory and *Hardening* name the fetch. New strings are in the working catalogues (en-GB, fr-FR, pt-PT as drafts) and Weblate (#349). From the list of improvements of 2026-10-02.

tiagoagueda added this to the 1.0.0 milestone

2026-10-03 01:58:51 +00:00

tiagoagueda added the

enhancement

internationalisation

labels

2026-10-03 01:58:51 +00:00

tiagoagueda referenced this issue

2026-10-03 01:58:53 +00:00

Warn when a company's identifiers disagree with each other, offline first and against Wikidata when asked #675

tiagoagueda referenced this issue

2026-10-03 01:58:59 +00:00