Warn when a company's identifiers disagree with each other, offline first and against Wikidata when asked #675

New issue

Closed

opened 2026-10-03 01:58:53 +00:00 by tiagoagueda · 1 comment

tiagoagueda commented

2026-10-03 01:58:53 +00:00

Owner

What exists today

Validation is per scheme. A scheme has a pattern, a tidy, and optionally a checksum (src/postulo/core/identifiers.py, Scheme; the table is src/postulo/plugins/identifiers/schemes.py:118-250). Only ORCID and LEI have a checksum (orcid_checks_out at :95, lei_checks_out at :110); ISNI has none (#637, open: the check character is never verified and its link carries spaces). #638 (open) lists values the table refuses or reads wrongly. CompanyIdentifier.clean applies the scheme and refuses what fails (src/postulo/jobs/models.py:543-549).
Nothing compares schemes. The only cross-record rule is uniqueness: one value per scheme per company, and one company per identifier per owner (jobs/models.py:512-537).
The plugin states it never looks anything up. src/postulo/plugins/identifiers/__init__.py:59 (“Nothing here is ever looked up online.”) and src/postulo/core/identifiers.py:21 (“Nothing here touches the network”); #42 deferred any lookup as “behind a deliberate per-request action and never automatically”. No code fetches from Wikidata, GLEIF, ISNI or OpenCorporates.
The pattern for a notice that tells and never refuses exists. jobs/duplicates.py works out possible duplicates when a page is drawn, with no table to keep true, and <c-duplicates> draws them as a note on the company's page (src/postulo/jobs/duplicates.py:1-27, src/postulo/templates/cotton/duplicates.html, used at src/postulo/templates/jobs/company_detail.html:71, directly above the identifiers list at :73-90).
What the company's own identifiers can be compared with, offline, is little. The schemes that could agree with each other are the register number (register, “PT 501234567”) and OpenCorporates (pt/501234567), which both carry a jurisdiction and a company number. A Wikidata item shape is already checked by the pattern (^Q[1-9]\d{0,11}$). A LEI does not encode a country: its first four characters are the issuing operator's prefix, characters 5 to 18 carry no embedded meaning, the last two are check digits (ISO 17442, as GLEIF and others describe it). The idea of checking “the LEI's country prefix” has no basis, and this issue does not do it.
What Wikidata can say, verified on 2026-10-02 (one read-only request to Special:EntityData/Q29671.json and one to the property labels): an item can hold the ISNI (P213), the LEI (P1278), the OpenCorporates id (P1320), the Crunchbase id (P2088), the LinkedIn company id (P4264), the official website (P856) and coordinates (P625). The University of Aveiro item has P213 (written without spaces), P856 and P2088 and no P1278 or P1320: an item often holds some and not the others, and anyone can edit it.

Checked against main at f09b6010b.

Proposal

Phase 1, offline, and part of this issue.

A small set of rules, each naming the schemes it compares and returning findings, in postulo.plugins.identifiers beside the schemes, exposed by postulo.core.identifiers (cross_checks()), the way the registry is. A finding is a sentence, the identifiers it concerns, and a severity that is only ever “warning”.
Rule: stored values that no longer pass their own scheme. After #637 and #638 change what is valid, rows saved earlier can fail the current rules (an ISNI with a wrong check character stays stored until its form is next saved). The rule re-runs Scheme.normalise and checksum over a company's stored rows and says which would be refused today.
Rule: register number against OpenCorporates. When a company has both and the jurisdiction in the register's two-letter country matches the OpenCorporates jurisdiction's country, the numbers (letters and digits, case-folded, separators removed) should agree; when the countries differ, say the two name different jurisdictions, which can be right for a company registered in more than one.
A note on the company's page, “Identifiers that do not fit together”, beside <c-duplicates>, in the same component shape, drawn only when there is a finding, each line saying which two identifiers and why, in words and not by colour. It never blocks saving, never changes a value and never fires in the background: it is computed when the page is drawn. The identifier rows on the form do not change.
Say on the identifiers section of the wiki what is and is not checked, so a clean page is not read as “verified”.

Phase 2, online, after #673 lands the guarded Wikidata read, and a separate issue when it does.

A Compare with Wikidata button on a company that has a Wikidata item, a deliberate act each time, through the guarded client (postulo.plugins.http, with size, time and address limits) and the same module and dialog shape as #673 and #674. It compares the item's ISNI, LEI, OpenCorporates, Crunchbase and LinkedIn ids and its website with what the company holds, shows both values, and offers to add an identifier the company lacks. It does not decide which side is right and never overwrites.
Nothing is stored from the comparison; the person decides, and the result is shown once.

Decisions

The plugin's promise. The plugin and the core module say nothing is looked up online. Recommended: they stay true. The offline rules live in the plugin; the Wikidata comparison lives in jobs beside #673's module and is not part of the registry. The plugin's description is unchanged.
Where the rules live. In the plugin, beside the schemes (they know which keys relate), or in jobs with the duplicate notice. Recommended: the plugin, because the relations are between schemes the plugin defines, with no third-party contract: the identifier kind advertises no entry-point group (plugins/identifiers/__init__.py, “Internal only, for now”), so no postulo.plugins.api name is added.
A warning, never an error. The list says “a warning be displayed”. Recommended: warnings only, never on the save path, because Wikidata is editable by anyone and a register may be right where Wikidata is wrong; the rule in jobs/duplicates.py (tell, never refuse) is the precedent.
Which side is right. Recommended: the message names both values and says they disagree; it does not say Wikidata is right. The person's own value stays.
Scope of phase 1. The maintainer's own example (a valid Wikidata item with an ISNI that is not that organisation's) cannot be found offline: nothing on the company says what its ISNI should be. Recommended: ship phase 1 as above, plainly smaller than the list suggests, and put the example into phase 2, which is where it can be caught. If the maintainer prefers a single issue, the milestone follows phase 2 (#673 is 1.0.0).
Conflict with #42's “not in the first cut”. #42 said a lookup should not validate that an item is an organisation. Recommended: still true; phase 2 is by press, on an item the person recorded.
Person identifiers. ORCID, ResearcherID, Scopus and an ISNI on a person have no sibling scheme to compare with, and an ORCID is itself an ISNI-compatible number, which is not a contradiction with a person's own ISNI. Recommended: companies only.

Prior art

No new dependency in phase 1: the rules are a few comparisons on values the registry already normalises. For the check digits that #637 adds, ISO 7064 MOD 11-2 is already hand-written for ORCID (schemes.py:95-107) and is what ISNI uses. Under the library-first rule, python-stdnum 2.2 (LGPL-2.1-or-later, which combines with AGPL-3.0; released 2026-01-04; no advisory on OSV) already has stdnum.isni, stdnum.lei and stdnum.iso7064, and is not a dependency today (absent from uv.lock). Recommended: leave that choice to #637, which adds the ISNI check and is where to decide between taking stdnum.isni and extending the short function that exists; the rules here compare values and need neither. For phase 2 the candidate clients, their licences and why none is adopted are in #673; the same document and the same guarded client are reused here. Not verified: whether OpenCorporates' and the registers' jurisdiction codes agree for every country.

Builds on

Issues #637 (ISNI check character and link), #638 (schemes that refuse real values), #109 (one registry and a subject matrix), #42 (identifiers, lookup deferred). #673 for phase 2 (the guarded Wikidata read). Related: #672 (which identifiers a person chooses to see; a warning concerns hidden ones too).

Done when

In tests/test_identifiers.py (and tests/test_identifier_registry.py for the registry hook): a company with a register number and an OpenCorporates id that agree has no finding; one with different numbers has a finding naming both; one with different countries has the jurisdiction finding; a stored ISNI with a wrong check character is reported once #637 lands; a company with one identifier has none.
The company page draws the note only when there is a finding, with the sentence and both values, passes axe in both themes (tests/e2e/test_accessibility.py), and never changes what saving does; tests/test_page_coverage.py is unchanged because no page is added.
A person's page never shows another's findings (tests/test_identifiers.py::test_another_persons_page_never_shows_my_ids is the model).
No format change: nothing new is stored, so the archive format (29 at f09b6010b) and the candidate format (4) stay as they are. New strings are in the working catalogues (en-GB, fr-FR, pt-PT as drafts) and Weblate (#349).
Phase 2, in its own issue: a test with MockTransport that the comparison goes through the guarded client and stores nothing.

From the list of improvements of 2026-10-02.

From the list: “Make the plugin perform a cross reference check and a warning be displayed if some identifiers are not coherent. For example, a company with a valid wikidata but references a wrong ISNI.” Identifiers are validated one at a time, by shape and where a scheme has one by check digit; nothing compares one identifier with another, and the plugin says it looks nothing up online. ## What exists today - **Validation is per scheme.** A scheme has a pattern, a `tidy`, and optionally a `checksum` (`src/postulo/core/identifiers.py`, `Scheme`; the table is `src/postulo/plugins/identifiers/schemes.py:118-250`). Only ORCID and LEI have a checksum (`orcid_checks_out` at `:95`, `lei_checks_out` at `:110`); ISNI has none (#637, open: the check character is never verified and its link carries spaces). #638 (open) lists values the table refuses or reads wrongly. `CompanyIdentifier.clean` applies the scheme and refuses what fails (`src/postulo/jobs/models.py:543-549`). - **Nothing compares schemes.** The only cross-record rule is uniqueness: one value per scheme per company, and one company per identifier per owner (`jobs/models.py:512-537`). - **The plugin states it never looks anything up.** `src/postulo/plugins/identifiers/__init__.py:59` (“Nothing here is ever looked up online.”) and `src/postulo/core/identifiers.py:21` (“Nothing here touches the network”); #42 deferred any lookup as “behind a deliberate per-request action and never automatically”. No code fetches from Wikidata, GLEIF, ISNI or OpenCorporates. - **The pattern for a notice that tells and never refuses exists.** `jobs/duplicates.py` works out possible duplicates when a page is drawn, with no table to keep true, and `<c-duplicates>` draws them as a note on the company's page (`src/postulo/jobs/duplicates.py:1-27`, `src/postulo/templates/cotton/duplicates.html`, used at `src/postulo/templates/jobs/company_detail.html:71`, directly above the identifiers list at `:73-90`). - **What the company's own identifiers can be compared with, offline, is little.** The schemes that could agree with each other are the register number (`register`, “PT 501234567”) and OpenCorporates (`pt/501234567`), which both carry a jurisdiction and a company number. A Wikidata item shape is already checked by the pattern (`^Q[1-9]\d{0,11}$`). **A LEI does not encode a country**: its first four characters are the issuing operator's prefix, characters 5 to 18 carry no embedded meaning, the last two are check digits (ISO 17442, as GLEIF and others describe it). The idea of checking “the LEI's country prefix” has no basis, and this issue does not do it. - **What Wikidata can say, verified on 2026-10-02** (one read-only request to `Special:EntityData/Q29671.json` and one to the property labels): an item can hold the ISNI (P213), the LEI (P1278), the OpenCorporates id (P1320), the Crunchbase id (P2088), the LinkedIn company id (P4264), the official website (P856) and coordinates (P625). The University of Aveiro item has P213 (written without spaces), P856 and P2088 and no P1278 or P1320: an item often holds some and not the others, and anyone can edit it. Checked against `main` at `f09b6010b`. ## Proposal **Phase 1, offline, and part of this issue.** 1. **A small set of rules, each naming the schemes it compares and returning findings**, in `postulo.plugins.identifiers` beside the schemes, exposed by `postulo.core.identifiers` (`cross_checks()`), the way the registry is. A finding is a sentence, the identifiers it concerns, and a severity that is only ever “warning”. 2. **Rule: stored values that no longer pass their own scheme.** After #637 and #638 change what is valid, rows saved earlier can fail the current rules (an ISNI with a wrong check character stays stored until its form is next saved). The rule re-runs `Scheme.normalise` and `checksum` over a company's stored rows and says which would be refused today. 3. **Rule: register number against OpenCorporates.** When a company has both and the jurisdiction in the register's two-letter country matches the OpenCorporates jurisdiction's country, the numbers (letters and digits, case-folded, separators removed) should agree; when the countries differ, say the two name different jurisdictions, which can be right for a company registered in more than one. 4. **A note on the company's page, “Identifiers that do not fit together”,** beside `<c-duplicates>`, in the same component shape, drawn only when there is a finding, each line saying which two identifiers and why, in words and not by colour. It **never blocks saving**, never changes a value and never fires in the background: it is computed when the page is drawn. The identifier rows on the form do not change. 5. **Say on the identifiers section of the wiki what is and is not checked**, so a clean page is not read as “verified”. **Phase 2, online, after #673 lands the guarded Wikidata read, and a separate issue when it does.** 6. **A *Compare with Wikidata* button on a company that has a Wikidata item**, a deliberate act each time, through the guarded client (`postulo.plugins.http`, with size, time and address limits) and the same module and dialog shape as #673 and #674. It compares the item's ISNI, LEI, OpenCorporates, Crunchbase and LinkedIn ids and its website with what the company holds, shows both values, and offers to add an identifier the company lacks. It does not decide which side is right and never overwrites. 7. **Nothing is stored from the comparison**; the person decides, and the result is shown once. ## Decisions - **The plugin's promise.** The plugin and the core module say nothing is looked up online. **Recommended: they stay true.** The offline rules live in the plugin; the Wikidata comparison lives in `jobs` beside #673's module and is not part of the registry. The plugin's description is unchanged. - **Where the rules live.** In the plugin, beside the schemes (they know which keys relate), or in `jobs` with the duplicate notice. **Recommended: the plugin**, because the relations are between schemes the plugin defines, with no third-party contract: the identifier kind advertises no entry-point group (`plugins/identifiers/__init__.py`, “Internal only, for now”), so no `postulo.plugins.api` name is added. - **A warning, never an error.** The list says “a warning be displayed”. **Recommended: warnings only, never on the save path**, because Wikidata is editable by anyone and a register may be right where Wikidata is wrong; the rule in `jobs/duplicates.py` (tell, never refuse) is the precedent. - **Which side is right.** **Recommended: the message names both values and says they disagree**; it does not say Wikidata is right. The person's own value stays. - **Scope of phase 1.** The maintainer's own example (a valid Wikidata item with an ISNI that is not that organisation's) cannot be found offline: nothing on the company says what its ISNI should be. **Recommended: ship phase 1 as above, plainly smaller than the list suggests, and put the example into phase 2**, which is where it can be caught. If the maintainer prefers a single issue, the milestone follows phase 2 (#673 is 1.0.0). - **Conflict with #42's “not in the first cut”.** #42 said a lookup should not validate that an item is an organisation. **Recommended: still true**; phase 2 is by press, on an item the person recorded. - **Person identifiers.** ORCID, ResearcherID, Scopus and an ISNI on a person have no sibling scheme to compare with, and an ORCID is itself an ISNI-compatible number, which is not a contradiction with a person's own ISNI. **Recommended: companies only.** ## Prior art No new dependency in phase 1: the rules are a few comparisons on values the registry already normalises. For the check digits that #637 adds, ISO 7064 MOD 11-2 is already hand-written for ORCID (`schemes.py:95-107`) and is what ISNI uses. Under the library-first rule, `python-stdnum` 2.2 (LGPL-2.1-or-later, which combines with AGPL-3.0; released 2026-01-04; no advisory on OSV) already has `stdnum.isni`, `stdnum.lei` and `stdnum.iso7064`, and is not a dependency today (absent from `uv.lock`). **Recommended: leave that choice to #637**, which adds the ISNI check and is where to decide between taking `stdnum.isni` and extending the short function that exists; the rules here compare values and need neither. For phase 2 the candidate clients, their licences and why none is adopted are in #673; the same document and the same guarded client are reused here. Not verified: whether OpenCorporates' and the registers' jurisdiction codes agree for every country. ## Builds on Issues #637 (ISNI check character and link), #638 (schemes that refuse real values), #109 (one registry and a subject matrix), #42 (identifiers, lookup deferred). #673 for phase 2 (the guarded Wikidata read). Related: #672 (which identifiers a person chooses to see; a warning concerns hidden ones too). ## Done when - In `tests/test_identifiers.py` (and `tests/test_identifier_registry.py` for the registry hook): a company with a register number and an OpenCorporates id that agree has no finding; one with different numbers has a finding naming both; one with different countries has the jurisdiction finding; a stored ISNI with a wrong check character is reported once #637 lands; a company with one identifier has none. - The company page draws the note only when there is a finding, with the sentence and both values, passes axe in both themes (`tests/e2e/test_accessibility.py`), and never changes what saving does; `tests/test_page_coverage.py` is unchanged because no page is added. - A person's page never shows another's findings (`tests/test_identifiers.py::test_another_persons_page_never_shows_my_ids` is the model). - No format change: nothing new is stored, so the archive format (29 at `f09b6010b`) and the candidate format (4) stay as they are. New strings are in the working catalogues (en-GB, fr-FR, pt-PT as drafts) and Weblate (#349). - Phase 2, in its own issue: a test with `MockTransport` that the comparison goes through the guarded client and stores nothing. From the list of improvements of 2026-10-02.

tiagoagueda added this to the 0.6.0 milestone

2026-10-03 01:58:53 +00:00

tiagoagueda added the

enhancement

label

2026-10-03 01:58:53 +00:00

tiagoagueda referenced this issue

2026-10-03 01:59:28 +00:00

Outbound requests: the documents say capture is the only one, and the container fetches GeoNames unasked from a fixed path #366

tiagoagueda referenced this issue

2026-10-03 01:59:28 +00:00

ISNI: the check character is never verified, and the link is written with spaces in it #637