Identifiers: accented LinkedIn names and newer ResearcherIDs are refused; a pasted Scopus link stores a wrong ID #638

New issue

Closed

opened 2026-10-01 10:48:34 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-01 10:48:34 +00:00

Owner

Found on 2026-09-29 in a read-only review of postulo at 95348908d, running each built-in identifier scheme against the forms of a value people paste.

Several schemes in the built-in table refuse a real value, or read a pasted address wrongly. Most of them cost a person a refusal they cannot explain; the Scopus one stores an identifier that is not theirs, without a warning.

Checked again on 2026-10-01 against main at 0a5cad587: still present. plugins/identifiers/schemes.py is unchanged; value_from_url is now at core/identifiers.py:128-139.

1. LinkedIn: a name with an accent is refused

Where: plugins/identifiers/schemes.py:178-189. The pattern is ^[a-z0-9][a-z0-9._\-]{0,99}$, and value_from_url (core/identifiers.py:123-134) lifts the slug from urlsplit(url).path without percent-decoding it.
What happens: https://www.linkedin.com/in/jo%C3%A3o-silva-1a2b3c/ normalises to jo%c3%a3o-silva-1a2b3c and is refused with "That does not look like a LinkedIn identifier". The same address written with ã, the bare slug joão-silva-1a2b3c and https://www.linkedin.com/company/cr%C3%A9dit-agricole/ are refused too. LinkedIn builds a profile's default address from the person's name, accents included, so this is the address many people outside the English-speaking world have.
Fix: percent-decode the lifted tail in value_from_url, and let the pattern accept letters of any script ([^\W_] for the first character, [\w.\-] after it). Percent-encode the value again where url_for builds the link.

2. ResearcherID: two- and three-letter prefixes, and the scheme's own link, are refused

Where: plugins/identifiers/schemes.py:138-146. The pattern is ^[A-Z]-\d{4}-\d{4}$, and the scheme has no hosts or url_paths.
What happens: AAB-1234-2020 and HGU-0221-2022 are refused. Web of Science has issued identifiers with two and three letters before the digits for several years; Wikidata's format constraint for the property (P1053) allows one to three. Because the scheme lifts nothing from a URL, pasting the address the scheme itself links to, https://www.webofscience.com/wos/author/record/J-7586-2012, is upper-cased whole and refused.
Fix: ^[A-Z]{1,3}-\d{4}-\d{4}$, with hosts=("webofscience.com", "researcherid.com") and url_paths=("/record/", "/rid/").

3. Scopus: digits from the rest of a pasted link are appended to the ID

Where: plugins/identifiers/schemes.py:147-155 and _digits_only at :79-80. The scheme has no hosts or url_paths, so a pasted link reaches tidy whole, and tidy deletes every character that is not a digit. The pattern accepts ten or eleven digits.
What happens: https://www.scopus.com/authid/detail.uri?authorId=7004212771&origin=1 is stored as 70042127711. The form of the link ORCID shows, …/inward/authorDetails.uri?authorID=7004212771&partnerID=MN8TOARS, is stored as 70042127718. Both pass the pattern, so the person sees no error and their CV links to an author ID that is not theirs. A link with no other digit in it happens to work.
Fix: when the value is an address on scopus.com, take the authorId (or authorID) query parameter and nothing else; tidy then removes only spaces. This needs value_from_url to be able to read a query parameter, or a lift callable on the scheme.

4. A fragment is appended to the path for every scheme

Where: core/identifiers.py:128-129: haystack = parts.path + ("#" + parts.fragment if parts.fragment else ""). The comment says it is there for GLEIF.
What happens: https://www.wikidata.org/wiki/Q95#P31, the address the browser shows after following a link to a statement, lifts Q95#P31 and is refused.
Fix: append the fragment only for a scheme that asks for it (a flag on the LEI scheme).

5. `\d` accepts digits that are not ASCII

Where: the patterns at plugins/identifiers/schemes.py:125, :141, :150, :160, :170 and :194 use \d on str patterns compiled without re.ASCII.
What happens: a Scopus ID typed in full-width digits (７００４２１２７７１, what a Japanese input method produces) or Arabic-Indic digits passes the pattern and is stored as typed. So does an LEI whose check digits are full-width, and an ORCID with full-width digits anywhere but its last place: both checksums pass, because int() reads those digits. The stored value builds a link that does not resolve, and it is not equal to the same identifier written in ASCII, so one_company_per_identifier_per_owner (jobs/models.py:525) does not see an LEI typed both ways as one.
Fix: fold digits to ASCII in Scheme.normalise (unicodedata.normalize("NFKC", …) covers the full-width forms; str(int(char)) per digit covers the rest), or compile the patterns with re.ASCII so the value is refused.

Why it matters: an identifier is typed once and printed on every CV. A refusal of a correct value sends the person to Other, which has no link; a wrong Scopus ID is printed as theirs.

Related to #311, which lets an administrator define schemes in JSON: these corrections belong to the built-in table and to value_from_url, whichever form the table takes.

Done when

tests/test_person_identifiers.py has a parametrised test that fails today: the two LinkedIn addresses above clean to joão-silva-1a2b3c; AAB-1234-2020 and the Web of Science address clean to their IDs; both Scopus links clean to 7004212771; a Scopus ID in full-width digits cleans to ASCII or is refused.
tests/test_identifiers.py has a test that https://www.wikidata.org/wiki/Q95#P31 cleans to Q95 for a company, and that the GLEIF address https://search.gleif.org/#/record/HWUPKR0MPOU8FGXBT394 still cleans to the LEI.

Found on 2026-09-29 in a read-only review of postulo at 95348908d, running each built-in identifier scheme against the forms of a value people paste. Several schemes in the built-in table refuse a real value, or read a pasted address wrongly. Most of them cost a person a refusal they cannot explain; the Scopus one stores an identifier that is not theirs, without a warning. Checked again on 2026-10-01 against `main` at `0a5cad587`: still present. `plugins/identifiers/schemes.py` is unchanged; `value_from_url` is now at `core/identifiers.py:128-139`. ## 1. LinkedIn: a name with an accent is refused - **Where:** `plugins/identifiers/schemes.py:178-189`. The pattern is `^[a-z0-9][a-z0-9._\-]{0,99}$`, and `value_from_url` (`core/identifiers.py:123-134`) lifts the slug from `urlsplit(url).path` without percent-decoding it. - **What happens:** `https://www.linkedin.com/in/jo%C3%A3o-silva-1a2b3c/` normalises to `jo%c3%a3o-silva-1a2b3c` and is refused with "That does not look like a LinkedIn identifier". The same address written with `ã`, the bare slug `joão-silva-1a2b3c` and `https://www.linkedin.com/company/cr%C3%A9dit-agricole/` are refused too. LinkedIn builds a profile's default address from the person's name, accents included, so this is the address many people outside the English-speaking world have. - **Fix:** percent-decode the lifted tail in `value_from_url`, and let the pattern accept letters of any script (`[^\W_]` for the first character, `[\w.\-]` after it). Percent-encode the value again where `url_for` builds the link. ## 2. ResearcherID: two- and three-letter prefixes, and the scheme's own link, are refused - **Where:** `plugins/identifiers/schemes.py:138-146`. The pattern is `^[A-Z]-\d{4}-\d{4}$`, and the scheme has no `hosts` or `url_paths`. - **What happens:** `AAB-1234-2020` and `HGU-0221-2022` are refused. Web of Science has issued identifiers with two and three letters before the digits for several years; Wikidata's format constraint for the property (P1053) allows one to three. Because the scheme lifts nothing from a URL, pasting the address the scheme itself links to, `https://www.webofscience.com/wos/author/record/J-7586-2012`, is upper-cased whole and refused. - **Fix:** `^[A-Z]{1,3}-\d{4}-\d{4}$`, with `hosts=("webofscience.com", "researcherid.com")` and `url_paths=("/record/", "/rid/")`. ## 3. Scopus: digits from the rest of a pasted link are appended to the ID - **Where:** `plugins/identifiers/schemes.py:147-155` and `_digits_only` at `:79-80`. The scheme has no `hosts` or `url_paths`, so a pasted link reaches `tidy` whole, and `tidy` deletes every character that is not a digit. The pattern accepts ten or eleven digits. - **What happens:** `https://www.scopus.com/authid/detail.uri?authorId=7004212771&origin=1` is stored as `70042127711`. The form of the link ORCID shows, `…/inward/authorDetails.uri?authorID=7004212771&partnerID=MN8TOARS`, is stored as `70042127718`. Both pass the pattern, so the person sees no error and their CV links to an author ID that is not theirs. A link with no other digit in it happens to work. - **Fix:** when the value is an address on `scopus.com`, take the `authorId` (or `authorID`) query parameter and nothing else; `tidy` then removes only spaces. This needs `value_from_url` to be able to read a query parameter, or a `lift` callable on the scheme. ## 4. A fragment is appended to the path for every scheme - **Where:** `core/identifiers.py:128-129`: `haystack = parts.path + ("#" + parts.fragment if parts.fragment else "")`. The comment says it is there for GLEIF. - **What happens:** `https://www.wikidata.org/wiki/Q95#P31`, the address the browser shows after following a link to a statement, lifts `Q95#P31` and is refused. - **Fix:** append the fragment only for a scheme that asks for it (a flag on the LEI scheme). ## 5. `\d` accepts digits that are not ASCII - **Where:** the patterns at `plugins/identifiers/schemes.py:125`, `:141`, `:150`, `:160`, `:170` and `:194` use `\d` on `str` patterns compiled without `re.ASCII`. - **What happens:** a Scopus ID typed in full-width digits (`７００４２１２７７１`, what a Japanese input method produces) or Arabic-Indic digits passes the pattern and is stored as typed. So does an LEI whose check digits are full-width, and an ORCID with full-width digits anywhere but its last place: both checksums pass, because `int()` reads those digits. The stored value builds a link that does not resolve, and it is not equal to the same identifier written in ASCII, so `one_company_per_identifier_per_owner` (`jobs/models.py:525`) does not see an LEI typed both ways as one. - **Fix:** fold digits to ASCII in `Scheme.normalise` (`unicodedata.normalize("NFKC", …)` covers the full-width forms; `str(int(char))` per digit covers the rest), or compile the patterns with `re.ASCII` so the value is refused. **Why it matters:** an identifier is typed once and printed on every CV. A refusal of a correct value sends the person to *Other*, which has no link; a wrong Scopus ID is printed as theirs. Related to #311, which lets an administrator define schemes in JSON: these corrections belong to the built-in table and to `value_from_url`, whichever form the table takes. ## Done when - `tests/test_person_identifiers.py` has a parametrised test that fails today: the two LinkedIn addresses above clean to `joão-silva-1a2b3c`; `AAB-1234-2020` and the Web of Science address clean to their IDs; both Scopus links clean to `7004212771`; a Scopus ID in full-width digits cleans to ASCII or is refused. - `tests/test_identifiers.py` has a test that `https://www.wikidata.org/wiki/Q95#P31` cleans to `Q95` for a company, and that the GLEIF address `https://search.gleif.org/#/record/HWUPKR0MPOU8FGXBT394` still cleans to the LEI.

tiagoagueda added this to the 0.6.0 milestone

2026-10-01 10:48:34 +00:00

tiagoagueda added the

bug

tier

labels

2026-10-01 10:48:34 +00:00

tiagoagueda referenced this issue from a commit

2026-10-01 23:12:36 +00:00

Put a web link on a service, with its icon, and check the address against it

tiagoagueda referenced this issue

2026-10-03 01:58:53 +00:00