Publications: suggest an entry from a DOI or from your ORCID record, only when asked, through the guarded client #688

New issue

Closed

opened 2026-10-03 01:59:02 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-03 01:59:02 +00:00

Owner

What exists today

Nothing looks up bibliographic data. There is no DOI scheme in the identifiers registry; the stored ORCID (#46, checksum-checked, src/postulo/plugins/identifiers/schemes.py:95-135) is never sent anywhere ("Nothing here asks ORCID whether an identifier exists", src/postulo/accounts/identifiers.py:21-23). The nearest network action on the career is one request per press (src/postulo/resume/links.py:35-86: 8 s, 3 redirects, "Postulo never fetches a link on its own").
The guarded client: postulo.plugins.http.public_only_client (src/postulo/plugins/http.py:206-222) refuses a non-public address on every request, redirects included, and pins the connection to the address it approved; until(deadline_in(n)) cuts every wait to a deadline; read_body(response, limit=, deadline=) (:342) stops at a byte and a time limit and unpacks only gzip and deflate (ACCEPT_ENCODING, :35). Its default User-Agent names the project, not the operator (src/postulo/plugins/public_addresses.py:29). The wiki calls it the client "for what is public by definition" (Writing-a-plugin.md:703); tests/security/test_outbound.py shows how it is tested (stubbed resolver, MockTransport).
Per-account throttling is core/throttle.py (consume, rate_for; POSTULO_CAPTURE_RATE=30/h, config/settings/base.py:411); per #366, link checks and logo fetches spend none. An operator address for third parties exists once: web push writes "the operator, never the project" (src/postulo/plugins/browser/webpush.py:107-116; DEFAULT_FROM_EMAIL, default postulo@localhost, config/settings/base.py:708). A per-person switch for a capability is a feature plugin (src/postulo/plugins/phone_numbers/__init__.py, plugins.policy.decide).
The documents say capture is the only outbound request; #366 asks for that to be corrected and for a table of every destination in Hardening. This issue adds three destinations to it.

What the services say, read on 2026-10-03.

DOI content negotiation (Crossref documentation, updated 2025-03-17): an Accept header on https://doi.org/<doi> returns one record in a chosen format (CSL JSON, BibTeX, RIS…) whichever agency registered it (Crossref, DataCite, mEDRA); "the mapping is not always direct". Observed: a Crossref DOI redirects once to api.crossref.org/v1/works/<doi>/transform, a DataCite DOI to data.crosscite.org/<doi>. A Crossref record was 21 KB, nearly all of it the reference list and abstract; a many-author physics paper's was 682,304 bytes with 5,156 authors. The returned URL is http://dx.doi.org/.... Crossref's own type (journal-article) came back, not CSL's article-journal; DataCite's was CSL (dataset): both vocabularies must be read.
Crossref REST API (updated 2025-10-16): no sign-up; public pool, no identification; polite pool, an email in the mailto parameter or agent header; Plus, a key. Limits, for a single record: public 5 requests a second and one concurrent, polite 10 and three, Plus 150 (a list query is 1 and 3 a second from 1 December 2025, per Crossref's announcement of the change; the reply said x-rate-limit-limit: 5, x-rate-limit-interval: 1s, x-api-pool: public-single); 429 when exceeded, 403 for a manual block. They ask for caching and an identifying User-Agent. An anonymous request can always be tied to an IP address, kept 90 days; contact details given in a request stay in the logs 90 days and are used only to contact whoever causes problems (rest-api-doc, api_tips.md). On reuse, the REST API page says almost none of the metadata is subject to copyright and may be used for any purpose, but some abstracts may be subject to copyright by publishers or authors.
DataCite: unidentified 500 requests per 5 minutes per IP, identified (email in User-Agent or mailto=) 1000, authenticated 3000; doi.org content negotiation 1000; 429 when exceeded. Metadata is CC0 (its data-file policy and POSI commitment).
ORCID: public data is readable through the Anonymous API (a GET https://pub.orcid.org/v3.0/<orcid>/works answered 200 JSON with no credentials) and the Public (registered) API. Limits: anonymous 12 requests a second, 40 burst, 25,000 reads a day per IP; registered 100,000 a day per client; over the burst is 503. Terms (Public APIs Terms of Service, updated 23 October 2024, which define the Anonymous API and the Public API together as the “Public APIs”): non-commercial ("may not charge any re-use fees… or make use of the public APIs in connection with any revenue-generating product or service"); credentials are "granted to individuals and not organizations", personal, not shareable; no continuous polling. A works summary has title, type, date, journal title and external ids (DOIs), and no authors.

Checked against main at f09b6010b.

Proposal

A lookup module (resume/lookups.py): from_doi(doi), one request to https://doi.org/<doi> for CSL JSON (2 MiB, 10 s in all), and from_orcid(orcid), one request to https://pub.orcid.org/v3.0/<orcid>/works asking for JSON (Accept: application/json; the default answer is XML; 4 MiB, 10 s). Both through public_only_client inside until(...) with read_body. Hosts are fixed; the only input in a URL is a DOI or an ORCID that passed its own check, percent-encoded into the path; at most 3 redirects, each approved by the guard.
What is kept is a short list, read as untrusted text: type, title, up to 100 authors (then "and others"), container, publisher, date, volume, issue, pages, DOI, language. The abstract, references, funders and links are never read into anything. Values are cut to their column's length, stripped of markup (titles carry <i> and MathML) and of control and bidirectional-override characters, and drawn escaped. The returned URL is ignored. An unknown type becomes misc, never a refusal; the type table accepts Crossref's and CSL's names.
A “Look up” button beside the DOI field (POST with CSRF, working without scripts): fills blank fields only, says which and from which service, saves nothing until the person saves. Its help says the DOI is sent to doi.org, which passes it to Crossref or DataCite, and nothing else about the person.
“Suggest from my ORCID record” on the Publications section, shown only where the person has stored an ORCID (their own, never a typed one, so it cannot read other people's lists). One request; a review page lists each public work, marks those already held (same DOI, else title and year), ticks none, and adds only what is ticked, from the summary alone (authors blank, each with the DOI button for the rest). At most 100 works shown and 20 added at once. ORCID returns only works set public.
Never automatic: no lookup on page load, typing, import, schedule or worker; no retry after a 429, 503 or timeout; nothing stored but what the person confirms.
A per-account allowance, POSTULO_LOOKUP_RATE (default 20/h), through core/throttle.consume, spent even when the service fails, as capture's is.
An operator's switch: a feature plugin publication-lookups, on by default; off hides both buttons and makes both endpoints answer as if absent. A row per destination in Hardening's table (what triggers it, whose address, which guard, which switch).
Wiki: Your career record (the buttons, what is sent, what is not kept) and Hardening.

Decisions

Which service answers a DOI. (a) doi.org content negotiation; (b) Crossref's and DataCite's own APIs, one then the other; (c) Crossref only. Recommended: (a): one request, any registrar, the use Crossref's page itself describes. Its cost is the lossy mapping (the person reviews) and two fixed hosts after the redirect, both checked by the guard. (b) means two clients and a guess about the agency.
ORCID anonymous or registered. Registered credentials belong to one individual and cannot be shared, so Postulo cannot ship them and an operator's would be personal. Recommended: the Anonymous API: nothing to store or leak, and 25,000 reads a day per IP is far above one click each.
Commercial use. ORCID's terms bar revenue-generating use. A homelab is outside that; an operator selling hosted accounts may not be. Recommended: the wiki says so beside the setting, and the switch exists; no code decides it.
Identification. The polite pool and DataCite's identified tier want an email, which the service logs. Recommended: none by default (the public pools allow a click); later, the operator's address by the web push helper's rule, only if POSTULO_DEFAULT_FROM_EMAIL is real and the operator opts in. A person's own email never leaves.
Fill blanks or overwrite. Recommended: blanks only.
ORCID: summary only or a DOI lookup per work. The latter is up to 20 requests at a click. Recommended: summary only; one outbound request per press.
Default on or off. Recommended: on, since off hides a feature the maintainer asked for; the switch and the wiki row do the explaining.
Trust. ORCID records are self-curated and DOI metadata is deposited by publishers; either can be wrong or hostile text. Recommended: suggestions only; the form is the review.

Prior art

Standards: DOI content negotiation; CSL JSON (field names read off real responses); the Crossref, DataCite and ORCID v3.0 APIs as above.
habanero (Crossref client): MIT; 2.9.2, 2026-06-17; needs httpx2 (BSD-3-Clause; OSV lists five advisories, two High, all fixed by 2.12.0), tqdm, urllib3; no advisory for habanero; not a dependency. Rejected: it opens its own connections, so the guard, pinning and body limits would not apply (docs/THREAT-MODEL.md rule 5), for what is two GETs.
pyorcid: MIT; 1.2.2, 2025-11-04; requests, python-dotenv, xmltodict; no advisory. orcid (python-orcid): BSD; 1.0.3, last released 2018-07-17. crossrefapi: 1.7.0, 2025-07-03; requests and ipython; licence not stated in its PyPI metadata (not verified); no advisory. All rejected, same reason. Every licence stated is MIT or BSD, so compatible with AGPL-3.0; the reason is the guard.
applications/ical.py and documents/docx.py are hand-written file formats and not the precedent here.

Builds on

#687 (the record), #366 (the outbound inventory), #46 (the person's ORCID), docs/THREAT-MODEL.md rule 5.

Needed by

Nothing. A DOI scheme in the identifiers registry (a work's, not a person's) is a separate idea.

Done when

A new tests/test_publication_lookup.py, every response from a MockTransport and the resolver stubbed: Crossref- and DataCite-shaped records fill blanks and leave typed values; both type vocabularies map and an unknown becomes misc; markup and control characters go; 5,000 authors become 100 and "and others"; abstract and references are never kept; the ORCID review marks held works, ticks none, adds only what is ticked, capped at 20; a person with no ORCID is not offered the button and a posted ORCID is ignored.
tests/security/test_outbound.py gains: a doi.org answer redirecting to a private address is refused, and the page says nothing of it; an oversize or trickling answer stops at its limit and deadline; the DOI is only in the path of a fixed host. tests/security/test_rate_limits.py gains the allowance (spent on failure, per account).
A switched-off publication-lookups hides both buttons and both endpoints refuse (tests/test_plugin_policy.py, extended); new pages join tests/test_page_coverage.py (as resume:link_check, :182-183) and tests/security/test_isolation_sweep.py.
Nothing new is stored: no archive or candidate change (the record's own bumps are #687's), no API, search or CV change; tests/test_wiki_surface.py untouched. New strings in the working catalogues (en-GB, fr-FR, pt-PT as drafts) and Weblate (#349); the wiki and Hardening row in the same commit.

From the list of improvements of 2026-10-02.

Typing a paper's authors, venue, volume and pages by hand is the part of adding a publication nobody wants to do. The maintainer's list says suggestions "can be obtained using ORCID or DOI search". #687 gives publications a record; this issue fills one from a public registry, by a button a person presses, never on its own. It is the first thing in Postulo that asks a scholarly service anything, so the limits matter as much as the lookup. ## What exists today - **Nothing looks up bibliographic data.** There is no DOI scheme in the identifiers registry; the stored ORCID (#46, checksum-checked, `src/postulo/plugins/identifiers/schemes.py:95-135`) is never sent anywhere ("Nothing here asks ORCID whether an identifier exists", `src/postulo/accounts/identifiers.py:21-23`). The nearest network action on the career is one request per press (`src/postulo/resume/links.py:35-86`: 8 s, 3 redirects, "Postulo never fetches a link on its own"). - **The guarded client**: `postulo.plugins.http.public_only_client` (`src/postulo/plugins/http.py:206-222`) refuses a non-public address on every request, redirects included, and pins the connection to the address it approved; `until(deadline_in(n))` cuts every wait to a deadline; `read_body(response, limit=, deadline=)` (`:342`) stops at a byte and a time limit and unpacks only gzip and deflate (`ACCEPT_ENCODING`, `:35`). Its default `User-Agent` names the project, not the operator (`src/postulo/plugins/public_addresses.py:29`). The wiki calls it the client "for what is public by definition" (`Writing-a-plugin.md:703`); `tests/security/test_outbound.py` shows how it is tested (stubbed resolver, `MockTransport`). - **Per-account throttling** is `core/throttle.py` (`consume`, `rate_for`; `POSTULO_CAPTURE_RATE=30/h`, `config/settings/base.py:411`); per #366, link checks and logo fetches spend none. **An operator address for third parties** exists once: web push writes "the operator, never the project" (`src/postulo/plugins/browser/webpush.py:107-116`; `DEFAULT_FROM_EMAIL`, default `postulo@localhost`, `config/settings/base.py:708`). **A per-person switch for a capability** is a feature plugin (`src/postulo/plugins/phone_numbers/__init__.py`, `plugins.policy.decide`). - The documents say capture is the only outbound request; #366 asks for that to be corrected and for a table of every destination in *Hardening*. This issue adds three destinations to it. **What the services say, read on 2026-10-03.** - **DOI content negotiation** (Crossref documentation, updated 2025-03-17): an `Accept` header on `https://doi.org/<doi>` returns one record in a chosen format (CSL JSON, BibTeX, RIS…) whichever agency registered it (Crossref, DataCite, mEDRA); "the mapping is not always direct". Observed: a Crossref DOI redirects once to `api.crossref.org/v1/works/<doi>/transform`, a DataCite DOI to `data.crosscite.org/<doi>`. A Crossref record was 21 KB, nearly all of it the `reference` list and abstract; a many-author physics paper's was **682,304 bytes with 5,156 authors**. The returned `URL` is `http://dx.doi.org/...`. Crossref's own `type` (`journal-article`) came back, not CSL's `article-journal`; DataCite's was CSL (`dataset`): both vocabularies must be read. - **Crossref REST API** (updated 2025-10-16): no sign-up; public pool, no identification; polite pool, an email in the `mailto` parameter or `agent` header; Plus, a key. Limits, for a single record: public 5 requests a second and one concurrent, polite 10 and three, Plus 150 (a list query is 1 and 3 a second from 1 December 2025, per Crossref's announcement of the change; the reply said `x-rate-limit-limit: 5`, `x-rate-limit-interval: 1s`, `x-api-pool: public-single`); 429 when exceeded, 403 for a manual block. They ask for caching and an identifying `User-Agent`. An anonymous request can always be tied to an IP address, kept 90 days; contact details given in a request stay in the logs 90 days and are used only to contact whoever causes problems (`rest-api-doc`, `api_tips.md`). On reuse, the REST API page says almost none of the metadata is subject to copyright and may be used for any purpose, but some abstracts may be subject to copyright by publishers or authors. - **DataCite**: unidentified 500 requests per 5 minutes per IP, identified (email in `User-Agent` or `mailto=`) 1000, authenticated 3000; doi.org content negotiation 1000; 429 when exceeded. Metadata is CC0 (its data-file policy and POSI commitment). - **ORCID**: public data is readable through the Anonymous API (a `GET https://pub.orcid.org/v3.0/<orcid>/works` answered 200 JSON with no credentials) and the Public (registered) API. Limits: anonymous 12 requests a second, 40 burst, **25,000 reads a day per IP**; registered 100,000 a day per client; over the burst is 503. Terms (Public APIs Terms of Service, updated 23 October 2024, which define the Anonymous API and the Public API together as the “Public APIs”): **non-commercial** ("may not charge any re-use fees… or make use of the public APIs in connection with any revenue-generating product or service"); credentials are "granted to individuals and not organizations", personal, not shareable; no continuous polling. A works *summary* has title, type, date, journal title and external ids (DOIs), and no authors. Checked against `main` at `f09b6010b`. ## Proposal 1. **A lookup module** (`resume/lookups.py`): `from_doi(doi)`, one request to `https://doi.org/<doi>` for CSL JSON (2 MiB, 10 s in all), and `from_orcid(orcid)`, one request to `https://pub.orcid.org/v3.0/<orcid>/works` asking for JSON (`Accept: application/json`; the default answer is XML; 4 MiB, 10 s). Both through `public_only_client` inside `until(...)` with `read_body`. Hosts are fixed; the only input in a URL is a DOI or an ORCID that passed its own check, percent-encoded into the path; at most 3 redirects, each approved by the guard. 2. **What is kept is a short list, read as untrusted text**: type, title, up to 100 authors (then "and others"), container, publisher, date, volume, issue, pages, DOI, language. The abstract, references, funders and links are never read into anything. Values are cut to their column's length, stripped of markup (titles carry `<i>` and MathML) and of control and bidirectional-override characters, and drawn escaped. The returned `URL` is ignored. An unknown type becomes `misc`, never a refusal; the type table accepts Crossref's and CSL's names. 3. **A “Look up” button beside the DOI field** (POST with CSRF, working without scripts): fills **blank fields only**, says which and from which service, saves nothing until the person saves. Its help says the DOI is sent to doi.org, which passes it to Crossref or DataCite, and nothing else about the person. 4. **“Suggest from my ORCID record”** on the Publications section, shown only where the person has stored an ORCID (their own, never a typed one, so it cannot read other people's lists). One request; a review page lists each public work, marks those already held (same DOI, else title and year), ticks none, and adds only what is ticked, **from the summary alone** (authors blank, each with the DOI button for the rest). At most 100 works shown and 20 added at once. ORCID returns only works set public. 5. **Never automatic**: no lookup on page load, typing, import, schedule or worker; no retry after a 429, 503 or timeout; nothing stored but what the person confirms. 6. **A per-account allowance**, `POSTULO_LOOKUP_RATE` (default `20/h`), through `core/throttle.consume`, spent even when the service fails, as capture's is. 7. **An operator's switch**: a feature plugin `publication-lookups`, on by default; off hides both buttons and makes both endpoints answer as if absent. A row per destination in *Hardening*'s table (what triggers it, whose address, which guard, which switch). 8. **Wiki**: *Your career record* (the buttons, what is sent, what is not kept) and *Hardening*. ## Decisions - **Which service answers a DOI.** (a) doi.org content negotiation; (b) Crossref's and DataCite's own APIs, one then the other; (c) Crossref only. **Recommended:** (a): one request, any registrar, the use Crossref's page itself describes. Its cost is the lossy mapping (the person reviews) and two fixed hosts after the redirect, both checked by the guard. (b) means two clients and a guess about the agency. - **ORCID anonymous or registered.** Registered credentials belong to one individual and cannot be shared, so Postulo cannot ship them and an operator's would be personal. **Recommended:** the Anonymous API: nothing to store or leak, and 25,000 reads a day per IP is far above one click each. - **Commercial use.** ORCID's terms bar revenue-generating use. A homelab is outside that; an operator selling hosted accounts may not be. **Recommended:** the wiki says so beside the setting, and the switch exists; no code decides it. - **Identification.** The polite pool and DataCite's identified tier want an email, which the service logs. **Recommended:** none by default (the public pools allow a click); later, the operator's address by the web push helper's rule, only if `POSTULO_DEFAULT_FROM_EMAIL` is real and the operator opts in. A person's own email never leaves. - **Fill blanks or overwrite.** **Recommended:** blanks only. - **ORCID: summary only or a DOI lookup per work.** The latter is up to 20 requests at a click. **Recommended:** summary only; one outbound request per press. - **Default on or off.** **Recommended:** on, since off hides a feature the maintainer asked for; the switch and the wiki row do the explaining. - **Trust.** ORCID records are self-curated and DOI metadata is deposited by publishers; either can be wrong or hostile text. **Recommended:** suggestions only; the form is the review. ## Prior art - **Standards**: DOI content negotiation; CSL JSON (field names read off real responses); the Crossref, DataCite and ORCID v3.0 APIs as above. - **`habanero`** (Crossref client): MIT; 2.9.2, 2026-06-17; needs `httpx2` (BSD-3-Clause; OSV lists five advisories, two High, all fixed by 2.12.0), `tqdm`, `urllib3`; no advisory for habanero; not a dependency. **Rejected**: it opens its own connections, so the guard, pinning and body limits would not apply (`docs/THREAT-MODEL.md` rule 5), for what is two GETs. - **`pyorcid`**: MIT; 1.2.2, 2025-11-04; `requests`, `python-dotenv`, `xmltodict`; no advisory. **`orcid`** (python-orcid): BSD; 1.0.3, last released 2018-07-17. **`crossrefapi`**: 1.7.0, 2025-07-03; `requests` and `ipython`; licence not stated in its PyPI metadata (not verified); no advisory. **All rejected**, same reason. Every licence stated is MIT or BSD, so compatible with AGPL-3.0; the reason is the guard. - `applications/ical.py` and `documents/docx.py` are hand-written file formats and not the precedent here. ## Builds on #687 (the record), #366 (the outbound inventory), #46 (the person's ORCID), `docs/THREAT-MODEL.md` rule 5. ## Needed by Nothing. A DOI scheme in the identifiers registry (a work's, not a person's) is a separate idea. ## Done when - A new `tests/test_publication_lookup.py`, every response from a `MockTransport` and the resolver stubbed: Crossref- and DataCite-shaped records fill blanks and leave typed values; both type vocabularies map and an unknown becomes `misc`; markup and control characters go; 5,000 authors become 100 and "and others"; abstract and references are never kept; the ORCID review marks held works, ticks none, adds only what is ticked, capped at 20; a person with no ORCID is not offered the button and a posted ORCID is ignored. - `tests/security/test_outbound.py` gains: a doi.org answer redirecting to a private address is refused, and the page says nothing of it; an oversize or trickling answer stops at its limit and deadline; the DOI is only in the path of a fixed host. `tests/security/test_rate_limits.py` gains the allowance (spent on failure, per account). - A switched-off `publication-lookups` hides both buttons and both endpoints refuse (`tests/test_plugin_policy.py`, extended); new pages join `tests/test_page_coverage.py` (as `resume:link_check`, `:182-183`) and `tests/security/test_isolation_sweep.py`. - Nothing new is stored: no archive or candidate change (the record's own bumps are #687's), no API, search or CV change; `tests/test_wiki_surface.py` untouched. New strings in the working catalogues (en-GB, fr-FR, pt-PT as drafts) and Weblate (#349); the wiki and *Hardening* row in the same commit. From the list of improvements of 2026-10-02.

tiagoagueda added this to the 1.0.0 milestone

2026-10-03 01:59:02 +00:00

tiagoagueda added the

enhancement

label

2026-10-03 01:59:02 +00:00

tiagoagueda referenced this issue

2026-10-03 01:59:08 +00:00

Your career: citations, read as how often your published work has been cited, and when it was counted #697

tiagoagueda referenced this issue

2026-10-03 01:59:28 +00:00

Outbound requests: the documents say capture is the only one, and the container fetches GeoNames unasked from a fixed path #366