Europass import: a JSON address with no country, an out-of-range year or an over-long number ends in a server error #634

New issue

Closed

opened 2026-10-01 10:48:32 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-01 10:48:32 +00:00

Owner

Found on 2026-09-29 in a read-only review of postulo at 95348908d, running the Europass reader on files with missing and out-of-range values.

Checked again on 2026-10-01 against main at 0a5cad587: still present. The code is now at plugins/europass/reader.py:134-136 (_json_country_code, called from :656; its XML twin _country_code is at :122-131) and :578-587 (read_json), and at plugins/europass/common.py:68-77 (_make_date) and :88-94 (_orcid_from). resume/views.py:441-461 is unchanged.

The reader's docstring promises that "no key is assumed to be present, and no value is assumed to have the type it should", and the import view catches only ImportRefused. Four kinds of input raise something else, so the upload answers 500 and the person is told nothing.

Where: resume/views.py:441-461 (EuropassImportView.post, whose only handler is except base.ImportRefused), and in plugins/europass/reader.py:
- :180-182 (_json_country_code), called from :780;
- :704-711 (read_json);
- :156-165 (_make_date);
- :197-203 (_orcid_from).
What happens:
1. An address with no country. _json_country_code calls _obj(address.get("Country")), which is None when the address has no Country, or when it is null or a string, and then None.get("Code"). A SkillsPassport JSON whose address is {"Contact": {"Municipality": "Lisboa"}} raises AttributeError: 'NoneType' object has no attribute 'get' at reader.py:182. The XML twin, _country_code (:168-177), checks for None first.
2. An over-long number. read_json catches json.JSONDecodeError and RecursionError. A number longer than Python's integer conversion limit raises a plain ValueError from json.loads ("Exceeds the limit (4300 digits) for integer string conversion").
3. A year no date can hold. _make_date catches TypeError and ValueError. A JSON year of Infinity or 1e400 (which json.loads accepts) raises OverflowError from int() at :157. A year too large for a C integer, for example 99999999999, raises OverflowError from dt.date at :163; this one is reachable from the XML as well (<From year="99999999999"/>).
4. A website address urlsplit rejects. _orcid_from catches ValidationError; a website containing orcid.org with an unbalanced [ in its host raises ValueError: Invalid IPv6 URL from identifiers.clean, in the JSON and the XML alike.
All four were run against the snapshot. The first is the plausible one in a real file; the others need an unusual or hand-edited file. Nothing is written in any of them.
Why it matters: a file that is half right should import the half that is right, as the docstring says. A 500 gives the person no sentence to act on.
Fix: in _json_country_code, use _obj(...) or {} (or _json_text(address, "Country", "Code")). In read_json, catch ValueError (which covers JSONDecodeError) and refuse with the existing "not readable JSON" sentence. In _make_date, catch OverflowError in both try blocks. In _orcid_from, catch ValueError as well as ValidationError.

Two other findings of this review touch the same paths: the import view catching only ImportRefused (a catch-all around read() would turn these into a message), and core/identifiers.py letting urlsplit's ValueError through (which is the root of case 4). These fixes make the reader keep its own promise whatever happens to those.

Done when

tests/test_europass.py has one test per case (an address without a country, a 5000-digit number, a year of Infinity, a year of 99999999999 in JSON and in XML, a website with an unbalanced bracket). Each fails today with an exception other than ImportRefused; afterwards each reads what it can or is refused with a sentence.

Found on 2026-09-29 in a read-only review of postulo at 95348908d, running the Europass reader on files with missing and out-of-range values. Checked again on 2026-10-01 against `main` at `0a5cad587`: still present. The code is now at `plugins/europass/reader.py:134-136` (`_json_country_code`, called from `:656`; its XML twin `_country_code` is at `:122-131`) and `:578-587` (`read_json`), and at `plugins/europass/common.py:68-77` (`_make_date`) and `:88-94` (`_orcid_from`). `resume/views.py:441-461` is unchanged. The reader's docstring promises that "no key is assumed to be present, and no value is assumed to have the type it should", and the import view catches only `ImportRefused`. Four kinds of input raise something else, so the upload answers 500 and the person is told nothing. - **Where:** `resume/views.py:441-461` (`EuropassImportView.post`, whose only handler is `except base.ImportRefused`), and in `plugins/europass/reader.py`: - `:180-182` (`_json_country_code`), called from `:780`; - `:704-711` (`read_json`); - `:156-165` (`_make_date`); - `:197-203` (`_orcid_from`). - **What happens:** 1. **An address with no country.** `_json_country_code` calls `_obj(address.get("Country"))`, which is `None` when the address has no `Country`, or when it is `null` or a string, and then `None.get("Code")`. A SkillsPassport JSON whose address is `{"Contact": {"Municipality": "Lisboa"}}` raises `AttributeError: 'NoneType' object has no attribute 'get'` at `reader.py:182`. The XML twin, `_country_code` (`:168-177`), checks for `None` first. 2. **An over-long number.** `read_json` catches `json.JSONDecodeError` and `RecursionError`. A number longer than Python's integer conversion limit raises a plain `ValueError` from `json.loads` ("Exceeds the limit (4300 digits) for integer string conversion"). 3. **A year no date can hold.** `_make_date` catches `TypeError` and `ValueError`. A JSON year of `Infinity` or `1e400` (which `json.loads` accepts) raises `OverflowError` from `int()` at `:157`. A year too large for a C integer, for example `99999999999`, raises `OverflowError` from `dt.date` at `:163`; this one is reachable from the XML as well (`<From year="99999999999"/>`). 4. **A website address `urlsplit` rejects.** `_orcid_from` catches `ValidationError`; a website containing `orcid.org` with an unbalanced `[` in its host raises `ValueError: Invalid IPv6 URL` from `identifiers.clean`, in the JSON and the XML alike. All four were run against the snapshot. The first is the plausible one in a real file; the others need an unusual or hand-edited file. Nothing is written in any of them. - **Why it matters:** a file that is half right should import the half that is right, as the docstring says. A 500 gives the person no sentence to act on. - **Fix:** in `_json_country_code`, use `_obj(...) or {}` (or `_json_text(address, "Country", "Code")`). In `read_json`, catch `ValueError` (which covers `JSONDecodeError`) and refuse with the existing "not readable JSON" sentence. In `_make_date`, catch `OverflowError` in both `try` blocks. In `_orcid_from`, catch `ValueError` as well as `ValidationError`. Two other findings of this review touch the same paths: the import view catching only `ImportRefused` (a catch-all around `read()` would turn these into a message), and `core/identifiers.py` letting `urlsplit`'s `ValueError` through (which is the root of case 4). These fixes make the reader keep its own promise whatever happens to those. ## Done when - `tests/test_europass.py` has one test per case (an address without a country, a 5000-digit number, a year of `Infinity`, a year of `99999999999` in JSON and in XML, a website with an unbalanced bracket). Each fails today with an exception other than `ImportRefused`; afterwards each reads what it can or is refused with a sentence.

tiagoagueda added this to the 0.6.0 milestone

2026-10-01 10:48:32 +00:00

tiagoagueda added the

bug

tier

labels

2026-10-01 10:48:32 +00:00

tiagoagueda referenced this issue from a commit

2026-10-04 00:24:28 +00:00

Europass import: a half-right file is read or refused, never a server error

tiagoagueda closed this issue

2026-10-04 00:24:28 +00:00