Europass import: the SkillsPassport readers keep the HTML markup that the Candidate reader strips #563

New issue

Closed

opened 2026-10-01 10:47:40 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-01 10:47:40 +00:00

Owner

Found on 2026-09-29 in a read-only review of postulo at 95348908d, comparing the three Europass readers and running the SkillsPassport ones on a file with markup in it.

Checked again on 2026-10-01 against main at 0a5cad587: still present. The code is now at plugins/europass/reader.py:398 (experience summary), :430 (education highlights), :461-462 (skill blocks) and :479 (achievements), with the JSON equivalents at :681, :705, :736 and :746. _json_text is at reader.py:532-533, _plain at plugins/europass/candidate.py:245-262, and _text moved to plugins/europass/common.py:146-157, the module both readers import since #248 and the place for _Words and _plain in the fix below.

The Candidate reader passes every long field through _plain (plugins/europass/candidate.py:244-261), which turns the editor's escaped HTML into lines of text, because "markup kept as it came would print on a CV as angle brackets". The SkillsPassport XML and JSON readers do not: they read the same kind of fields through _text and _json_text, which only collapse whitespace. Cedefop's editor wrote markup into these fields too; its own example file, europass-cl-cv-example-v3.3.0.xml in the europass/rest-examples-java repository, has escaped   in the work experience Activities, the education Activities and the communication skills Description. The module docstring promises that "a career arrives the same way from either" format.

Where: plugins/europass/reader.py:522 (experience summary) and :554 (education highlights), :585-586 (skill blocks), :603 (achievements); the JSON equivalents at :805, :829, :860 and :870. The helpers are _text (reader.py:368-379) and _json_text (:656-657).
What happens: with the fixture tests/data/europass.xml changed so that an Activities holds Evaluation of support measures and young people, reader.read() returns the summary Evaluation of support measures and young people. With the Organisational skills Label holding an escaped <ul><li>Mentoring</li><li>Planning</li></ul>, the group comes back as one skill, '<ul><li>Mentoring</li><li>Planning</li></ul>', because _split_skills splits only on newlines and semicolons. The JSON reader does the same with …<ul><li>…</li></ul> in Activities. The same file read through the Candidate path would come back as plain lines.
Why it matters: somebody rescuing a pre-2020 Europass CV (the reason the reader exists) gets literal tags in every experience and education entry and in their skills, which then print on every CV they make from that career unless they find and clean each field by hand. A skill block written as a list also arrives as one long "skill".
Fix: move _Words and _plain out of candidate.py into a small module of the europass package that both readers import, and apply _plain to the raw text of every SkillsPassport free-text field above, in the XML reader and the JSON reader, before _split_skills where a block is split. Keep the whitespace collapsing for short fields such as titles and names.

Related to #318, which checks the Candidate reader against a real download and does not touch the SkillsPassport readers.

Done when

tests/test_europass.py reads a SkillsPassport XML fixture and a JSON fixture whose Activities and a skills Description carry escaped ,  , <ul> and <li>, and asserts plain lines with no < in the summary, the highlights and each skill, and that a listed skill block splits into one skill per item.

Found on 2026-09-29 in a read-only review of postulo at 95348908d, comparing the three Europass readers and running the SkillsPassport ones on a file with markup in it. Checked again on 2026-10-01 against `main` at `0a5cad587`: still present. The code is now at `plugins/europass/reader.py:398` (experience summary), `:430` (education highlights), `:461-462` (skill blocks) and `:479` (achievements), with the JSON equivalents at `:681`, `:705`, `:736` and `:746`. `_json_text` is at `reader.py:532-533`, `_plain` at `plugins/europass/candidate.py:245-262`, and `_text` moved to `plugins/europass/common.py:146-157`, the module both readers import since #248 and the place for `_Words` and `_plain` in the fix below. The Candidate reader passes every long field through `_plain` (`plugins/europass/candidate.py:244-261`), which turns the editor's escaped HTML into lines of text, because "markup kept as it came would print on a CV as angle brackets". The SkillsPassport XML and JSON readers do not: they read the same kind of fields through `_text` and `_json_text`, which only collapse whitespace. Cedefop's editor wrote markup into these fields too; its own example file, `europass-cl-cv-example-v3.3.0.xml` in the `europass/rest-examples-java` repository, has escaped ` ` in the work experience `Activities`, the education `Activities` and the communication skills `Description`. The module docstring promises that "a career arrives the same way from either" format. - **Where:** `plugins/europass/reader.py:522` (experience summary) and `:554` (education highlights), `:585-586` (skill blocks), `:603` (achievements); the JSON equivalents at `:805`, `:829`, `:860` and `:870`. The helpers are `_text` (`reader.py:368-379`) and `_json_text` (`:656-657`). - **What happens:** with the fixture `tests/data/europass.xml` changed so that an `Activities` holds `Evaluation of support measures and young people`, `reader.read()` returns the summary `Evaluation of support measures and young people`. With the Organisational skills `Label` holding an escaped `<ul><li>Mentoring</li><li>Planning</li></ul>`, the group comes back as one skill, `'<ul><li>Mentoring</li><li>Planning</li></ul>'`, because `_split_skills` splits only on newlines and semicolons. The JSON reader does the same with `…<ul><li>…</li></ul>` in `Activities`. The same file read through the Candidate path would come back as plain lines. - **Why it matters:** somebody rescuing a pre-2020 Europass CV (the reason the reader exists) gets literal tags in every experience and education entry and in their skills, which then print on every CV they make from that career unless they find and clean each field by hand. A skill block written as a list also arrives as one long "skill". - **Fix:** move `_Words` and `_plain` out of `candidate.py` into a small module of the `europass` package that both readers import, and apply `_plain` to the raw text of every SkillsPassport free-text field above, in the XML reader and the JSON reader, before `_split_skills` where a block is split. Keep the whitespace collapsing for short fields such as titles and names. Related to #318, which checks the Candidate reader against a real download and does not touch the SkillsPassport readers. ## Done when - `tests/test_europass.py` reads a SkillsPassport XML fixture and a JSON fixture whose `Activities` and a skills `Description` carry escaped ``, ` `, `<ul>` and `<li>`, and asserts plain lines with no `<` in the summary, the highlights and each skill, and that a listed skill block splits into one skill per item.

tiagoagueda added this to the 0.5.0 milestone

2026-10-01 10:47:40 +00:00

tiagoagueda added the

bug

tier

labels

2026-10-01 10:47:40 +00:00

tiagoagueda referenced this issue from a commit

2026-10-04 10:04:57 +00:00

Strip the markup from the older Europass readers

tiagoagueda closed this issue

2026-10-04 10:04:57 +00:00