Basic Markdown in a listing's details, rendered on the server and sanitised #665

New issue

Closed

opened 2026-10-03 01:58:44 +00:00 by tiagoagueda · 1 comment

tiagoagueda commented

2026-10-03 01:58:44 +00:00

Owner

What exists today

The description is plain text. JobPosting.description is a TextField (src/postulo/jobs/models.py:898). The form's help text promises it "is kept exactly as pasted" (jobs/forms.py:71-74); the field is a ten-row Textarea (jobs/forms.py:715-720), and the capture review form edits the same value (templates/jobs/capture_review.html:150, jobs/capture_views.py:250).
Where it is shown. The listing page prints it escaped inside whitespace-pre-line (templates/jobs/posting_detail.html:61-62); the Listings table prints it clamped to two lines (templates/jobs/partials/listing_row.html:80-84); search takes an excerpt from it (core/search.py:165-178); the API returns it on the detail only (api/schemas.py:1012); the archive carries it as description (core/export.py:208, POSTING_FIELDS); CSV import appends notes to it (core/csv_import.py:991).
What a capture puts there is flattened text, capped. The parsers' schema truncates at 40,000 characters and adds "[…truncated]" (plugins/base.py:30, :91-92). It is text read off somebody else's page, not Markdown, so a line that begins with -, *, # or a number is, in the reader's mind, a line and not markup.
No Markdown exists in the project. Nothing under src/postulo parses it, and uv.lock holds none of markdown-it-py, mistune, nh3, bleach or markdown. #480 (open) lists Markdown as a candidate export format, which is the other direction.
The policy the page runs under. SECURE_CSP allows scripts and styles only from 'self' (plus a nonce for style on one kind of page, the CV and letter previews) and images from 'self' and data: (config/settings/base.py:671-698). Nothing here may need script, and a rendered image must never reach another host. The project already treats untrusted markup as two halves of one defence (core/pictures.py for SVG, core/files.py:51-62).
Where the project hand-writes a standard, and why. applications/ical.py:1-12 and documents/docx.py:1-12 write a format out from structured data in a page of code, to avoid a dependency that would "bring the rest of the standard along" or a C extension; docs/PLAN.md:132-136 kept BeautifulSoup and lxml out for the same reason. Both are writers of a fixed shape. This is a parser of arbitrary input whose output is HTML, and so the precedent points the other way here.

Checked against main at f09b6010b.

Proposal

A per-listing format. JobPosting.description_format with choices Plain text and Markdown, default Plain text (a migration with no data change). Existing and captured text stays plain; a person who wants Markdown on one listing chooses it on the listing form and on the capture review form, whose help text says what is supported.
One renderer, in one module (core/markdown.py, so a later field can use it): render(text) -> SafeString and plain(text) -> str. Parser: markdown-it-py with the zero preset, html off, linkify off, breaks on, and exactly these rules enabled: paragraphs, line breaks, emphasis and strong emphasis, inline code, ordered and unordered lists, block quotes, links (inline, reference and <https://…>), headings. Not enabled: raw HTML, images, tables, task lists, footnotes, fenced code, autolinking of bare addresses. Headings are shifted down so they cannot compete with the page's own: # renders as <h3>, and nothing above <h6>.
A second pass over the output with an allowlist (nh3): p br strong em code ul ol li blockquote h3 h4 h5 h6 a; href on a only, with the schemes http, https and mailto; rel="noopener noreferrer external" and target="_blank" forced, as the page's existing "Original posting" link has (posting_detail.html:146). The CSP is the third layer and is not relied on.
A style for it. The rendered block carries one class, styled in assets/css/app.css with the same tokens as the rest (list markers, link underline in the brand colour, heading sizes, quote border), then npm run build:css and the committed stylesheet. Plain text keeps whitespace-pre-line.
Everything that is not the listing page gets text, not HTML. The table cell and the search excerpt use plain(text) (the same parse, rendered as text, so ** and [x](y) markers do not show). The API returns the stored source and a new description_format; it does not return HTML (a client would have to sanitise it again). The archive carries description_format.
Wording. The form help (jobs/forms.py:71) says what Markdown is supported and that the text is stored as typed; the wiki's Tracking applications gets a short section.

Decisions

A flag, or Markdown for every listing. Options: always render Markdown; render it only where the person chose it. Recommended: the flag, default plain. Captured adverts are flattened text, and rendering *, # or 1) at the start of a line as markup changes what hundreds of existing listings say. The cost is a second field on the form.
Which syntax. Options: CommonMark's core only; GitHub-flavoured (tables, task lists, autolinks, strikethrough). Recommended: the core subset in part 2. The note says "basic"; a table in an advert's description is rare, and each extension is a rule to keep sanitised.
Headings. Allowed but shifted (above), or not offered. Recommended: allowed and shifted: an advert has sections, and a heading that outranks the page's own breaks the outline for a screen reader.
Images. None. img-src includes data:, and a remote image would tell the host who looked; the project's rule is that a page makes no request on a reader's behalf (accounts/avatars.py:1-10).
A sanitiser as well as a parser. Options: markdown-it-py with HTML off (its documented safe mode) and nothing more; plus nh3 over the output. Recommended: both. With raw HTML off the second pass should never change anything, and that is the reason it is worth having: it holds if somebody later turns html on or adds a plugin, as the SVG allowlist and the file policy do for pictures. Conflict to record: docs/PLAN.md:132-136 kept C extensions out of the core, and nh3 is a compiled (Rust) wheel. The core already depends on compiled wheels (Pillow, cryptography, psycopg-binary, brotli, zopfli, in uv.lock), so the argument that was made about lxml in M4 has not been applied since; if the maintainer would rather not add one, the fallback is the parser alone with the tests below.
Preview. A toolbar or a live preview needs script. Recommended: none in this issue; if wanted, a Preview button that posts the form and renders on the server is its own issue.
Hand-written subset. See Prior art. Recommended: not hand-written.

Prior art

Versions and advisories checked on 2026-10-02 against PyPI and OSV (api.osv.dev). OSV lists advisories over all versions; the second number is those that affect the latest release.

markdown-it-py (https://github.com/executablebooks/markdown-it-py): a CommonMark parser, a port of markdown-it. MIT. 4.2.0, released 2026-05-07. Pure Python, requires Python 3.10 or later, one dependency (mdurl, MIT, 0.1.2 from 2022-08-14, pure Python). 4 entries on record; the two GitHub advisories are denial-of-service issues fixed in 2.2.0 (2023); 0 affect 4.2.0. Its documentation says the default commonmark preset enables HTML and is "less secure", that the js-default preset is recommended for untrusted input, and that it blocks javascript:, vbscript:, file: and data: links (with limited image exceptions). Not a dependency today. Adopt, configured as in part 2 (not from a preset that enables HTML).
mistune (https://github.com/lepture/mistune): BSD-3-Clause; 3.3.4, released 2026-07-22; pure Python. 40 advisories on record, among them XSS through percent-encoded javascript: addresses in safe_url() (published 2026-07-20, fixed in 3.3.0) and several directive and denial-of-service issues; 0 affect 3.3.4. A record of cross-site scripting in its URL handling that recent is the reason to prefer the other parser. Not adopted.
Python-Markdown (markdown) (BSD-3-Clause; 3.11, released 2026-09-25; 2 advisories, fixed in 3.8.1): passes raw HTML through by default (not re-checked here against its current documentation), so it would depend on the sanitiser for everything. Not adopted.
cmarkgfm (MIT; 2025.10.22; C extension over cmark-gfm; 0 advisories) and commonmark (BSD-3-Clause; 0.9.2; 0 advisories): the first is a compiled dependency with nothing the parser above lacks; the second's maintenance state is not verified. Not adopted.
nh3 (https://github.com/messense/nh3): Python bindings for the Rust crate ammonia. MIT (the Rust crate ammonia: MIT or Apache-2.0). 0.3.7, released 2026-08-23; 0 advisories on OSV for the package (RustSec for ammonia itself was not checked: not verified). Wheels exist for manylinux on x86_64 and aarch64 (abi3, Python 3.8 and later), which is what the Debian-based image installs, so it needs no compiler. No runtime dependencies. Not a dependency today. Adopt as the second layer, subject to the decision above.
bleach: 6.4.0, released 2026-06-05. Its README now says that Bleach "is no longer maintained" with "no future releases including for security issues", and the repository was archived on 2026-06-10. 13 advisories on record, 0 affect 6.4.0. Rejected: nothing will fix the next one.
markdown2: MIT; 2.5.5; 8 advisories on record, several of them cross-site scripting. Not adopted.
Hand-written (a few regular expressions for bold, italic, lists and links, then escaping): what ical.py and docx.py do for their formats. The rule the maintainer set is to look for a library first; the library is there, maintained and clean, and a hand-written Markdown-to-HTML path is exactly where cross-site scripting is introduced. Rejected.

Builds on

#480 (Markdown as an export format, the other direction), #264 (an SVG sanitiser and a hardened file policy as two halves of one defence).

Needed by

Nothing.

Done when

A new tests/test_markdown.py: each rule of the subset renders as specified; raw HTML, a <script> element, an onerror= attribute, an ![](…) image, [x](javascript:…), a data: link and a percent-encoded or entity-encoded scheme come out as text or as nothing; every a has the forced rel and target; plain() strips the markers; the markdown-it configuration itself is asserted (html off, no image rule, no table rule), so a change to it fails a test.
A new tests/security/test_markdown.py renders the 40,000-character worst cases (long runs of **x**, [[[[, >>>>, deeply nested lists, a very long link title) and asserts a bounded time and an output that still passes the allowlist.
tests/test_listings.py: a listing in Markdown shows a list and a link on its page and plain text in the table, the search excerpt and the API; one in plain text is unchanged, and so is an existing listing after the migration.
The browser suite (tests/e2e/test_accessibility.py, tests/e2e/) visits a Markdown listing under the real policy with no violation, axe-clean in both themes, with scripts off as well; tests/test_stylesheet.py passes with the rebuilt CSS and tests/test_template_lint.py with logical utilities only.
The archive format goes up by one (core/export.py FORMAT_VERSION, 29 on f09b6010b; one more than whatever it is when the commit lands) for description_format, and an older archive imports as plain; the candidate format (4) is unchanged. API: description_format on the listing detail and on writes (tests/test_api.py). Processing record (core/gdpr.py) unchanged. Search needs no change beyond the excerpt. CV print choice (#308) does not apply. tests/test_wiki_surface.py does not apply (no new plugin-surface name).
pyproject.toml and uv.lock gain markdown-it-py (with mdurl) and nh3; THIRD-PARTY.md lists only works committed to the tree and says dependencies arrive under their own licences, so it needs no row, and the image's bill of materials carries them; strings in the working catalogues (en-GB, fr-FR, pt-PT as drafts) and Weblate (#349).

From the list of improvements of 2026-10-02.

From the list of 2026-10-02: "Listing details can support basic MD syntax." A listing's description is plain text today; a person who types or edits one cannot make a list, a link or a heading of it. The page must keep working with scripts off and under the strict content security policy, so the rendering is the server's. ## What exists today - **The description is plain text.** `JobPosting.description` is a `TextField` (`src/postulo/jobs/models.py:898`). The form's help text promises it "is kept exactly as pasted" (`jobs/forms.py:71-74`); the field is a ten-row `Textarea` (`jobs/forms.py:715-720`), and the capture review form edits the same value (`templates/jobs/capture_review.html:150`, `jobs/capture_views.py:250`). - **Where it is shown.** The listing page prints it escaped inside `whitespace-pre-line` (`templates/jobs/posting_detail.html:61-62`); the Listings table prints it clamped to two lines (`templates/jobs/partials/listing_row.html:80-84`); search takes an excerpt from it (`core/search.py:165-178`); the API returns it on the detail only (`api/schemas.py:1012`); the archive carries it as `description` (`core/export.py:208`, `POSTING_FIELDS`); CSV import appends notes to it (`core/csv_import.py:991`). - **What a capture puts there is flattened text, capped.** The parsers' schema truncates at 40,000 characters and adds "[…truncated]" (`plugins/base.py:30`, `:91-92`). It is text read off somebody else's page, not Markdown, so a line that begins with `-`, `*`, `#` or a number is, in the reader's mind, a line and not markup. - **No Markdown exists in the project.** Nothing under `src/postulo` parses it, and `uv.lock` holds none of `markdown-it-py`, `mistune`, `nh3`, `bleach` or `markdown`. #480 (open) lists Markdown as a candidate export format, which is the other direction. - **The policy the page runs under.** `SECURE_CSP` allows scripts and styles only from `'self'` (plus a nonce for style on one kind of page, the CV and letter previews) and images from `'self'` and `data:` (`config/settings/base.py:671-698`). Nothing here may need script, and a rendered image must never reach another host. The project already treats untrusted markup as two halves of one defence (`core/pictures.py` for SVG, `core/files.py:51-62`). - **Where the project hand-writes a standard, and why.** `applications/ical.py:1-12` and `documents/docx.py:1-12` write a format out from structured data in a page of code, to avoid a dependency that would "bring the rest of the standard along" or a C extension; `docs/PLAN.md:132-136` kept BeautifulSoup and lxml out for the same reason. Both are writers of a fixed shape. This is a **parser of arbitrary input whose output is HTML**, and so the precedent points the other way here. Checked against `main` at `f09b6010b`. ## Proposal 1. **A per-listing format.** `JobPosting.description_format` with choices *Plain text* and *Markdown*, default *Plain text* (a migration with no data change). Existing and captured text stays plain; a person who wants Markdown on one listing chooses it on the listing form and on the capture review form, whose help text says what is supported. 2. **One renderer, in one module** (`core/markdown.py`, so a later field can use it): `render(text) -> SafeString` and `plain(text) -> str`. Parser: markdown-it-py with the `zero` preset, `html` off, `linkify` off, `breaks` on, and exactly these rules enabled: paragraphs, line breaks, emphasis and strong emphasis, inline code, ordered and unordered lists, block quotes, links (inline, reference and `<https://…>`), headings. **Not** enabled: raw HTML, images, tables, task lists, footnotes, fenced code, autolinking of bare addresses. Headings are shifted down so they cannot compete with the page's own: `#` renders as `<h3>`, and nothing above `<h6>`. 3. **A second pass over the output** with an allowlist (`nh3`): `p br strong em code ul ol li blockquote h3 h4 h5 h6 a`; `href` on `a` only, with the schemes `http`, `https` and `mailto`; `rel="noopener noreferrer external"` and `target="_blank"` forced, as the page's existing "Original posting" link has (`posting_detail.html:146`). The CSP is the third layer and is not relied on. 4. **A style for it.** The rendered block carries one class, styled in `assets/css/app.css` with the same tokens as the rest (list markers, link underline in the brand colour, heading sizes, quote border), then `npm run build:css` and the committed stylesheet. Plain text keeps `whitespace-pre-line`. 5. **Everything that is not the listing page gets text, not HTML.** The table cell and the search excerpt use `plain(text)` (the same parse, rendered as text, so `**` and `[x](y)` markers do not show). The API returns the stored source and a new `description_format`; it does not return HTML (a client would have to sanitise it again). The archive carries `description_format`. 6. **Wording.** The form help (`jobs/forms.py:71`) says what Markdown is supported and that the text is stored as typed; the wiki's *Tracking applications* gets a short section. ## Decisions - **A flag, or Markdown for every listing.** Options: always render Markdown; render it only where the person chose it. **Recommended:** the flag, default plain. Captured adverts are flattened text, and rendering `*`, `#` or `1)` at the start of a line as markup changes what hundreds of existing listings say. The cost is a second field on the form. - **Which syntax.** Options: CommonMark's core only; GitHub-flavoured (tables, task lists, autolinks, strikethrough). **Recommended:** the core subset in part 2. The note says "basic"; a table in an advert's description is rare, and each extension is a rule to keep sanitised. - **Headings.** Allowed but shifted (above), or not offered. **Recommended:** allowed and shifted: an advert has sections, and a heading that outranks the page's own breaks the outline for a screen reader. - **Images.** None. `img-src` includes `data:`, and a remote image would tell the host who looked; the project's rule is that a page makes no request on a reader's behalf (`accounts/avatars.py:1-10`). - **A sanitiser as well as a parser.** Options: markdown-it-py with HTML off (its documented safe mode) and nothing more; plus `nh3` over the output. **Recommended:** both. With raw HTML off the second pass should never change anything, and that is the reason it is worth having: it holds if somebody later turns `html` on or adds a plugin, as the SVG allowlist and the file policy do for pictures. Conflict to record: `docs/PLAN.md:132-136` kept C extensions out of the core, and `nh3` is a compiled (Rust) wheel. The core already depends on compiled wheels (Pillow, cryptography, psycopg-binary, brotli, zopfli, in `uv.lock`), so the argument that was made about lxml in M4 has not been applied since; if the maintainer would rather not add one, the fallback is the parser alone with the tests below. - **Preview.** A toolbar or a live preview needs script. **Recommended:** none in this issue; if wanted, a *Preview* button that posts the form and renders on the server is its own issue. - **Hand-written subset.** See *Prior art*. **Recommended:** not hand-written. ## Prior art Versions and advisories checked on 2026-10-02 against PyPI and OSV (`api.osv.dev`). OSV lists advisories over all versions; the second number is those that affect the latest release. - **`markdown-it-py`** (<https://github.com/executablebooks/markdown-it-py>): a CommonMark parser, a port of markdown-it. MIT. 4.2.0, released 2026-05-07. Pure Python, requires Python 3.10 or later, one dependency (`mdurl`, MIT, 0.1.2 from 2022-08-14, pure Python). 4 entries on record; the two GitHub advisories are denial-of-service issues fixed in 2.2.0 (2023); 0 affect 4.2.0. Its documentation says the default `commonmark` preset enables HTML and is "less secure", that the `js-default` preset is recommended for untrusted input, and that it blocks `javascript:`, `vbscript:`, `file:` and `data:` links (with limited image exceptions). Not a dependency today. **Adopt**, configured as in part 2 (not from a preset that enables HTML). - **`mistune`** (<https://github.com/lepture/mistune>): BSD-3-Clause; 3.3.4, released 2026-07-22; pure Python. 40 advisories on record, among them XSS through percent-encoded `javascript:` addresses in `safe_url()` (published 2026-07-20, fixed in 3.3.0) and several directive and denial-of-service issues; 0 affect 3.3.4. A record of cross-site scripting in its URL handling that recent is the reason to prefer the other parser. **Not adopted.** - **Python-Markdown (`markdown`)** (BSD-3-Clause; 3.11, released 2026-09-25; 2 advisories, fixed in 3.8.1): passes raw HTML through by default (not re-checked here against its current documentation), so it would depend on the sanitiser for everything. **Not adopted.** - **`cmarkgfm`** (MIT; 2025.10.22; C extension over cmark-gfm; 0 advisories) and **`commonmark`** (BSD-3-Clause; 0.9.2; 0 advisories): the first is a compiled dependency with nothing the parser above lacks; the second's maintenance state is not verified. **Not adopted.** - **`nh3`** (<https://github.com/messense/nh3>): Python bindings for the Rust crate `ammonia`. MIT (the Rust crate `ammonia`: MIT or Apache-2.0). 0.3.7, released 2026-08-23; 0 advisories on OSV for the package (RustSec for `ammonia` itself was not checked: not verified). Wheels exist for manylinux on x86_64 and aarch64 (`abi3`, Python 3.8 and later), which is what the Debian-based image installs, so it needs no compiler. No runtime dependencies. Not a dependency today. **Adopt** as the second layer, subject to the decision above. - **`bleach`**: 6.4.0, released 2026-06-05. Its README now says that Bleach "is no longer maintained" with "no future releases including for security issues", and the repository was archived on 2026-06-10. 13 advisories on record, 0 affect 6.4.0. **Rejected**: nothing will fix the next one. - **`markdown2`**: MIT; 2.5.5; 8 advisories on record, several of them cross-site scripting. **Not adopted.** - **Hand-written** (a few regular expressions for bold, italic, lists and links, then escaping): what `ical.py` and `docx.py` do for their formats. The rule the maintainer set is to look for a library first; the library is there, maintained and clean, and a hand-written Markdown-to-HTML path is exactly where cross-site scripting is introduced. **Rejected.** ## Builds on #480 (Markdown as an export format, the other direction), #264 (an SVG sanitiser and a hardened file policy as two halves of one defence). ## Needed by Nothing. ## Done when - A new `tests/test_markdown.py`: each rule of the subset renders as specified; raw HTML, a `<script>` element, an `onerror=` attribute, an `![](…)` image, `[x](javascript:…)`, a `data:` link and a percent-encoded or entity-encoded scheme come out as text or as nothing; every `a` has the forced `rel` and `target`; `plain()` strips the markers; the markdown-it configuration itself is asserted (`html` off, no image rule, no table rule), so a change to it fails a test. - A new `tests/security/test_markdown.py` renders the 40,000-character worst cases (long runs of `**x**`, `[[[[`, `>>>>`, deeply nested lists, a very long link title) and asserts a bounded time and an output that still passes the allowlist. - `tests/test_listings.py`: a listing in Markdown shows a list and a link on its page and plain text in the table, the search excerpt and the API; one in plain text is unchanged, and so is an existing listing after the migration. - The browser suite (`tests/e2e/test_accessibility.py`, `tests/e2e/`) visits a Markdown listing under the real policy with no violation, axe-clean in both themes, with scripts off as well; `tests/test_stylesheet.py` passes with the rebuilt CSS and `tests/test_template_lint.py` with logical utilities only. - The archive format goes up by one (`core/export.py` `FORMAT_VERSION`, 29 on `f09b6010b`; one more than whatever it is when the commit lands) for `description_format`, and an older archive imports as plain; the candidate format (4) is unchanged. API: `description_format` on the listing detail and on writes (`tests/test_api.py`). Processing record (`core/gdpr.py`) unchanged. Search needs no change beyond the excerpt. CV print choice (#308) does not apply. `tests/test_wiki_surface.py` does not apply (no new plugin-surface name). - `pyproject.toml` and `uv.lock` gain `markdown-it-py` (with `mdurl`) and `nh3`; `THIRD-PARTY.md` lists only works committed to the tree and says dependencies arrive under their own licences, so it needs no row, and the image's bill of materials carries them; strings in the working catalogues (en-GB, fr-FR, pt-PT as drafts) and Weblate (#349). From the list of improvements of 2026-10-02.

tiagoagueda added this to the 0.6.0 milestone

2026-10-03 01:58:44 +00:00

tiagoagueda added the

interface

enhancement

labels

2026-10-03 01:58:44 +00:00

tiagoagueda commented

2026-10-05 20:48:51 +00:00

Author

Owner

Done in 9e5d2ad62 (on main; CI green at 15e6d7ee4). A listing's description can be set to Markdown per listing, rendered on the server and sanitised (no images, raw HTML or relative links). Wiki: Listings page.

Done in 9e5d2ad62 (on main; CI green at 15e6d7ee4). A listing's description can be set to Markdown per listing, rendered on the server and sanitised (no images, raw HTML or relative links). Wiki: Listings page.

tiagoagueda closed this issue

2026-10-05 20:48:51 +00:00

tiagoagueda referenced this issue

2026-10-30 21:25:13 +00:00

Listings: show which of the listing's skills the person has and which they do not, from ESCO, with no model #1093