Every document should record the language it is in, uploads and sent snapshots included #283

New issue

Closed

opened 2026-09-19 09:59:37 +00:00 by tiagoagueda · 1 comment

tiagoagueda commented

2026-09-19 09:59:37 +00:00

Owner

#280 asks for a flag on the cards, and scoped uploads and sent documents out on the grounds that neither carries a language. This issue settles that the other way: every document Postulo holds should record the language it is in, and every one should have somewhere to see it and — where it can honestly be changed — to set it. #280 then has something to draw for all four kinds instead of two.

The field decides the lang of a PDF that goes to an employer, which is what a screen reader reads it out with and what WeasyPrint hyphenates by (#223); it decides where a store files it (#130); and for a sent document it is part of the record of what was actually sent. Two of the four kinds have it. The other two guess.

What each kind has today

Model	Language	Where it is seen	Where it is set
`CV` (`models.py:120`)	yes, blank = follow profile	nowhere but the form (#280)	its form
`CoverLetter` (`models.py:333`)	yes, blank = follow profile	nowhere but the form (#280)	its form
`UploadedDocument` (`models.py:383`)	no field	—	—
`RenderedDocument` (`models.py:509`)	no field	—	—

Three things that are wrong today, not merely missing

1. An upload's language is guessed, and the guess is written into other people's systems.

documents/stores.py:50, _language_of, says so in its own docstring:

"An upload has no language of its own — nobody has told Postulo what is inside it — so it keeps falling back to the person, which is the best available answer for a file they chose themselves."

A scanned German certificate uploaded by somebody who reads Postulo in English is handed to Paperless as English. The comment three lines below the call site (stores.py:105) states the purpose that defeats: "A French CV filed in Paperless under the language its owner happens to read Postulo in is filed wrongly, and the whole point of sending it there is to find it again." The reasoning was right and the upload case was left as the best guess available. Now there can be a better one: ask.

2. A sent document's language is read from its source, live, and the source can change.

For a render, _language_of falls through to document_language(source) — evaluated whenever a store is told about it, not when the PDF was made. Change the CV's language after sending it and the frozen PDF is suddenly described differently. This contradicts what the model is for: RenderedDocument's own docstring is "A PDF exactly as it was sent, kept unchanged", and sent_to is stored as text rather than followed as a link precisely so a later edit cannot rewrite history (models.py:544).

3. When the source is deleted, the language of what the employer received is lost outright.

source is legitimately empty — an upload never had one, a report has none, and signals.py clears the link when a CV is deleted so that the frozen PDF survives it (models.py:552-560). _language_of then falls back to the owner's profile. So the one document that can never be regenerated is the one whose language is least recoverable.

The value is already computed at the right moment and thrown away: rendering.py:435 does with translation.override(document_language(cv)) to build the title, two lines above RenderedDocument(...). Storing it is a one-line change to something already in hand.

What to build

UploadedDocument.language — a real field, blank=True, same shape as the other two, and a picker on the form via the existing LanguageChoiceMixin (documents/forms.py:49). Blank must mean "not said", not "follow your profile": for a CV that fallback is a sensible default about text Postulo renders, but for a file somebody else made it is a claim about contents Postulo has never read. _language_of should return "" for an upload nobody has told, and a store should be sent no language rather than a wrong one. The upload form already allows editing everything except the bytes (#217), so an existing upload can be corrected.

RenderedDocument.language — captured at snapshot time in snapshot_cv, snapshot_letter and snapshot_report, alongside sent_to and checksum, and editable=False like them. _language_of then reads the stored value and stops consulting the source at all.

_language_of becomes a reader, not a resolver: stored value, else nothing. Its present cleverness is the bug.

The UI half

Uploads (templates/documents/upload_list.html:34) — the row already reads kind · version · date; the language joins it, and "not said" is worth showing as such rather than as a blank, because it is the prompt to fill it in.
Sent documents (templates/documents/rendered_list.html:34) — beside the kind, read-only.
CVs and letters — #280's flag on the card, which this issue makes possible everywhere rather than in two places out of four.
The upload form gains the picker. Nothing else gains an input, because nothing else should.

Points to settle

Required, or optional-and-visible? The title says must. Making it required on upload is defensible — it is one picker, and the person is right there looking at the file. It also means an upload can no longer be created by anything that does not supply one, including the API and a restore. Recommendation: optional in the database, prompted in the interface, and shown as "not said" until answered — which gets the behaviour without making an old archive unrestorable. If it should be genuinely mandatory, say so and the form enforces it while the column stays blankable for the backfill.
What happens to the rows that exist. Several thousand uploads and renders with no value. A data migration cannot know an upload's language and must leave it blank. For a render it can do better: where source still points at something, document_language(source) today is a better answer than nothing — it is the same value the store is already being given. Freezing that at migration time is an improvement even though it is not certain, but it should be a deliberate decision, not a side effect.
May a frozen render's language be corrected? A sent document has no edit view at all (documents/urls.py gives it list, download and archive only). If the captured value is ever wrong, there is no way to fix it. Leaving it that way is consistent; allowing a correction to metadata about the artefact while the artefact stays frozen is also defensible. Decide it here rather than discovering it later.
Detecting an upload's language was considered and is not proposed: it needs a text layer a scan does not have and a dependency this project does not carry, and a wrong automatic answer is exactly the failure being fixed. Asking is cheaper and honest.

Notes for whoever takes it

Two migrations (one per model), both additive.
The export format goes 16 → 17. core/export.py:45; UPLOAD_FIELDS (:188) and SENT_FIELDS (:189) gain the field, and the importer must keep reading 16, where it is absent. CV_FIELDS and LETTER_FIELDS already carry it.
The API is inconsistent today and should be fixed in passing: CVOut has language (api/schemas.py:370) and LetterOut does not, although CoverLetter has had the field all along. DocumentOut (:406), which serves both uploads and renders, has none. All three want it.
documents/kinds.py is the vocabulary of kinds; a language is not a kind and nothing there needs touching.
Adding fields to documents/models.py shifts the #: source references of every translatable string below them and breaks scripts/messages.py extract --check with no new string — #193 was this exact file. Run extract, then scope the revert to src/postulo/plugins/*/locale.
Tests worth writing, beyond the obvious: a store is told a render's language as it was at snapshot, after the source's language has since been changed; and a render whose source has been deleted still reports one.
Three catalogues filled draft (fr-fr, pt-pt, pt-br); the other 36 at the release sweep.

#280 asks for a flag on the cards, and scoped uploads and sent documents out on the grounds that neither carries a language. This issue settles that the other way: **every document Postulo holds should record the language it is in, and every one should have somewhere to see it and — where it can honestly be changed — to set it.** #280 then has something to draw for all four kinds instead of two. The field decides the `lang` of a PDF that goes to an employer, which is what a screen reader reads it out with and what WeasyPrint hyphenates by (#223); it decides where a store files it (#130); and for a sent document it is part of the record of what was actually sent. Two of the four kinds have it. The other two guess. ## What each kind has today | Model | Language | Where it is seen | Where it is set | |---|---|---|---| | `CV` (`models.py:120`) | **yes**, blank = follow profile | nowhere but the form (#280) | its form | | `CoverLetter` (`models.py:333`) | **yes**, blank = follow profile | nowhere but the form (#280) | its form | | `UploadedDocument` (`models.py:383`) | **no field** | — | — | | `RenderedDocument` (`models.py:509`) | **no field** | — | — | ## Three things that are wrong today, not merely missing **1. An upload's language is guessed, and the guess is written into other people's systems.** `documents/stores.py:50`, `_language_of`, says so in its own docstring: > *"An upload has no language of its own — nobody has told Postulo what is inside it — so it keeps falling back to the person, which is the best available answer for a file they chose themselves."* A scanned German certificate uploaded by somebody who reads Postulo in English is handed to Paperless as **English**. The comment three lines below the call site (`stores.py:105`) states the purpose that defeats: *"A French CV filed in Paperless under the language its owner happens to read Postulo in is filed wrongly, and the whole point of sending it there is to find it again."* The reasoning was right and the upload case was left as the best guess available. Now there can be a better one: ask. **2. A sent document's language is read from its source, live, and the source can change.** For a render, `_language_of` falls through to `document_language(source)` — evaluated whenever a store is told about it, not when the PDF was made. Change the CV's language after sending it and the frozen PDF is suddenly described differently. This contradicts what the model is for: `RenderedDocument`'s own docstring is *"A PDF exactly as it was sent, kept unchanged"*, and `sent_to` is stored as text rather than followed as a link precisely so a later edit cannot rewrite history (`models.py:544`). **3. When the source is deleted, the language of what the employer received is lost outright.** `source` is legitimately empty — an upload never had one, a report has none, and `signals.py` clears the link when a CV is deleted so that the frozen PDF survives it (`models.py:552-560`). `_language_of` then falls back to the owner's profile. So the one document that can never be regenerated is the one whose language is least recoverable. The value is **already computed at the right moment and thrown away**: `rendering.py:435` does `with translation.override(document_language(cv))` to build the title, two lines above `RenderedDocument(...)`. Storing it is a one-line change to something already in hand. ## What to build **`UploadedDocument.language`** — a real field, `blank=True`, same shape as the other two, and a picker on the form via the existing `LanguageChoiceMixin` (`documents/forms.py:49`). Blank must mean **"not said"**, not "follow your profile": for a CV that fallback is a sensible default about text Postulo renders, but for a file somebody else made it is a claim about contents Postulo has never read. `_language_of` should return `""` for an upload nobody has told, and a store should be sent no language rather than a wrong one. The upload form already allows editing everything except the bytes (#217), so an existing upload can be corrected. **`RenderedDocument.language`** — captured at snapshot time in `snapshot_cv`, `snapshot_letter` and `snapshot_report`, alongside `sent_to` and `checksum`, and `editable=False` like them. `_language_of` then reads the stored value and stops consulting the source at all. **`_language_of` becomes a reader**, not a resolver: stored value, else nothing. Its present cleverness is the bug. ## The UI half - **Uploads** (`templates/documents/upload_list.html:34`) — the row already reads *kind · version · date*; the language joins it, and "not said" is worth showing as such rather than as a blank, because it is the prompt to fill it in. - **Sent documents** (`templates/documents/rendered_list.html:34`) — beside the kind, read-only. - **CVs and letters** — #280's flag on the card, which this issue makes possible everywhere rather than in two places out of four. - **The upload form** gains the picker. Nothing else gains an input, because nothing else should. ## Points to settle - **Required, or optional-and-visible?** The title says *must*. Making it required on upload is defensible — it is one picker, and the person is right there looking at the file. It also means an upload can no longer be created by anything that does not supply one, including the API and a restore. Recommendation: **optional in the database, prompted in the interface, and shown as "not said" until answered** — which gets the behaviour without making an old archive unrestorable. If it should be genuinely mandatory, say so and the form enforces it while the column stays blankable for the backfill. - **What happens to the rows that exist.** Several thousand uploads and renders with no value. A data migration cannot know an upload's language and must leave it blank. For a **render** it can do better: where `source` still points at something, `document_language(source)` today is a better answer than nothing — it is the same value the store is already being given. Freezing that at migration time is an improvement even though it is not certain, but it should be a deliberate decision, not a side effect. - **May a frozen render's language be corrected?** A sent document has no edit view at all (`documents/urls.py` gives it list, download and archive only). If the captured value is ever wrong, there is no way to fix it. Leaving it that way is consistent; allowing a correction to *metadata about* the artefact while the artefact stays frozen is also defensible. Decide it here rather than discovering it later. - **Detecting an upload's language** was considered and is not proposed: it needs a text layer a scan does not have and a dependency this project does not carry, and a wrong automatic answer is exactly the failure being fixed. Asking is cheaper and honest. ## Notes for whoever takes it - **Two migrations** (one per model), both additive. - **The export format goes 16 → 17.** `core/export.py:45`; `UPLOAD_FIELDS` (`:188`) and `SENT_FIELDS` (`:189`) gain the field, and the importer must keep reading 16, where it is absent. `CV_FIELDS` and `LETTER_FIELDS` already carry it. - **The API is inconsistent today and should be fixed in passing:** `CVOut` has `language` (`api/schemas.py:370`) and **`LetterOut` does not**, although `CoverLetter` has had the field all along. `DocumentOut` (`:406`), which serves both uploads and renders, has none. All three want it. - `documents/kinds.py` is the vocabulary of *kinds*; a language is not a kind and nothing there needs touching. - Adding fields to `documents/models.py` shifts the `#:` source references of every translatable string below them and breaks `scripts/messages.py extract --check` with no new string — #193 was this exact file. Run `extract`, then scope the revert to `src/postulo/plugins/*/locale`. - Tests worth writing, beyond the obvious: a store is told a render's language *as it was at snapshot*, after the source's language has since been changed; and a render whose source has been deleted still reports one. - Three catalogues filled `draft` (`fr-fr`, `pt-pt`, `pt-br`); the other 36 at the release sweep.

tiagoagueda added this to the 0.4.0 milestone

2026-09-19 09:59:37 +00:00

tiagoagueda added the

enhancement

internationalisation

interface

labels

2026-09-19 09:59:37 +00:00

tiagoagueda referenced this issue from a commit

2026-09-20 10:24:04 +00:00

Let a file and a sent PDF say what language they are in

tiagoagueda commented