Export and import one kind of record as JSON: companies, contacts, listings and applications, each as a file of its own #659

New issue

Closed

opened 2026-10-03 01:58:39 +00:00 by tiagoagueda · 1 comment

tiagoagueda commented

2026-10-03 01:58:39 +00:00

Owner

What exists today

The archive. src/postulo/core/export.py:80 (FORMAT_VERSION = 29), build_document (:619), a zip with the files beside it. Its import, src/postulo/core/importer.py:406 (load), is reached only by manage.py import_data (core/management/commands/import_data.py): “Creates records; never merges”, and into an account that already holds a search only with --force, which makes a second copy of everything. Open defects: #354 (keys reach the model constructors), #375, #464, #466, #334.
The candidate file, the pattern to copy. CANDIDATE_FORMAT = 4 (core/export.py:104), written by build_candidate_document (:868) from the same builders as the archive; read by resume/candidate.py as read (:353), plan (:1289) and apply (:1298) on a review page (resume/urls.py:13). Its stance (candidate.py:1-40): a review, not an action; it merges and never replaces; two entries are the same when they say the same thing, because ids in a file are local to it; every value goes through the form a person would have typed it in; the file is a stranger's, so its size (MAX_BYTES, :88) and rows (MAX_ROWS, :93) are bounded first.
One contact as JSON. ContactExportView (jobs/views.py:818-841, core/gdpr.py:155), offered only when the data-protection notice is (#297): the answer to “what do you have on me?”.
The API is JSON per kind: companies, applications, listings, interviews, reminders, offers, documents, profile (api/routers/), token-authenticated, with an updated_since cursor (#230). Its shapes are written by hand so that a new field never leaks (api/schemas.py:1-8). Contacts travel inside companies. There is no import there beyond creating one record.
The spreadsheet import (core/csv_import.py, core/views_import.py) is the other file import: upload, map and preview, import; duplicates reported and not created; MAX_BYTES 2 MiB and MAX_ROWS 5,000 (:37-38).
Matching today. A company is unique per owner by name (jobs/models.py:313, get_or_create_company in applications/services.py:307); jobs/duplicates.py finds likely duplicates of a company (:222) and a contact (:270).

Checked against main at f09b6010b.

Proposal

This is the design and an order of work; each kind is a commit of its own after the shared part.

The shared part. Factor out of the candidate file what is not about careers: the bounds, the plan's outcomes (will be added, already there, cannot be added), the stash of a held file in the session, and the review page. Nothing about the candidate file's behaviour changes.
One file per kind, each with its own marker, as the candidate file has candidate_format: companies_format, contacts_format, listings_format, applications_format, each starting at 1, under the same postulo header (version, exported_at, a note). The archive's reader looks for format and refuses these, as it refuses a candidate file (tests/test_candidate_file.py:425).
The same blocks as the archive, from the same builders, so there is one spelling of a company and not two to keep in step. A company carries its departments; an application carries its timeline and its interviews as the archive nests them.
Export from Settings → Your data, a card per kind with its count, as the export page counts today. The same files, filtered, from a list page belong to #341.
Import through the review page: what the file holds, what would be added, what is already there (matched by what a record says: a company by name or Wikidata id; a contact by email, else by name at the company; a listing by its address, else company, title and date; an application by its listing and its dates), what cannot be added and why. Then confirm. Nothing is replaced, and the same file twice adds nothing the second time.
Every imported application carries a timeline entry saying which file it came from, as the spreadsheet import writes one.

Decisions

Which kinds. The plan listed events. Recommended: no separate events file. Interviews and timeline entries travel inside their application, as in the archive; calendar events as a file are iCalendar (#661), by the maintainer's own rule.
Contacts: JSON and vCard. The full record, with its notes and its company, is JSON; interchange with an address book is vCard 4 (#660). Recommended: both, and the JSON stays the one that loses nothing.
A schema. Recommended: none published in the first step. A JSON Schema can be generated from pydantic models later with no new dependency (pydantic 2.13.5 is locked); the stance that every value goes through the form is what validates today, and is stricter.
What is left out, and said. Pictures, logos, uploaded files and plugin data: JSON cannot carry files, and the archive is where plugin rows go. Recommended: each file says so in its note, as the candidate file does of the picture.
Limits. Recommended: 2 MiB and 5,000 rows, the spreadsheet import's, for companies, contacts and listings; the candidate's 1 MiB and 200 for nothing here.
The importer's known weakness. The archive importer hands unknown keys to the model constructors (#354). Recommended: these readers take only the keys the writer writes, from the start.

Prior art

No library is needed to read or write JSON. Considered for validation: pydantic 2.13.5 (MIT, 2026-08-28, already locked; the API's schemas are pydantic; OSV lists four advisories, all fixed before 2.4.0) and jsonschema 4.26.0 (MIT, 2026-01-07, no OSV advisory, not a dependency). The project's own habit, every value through the page's form, is stricter than either and already tested, so neither is adopted. Django's serializers would tie the file to model fields and primary keys, which the archive's own module says it is not (core/export.py:1-12): rejected. Hand-written precedent: the archive and candidate readers, in the standard library.

Builds on / Needed by

Builds on #181, #334, #354, #375, #464, #466, #297 and #341. Needed by #660 and #661, which reuse the review page.

Done when

tests/test_kind_files.py (new): each kind's file round-trips between two accounts with nothing of the first in the second; a second import adds nothing; a block that changes shape is a new version of its file (the fingerprint test of tests/test_candidate_file.py:380); unknown keys, an owner_id and a foreign id write nothing; the archive's reader refuses each file.
tests/security/test_kind_files.py (new) in the style of tests/security/test_candidate_file.py: size and row bounds, another account's records never in a file, a crafted file cannot name a record.
tests/e2e/test_kind_files.py (new): export, upload, read the plan, confirm. tests/test_page_coverage.py, tests/security/test_isolation_sweep.py and axe cover the new pages.
The archive format (29 today) and the candidate format (4) are untouched: no new record type and no shape changed. Each new file starts at its own format 1. The processing record is built from plugins and connections and needs nothing. Strings in en-GB, fr-FR and pt-PT (drafts) and Weblate (#349); the wiki's Backups and your data.

From the list of improvements of 2026-10-02.

From the maintainer's list: “Preferable export/import format of ALL KIND data is JSON.” Postulo writes JSON for the whole account and for the candidate's own record, and serves it through the API, but nobody can take out “my companies” or “my contacts” as a file, and nothing reads such a file back. The whole-account archive is all or nothing, and its import only creates. ## What exists today - **The archive.** `src/postulo/core/export.py:80` (`FORMAT_VERSION = 29`), `build_document` (`:619`), a zip with the files beside it. Its import, `src/postulo/core/importer.py:406` (`load`), is reached only by `manage.py import_data` (`core/management/commands/import_data.py`): “Creates records; never merges”, and into an account that already holds a search only with `--force`, which makes a second copy of everything. Open defects: #354 (keys reach the model constructors), #375, #464, #466, #334. - **The candidate file, the pattern to copy.** `CANDIDATE_FORMAT = 4` (`core/export.py:104`), written by `build_candidate_document` (`:868`) from the same builders as the archive; read by `resume/candidate.py` as `read` (`:353`), `plan` (`:1289`) and `apply` (`:1298`) on a review page (`resume/urls.py:13`). Its stance (`candidate.py:1-40`): a review, not an action; it merges and never replaces; two entries are the same when they *say* the same thing, because ids in a file are local to it; every value goes through the form a person would have typed it in; the file is a stranger's, so its size (`MAX_BYTES`, `:88`) and rows (`MAX_ROWS`, `:93`) are bounded first. - **One contact as JSON.** `ContactExportView` (`jobs/views.py:818-841`, `core/gdpr.py:155`), offered only when the data-protection notice is (#297): the answer to “what do you have on me?”. - **The API** is JSON per kind: companies, applications, listings, interviews, reminders, offers, documents, profile (`api/routers/`), token-authenticated, with an `updated_since` cursor (#230). Its shapes are written by hand so that a new field never leaks (`api/schemas.py:1-8`). Contacts travel inside companies. There is no import there beyond creating one record. - **The spreadsheet import** (`core/csv_import.py`, `core/views_import.py`) is the other file import: upload, map and preview, import; duplicates reported and not created; `MAX_BYTES` 2 MiB and `MAX_ROWS` 5,000 (`:37-38`). - **Matching today.** A company is unique per owner by name (`jobs/models.py:313`, `get_or_create_company` in `applications/services.py:307`); `jobs/duplicates.py` finds likely duplicates of a company (`:222`) and a contact (`:270`). Checked against `main` at `f09b6010b`. ## Proposal This is the design and an order of work; each kind is a commit of its own after the shared part. 1. **The shared part.** Factor out of the candidate file what is not about careers: the bounds, the plan's outcomes (*will be added*, *already there*, *cannot be added*), the stash of a held file in the session, and the review page. Nothing about the candidate file's behaviour changes. 2. **One file per kind, each with its own marker,** as the candidate file has `candidate_format`: `companies_format`, `contacts_format`, `listings_format`, `applications_format`, each starting at 1, under the same `postulo` header (`version`, `exported_at`, a `note`). The archive's reader looks for `format` and refuses these, as it refuses a candidate file (`tests/test_candidate_file.py:425`). 3. **The same blocks as the archive,** from the same builders, so there is one spelling of a company and not two to keep in step. A company carries its departments; an application carries its timeline and its interviews as the archive nests them. 4. **Export** from *Settings → Your data*, a card per kind with its count, as the export page counts today. The same files, filtered, from a list page belong to #341. 5. **Import** through the review page: what the file holds, what would be added, what is already there (matched by what a record says: a company by name or Wikidata id; a contact by email, else by name at the company; a listing by its address, else company, title and date; an application by its listing and its dates), what cannot be added and why. Then confirm. Nothing is replaced, and the same file twice adds nothing the second time. 6. **Every imported application carries a timeline entry saying which file it came from,** as the spreadsheet import writes one. ## Decisions - **Which kinds.** The plan listed events. **Recommended:** no separate events file. Interviews and timeline entries travel inside their application, as in the archive; calendar events as a file are iCalendar (#661), by the maintainer's own rule. - **Contacts: JSON and vCard.** The full record, with its notes and its company, is JSON; interchange with an address book is vCard 4 (#660). **Recommended:** both, and the JSON stays the one that loses nothing. - **A schema.** **Recommended:** none published in the first step. A JSON Schema can be generated from pydantic models later with no new dependency (pydantic 2.13.5 is locked); the stance that every value goes through the form is what validates today, and is stricter. - **What is left out, and said.** Pictures, logos, uploaded files and plugin data: JSON cannot carry files, and the archive is where plugin rows go. **Recommended:** each file says so in its `note`, as the candidate file does of the picture. - **Limits.** **Recommended:** 2 MiB and 5,000 rows, the spreadsheet import's, for companies, contacts and listings; the candidate's 1 MiB and 200 for nothing here. - **The importer's known weakness.** The archive importer hands unknown keys to the model constructors (#354). **Recommended:** these readers take only the keys the writer writes, from the start. ## Prior art No library is needed to read or write JSON. Considered for validation: **pydantic** 2.13.5 (MIT, 2026-08-28, already locked; the API's schemas are pydantic; OSV lists four advisories, all fixed before 2.4.0) and **jsonschema** 4.26.0 (MIT, 2026-01-07, no OSV advisory, not a dependency). The project's own habit, every value through the page's form, is stricter than either and already tested, so neither is adopted. Django's `serializers` would tie the file to model fields and primary keys, which the archive's own module says it is not (`core/export.py:1-12`): rejected. Hand-written precedent: the archive and candidate readers, in the standard library. ## Builds on / Needed by Builds on #181, #334, #354, #375, #464, #466, #297 and #341. Needed by #660 and #661, which reuse the review page. ## Done when - `tests/test_kind_files.py` (new): each kind's file round-trips between two accounts with nothing of the first in the second; a second import adds nothing; a block that changes shape is a new version of its file (the fingerprint test of `tests/test_candidate_file.py:380`); unknown keys, an `owner_id` and a foreign id write nothing; the archive's reader refuses each file. - `tests/security/test_kind_files.py` (new) in the style of `tests/security/test_candidate_file.py`: size and row bounds, another account's records never in a file, a crafted file cannot name a record. - `tests/e2e/test_kind_files.py` (new): export, upload, read the plan, confirm. `tests/test_page_coverage.py`, `tests/security/test_isolation_sweep.py` and axe cover the new pages. - The archive format (29 today) and the candidate format (4) are untouched: no new record type and no shape changed. Each new file starts at its own format 1. The processing record is built from plugins and connections and needs nothing. Strings in en-GB, fr-FR and pt-PT (drafts) and Weblate (#349); the wiki's *Backups and your data*. From the list of improvements of 2026-10-02.

tiagoagueda added this to the 0.6.0 milestone

2026-10-03 01:58:39 +00:00

tiagoagueda added the

enhancement

label

2026-10-03 01:58:39 +00:00

tiagoagueda referenced this issue

2026-10-03 01:58:40 +00:00

Contacts as vCard 4: export and import .vcf, through one mapping the DAV plugin can reuse #660

tiagoagueda referenced this issue

2026-10-03 01:58:41 +00:00

Events as iCalendar: export every dated thing, import an .ics file, and replace the hand-written reader and writer with a library #661