A kept page follows its capture, then the listing it became, and goes when that listing goes #664

New issue

Closed

opened 2026-10-03 01:58:43 +00:00 by tiagoagueda · 1 comment

tiagoagueda commented

2026-10-03 01:58:43 +00:00

Owner

What exists today

What is kept, and where. CapturedPage (src/postulo/jobs/models.py:1212-1294) is one row per capture (OneToOneField, on_delete=CASCADE, :1235-1240) holding two optional files: the source, gzipped text capped at 2,000,000 bytes (POSTULO_CAPTURE_SOURCE_MAX_BYTES, config/settings/base.py:594), and a rendering (PNG, JPEG, WebP or PDF) capped at 10 MiB (:595-597). An account's kept pages may weigh 1 GiB together (:602-604). Both switches are off by default (:587-588). They are files under captures/<owner>/<YYYY>/<MM>/ (page_upload_to, :1178-1180), not rows. Where the maintainer's rule says "database", this is the same question as #662.
A capture has no deletion of its own. The review page's three answers are save as a listing, bind to a listing, and discard (jobs/capture_views.py:326-380, :383-440, :443-460). Discard sets status = DISCARDED and nothing else (:449-452); the bulk version is an UPDATE (:475-480). Nothing in the interface deletes a Capture. The only ways a row goes are deleting the account and the admin (jobs/admin.py:57). The list of discarded captures that #260 promised does not exist (#380, open).
The kept page can be forgotten on its own. CapturedPageForgetView (jobs/page_views.py:174-200, jobs/urls.py:90-92, linked from templates/jobs/capture_page.html:114, :181, :191) asks, then deletes the source, the rendering or both of one capture, for any capture, a confirmed one included; “the capture itself stays” (pages.forget). So the person can already delete the stored data by hand; what they cannot do is delete the capture, or have the data follow the listing.
An unconfirmed capture's page does expire. pages.expire_unconfirmed deletes the page of every capture that is not ACCEPTED once the capture is older than POSTULO_CAPTURE_PAGE_KEEP_DAYS (30 by default; 0 keeps all), run by the scheduler (jobs/pages.py:800-823, notifications/management/commands/send_due_reminders.py:203-205). The capture row stays. Its files go through the post_delete receiver (jobs/signals.py:57-84, after commit and only where no other row uses the name), which also runs for a cascade, so deleting a Capture row takes its files (tests/test_captured_pages.py:880).
A confirmed capture's page stays on the capture, by design. Saving as a listing sets status = ACCEPTED and posting = listing (jobs/capture_views.py:351-352); binding does the same (jobs/history.py:239-241). bind_capture's docstring says the capture "is kept and pointed at, not copied: what it read, the address it was read at, and whatever it kept of the page (#256) stay on the capture, which becomes the listing's" (jobs/history.py:206-219, #270). The listing reaches the page through the capture: capture__posting=posting (jobs/views.py:865-869, shown as "The page as captured" in templates/jobs/posting_detail.html:151-163). An accepted capture is excluded from expiry, "for as long as it exists" (pages.py:800-807, the wiki's Capturing postings: "A capture you saved keeps its page").
Deleting the listing leaves the capture and its page. Capture.posting is SET_NULL (jobs/models.py:1110-1118). PostingDeleteView says so before it happens: "The %(count)d file or capture its history points at is kept where it is." (jobs/views.py:917, :972-982, #270). The capture is then ACCEPTED with no listing, which the expiry skips, which the interface lists nowhere, and which nothing but the account's deletion removes. Its page goes on counting against the account's 1 GiB, and the "Settings → Capture" page's total.
Already tracked, not repeated here. #380 (discarded captures cannot be listed or restored), #523 (binding leaves learning on the capture), #524 (expiry loses what the review would learn), #336 (the review page renders a decided capture), #355 (account deletion's hand-kept lists).

Checked against main at f09b6010b.

Proposal

A listing's deletion deletes the captures that became it, and with them their pages. Capture.posting goes from SET_NULL to CASCADE (one migration, no data change). The existing post_delete receiver removes the files after the commit; the cascade runs the collector, which sends the signal for every row (jobs/signals.py:1-15). bind_capture's captures follow, because they set the same field. No new ownership model, no archive change: captures[].posting_id and captures[].page already say it.
Say what will go. PostingDeleteView counts the captures and their kept pages among the consequences ("%(count)d kept page of a capture") and no longer says a capture "is kept where it is"; the sentence stays true for an uploaded file, which is still not deleted with a listing.
A capture can be deleted, while it is not a listing's. A confirmed POST to a new jobs:capture_delete for a pending or discarded capture removes the row and, through the receiver, its page, at once. Offered on the review page and on the list of discarded captures that #380 restores; it is refused for an ACCEPTED capture, whose data is the listing's.
Find what is already stranded. manage.py prune_captures (listing by default, --remove to act, as prune_media does) names every ACCEPTED capture that has no listing and no application, with its page and what it weighs. Not a migration: the files are somebody's, and the rule that applies to those rows was not the rule when they were made.
Wiki and wording. Capturing postings (the "How long" and "Where it is" paragraphs), Configuration (POSTULO_CAPTURE_PAGE_KEEP_DAYS: "A capture somebody saved keeps its page" becomes "keeps its page while the listing exists"), and the docstrings of jobs/pages.py, jobs/history.py and CapturedPage.

Decisions

Where the data lives: database or files. The maintainer's rule says the data is stored in the database and migrates to the listing's entry. A source is gzipped text (up to 2,000,000 bytes) and a rendering may be 10 MiB, with up to 1 GiB per account. Recommended: keep files. The lifecycle the rule asks for (goes with the capture, then with the listing, nowhere else) is delivered by a foreign key and the receiver that already exists; a database column would put up to 10 MiB per capture into every dump, make every backup carry what the operator may have chosen not to keep, and be read whole into memory by Django. If #662 chooses the database for pictures after measuring, that answer concerns 1 MiB blobs and is not evidence about these.
Who owns the page once confirmed. Options: (A) the page's foreign key moves from the capture to the listing, with the capture link made optional; (A') the page stays on the capture, and the listing owns the capture (CASCADE); (B) the page stays on the capture and a pre_delete receiver on the listing deletes just the pages. Recommended: A'. It makes the rule true with one migration and no new URLs (the page routes are keyed by capture id, jobs/urls.py:70-92), keeps #270's design, and leaves the archive untouched. B leaves a decided capture with no listing and no page, a row that means nothing. A is the literal reading of "migrated to the listing" and is the choice if decided captures are ever to be purged while their pages stay; it needs the page routes re-addressed, an archive bump (29) and the importer's _restore_kept_page moved. A' does not stop A being done later.
Which "listing entry". The note says the data migrates "to database listing entry"; this reads it as the listing itself (JobPosting). The other candidate is an entry in the listing's history (ListingEvent, #270), which a bound capture already has and which points at the capture. Under A' the page is reachable from both, and goes with both.
Discarding. The rule says a capture deletion deletes the data. Today discard is a recoverable status (#260's design) and its page expires after 30 days. Options: discard deletes the data at once; discard keeps the window and an explicit Delete (part 3) deletes at once. Recommended: the second. #380 asks for the way back, and a window that ends by itself is already the stated retention (POSTULO_CAPTURE_PAGE_KEEP_DAYS).
Conflict with an earlier decision. #270 decided that what a capture kept "stays on the capture, which becomes the listing's", and PostingDeleteView tells the person a capture survives its listing. This changes that, for pages and captures, and not for uploaded files. Record in the changelog under Changed.
Stranded rows already in a database. Options: a migration deletes them; a command lists and removes them; leave them to expire when the account goes. Recommended: the command, the same default prune_media has: "the failure mode of the opposite default is somebody's CV".

Builds on

#256 (closed), #270 (closed), #260 (closed), #217 (closed), #380, #523, #524, #336, #355.

Needed by

Part 3 needs a list of discarded captures, which is #380; part 3 may ship with a minimal one if #380 has not landed.

Done when

tests/test_captured_pages.py: deleting a listing that a capture became removes the capture, the page row and both files after commit (beside test_deleting_a_capture_takes_its_files; fails today); a bound capture's page goes with the listing it was bound to; a rolled-back listing deletion keeps them; a pending or discarded capture's page is untouched by deleting somebody else's listing; deleting a pending capture through jobs:capture_delete removes its files, and the same request on an ACCEPTED capture is refused.
tests/test_listing_history.py: the history entries that pointed at the deleted capture go with the listing and nothing dangles.
tests/test_listing_history.py:493 pins today's sentence ("The 1 file or capture its history points at is kept where it is."); it changes with part 2, and a new test there asserts that the confirmation page counts the kept pages.
prune_captures lists and removes a stranded page and leaves a live one (tests/test_captured_pages.py).
tests/security/test_isolation_sweep.py and tests/test_page_coverage.py for jobs:capture_delete (another person's capture is a 404; a sign-in is needed).
Export and import: no new key, so FORMAT_VERSION (29 on f09b6010b, core/export.py:80) is unchanged under A'; under A it goes up by one (one more than whatever it is when the commit lands). API: GET /captures is unchanged (pending only); a capture of a deleted listing is a 404. Processing record (core/gdpr.py) unchanged: nothing new is kept, less is kept for less time. Strings in the working catalogues (en-GB, fr-FR, pt-PT as drafts) and Weblate (#349); the wiki pages in part 5.

From the list of improvements of 2026-10-02.

The maintainer's rule, from the list of 2026-10-02: "When keep capture source and/or rendering is set, such data is stored on the database. A capture deletion, delete the corresponding stored data. Unless capture translated to confirmed listing, in such case, data is migrated to database listing entry. Only if the listing is deleted the data is permanently deleted." Today the data is kept as files, and for a *confirmed* capture the code does the opposite of the last sentence: deleting the listing leaves the capture and its page behind for good. ## What exists today - **What is kept, and where.** `CapturedPage` (`src/postulo/jobs/models.py:1212-1294`) is one row per capture (`OneToOneField`, `on_delete=CASCADE`, `:1235-1240`) holding two optional files: the source, gzipped text capped at 2,000,000 bytes (`POSTULO_CAPTURE_SOURCE_MAX_BYTES`, `config/settings/base.py:594`), and a rendering (PNG, JPEG, WebP or PDF) capped at 10 MiB (`:595-597`). An account's kept pages may weigh 1 GiB together (`:602-604`). Both switches are off by default (`:587-588`). They are files under `captures/<owner>/<YYYY>/<MM>/` (`page_upload_to`, `:1178-1180`), not rows. Where the maintainer's rule says "database", this is the same question as #662. - **A capture has no deletion of its own.** The review page's three answers are save as a listing, bind to a listing, and discard (`jobs/capture_views.py:326-380`, `:383-440`, `:443-460`). *Discard* sets `status = DISCARDED` and nothing else (`:449-452`); the bulk version is an `UPDATE` (`:475-480`). Nothing in the interface deletes a `Capture`. The only ways a row goes are deleting the account and the admin (`jobs/admin.py:57`). The list of discarded captures that #260 promised does not exist (#380, open). - **The kept page can be forgotten on its own.** `CapturedPageForgetView` (`jobs/page_views.py:174-200`, `jobs/urls.py:90-92`, linked from `templates/jobs/capture_page.html:114, :181, :191`) asks, then deletes the source, the rendering or both of one capture, for any capture, a confirmed one included; “the capture itself stays” (`pages.forget`). So the person can already delete the stored data by hand; what they cannot do is delete the capture, or have the data follow the listing. - **An unconfirmed capture's page does expire.** `pages.expire_unconfirmed` deletes the page of every capture that is not `ACCEPTED` once the capture is older than `POSTULO_CAPTURE_PAGE_KEEP_DAYS` (30 by default; 0 keeps all), run by the scheduler (`jobs/pages.py:800-823`, `notifications/management/commands/send_due_reminders.py:203-205`). The capture row stays. Its files go through the `post_delete` receiver (`jobs/signals.py:57-84`, after commit and only where no other row uses the name), which also runs for a cascade, so deleting a `Capture` row takes its files (`tests/test_captured_pages.py:880`). - **A confirmed capture's page stays on the capture, by design.** Saving as a listing sets `status = ACCEPTED` and `posting = listing` (`jobs/capture_views.py:351-352`); binding does the same (`jobs/history.py:239-241`). `bind_capture`'s docstring says the capture "is **kept and pointed at**, not copied: what it read, the address it was read at, and whatever it kept of the page (#256) stay on the capture, which becomes the listing's" (`jobs/history.py:206-219`, #270). The listing reaches the page through the capture: `capture__posting=posting` (`jobs/views.py:865-869`, shown as "The page as captured" in `templates/jobs/posting_detail.html:151-163`). An accepted capture is excluded from expiry, "for as long as it exists" (`pages.py:800-807`, the wiki's *Capturing postings*: "A capture you saved keeps its page"). - **Deleting the listing leaves the capture and its page.** `Capture.posting` is `SET_NULL` (`jobs/models.py:1110-1118`). `PostingDeleteView` says so before it happens: "The %(count)d file or capture its history points at is kept where it is." (`jobs/views.py:917`, `:972-982`, #270). The capture is then `ACCEPTED` with no listing, which the expiry skips, which the interface lists nowhere, and which nothing but the account's deletion removes. Its page goes on counting against the account's 1 GiB, and the "Settings → Capture" page's total. - **Already tracked, not repeated here.** #380 (discarded captures cannot be listed or restored), #523 (binding leaves `learning` on the capture), #524 (expiry loses what the review would learn), #336 (the review page renders a decided capture), #355 (account deletion's hand-kept lists). Checked against `main` at `f09b6010b`. ## Proposal 1. **A listing's deletion deletes the captures that became it, and with them their pages.** `Capture.posting` goes from `SET_NULL` to `CASCADE` (one migration, no data change). The existing `post_delete` receiver removes the files after the commit; the cascade runs the collector, which sends the signal for every row (`jobs/signals.py:1-15`). `bind_capture`'s captures follow, because they set the same field. No new ownership model, no archive change: `captures[].posting_id` and `captures[].page` already say it. 2. **Say what will go.** `PostingDeleteView` counts the captures and their kept pages among the consequences ("%(count)d kept page of a capture") and no longer says a capture "is kept where it is"; the sentence stays true for an uploaded file, which is still not deleted with a listing. 3. **A capture can be deleted, while it is not a listing's.** A confirmed `POST` to a new `jobs:capture_delete` for a pending or discarded capture removes the row and, through the receiver, its page, at once. Offered on the review page and on the list of discarded captures that #380 restores; it is refused for an `ACCEPTED` capture, whose data is the listing's. 4. **Find what is already stranded.** `manage.py prune_captures` (listing by default, `--remove` to act, as `prune_media` does) names every `ACCEPTED` capture that has no listing and no application, with its page and what it weighs. Not a migration: the files are somebody's, and the rule that applies to those rows was not the rule when they were made. 5. **Wiki and wording.** *Capturing postings* (the "How long" and "Where it is" paragraphs), *Configuration* (`POSTULO_CAPTURE_PAGE_KEEP_DAYS`: "A capture somebody saved keeps its page" becomes "keeps its page while the listing exists"), and the docstrings of `jobs/pages.py`, `jobs/history.py` and `CapturedPage`. ## Decisions - **Where the data lives: database or files.** The maintainer's rule says the data is stored in the database and migrates to the listing's entry. A source is gzipped text (up to 2,000,000 bytes) and a rendering may be 10 MiB, with up to 1 GiB per account. **Recommended:** keep files. The lifecycle the rule asks for (goes with the capture, then with the listing, nowhere else) is delivered by a foreign key and the receiver that already exists; a database column would put up to 10 MiB per capture into every dump, make every backup carry what the operator may have chosen not to keep, and be read whole into memory by Django. If #662 chooses the database for pictures after measuring, that answer concerns 1 MiB blobs and is not evidence about these. - **Who owns the page once confirmed.** Options: (A) the page's foreign key moves from the capture to the listing, with the capture link made optional; (A') the page stays on the capture, and the listing owns the capture (`CASCADE`); (B) the page stays on the capture and a `pre_delete` receiver on the listing deletes just the pages. **Recommended:** A'. It makes the rule true with one migration and no new URLs (the page routes are keyed by capture id, `jobs/urls.py:70-92`), keeps #270's design, and leaves the archive untouched. B leaves a decided capture with no listing and no page, a row that means nothing. A is the literal reading of "migrated to the listing" and is the choice if decided captures are ever to be purged while their pages stay; it needs the page routes re-addressed, an archive bump (29) and the importer's `_restore_kept_page` moved. A' does not stop A being done later. - **Which "listing entry".** The note says the data migrates "to database listing entry"; this reads it as the listing itself (`JobPosting`). The other candidate is an entry in the listing's history (`ListingEvent`, #270), which a bound capture already has and which points at the capture. Under A' the page is reachable from both, and goes with both. - **Discarding.** The rule says a capture deletion deletes the data. Today *discard* is a recoverable status (#260's design) and its page expires after 30 days. Options: discard deletes the data at once; discard keeps the window and an explicit *Delete* (part 3) deletes at once. **Recommended:** the second. #380 asks for the way back, and a window that ends by itself is already the stated retention (`POSTULO_CAPTURE_PAGE_KEEP_DAYS`). - **Conflict with an earlier decision.** #270 decided that what a capture kept "stays on the capture, which becomes the listing's", and `PostingDeleteView` tells the person a capture survives its listing. This changes that, for pages and captures, and not for uploaded files. Record in the changelog under *Changed*. - **Stranded rows already in a database.** Options: a migration deletes them; a command lists and removes them; leave them to expire when the account goes. **Recommended:** the command, the same default `prune_media` has: "the failure mode of the opposite default is somebody's CV". ## Builds on #256 (closed), #270 (closed), #260 (closed), #217 (closed), #380, #523, #524, #336, #355. ## Needed by Part 3 needs a list of discarded captures, which is #380; part 3 may ship with a minimal one if #380 has not landed. ## Done when - `tests/test_captured_pages.py`: deleting a listing that a capture became removes the capture, the page row and both files after commit (beside `test_deleting_a_capture_takes_its_files`; fails today); a bound capture's page goes with the listing it was bound to; a rolled-back listing deletion keeps them; a pending or discarded capture's page is untouched by deleting somebody else's listing; deleting a pending capture through `jobs:capture_delete` removes its files, and the same request on an `ACCEPTED` capture is refused. - `tests/test_listing_history.py`: the history entries that pointed at the deleted capture go with the listing and nothing dangles. - `tests/test_listing_history.py:493` pins today's sentence ("The 1 file or capture its history points at is kept where it is."); it changes with part 2, and a new test there asserts that the confirmation page counts the kept pages. - `prune_captures` lists and removes a stranded page and leaves a live one (`tests/test_captured_pages.py`). - `tests/security/test_isolation_sweep.py` and `tests/test_page_coverage.py` for `jobs:capture_delete` (another person's capture is a 404; a sign-in is needed). - Export and import: no new key, so `FORMAT_VERSION` (29 on `f09b6010b`, `core/export.py:80`) is unchanged under A'; under A it goes up by one (one more than whatever it is when the commit lands). API: `GET /captures` is unchanged (pending only); a capture of a deleted listing is a 404. Processing record (`core/gdpr.py`) unchanged: nothing new is kept, less is kept for less time. Strings in the working catalogues (en-GB, fr-FR, pt-PT as drafts) and Weblate (#349); the wiki pages in part 5. From the list of improvements of 2026-10-02.

tiagoagueda added this to the 0.6.0 milestone

2026-10-03 01:58:43 +00:00

tiagoagueda added the

enhancement

label

2026-10-03 01:58:43 +00:00

tiagoagueda referenced this issue from a commit

2026-10-06 20:09:43 +00:00

A kept page follows its capture, then the listing it became, and goes when that listing goes

tiagoagueda commented

2026-10-06 20:34:28 +00:00

Author