Export archives: without the optional scheduler none is ever deleted, though the page promises it #539

New issue

Closed

opened 2026-10-01 10:47:29 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-01 10:47:29 +00:00

Owner

Found on 2026-09-29 in a read-only review of postulo at 95348908d, following the export archive from the button to whatever deletes it.

Related to #247.

Checked again on 2026-10-01 against main at 0a5cad587: still present. The code is now at notifications/management/commands/send_due_reminders.py:202 (the call to reap_archives()) and :205 (pages.expire_unconfirmed()); the other locations have not moved.

Every export now leaves a file: with or without a worker, build_an_export stores the zip as an ExportArchive. Only reap_archives() deletes those, and only the scheduler calls it. The scheduler is optional and described as needed only for notifications, so an instance run the way the first install command runs it keeps every archive forever, while the export page tells the person it is deleted after a day.

Where: core/slow.py:35-67 (build_an_export, which every export takes: core/views_export.py:55 sends the errand, and with no worker it runs in the request) and core/slow.py:70-83 (reap_archives). Its only caller is notifications/management/commands/send_due_reminders.py:200. The promise: templates/core/export.html:64-67 ("A built archive is kept for {{ hours }} hours and then deleted") and Backups-and-your-data.md:17-19 ("It is deleted a day later"). The scheduler behind a profile: docker/compose.yml:41-46, whose comment says it is "Only needed once somebody has set up a notification connection". The bare-metal steps say "There is no background worker to run" (Installing-Postulo.md:236).
What happens: on an instance started with docker compose -f docker/compose.yml up -d (Installing-Postulo.md:9) and no notification connection, each Export my data writes a full copy of the account (every record and every document) under media/exports/<pk>/. After 24 hours export_archive refuses to hand it over as expired, but the file stays on the disk and in every backup, one per export, for good. POSTULO_CAPTURE_PAGE_KEEP_DAYS has the same dependency: pages.expire_unconfirmed() (send_due_reminders.py:203) runs on the same pass.
Why it matters: the person was told the copy of their whole job search would be gone after a day, and it is not; disk use grows with every export. Only Configuration.md:433 mentions that "the scheduler does the deleting", and nothing on the pages a person or an operator reads first says the scheduler is needed for it.
Fix: make the deletion not depend on an optional process: reap expired archives (and unconfirmed captured pages) opportunistically, for example in export_overview and export_download and at most once an hour from a request, as well as on the scheduler's pass. Change the compose comment to say the scheduler also expires archives and captured pages, and add it to the bare-metal steps (the wording of Installing-Postulo.md:236 is reported separately, with the rest of those steps).

Done when

A test in tests/test_export.py builds an archive, moves the clock past POSTULO_EXPORT_KEEP_HOURS, requests the export page without running the scheduler, and finds the file and the row gone; it fails today.

Found on 2026-09-29 in a read-only review of postulo at 95348908d, following the export archive from the button to whatever deletes it. Related to #247. Checked again on 2026-10-01 against `main` at `0a5cad587`: still present. The code is now at `notifications/management/commands/send_due_reminders.py:202` (the call to `reap_archives()`) and `:205` (`pages.expire_unconfirmed()`); the other locations have not moved. Every export now leaves a file: with or without a worker, `build_an_export` stores the zip as an `ExportArchive`. Only `reap_archives()` deletes those, and only the scheduler calls it. The scheduler is optional and described as needed only for notifications, so an instance run the way the first install command runs it keeps every archive forever, while the export page tells the person it is deleted after a day. - **Where:** `core/slow.py:35-67` (`build_an_export`, which every export takes: `core/views_export.py:55` sends the errand, and with no worker it runs in the request) and `core/slow.py:70-83` (`reap_archives`). Its only caller is `notifications/management/commands/send_due_reminders.py:200`. The promise: `templates/core/export.html:64-67` ("A built archive is kept for {{ hours }} hours and then deleted") and `Backups-and-your-data.md:17-19` ("It is deleted a day later"). The scheduler behind a profile: `docker/compose.yml:41-46`, whose comment says it is "Only needed once somebody has set up a notification connection". The bare-metal steps say "There is no background worker to run" (`Installing-Postulo.md:236`). - **What happens:** on an instance started with `docker compose -f docker/compose.yml up -d` (`Installing-Postulo.md:9`) and no notification connection, each *Export my data* writes a full copy of the account (every record and every document) under `media/exports/<pk>/`. After 24 hours `export_archive` refuses to hand it over as expired, but the file stays on the disk and in every backup, one per export, for good. `POSTULO_CAPTURE_PAGE_KEEP_DAYS` has the same dependency: `pages.expire_unconfirmed()` (`send_due_reminders.py:203`) runs on the same pass. - **Why it matters:** the person was told the copy of their whole job search would be gone after a day, and it is not; disk use grows with every export. Only `Configuration.md:433` mentions that "the scheduler does the deleting", and nothing on the pages a person or an operator reads first says the scheduler is needed for it. - **Fix:** make the deletion not depend on an optional process: reap expired archives (and unconfirmed captured pages) opportunistically, for example in `export_overview` and `export_download` and at most once an hour from a request, as well as on the scheduler's pass. Change the compose comment to say the scheduler also expires archives and captured pages, and add it to the bare-metal steps (the wording of `Installing-Postulo.md:236` is reported separately, with the rest of those steps). ## Done when - A test in `tests/test_export.py` builds an archive, moves the clock past `POSTULO_EXPORT_KEEP_HOURS`, requests the export page without running the scheduler, and finds the file and the row gone; it fails today.

tiagoagueda added this to the 0.5.0 milestone

2026-10-01 10:47:29 +00:00

tiagoagueda added the

bug

label

2026-10-01 10:47:29 +00:00

tiagoagueda referenced this issue from a commit

2026-10-04 10:05:00 +00:00

Delete expired export archives without the scheduler

tiagoagueda closed this issue

2026-10-04 10:05:00 +00:00