Data export: the archive is built in /tmp and then copied, and each kept file is read whole into memory on the way #1125

New issue

Closed

opened 2026-10-31 03:04:12 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-31 03:04:12 +00:00

Owner

What exists today

src/postulo/core/slow.py:47-58 (the export errand): tempfile.mkstemp(prefix="postulo-export-", suffix=".zip") in the system temporary directory, write_archive(owner, writing) into it, then archive.file.save(name, File(reading)), which for FileSystemStorage copies the bytes chunk by chunk into the media root (django/core/files/storage/filesystem.py:68-110; it only renames when the content has temporary_file_path, which a plain File does not), then the temporary file is unlinked. Every export is written twice.

docker/compose.hardened.yml:22-27 mounts /tmp as tmpfs for every service, and the worker has mem_limit: 1g (line 39). Tmpfs pages are charged to the container's memory cgroup, so under the hardened file the whole archive sits in the worker's memory allowance while it is built and copied. Captured pages are part of the export (core/export.py:1738-1744) and an account may keep up to 1 GB of them (POSTULO_CAPTURE_ACCOUNT_MAX_BYTES, config/settings/base.py:775-776), so a large account can push the worker towards its limit and the OOM killer, which B6 then reports a day later.

core/export.py:1774-1777 copies each media file with archive.writestr(name, handle.read()), reading it whole. That is bounded by the largest single file (uploads are capped at 20 MB by the form, kept page renderings at 10 MB), so this part is a small cost, not the scout's main point. The docstring at lines 1751-1753 says holding the export in memory was chosen deliberately because an export is "measured in megabytes"; with captures counted that is no longer always true, and streaming here is two lines, not new machinery.

manage.py export_data (core/management/commands/export_data.py:27) writes to a file the operator names and is not affected by the temporary directory.

What would change

In write_archive, stream each file: with default_storage.open(name, "rb") as src, archive.open(f"{MEDIA_PREFIX}{name}", "w") as dst: shutil.copyfileobj(src, dst, 1024 * 1024). Same archive, same members.
In build_an_export, create the temporary file on the data volume rather than in /tmp (for example a .tmp directory under MEDIA_ROOT's parent, or the directory exports are kept in), and hand it to storage as an object with temporary_file_path() (Django's TemporaryUploadedFile pattern) so FileSystemStorage moves it instead of copying. A remote storage backend still copies, as it must.
Update the docstring at core/export.py:1751-1753.

No library needed: zipfile.ZipFile.open(..., "w") and shutil.copyfileobj are the standard library.

Tests: the existing export tests keep passing byte for byte on the manifest and members; a test that the errand leaves nothing in the temporary directory.

What it saves or protects

medium: an export stops occupying the worker's memory under the hardened compose file (where /tmp is tmpfs counted against mem_limit: 1g) and stops being written twice; per-file peak memory drops from the largest file (up to 20 MB for an upload, 10 MB for a kept page rendering) to a copy buffer

Accessibility

none

Decisions

Where may the scratch file live? Recommended: inside the data volume next to the export archives, as a dot-file the orphan sweep ignores until it is past the grace period (B3), so a killed export does not leave a stray for long.

Found by the efficiency and reliability audit of 2026-10-30 (finding D5; index postulo/postulo#1108). Checked against main at daee30b15 (2026-10-30).

## What exists today `src/postulo/core/slow.py:47-58` (the `export` errand): `tempfile.mkstemp(prefix="postulo-export-", suffix=".zip")` in the system temporary directory, `write_archive(owner, writing)` into it, then `archive.file.save(name, File(reading))`, which for `FileSystemStorage` copies the bytes chunk by chunk into the media root (`django/core/files/storage/filesystem.py:68-110`; it only renames when the content has `temporary_file_path`, which a plain `File` does not), then the temporary file is unlinked. Every export is written twice. `docker/compose.hardened.yml:22-27` mounts `/tmp` as tmpfs for every service, and the worker has `mem_limit: 1g` (line 39). Tmpfs pages are charged to the container's memory cgroup, so under the hardened file the whole archive sits in the worker's memory allowance while it is built and copied. Captured pages are part of the export (`core/export.py:1738-1744`) and an account may keep up to 1 GB of them (`POSTULO_CAPTURE_ACCOUNT_MAX_BYTES`, `config/settings/base.py:775-776`), so a large account can push the worker towards its limit and the OOM killer, which `B6` then reports a day later. `core/export.py:1774-1777` copies each media file with `archive.writestr(name, handle.read())`, reading it whole. That is bounded by the largest single file (uploads are capped at 20 MB by the form, kept page renderings at 10 MB), so this part is a small cost, not the scout's main point. The docstring at lines 1751-1753 says holding the export in memory was chosen deliberately because an export is "measured in megabytes"; with captures counted that is no longer always true, and streaming here is two lines, not new machinery. `manage.py export_data` (`core/management/commands/export_data.py:27`) writes to a file the operator names and is not affected by the temporary directory. ## What would change 1. In `write_archive`, stream each file: `with default_storage.open(name, "rb") as src, archive.open(f"{MEDIA_PREFIX}{name}", "w") as dst: shutil.copyfileobj(src, dst, 1024 * 1024)`. Same archive, same members. 2. In `build_an_export`, create the temporary file on the data volume rather than in `/tmp` (for example a `.tmp` directory under `MEDIA_ROOT`'s parent, or the directory exports are kept in), and hand it to storage as an object with `temporary_file_path()` (Django's `TemporaryUploadedFile` pattern) so `FileSystemStorage` moves it instead of copying. A remote storage backend still copies, as it must. 3. Update the docstring at `core/export.py:1751-1753`. No library needed: `zipfile.ZipFile.open(..., "w")` and `shutil.copyfileobj` are the standard library. Tests: the existing export tests keep passing byte for byte on the manifest and members; a test that the errand leaves nothing in the temporary directory. ## What it saves or protects medium: an export stops occupying the worker's memory under the hardened compose file (where /tmp is tmpfs counted against `mem_limit: 1g`) and stops being written twice; per-file peak memory drops from the largest file (up to 20 MB for an upload, 10 MB for a kept page rendering) to a copy buffer ## Accessibility none ## Decisions - **Where may the scratch file live?** Recommended: inside the data volume next to the export archives, as a dot-file the orphan sweep ignores until it is past the grace period (B3), so a killed export does not leave a stray for long. --- Found by the efficiency and reliability audit of 2026-10-30 (finding D5; index postulo/postulo#1108). Checked against `main` at `daee30b15` (2026-10-30).

tiagoagueda added this to the 0.6.0 milestone

2026-10-31 03:04:12 +00:00

tiagoagueda added the

performance

enhancement

labels

2026-10-31 03:04:12 +00:00

tiagoagueda referenced this issue

2026-10-31 03:04:40 +00:00

Efficiency and reliability audit of 2026-10-30: index #1108

tiagoagueda referenced this issue from a pull request that will close it,

2026-11-11 13:40:10 +00:00