Avatar and company logo: database storage that goes with the owner, or files with a guaranteed lifecycle #662

New issue

Closed

opened 2026-10-03 01:58:41 +00:00 by tiagoagueda · 1 comment

tiagoagueda commented

2026-10-03 01:58:41 +00:00

Owner

What exists today

The avatar is a file. Profile.avatar is an ImageField stored under avatars/<user pk>/ (src/postulo/accounts/models.py:152-154, :410). The Gravatar copy is a second file in the same directory (models.py:414-416); Profile.picture chooses between them (models.py:443-450). The picture is decoded, cropped square and re-encoded as PNG (accounts/avatars.py:67-80), with a 5 MB input cap (:48) and a 1 MB stored budget (:52).
The logo is a file. Company.logo is a FileField under logos/<owner pk>/ (jobs/models.py:204-206, :299). A raster is re-encoded as PNG within the same 1 MB budget (jobs/logos.py:57, :90-107); an SVG is sanitised and stays an SVG (core/pictures.py:290-315, at most 512 KiB, :244). So no stored picture weighs more than 1 MiB.
Serving. AvatarView (accounts/views.py:297-315) and CompanyLogoView (jobs/views.py:412-435) call serve_private_file (core/files.py:65-117), which can hand the file to X-Accel-Redirect or X-Sendfile and otherwise streams it. {% company_logo %} draws one <img> per company, each a request to that view with a one-day cache and a ?v= stamp (core/templatetags/postulo.py:231-248).
Deleting the owner. Nothing deletes these files when their row goes, except by hand-written lists. accounts/deletion.py:61-81 (files_of) names the avatar and the Gravatar copy but not the logo, and media_directories_of (:84-91) prunes avatars/<pk> but not logos/<pk>. There is no post_delete receiver for Profile or Company (the only receivers that remove files are documents/signals.py:38-39 and jobs/signals.py:57). The result is the opposite of the maintainer's second sentence: a company deleted from its own page leaves its logo (jobs/views.py:574).
Already tracked, not repeated here. #355 (open, 0.5.0): account deletion and company deletion leave the logo, and prune_media deletes live exports. #525 (open, 0.6.0): logos.store and logos.clear delete the old file before the row is saved (jobs/logos.py:214-244). #466 (open, 0.5.0): the archive importer stores an avatar or logo without running avatars.process or logos.process (core/importer.py:504-508, :666, :697). #482 (open, 0.6.0): re-encoding a phone photo holds a web worker for seconds.
Also not covered by #355. A user deleted from Django's admin (accounts/admin.py:14-15; the admin is mounted only when an operator sets POSTULO_ADMIN_URL, config/urls.py:33-36) loses the Profile row by cascade and keeps the avatar and the Gravatar copy on disk, because no receiver exists for Profile. A comment on #355 records it.
The media tree and the database are backed up together (core/backup.py:1-12, include_media at :328; manage.py backup --no-media leaves the tree out), so "one thing to back up" is already nearly true for an operator who does not pass --no-media.
The per-person archive carries both as files in the zip, under a media/ prefix, named in the JSON by account.avatar_file and companies[].logo_file (core/export.py:655-657, :693, :984-989); the importer saves them back (core/importer.py:504-508, :697). prune_media lists them as referenced (documents/management/commands/prune_media.py:29-31).
Why files. docs/PLAN.md section 6 puts private media behind an ownership-checked view, and #217 settled that deleting a row deletes its file. Neither says why a picture of at most 1 MiB has to be a file rather than a column; the pictures were files because uploads were.
Where the code touches the field: the profile form (accounts/forms.py:448, :513, :611, :649-651), profile.html:53-76, the template tag above, company merging (jobs/merging.py:401-407, :491-507), the company form (jobs/forms.py:256), accounts/deletion.py, prune_media, export and import.

Checked against main at f09b6010b.

Proposal

Say in the wiki and in CONTRIBUTING.md which pictures live where. This is the one decision that the rest depends on: see the first decision below.
If the answer is the database: a side table per owner, not a column on the owner's row, one BinaryField plus its media type and the moment it was stored. A profile has up to two pictures (the upload and the Gravatar copy), a company has up to one. The side table is OneToOne/ForeignKey with on_delete=CASCADE, so deleting the owner deletes the picture in the same statement, in the same transaction, with no receiver, no on_commit, no files_of entry and no orphan sweep.
One function in core/pictures.py (or beside it) that takes the owner and the bytes after as_stored or sanitise_svg, and is the only place a picture is written. The upload form, Find logo, the Gravatar fetch, company merging and the archive importer all go through it, which also puts the importer's picture through the same checks (#466).
Serve from the table: a view answering the bytes with the media type, Cache-Control: private, max-age=86400, nosniff and the FILE_POLICY header, as serve_private_text already does for text (core/files.py:125-153). {% company_logo %} and AvatarView keep their addresses and their ?v= stamp. The company list must not load the bytes: reading a company never touches the side table.
Move what exists. A data migration reads each avatar, Gravatar copy and logo through the configured storage, writes the rows, and reports a file that is missing or no longer decodes (that picture becomes "none", as if removed). The old fields are dropped in the next release, so a rollback finds its files, and prune_media then lists the old files as orphans for the operator to remove.
Delete avatar and gravatar_image from deletion.files_of, avatars/<pk> from media_directories_of, and the two entries from prune_media._referenced. Company.logo leaves merging.py's file handling: a merge moves a row or deletes one.
The per-person archive: keep avatar_file and logo_file as members of the zip so older archives still import and the format number does not move; only the name inside media/ stops being a storage path.

If the answer is files: do not build the table. Finish #355 (a post_delete receiver on Company and one on Profile, run in transaction.on_commit, and apps.get_models() walked for every file field so the next one is covered), #525 (write, save, then remove the old name on commit), and #466 (through process). This issue then closes with a note in the wiki.

Decisions

Database or files. The maintainer has stated: database. Options: a column or side table, or files whose lifecycle is guaranteed by receivers and a test that walks every file field. Recommended: the database, in a side table. The stated rule is a lifecycle rule (goes with its owner), and a cascade is the only mechanism that cannot be forgotten: #355 and #525, and the admin deletion found here, are all about a file outliving or disagreeing with its row, and none of them can occur for a row. #466 is about unchecked bytes, and the one write function in part 3 fixes it whichever way this goes. What it costs: (1) X-Accel-Redirect and X-Sendfile are no longer available for these two pictures, so a web worker answers each one. The image runs three synchronous workers (docker/Dockerfile:360-362), but a picture is at most 1 MiB, the answer is cached for a day, and FileResponse already occupies a worker when no proxy offer is configured. (2) The database grows by at most 1 MiB per picture. SQLite's own article on this says the database is more efficient than files as blobs get smaller and files win as they get larger, and the default blob limit is 1,000,000,000 bytes (https://www.sqlite.org/fasterthanfs.html, https://www.sqlite.org/limits.html); the exact break-even is not quoted there and is not verified for this workload. (3) A write to the table holds the write lock for the time of one small INSERT, which is what logos.store already does for the row (jobs/logos.py:233-234). If a measurement on a real instance shows the list pages slower, the files option remains open and #355's receivers are the fix.
The Gravatar copy. The note names only the uploaded avatar. Options: move it with the avatar; leave it as a file. Recommended: move it with the avatar, as a second row of the same table. A fetched copy is derived and replaceable, but two stores for one property (Profile.picture) is the kind of split that produced the bugs above.
Size. Options: keep the 1 MiB stored budget and the 512 KiB SVG ceiling; lower them for the database. Recommended: keep them, and add a line to the wiki's Hardening page stating what each picture can add to the database.
One table or two. Options: one generic "stored picture" table keyed by owner type and id, or one per owner model. Recommended: one per owner model (ProfilePicture, CompanyLogo): a real foreign key is what makes the cascade a database fact and not an application habit, and a generic key is what forget_the_source_of_a_render has to write out by hand (documents/signals.py:72-89).
Existing library. django-db-file-storage (MIT, latest release 2024-01-30, 0 advisories on OSV) stores files in the database behind Django's storage interface. Its classifiers list Django up to 5.0 and Postulo requires Django 6.1 (pyproject.toml); the repository was last pushed to on 2026-09-02, and whether it runs on Django 6.1 is not verified. Recommended: not adopted. It would give a storage facade, where the point here is an ownership relation, and BinaryField is in Django.
Conflicts with earlier decisions. #217 and docs/PLAN.md section 6 describe private media as files behind a view; this changes that for two pictures only and leaves documents, renders, kept pages and exports where they are. The comment on Company.logo (jobs/models.py:291-298) and the docstrings of accounts/avatars.py and jobs/logos.py ("keeps it under private media") are updated with it. #663 (the storage boundary for uploaded documents) is unaffected: a storage plugin never sees these pictures.

Builds on

#217 (closed), #355, #466, #525, #482, #265 (closed, why the stored size is not capped at 256).

Needed by

#663 states the storage rule for uploaded documents and refers here for the two pictures. #664 concerns kept pages, which are files too; it does not wait for this answer.

Done when

The answer is written in the wiki (Files and what you sent and Backups and your data) and in the docstrings named above.
tests/test_account_deletion.py::test_deleting_removes_every_owned_row_and_every_file (extended) gives the person a company with a logo and a Gravatar copy and asserts, after delete_account(), that no picture row remains and that no file of theirs remains under MEDIA_ROOT. A new test in tests/test_logos.py deletes a company through jobs:company_delete and finds no picture row. A test deletes a user through the admin and finds none either.
A merge in tests/test_merging.py leaves the kept company with one logo row and the other's gone.
tests/security/test_feeds_and_pictures.py still passes unchanged for the two serving views (an owner, an administrator for avatars, nobody else, sign-in required), and a test reads a list page of companies and counts queries, to prove no picture bytes are loaded with a company.
The data migration has a test in tests/test_logos.py and tests/test_avatars.py over a file, a missing file and an undecodable one; prune_media (tests/test_document_record.py) no longer names the two pictures and lists the old files once the fields are gone.
Export and import: tests/test_avatars.py:344 and tests/test_logos.py:449-470 pass unchanged; the importer's pictures go through process (shared with #466). FORMAT_VERSION (29 on f09b6010b, core/export.py:80) stays unless the member naming changes; if it does, it goes up by one, to one more than whatever it is when the commit lands.
Strings in the working catalogues (en-GB, fr-FR, pt-PT as drafts) and Weblate (#349).

From the list of improvements of 2026-10-02.

The maintainer's rule, from the list of 2026-10-02: "An uploaded avatar photo is stored on the database. If the user is deleted, the uploaded avatar go with it." and "A company logo is stored on the database. If the company is deleted, the logo go with it." Today both are files under private media, not rows, so neither sentence is true yet. This issue records the rule, the facts, and the two ways to make it true. ## What exists today - **The avatar is a file.** `Profile.avatar` is an `ImageField` stored under `avatars/<user pk>/` (`src/postulo/accounts/models.py:152-154`, `:410`). The Gravatar copy is a second file in the same directory (`models.py:414-416`); `Profile.picture` chooses between them (`models.py:443-450`). The picture is decoded, cropped square and re-encoded as PNG (`accounts/avatars.py:67-80`), with a 5 MB input cap (`:48`) and a 1 MB stored budget (`:52`). - **The logo is a file.** `Company.logo` is a `FileField` under `logos/<owner pk>/` (`jobs/models.py:204-206`, `:299`). A raster is re-encoded as PNG within the same 1 MB budget (`jobs/logos.py:57`, `:90-107`); an SVG is sanitised and stays an SVG (`core/pictures.py:290-315`, at most 512 KiB, `:244`). So no stored picture weighs more than 1 MiB. - **Serving.** `AvatarView` (`accounts/views.py:297-315`) and `CompanyLogoView` (`jobs/views.py:412-435`) call `serve_private_file` (`core/files.py:65-117`), which can hand the file to `X-Accel-Redirect` or `X-Sendfile` and otherwise streams it. `{% company_logo %}` draws one `<img>` per company, each a request to that view with a one-day cache and a `?v=` stamp (`core/templatetags/postulo.py:231-248`). - **Deleting the owner.** Nothing deletes these files when their row goes, except by hand-written lists. `accounts/deletion.py:61-81` (`files_of`) names the avatar and the Gravatar copy but not the logo, and `media_directories_of` (`:84-91`) prunes `avatars/<pk>` but not `logos/<pk>`. There is no `post_delete` receiver for `Profile` or `Company` (the only receivers that remove files are `documents/signals.py:38-39` and `jobs/signals.py:57`). The result is the opposite of the maintainer's second sentence: a company deleted from its own page leaves its logo (`jobs/views.py:574`). - **Already tracked, not repeated here.** #355 (open, 0.5.0): account deletion and company deletion leave the logo, and `prune_media` deletes live exports. #525 (open, 0.6.0): `logos.store` and `logos.clear` delete the old file before the row is saved (`jobs/logos.py:214-244`). #466 (open, 0.5.0): the archive importer stores an avatar or logo without running `avatars.process` or `logos.process` (`core/importer.py:504-508`, `:666`, `:697`). #482 (open, 0.6.0): re-encoding a phone photo holds a web worker for seconds. - **Also not covered by #355.** A user deleted from Django's admin (`accounts/admin.py:14-15`; the admin is mounted only when an operator sets `POSTULO_ADMIN_URL`, `config/urls.py:33-36`) loses the `Profile` row by cascade and keeps the avatar and the Gravatar copy on disk, because no receiver exists for `Profile`. A comment on #355 records it. - **The media tree and the database are backed up together** (`core/backup.py:1-12`, `include_media` at `:328`; `manage.py backup --no-media` leaves the tree out), so "one thing to back up" is already nearly true for an operator who does not pass `--no-media`. - **The per-person archive carries both as files in the zip**, under a `media/` prefix, named in the JSON by `account.avatar_file` and `companies[].logo_file` (`core/export.py:655-657`, `:693`, `:984-989`); the importer saves them back (`core/importer.py:504-508`, `:697`). `prune_media` lists them as referenced (`documents/management/commands/prune_media.py:29-31`). - **Why files.** `docs/PLAN.md` section 6 puts private media behind an ownership-checked view, and #217 settled that deleting a row deletes its file. Neither says why a picture of at most 1 MiB has to be a file rather than a column; the pictures were files because uploads were. - **Where the code touches the field:** the profile form (`accounts/forms.py:448`, `:513`, `:611`, `:649-651`), `profile.html:53-76`, the template tag above, company merging (`jobs/merging.py:401-407`, `:491-507`), the company form (`jobs/forms.py:256`), `accounts/deletion.py`, `prune_media`, export and import. Checked against `main` at `f09b6010b`. ## Proposal 1. Say in the wiki and in `CONTRIBUTING.md` which pictures live where. This is the one decision that the rest depends on: see the first decision below. 2. If the answer is the database: a side table per owner, not a column on the owner's row, one `BinaryField` plus its media type and the moment it was stored. A profile has up to two pictures (the upload and the Gravatar copy), a company has up to one. The side table is `OneToOne`/`ForeignKey` with `on_delete=CASCADE`, so deleting the owner deletes the picture in the same statement, in the same transaction, with no receiver, no `on_commit`, no `files_of` entry and no orphan sweep. 3. One function in `core/pictures.py` (or beside it) that takes the owner and the bytes **after** `as_stored` or `sanitise_svg`, and is the only place a picture is written. The upload form, *Find logo*, the Gravatar fetch, company merging and the archive importer all go through it, which also puts the importer's picture through the same checks (#466). 4. Serve from the table: a view answering the bytes with the media type, `Cache-Control: private, max-age=86400`, `nosniff` and the `FILE_POLICY` header, as `serve_private_text` already does for text (`core/files.py:125-153`). `{% company_logo %}` and `AvatarView` keep their addresses and their `?v=` stamp. The company list must not load the bytes: reading a company never touches the side table. 5. Move what exists. A data migration reads each avatar, Gravatar copy and logo through the configured storage, writes the rows, and reports a file that is missing or no longer decodes (that picture becomes "none", as if removed). The old fields are dropped in the **next** release, so a rollback finds its files, and `prune_media` then lists the old files as orphans for the operator to remove. 6. Delete `avatar` and `gravatar_image` from `deletion.files_of`, `avatars/<pk>` from `media_directories_of`, and the two entries from `prune_media._referenced`. `Company.logo` leaves `merging.py`'s file handling: a merge moves a row or deletes one. 7. The per-person archive: keep `avatar_file` and `logo_file` as members of the zip so older archives still import and the format number does not move; only the name inside `media/` stops being a storage path. If the answer is files: do not build the table. Finish #355 (a `post_delete` receiver on `Company` and one on `Profile`, run in `transaction.on_commit`, and `apps.get_models()` walked for every file field so the next one is covered), #525 (write, save, then remove the old name on commit), and #466 (through `process`). This issue then closes with a note in the wiki. ## Decisions - **Database or files.** The maintainer has stated: database. Options: a column or side table, or files whose lifecycle is guaranteed by receivers and a test that walks every file field. **Recommended:** the database, in a side table. The stated rule is a lifecycle rule (goes with its owner), and a cascade is the only mechanism that cannot be forgotten: #355 and #525, and the admin deletion found here, are all about a file outliving or disagreeing with its row, and none of them can occur for a row. #466 is about unchecked bytes, and the one write function in part 3 fixes it whichever way this goes. What it costs: (1) `X-Accel-Redirect` and `X-Sendfile` are no longer available for these two pictures, so a web worker answers each one. The image runs three synchronous workers (`docker/Dockerfile:360-362`), but a picture is at most 1 MiB, the answer is cached for a day, and `FileResponse` already occupies a worker when no proxy offer is configured. (2) The database grows by at most 1 MiB per picture. SQLite's own article on this says the database is more efficient than files as blobs get smaller and files win as they get larger, and the default blob limit is 1,000,000,000 bytes (<https://www.sqlite.org/fasterthanfs.html>, <https://www.sqlite.org/limits.html>); the exact break-even is not quoted there and is not verified for this workload. (3) A write to the table holds the write lock for the time of one small `INSERT`, which is what `logos.store` already does for the row (`jobs/logos.py:233-234`). If a measurement on a real instance shows the list pages slower, the files option remains open and #355's receivers are the fix. - **The Gravatar copy.** The note names only the uploaded avatar. Options: move it with the avatar; leave it as a file. **Recommended:** move it with the avatar, as a second row of the same table. A fetched copy is derived and replaceable, but two stores for one property (`Profile.picture`) is the kind of split that produced the bugs above. - **Size.** Options: keep the 1 MiB stored budget and the 512 KiB SVG ceiling; lower them for the database. **Recommended:** keep them, and add a line to the wiki's Hardening page stating what each picture can add to the database. - **One table or two.** Options: one generic "stored picture" table keyed by owner type and id, or one per owner model. **Recommended:** one per owner model (`ProfilePicture`, `CompanyLogo`): a real foreign key is what makes the cascade a database fact and not an application habit, and a generic key is what `forget_the_source_of_a_render` has to write out by hand (`documents/signals.py:72-89`). - **Existing library.** `django-db-file-storage` (MIT, latest release 2024-01-30, 0 advisories on OSV) stores files in the database behind Django's storage interface. Its classifiers list Django up to 5.0 and Postulo requires Django 6.1 (`pyproject.toml`); the repository was last pushed to on 2026-09-02, and whether it runs on Django 6.1 is not verified. **Recommended:** not adopted. It would give a storage facade, where the point here is an ownership relation, and `BinaryField` is in Django. - **Conflicts with earlier decisions.** #217 and `docs/PLAN.md` section 6 describe private media as files behind a view; this changes that for two pictures only and leaves documents, renders, kept pages and exports where they are. The comment on `Company.logo` (`jobs/models.py:291-298`) and the docstrings of `accounts/avatars.py` and `jobs/logos.py` ("keeps it under private media") are updated with it. #663 (the storage boundary for uploaded documents) is unaffected: a storage plugin never sees these pictures. ## Builds on #217 (closed), #355, #466, #525, #482, #265 (closed, why the stored size is not capped at 256). ## Needed by #663 states the storage rule for uploaded documents and refers here for the two pictures. #664 concerns kept pages, which are files too; it does not wait for this answer. ## Done when - The answer is written in the wiki (*Files and what you sent* and *Backups and your data*) and in the docstrings named above. - `tests/test_account_deletion.py::test_deleting_removes_every_owned_row_and_every_file` (extended) gives the person a company with a logo and a Gravatar copy and asserts, after `delete_account()`, that no picture row remains and that no file of theirs remains under `MEDIA_ROOT`. A new test in `tests/test_logos.py` deletes a company through `jobs:company_delete` and finds no picture row. A test deletes a user through the admin and finds none either. - A merge in `tests/test_merging.py` leaves the kept company with one logo row and the other's gone. - `tests/security/test_feeds_and_pictures.py` still passes unchanged for the two serving views (an owner, an administrator for avatars, nobody else, sign-in required), and a test reads a list page of companies and counts queries, to prove no picture bytes are loaded with a company. - The data migration has a test in `tests/test_logos.py` and `tests/test_avatars.py` over a file, a missing file and an undecodable one; `prune_media` (`tests/test_document_record.py`) no longer names the two pictures and lists the old files once the fields are gone. - Export and import: `tests/test_avatars.py:344` and `tests/test_logos.py:449-470` pass unchanged; the importer's pictures go through `process` (shared with #466). `FORMAT_VERSION` (29 on `f09b6010b`, `core/export.py:80`) stays unless the member naming changes; if it does, it goes up by one, to one more than whatever it is when the commit lands. - Strings in the working catalogues (en-GB, fr-FR, pt-PT as drafts) and Weblate (#349). From the list of improvements of 2026-10-02.

tiagoagueda added this to the 0.6.0 milestone

2026-10-03 01:58:41 +00:00

tiagoagueda added the

enhancement

label

2026-10-03 01:58:41 +00:00

tiagoagueda referenced this issue

2026-10-03 01:58:42 +00:00

Uploaded documents: no orphans, no corrupted files, and one boundary for where the bytes live #663

tiagoagueda referenced this issue

2026-10-03 01:58:43 +00:00

A kept page follows its capture, then the listing it became, and goes when that listing goes #664