Donate to Postulo/postulo

• https://buymeacoffee.com/tiagoagueda

---

See the documentation for more details on donation links.

0.6.0 fixes, bunch 6: scheduler and worker #1228

Merged

tiagoagueda merged 11 commits from b6-scheduler into main 2026-11-11 11:52:17 +00:00

Conversation 0 Commits 11 Files changed 113 +2002 -154

tiagoagueda commented 2026-11-11 10:38:54 +00:00

Owner

• Copy link
• Copy Markdown

Bunch 6 of the 0.6.0 fixes: the scheduler pass and the worker. One signed commit per issue, then the changelog lines, one CI line and the fix to a test that line showed up; merge fast-forward-only.

• Closes #1118: the scheduled scrub reads only uploads never checked or last checked more than POSTULO_SCRUB_EVERY_DAYS ago (default 7; 0 is every pass, as before). manage.py verify_media still reads everything. New index on verified_at (migration documents 0033_upload_verified_at_index). With 100 uploads: 7,200 files hashed a day to 100.
• Closes #1119: the orphan sweep runs once an hour from the scheduler, and counts the kept files on the walk it makes anyway (13 walks an hour to 1). A sweep asked for outright is not gated.
• Closes #1121: the scheduler deletes django_tasks_db's finished results after the same seven days as errands, through the library's own finished().
• Closes #1167: expired rows of the database cache and expired files of the rate-limit file cache are deleted once an hour, by the scheduler and by the hourly request-time tidy; only entries already past their expiry, and nothing at all with other backends. The file purge is held to five seconds a run, since one caller is a request.
• Closes #1163: a notifier send that fails with an ordinary exception waits in the webhook delivery queue, already one attempt old, and the pass that delivers webhooks sends it again on the same schedule, with the same notification and key. No new table, no migration, no change to the archive; the rules #752 put on webhook rows hold for these rows too. ConnectionUnusable still retires the connection and queues nothing.
• Closes #1117: django_tasks_db logs at WARNING, the worker says once that it started, and the heartbeat is written on the first pass and then once a minute (100 idle passes: 200 log lines and 100 writes to 1 and 4).
• Closes #1120: the gone-quiet check reads the owners once with their profiles and filters "already announced" in SQL (nothing new for three accounts: 7 queries to 4; announced applications are no longer loaded).
• Closes #1172: archiving.send_pending stops starting copies once 60 s have gone, as the webhooks do; what is left keeps its place and goes first on the next pass.

Tested locally: lint, migration and catalogue checks, the full unit suite (21,402 passed, 189 skipped), and the two browser test files that touch these areas (16 passed; no template, style or script changed).

Not exercised locally: PostgreSQL. The cache purge's tests now run in the PostgreSQL job as well, because its one hand-written statement must never delete a live counter; the new index is applied by that job's migrate. The gone-quiet filter and the notifier retries are proven on SQLite only.

🤖 Generated with Claude Code

Bunch 6 of the 0.6.0 fixes: the scheduler pass and the worker. One signed commit per issue, then the changelog lines, one CI line and the fix to a test that line showed up; merge fast-forward-only. - Closes #1118: the scheduled scrub reads only uploads never checked or last checked more than `POSTULO_SCRUB_EVERY_DAYS` ago (default 7; 0 is every pass, as before). `manage.py verify_media` still reads everything. New index on `verified_at` (migration `documents 0033_upload_verified_at_index`). With 100 uploads: 7,200 files hashed a day to 100. - Closes #1119: the orphan sweep runs once an hour from the scheduler, and counts the kept files on the walk it makes anyway (13 walks an hour to 1). A sweep asked for outright is not gated. - Closes #1121: the scheduler deletes `django_tasks_db`'s finished results after the same seven days as errands, through the library's own `finished()`. - Closes #1167: expired rows of the database cache and expired files of the rate-limit file cache are deleted once an hour, by the scheduler and by the hourly request-time tidy; only entries already past their expiry, and nothing at all with other backends. The file purge is held to five seconds a run, since one caller is a request. - Closes #1163: a notifier send that fails with an ordinary exception waits in the webhook delivery queue, already one attempt old, and the pass that delivers webhooks sends it again on the same schedule, with the same notification and key. No new table, no migration, no change to the archive; the rules #752 put on webhook rows hold for these rows too. `ConnectionUnusable` still retires the connection and queues nothing. - Closes #1117: `django_tasks_db` logs at WARNING, the worker says once that it started, and the heartbeat is written on the first pass and then once a minute (100 idle passes: 200 log lines and 100 writes to 1 and 4). - Closes #1120: the gone-quiet check reads the owners once with their profiles and filters "already announced" in SQL (nothing new for three accounts: 7 queries to 4; announced applications are no longer loaded). - Closes #1172: `archiving.send_pending` stops starting copies once 60 s have gone, as the webhooks do; what is left keeps its place and goes first on the next pass. Tested locally: lint, migration and catalogue checks, the full unit suite (21,402 passed, 189 skipped), and the two browser test files that touch these areas (16 passed; no template, style or script changed). Not exercised locally: PostgreSQL. The cache purge's tests now run in the PostgreSQL job as well, because its one hand-written statement must never delete a live counter; the new index is applied by that job's migrate. The gone-quiet filter and the notifier retries are proven on SQLite only. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

tiagoagueda added 10 commits 2026-11-11 10:38:54 +00:00

Leave an upload the scrub has read alone for a week ... da32ed6f4e

Every scheduler pass, five minutes apart, read and hashed the 25 uploads
checked longest ago, however lately that was. An instance with a hundred
uploads read each of them 72 times a day: 7,200 files hashed and 7,200
rows written, where once a week reports damage within days all the same.

The scheduled slice now takes only the uploads never read or last read
more than POSTULO_SCRUB_EVERY_DAYS ago (default 7; 0 is every pass, as
before). `manage.py verify_media` passes no limit and still reads every
file. An index on `verified_at` answers the question the pass asks, so a
pass with nothing due no longer reads and sorts every row of the table.

Measured over a day of passes with 100 uploads: 7,200 files read and
7,200 UPDATEs before, 100 of each after; over eight days 57,600 and 200.

Closes #1118

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

Sweep the media for orphans once an hour, in one walk ... 07e6cb3fa9

Every scheduler pass, five minutes apart, read every file field of
every table and walked the whole media tree, looking for files that
must be older than the grace period, a day by default, before they go.
Each sweep could find only what had come of age since the last one. And
a sweep that found more than ten strays walked the tree a second time,
only to count the files it measures them against.

The pass now calls `scrub.sweep_if_due`, which sweeps when an hour has
gone by since the last sweep: a key in the shared cache, as the tidying
of export archives already has. An orphan goes an hour later at most. A
cache that does not answer sweeps on every pass, as before, and a sweep
that is switched off does not take the key. `scrub.sweep` itself is not
held to the hour, and `media.survey` gives the strays and the number of
files kept from the same walk.

Measured over twelve passes, an hour, on a media root of 2,012 files:
13 walks and 12 readings of every file field before, one of each after;
a pass that does not sweep takes under a millisecond where it took
about 380.

Closes #1119

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

Forget the queue's finished task results after a week ... 67fc12f59d

Every task a worker runs leaves a row in django_tasks_db's table: its
arguments, what it returned and, when it failed, the whole traceback.
The library ships prune_db_task_results and nothing ran it, so the table
grew by a row per errand, archive copy and backup for the life of the
instance, each row outliving the errand that gave it meaning.

The scheduler's pass now deletes them beside errands.forget_old(), in one
statement, for every queue, after the same seven days (now one constant,
errands.KEEP_DAYS). "Finished" is the library's own finished() manager
method, the one its command uses, not a status list of ours; work still
waiting or running is never touched. The count joins the pass's total.

Closes #1121

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

Delete expired cache entries once an hour ... 45b29fe000

Django's database cache and its file cache delete an expired entry only
when that same key is asked for again, or when the store reaches
MAX_ENTRIES, which is 200,000 here so that a counter cannot be pushed
out (#755). Keys that are never asked for again (the last hour's window
of every allowance, the dashboard figures of a fingerprint that has
since changed, allauth's history of an address) therefore stayed for
good, while every write counted the whole table or listed the whole
directory.

core.cache_purge removes what is already past its expiry, at most once
an hour behind a cache.add gate, from the scheduler's pass and from the
hourly tidy a request does:

- the rows of the default cache when it is the database backend, in one
  DELETE, the statement DatabaseCache._cull issues;
- the files of the allowances' cache when it is a FileBasedCache, each
  judged by the backend's own _is_expired, and held to five seconds a
  run so a request never waits on a directory built up over a year.

A live entry is never touched, so no allowance starts again and #755's
protection is unchanged. With Redis or Memcached it does nothing, not
even take the gate.

Closes #1167

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

Try a failed notifier send again on the webhook's schedule ... 5e4905e12e

A notifier's send was called once. When it raised, the failure was written
on the connection and the notification was gone: whoever announced it had
already stamped it as announced, so a reminder that fell due in the minute
the mail server, Telegram or a push service could not be reached never
arrived, while a webhook delivery was retried for about ten hours.

A send that fails with an ordinary exception now leaves a row in the queue
a webhook's deliveries wait in, already one attempt old, and the pass that
delivers webhooks sends it again: five minutes on, then doubling, eight
attempts in all, claimed with the same conditional UPDATE so two schedulers
send a retry once between them. The row holds the notification as a webhook
is sent it, which is every field, so the notifier is handed the same
notification with the same key, in the language its words are in.

The outbox was generalised rather than given a second table beside it: no
model, no migration and no change to the archive, and one retry mechanism
to reason about. A retry row therefore obeys the rules #752 put on a
webhook's row without a line written for it: it goes with its connection
and its account, its text is cleared once it has arrived, it is forgotten a
week after it was sent or given up on, and it was never in an archive. A
connection that is switched off or retired is not dialled, nor is a
deactivated account's. ConnectionUnusable still retires the connection and
queues nothing; met on a retry, it retires the connection and gives the row
up. A notifier that only queues (the webhook) is left to what delivers for
it. The failure is still written on the connection at once.

A send that reached some of a connection's destinations before raising will
reach those again on the retry; that is accepted, and the notifier protocol
says so.

The one query the queue made without an owner now names it, so its line
leaves the list of unscoped queries.

Closes #1163

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

Quieten the idle worker: no log line and no heartbeat write on every pass ... 1316784cc5

A worker with nothing to do goes round every two seconds, and each time the
library under it logged "Starting worker" and "No more tasks to run" at INFO
and the command rewrote its heartbeat file. That is 86,400 lines a day, about
14.5 MB of JSON into a kept log of 20 MB that the web process shares, so the
web's errors were pushed out of it in about a day and a half; and 43,200 file
writes a day to say what its readers only ask about every five minutes.

`LOGGING` now hears `django_tasks_db` from WARNING up, so a stop asked for and
a task that failed unexpectedly are still written. In place of the library's
lines, `manage.py work` says once that it has started (not with `--once`: run
from cron every minute that would be the noise back again).

The heartbeat is written on the first pass, so a worker that has just started
is seen at once, and then when the last write is a minute old, counted on the
monotonic clock. The healthcheck and the page that watches slow work both allow
five minutes, and a test holds them at twice the interval or more.

Measured over 100 idle passes (200 s on a faked clock): 200 log lines and
33,600 bytes of JSON before, 1 line and 191 bytes after; 100 heartbeat writes
before, 4 after.

Closes #1117

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

Ask the gone-quiet pass only for what is still to be said ... 450c89487b

Every five minutes the scheduler asked for the owners of any application,
then for each of them read the account a second time just for its
threshold, and loaded every quiet application of theirs, with its listing
and its company, to find in Python that it had already been announced. A
pass with nothing new cost 1 + 2N queries for N accounts, including the
accounts that have only drafts or applications that ended.

The owners are now read once with their profiles, and only the active ones
who have an application in a status that can go quiet. The "not yet said
for this silence" test is part of the query, so an application already
announced is not loaded at all. A pass with nothing new is 1 + N queries
(7 to 4 for three accounts) and reads no application.

What is announced does not change: the same applications, once per
silence, with each person's own threshold and in their language, and the
claim that keeps two schedulers from saying it twice is as it was.

Closes #1120

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

Give the document copies a time budget in the scheduler pass ... 07bf8d907d

`archiving.send_pending` sent every due copy, up to fifty, before the
pass went on to the webhooks and the syncs. A store that takes twenty
seconds over each document held all of that for a quarter of an hour.

The pass now stops starting copies once sixty seconds have gone, as the
webhooks do, and leaves the rest to the next pass. A copy already
started is never cut off, so a pass spends the budget and one copy more
at most. What is left is neither claimed nor counted as tried: it keeps
the time it was due and stands first next time, ahead of any copy that
failed and waits for its retry, so nothing is starved by a slow store.
The counts the pass returns, and the line the scheduler writes from
them, are of the copies it started.

With fifty copies at twenty seconds each, one pass held the scheduler
for 1000 s; it now holds it for 60 s, and the fifty go out over
seventeen passes.

Closes #1172

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

Record bunch 6 in the changelog ... a6554cc11d

One line for each of the eight scheduler and worker issues: the upload
scrub, the orphan sweep, the gone-quiet check, the idle worker, the
document copies' time budget, the notifier retries, the queue's finished
task rows and the caches' expired entries.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

Run the cache purge's tests on PostgreSQL as well ... 2e3064405b

The purge of expired cache rows is one statement written by hand, the one
Django's own cull issues: it compares the expiry with the present moment in
the form the database keeps a date. The unit job proves it on SQLite only,
and a comparison that went wrong on the other engine would delete a live
rate-limit counter, which #755 exists to prevent. Its test file joins the
six the PostgreSQL job already runs; it takes seconds.

Refs #1167

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

tiagoagueda added 1 commit 2026-11-11 11:19:11 +00:00

Keep the test's connection when a cache purge test runs a scheduler pass ... f0f0b5d803

The scheduler command closes old connections when a pass ends. Under the
test settings a connection is old at once, and it is the one holding the
test's transaction. In-memory SQLite is never closed, so the unit job saw
nothing; on PostgreSQL, where this file now runs as well, the statement
after the pass found its connection gone and the test failed on that, not
on what it checks. The file's other nine tests passed there, the ones that
run the purge's statement without the command among them.

The file's tests now leave the command's closing out, which is not what
they are about.

Refs #1167

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

tiagoagueda merged commit f0f0b5d803 into main 2026-11-11 11:52:17 +00:00

tiagoagueda deleted branch b6-scheduler 2026-11-11 11:52:19 +00:00

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

accessibility

Usable by everyone, including people with disabilities: one of the project's stated missions

  

authentication

Signing in, and proving who you are

  

breaking change

Changes behaviour or data in a way operators must act on

  

bug

Something does not do what it says it does

  

documentation

The wiki, the docs, or the words in the interface

  

enhancement

An improvement or a new capability

  

interface

The shape of the pages: what is where, and how it reads

  

internationalisation

Languages, regions, and the formats and symbols that go with them

  

observability

What an operator can see about a running instance: metrics, logs, health

  

performance

Uses fewer resources, or is more reliable, with no loss of features or accessibility

  

security

Keeping the code and the data secure: one of the project's stated missions

  

tier

1

Easy: small, self-contained, no prerequisites

  

tier

2

High-severity security: account access, personal data, the boundaries

  

tier

3

Foundations: prerequisite work that unblocks other issues

  

tier/4

Later: large features and long-horizon work

No labels

accessibility

authentication

breaking change

bug

documentation

enhancement

interface

internationalisation

observability

performance

security

tier

1

tier

2

tier

3

tier/4

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

Postulo/postulo!1228

No description provided.