Postulo/postulo

Watch

Fork 0

0.6.0 fixes, bunch 3: reliability bugs #1225

Merged

tiagoagueda merged 8 commits from b3-reliability into main

2026-11-10 14:12:34 +00:00

tiagoagueda commented

2026-11-10 13:37:56 +00:00

Owner

Bunch 3 of the 0.6.0 fixes: reliability bugs. One signed commit per issue, then one with the changelog lines, the extracted strings and the PostgreSQL job's file list; merge fast-forward-only.

Closes #1164: add_reminder writes nothing when the entry is unchanged, and says whether it changed.
Closes #1129: five read-only views no longer run inside the request's transaction on SQLite.
Closes #1162: a failing sync backs off, capped at one day; ConnectionUnusable is honoured for syncs.
Closes #1123: a backup is written under a partial name and renamed once verified.
Closes #1130: the scheduler, backup and sync leases live in a table, taken with one conditional statement (migration core 0041).
Closes #1131: the backup lease is short and renewed by the running backup, ported onto #1130's table.
Closes #1122: an errand page says when the worker has not been seen, polls less often, and work interrupted mid-task is reported after an hour.

Tested locally: lint, migration and catalogue checks, the full unit suite (21,257 passed after the one guard it found was recorded), and the browser tests of the touched pages, without the accessibility walks (no axe-core in the worktree): the browser job here runs those. The race fixed in #1130 is only really run by the PostgreSQL job, which now includes tests/test_leases.py.

Two new strings for Weblate: "Next attempt %(when)s" and the errand page's sentence for a silent worker.

Known limit, recorded in #1131's commit: a single backup step longer than ten minutes lets the lease lapse.

🤖 Generated with Claude Code

Bunch 3 of the 0.6.0 fixes: reliability bugs. One signed commit per issue, then one with the changelog lines, the extracted strings and the PostgreSQL job's file list; merge fast-forward-only. - Closes #1164: `add_reminder` writes nothing when the entry is unchanged, and says whether it changed. - Closes #1129: five read-only views no longer run inside the request's transaction on SQLite. - Closes #1162: a failing sync backs off, capped at one day; `ConnectionUnusable` is honoured for syncs. - Closes #1123: a backup is written under a partial name and renamed once verified. - Closes #1130: the scheduler, backup and sync leases live in a table, taken with one conditional statement (migration `core` 0041). - Closes #1131: the backup lease is short and renewed by the running backup, ported onto #1130's table. - Closes #1122: an errand page says when the worker has not been seen, polls less often, and work interrupted mid-task is reported after an hour. Tested locally: lint, migration and catalogue checks, the full unit suite (21,257 passed after the one guard it found was recorded), and the browser tests of the touched pages, without the accessibility walks (no axe-core in the worktree): the browser job here runs those. The race fixed in #1130 is only really run by the PostgreSQL job, which now includes `tests/test_leases.py`. Two new strings for Weblate: "Next attempt %(when)s" and the errand page's sentence for a silent worker. Known limit, recorded in #1131's commit: a single backup step longer than ten minutes lets the lease lapse. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

tiagoagueda added 8 commits

2026-11-10 13:37:56 +00:00

Leave an unchanged calendar entry alone in add_reminder 0613e794bd

A plugin that syncs a calendar calls add_reminder for every entry it
knows on every run. Each call saved the reminder whether or not anything
differed, which moved updated_at, and with it the fingerprint the
dashboard's figures are cached under: the first view of the dashboard
after every sync rebuilt them from every application.

add_reminder now compares the summary, note, due date, actor and whether
the entry is retired with what is stored, and writes nothing when they
are the same. The reminder it returns carries `changed`, true when the
call made or altered the entry, so a caller can count what it really
pushed; the return type is as before.

Closes #1164

Co-Authored-By: Claude <noreply@anthropic.com>

Serve pictures, the worker, the manifest and the errand poll outside a transaction b9d373fb02

On SQLite every request's transaction begins IMMEDIATE, so it takes the
write lock on entering the view and keeps it until the view returns. Five
views that only read were paying for that: a company's logo (one request
per card on a board), a person's picture, the service worker, the web app
manifest, and the fragment that polls a piece of slow work every two
seconds, which queued behind the very work it was watching. Under WAL a
read outside a transaction waits for no writer.

Each now carries transaction.non_atomic_requests, with its reason beside
it, and config/sqlite.py lists them. None of them writes: a test asks for
each one and looks at every statement it sent, and another holds the
decorator in place so a refactor does not quietly put them back. The
notices poll stays in its transaction, since it deletes and marks together.

Closes #1129

Co-Authored-By: Claude <noreply@anthropic.com>

Try a failing sync less and less often, and let a sync end its connection da61dae70c

A sync connection that failed was due again one interval later, for ever:
after a password change, the mail, calendar or France Travail server got a
refused sign-in every interval, up to 96 a day, until the person noticed.
The scheduler now waits the longer of the interval and the time the
connection has been failing (since it last worked, or since it was made),
never more than a day. That doubles the wait at each failure with no new
column, and the first run that works puts it back on its interval. Sync
now is not held back, and the connection's card says when the next attempt
is, so the longer wait does not read as a sync that stopped.

A sync that raised ConnectionUnusable was logged as a crash and dialled
again. It is now heard as it is for stores and notifiers: the connection is
switched off with the plugin's sentence. Only a plugin decides that; the
core never switches a failing connection off by itself.

Closes #1162

Co-Authored-By: Claude <noreply@anthropic.com>

Give a backup its name only once it is written and verified 7e72210521

An archive was opened under its final name and verified afterwards, and
nothing removed it when the write or the check failed. A run cut short by a
full disk, a killed container or a failed check therefore left a file that
the Backups page listed as an ordinary backup (its manifest is the first
member, so the counts were there) and that retention counted among the
newest kept, deleting one more good archive than intended.

write_backup now writes to a dot-file beside the target, verifies that, and
only then renames it into place; any failure removes the file. A process
killed outright cannot remove its own, so the scheduler removes such files
once they are six hours old and logs how many.

Closes #1123

Co-Authored-By: Claude <noreply@anthropic.com>

Keep the scheduler, backup and sync leases in a table of their own f97d1e3c50

The three leases were taken with cache.add. On the database cache that is
a SELECT and then an UPDATE by key alone, so on PostgreSQL two processes
arriving together over an expired lease both read it as expired and both
wrote their token: two scheduler passes, two backups or two syncs of one
connection ran side by side, which is what the leases exist to prevent.
An expired lease is what a killed worker leaves behind.

core.leases keeps one row per lease and takes, releases and renews it
with one conditional statement each, so the database decides: a taker
that waited on another's update finds the row no longer expired and
changes nothing. Releasing by token is the compare-and-delete the cache
lacked, and the lease no longer goes with a cache that is flushed or
pointed elsewhere.

Closes #1130

Co-Authored-By: Claude <noreply@anthropic.com>

Hold the backup claim as a short lease the running backup renews 1cfd963e64

The claim on "a backup is running" lasted six hours so that it outlived the
slowest backup. A worker killed mid-backup never released it, so the
scheduled backup was refused for up to six hours and a daily slot inside
them was lost. Back up now was refused for longer still: the page also
trusted any unfinished backup errand of the last day, and the killed one
stayed "working" until the day-long sweep, with nothing to clear it.

The claim now lasts ten minutes and the backup renews it as it works:
write_backup takes an optional callable it calls between its steps and for
every file it adds, and take() passes one that touches the cache key at
most once a minute. The page asks the claim, and counts an unfinished
errand only while it is younger than the claim lasts, so one just queued
still stops a second press. The scheduler marks a backup errand that is
working with no claim behind it as interrupted within minutes, not a day.

Closes #1131

Co-Authored-By: Claude <noreply@anthropic.com>

Say so on an errand's page when the worker has not been seen dc07d5c971

With the worker stopped, crashed or never started, a page watching a piece
of slow work went on saying "The work carries on without you" and asked
again every two seconds, 1,800 requests an hour, until the scheduler failed
the errand a day after it was asked for.

The page now reads the worker's heartbeat. When there is none, or it is
older than the five minutes the worker's own healthcheck allows, an
unfinished errand says that the background worker has not been seen for a
while and that the work will start when it is back, and asks again every
thirty seconds instead of two. The sentence takes the place of the old one
inside the fragment, so the page's live region announces it as it does any
other change of state, and the Check again link stays.

An errand that is working is never begun twice, so the scheduler now calls
it interrupted an hour after it began rather than a day after it was asked
for. One still waiting keeps the day: a worker that comes back will do it.

The suite gives every test a worker that has just been round, in a file of
its own, so that the heartbeat in a checkout's data directory no longer
decides what a test sees.

Closes #1122

Co-Authored-By: Claude <noreply@anthropic.com>

Record bunch 3, extract its strings and test the leases on PostgreSQL

CI / Unit tests and coverage (Python 3.14) (pull_request) Successful in 15m51s

Details

CI / Checks: lint, migrations, catalogues, settings, built files (pull_request) Successful in 1m53s

Details

CI / Database tests on PostgreSQL (pull_request) Successful in 4m5s

Details

CI / Security audit (pull_request) Successful in 19s

Details

CI / Browser tests (Chromium) (pull_request) Successful in 33m38s

Details

CI / Unit tests and coverage (Python 3.14) (push) Successful in 17m5s

Details

CI / Checks: lint, migrations, catalogues, settings, built files (push) Successful in 2m0s

Details

CI / Database tests on PostgreSQL (push) Successful in 4m8s

Details

CI / Security audit (push) Successful in 19s

Details

README / README language tables (push) Successful in 20s

Details

CI / Browser tests (Chromium) (push) Successful in 35m8s

Details

Dev image / Dev image: build, scan, push (push) Successful in 47m28s

Details

351100a878

The changelog lines of the seven reliability fixes before this commit, one
per issue. The catalogues gain the two strings those fixes added ("Next
attempt ..." on a failing sync's card, and the errand page's sentence when
the worker has not been seen), untranslated: they are translated in Weblate.

tests/test_leases.py joins the files CI's PostgreSQL job runs. Its test of
two connections taking one expired lease skips itself on SQLite, so that job
is the only place the race fixed in #1130 is really run.

Refs #1164, #1129, #1162, #1123, #1130, #1131, #1122

Co-Authored-By: Claude <noreply@anthropic.com>