Donate to Postulo/postulo

• https://buymeacoffee.com/tiagoagueda

---

See the documentation for more details on donation links.

0.6.0 fixes, bunch 7: storage, search and SQLite #1229

Merged

tiagoagueda merged 7 commits from b7-storage into main 2026-11-11 14:15:12 +00:00

Conversation 0 Commits 7 Files changed 113 +3352 -109

tiagoagueda commented 2026-11-11 13:40:10 +00:00

Owner

• Copy link
• Copy Markdown

Bunch 7 of the 0.6.0 fixes: storage, search and SQLite. One signed commit per issue, then the changelog lines; merge fast-forward-only.

• Closes #1116: a search group counts its matches in the query that fetches them, with a window count read off the first row (42 queries to 22 on the test fixture). A queryset a window would miscount (distinct, sliced, a union, values()) keeps the separate count. Totals and hits are compared with the old two queries on data built to inflate a join.
• Closes #1134: SQLite's LIKE answers plain ASCII text for the four lookup shapes without the regular expression (20,000 rows and two columns: 42 ms to 29 ms). Anything else, any non-ASCII value or pattern included, goes the way it did. The two ways are compared over 11,664 fixed cases and a seeded sweep of 20,000.
• Closes #1124: backups are written at gzip level 6, a constant. Reading, verifying and restoring are untouched; the backup format number does not change.
• Closes #1125: an export is written once, as a dot-file beside the export archives on the data volume, and moved into place; each kept file is streamed in 1 MiB pieces (one 20 MB upload: 71 MB peak to 5 MB, nothing in /tmp). A killed export's leftover goes with the orphan sweep after its grace period. The backup no longer stops on a file that leaves the media tree while it is walked.
• Closes #1127: fetch_geonames removes the city and postal-code text tables once the lookup built from each has passed its checks (about 32 MB freed). The lookup alone is the dataset; a table placed by hand is still indexed and left where it is. A lookup standing alone writes its country names again in place when they are out of date (a newer CLDR, a new language, a lookup from before #1109), so postulo.W003 now means it could not.
• Closes #1128: file-backed SQLite connections hold the write-ahead log to 64 MB; PRAGMA optimize runs at most once a day from the scheduler or the hourly tidy; manage.py compact_database (with --dry-run) runs VACUUM after checking the free disk, and Server Overview shows the unused space. Two new strings for Weblate. Nothing at all on PostgreSQL.

Tested locally: lint, migration and catalogue checks, the full unit suite (21,502 passed, 189 skipped), and the browser test files that visit the Server Overview (31 passed; the axe walks skip on this machine and run here).

Not exercised locally: PostgreSQL (the window count is standard SQL and runs there only in production; the SQLite work does not apply), and the image itself, whose SQLite may be older than the 3.50 these were measured on: everything used exists since 3.40, and the two newer bits of the optimize mask should simply be ignored before 3.46 (not tested on an older SQLite).

🤖 Generated with Claude Code

Bunch 7 of the 0.6.0 fixes: storage, search and SQLite. One signed commit per issue, then the changelog lines; merge fast-forward-only. - Closes #1116: a search group counts its matches in the query that fetches them, with a window count read off the first row (42 queries to 22 on the test fixture). A queryset a window would miscount (distinct, sliced, a union, `values()`) keeps the separate count. Totals and hits are compared with the old two queries on data built to inflate a join. - Closes #1134: SQLite's `LIKE` answers plain ASCII text for the four lookup shapes without the regular expression (20,000 rows and two columns: 42 ms to 29 ms). Anything else, any non-ASCII value or pattern included, goes the way it did. The two ways are compared over 11,664 fixed cases and a seeded sweep of 20,000. - Closes #1124: backups are written at gzip level 6, a constant. Reading, verifying and restoring are untouched; the backup format number does not change. - Closes #1125: an export is written once, as a dot-file beside the export archives on the data volume, and moved into place; each kept file is streamed in 1 MiB pieces (one 20 MB upload: 71 MB peak to 5 MB, nothing in /tmp). A killed export's leftover goes with the orphan sweep after its grace period. The backup no longer stops on a file that leaves the media tree while it is walked. - Closes #1127: `fetch_geonames` removes the city and postal-code text tables once the lookup built from each has passed its checks (about 32 MB freed). The lookup alone is the dataset; a table placed by hand is still indexed and left where it is. A lookup standing alone writes its country names again in place when they are out of date (a newer CLDR, a new language, a lookup from before #1109), so `postulo.W003` now means it could not. - Closes #1128: file-backed SQLite connections hold the write-ahead log to 64 MB; `PRAGMA optimize` runs at most once a day from the scheduler or the hourly tidy; `manage.py compact_database` (with `--dry-run`) runs `VACUUM` after checking the free disk, and Server Overview shows the unused space. Two new strings for Weblate. Nothing at all on PostgreSQL. Tested locally: lint, migration and catalogue checks, the full unit suite (21,502 passed, 189 skipped), and the browser test files that visit the Server Overview (31 passed; the axe walks skip on this machine and run here). Not exercised locally: PostgreSQL (the window count is standard SQL and runs there only in production; the SQLite work does not apply), and the image itself, whose SQLite may be older than the 3.50 these were measured on: everything used exists since 3.40, and the two newer bits of the optimize mask should simply be ignored before 3.46 (not tested on an older SQLite). 🤖 Generated with [Claude Code](https://claude.com/claude-code)

tiagoagueda added 7 commits 2026-11-11 13:40:10 +00:00

Write a backup at gzip level 6 ... 046cd944e6

The archive was opened with tarfile's "w:gz", which compresses at level 9
unless told. On the database that took about 45% longer than level 6 for
an archive under 4% smaller, and on media, which does not compress, the
level changes nothing. Measured through the same writer on a 30.5 MB
SQLite file: about 1.2 s and 15.91 MB at level 9, about 0.85 s and
16.49 MB at level 6.

The level is a named constant, GZIP_LEVEL, not a setting. It is the one
place an archive is compressed: pg_dump already compresses its dump at
gzip's default, the offsite copy encrypts the file as it is, and an
uploaded archive is kept as it arrived. Reading does not depend on the
level, so the format number stays 2 and archives written at level 9 are
verified and restored as before.

Closes #1124

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

Write an export once, on the data volume, a piece of each file at a time ... 96436e2073

An export was built in the system's temporary directory and then copied
into the store, so every archive was written twice. Under the hardened
compose file /tmp is tmpfs and counts against the worker's 1 GB, and an
account may keep a gigabyte of captured pages, so a large export sat in
the worker's memory allowance while it was built and copied. Each kept
file was also read whole to be put in the zip.

The archive is now written as a dot-file in the person's own folder of
archives, made by the store so it has the modes every kept file has, and
handed to the store as an object with temporary_file_path(), which
FileSystemStorage renames into place. A store that is not a directory on
this machine still gets a file in the temporary directory to copy. Each
kept file is copied into the zip one megabyte at a time; its member is
given the entry writestr made (dated now, compressed as the archive is),
so the archive reads as before.

Measured with one 20 MB upload: peak traced memory while the archive is
written 71.0 MB before, 5.0 MB after; 20 MB in the temporary directory
and 20 MB copied by the store before, none and one rename after.

A worker killed half way leaves the dot-file. No row names it, so the
sweep of unnamed files removes it once it is past the grace period, and
leaves alone one that is still being written, which is as old as its
last write. A failed export removes it at once. Nothing lists or serves
it. A backup walks the media tree, where a file can now be renamed away
between being listed and being read: the walk and the count before it
leave such a file out instead of stopping the backup.

Closes #1125

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

Remove the downloaded GeoNames tables once their lookups are built ... 80c6df324f

`fetch_geonames` kept the city table beside the lookup built from it:
32 MB on every data volume, and the postal codes' table with it where
those are on. After the build the city table was read for its presence
and its date only, and the postal codes' table for nothing.

The lookup alone is the dataset now. `available()` and `_index()` take
it without a table, and the entrypoint downloads only where neither the
lookup nor a table is there, so a start does not fetch again what the
last one removed. The command removes a table after the lookup built
from it has answered what the download was asked (the cities it must
hold, their count, what its country names were read from): a build that
stops, or a lookup that does not answer, leaves the table where the
next start indexes it, and a build that stops leaves the lookup before
it too.

A table an operator puts there by hand is still indexed when it is
newer than the lookup, and is left alone: only the command removes a
table, and only the one it downloaded. So an instance from before this
keeps its table, and its lookup from before #1109 is still built again
from it at its first use.

Without the table a lookup cannot be built again, and it need not be:
the cities do not go out of date, and the countries' names, which do
(another CLDR, another language, a newer country table, or a lookup
from before #1109, which added those two tables and nothing else), are
tables of their own, read from CLDR and the country table that is kept.
A lookup that stands alone has them written again at its first use, in
place and in one transaction, by the same code that writes them at a
build; the cities and their names are not touched. The write lock is
taken first, so of the workers that start together one writes and the
others wait and find it done. Otherwise every upgrade that brings a
newer CLDR or a language would have asked for the 32 MB again.

Where the names cannot be written (no country table to read them from,
a volume mounted read-only, a lock nobody lets go of) the lookup is
read as it is: the log says so once, and postulo.W003 says so at
start-up, both with the command that downloads the tables and builds
from them. W003 says nothing of names the first use puts right. A file
under the lookup's name that is not a lookup is an absent dataset
rather than an error on every save, and a worker that saw the table a
moment before the command removed it reads the lookup.

On the tests' tables (10,004 cities; 100,003 postal codes) a fetch
with the postal codes left 5,097,878 bytes beside the two lookups, and
leaves 20,794: the country table.

Closes #1127

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

Count a search group's matches in the query that fetches them ... db9b22b68f

Each group of the search asked two questions: a COUNT for the "All N" line,
then the first five rows. Both filter the same rows with the same LIKEs, and
on SQLite every LIKE is a call into Python (#505), so every row of every
searched table went through it twice. One search over the nine groups and
the eleven models of the career made 42 queries of its own.

take() now annotates the rows with COUNT(*) OVER () and reads the total off
the first row, or 0 when there is none: 22 queries for the same search, and
the search page with 300 companies goes from 49 queries to 29. The number
is the one COUNT gave, since a window is worked out after the WHERE and
before the LIMIT, and no group joins a table that could repeat a row (they
use Exists). A queryset where that would not hold, one that is distinct,
already sliced, a union or values(), keeps the separate COUNT.

Closes #1116

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

Answer SQLite's LIKE for plain ASCII text without the regular expression ... 09430efb26

SQLite calls Postulo's LIKE (#505) once for every row and every searched
column, and each call ran a Unicode case-insensitive regular expression:
about 630 ns, whatever the text was. For ASCII text, folding case is
turning A-Z into a-z, and the four lookups Django compiles to LIKE ask one
of four plain questions.

like() now answers those with str methods when the value is an ASCII str
and the pattern, read with the escape rule of _compile, is an ASCII literal
with a % at the ends only: contains, starts with, ends with, equals. About
375 ns a call; 20,000 ASCII rows over two columns go from 42 ms to 29 ms.
A value or pattern with a letter outside ASCII, a _ wildcard, a % in the
middle, a number or a blob is answered by the regular expression as before,
so the Kelvin sign, the long s and the dotted capital I fold as they did.
The pattern's two readings are cached together, so a call still looks one
thing up.

The function as it was stays in the module as like_by_expression, and the
tests hold like() to it over a fixed table and a seeded sweep.

Closes #1134

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

Cap SQLite's log, refresh its statistics daily and add compact_database ... d6dd0da6b2

A file-backed instance opened its database with WAL, a longer wait and
immediate transactions, and did nothing else to it, ever. Three things
followed, each small and each for good.

The write-ahead log never got smaller. SQLite checkpoints it but keeps the
file at its largest until the last connection closes, and with three
workers, a scheduler and a worker one is nearly always open: after one
large write (an import, a restore, a first purge) the log stayed that size.
Every connection to a file now sets journal_size_limit to 64 MB, in the
same init command as the other pragmas, so the log is cut back the next
time a write starts it again.

The query planner had no statistics. Nothing ran ANALYZE, so sqlite_stat1
was whatever a migration left, which is nothing. The scheduler's pass and
the hourly tidy a request does now run PRAGMA optimize at most once a day,
behind a cache key of its own, as the cache purge is (#1167). It is asked
with the mask 0x10012 and not bare: the bare form judges a table by what
the connection running it has queried, and here that connection lives for
one pass or one request. Measured on SQLite 3.50, a table that had grown
two hundredfold was left alone by a fresh connection, and analysed with
the mask; the sampling bit keeps it to milliseconds (18 ms for 400,000
rows). It runs inside a request's transaction as well as outside one. A
database it cannot write to in time is logged and left for the next day.
On PostgreSQL nothing is issued and no key is written.

The file never shrank. Deleted rows leave free pages that are used again
but never given back, and a backup copies them. `manage.py
compact_database` says how much they come to and then runs VACUUM;
--dry-run stops at the figure. It is a command and not a schedule because
VACUUM holds every other writer up for its whole run and writes the pages
in use twice before the file shrinks, to a temporary file and to the log
beside the database: it refuses, with the figures, when either place has
too little free disk. It empties the log afterwards, or the saving would
not show until later writes had gone round it. The Server Overview page
shows the same figure for SQLite, from two pragmas, so an administrator
can see when it is worth running.

Closes #1128

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

Record bunch 7 in the changelog ... 2f7d26ecd4

One line for each of the six storage, search and SQLite issues: the search
that counts beside its rows, the ASCII way through SQLite's LIKE, backups
at gzip level 6, the export written once on the data volume, the GeoNames
tables removed after their lookups are built, and SQLite's upkeep. The
catalogues gain the two strings of the Server Overview's new row, empty,
for Weblate.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

tiagoagueda merged commit 2f7d26ecd4 into main 2026-11-11 14:15:12 +00:00

tiagoagueda deleted branch b7-storage 2026-11-11 14:15:14 +00:00

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

accessibility

Usable by everyone, including people with disabilities: one of the project's stated missions

  

authentication

Signing in, and proving who you are

  

breaking change

Changes behaviour or data in a way operators must act on

  

bug

Something does not do what it says it does

  

documentation

The wiki, the docs, or the words in the interface

  

enhancement

An improvement or a new capability

  

interface

The shape of the pages: what is where, and how it reads

  

internationalisation

Languages, regions, and the formats and symbols that go with them

  

observability

What an operator can see about a running instance: metrics, logs, health

  

performance

Uses fewer resources, or is more reliable, with no loss of features or accessibility

  

security

Keeping the code and the data secure: one of the project's stated missions

  

tier

1

Easy: small, self-contained, no prerequisites

  

tier

2

High-severity security: account access, personal data, the boundaries

  

tier

3

Foundations: prerequisite work that unblocks other issues

  

tier/4

Later: large features and long-horizon work

No labels

accessibility

authentication

breaking change

bug

documentation

enhancement

interface

internationalisation

observability

performance

security

tier

1

tier

2

tier

3

tier/4

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

Postulo/postulo!1229

No description provided.