Donate to Postulo/postulo

• https://buymeacoffee.com/tiagoagueda

---

See the documentation for more details on donation links.

Keep the browser tests' database in a file, a connection per request (#1024) #1074

Merged

tiagoagueda merged 1 commit from 1024-live-server-db into main 2026-10-29 10:18:31 +00:00

Conversation 0 Commits 1 Files changed 5 +235 -12

tiagoagueda commented 2026-10-28 14:37:43 +00:00

Owner

• Copy link
• Copy Markdown

On an in-memory SQLite database pytest-django's live server hands the test
thread's one connection to every request thread, and every request is a
transaction (ATOMIC_REQUESTS). Requests in flight at once tangled their
transactions in that one wrapper's bookkeeping: a commit took away another
request's savepoint, and the browser job failed at random with "no such
savepoint" and then a TransactionManagementError on every request and
teardown after it.

A run with an e2e test in it now gives the test database a file of its own
(django_db_modify_db_settings in tests/conftest.py), one per xdist worker in
a temporary directory removed at the end, opened with the instance's own
options (IMMEDIATE transactions, WAL, a 20-second busy timeout; #206), so a
second writer waits its turn. pytest-django shares only in-memory
connections, so each request thread connects for itself. The live server
now waits for the test database (tests/e2e/conftest.py): the font check
started it first, while the connection still named :memory:, and it shared
the connection all the same. The unit suite and a PostgreSQL run are left
as they were.

tests/e2e/test_live_server_connections.py plays the failing order out with
three requests and a pause held open between SAVEPOINT and Django noting it
down; it failed with the CI's errors before the change and passes after.

Closes #1024

Notes for review

• Root cause: pytest-django builds connections_override when the live server starts. The autouse font check started the server before django_db_setup, while the database was still :memory:, so every request thread used the test thread's own connection. With ATOMIC_REQUESTS, the savepoints of overlapping requests got released by the wrong thread: "no such savepoint", then a TransactionManagementError cascade (1,829 errors in one CI run).

• Fix:

  • When a run contains browser tests, the test database is a file, one per xdist worker, with the production SQLite options (IMMEDIATE, WAL, 20 s timeout). It is removed at the end.
  • live_server now depends on django_db_setup, so the override is empty.
  • Unit runs and PostgreSQL runs are untouched.

• Regression test: tests/e2e/test_live_server_connections.py reproduces the failure deterministically (failed before the fix, passes after) and asserts the override is empty.

• Timings, each file alone, run twice, in seconds:

  File	Before	After
  test_reflow	180 / 149	121 / 118
  test_every_page	443 / 342	317 / 343
  test_accessibility	91 / 111	88 / 83

• Not yet run: with -n, where the per-worker file is reasoned from pytest-django's source. That arrives with #1033's second worker.

🤖 Generated with Claude Code

On an in-memory SQLite database pytest-django's live server hands the test thread's one connection to every request thread, and every request is a transaction (ATOMIC_REQUESTS). Requests in flight at once tangled their transactions in that one wrapper's bookkeeping: a commit took away another request's savepoint, and the browser job failed at random with "no such savepoint" and then a TransactionManagementError on every request and teardown after it. A run with an e2e test in it now gives the test database a file of its own (django_db_modify_db_settings in tests/conftest.py), one per xdist worker in a temporary directory removed at the end, opened with the instance's own options (IMMEDIATE transactions, WAL, a 20-second busy timeout; #206), so a second writer waits its turn. pytest-django shares only in-memory connections, so each request thread connects for itself. The live server now waits for the test database (tests/e2e/conftest.py): the font check started it first, while the connection still named :memory:, and it shared the connection all the same. The unit suite and a PostgreSQL run are left as they were. tests/e2e/test_live_server_connections.py plays the failing order out with three requests and a pause held open between SAVEPOINT and Django noting it down; it failed with the CI's errors before the change and passes after. Closes #1024 ## Notes for review - **Root cause:** pytest-django builds `connections_override` when the live server starts. The autouse font check started the server before `django_db_setup`, while the database was still `:memory:`, so every request thread used the test thread's own connection. With `ATOMIC_REQUESTS`, the savepoints of overlapping requests got released by the wrong thread: "no such savepoint", then a `TransactionManagementError` cascade (1,829 errors in one CI run). - **Fix:** - When a run contains browser tests, the test database is a file, one per xdist worker, with the production SQLite options (IMMEDIATE, WAL, 20 s timeout). It is removed at the end. - `live_server` now depends on `django_db_setup`, so the override is empty. - Unit runs and PostgreSQL runs are untouched. - **Regression test:** `tests/e2e/test_live_server_connections.py` reproduces the failure deterministically (failed before the fix, passes after) and asserts the override is empty. - **Timings, each file alone, run twice, in seconds:** | File | Before | After | |---|---|---| | `test_reflow` | 180 / 149 | 121 / 118 | | `test_every_page` | 443 / 342 | 317 / 343 | | `test_accessibility` | 91 / 111 | 88 / 83 | - **Not yet run:** with `-n`, where the per-worker file is reasoned from pytest-django's source. That arrives with #1033's second worker. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

tiagoagueda added 1 commit 2026-10-28 14:37:43 +00:00

Keep the browser tests' database in a file, a connection per request (#1024) ... 241bbf93b7

On an in-memory SQLite database pytest-django's live server hands the test
thread's one connection to every request thread, and every request is a
transaction (ATOMIC_REQUESTS). Requests in flight at once tangled their
transactions in that one wrapper's bookkeeping: a commit took away another
request's savepoint, and the browser job failed at random with "no such
savepoint" and then a TransactionManagementError on every request and
teardown after it.

A run with an e2e test in it now gives the test database a file of its own
(django_db_modify_db_settings in tests/conftest.py), one per xdist worker in
a temporary directory removed at the end, opened with the instance's own
options (IMMEDIATE transactions, WAL, a 20-second busy timeout; #206), so a
second writer waits its turn. pytest-django shares only in-memory
connections, so each request thread connects for itself. The live server
now waits for the test database (tests/e2e/conftest.py): the font check
started it first, while the connection still named :memory:, and it shared
the connection all the same. The unit suite and a PostgreSQL run are left
as they were.

tests/e2e/test_live_server_connections.py plays the failing order out with
three requests and a pause held open between SAVEPOINT and Django noting it
down; it failed with the CI's errors before the change and passes after.

Closes #1024

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

tiagoagueda referenced this pull request 2026-10-28 18:20:13 +00:00

Cache what every CI job rebuilt, and fetch no map tiles in tests (#1033) #1083

tiagoagueda force-pushed 1024-live-server-db from 241bbf93b7 to ece1170f6a 2026-10-28 19:48:39 +00:00 Compare

tiagoagueda force-pushed 1024-live-server-db from ece1170f6a to 9691831204 2026-10-29 10:13:32 +00:00 Compare

tiagoagueda merged commit 76fbda5110 into main 2026-10-29 10:18:31 +00:00

tiagoagueda deleted branch 1024-live-server-db 2026-10-29 10:18:32 +00:00

tiagoagueda referenced this pull request from a commit 2026-10-29 10:18:34 +00:00

Keep the browser tests' database in a file, a connection per request (#1024) (#1074)

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

accessibility

Usable by everyone, including people with disabilities: one of the project's stated missions

  

authentication

Signing in, and proving who you are

  

breaking change

Changes behaviour or data in a way operators must act on

  

bug

Something does not do what it says it does

  

documentation

The wiki, the docs, or the words in the interface

  

enhancement

An improvement or a new capability

  

interface

The shape of the pages: what is where, and how it reads

  

internationalisation

Languages, regions, and the formats and symbols that go with them

  

observability

What an operator can see about a running instance: metrics, logs, health

  

performance

Uses fewer resources, or is more reliable, with no loss of features or accessibility

  

security

Keeping the code and the data secure: one of the project's stated missions

  

tier

1

Easy: small, self-contained, no prerequisites

  

tier

2

High-severity security: account access, personal data, the boundaries

  

tier

3

Foundations: prerequisite work that unblocks other issues

  

tier/4

Later: large features and long-horizon work

No labels

accessibility

authentication

breaking change

bug

documentation

enhancement

interface

internationalisation

observability

performance

security

tier

1

tier

2

tier

3

tier/4

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

Postulo/postulo!1074

No description provided.