CI: a unit worker killed for memory under -n 3 leaves the job hanging until the runner's deadline #1184

New issue

Closed

opened 2026-10-31 17:56:55 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-31 17:56:55 +00:00

Owner

Twice on 2026-10-31 the unit job died on the runner's deadline after every test it reached had passed.

Run 1778 (PR #1058, fe0580495): [gw2] node down: Not properly terminated at 12:04, then replacing crashed worker gw2. The progress reached [ 99%] at 12:05 and nothing followed until context deadline exceeded at 12:22.
Run 1793 (main push, 7332677b2): [gw1] node down: Not properly terminated at 63%, then replacing crashed worker gw1. The run reached 99% at 17:23 and hit the deadline at 17:39.

Cause:

Both times, a worker was killed outright in the coverage step (the 3.14 leg) under -n 3, inside the job container's 2 GB.
That is the failure #1085 measured three workers against (run 1658: "the kernel killed two of them at the same second"). Since then, tiers 3–4 have added tests and pages, and --cov adds to each worker.
Once a worker crashes, xdist under --dist loadgroup replaces it but never finishes. So the job does not fail; it hangs for 16 minutes and says nothing about which test was running.

Fix (release blocker, since the tag needs a green main):

Go back to -n 2 in ci.yml and every-python.yml, which was stable before #1085. This costs about 3.5 minutes per run.
Add --max-worker-restart=0, so a killed worker fails the job at once, with the test it was running in the report.
Re-measuring three workers belongs with #1033 (0.6.0).

Twice on 2026-10-31 the unit job died on the runner's deadline after every test it reached had passed. - **Run 1778** (PR #1058, `fe0580495`): `[gw2] node down: Not properly terminated` at 12:04, then `replacing crashed worker gw2`. The progress reached `[ 99%]` at 12:05 and nothing followed until `context deadline exceeded` at 12:22. - **Run 1793** (`main` push, `7332677b2`): `[gw1] node down: Not properly terminated` at 63%, then `replacing crashed worker gw1`. The run reached 99% at 17:23 and hit the deadline at 17:39. **Cause:** - Both times, a worker was killed outright in the coverage step (the 3.14 leg) under `-n 3`, inside the job container's 2 GB. - That is the failure #1085 measured three workers against (run 1658: "the kernel killed two of them at the same second"). Since then, tiers 3–4 have added tests and pages, and `--cov` adds to each worker. - Once a worker crashes, xdist under `--dist loadgroup` replaces it but never finishes. So the job does not fail; it hangs for 16 minutes and says nothing about which test was running. **Fix (release blocker, since the tag needs a green `main`):** - Go back to `-n 2` in `ci.yml` and `every-python.yml`, which was stable before #1085. This costs about 3.5 minutes per run. - Add `--max-worker-restart=0`, so a killed worker fails the job at once, with the test it was running in the report. - Re-measuring three workers belongs with #1033 (0.6.0).

tiagoagueda referenced this issue from a pull request that will close it,

2026-10-31 17:59:08 +00:00

Run the unit tests on two workers again, and fail at once when one is killed (#1184) #1185

tiagoagueda referenced this issue from a commit

2026-10-31 17:59:09 +00:00

Run the unit tests on two workers again, and fail at once when one is killed (#1184)

tiagoagueda added this to the 0.5.0 milestone

2026-10-31 17:59:09 +00:00

tiagoagueda added the

bug

label

2026-10-31 17:59:09 +00:00