CI: install and download steps have no time limit and curl does not retry #1138

New issue

Closed

opened 2026-10-31 03:04:17 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-31 03:04:17 +00:00

Owner

What exists today

.forgejo/workflows/ci.yml:22-25 records that forgejo-runner 6.4 ignores a job's timeout-minutes and honours a step's. Step limits exist only on the test steps (:163, :312, :319, :326, :568; every-python.yml:151, :158, :165). None of the network steps has one:

apt-get update / install at ci.yml:106-107, :232-233, :506-507, :516-522; every-python.yml:82-83; dependencies.yml:62-63.
The uv installer, curl -LsSf at ci.yml:112, :238, :351, :545, :666 and in the other two workflows; the script then downloads the uv binary itself.
uv sync (:131, :257, :365, :552), playwright install (:134), npm ci (:137, :422, :695), fetch_esco (:284), the PGDG key and lists, and actions/cache restores.

So a stalled mirror or a hung TLS connection runs until the runner's configured job timeout (forgejo-runner's default is 3 h), with one of the two slots taken and the next push queued behind it.

Partly mitigated already, which the candidate did not say: uv retries its own HTTP requests (3 by default, UV_HTTP_RETRIES) with a read timeout (UV_HTTP_TIMEOUT, 30 s); apt on bookworm (apt 2.6) retries a failed fetch 3 times by default (Acquire::Retries). The gaps are a connection that stalls without failing, curl (no --retry, no --max-time), npm ci and playwright install.

No measurement was possible locally; the runner's history of hung set-up steps was not read.

What would change

timeout-minutes: on every set-up step (5 for apt, uv install, uv sync, npm ci; 10 for playwright install and fetch_esco). Step level only, as the file's header already prescribes.
curl --retry 3 --retry-all-errors --connect-timeout 10 --max-time 120 on the installer download.
Optionally -o Acquire::http::Timeout=30 on apt-get, so a stalled mirror fails fast into apt's own retries.
tests/test_release_tools.py reads the workflows (the unit legs are held equal between ci.yml and every-python.yml) and must still pass. The same pattern in the plugin repos' workflows is a separate, org-wide finding.

What it saves or protects

small (reliability): a hung download ends in minutes with its step named, instead of holding one of the runner's two slots until the runner's own job limit

Accessibility

none

Decisions

Limits per step, or one shared value? Recommended: per step, written next to the step, since a variable cannot be used in timeout-minutes on every Forgejo version (keep it literal unless one run proves an expression works).

Found by the efficiency and reliability audit of 2026-10-30 (finding C11; index postulo/postulo#1108). Checked against main at daee30b15 (2026-10-30).

## What exists today `.forgejo/workflows/ci.yml:22-25` records that forgejo-runner 6.4 ignores a job's `timeout-minutes` and honours a step's. Step limits exist only on the test steps (`:163`, `:312`, `:319`, `:326`, `:568`; `every-python.yml:151`, `:158`, `:165`). None of the network steps has one: - `apt-get update` / `install` at `ci.yml:106-107`, `:232-233`, `:506-507`, `:516-522`; `every-python.yml:82-83`; `dependencies.yml:62-63`. - The uv installer, `curl -LsSf` at `ci.yml:112`, `:238`, `:351`, `:545`, `:666` and in the other two workflows; the script then downloads the uv binary itself. - `uv sync` (`:131`, `:257`, `:365`, `:552`), `playwright install` (`:134`), `npm ci` (`:137`, `:422`, `:695`), `fetch_esco` (`:284`), the PGDG key and lists, and `actions/cache` restores. So a stalled mirror or a hung TLS connection runs until the runner's configured job timeout (forgejo-runner's default is 3 h), with one of the two slots taken and the next push queued behind it. Partly mitigated already, which the candidate did not say: uv retries its own HTTP requests (3 by default, `UV_HTTP_RETRIES`) with a read timeout (`UV_HTTP_TIMEOUT`, 30 s); apt on bookworm (apt 2.6) retries a failed fetch 3 times by default (`Acquire::Retries`). The gaps are a connection that stalls without failing, `curl` (no `--retry`, no `--max-time`), `npm ci` and `playwright install`. No measurement was possible locally; the runner's history of hung set-up steps was not read. ## What would change - `timeout-minutes:` on every set-up step (5 for apt, uv install, `uv sync`, `npm ci`; 10 for `playwright install` and `fetch_esco`). Step level only, as the file's header already prescribes. - `curl --retry 3 --retry-all-errors --connect-timeout 10 --max-time 120` on the installer download. - Optionally `-o Acquire::http::Timeout=30` on `apt-get`, so a stalled mirror fails fast into apt's own retries. - `tests/test_release_tools.py` reads the workflows (the unit legs are held equal between `ci.yml` and `every-python.yml`) and must still pass. The same pattern in the plugin repos' workflows is a separate, org-wide finding. ## What it saves or protects small (reliability): a hung download ends in minutes with its step named, instead of holding one of the runner's two slots until the runner's own job limit ## Accessibility none ## Decisions - **Limits per step, or one shared value?** Recommended: per step, written next to the step, since a variable cannot be used in `timeout-minutes` on every Forgejo version (keep it literal unless one run proves an expression works). --- Found by the efficiency and reliability audit of 2026-10-30 (finding C11; index postulo/postulo#1108). Checked against `main` at `daee30b15` (2026-10-30).

tiagoagueda added this to the 0.6.0 milestone

2026-10-31 03:04:17 +00:00

tiagoagueda added the

performance

bug

labels

2026-10-31 03:04:17 +00:00

tiagoagueda referenced this issue

2026-10-31 03:04:41 +00:00

Efficiency and reliability audit of 2026-10-30: index #1108

tiagoagueda referenced this issue from a pull request that will close it,

2026-11-10 10:31:54 +00:00