CI: keep the ESCO classification between runs, and download it live on the weekly run #714

New issue

Closed

opened 2026-10-04 20:43:11 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-04 20:43:11 +00:00

Owner

Every unit leg runs manage.py fetch_esco --revision 1.2.1: three API pages of 10,000 concepts, 27–30 s, and a request to an outside service on every push. If the service is slow or rate-limits, the whole leg fails (fetch_esco.py:337). Only 22 tests read the result (test_esco.py, five in test_listings.py, five in test_suggestions.py), and they skip without it.

Proposal

actions/cache over src/postulo/jobs/data/esco-*, keyed on the revision alone: the loader refuses a directory holding two versions (esco.py:120-124), so a key that could restore an older one must not exist.
The download runs on a cache miss, and always on the weekly schedule, because the fetch validates what it writes and is the check that the API still answers in the shape the loader reads.

Done when

A push with the cache warm makes no request to the ESCO API, and the weekly run still does.

Measured from Forgejo's task API and job logs, runs 561–710 (24 Sep – 4 Oct 2026): wall time per push about 52–55 min, up from about 15; the browser suite grew from 149 to 645 tests in five days and runs serially (29 min); the runner takes **two jobs at a time, in the order the workflow file lists them**, so the browser job starts 19–22 min after the push; and Forgejo cancels the older push run when a newer push lands, which stopped 18 of 56 browser jobs before they gave a verdict. Every unit leg runs `manage.py fetch_esco --revision 1.2.1`: three API pages of 10,000 concepts, 27–30 s, and a request to an outside service on every push. If the service is slow or rate-limits, the whole leg fails (`fetch_esco.py:337`). Only 22 tests read the result (`test_esco.py`, five in `test_listings.py`, five in `test_suggestions.py`), and they skip without it. ## Proposal - `actions/cache` over `src/postulo/jobs/data/esco-*`, keyed on the revision alone: the loader refuses a directory holding two versions (`esco.py:120-124`), so a key that could restore an older one must not exist. - The download runs on a cache miss, and always on the weekly schedule, because the fetch validates what it writes and is the check that the API still answers in the shape the loader reads. ## Done when - A push with the cache warm makes no request to the ESCO API, and the weekly run still does.

tiagoagueda added the

enhancement

label

2026-10-04 20:43:11 +00:00

tiagoagueda referenced this issue from a commit

2026-10-04 21:23:08 +00:00

Keep the ESCO classification between runs, and download it live weekly

tiagoagueda closed this issue

2026-10-04 21:23:08 +00:00