Browser tests: check each page once with every instrument, instead of walking the site ten times

tiagoagueda commented

2026-10-04 20:43:13 +00:00

Owner

Measured from Forgejo's task API and job logs, runs 561–710 (24 Sep – 4 Oct 2026): wall time per push about 52–55 min, up from about 15; the browser suite grew from 149 to 645 tests in five days and runs serially (29 min); the runner takes two jobs at a time, in the order the workflow file lists them, so the browser job starts 19–22 min after the push; and Forgejo cancels the older push run when a newer push lands, which stopped 18 of 56 browser jobs before they gave a verdict.

Ten tests walk every page in signed_in_paths() (150 paths, test_accessibility.py:407-640): axe in light and dark, reflow at 320 in three languages, text spacing twice, target size twice, and the aria-describedby check. That is about 1,500 page loads, each with its own instrument injected (axe alone is 580 KB per call), and each new page the coverage check requires (tests/test_page_coverage.py) lengthens all ten.

Proposal

Five tests, each loading every page once and running several instruments on it, reporting each instrument's findings separately:

A (English, 1280×900, reduced motion), in this order on each page: aria-describedby; axe in light; dark scheme, wait until no animation is running, axe in dark, back to light; target size with its menus; resize to 640×450 for the 200% zoom check and back; text spacing last, because its stylesheet stays on the page. Two animation frames after each resize, because the page answers a resize asynchronously.
B: target size with the profile set to compact.
C, D, E: reflow at 320 pixels in English, Greek and German.

About 750 loads instead of 1,500, in five tests that spread across workers. Reduced motion matters here: the stylesheet fades colours (transition-colors), and axe must not measure halfway through a fade. VISITED_URL_NAMES stays importable for the coverage check.

Done when

Each page is loaded once per pass, every finding the ten walks reported is still reported, and a deliberately broken page still fails the right instrument.

Measured from Forgejo's task API and job logs, runs 561–710 (24 Sep – 4 Oct 2026): wall time per push about 52–55 min, up from about 15; the browser suite grew from 149 to 645 tests in five days and runs serially (29 min); the runner takes **two jobs at a time, in the order the workflow file lists them**, so the browser job starts 19–22 min after the push; and Forgejo cancels the older push run when a newer push lands, which stopped 18 of 56 browser jobs before they gave a verdict. Ten tests walk every page in `signed_in_paths()` (150 paths, `test_accessibility.py:407-640`): axe in light and dark, reflow at 320 in three languages, text spacing twice, target size twice, and the `aria-describedby` check. That is about 1,500 page loads, each with its own instrument injected (axe alone is 580 KB per call), and each new page the coverage check requires (`tests/test_page_coverage.py`) lengthens all ten. ## Proposal Five tests, each loading every page once and running several instruments on it, reporting each instrument's findings separately: - **A** (English, 1280×900, reduced motion), in this order on each page: `aria-describedby`; axe in light; dark scheme, wait until no animation is running, axe in dark, back to light; target size with its menus; resize to 640×450 for the 200% zoom check and back; text spacing last, because its stylesheet stays on the page. Two animation frames after each resize, because the page answers a resize asynchronously. - **B**: target size with the profile set to compact. - **C, D, E**: reflow at 320 pixels in English, Greek and German. About 750 loads instead of 1,500, in five tests that spread across workers. Reduced motion matters here: the stylesheet fades colours (`transition-colors`), and axe must not measure halfway through a fade. `VISITED_URL_NAMES` stays importable for the coverage check. ## Done when - Each page is loaded once per pass, every finding the ten walks reported is still reported, and a deliberately broken page still fails the right instrument.

tiagoagueda added the

enhancement

label

2026-10-04 20:43:13 +00:00

tiagoagueda referenced this issue from a commit

2026-10-05 01:16:48 +00:00

Walk every page three times instead of six, with every instrument on a load

tiagoagueda closed this issue

2026-10-05 01:16:48 +00:00

tiagoagueda referenced this issue from a commit

2026-10-05 01:52:24 +00:00

Ask once more for a page a walk stalls on, and list the stalls