Fix the open bug and security issues (#8-#22, #24) #25

Merged

tiagoagueda merged 3 commits from sec2-fixes into main

2026-10-19 09:55:26 +00:00

tiagoagueda commented

2026-10-19 07:00:17 +00:00

Owner

Fixes every open bug and security issue of this repository in three commits: the dev lock and CI audit (#24), the code and its tests (#8-#22), and the catalogue extraction.

What and why

#24 (SEC2-049): core pin moved to 42be52048; uv lock --upgrade (pyjwt 2.15.1, urllib3 2.8.0, oauthlib 4.0.0, django 6.1.2, allauth 65.20.0, imapclient 4.1.0). pip-audit 2.10.1 --strict over the exported lock reports no known vulnerabilities. CI and the release gate now run that audit (Postulo itself is --no-emit-packaged: git dependency, not on PyPI) and install uv from a pinned version through the core's checksum-verified installer pattern. No action is SHA-pinned.
#8: one message that cannot be parsed, classified, matched or suggested no longer fails the pass: each is isolated (own try, own savepoint), counted, noted and flagged with the rest. Bad Date bytes, charset=idna/undefined and NUL characters are handled in parse.
#9: the "Matched by" / "Recognised" values are translatable labels (HOW_LABELS, KIND_LABELS); Match.how and Verdict.kind stay identifiers. The language the scheduler runs in is the core's to set (see below).
#10: ngettext for both count sentences (and the new "could not be read" one).
#11: no Message-ID -> sha256-<hash of the message>, not the UID.
#12: INBOX refused (form, Test and the pass); move_to == folder refused; help text, validation text and README say every message read is flagged or moved, recognised or not.
#13: empty port follows the security (143 for STARTTLS); port 1-65535 and look-back 1-3650 validated; saved out-of-range values fall back.
#15: HTML read by a linear HTMLParser over at most 200 KB; headers cut to 4 KB; bounded identifier regex; threading keeps the In-Reply-To parent and the last 8 References, so the thread query binds at most nine values.
#16: offer patterns name the post instead of matching a vacancy word or any "pleased to offer"; the matching interview wordings were added.
#17: one normalised spelling for rules and company/role search (entities, NFKC, apostrophes, whitespace), <head>/<title> dropped, inline vs block tags handled; the stored excerpt keeps its line breaks.
#18: kinds tried in MOST_CONSEQUENTIAL_FIRST order across all languages.
#19: public-suffix-aware registrable() from a short built-in table (no dependency, no download), bare suffixes reduce to nothing, IDNA-normalised hosts, public providers checked in full and reduced.
#20: batches of 25, sizes first, messages above 1 MB fetched as headers + first 64 KiB of text, and in untouched/move mode a message already suggested is recognised from its Message-ID and not downloaded again.
#21: headers read as text from the raw string (UTF-8, else Latin-1).
#22: MailboxError becomes SyncReport(error=...) with the plain sentence; a refused flag/move keeps the report and adds a note (server words only in the log); MOVE fallback (copy, \Deleted, UID EXPUNGE) or a clear refusal; PERMANENTFLAGS read; folder_exists wrapped in Test.

Tests

uv run pytest: 280 passed (23 release-gate tests deselected as usual); ruff check, ruff format --check, postulo-messages extract --check / check / guard clean. New cases are in tests/test_review_fixes.py and use the sentences and inputs from the issues; tests/test_imap.py got a fake mailbox that records fetches, answers sizes/partial fetches and can refuse a call. Without the source changes the suite does not even import (it needs MOST_CONSEQUENTIAL_FIRST, HOW_LABELS).

Left open / notes

Nothing left open among the listed issues. Two caveats:
- #9: the stored labels are in the language active when the pass runs; for a scheduled pass that is LANGUAGE_CODE until the core runs each sync in the owner's language (the half the issue assigns to the core). The test therefore asserts the labels are the lazy translated ones rather than a Portuguese string: no translation is written here (Weblate only), so the new msgids have empty slots until Weblate drafts them.
- #19 uses a built-in table, not the full Public Suffix List; an unlisted multi-label suffix still reduces to its last two labels.
Catalogues: the extraction also brought in the seven languages the core added (az, be, cnr, fo, kk, nn, rm). All new strings are empty slots for Weblate.

Closes #8, #9, #10, #11, #12, #13, #15, #16, #17, #18, #19, #20, #21, #22, #24

🤖 Generated with Claude Code

Fixes every open `bug` and `security` issue of this repository in three commits: the dev lock and CI audit (#24), the code and its tests (#8-#22), and the catalogue extraction. ## What and why - **#24 (SEC2-049)**: core pin moved to `42be52048`; `uv lock --upgrade` (pyjwt 2.15.1, urllib3 2.8.0, oauthlib 4.0.0, django 6.1.2, allauth 65.20.0, imapclient 4.1.0). `pip-audit 2.10.1 --strict` over the exported lock reports no known vulnerabilities. CI and the release gate now run that audit (Postulo itself is `--no-emit-package`d: git dependency, not on PyPI) and install uv from a pinned version through the core's checksum-verified installer pattern. No action is SHA-pinned. - **#8**: one message that cannot be parsed, classified, matched or suggested no longer fails the pass: each is isolated (own `try`, own savepoint), counted, noted and flagged with the rest. Bad Date bytes, `charset=idna`/`undefined` and NUL characters are handled in `parse`. - **#9**: the "Matched by" / "Recognised" values are translatable labels (`HOW_LABELS`, `KIND_LABELS`); `Match.how` and `Verdict.kind` stay identifiers. The language the scheduler runs in is the core's to set (see below). - **#10**: `ngettext` for both count sentences (and the new "could not be read" one). - **#11**: no `Message-ID` -> `sha256-<hash of the message>`, not the UID. - **#12**: `INBOX` refused (form, Test and the pass); `move_to == folder` refused; help text, validation text and README say every message read is flagged or moved, recognised or not. - **#13**: empty port follows the security (143 for STARTTLS); port 1-65535 and look-back 1-3650 validated; saved out-of-range values fall back. - **#15**: HTML read by a linear `HTMLParser` over at most 200 KB; headers cut to 4 KB; bounded identifier regex; threading keeps the In-Reply-To parent and the last 8 References, so the thread query binds at most nine values. - **#16**: offer patterns name the post instead of matching a vacancy word or any "pleased to offer"; the matching interview wordings were added. - **#17**: one normalised spelling for rules and company/role search (entities, NFKC, apostrophes, whitespace), `<head>`/`<title>` dropped, inline vs block tags handled; the stored excerpt keeps its line breaks. - **#18**: kinds tried in `MOST_CONSEQUENTIAL_FIRST` order across all languages. - **#19**: public-suffix-aware `registrable()` from a short built-in table (no dependency, no download), bare suffixes reduce to nothing, IDNA-normalised hosts, public providers checked in full and reduced. - **#20**: batches of 25, sizes first, messages above 1 MB fetched as headers + first 64 KiB of text, and in untouched/move mode a message already suggested is recognised from its `Message-ID` and not downloaded again. - **#21**: headers read as text from the raw string (UTF-8, else Latin-1). - **#22**: `MailboxError` becomes `SyncReport(error=...)` with the plain sentence; a refused flag/move keeps the report and adds a note (server words only in the log); MOVE fallback (copy, `\Deleted`, UID EXPUNGE) or a clear refusal; `PERMANENTFLAGS` read; `folder_exists` wrapped in Test. ## Tests `uv run pytest`: 280 passed (23 release-gate tests deselected as usual); `ruff check`, `ruff format --check`, `postulo-messages extract --check / check / guard` clean. New cases are in `tests/test_review_fixes.py` and use the sentences and inputs from the issues; `tests/test_imap.py` got a fake mailbox that records fetches, answers sizes/partial fetches and can refuse a call. Without the source changes the suite does not even import (it needs `MOST_CONSEQUENTIAL_FIRST`, `HOW_LABELS`). ## Left open / notes - Nothing left open among the listed issues. Two caveats: - #9: the stored labels are in the language active when the pass runs; for a scheduled pass that is `LANGUAGE_CODE` until the core runs each sync in the owner's language (the half the issue assigns to the core). The test therefore asserts the labels are the lazy translated ones rather than a Portuguese string: no translation is written here (Weblate only), so the new msgids have empty slots until Weblate drafts them. - #19 uses a built-in table, not the full Public Suffix List; an unlisted multi-label suffix still reduces to its last two labels. - Catalogues: the extraction also brought in the seven languages the core added (az, be, cnr, fo, kk, nn, rm). All new strings are empty slots for Weblate. Closes #8, #9, #10, #11, #12, #13, #15, #16, #17, #18, #19, #20, #21, #22, #24 🤖 Generated with [Claude Code](https://claude.com/claude-code)

tiagoagueda added 3 commits

2026-10-19 07:00:17 +00:00

Repin the core, upgrade the lock, audit it in CI ffc15f7aa9

The dev lock had never been re-resolved: pyjwt 2.13.0, urllib3 2.7.0, oauthlib 3.3.1,
django 6.1.1 and django-allauth 65.19.2 carried published advisories. The core pin moves
to 42be52048 and `uv lock --upgrade` takes everything to the current release (pyjwt
2.15.1, urllib3 2.8.0, oauthlib 4.0.0, django 6.1.2, allauth 65.20.0, imapclient 4.1.0).

CI and the release gate now install uv from a pinned version whose installer is checked
against its sha256 (the core's UV_VERSION / UV_INSTALLER_SHA256 pattern), and CI and the
release gate run pip-audit 2.10.1 --strict over the exported lock, so a new advisory fails
the build. Postulo itself is left out of the export: it is a git dependency, not on PyPI.

Closes #24

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Fix the review findings: reading, classifying, matching, the form and what a refusal does 3769c94140

Reading a message (mailbox.py, normalising.py)
- #8: a message that cannot be read or filed is that message's problem. Each message is
  parsed, classified, matched and suggested under its own try and savepoint; it is counted,
  noted ("N messages could not be read and were skipped") and flagged with the rest, so one
  bad message no longer fails every pass until it leaves the look-back. Dates, charsets that
  are not charsets (idna, undefined) and NUL characters no longer escape parse.
- #11: a message with no Message-ID is keyed on a hash of what it says, not on its UID.
- #15: the HTML stripper is an HTMLParser subclass over at most 200 KB; headers are cut to
  4 KB, the identifier regex is bounded, and threading keeps the In-Reply-To parent and the
  last 8 of References, so the thread query binds at most nine values.
- #17: the rules and the company/role search read one spelling (entities decoded, NFKC,
  typographic apostrophes plain, every run of whitespace one space), <head> and <title> are
  dropped, inline tags insert nothing and block tags a line break. The stored excerpt keeps
  its line breaks.
- #20: a pass asks for 25 messages at a time, sizes first; a message above 1 MB is fetched
  as its headers plus the first 64 KiB of its text; in untouched and move mode a message
  that already has a suggestion is recognised from its Message-ID and not downloaded again.
- #21: every header is read as text from the raw string (UTF-8, else Latin-1), so a raw
  UTF-8 From or Subject keeps the sender and the subject.

Classifying and matching
- #16: the offer lists no longer match the everyday words for a vacancy ("oferta de empleo",
  "job offer", "propozycja współpracy") nor "pleased to offer" anything: they name the post.
  The interview lists gained the "pleased to offer you an interview" wordings.
- #18: kinds are tried in MOST_CONSEQUENTIAL_FIRST order across every language, so a
  rejection is never read as an earlier-sorting language's interview.
- #19: a domain is reduced with a table of shared hosting suffixes and the second-level
  labels country codes register under (co.uk, com.br, com.pl ...); a bare suffix reduces to
  nothing; hosts are compared in their IDNA form; the sender is checked against the public
  providers in full and reduced.

The form, the labels and the refusals (sync.py)
- #9: "Matched by" and "Recognised" store translated labels (HOW_LABELS, KIND_LABELS);
  Match.how and Verdict.kind stay identifiers.
- #10: the two sentences that count messages use ngettext.
- #12: INBOX is refused by the form, Test and the pass (it may have been saved earlier);
  move_to equal to the folder is refused; the help text and README say that every message
  read is flagged or moved, recognised or not.
- #13: an empty port follows the security (143 for STARTTLS); port 1-65535 and look-back
  1-3650 are validated and a saved out-of-range value falls back instead of raising.
- #22: sync returns the plain sentence for a MailboxError; a refused flag or move keeps the
  report and adds a note that the next pass will read the messages again, with the server's
  words in the log only; a server without MOVE copies, marks deleted and UID-expunges those
  messages, or says it cannot; PERMANENTFLAGS is read and a folder that keeps no keywords
  is read as untouched with a note; Test survives a refused folder check.

Closes #8, #9, #10, #11, #12, #13, #15, #16, #17, #18, #19, #20, #21, #22

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Extract the new and changed strings into the catalogues

CI / test (pull_request) Successful in 2m30s

Details

91bc315e4a

`postulo-messages extract` leaves an empty slot for each new string in every catalogue,
including the seven languages the core added since the last extraction (az, be, cnr, fo, kk,
nn, rm), and the plural forms of the three sentences that count messages. Nothing is
translated here: Weblate drafts the new strings and sends them back as a pull request.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

tiagoagueda merged commit f08acc7cd8 into main

2026-10-19 09:55:26 +00:00

tiagoagueda referenced this pull request from a commit

2026-10-19 09:55:27 +00:00

Merge pull request 'Fix the open bug and security issues (#8-#22, #24)' (#25) from sec2-fixes into main

tiagoagueda referenced this pull request

2026-10-19 09:55:29 +00:00

Suggestions: “Matched by” and “Recognised” carry English literals, and the labels are in the scheduler's language #9

tiagoagueda referenced this pull request

2026-10-19 09:55:32 +00:00

Test and sync: the two sentences that count messages have no plural forms and read “1 messages” #10

tiagoagueda referenced this pull request