Reading a listing: no tool returns the job advert, and the fence tests use a response shape the API never sends #10

New issue

Closed

opened 2026-10-01 10:48:03 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-01 10:48:03 +00:00

Owner

Found on 2026-09-29 in a read-only review of postulo-mcp at 0b29e70b9, comparing every tool's address with the API description in tests/openapi.json and with the test fixtures.

Related to #3.

The job advert is the text the README and the server's docstring present as the main thing an agent reads and the main thing fenced, but no tool or resource can fetch it. In the API, description exists only on GET /listings/{pk} (ListingDetailOut), and nothing in server.py calls that address. The tests hide the gap because their fixtures put a description where the API has none.

Where: src/postulo_mcp/server.py:164-166 (get_application), src/postulo_mcp/server.py:172-181 (search_postings reads /listings, which answers ListingOut without description), src/postulo_mcp/server.py:346-355 (prepare_for_interview), README.md:86-106; tests/test_server.py:26-41 (APPLICATION with a posting key and posting.description), tests/test_server.py:100-101 (/listings answering a description), tests/test_server.py:250-260 and :372-379; tests/fake_api.py:104-109 (Operation.answer).
What happens:
1. get_application returns listing: {id, title, company} (ListingRef); search_postings returns ListingOut. Neither carries the advert, and there is no get_listing tool or postulo://listings/{id} resource. A person who runs prepare_for_interview, which asks the agent to say "what they asked for", gets an agent that has no way to read the advert, and the README's description of that prompt ("gathers the posting") is not true.
2. The application fixture has a posting object; ApplicationDetailOut has listing and no posting. The /listings fixture adds description to ListingOut, which has none. test_somebody_elses_words_reach_the_model_inside_a_fence and test_resources_return_readable_text_not_json therefore check the fence on a field in a place the API never puts it.
3. Operation.answer wraps whatever the test wrote in {items, count} without checking it against the response schema, and nothing checks request bodies either. Operation.check covers query parameters only, so a misspelt write field (application_id on POST /reminders) or a response field the API dropped would still pass.
4. The snapshot itself is stale: it was taken at core 5ee31c66a (tests/fake_api.py:25), and the core's ApplicationDetailOut has since gained offers and ListingDetailOut gained events (core api/schemas.py:200-204 and :483-488).
Why it matters: the interview prompt, one of the two the server ships, cannot do what it says, and an agent without the advert is pushed to guess, which the server's own instructions forbid. The fence tests, the fake's promise in README.md:159-162 and the stale snapshot together mean a change in the API's shape reaches users before it reaches CI.
Fix: add a get_listing(listing_id) read tool and a postulo://listings/{listing_id} resource over GET /listings/{pk}, fenced as usual (description is already in UNTRUSTED_FIELDS), and name it in prepare_for_interview and the README. Rewrite the fixtures in the real shapes (listing, not posting; no description on ListingOut) and move the advert fence test onto get_listing. Make Operation.answer validate the fixture against the operation's success schema and the client's JSON body against its request schema (a small recursive check over properties, required and $ref, or jsonschema), so a fixture or payload in the wrong shape fails. Refresh tests/openapi.json from the current core.

Done when

A test in tests/test_server.py calls get_listing and finds the advert inside a fence.
A test in tests/test_server.py shows that a fixture carrying a field the response schema does not have, and a write body carrying a field the request schema does not have, both fail in tests/fake_api.py.

Found on 2026-09-29 in a read-only review of postulo-mcp at 0b29e70b9, comparing every tool's address with the API description in `tests/openapi.json` and with the test fixtures. Related to #3. The job advert is the text the README and the server's docstring present as the main thing an agent reads and the main thing fenced, but no tool or resource can fetch it. In the API, `description` exists only on `GET /listings/{pk}` (`ListingDetailOut`), and nothing in `server.py` calls that address. The tests hide the gap because their fixtures put a `description` where the API has none. - **Where:** `src/postulo_mcp/server.py:164-166` (`get_application`), `src/postulo_mcp/server.py:172-181` (`search_postings` reads `/listings`, which answers `ListingOut` without `description`), `src/postulo_mcp/server.py:346-355` (`prepare_for_interview`), `README.md:86-106`; `tests/test_server.py:26-41` (`APPLICATION` with a `posting` key and `posting.description`), `tests/test_server.py:100-101` (`/listings` answering a `description`), `tests/test_server.py:250-260` and `:372-379`; `tests/fake_api.py:104-109` (`Operation.answer`). - **What happens:** 1. `get_application` returns `listing: {id, title, company}` (`ListingRef`); `search_postings` returns `ListingOut`. Neither carries the advert, and there is no `get_listing` tool or `postulo://listings/{id}` resource. A person who runs `prepare_for_interview`, which asks the agent to say "what they asked for", gets an agent that has no way to read the advert, and the README's description of that prompt ("gathers the posting") is not true. 2. The application fixture has a `posting` object; `ApplicationDetailOut` has `listing` and no `posting`. The `/listings` fixture adds `description` to `ListingOut`, which has none. `test_somebody_elses_words_reach_the_model_inside_a_fence` and `test_resources_return_readable_text_not_json` therefore check the fence on a field in a place the API never puts it. 3. `Operation.answer` wraps whatever the test wrote in `{items, count}` without checking it against the response schema, and nothing checks request bodies either. `Operation.check` covers query parameters only, so a misspelt write field (`application_id` on `POST /reminders`) or a response field the API dropped would still pass. 4. The snapshot itself is stale: it was taken at core 5ee31c66a (`tests/fake_api.py:25`), and the core's `ApplicationDetailOut` has since gained `offers` and `ListingDetailOut` gained `events` (core `api/schemas.py:200-204` and `:483-488`). - **Why it matters:** the interview prompt, one of the two the server ships, cannot do what it says, and an agent without the advert is pushed to guess, which the server's own instructions forbid. The fence tests, the fake's promise in `README.md:159-162` and the stale snapshot together mean a change in the API's shape reaches users before it reaches CI. - **Fix:** add a `get_listing(listing_id)` read tool and a `postulo://listings/{listing_id}` resource over `GET /listings/{pk}`, fenced as usual (`description` is already in `UNTRUSTED_FIELDS`), and name it in `prepare_for_interview` and the README. Rewrite the fixtures in the real shapes (`listing`, not `posting`; no `description` on `ListingOut`) and move the advert fence test onto `get_listing`. Make `Operation.answer` validate the fixture against the operation's success schema and the client's JSON body against its request schema (a small recursive check over `properties`, `required` and `$ref`, or `jsonschema`), so a fixture or payload in the wrong shape fails. Refresh `tests/openapi.json` from the current core. ## Done when - A test in `tests/test_server.py` calls `get_listing` and finds the advert inside a fence. - A test in `tests/test_server.py` shows that a fixture carrying a field the response schema does not have, and a write body carrying a field the request schema does not have, both fail in `tests/fake_api.py`.

tiagoagueda added this to the 0.5.0 milestone

2026-10-01 10:48:03 +00:00

tiagoagueda added the

bug

label

2026-10-01 10:48:03 +00:00

tiagoagueda referenced this issue from a commit

2026-10-19 06:50:34 +00:00

Security review fixes: one fence for answers and errors, honest API calls, a bounded and throttled port

tiagoagueda referenced this issue

2026-10-19 06:50:42 +00:00

Security review fixes: one fence, honest API calls, a bounded and throttled port, CI that runs #22