Search: highlighting and excerpts land on the wrong letters after a Turkish İ #377

New issue

Closed

opened 2026-10-01 10:46:15 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-01 10:46:15 +00:00

Owner

Found on 2026-09-29 in a read-only review of postulo at 95348908d, reading the search results' highlighting and checking it with a Turkish place name.

Checked again on 2026-10-01 against main at 0a5cad587: still present. The code is now at core/templatetags/postulo.py:468-490 for highlight (#248 moved the flag lookup out of that file); excerpt is unchanged at core/search.py:84-100.

highlight and excerpt find the term in text.lower() and then slice the original text with the positions found there. That assumes lower-casing keeps a string's length. It does not for İ (U+0130, the capital dotted I), which Python lowers to two code points, so every position after one is shifted. Turkish is a shipped language, and İstanbul and İzmir are common in job locations.

Where: core/templatetags/postulo.py:486-508 (highlight, used at templates/core/search.html:49-54 on every hit's title, subtitle and excerpt); core/search.py:84-100 (excerpt).
What happens: checked against the snapshot:
- highlight("İstanbul office", "stan") returns İstanbul office: the mark covers letters that are not the term.
- highlight("Office in İzmir and İstanbul, stand-up daily", "stand") marks and-u; the shift grows with each İ before the match.
- excerpt("İ" * 100 + " needle here", "needle", radius=5) returns "…": the window is computed past the end of the text, and the passage that matched is not shown at all.
 The output stays escaped, so nothing unsafe is produced; the result is simply wrong. U+0130 is the only code point whose str.lower() changes length.
Why it matters: a person reading Turkish (or any text naming a Turkish place) sees the wrong words marked in search results, and a long Turkish description can come back with an empty excerpt.
Fix: find matches on the original text without changing its length, for example re.finditer(re.escape(query), text, re.IGNORECASE) in highlight, and re.search in excerpt for the position. The query itself can contain İ, so use the match's own span rather than len(query) for the end.

Done when

A test in tests/test_search.py (or the template tag's tests) checks that highlight("İstanbul office", "stan") returns İstanbul office, and that an excerpt of a text with several İ before the match contains the match.

Found on 2026-09-29 in a read-only review of postulo at 95348908d, reading the search results' highlighting and checking it with a Turkish place name. Checked again on 2026-10-01 against `main` at `0a5cad587`: still present. The code is now at `core/templatetags/postulo.py:468-490` for `highlight` (#248 moved the flag lookup out of that file); `excerpt` is unchanged at `core/search.py:84-100`. `highlight` and `excerpt` find the term in `text.lower()` and then slice the original `text` with the positions found there. That assumes lower-casing keeps a string's length. It does not for `İ` (U+0130, the capital dotted I), which Python lowers to two code points, so every position after one is shifted. Turkish is a shipped language, and *İstanbul* and *İzmir* are common in job locations. - **Where:** `core/templatetags/postulo.py:486-508` (`highlight`, used at `templates/core/search.html:49-54` on every hit's title, subtitle and excerpt); `core/search.py:84-100` (`excerpt`). - **What happens:** checked against the snapshot: - `highlight("İstanbul office", "stan")` returns `İstanbul office`: the mark covers letters that are not the term. - `highlight("Office in İzmir and İstanbul, stand-up daily", "stand")` marks `and-u`; the shift grows with each `İ` before the match. - `excerpt("İ" * 100 + " needle here", "needle", radius=5)` returns `"…"`: the window is computed past the end of the text, and the passage that matched is not shown at all. The output stays escaped, so nothing unsafe is produced; the result is simply wrong. U+0130 is the only code point whose `str.lower()` changes length. - **Why it matters:** a person reading Turkish (or any text naming a Turkish place) sees the wrong words marked in search results, and a long Turkish description can come back with an empty excerpt. - **Fix:** find matches on the original text without changing its length, for example `re.finditer(re.escape(query), text, re.IGNORECASE)` in `highlight`, and `re.search` in `excerpt` for the position. The query itself can contain `İ`, so use the match's own span rather than `len(query)` for the end. ## Done when - A test in `tests/test_search.py` (or the template tag's tests) checks that `highlight("İstanbul office", "stan")` returns `İstanbul office`, and that an excerpt of a text with several `İ` before the match contains the match.

tiagoagueda added this to the 0.6.0 milestone

2026-10-01 10:46:15 +00:00

tiagoagueda added the

internationalisation

tier

bug

labels

2026-10-01 10:46:15 +00:00

tiagoagueda referenced this issue from a commit

2026-10-04 09:26:54 +00:00

Match search terms on the original text, not on its lower-cased copy

tiagoagueda closed this issue