Reading a page: “up to” is only looked for before the figure, and the whole-word guard matches inside Hindi and Bengali words #1106

New issue

Open

opened 2026-10-31 02:37:36 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-31 02:37:36 +00:00

Owner

Found while reviewing the Hindi, Urdu and Bengali translations of the builtin plugin's keyword lists (2026-10-30). The page reader in src/postulo/plugins/builtin/patterns.py (#267) cannot work for these languages as written, for two reasons. Translating the lists does not fix either.

1. "Up to" is only looked for before the figure

UP_TO_WORDS ("up to|maximum|max") is searched only in the text before the figure: before at line 578, filler at line 608. Languages with postpositions put the word after the number:

Hindi "40,000 तक", Urdu "40,000 تک", Bengali "40,000 পর্যন্ত"
Turkish "40.000'e kadar", Japanese "40万円まで", Korean "4만 원까지"

On such a page, "up to 40,000" is read as a minimum of 40,000, which is the opposite of what the advert says. The translators of hi, ur and bn marked these lines unsure for this reason. RANGE_WORDS has the same shape: "30,000 से 40,000 तक" works only because से happens to sit between the figures.

2. The whole-word guard fails inside Indic words

_word() (line 236) wraps each spelling in (?<![^\W\d_])…(?![^\W\d_]). Python's \w does not count combining vowel signs (matras: U+093E ा, U+0947 े, U+0948 ै, U+093F ि, U+0902 ं, U+094D ्…) as word characters. So a spelling that starts or ends next to a matra is matched in the middle of a word:

pat = re.compile(r'(?<![^\W\d_])से(?![^\W\d_])')
[m.start() for m in pat.finditer('कैसे')]       # [2]  — "how"
[m.start() for m in pat.finditer('पैसे भेजे')]  # [2]  — "money"

This affects every list read through _word(): closing words, salary words and periods, not only pay ranges. It also hits every script that uses combining marks: Devanagari, Bengali, Gurmukhi, Tamil, Thai, and Arabic with harakat.

What to do

Let a list say where its word goes, before or after the figure. For example, add a second string, "words that follow a pay figure to make it the most on offer", next to UP_TO_WORDS, and the same for the closing word of a range. That keeps the lists translatable without code per language.
Make the guard treat combining marks as part of a word: (?<![\w\p{M}]) with the regex module, or check unicodedata.category(ch)[0] in "LMN" around the match.
Tests: "40,000 तक" read as a maximum, "कैसे" not matched by से, and one Bengali and one Arabic case.

Not a 0.5.0 blocker: a page in these languages is still read in English as well, and nothing is invented. But hi, ur and bn are selected for 0.5, so their lists will be wrong until this lands.

Found while reviewing the Hindi, Urdu and Bengali translations of the builtin plugin's keyword lists (2026-10-30). The page reader in `src/postulo/plugins/builtin/patterns.py` (#267) cannot work for these languages as written, for two reasons. Translating the lists does not fix either. ## 1. "Up to" is only looked for before the figure `UP_TO_WORDS` ("up to|maximum|max") is searched only in the text *before* the figure: `before` at line 578, `filler` at line 608. Languages with postpositions put the word *after* the number: - Hindi "40,000 तक", Urdu "40,000 تک", Bengali "40,000 পর্যন্ত" - Turkish "40.000'e kadar", Japanese "40万円まで", Korean "4만 원까지" On such a page, "up to 40,000" is read as a minimum of 40,000, which is the opposite of what the advert says. The translators of hi, ur and bn marked these lines unsure for this reason. `RANGE_WORDS` has the same shape: "30,000 से 40,000 तक" works only because से happens to sit between the figures. ## 2. The whole-word guard fails inside Indic words `_word()` (line 236) wraps each spelling in `(?<![^\W\d_])…(?![^\W\d_])`. Python's `\w` does not count combining vowel signs (matras: U+093E ा, U+0947 े, U+0948 ै, U+093F ि, U+0902 ं, U+094D ्…) as word characters. So a spelling that starts or ends next to a matra is matched in the middle of a word: ```python pat = re.compile(r'(?<![^\W\d_])से(?![^\W\d_])') [m.start() for m in pat.finditer('कैसे')] # [2] — "how" [m.start() for m in pat.finditer('पैसे भेजे')] # [2] — "money" ``` This affects every list read through `_word()`: closing words, salary words and periods, not only pay ranges. It also hits every script that uses combining marks: Devanagari, Bengali, Gurmukhi, Tamil, Thai, and Arabic with harakat. ## What to do - Let a list say where its word goes, before or after the figure. For example, add a second string, "words that follow a pay figure to make it the most on offer", next to `UP_TO_WORDS`, and the same for the closing word of a range. That keeps the lists translatable without code per language. - Make the guard treat combining marks as part of a word: `(?<![\w\p{M}])` with the `regex` module, or check `unicodedata.category(ch)[0] in "LMN"` around the match. - Tests: "40,000 तक" read as a maximum, "कैसे" not matched by से, and one Bengali and one Arabic case. Not a 0.5.0 blocker: a page in these languages is still read in English as well, and nothing is invented. But hi, ur and bn are selected for 0.5, so their lists will be wrong until this lands.

tiagoagueda added the

bug

internationalisation

labels

2026-10-31 02:37:46 +00:00