Translations: the .po reader turns a literal backslash before n or t into a newline or tab, losing the entry #498

New issue

Closed

opened 2026-10-01 10:47:07 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-01 10:47:07 +00:00

Owner

Found on 2026-09-29 in a read-only review of postulo at 95348908d, by round-tripping strings through the snapshot's dump and parse.

Checked again on 2026-10-01 against main at 0a5cad587: still present.

_unquote decodes escapes with chained str.replace calls: \n first and \\ last. A string that _quote wrote with an escaped backslash followed by n or t is therefore read back wrong. Separately, _quote does not escape a carriage return, so it is written raw, and parse splits it as a line ending. Either way the entry comes back under a different msgid than the one written, and its translation no longer matches the string in the source.

Where: core/messages_tool.py:524-528 (_unquote: .replace("\\n", "\n").replace("\\t", "\t").replace('\\"', '"').replace("\\\\", "\\")); core/messages_tool.py:391-395 (_quote, which escapes \\, ", \n and \t only); core/messages_tool.py:491 (for raw in text.splitlines() in parse).
What happens: with the snapshot's functions:
- _unquote(_quote("C:\\new folder")) returns C:\ followed by a newline and ew folder;
- _unquote(_quote("tab\\t")) returns tab\ followed by a tab;
- a catalogue holding Type C:\new that goes through dump and then parse comes back with a msgid of backslash, newline, ew, so the original entry is lost;
- one holding a carriage return comes back cut at the carriage return.
On the next extract, the existing entry no longer matches the extracted string, so it is dropped with its translation. extract --check then reports the catalogue as stale, and extracting again drops the translation again. The .mo compiled from the parsed catalogue carries the corrupted key, so the runtime does not find the translation either.
Why it matters: no message in today's catalogues contains a backslash, so nothing is lost yet. The first help text that mentions a Windows path, a regular expression or an escape sequence would lose its translation in every language on every run. The tool is also published as the postulo-messages command (pyproject.toml:86) for plugin repositories, whose strings Postulo does not control.
Fix: decode in one left-to-right pass, re.sub(r'\\(.)', lambda m: {"n": "\n", "t": "\t", "r": "\r", '"': '"', "\\": "\\"}.get(m.group(1), m.group(0)), chunk), and have _quote escape \r as \\r.

Done when

tests/test_messages_tool.py round-trips, through dump and parse, a catalogue whose msgid and msgstr contain a literal backslash before n, t and ", and a carriage return. It asserts every entry comes back unchanged. Today the backslash and carriage-return cases fail.

Found on 2026-09-29 in a read-only review of postulo at 95348908d, by round-tripping strings through the snapshot's `dump` and `parse`. Checked again on 2026-10-01 against `main` at `0a5cad587`: still present. `_unquote` decodes escapes with chained `str.replace` calls: `\n` first and `\\` last. A string that `_quote` wrote with an escaped backslash followed by `n` or `t` is therefore read back wrong. Separately, `_quote` does not escape a carriage return, so it is written raw, and `parse` splits it as a line ending. Either way the entry comes back under a different msgid than the one written, and its translation no longer matches the string in the source. - **Where:** `core/messages_tool.py:524-528` (`_unquote`: `.replace("\\n", "\n").replace("\\t", "\t").replace('\\"', '"').replace("\\\\", "\\")`); `core/messages_tool.py:391-395` (`_quote`, which escapes `\\`, `"`, `\n` and `\t` only); `core/messages_tool.py:491` (`for raw in text.splitlines()` in `parse`). - **What happens:** with the snapshot's functions: - `_unquote(_quote("C:\\new folder"))` returns `C:\` followed by a newline and `ew folder`; - `_unquote(_quote("tab\\t"))` returns `tab\` followed by a tab; - a catalogue holding `Type C:\new` that goes through `dump` and then `parse` comes back with a msgid of backslash, newline, `ew`, so the original entry is lost; - one holding a carriage return comes back cut at the carriage return. On the next `extract`, the existing entry no longer matches the extracted string, so it is dropped with its translation. `extract --check` then reports the catalogue as stale, and extracting again drops the translation again. The `.mo` compiled from the parsed catalogue carries the corrupted key, so the runtime does not find the translation either. - **Why it matters:** no message in today's catalogues contains a backslash, so nothing is lost yet. The first help text that mentions a Windows path, a regular expression or an escape sequence would lose its translation in every language on every run. The tool is also published as the `postulo-messages` command (`pyproject.toml:86`) for plugin repositories, whose strings Postulo does not control. - **Fix:** decode in one left-to-right pass, `re.sub(r'\\(.)', lambda m: {"n": "\n", "t": "\t", "r": "\r", '"': '"', "\\": "\\"}.get(m.group(1), m.group(0)), chunk)`, and have `_quote` escape `\r` as `\\r`. ## Done when - `tests/test_messages_tool.py` round-trips, through `dump` and `parse`, a catalogue whose msgid and msgstr contain a literal backslash before `n`, `t` and `"`, and a carriage return. It asserts every entry comes back unchanged. Today the backslash and carriage-return cases fail.

tiagoagueda added this to the 0.6.0 milestone

2026-10-01 10:47:07 +00:00

tiagoagueda added the

internationalisation

tier

bug

labels

2026-10-01 10:47:07 +00:00

tiagoagueda referenced this issue from a commit

2026-10-04 04:15:10 +00:00

Read .po escapes in one pass so backslashes and carriage returns round-trip

tiagoagueda closed this issue