Company map: the country hint reads GeoNames' capital and postal-code columns as country names #534

New issue

Closed

opened 2026-10-01 10:47:27 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-01 10:47:27 +00:00

Owner

Found on 2026-09-29 in a read-only review of postulo at 95348908d, reading jobs/places.py against the real countryInfo.txt and resolving sample locations with it.

Related to #108.

Checked again on 2026-10-01 against main at 0a5cad587: still present.

_countries() builds the table of country names that narrows a city match, but it reads the wrong columns. In GeoNames' countryInfo.txt column 4 is the country's English name, column 5 is its capital and column 13 is the postal-code format; there is no column of names by language. So every capital city becomes a "country", and no country name other than the English one is ever a hint. The tests pass because their fixture invents the layout the code expects.

Where: jobs/places.py:111-136 (_countries: candidates = [columns[4], columns[5]], then columns[13].split(";") parsed as Language:Name; the header line, which starts with #ISO, is not skipped); jobs/places.py:169-175 (resolve: a hint whose codes exclude every candidate returns None); tests/test_places.py:52-73 and :88-101, tests/test_fetch_geonames.py:52-65 and :81-93 (country_row repeats the name in the capital column and puts English:Germany;German:Deutschland where the real file has the postal-code format).
What happens: with the snapshot code on the real tables, _countries()["victoria"] is {"SC"}, ["kingston"] is {"JM", "NF"}, ["washington"] is {"US"} and ["country"] is {"#ISO"}. So:
1. A second part that names a region sharing its name with a capital drops the location: resolve("Melbourne, Victoria") and resolve("Geelong, Victoria") are None, because Victoria is read as the Seychelles.
2. A country written in any language but English narrows nothing, and the largest city of that name wins: resolve("Valencia, España") is Valencia in Venezuela, resolve("Córdoba, España") is Córdoba in Argentina and resolve("León, España") is León in Mexico, while "Valencia, Spain" is placed in Spain.
Why it matters: the docstring promises a country name "in the languages its table gives it", and in the languages Postulo is used in, the companies of a Spanish, German or Portuguese reader land on the wrong continent without a word.
Fix: read only the ISO code (column 0), ISO3 (column 1) and the English name (column 4), and skip lines starting with #; drop the capital and the column-13 parsing. Names in other languages need a real source, fetched with the tables: GeoNames' alternate names for the countries' own geonameid (column 16), filtered to those ids when fetch_geonames runs, or CLDR's territory names; either is folded and mapped to the ISO code. Rewrite the fixtures in both test files from real countryInfo.txt rows, and correct the _countries docstring.

Done when

tests/test_places.py builds its country table from real countryInfo.txt rows and has tests that resolve("Melbourne, Victoria") is placed in Australia and resolve("Valencia, España") in Spain; both fail today.

Found on 2026-09-29 in a read-only review of postulo at 95348908d, reading `jobs/places.py` against the real `countryInfo.txt` and resolving sample locations with it. Related to #108. Checked again on 2026-10-01 against `main` at `0a5cad587`: still present. `_countries()` builds the table of country names that narrows a city match, but it reads the wrong columns. In GeoNames' `countryInfo.txt` column 4 is the country's English name, column 5 is its capital and column 13 is the postal-code format; there is no column of names by language. So every capital city becomes a "country", and no country name other than the English one is ever a hint. The tests pass because their fixture invents the layout the code expects. - **Where:** `jobs/places.py:111-136` (`_countries`: `candidates = [columns[4], columns[5]]`, then `columns[13].split(";")` parsed as `Language:Name`; the header line, which starts with `#ISO`, is not skipped); `jobs/places.py:169-175` (`resolve`: a hint whose codes exclude every candidate returns `None`); `tests/test_places.py:52-73` and `:88-101`, `tests/test_fetch_geonames.py:52-65` and `:81-93` (`country_row` repeats the name in the capital column and puts `English:Germany;German:Deutschland` where the real file has the postal-code format). - **What happens:** with the snapshot code on the real tables, `_countries()["victoria"]` is `{"SC"}`, `["kingston"]` is `{"JM", "NF"}`, `["washington"]` is `{"US"}` and `["country"]` is `{"#ISO"}`. So: 1. A second part that names a region sharing its name with a capital drops the location: `resolve("Melbourne, Victoria")` and `resolve("Geelong, Victoria")` are `None`, because Victoria is read as the Seychelles. 2. A country written in any language but English narrows nothing, and the largest city of that name wins: `resolve("Valencia, España")` is Valencia in Venezuela, `resolve("Córdoba, España")` is Córdoba in Argentina and `resolve("León, España")` is León in Mexico, while `"Valencia, Spain"` is placed in Spain. - **Why it matters:** the docstring promises a country name "in the languages its table gives it", and in the languages Postulo is used in, the companies of a Spanish, German or Portuguese reader land on the wrong continent without a word. - **Fix:** read only the ISO code (column 0), ISO3 (column 1) and the English name (column 4), and skip lines starting with `#`; drop the capital and the column-13 parsing. Names in other languages need a real source, fetched with the tables: GeoNames' alternate names for the countries' own `geonameid` (column 16), filtered to those ids when `fetch_geonames` runs, or CLDR's territory names; either is folded and mapped to the ISO code. Rewrite the fixtures in both test files from real `countryInfo.txt` rows, and correct the `_countries` docstring. ## Done when - `tests/test_places.py` builds its country table from real `countryInfo.txt` rows and has tests that `resolve("Melbourne, Victoria")` is placed in Australia and `resolve("Valencia, España")` in Spain; both fail today.

tiagoagueda added this to the 0.5.0 milestone

2026-10-01 10:47:27 +00:00

tiagoagueda added the

internationalisation

bug

labels

2026-10-01 10:47:27 +00:00

tiagoagueda referenced this issue

2026-10-03 01:59:10 +00:00

Maps on OpenStreetMap data: choose the engine, the precision and where the tiles come from #700

tiagoagueda referenced this issue

2026-10-03 01:59:11 +00:00