Map data: the downloaded GeoNames text tables are kept beside the lookup databases built from them #1127

New issue

Closed

opened 2026-10-31 03:04:13 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-31 03:04:13 +00:00

Owner

What exists today

manage.py fetch_geonames (src/postulo/jobs/management/commands/fetch_geonames.py:91-94) writes geonames-cities1000.txt and geonames-countryinfo.txt, then builds geonames-cities1000.sqlite from the text. With postcodes on, it writes geonames-postcodes.txt (line 161) and builds geonames-postcodes.sqlite (line 162). In the dev tree, src/postulo/jobs/data/geonames-cities1000.txt is 32,116,760 bytes and the lookup built from it 30,511,104 bytes.

After the build, the cities text is read for three things only, none of which needs its content:

jobs/places.py:87-89 _cities_file() returns it if present, and available() (line 364-366) answers "is the dataset here" with it;
_index() (lines 148-176) rebuilds the lookup when the text's mtime is newer than the lookup's, which matters only when an operator drops a new text in by hand;
docker/entrypoint.sh:32 downloads the dataset on start when the text is missing.

The country table (geonames-countryinfo.txt, read at places.py:211) is small and is read for its content; it is not part of this finding. The scout's "only for mtime comparison" is corrected: the text is also the presence marker, which is why it cannot simply be deleted without changing those three places.

What would change

fetch_geonames deletes the cities text (and the postal-code text) once the lookup it built has passed its own checks.
available() and _index() accept the lookup alone: present lookup and no text means "use the lookup"; a text newer than the lookup (an operator's hand-placed table, as entrypoint.sh:28-29 describes) still triggers a rebuild, after which the text may stay, since the operator put it there.
entrypoint.sh:32 tests for either file.

The download is unchanged; a re-fetch still writes the text, builds, and removes it.

What it saves or protects

small: 32 MB of geonames-cities1000.txt on every instance's data volume (measured in the dev tree), plus the postal-code text where postcodes are switched on (not measured here)

Accessibility

none

Decisions

Keep a hand-placed text after rebuilding from it? Recommended: yes, leave the operator's file alone; only delete what fetch_geonames itself downloaded.

Found by the efficiency and reliability audit of 2026-10-30 (finding D7; index postulo/postulo#1108). Checked against main at daee30b15 (2026-10-30).

## What exists today `manage.py fetch_geonames` (`src/postulo/jobs/management/commands/fetch_geonames.py:91-94`) writes `geonames-cities1000.txt` and `geonames-countryinfo.txt`, then builds `geonames-cities1000.sqlite` from the text. With postcodes on, it writes `geonames-postcodes.txt` (line 161) and builds `geonames-postcodes.sqlite` (line 162). In the dev tree, `src/postulo/jobs/data/geonames-cities1000.txt` is 32,116,760 bytes and the lookup built from it 30,511,104 bytes. After the build, the cities text is read for three things only, none of which needs its content: - `jobs/places.py:87-89` `_cities_file()` returns it if present, and `available()` (line 364-366) answers "is the dataset here" with it; - `_index()` (lines 148-176) rebuilds the lookup when the text's mtime is newer than the lookup's, which matters only when an operator drops a new text in by hand; - `docker/entrypoint.sh:32` downloads the dataset on start when the text is missing. The country table (`geonames-countryinfo.txt`, read at `places.py:211`) is small and is read for its content; it is not part of this finding. The scout's "only for mtime comparison" is corrected: the text is also the presence marker, which is why it cannot simply be deleted without changing those three places. ## What would change 1. `fetch_geonames` deletes the cities text (and the postal-code text) once the lookup it built has passed its own checks. 2. `available()` and `_index()` accept the lookup alone: present lookup and no text means "use the lookup"; a text newer than the lookup (an operator's hand-placed table, as `entrypoint.sh:28-29` describes) still triggers a rebuild, after which the text may stay, since the operator put it there. 3. `entrypoint.sh:32` tests for either file. The download is unchanged; a re-fetch still writes the text, builds, and removes it. ## What it saves or protects small: 32 MB of `geonames-cities1000.txt` on every instance's data volume (measured in the dev tree), plus the postal-code text where postcodes are switched on (not measured here) ## Accessibility none ## Decisions - **Keep a hand-placed text after rebuilding from it?** Recommended: yes, leave the operator's file alone; only delete what `fetch_geonames` itself downloaded. --- Found by the efficiency and reliability audit of 2026-10-30 (finding D7; index postulo/postulo#1108). Checked against `main` at `daee30b15` (2026-10-30).

tiagoagueda added this to the 0.6.0 milestone

2026-10-31 03:04:13 +00:00

tiagoagueda added the

performance

enhancement

labels

2026-10-31 03:04:13 +00:00

tiagoagueda referenced this issue

2026-10-31 03:04:40 +00:00

Efficiency and reliability audit of 2026-10-30: index #1108

tiagoagueda referenced this issue from a pull request that will close it,

2026-11-11 13:40:10 +00:00