Tags and industries: an ASCII-only slug as identity drops non-Latin names, merges C# into C++ and answers 500 #356

New issue

Closed

opened 2026-10-01 10:46:06 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-01 10:46:06 +00:00

Owner

Found on 2026-09-29 in a read-only review of postulo at 95348908d, following every path that makes a tag or an industry (the two pages, the application and company forms, the API, the spreadsheet import, the archive) and running each against the snapshot with the test client.

Checked again on 2026-10-01 against main at 0a5cad587: still present. Two files gained or lost lines above the code, which is otherwise unchanged: the spreadsheet import's tag loop is now at core/csv_import.py:1047-1052 and its transaction at :941; in jobs/forms.py the Industry.named call is now at :334, IndustryForm.clean_name's lookup at :551 and IndustryForm.save's slug at :566. Everything else is where it was.

A tag and an industry are unique per (owner, slug), and the slug comes from django.utils.text.slugify() without allow_unicode, which deletes every character outside ASCII. That slug is computed in seven places that do not agree: some match an existing record by slug and some by name, one cuts at 60 characters where the others cut at 160, one falls back to other and the rest to the empty string, and a renamed tag keeps the slug of its old name. C++, C# and C all become c; any name written wholly in Greek, Cyrillic, Georgian, Armenian, Arabic, Hebrew, Ethiopic or CJK becomes "". Depending on the path, the person's label is silently dropped, attached to a different tag, duplicated, lost on an export and import, or refused by the database as an IntegrityError, a 500. Postulo ships catalogues for el, bg, uk, mk, sr, ka, hy, ar, am and ti, among others.

Where:
- core/models.py:195-197, the unique_tag_slug_per_owner constraint; jobs/models.py:67-69, unique_industry_slug_per_owner.
- core/models.py:202-205, Tag.save: self.slug = slugify(self.name)[:60], only while the slug is empty, with no fallback. A rename keeps the old slug; a non-Latin name is stored with "".
- core/models.py:232-258, Tag.named (the application forms' New tags box, applications/forms.py:289 and :368): matches by slug, and if not name or not slug or slug in seen: continue skips a name whose slug is empty.
- applications/forms.py:615-624, TagForm.clean_name: looks for a clash with name__iexact only. The slug is not a form field, so model validation never looks at the constraint.
- api/routers/common.py:71-82, tags_named (called at api/routers/applications.py:104 and api/routers/listings.py:208): filter(name__iexact=name).first(), else Tag.objects.create(owner=owner, name=name), with no cut to the column's 60 characters.
- core/csv_import.py:1052-1057, in perform: the same by-name lookup, then Tag.objects.create(owner=user, name=name[:60]), looking up the uncut name and storing the cut one.
- jobs/models.py:74-76, Industry.save: slugify(self.name)[:160] or "other", only while the slug is empty.
- jobs/models.py:83-102, Industry.named (called at jobs/forms.py:333, api/routers/companies.py:67 and :99, and core/importer.py:585): cuts at 160 and skips an empty slug.
- jobs/forms.py:554, IndustryForm.clean_name: looks for slug=slugify(name)[:60]; jobs/forms.py:569, IndustryForm.save: industry.slug = slugify(industry.name)[:60] or "other", on every save. The 60 is the length from before #140 widened the field to 160.
- core/export.py:703 writes an application's tags as slugs; core/importer.py:404-413 remakes a tag under entry.get("slug") or entry.get("name", "").lower() and keys tags_by_slug on the result; core/importer.py:667-670 looks the application's slugs up in it.
- applications/views.py:102 filters by tags__slug; templates/applications/partials/filters.html:35 uses the slug as the option's value.
What happens (each run against the snapshot unless marked as read):
1. Non-Latin names are dropped without a word. Tag.named(user, ["Κάτι", "日本語"]) returns [], so a tag typed into New tags in Greek, Russian or Japanese is not made and nothing says so. Industry.named(user, ["Банки", "Τράπεζες", "Software"]) returns only Software: the company form's Other industries box drops the name, including one picked from the suggestions the form itself offers in Bulgarian, and POST /api/v1/companies with "industries": ["Πληροφορική"] answers 201 with industries: [].
2. The second non-Latin name on either page is a 500. On the Tags page Επείγον is saved with the slug ""; Κάτι then passes clean_name and raises IntegrityError: UNIQUE constraint failed: core_tag.owner_id, core_tag.slug. On the Industries page Ενέργεια is saved as other; Υγεία passes clean_name, which looked for the slug "", and fails the same way. So does the first non-Latin industry when one called Other already exists.
3. Latin names that share a slug collide. With C++ present, C# (or C) on the Tags page is a 500, and typed into New tags it attaches the C++ tag. Remote! beside remote is a 500 too: name__iexact does not match and the slug does.
4. A renamed tag answers to its old name. Rename Remote to Hybrid and the slug stays remote. Typing Remote into New tags attaches the tag now called Hybrid; typing Hybrid makes a second tag named Hybrid; making Remote on the Tags page is a 500.
5. The API fails on ordinary tags. With a tag remote, POST /api/v1/applications with "tags": ["Remote!"] answers 500; so does "tags": ["C++", "C#"], and a second Greek or Cyrillic tag. POST /api/v1/listings/{id}/apply goes through the same helper. A tag name longer than 60 characters is stored whole on SQLite; the column is varchar(60), which PostgreSQL refuses (read, not run).
6. The spreadsheet import fails as a whole. A Tags cell holding C++; C#, Remote!; remote or удалённо; мечта raises the same IntegrityError, on a single row. So does a 70-character tag on two rows: the second row looks up the full name, misses the cut tag the first row made, and makes it again. perform runs in one transaction (core/csv_import.py:946), so the page answers 500 and nothing from the file is imported.
7. The Industries page and the model compute different keys for a long name. "Undifferentiated goods- and services-producing activities of private households for own use" gets a 90-character slug through Industry.named; adding the same name on the Industries page passes clean_name, which looks for the 60-character slug, and makes a second industry with the same name. Opening an industry with an 89-character slug on the Industries page and saving it unchanged rewrites the slug to 60 characters, and the next Industry.named call with that name makes a duplicate. Many of the NACE division names offered as suggestions (#140) are this long.
8. An export and import loses them. A tag whose slug is "" is exported as "" on its applications; the importer remakes the tag under name.lower(), the reference to "" matches nothing, and the tag falls off every application. A company's industries are exported by name and imported through Industry.named, so the one Cyrillic industry that did exist (as other) is dropped from its company.
9. The filter cannot select it (read). A tag whose slug is "" renders as <option value="">, the same value as Any.
Why it matters: somebody using Postulo in a non-Latin script loses what they type on the application and company forms, cannot have a second tag or a second industry, meets an error page on two plain forms, and loses tags and industries when restoring from an archive. Anybody at all meets the 500 with C# beside C++ or with a renamed tag's old name, an agent using the API gets a 500 for an ordinary tag, and a whole spreadsheet fails to import with no explanation. Duplicate industries split a company list in two: the ?industry= filter and the counts on the Industries page each show half, until the person finds and merges them by hand.
Fix: one rule, in one place per model, used by every path.
1. Add a single slug function (for example Tag.slug_for(name) and Industry.slug_for(name), or one shared helper): slugify(name, allow_unicode=True), cut to cls._meta.get_field("slug").max_length, never empty (a fixed word such as tag or industry when nothing is left, as for ++), and made unique for the owner with a numeric suffix (c, c-2). Declare both fields SlugField(allow_unicode=True).
2. Make the name the identity and the slug a derived handle for URLs and the filter: Tag.named and Industry.named match an existing record by its name, cut to the column's length and compared case-insensitively with whitespace collapsed, and otherwise create one through the slug function. This is what lets C# and C++ coexist. Industry.named promises that "capitals and accents do not breed duplicates"; if that is to hold, strip combining marks in the comparison as well.
3. Call them everywhere: tags_named and the loop in core/csv_import.py:1052-1057 become Tag.named(...); TagForm.clean_name and IndustryForm.clean_name check the clash by the same comparison, so a clash is a form error and never an IntegrityError; IndustryForm.save stops setting the slug itself.
4. Recompute the slug in save whenever the name has changed, with the same uniqueness step.
5. In core/importer.py:404-413, make tags through the same helper and key tags_by_slug on the slug the archive gave (entry.get("slug", "")), not on the slug of the row that was made, so archives already written with "" keep their tags.
6. A data migration re-derives the slug of every tag and industry whose slug is "", is other for a name that is not Other, was cut at 60, or no longer matches its name, and merges industries left with identical names, moving their companies.

Done when

tests/test_tags.py: C++ then C#, remote then Remote!, and Επείγον then Κάτι each end as two distinct tags with no error, through the Tags page (or a form error where the names are equal) and through the New tags box; Tag.named returns a tag for a Greek, a Cyrillic and a Japanese name; after renaming Remote to Hybrid, Tag.named with Hybrid returns the renamed tag and with Remote a new one, and making Remote on the Tags page succeeds; a non-Latin tag can be chosen in the applications filter and is still on its application after an export and import. Each fails today.
tests/test_industries.py: two Greek industries can be made on the Industries page; Industry.named returns a row for Банки and Τράπεζες; a company saved with Банки under Other industries has that industry, and still has it after an export and import; a name with more than 60 slug characters made through Industry.named, saved unchanged through jobs:industry_update and passed to Industry.named again is one industry, and posting it to jobs:industry_create gives the "You already have that industry." error.
tests/test_api.py: POST /api/v1/applications with ["Remote!"] while remote exists, with ["C++", "C#"] and with a 70-character tag answers 201; POST /api/v1/companies with a Greek industry returns it.
tests/test_csv_import.py: a sheet whose Tags cells hold C++; C#, two Cyrillic names, and a 70-character tag repeated on two rows imports with the expected tags.
A migration test: rows with the slugs "" and other, a tag renamed before the migration, and two industries with one name come out with distinct, non-empty slugs and one industry.

Found on 2026-09-29 in a read-only review of postulo at 95348908d, following every path that makes a tag or an industry (the two pages, the application and company forms, the API, the spreadsheet import, the archive) and running each against the snapshot with the test client. Checked again on 2026-10-01 against `main` at `0a5cad587`: still present. Two files gained or lost lines above the code, which is otherwise unchanged: the spreadsheet import's tag loop is now at `core/csv_import.py:1047-1052` and its transaction at `:941`; in `jobs/forms.py` the `Industry.named` call is now at `:334`, `IndustryForm.clean_name`'s lookup at `:551` and `IndustryForm.save`'s slug at `:566`. Everything else is where it was. A tag and an industry are unique per `(owner, slug)`, and the slug comes from `django.utils.text.slugify()` without `allow_unicode`, which deletes every character outside ASCII. That slug is computed in seven places that do not agree: some match an existing record by slug and some by name, one cuts at 60 characters where the others cut at 160, one falls back to `other` and the rest to the empty string, and a renamed tag keeps the slug of its old name. `C++`, `C#` and `C` all become `c`; any name written wholly in Greek, Cyrillic, Georgian, Armenian, Arabic, Hebrew, Ethiopic or CJK becomes `""`. Depending on the path, the person's label is silently dropped, attached to a different tag, duplicated, lost on an export and import, or refused by the database as an `IntegrityError`, a 500. Postulo ships catalogues for el, bg, uk, mk, sr, ka, hy, ar, am and ti, among others. - **Where:** - `core/models.py:195-197`, the `unique_tag_slug_per_owner` constraint; `jobs/models.py:67-69`, `unique_industry_slug_per_owner`. - `core/models.py:202-205`, `Tag.save`: `self.slug = slugify(self.name)[:60]`, only while the slug is empty, with no fallback. A rename keeps the old slug; a non-Latin name is stored with `""`. - `core/models.py:232-258`, `Tag.named` (the application forms' *New tags* box, `applications/forms.py:289` and `:368`): matches by slug, and `if not name or not slug or slug in seen: continue` skips a name whose slug is empty. - `applications/forms.py:615-624`, `TagForm.clean_name`: looks for a clash with `name__iexact` only. The slug is not a form field, so model validation never looks at the constraint. - `api/routers/common.py:71-82`, `tags_named` (called at `api/routers/applications.py:104` and `api/routers/listings.py:208`): `filter(name__iexact=name).first()`, else `Tag.objects.create(owner=owner, name=name)`, with no cut to the column's 60 characters. - `core/csv_import.py:1052-1057`, in `perform`: the same by-name lookup, then `Tag.objects.create(owner=user, name=name[:60])`, looking up the uncut name and storing the cut one. - `jobs/models.py:74-76`, `Industry.save`: `slugify(self.name)[:160] or "other"`, only while the slug is empty. - `jobs/models.py:83-102`, `Industry.named` (called at `jobs/forms.py:333`, `api/routers/companies.py:67` and `:99`, and `core/importer.py:585`): cuts at 160 and skips an empty slug. - `jobs/forms.py:554`, `IndustryForm.clean_name`: looks for `slug=slugify(name)[:60]`; `jobs/forms.py:569`, `IndustryForm.save`: `industry.slug = slugify(industry.name)[:60] or "other"`, on every save. The 60 is the length from before #140 widened the field to 160. - `core/export.py:703` writes an application's tags as slugs; `core/importer.py:404-413` remakes a tag under `entry.get("slug") or entry.get("name", "").lower()` and keys `tags_by_slug` on the result; `core/importer.py:667-670` looks the application's slugs up in it. - `applications/views.py:102` filters by `tags__slug`; `templates/applications/partials/filters.html:35` uses the slug as the option's value. - **What happens** (each run against the snapshot unless marked as read): 1. *Non-Latin names are dropped without a word.* `Tag.named(user, ["Κάτι", "日本語"])` returns `[]`, so a tag typed into *New tags* in Greek, Russian or Japanese is not made and nothing says so. `Industry.named(user, ["Банки", "Τράπεζες", "Software"])` returns only *Software*: the company form's *Other industries* box drops the name, including one picked from the suggestions the form itself offers in Bulgarian, and `POST /api/v1/companies` with `"industries": ["Πληροφορική"]` answers 201 with `industries: []`. 2. *The second non-Latin name on either page is a 500.* On the Tags page *Επείγον* is saved with the slug `""`; *Κάτι* then passes `clean_name` and raises `IntegrityError: UNIQUE constraint failed: core_tag.owner_id, core_tag.slug`. On the Industries page *Ενέργεια* is saved as `other`; *Υγεία* passes `clean_name`, which looked for the slug `""`, and fails the same way. So does the first non-Latin industry when one called *Other* already exists. 3. *Latin names that share a slug collide.* With `C++` present, `C#` (or `C`) on the Tags page is a 500, and typed into *New tags* it attaches the `C++` tag. `Remote!` beside `remote` is a 500 too: `name__iexact` does not match and the slug does. 4. *A renamed tag answers to its old name.* Rename *Remote* to *Hybrid* and the slug stays `remote`. Typing *Remote* into *New tags* attaches the tag now called *Hybrid*; typing *Hybrid* makes a second tag named *Hybrid*; making *Remote* on the Tags page is a 500. 5. *The API fails on ordinary tags.* With a tag `remote`, `POST /api/v1/applications` with `"tags": ["Remote!"]` answers 500; so does `"tags": ["C++", "C#"]`, and a second Greek or Cyrillic tag. `POST /api/v1/listings/{id}/apply` goes through the same helper. A tag name longer than 60 characters is stored whole on SQLite; the column is `varchar(60)`, which PostgreSQL refuses (read, not run). 6. *The spreadsheet import fails as a whole.* A Tags cell holding `C++; C#`, `Remote!; remote` or `удалённо; мечта` raises the same `IntegrityError`, on a single row. So does a 70-character tag on two rows: the second row looks up the full name, misses the cut tag the first row made, and makes it again. `perform` runs in one transaction (`core/csv_import.py:946`), so the page answers 500 and nothing from the file is imported. 7. *The Industries page and the model compute different keys for a long name.* "Undifferentiated goods- and services-producing activities of private households for own use" gets a 90-character slug through `Industry.named`; adding the same name on the Industries page passes `clean_name`, which looks for the 60-character slug, and makes a second industry with the same name. Opening an industry with an 89-character slug on the Industries page and saving it unchanged rewrites the slug to 60 characters, and the next `Industry.named` call with that name makes a duplicate. Many of the NACE division names offered as suggestions (#140) are this long. 8. *An export and import loses them.* A tag whose slug is `""` is exported as `""` on its applications; the importer remakes the tag under `name.lower()`, the reference to `""` matches nothing, and the tag falls off every application. A company's industries are exported by name and imported through `Industry.named`, so the one Cyrillic industry that did exist (as `other`) is dropped from its company. 9. *The filter cannot select it* (read). A tag whose slug is `""` renders as `<option value="">`, the same value as *Any*. - **Why it matters:** somebody using Postulo in a non-Latin script loses what they type on the application and company forms, cannot have a second tag or a second industry, meets an error page on two plain forms, and loses tags and industries when restoring from an archive. Anybody at all meets the 500 with `C#` beside `C++` or with a renamed tag's old name, an agent using the API gets a 500 for an ordinary tag, and a whole spreadsheet fails to import with no explanation. Duplicate industries split a company list in two: the `?industry=` filter and the counts on the Industries page each show half, until the person finds and merges them by hand. - **Fix:** one rule, in one place per model, used by every path. 1. Add a single slug function (for example `Tag.slug_for(name)` and `Industry.slug_for(name)`, or one shared helper): `slugify(name, allow_unicode=True)`, cut to `cls._meta.get_field("slug").max_length`, never empty (a fixed word such as `tag` or `industry` when nothing is left, as for `++`), and made unique for the owner with a numeric suffix (`c`, `c-2`). Declare both fields `SlugField(allow_unicode=True)`. 2. Make the name the identity and the slug a derived handle for URLs and the filter: `Tag.named` and `Industry.named` match an existing record by its name, cut to the column's length and compared case-insensitively with whitespace collapsed, and otherwise create one through the slug function. This is what lets `C#` and `C++` coexist. `Industry.named` promises that "capitals and accents do not breed duplicates"; if that is to hold, strip combining marks in the comparison as well. 3. Call them everywhere: `tags_named` and the loop in `core/csv_import.py:1052-1057` become `Tag.named(...)`; `TagForm.clean_name` and `IndustryForm.clean_name` check the clash by the same comparison, so a clash is a form error and never an `IntegrityError`; `IndustryForm.save` stops setting the slug itself. 4. Recompute the slug in `save` whenever the name has changed, with the same uniqueness step. 5. In `core/importer.py:404-413`, make tags through the same helper and key `tags_by_slug` on the slug the archive gave (`entry.get("slug", "")`), not on the slug of the row that was made, so archives already written with `""` keep their tags. 6. A data migration re-derives the slug of every tag and industry whose slug is `""`, is `other` for a name that is not *Other*, was cut at 60, or no longer matches its name, and merges industries left with identical names, moving their companies. ## Done when - `tests/test_tags.py`: `C++` then `C#`, `remote` then `Remote!`, and *Επείγον* then *Κάτι* each end as two distinct tags with no error, through the Tags page (or a form error where the names are equal) and through the *New tags* box; `Tag.named` returns a tag for a Greek, a Cyrillic and a Japanese name; after renaming *Remote* to *Hybrid*, `Tag.named` with *Hybrid* returns the renamed tag and with *Remote* a new one, and making *Remote* on the Tags page succeeds; a non-Latin tag can be chosen in the applications filter and is still on its application after an export and import. Each fails today. - `tests/test_industries.py`: two Greek industries can be made on the Industries page; `Industry.named` returns a row for *Банки* and *Τράπεζες*; a company saved with *Банки* under *Other industries* has that industry, and still has it after an export and import; a name with more than 60 slug characters made through `Industry.named`, saved unchanged through `jobs:industry_update` and passed to `Industry.named` again is one industry, and posting it to `jobs:industry_create` gives the "You already have that industry." error. - `tests/test_api.py`: `POST /api/v1/applications` with `["Remote!"]` while `remote` exists, with `["C++", "C#"]` and with a 70-character tag answers 201; `POST /api/v1/companies` with a Greek industry returns it. - `tests/test_csv_import.py`: a sheet whose Tags cells hold `C++; C#`, two Cyrillic names, and a 70-character tag repeated on two rows imports with the expected tags. - A migration test: rows with the slugs `""` and `other`, a tag renamed before the migration, and two industries with one name come out with distinct, non-empty slugs and one industry.

tiagoagueda added this to the 0.5.0 milestone

2026-10-01 10:46:06 +00:00

tiagoagueda added the

bug

internationalisation

labels

2026-10-01 10:46:06 +00:00

tiagoagueda referenced this issue from a commit

2026-10-04 10:05:00 +00:00

Make a tag or an industry its name, with a Unicode slug derived from it

tiagoagueda closed this issue

2026-10-04 10:05:00 +00:00