Spreadsheet import: deadlines and tags are lost without a note, on listings always and on any row whose date is not read #394

New issue

Closed

opened 2026-10-01 10:46:23 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-01 10:46:23 +00:00

Owner

Found on 2026-09-29 in a read-only review of postulo at 95348908d, reading core/csv_import.py, importing a small sheet on an in-memory database, and running parse_date on dates written the way several of the offered languages write them.

Checked again on 2026-10-01 against main at 0a5cad587: still present. core/csv_import.py shifted by a few lines when the currency list moved to core/currencies.py (#248). The code is now at core/csv_import.py:984 ("closes_at": None), :988-994 (the listing branch of perform), :1047-1052 (the tags), :265-272 (the aliases that mean draft) and :894 ("Status in the spreadsheet: …") for section 1; :860-865 (the two dates in parse_rows, the applied date's note at :862-863, the deadline at :865) for section 2; :537-557 (_MONTHS), :587-597 (the month-name branch of parse_date) and :1024-1038 ("Applied on an unknown date") for section 3; and :812-830 (ParsedRow.becomes, its last line at :830) and :796 (the default status) for the help text. The templates and jobs/closing.py have not moved, and the same sheet and the same dates give the same results.

Related to #222 (closed), which made the status decide what a row becomes and kept "the status, the channel, the tags and the deadline" for rows that become applications. This issue adds the three places where a deadline or the tags are still dropped: rows that become listings, deadlines parse_date cannot read, and the month names parse_date does not know. In all three the import succeeds and nothing on the preview, the result page or the record says a value was in the file.

1. A row that becomes a listing never keeps its deadline or its tags

Where: core/csv_import.py:989 ("closes_at": None, in posting_data, for every row); :993-999 (the listing branch of perform, which uses only row.notes); :1052-1057 (row.tags is used on the application path only).
What happens: a row whose status is draft, to apply, wishlist or one of their French and Portuguese aliases (:261-268) becomes a listing. For the sheet Company,Role,Status,Deadline,Tags with the row Acme,Dev,Wishlist,2026-10-15,remote; dream, the listing is created with closes_at empty and an empty description. The same row with the status Applied keeps the deadline on the application and gets both tags.
Why it matters: a wish list with closing dates is exactly what somebody moves into Postulo. The closing-soon notice (jobs/closing.py:55-66, closing_for, which filters undecided listings on closes_at) never fires for any imported listing. JobPosting has no tags, so the tags need somewhere to go.
Fix: for a row that becomes a listing, set closes_at from row.deadline. Since a listing cannot carry tags, add them to the notes that the listing branch already appends to its description (for example "Tags in the spreadsheet: remote, dream", translated, as "Status in the spreadsheet: …" is at :899).

2. A deadline that cannot be read leaves no trace

Where: core/csv_import.py:865-870 in parse_rows. An unreadable applied date is kept (row.notes.append(f"{sheet.headers[index]}: {value}") at :867-868); the deadline at :870 has no such branch. The preview table in templates/core/import_csv_map.html:74-83 has an Applied column and no Deadline column, so the loss cannot be seen before importing either.
What happens: a deadline cell reading "end of October", or any of the date forms in section 3, gives row.deadline = None and the text is gone, for applications and listings alike.
Fix: when parse_date returns None for a non-empty deadline, append the header and the raw value to row.notes, as the applied date does. Add a Deadline column to the preview.

3. `parse_date` reads month names in three languages only, and needs a four-digit year beside them

Where: core/csv_import.py:533-553 (_MONTHS: English, French and Portuguese) and :583-593 (the month-name branch of parse_date, where the year is next((n for n in numbers if n > 31), None), so a two-digit year below 32 is never taken as one).
What happens: parse_date returns None, whichever date order is chosen, for "5 enero 2026", "5 de enero de 2026", "5. Januar 2026", "15. Oktober 2026", "5 gennaio 2026", "5 stycznia 2026", "5 januari 2026", "2026. január 5." and for Excel's own "5-Jan-26". "5 janvier 2026", "5 de março de 2026", "Jan 5, 2026" and numeric forms such as "31.12.2026" all parse. A German sheet whose deadline column reads "15. Oktober 2026" loses every deadline silently (section 2). If its applied column is written the same way, the text survives in the notes but every application arrives with no applied date, recorded as "Applied on an unknown date" (:1029-1043), so the report and Insights do not count it in figures measured from the date sent.
Fix: build the month-name table from Django's translated MONTHS, MONTHS_3 and MONTHS_ALT (django.utils.dates) for every offered language, each under translation.override, computed once and cached; keep the hand-written abbreviations that Django does not have ("janv", "juil", "sept"). Accept a two-digit year after a month name when the text holds exactly a day, a month name and one other number.

templates/core/import_csv.html:34 and the wiki page Backups-and-your-data.md:65-66 say "Rows with a date applied become applications … rows without one become listings". Since #222 the status decides (ParsedRow.becomes, core/csv_import.py:817-835): a row with no date and the status Rejected becomes an application, and a dated draft row becomes a listing. A row with no status at all is read as applied (ParsedRow.status defaults to "applied" at :801 and map_status never returns an empty string), so it becomes an application too, and the last line of becomes (:835) is never reached. Correct both texts in the same change, and say there what happens to a listing's deadline and tags.

Done when

A test in tests/test_csv_import.py imports a Wishlist row with a deadline and tags and asserts the listing's closes_at and that the tags appear in its description.
A test there imports a row whose deadline is "end of October" and asserts the header and the raw value are kept, in the timeline entry for an application and in the description for a listing.
A test there asserts that parse_date reads "15. Oktober 2026", "5 de enero de 2026", "5 gennaio 2026" and "5-Jan-26" as their dates, and still returns None for "end of October".
The mapping page's preview shows the deadline, and the help text and the wiki page describe the status rule.

Found on 2026-09-29 in a read-only review of postulo at 95348908d, reading `core/csv_import.py`, importing a small sheet on an in-memory database, and running `parse_date` on dates written the way several of the offered languages write them. Checked again on 2026-10-01 against `main` at `0a5cad587`: still present. `core/csv_import.py` shifted by a few lines when the currency list moved to `core/currencies.py` (#248). The code is now at `core/csv_import.py:984` (`"closes_at": None`), `:988-994` (the listing branch of `perform`), `:1047-1052` (the tags), `:265-272` (the aliases that mean *draft*) and `:894` ("Status in the spreadsheet: …") for section 1; `:860-865` (the two dates in `parse_rows`, the applied date's note at `:862-863`, the deadline at `:865`) for section 2; `:537-557` (`_MONTHS`), `:587-597` (the month-name branch of `parse_date`) and `:1024-1038` ("Applied on an unknown date") for section 3; and `:812-830` (`ParsedRow.becomes`, its last line at `:830`) and `:796` (the default status) for the help text. The templates and `jobs/closing.py` have not moved, and the same sheet and the same dates give the same results. Related to #222 (closed), which made the status decide what a row becomes and kept "the status, the channel, the tags and the deadline" for rows that become applications. This issue adds the three places where a deadline or the tags are still dropped: rows that become listings, deadlines `parse_date` cannot read, and the month names `parse_date` does not know. In all three the import succeeds and nothing on the preview, the result page or the record says a value was in the file. ## 1. A row that becomes a listing never keeps its deadline or its tags - **Where:** `core/csv_import.py:989` (`"closes_at": None,` in `posting_data`, for every row); `:993-999` (the listing branch of `perform`, which uses only `row.notes`); `:1052-1057` (`row.tags` is used on the application path only). - **What happens:** a row whose status is *draft*, *to apply*, *wishlist* or one of their French and Portuguese aliases (`:261-268`) becomes a listing. For the sheet `Company,Role,Status,Deadline,Tags` with the row `Acme,Dev,Wishlist,2026-10-15,remote; dream`, the listing is created with `closes_at` empty and an empty description. The same row with the status *Applied* keeps the deadline on the application and gets both tags. - **Why it matters:** a wish list with closing dates is exactly what somebody moves into Postulo. The closing-soon notice (`jobs/closing.py:55-66`, `closing_for`, which filters undecided listings on `closes_at`) never fires for any imported listing. `JobPosting` has no tags, so the tags need somewhere to go. - **Fix:** for a row that becomes a listing, set `closes_at` from `row.deadline`. Since a listing cannot carry tags, add them to the notes that the listing branch already appends to its description (for example "Tags in the spreadsheet: remote, dream", translated, as "Status in the spreadsheet: …" is at `:899`). ## 2. A deadline that cannot be read leaves no trace - **Where:** `core/csv_import.py:865-870` in `parse_rows`. An unreadable applied date is kept (`row.notes.append(f"{sheet.headers[index]}: {value}")` at `:867-868`); the deadline at `:870` has no such branch. The preview table in `templates/core/import_csv_map.html:74-83` has an *Applied* column and no *Deadline* column, so the loss cannot be seen before importing either. - **What happens:** a deadline cell reading "end of October", or any of the date forms in section 3, gives `row.deadline = None` and the text is gone, for applications and listings alike. - **Fix:** when `parse_date` returns `None` for a non-empty deadline, append the header and the raw value to `row.notes`, as the applied date does. Add a *Deadline* column to the preview. ## 3. `parse_date` reads month names in three languages only, and needs a four-digit year beside them - **Where:** `core/csv_import.py:533-553` (`_MONTHS`: English, French and Portuguese) and `:583-593` (the month-name branch of `parse_date`, where the year is `next((n for n in numbers if n > 31), None)`, so a two-digit year below 32 is never taken as one). - **What happens:** `parse_date` returns `None`, whichever date order is chosen, for "5 enero 2026", "5 de enero de 2026", "5. Januar 2026", "15. Oktober 2026", "5 gennaio 2026", "5 stycznia 2026", "5 januari 2026", "2026. január 5." and for Excel's own "5-Jan-26". "5 janvier 2026", "5 de março de 2026", "Jan 5, 2026" and numeric forms such as "31.12.2026" all parse. A German sheet whose deadline column reads "15. Oktober 2026" loses every deadline silently (section 2). If its applied column is written the same way, the text survives in the notes but every application arrives with no applied date, recorded as "Applied on an unknown date" (`:1029-1043`), so the report and Insights do not count it in figures measured from the date sent. - **Fix:** build the month-name table from Django's translated `MONTHS`, `MONTHS_3` and `MONTHS_ALT` (`django.utils.dates`) for every offered language, each under `translation.override`, computed once and cached; keep the hand-written abbreviations that Django does not have ("janv", "juil", "sept"). Accept a two-digit year after a month name when the text holds exactly a day, a month name and one other number. ## Related: the help text still describes the rule #222 replaced `templates/core/import_csv.html:34` and the wiki page `Backups-and-your-data.md:65-66` say "Rows with a date applied become applications … rows without one become listings". Since #222 the status decides (`ParsedRow.becomes`, `core/csv_import.py:817-835`): a row with no date and the status *Rejected* becomes an application, and a dated *draft* row becomes a listing. A row with no status at all is read as *applied* (`ParsedRow.status` defaults to `"applied"` at `:801` and `map_status` never returns an empty string), so it becomes an application too, and the last line of `becomes` (`:835`) is never reached. Correct both texts in the same change, and say there what happens to a listing's deadline and tags. ## Done when - A test in `tests/test_csv_import.py` imports a *Wishlist* row with a deadline and tags and asserts the listing's `closes_at` and that the tags appear in its description. - A test there imports a row whose deadline is "end of October" and asserts the header and the raw value are kept, in the timeline entry for an application and in the description for a listing. - A test there asserts that `parse_date` reads "15. Oktober 2026", "5 de enero de 2026", "5 gennaio 2026" and "5-Jan-26" as their dates, and still returns `None` for "end of October". - The mapping page's preview shows the deadline, and the help text and the wiki page describe the status rule.

tiagoagueda added this to the 0.5.0 milestone

2026-10-01 10:46:23 +00:00

tiagoagueda added the

internationalisation

bug

labels

2026-10-01 10:46:23 +00:00

tiagoagueda referenced this issue from a commit

2026-10-04 10:04:57 +00:00

Keep spreadsheet deadlines and tags, and read more month names

tiagoagueda closed this issue

2026-10-04 10:04:57 +00:00