Backups: the archive is gzipped at level 9, which costs about 45% more CPU on the database than level 6 for 4% less size #1124

New issue

Closed

opened 2026-10-31 03:04:11 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-31 03:04:11 +00:00

Owner

What exists today

src/postulo/core/backup.py:548 writes the archive with tarfile.open(path, "w:gz"), whose compresslevel defaults to 9. The archive holds the SQLite copy (database.sqlite3, made with SQLite's backup API, lines 327-330), the media tree and the plugins directory (lines 553-556).

Measured with zlib.compress at levels 1, 6 and 9 (python -I, this machine):

Sample	Size	L1	L6	L9
a 30.5 MB SQLite file (`jobs/data/geonames-cities1000.sqlite`)	30.5 MB	19.10 MB, 375 ms	16.49 MB, 942 ms	15.91 MB, 1,356 ms
the dev database `data/postulo.sqlite3`	1.4 MB	0.10 MB, 4 ms	0.07 MB, 8 ms	0.07 MB, 22 ms
20 MB of random bytes, standing for JPEG/PNG/PDF uploads	21.0 MB	22.12 MB, 356 ms	20.98 MB, 678 ms	20.98 MB, 593 ms
the six local media PDFs	0.4 MB	0.43 MB, 7 ms	0.41 MB, 13 ms	0.41 MB, 11 ms

Correction to the scout: the cost is not "level 9 over already-compressed media". On data that does not compress, deflate at level 9 is no slower than level 6 (593 against 678 ms). The extra cost of level 9 is on the database, where it is about 45% more time for 3.6% less size. Level 1 is faster still but expands incompressible media by 5% and leaves the database 16% larger than level 6, so it is not recommended.

What would change

tarfile.open(path, "w:gz", compresslevel=6). Reading is unchanged (gzip decompression does not depend on the level), so older archives, verify_backup, restore and the offsite copy are untouched, and the archive format number does not change. A modern alternative (zstd) would need Python 3.14's compression.zstd or a new dependency and a new archive format; not worth it for this.

What it saves or protects

small: about a third less compression time on the database part of every backup (measured 1.36 s at level 9 against 0.94 s at level 6 on a 30 MB SQLite file), for archives about 4% larger on that part and no different on media

Accessibility

none

Decided by the maintainer (2026-10-31)

backups at gzip level 6, a constant

Decisions

Level 6 instead of 9? Recommended: yes; it is gzip's own default, and the size difference is a few per cent of the database only.
Make it a setting? Recommended: no; one sensible constant.

Found by the efficiency and reliability audit of 2026-10-30 (finding D4; index postulo/postulo#1108). Checked against main at daee30b15 (2026-10-30).

## What exists today `src/postulo/core/backup.py:548` writes the archive with `tarfile.open(path, "w:gz")`, whose `compresslevel` defaults to 9. The archive holds the SQLite copy (`database.sqlite3`, made with SQLite's backup API, lines 327-330), the media tree and the plugins directory (lines 553-556). Measured with `zlib.compress` at levels 1, 6 and 9 (`python -I`, this machine): | Sample | Size | L1 | L6 | L9 | |---|---|---|---|---| | a 30.5 MB SQLite file (`jobs/data/geonames-cities1000.sqlite`) | 30.5 MB | 19.10 MB, 375 ms | 16.49 MB, 942 ms | 15.91 MB, 1,356 ms | | the dev database `data/postulo.sqlite3` | 1.4 MB | 0.10 MB, 4 ms | 0.07 MB, 8 ms | 0.07 MB, 22 ms | | 20 MB of random bytes, standing for JPEG/PNG/PDF uploads | 21.0 MB | 22.12 MB, 356 ms | 20.98 MB, 678 ms | 20.98 MB, 593 ms | | the six local media PDFs | 0.4 MB | 0.43 MB, 7 ms | 0.41 MB, 13 ms | 0.41 MB, 11 ms | Correction to the scout: the cost is not "level 9 over already-compressed media". On data that does not compress, deflate at level 9 is no slower than level 6 (593 against 678 ms). The extra cost of level 9 is on the database, where it is about 45% more time for 3.6% less size. Level 1 is faster still but expands incompressible media by 5% and leaves the database 16% larger than level 6, so it is not recommended. ## What would change `tarfile.open(path, "w:gz", compresslevel=6)`. Reading is unchanged (gzip decompression does not depend on the level), so older archives, `verify_backup`, restore and the offsite copy are untouched, and the archive format number does not change. A modern alternative (zstd) would need Python 3.14's `compression.zstd` or a new dependency and a new archive format; not worth it for this. ## What it saves or protects small: about a third less compression time on the database part of every backup (measured 1.36 s at level 9 against 0.94 s at level 6 on a 30 MB SQLite file), for archives about 4% larger on that part and no different on media ## Accessibility none ## Decided by the maintainer (2026-10-31) backups at gzip level 6, a constant ## Decisions - **Level 6 instead of 9?** Recommended: yes; it is gzip's own default, and the size difference is a few per cent of the database only. - **Make it a setting?** Recommended: no; one sensible constant. --- Found by the efficiency and reliability audit of 2026-10-30 (finding D4; index postulo/postulo#1108). Checked against `main` at `daee30b15` (2026-10-30).

tiagoagueda added this to the 0.6.0 milestone

2026-10-31 03:04:11 +00:00

tiagoagueda added the

performance

enhancement

labels

2026-10-31 03:04:11 +00:00

tiagoagueda referenced this issue

2026-10-31 03:04:40 +00:00

Efficiency and reliability audit of 2026-10-30: index #1108

tiagoagueda referenced this issue from a pull request that will close it,

2026-11-11 13:40:10 +00:00