Repository navigation
Conversation
Hugo makes every file a reader gets, so a source only has to be as large as
the biggest variant a template asks for. These had drifted far past that.
orpheus.png was 23944x5753: 138 megapixels, about 551 MB of RAM to decode,
for a logo the page shows at 560x135. Pillow refuses to open an image that
size at all, as a decompression bomb, and the build paid the cost on every
run.
26 sources are rewritten. themes/pybcn_theme/assets/images goes from
34.4 MB to 24.8 MB and from 392 megapixels to 139. The largest file in it
was orpheus.png at 138 Mpx and is now okta.png at 3.6.
The caps come from the templates:
sponsors Fit 1120x300 sponsor_summary.html, the 2x variant
photos 1600 wide image_hero.html, Fill 1600x450; the carousel
asks 1200 through image_resize.html
logo.png 920 wide nav.html, Resize 460x, the 2x of a 230px slot
Being over a cap is not enough on its own. A 512x512 logo shown at 300x300
costs almost nothing and rewriting it would throw away the only copy the
repository has, so a file is left alone unless it is also over 150 KB or
over 4 megapixels. That is what picks out the 23944px logo and leaves the
512px one.
bin/fit-sources is the script, in the shape of bin/square-photos: run by
hand when an image is added, not by the build and not by CI, with a dry run
and a list of named paths. The paths matter here, because one file to a
process gives the memory back between them: a run that held orpheus next to
the rest was killed on a machine with 8 GB.
Fewer pixels do not always mean fewer bytes. A JPEG saved at a low quality,
resized and written back at a high one, comes out larger, and eight of the
hero photos did, one of them by half again. The quality steps down from 88
until the file fits the weight it had. One still does not:
sponsors-area-pyday-2019.jpg goes from 372 KB to 406 KB at q76, and keeps
the change, because 1.7 megapixels instead of 4.2 is worth 34 KB.
bin/check-content fails on the next one, with the same two tables and the
same thresholds, pointing at the script. Checked against the old file: it
reports orpheus.png at 23944x5753 as an error.
Verified against the build: the 430 files Hugo generates are the same 430,
at the same pixel sizes, with one exception. The 1x sponsor variant of
backmarket goes from 225x150 to 226x150, because rounding its source to
451x300 moved the aspect ratio by a third of a pixel.
Three photographs under images/photos were PNG, which is the format for a drawing and not for a photograph. They were 2.3 MB between them, and none of the three used its alpha channel. canodrom.png and canodrom_header.png become JPEG at quality 88, the same pixels: 768 KB to 133 KB, and 923 KB to 135 KB. The five pages that name them follow. people-pyday-2019_1600_450.png goes with no replacement. It is the same photograph as people-pyday-2019.jpg, cropped to 1600x450 by hand before image_hero.html existed, and used by one page while three others use the JPEG. PyDay BCN 2020 now names the JPEG, which the pipeline crops to the same box: that page and /pybcn_association/collaborate/ now serve one generated file between them, where they served two. assets/images goes from 24.8 MB to 22.9 MB, and from 34.4 MB before the commit under this one. The logo at images/sponsors/canodrom.png is a different file and stays a PNG, as every logo does.
|
Are the old heavy images still kept in the git history? |
| note = f"{width}x{height} -> {target[0]}x{target[1]}" | ||
| if dry_run: | ||
| return f"{note} {before / 1024:.0f} KB {width * height / 1e6:.1f} Mpx" | ||
| quality = save(image.resize(target, Image.LANCZOS), path, before) |
There was a problem hiding this comment.
This might not be using LANCZOS if the image mode is P; see this line in the source code of PIL:
https://github.com/python-pillow/Pillow/blob/424d329c6fa43d3e577c1f83dd55f1490ca259a7/src/PIL/Image.py#L2418
Image.resize() ignores the resample argument for mode P and mode 1 and uses NEAREST, because there is no sensible way to average two palette indices. The script asked for LANCZOS and did not get it. One of the 26 sources is a palette PNG, sponsors/jobfluent.png, and it was scaled down by five and a half with nearest neighbour. What that cost a reader, measured on the 546x150 WebP the page actually serves, is almost nothing. Against the ideal, which is the original resized straight to that size, the nearest neighbour route is 6.88 of 255 away and the correct one 5.80. The second reduction, from 1092 to 546, averages away most of the damage, and this logo is the mildest possible case: a flat wordmark on a palette of 33 colours, where nearest neighbour has little to spoil. Zoomed to twice the served size the three are hard to tell apart. The fix is worth having for the next one rather than for this one. A palette PNG can be a screenshot or a diagram, and there the filter is the difference between a readable image and a broken one. A script that says LANCZOS should use LANCZOS. The image is converted before the resize, to RGBA when the palette carries transparency and to RGB when it does not, and quantised back afterwards. A palette packs a flat logo into a fraction of what full colour needs: this file is 11 KB on a palette and 38 KB in RGBA, which is what the original weighed. The quantised version sits 2.07 from the ideal against the RGBA one's 1.71, a quarter of a point of mean difference for a quarter of the bytes. FASTOCTREE is the method that takes RGBA and keeps the alpha: the count of fully transparent pixels is unchanged at 229,577. Found by Daniel Mesejo in review, with the line of the Pillow source.
|
Yes, they are, and they will stay there unless we decide otherwise. Where the weight is: the repository is 139 MB. The eight largest still in it:
So what this pull request saves is on every build and on the working tree, not on the clone. A fresh clone still carries all of it, once. Removing them means rewriting the history with My view: not now, and probably not for 139 MB. GitHub starts warning at 1 GB. It is a one-off disruption that we would be paying while six pull requests are in flight, for a number that hurts nobody today, and it costs a one-time clone rather than anything recurring. If we ever do it, the moment is an empty queue and a day when everyone can re-clone. Worth an issue so it is written down rather than forgotten? |
|
@mesejo both of your points are answered, and the second one was a real bug. If you are happy with the answers, this one is ready for your approval. The LANCZOS catch. You were right, and it bit exactly one of the 26 files: I also measured what it had actually cost a reader, because my first answer overstated it. On the 546x150 WebP the page serves, the nearest neighbour route sits 6.88 of 255 from the ideal and the correct one 5.80: the second reduction averages away most of the damage, and this logo is the mildest possible case, a flat wordmark on a palette of 33 colours. So the fix is worth having for the next palette image, which could be a screenshot or a diagram, rather than for this one. The git history. Answered in the comment above: the heavy blobs stay, they are all on The three checks are green and the branch is up to date with |
…e-sources # Conflicts: # bin/check-content
This branch has not been deployed
2 participants
Hugo makes every file a reader gets, so an image source only has to be as large as the biggest variant a template asks for. These had drifted far past that.
orpheus.pngwas 23,944 x 5,753: 138 megapixels, about 551 MB of RAM to decode, for a logo the page shows at 560x135. Pillow refuses to open an image that size at all, as a decompression bomb, and the build paid the cost on every run. This is the open item of #190.26 sources are rewritten.
themes/pybcn_theme/assets/imagesgoes from 34.4 MB to 24.8 MB and from 392 megapixels to 139. The largest file in it wasorpheus.pngat 138 Mpx and is nowokta.pngat 3.6.The caps come from the templates
sponsors/sponsor_summary.html, the 2x variantphotos/image_hero.html, Fill 1600x450; the carousel asks 1200logo.pngnav.html, Resize 460x, the 2x of a 230px slotBeing over a cap is not enough on its own. A 512x512 logo shown at 300x300 costs almost nothing, and rewriting it would throw away the only copy the repository has, so a file is left alone unless it is also over 150 KB or over 4 megapixels. That is what picks out the 23,944px logo and leaves the 512px one.
The script
bin/fit-sources, in the shape ofbin/square-photos: run by hand when an image is added, not by the build and not by CI, with a dry run and a list of named paths. The paths matter here, because one file to a process gives the memory back between them: a run that heldorpheus.pngnext to the rest was killed on a machine with 8 GB.Fewer pixels do not always mean fewer bytes. A JPEG saved at a low quality, resized and written back at a high one, comes out larger, and eight of the hero photos did, one of them by half again. The quality steps down from 88 until the file fits the weight it had. One still does not:
sponsors-area-pyday-2019.jpggoes from 372 KB to 406 KB at q76, and keeps the change, because 1.7 megapixels instead of 4.2 is worth 34 KB.The check
bin/check-contentfails on the next one, with the same two tables and the same thresholds, pointing at the script. Checked against the old file, it reports:Verified
The 430 files Hugo generates are the same 430, at the same pixel sizes, with one exception: the 1x sponsor variant of Back Market goes from 225x150 to 226x150, because rounding its source to 451x300 moved the aspect ratio by a third of a pixel.
Both checks green, build clean.
Issues
Closes #169, "Reduir pes imatges participants PyDay": the person photographs went from 38.7 MB to 12.00 MB in #195 and the organizers page now ships 0.25 MB of images, and this cuts what was left of the oversized sources.
Part of #171, "Actualitzar imatges carrusel". It does the weight half: the Canodrom photograph the issue names was 981 KB as a PNG and is 133 KB as a JPEG. It does not do the other half, which is choosing newer photographs for the carousel, so the issue stays open for that.