Repository navigation
Conversation
Member
|
We don't really like to install software from source unless it's essential so I need to think about this one. |
Contributor
|
Do tile servers collect performance metrics? This is the sort of stuff that would be great to A/B test. |
Member
|
You mean something other that what's in prometheus? What did you have in mind? |
Contributor
|
Ideally there would be some identical tile servers, differing only in which variant of zlib they are using. This detail (zlib version) would be a tag fed up to Prometheus, resulting in comparable metrics that can inform whether the difference is significant enough to consider the change of libraries. Having said that, looking at https://hardware.openstreetmap.org I'm guessing that "identical tile servers" is a non-starter. Unless the physical file servers have some sort of logical split into several tile servers with containers or something. No idea if that's a thing; out of my depth at that point. |
Contributor
|
Hm, perhaps the two azures. They are described slightly differently in the hardware page ("Lenovo HR630X" and "Lenovo ThinkSystem HR630X") and that threw me off. |
Member
|
I don't think there is any disagreement about switching to using |
Member
|
I mean LD_PRELOAD is a bit of a smell as well really... The azure's probably aren't the best for comparisons though - odin and ysera are likely better. But yes I don't have a problem with zlib-ng in principle - to be honest I'm astonished it's not at least packaged for Debian as I kind of assumed everybody else had switched to it as a default like Fedora did ages ago. |
Member
|
Collaborator
Member
|
So I built a package successfully using that repo, and then updated it to the latest zlib-ng version and built that successfully. One issue is that the package it builds for the library with the zlib compatibility interface is designed to replace the normal zlib library so can't be installed alongside and it will become the zlib library for the whole system - that's probably fine (after all Fedora uses it like that) and it avoids having to rebuild mod_tile but it is quite a big step to take. |
Author
|
The system-wide replacement is just a choice of that particular packaging, because it was intended for Debian's switchover (#1002056). It's not inherent to zlib-ng. For us, the ZLIB_COMPAT build might be better installed into a separate/private directory, say, Pushed a commit that assumes it's built this way. Patches: Patches (click to expand)From 422bfaabfc531d5b4da94fbc82beee12b0a59dcd Mon Sep 17 00:00:00 2001
From: Leijurv <leijurv@gmail.com>
Date: Thu, 8 Oct 2026 13:13:08 -0700
Subject: [PATCH 1/2] New upstream release 2.3.3
---
debian/changelog | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/debian/changelog b/debian/changelog
index 1c25ecd..c3a1859 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,9 @@
+zlib-ng (2.3.3-1~osm1) UNRELEASED; urgency=medium
+
+ * New upstream release.
+
+ -- Leijurv <leijurv@gmail.com> Thu, 08 Oct 2026 13:13:08 -0700
+
zlib-ng (2.2.2-1) UNRELEASED; urgency=medium
* Initial release. (Closes: #1002056)
--
2.43.0Then From 8cb33034925132fa769c87f4db9f454866b6ab43 Mon Sep 17 00:00:00 2001
From: Leijurv <leijurv@gmail.com>
Date: Thu, 8 Oct 2026 13:27:57 -0700
Subject: [PATCH 2/2] Ship the zlib compatible library alongside zlib1g
Install the ZLIB_COMPAT build as libz-ng-compat in the private directory
/usr/lib/<triplet>/zlib-ng instead of as libz1, which conflicted with and
replaced zlib1g for the whole system. Programs only use it when asked to,
e.g. with LD_LIBRARY_PATH. Drop libz-dev, since nothing should build
against the private library, and skip dh_makeshlibs for it, since it needs
neither an ldconfig trigger nor a shlibs file.
---
debian/changelog | 3 +
debian/control | 40 ++-----------
debian/libz-dev.docs | 4 --
debian/libz-dev.install | 6 --
debian/libz-ng-compat.install | 2 +
debian/libz1.install | 2 -
debian/libz1.symbols | 109 ----------------------------------
debian/rules | 18 +++++-
8 files changed, 28 insertions(+), 156 deletions(-)
delete mode 100644 debian/libz-dev.docs
delete mode 100644 debian/libz-dev.install
create mode 100644 debian/libz-ng-compat.install
delete mode 100644 debian/libz1.install
delete mode 100644 debian/libz1.symbols
diff --git a/debian/changelog b/debian/changelog
index c3a1859..ed3916e 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,6 +1,9 @@
zlib-ng (2.3.3-1~osm1) UNRELEASED; urgency=medium
* New upstream release.
+ * Ship the zlib compatible library as libz-ng-compat, in the private
+ directory /usr/lib/<triplet>/zlib-ng, so that it can be installed
+ alongside zlib1g instead of replacing it. Drop libz1 and libz-dev.
-- Leijurv <leijurv@gmail.com> Thu, 08 Oct 2026 13:13:08 -0700
diff --git a/debian/control b/debian/control
index 43dc4fa..87f3c56 100644
--- a/debian/control
+++ b/debian/control
@@ -47,48 +47,20 @@ Description: optimized zlib compression library - development files
This package contains the header files and the static library needed to
compile applications that use zlib-ng.
-Package: libz1
+Package: libz-ng-compat
Architecture: any
Multi-Arch: same
Depends:
${shlibs:Depends},
${misc:Depends},
-Conflicts:
- zlib1g,
-Replaces:
- zlib1g,
-Provides:
- zlib1g (= 1:1.3.1),
-Description: optimized API/ABI compatible zlib compression library - shared library
+Description: optimized API/ABI compatible zlib compression library - private library
zlib-ng is an optimized fork of the zlib library implementing the deflate
compression method found in gzip and PKZIP.
.
It includes and consolidates many optimizations found in alternative forks,
that have not been merged in the official zlib library.
.
- This package contains the ABI/API compatible shared library.
-
-Package: libz-dev
-Section: libdevel
-Architecture: any
-Multi-Arch: same
-Depends:
- libz1 (= ${binary:Version}),
- ${misc:Depends},
-Conflicts:
- zlib1g-dev,
-Replaces:
- zlib1g-dev,
-Provides:
- zlib1g-dev (= 1:1.3.1),
-Description: optimized API/ABI compatible zlib compression library - development files
- zlib-ng is an optimized fork of the zlib library implementing the deflate
- compression method found in gzip and PKZIP.
- .
- It includes and consolidates many optimizations found in alternative forks,
- that have not been merged in the official zlib library, while providing a
- cleaned up API, not compatible with zlib, although diverging mostly in the
- symbols being prefixed and some data type improvements.
- .
- This package contains the API/ABI compatible header files and the static
- library needed to compile applications that use zlib.
+ This package contains the ABI/API compatible shared library, installed in
+ the private directory /usr/lib/<triplet>/zlib-ng rather than replacing the
+ system zlib library. Programs only use it when asked to, for example with
+ LD_LIBRARY_PATH=/usr/lib/<triplet>/zlib-ng.
diff --git a/debian/libz-dev.docs b/debian/libz-dev.docs
deleted file mode 100644
index 45fde7e..0000000
--- a/debian/libz-dev.docs
+++ /dev/null
@@ -1,4 +0,0 @@
-doc/algorithm.txt
-doc/crc-doc.1.0.pdf
-doc/crc-pclmulqdq.pdf
-doc/txtvsbin.txt
diff --git a/debian/libz-dev.install b/debian/libz-dev.install
deleted file mode 100644
index 19c82a7..0000000
--- a/debian/libz-dev.install
+++ /dev/null
@@ -1,6 +0,0 @@
-usr/include/*
-usr/lib/*/cmake/ZLIB/ZLIB*.cmake
-usr/lib/*/cmake/ZLIB/zlib*.cmake
-usr/lib/*/libz.a
-usr/lib/*/libz.so
-usr/lib/*/pkgconfig/zlib.pc
diff --git a/debian/libz-ng-compat.install b/debian/libz-ng-compat.install
new file mode 100644
index 0000000..b463d06
--- /dev/null
+++ b/debian/libz-ng-compat.install
@@ -0,0 +1,2 @@
+usr/lib/*/zlib-ng/libz.so.1
+usr/lib/*/zlib-ng/libz.so.1.*
diff --git a/debian/libz1.install b/debian/libz1.install
deleted file mode 100644
index d94a6c4..0000000
--- a/debian/libz1.install
+++ /dev/null
@@ -1,2 +0,0 @@
-usr/lib/*/libz.so.1
-usr/lib/*/libz.so.1.*
diff --git a/debian/libz1.symbols b/debian/libz1.symbols
deleted file mode 100644
index 4114d3c..0000000
--- a/debian/libz1.symbols
+++ /dev/null
@@ -1,109 +0,0 @@
-libz.so.1 libz1 #MINVER# | zlib1g (>= 1:1.2.12)
-# For symbols added after libz 1.2.12, we use the following dependency
-# template instead, otherwise we cannot declare a proper dependency against
-# the expected legacy zlib library.
-| libz1 #MINVER#
-* Build-Depends-Packages: libz-dev, zlib1g-dev
- ZLIB_1.2.0.2@ZLIB_1.2.0.2 1.2.0.2
- ZLIB_1.2.0.8@ZLIB_1.2.0.8 1.2.0.8
- ZLIB_1.2.0@ZLIB_1.2.0 1.2.0
- ZLIB_1.2.12@ZLIB_1.2.12 1.2.12
- ZLIB_1.2.2.3@ZLIB_1.2.2.3 1.2.2.3
- ZLIB_1.2.2.4@ZLIB_1.2.2.4 1.2.2.4
- ZLIB_1.2.2@ZLIB_1.2.2 1.2.2
- ZLIB_1.2.3.3@ZLIB_1.2.3.3 1.2.3.3
- ZLIB_1.2.3.4@ZLIB_1.2.3.4 1.2.3.4
- ZLIB_1.2.3.5@ZLIB_1.2.3.5 1.2.3.5
- ZLIB_1.2.5.1@ZLIB_1.2.5.1 1.2.5.1
- ZLIB_1.2.5.2@ZLIB_1.2.5.2 1.2.5.2
- ZLIB_1.2.7.1@ZLIB_1.2.7.1 1.2.7.1
- ZLIB_1.2.9@ZLIB_1.2.9 1.2.9
- adler32@Base 1.2.0
- adler32_combine64@ZLIB_1.2.3.3 1.2.3.3
- adler32_combine@ZLIB_1.2.2 1.2.2
- adler32_z@ZLIB_1.2.9 1.2.9
- compress2@Base 1.2.0
- compress@Base 1.2.0
- compressBound@ZLIB_1.2.0 1.2.0
- crc32@Base 1.2.0
- crc32_combine64@ZLIB_1.2.3.3 1.2.3.3
- crc32_combine@ZLIB_1.2.2 1.2.2
- crc32_combine_gen64@ZLIB_1.2.12 1.2.12
- crc32_combine_gen@ZLIB_1.2.12 1.2.12
- crc32_combine_op@ZLIB_1.2.12 1.2.12
- crc32_z@ZLIB_1.2.9 1.2.9
- deflate@Base 1.2.0
- deflateBound@ZLIB_1.2.0 1.2.0
- deflateCopy@Base 1.2.0
- deflateEnd@Base 1.2.0
- deflateGetDictionary@ZLIB_1.2.9 1.2.9
- deflateInit2_@Base 1.2.0
- deflateInit_@Base 1.2.0
- deflateParams@Base 1.2.0
- deflatePending@ZLIB_1.2.5.1 1.2.5.1
- deflatePrime@ZLIB_1.2.0.8 1.2.0.8
- deflateReset@Base 1.2.0
- deflateResetKeep@ZLIB_1.2.5.2 1.2.5.2
- deflateSetDictionary@Base 1.2.0
- deflateSetHeader@ZLIB_1.2.2 1.2.2
- deflateTune@ZLIB_1.2.2.3 1.2.2.3
- get_crc_table@Base 1.2.0
- gzbuffer@ZLIB_1.2.3.5 1.2.3.5
- gzclearerr@ZLIB_1.2.0.2 1.2.0.2
- gzclose@Base 1.2.0
- gzclose_r@ZLIB_1.2.3.5 1.2.3.5
- gzclose_w@ZLIB_1.2.3.5 1.2.3.5
- gzdirect@ZLIB_1.2.2.3 1.2.2.3
- gzdopen@Base 1.2.0
- gzeof@Base 1.2.0
- gzerror@Base 1.2.0
- gzflush@Base 1.2.0
- gzfread@ZLIB_1.2.9 1.2.9
- gzfwrite@ZLIB_1.2.9 1.2.9
- gzgetc@Base 1.2.0
- gzgetc_@ZLIB_1.2.5.2 1.2.5.2
- gzgets@Base 1.2.0
- gzoffset64@ZLIB_1.2.3.5 1.2.3.5
- gzoffset@ZLIB_1.2.3.5 1.2.3.5
- gzopen64@ZLIB_1.2.3.3 1.2.3.3
- gzopen@Base 1.2.0
- gzprintf@Base 1.2.0
- gzputc@Base 1.2.0
- gzputs@Base 1.2.0
- gzread@Base 1.2.0
- gzrewind@Base 1.2.0
- gzseek64@ZLIB_1.2.3.3 1.2.3.3
- gzseek@Base 1.2.0
- gzsetparams@Base 1.2.0
- gztell64@ZLIB_1.2.3.3 1.2.3.3
- gztell@Base 1.2.0
- gzungetc@ZLIB_1.2.0.2 1.2.0.2
- gzvprintf@ZLIB_1.2.7.1 1.2.7.1
- gzwrite@Base 1.2.0
- inflate@Base 1.2.0
- inflateBack@ZLIB_1.2.0 1.2.0
- inflateBackEnd@ZLIB_1.2.0 1.2.0
- inflateBackInit_@ZLIB_1.2.0 1.2.0
- inflateCodesUsed@ZLIB_1.2.9 1.2.9
- inflateCopy@ZLIB_1.2.0 1.2.0
- inflateEnd@Base 1.2.0
- inflateGetDictionary@ZLIB_1.2.7.1 1.2.7.1
- inflateGetHeader@ZLIB_1.2.2 1.2.2
- inflateInit2_@Base 1.2.0
- inflateInit_@Base 1.2.0
- inflateMark@ZLIB_1.2.3.4 1.2.3.4
- inflatePrime@ZLIB_1.2.2.4 1.2.2.4
- inflateReset2@ZLIB_1.2.3.4 1.2.3.4
- inflateReset@Base 1.2.0
- inflateResetKeep@ZLIB_1.2.5.2 1.2.5.2
- inflateSetDictionary@Base 1.2.0
- inflateSync@Base 1.2.0
- inflateSyncPoint@Base 1.2.0
- inflateUndermine@ZLIB_1.2.3.3 1.2.3.3
- inflateValidate@ZLIB_1.2.9 1.2.9
- uncompress2@ZLIB_1.2.9 1.2.9
- uncompress@Base 1.2.0
- zError@Base 1.2.0
- z_vstring@Base 1.2.0
- zlibCompileFlags@ZLIB_1.2.0.2 1.2.0.2
- zlibVersion@Base 1.2.0
diff --git a/debian/rules b/debian/rules
index a689550..621dce8 100755
--- a/debian/rules
+++ b/debian/rules
@@ -3,6 +3,8 @@
# Output every command that modifies files on the build system.
#export DH_VERBOSE = 1
+include /usr/share/dpkg/architecture.mk
+
export DEB_BUILD_MAINT_OPTIONS = hardening=+all
export DEB_CFLAGS_MAINT_APPEND = -Wall -Wextra
@@ -17,8 +19,12 @@ BZ = $(D)/build-z
DNG = $(D)/destdir-ng
DZ = $(D)/destdir-z
+# The zlib compatible library goes in a private directory, so that it can be
+# installed alongside zlib1g and only used by programs that ask for it.
+ZLIBDIR = lib/$(DEB_HOST_MULTIARCH)/zlib-ng
+
PKGS_NG = -plibz-ng2 -plibz-ng-dev
-PKGS_Z = -plibz1 -plibz-dev
+PKGS_Z = -plibz-ng-compat
%:
dh $@ --buildsystem=cmake
@@ -34,6 +40,7 @@ override_dh_auto_configure:
dh_auto_configure --builddir=$(BZ) -- \
$(confflags) \
-DZLIB_COMPAT=ON \
+ -DCMAKE_INSTALL_LIBDIR=$(ZLIBDIR) \
# EOL
override_dh_auto_build:
@@ -43,6 +50,10 @@ override_dh_auto_build:
override_dh_auto_install:
dh_auto_install --builddir=$(BNG) --destdir=$(DNG)
dh_auto_install --builddir=$(BZ) --destdir=$(DZ)
+ # Nothing should build against the private library.
+ rm -r $(DZ)/usr/include $(DZ)/usr/$(ZLIBDIR)/libz.a \
+ $(DZ)/usr/$(ZLIBDIR)/libz.so $(DZ)/usr/$(ZLIBDIR)/pkgconfig \
+ $(DZ)/usr/$(ZLIBDIR)/cmake
override_dh_install:
dh_install --sourcedir=$(DNG) $(PKGS_NG)
@@ -56,5 +67,10 @@ override_dh_missing:
dh_missing --sourcedir=$(DNG) $(PKGS_NG)
dh_missing --sourcedir=$(DZ) $(PKGS_Z)
+# The private library is not in the loader's search path, so it needs neither
+# an ldconfig trigger nor a shlibs file that would offer it to other packages.
+override_dh_makeshlibs:
+ dh_makeshlibs -Nlibz-ng-compat
+
override_dh_installchangelogs:
dh_installchangelogs --no-trim
--
2.43.0 |
Member
|
Yes I realise it's a packaging choice, but that's the packaging we have available and I'm not competent to change it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
zlib-ng is a modern fork of zlib. It's reputable, for example it's used in Python and it's the system zlib on Fedora.
I've measured it as -5% mapnik CPU on x86 locally, -10.4% mapnik CPU on ARM (
m7gd.2xlargesimilar to palulukon). The tile filesize varies but tentatively it seems to perhaps 2% smaller than before (based on a sample of ~200 metatiles (~13,000 PNGs), z10-z12 random across the world, z13-z19 sampled from request logs. 84% of the tiles got smaller. median change was -2.6%. range was -12.1% to +1.0%).This PR adds it as a second
LD_PRELOAD(alongsidetcmalloc) for renderd. It's not packaged for Debian anywhere it appears. But it's pretty small and takes <1 min to build, so this recipe builds it from pinned source. Of course, an alternative would be to publish it on apt.osm.org.