diff --git a/README.rst b/README.rst index 43f1b4b..2340975 100644 --- a/README.rst +++ b/README.rst @@ -11,6 +11,14 @@ libasdf :target: https://libasdf.readthedocs.io/en/latest/ :alt: Documentation Status +.. image:: https://anaconda.org/conda-forge/libasdf/badges/version.svg + :target: https://anaconda.org/channels/conda-forge/packages/libasdf + :alt: Conda Package + +.. image:: https://img.shields.io/badge/dynamic/regex?url=https%3A%2F%2Fraw.githubusercontent.com%2Fasdf-format%2Fhomebrew-tap%2Fmain%2FFormula%2Flibasdf.rb&search=releases%2Fdownload%2F%28%5B0-9%5D%5B0-9a-zA-Z.%5D%2A%29%2F&replace=%241&label=Homebrew%20Tap&color=orange + :target: https://github.com/asdf-format/homebrew-tap + :alt: Homebrew Tap + .. _end-badges: A C library for reading (and eventually writing) `ASDF @@ -20,21 +28,160 @@ A C library for reading (and eventually writing) `ASDF Introduction ============ -libasdf is largely a wrapper around `libfyaml `__ -but with an understanding of the structure of ASDF files, with the capability to read and -extract binary block data, as well as typed getters for metadata in the ASDF tree. +libasdf is largely a wrapper around `libfyaml +`__ but with an understanding of the +structure of ASDF files, with the capability to read and extract binary block +data, as well as typed getters for metadata in the ASDF tree. + +It also features an extension mechanism for reading ASDF schemas, including the +core schemas such as ``core/ndarray-`` into C-native datastructures. + +libasdf additionally installs a companion command-line tool, ``asdf``: a +wrapper around the library providing utilities for inspecting and extracting +data from ASDF files. Its capabilities are currently modest but will be +expanded in the future; see the `command-line tool documentation +`__ for details. + + +.. _installation: + +Installation +============ + +libasdf is packaged for conda and Homebrew. Both install the shared library, +the public headers, and the ``asdf`` command-line tool. It can also be built +from source. If you would rather see what using the library looks like first, +skip ahead to `Getting started`_. + +conda +----- + +libasdf is available from `conda-forge +`__:: + + conda install -c conda-forge libasdf + +Packages are built for Linux and macOS; there is currently no Windows build. + +Homebrew +-------- + +libasdf is distributed through the asdf-format `tap +`__:: + + brew tap asdf-format/tap + brew install libasdf + +or, equivalently, in a single step:: + + brew install asdf-format/tap/libasdf + +Bottles (pre-built binaries) are provided for Apple Silicon macOS and x86-64 +Linux. On Intel macOS the formula builds from source, which takes +considerably longer. + +.. note:: + + The formula declares ``conflicts_with "asdf"``: libasdf's command-line tool + and the `asdf `__ version manager both install a + binary named ``asdf``, so Homebrew will not link the two at once. + +From source +----------- + +libasdf's build system is built with CMake. To build from a release tarball +or from a git checkout, you'll need the following software installed on your +system: + +Requirements +^^^^^^^^^^^^ + +- **CMake** (for generating the build system) +- **C compiler** (e.g., ``gcc`` or ``clang``) +- **Make** (e.g., ``GNU make``) +- **pkg-config** +- **libfyaml** + - Version >=0.8 is tested to work +- **zlib**, **bzip2**, and **lz4** (for compression support) +- **libmd** (required for MD5 checksum support) +- **libstatgrab** (optional, for system resource heuristics) +- **argp** (this is a feature of glibc, but if compiling with a different libc you need a + standalone version of this; also it is only needed if building the command-line tool) + +On **Debian/Ubuntu**:: + + sudo apt install build-essential pkg-config libfyaml-dev \ + zlib1g-dev libbz2-dev liblz4-dev libstatgrab-dev libmd-dev + +On **Fedora**:: + + sudo dnf install gcc make pkgconf libfyaml-devel \ + zlib-devel bzip2-devel lz4-devel libstatgrab-devel libmd-devel + +On **macOS** (with Homebrew):: + + brew install pkg-config libfyaml argp-standalone \ + zlib bzip2 lz4 libstatgrab libmd + +Building +^^^^^^^^ + +Clone the repository and build the project as follows (if you are building +from a release tarball, unpack it and skip the ``git clone``):: -It also features an extension mechanism (still nascent) for reading ASDF schemas, including -the core schemas such as ``core/ndarray-`` into C-native datastructures. + git clone https://github.com/asdf-format/libasdf.git + cd libasdf + mkdir build + cd build + cmake .. \ + -D ENABLE_TESTING=[YES/NO] \ + -D ENABLE_TESTING_SHELL=[YES/NO] \ + -D ENABLE_TOOL=[YES/NO] \ + -D ENABLE_ASAN=[YES/NO] \ + -D FYAML_NO_PKGCONFIG=[YES/NO] \ + # If YES \ + -D FYAML_LIBDIR=[path/lib] \ + -D FYAML_INCLUDEDIR=[path/include] \ + -D ARGP_NO_PKGCONFIG=[YES/NO] \ + # If YES \ + -D ARGP_LIBDIR=[path/lib] \ + -D ARGP_INCLUDEDIR=[path/include] + make + sudo make install # Optional, installs the binary system-wide + +If doing a system install, as usual it's recommended to install to +``/usr/local`` by providing ``-DCMAKE_INSTALL_PREFIX=/usr/local`` when running +``cmake``. Or, if you have a ``${HOME}/.local`` you can set the prefix there, +etc. + +Logging +^^^^^^^ + +libasdf can emit diagnostic log messages, controlled by the following options: + +- ``-D ENABLE_LOG=[YES/NO]`` -- compile libasdf's internal log statements into + the library (default ``YES``). When ``NO`` they compile to nothing. +- ``-D ENABLE_LOG_COLOR=[YES/NO]`` -- colorize log output (default ``YES``). +- ``-D LOG_DEFAULT=LEVEL`` -- the default runtime log level, used when none is + set explicitly; one of ``TRACE``, ``DEBUG``, ``INFO``, ``WARN`` (the + default), ``ERROR``, ``FATAL``, or ``NONE``. +- ``-D LOG_MIN=LEVEL`` -- the compile-time minimum level; messages below it are + compiled out entirely (default ``TRACE``). -libasdf additionally installs a companion command-line tool, ``asdf``: a wrapper around the -library providing utilities for inspecting and extracting data from ASDF files. Its -capabilities are currently modest but will be expanded in the future; see the -`command-line tool documentation `__ +At runtime the default level can also be overridden through the +``ASDF_LOG_LEVEL`` environment variable. See the +`logging documentation `__ for details. -Getting Started ---------------- +Notes +^^^^^ + +- Run ``make clean`` to clean build artifacts. +- Run ``ctest --output-on-failure`` to execute unit tests + + +Getting started +=============== To open an ASDF file with libasdf the simplest way is to use the ``asdf_open`` function. This returns an ``asdf_file_t *`` which is your main interface to the ASDF file. @@ -126,7 +273,7 @@ to a new file: return 0; } -With libasdf installed on your system (see `Development`_) you can compile +With libasdf installed on your system (see `Installation`_) you can compile and run this test like: .. code:: console @@ -244,105 +391,29 @@ Additional examples can be found in the `libasdf documentation `__. -.. _development: - -Development -=========== - -Building from git ------------------ - -libasdf's build system is built with CMake. To build this project -from source, you'll need the following software installed on your system: - -Requirements -^^^^^^^^^^^^ - -To build this project from source, you'll need the following software installed -on your system: - -- **CMake** (for generating the build system) -- **C compiler** (e.g., ``gcc`` or ``clang``) -- **Make** (e.g., ``GNU make``) -- **pkg-config** -- **libfyaml** - - Version >=0.8 is tested to work -- **zlib**, **bzip2**, and **lz4** (for compression support) -- **libmd** (required for MD5 checksum support) -- **libstatgrab** (optional, for system resource heuristics) -- **argp** (this is a feature of glibc, but if compiling with a different libc you need a - standalone version of this; also it is only needed if building the command-line tool) - -On **Debian/Ubuntu**:: - - sudo apt install build-essential pkg-config libfyaml-dev \ - zlib1g-dev libbz2-dev liblz4-dev libstatgrab-dev libmd-dev - -On **Fedora**:: - - sudo dnf install gcc make pkgconf libfyaml-devel \ - zlib-devel bzip2-devel lz4-devel libstatgrab-devel libmd-devel - -On **macOS** (with Homebrew):: - - brew install pkg-config libfyaml argp-standalone \ - zlib bzip2 lz4 libstatgrab libmd - -Building -^^^^^^^^ - -Clone the repository and build the project as follows:: +Versioning and stability +======================== - git clone https://github.com/asdf-format/libasdf.git - cd libasdf - mkdir build - cd build - cmake .. \ - -D ENABLE_TESTING=[YES/NO] \ - -D ENABLE_TESTING_SHELL=[YES/NO] \ - -D ENABLE_TOOL=[YES/NO] \ - -D ENABLE_ASAN=[YES/NO] \ - -D FYAML_NO_PKGCONFIG=[YES/NO] \ - # If YES \ - -D FYAML_LIBDIR=[path/lib] \ - -D FYAML_INCLUDEDIR=[path/include] \ - -D ARGP_NO_PKGCONFIG=[YES/NO] \ - # If YES \ - -D ARGP_LIBDIR=[path/lib] \ - -D ARGP_INCLUDEDIR=[path/include] - make - sudo make install # Optional, installs the binary system-wide +libasdf follows `semantic versioning `__, and is +currently in the ``0.x`` series. -If doing a system install, as usual it's recommended to install to -``/usr/local`` by providing ``-DCMAKE_INSTALL_PREFIX=/usr/local`` when running -``cmake``. Or, if you have a ``${HOME}/.local`` you can set the prefix there, -etc. +**API stability.** Source compatibility is maintained between ``0.x`` +releases: code that compiles against one release compiles against the next. -Logging -^^^^^^^ +**ABI stability.** The shared library carries a SONAME (``libasdf.so.0``) +derived from an interface version that moves independently of the package +version. It changes only when an interface is removed or altered, never when +interfaces are merely added, so a binary linked against one release keeps +working with later releases that only add to the API. This is guaranteed for +64-bit targets; 32-bit ABI compatibility is not promised. See `shared library +versioning +`__ +for how this is managed. -libasdf can emit diagnostic log messages, controlled by the following options: +**Road to 1.0.** Version 1.0 aims for at least complete *read* support for +every feature of the ASDF standard and the core schemas. libasdf already +supports reading *most* ASDF files one is likely to encounter in the wild. -- ``-D ENABLE_LOG=[YES/NO]`` -- compile libasdf's internal log statements into - the library (default ``YES``). When ``NO`` they compile to nothing. -- ``-D ENABLE_LOG_COLOR=[YES/NO]`` -- colorize log output (default ``YES``). -- ``-D LOG_DEFAULT=LEVEL`` -- the default runtime log level, used when none is - set explicitly; one of ``TRACE``, ``DEBUG``, ``INFO``, ``WARN`` (the - default), ``ERROR``, ``FATAL``, or ``NONE``. -- ``-D LOG_MIN=LEVEL`` -- the compile-time minimum level; messages below it are - compiled out entirely (default ``TRACE``). - -At runtime the default level can also be overridden through the -``ASDF_LOG_LEVEL`` environment variable. See the -`logging documentation `__ -for details. - -Notes -^^^^^ - -- Run ``make clean`` to clean build artifacts. -- Run ``make project_source`` to generate a source archive with CPack -- Run ``ctest --output-on-failure`` to execute unit tests Official Extensions =================== diff --git a/changes/+symbol-leakage-linker-syms.misc b/changes/+symbol-leakage-linker-syms.misc new file mode 100644 index 0000000..b9de0ab --- /dev/null +++ b/changes/+symbol-leakage-linker-syms.misc @@ -0,0 +1,6 @@ +``tests/test-symbol-leakage.sh`` no longer reports linker- and CRT-generated +symbols (``_init``, ``_fini``, ``__bss_start``, ``_edata``, ``_end`` and +friends) as leaked. + +The workarounds for these can be hopefully now be removed from downstream +packages (i.e. the conda-forge and homebrew receipes). diff --git a/changes/267.doc b/changes/267.doc new file mode 100644 index 0000000..c311b31 --- /dev/null +++ b/changes/267.doc @@ -0,0 +1,5 @@ +Replaced the README's ``Development`` section with a more prominent +``Installation`` section covering conda-forge and Homebrew as well as building +from source, and added a brief ``Versioning and stability`` section describing +the version scheme, the API and ABI compatibility guarantees, and what is +required for 1.0. diff --git a/docs/development.rst b/docs/development.rst index 547b109..fb3f843 100644 --- a/docs/development.rst +++ b/docs/development.rst @@ -5,7 +5,7 @@ Development resources This page covers building libasdf from a git checkout, the conventions the project follows, and how releases are made. If you only want to *use* the -library, the build instructions in the :ref:`README ` *should* +library, the build instructions in the :ref:`README ` *should* be sufficient. @@ -586,6 +586,10 @@ Cutting the release #. Review the draft release on GitHub and publish it. +#. Update the downstream packages; see `Downstream packaging`_ below. Neither + happens automatically, and both consume the release tarball, so nothing can + be done until the release is actually published. + The release workflow refuses to run unless the tag matches the version recorded in ``.bumpver.toml`` and ``configure.ac``, and unless the top section of ``CHANGES.rst`` is the one for that version, so a mistagged or half-finished @@ -600,3 +604,78 @@ itself, it can be triggered by hand, e.g. with the GitHub CLI: $ gh workflow run release.yml -f tag= Re-running updates the existing draft rather than failing. + + +.. _downstream-packaging: + +Downstream packaging +-------------------- + +libasdf is published through two channels that live in their own repositories, +and each needs a version bump of its own after a release. Both fetch:: + + https://github.com/asdf-format/libasdf/releases/download//libasdf-.tar.gz + +so that asset name is effectively part of the release contract: renaming it, or +attaching the tarball under a different name, breaks both packagers at once. + +conda-forge +^^^^^^^^^^^ + +The feedstock is `conda-forge/libasdf-feedstock +`__. + +Usually there is nothing to do: conda-forge's autotick bot notices the new +GitHub release, opens a pull request updating the version and checksum, and a +maintainer reviews and merges it once CI is green. It can take a few hours to +appear. + +To do it by hand, in ``recipe/recipe.yaml``: set ``context.version``, replace +``source.sha256`` with the checksum of the new tarball, and reset +``build.number`` to ``0``. + +One thing about that recipe is worth knowing: its ``run_exports`` pin uses +``upper_bound='x.x'``, which injects ``libasdf >=0.1.0.0.0.0,<0.2.0a0`` into +the run requirements of anything built against libasdf ``0.1.x``. That is +deliberately stricter than libasdf's own ABI guarantee, which lets a binary +keep working across any release that only adds interfaces. + +It is a packaging policy rather than a restatement of the C-level promise, and +no pin can express that promise exactly: the pin is written in version-number +space, while the ABI boundary lives in soname space, and +:ref:`the two are decoupled on purpose `. A looser pin would +be wrong in the dangerous direction the first time ``age`` resets and the +soname moves mid-series; this one is wrong only in the conservative direction, +where the cost is an unnecessary rebuild of everything downstream. While we +are in ``0.x`` and reserve the right to break ABI at a minor release, that is +the right trade. Once the SONAME and the version number have a settled +relationship at 1.0, ``upper_bound='x'``--the rattler-build default, which +permits any release sharing a major version--becomes a safe place to pin it. + + +Homebrew +^^^^^^^^ + +The tap is `asdf-format/homebrew-tap +`__, providing +``Formula/libasdf.rb``. + +#. Open a pull request bumping ``url`` and ``sha256`` in the formula. + ``brew bump-formula-pr`` will do this for you. + +#. The tap's ``tests.yml`` workflow builds bottles for the PR. Wait for it. + +#. Label the pull request ``pr-pull``. That triggers ``publish.yml``, which + runs ``brew pr-pull`` to fetch the built bottles, commit the bottle block, + and push to ``main``. + +Bottles are built for Apple Silicon macOS and x86-64 Linux. Intel macOS is not +bottled--Homebrew has moved it to Tier 3--so users there build from source. + +The formula's Linux-only linker version script is the workaround described +under conda-forge above, and can go the same way. + +Note also that the formula declares ``conflicts_with "asdf"``, since our +command-line tool collides with the ``asdf`` version manager. Keep that in +place: dropping it would let Homebrew link both and leave whichever came second +shadowed. diff --git a/tests/test-symbol-leakage.sh b/tests/test-symbol-leakage.sh index 68dd72b..f9ff9d8 100755 --- a/tests/test-symbol-leakage.sh +++ b/tests/test-symbol-leakage.sh @@ -53,16 +53,30 @@ if [ -z "${symbols}" ]; then exit 77 fi -# Ignore decorations that are not part of the symbol name itself: +# Ignore decorations and symbols that are not libasdf's to begin with: +# # - ASan emits various aliases for globals depending on the compiler version: # - __odr_asan. # - __odr_asan_gen_ # - __start_asan_globals, __stop_asan_globals +# +# - Linker- and CRT-generated symbols (the section boundary markers the GNU +# linker provides, plus _init/_fini from crti.o/crtn.o). These are not +# emitted by libasdf at all, and every shared object on the platform may +# define its own, so they cannot collide in the way this test exists to +# catch. Current binutils and glibc keep them hidden, which is why this +# never fires locally; the older toolchains used by conda-forge and by +# Homebrew on Linux export them with default visibility. Both carried +# linker version-script workarounds to get past this test before the +# exception below was added: conda-forge for _init/_fini, Homebrew for +# __bss_start/_edata/_end. +# # - Mach-O prefixes every C symbol with an underscore, hence the optional -# leading _ in the pattern below +# leading _ in the asdf_ pattern below leaked=$(echo "${symbols}" \ - | sed -e 's/^__odr_asan.*//' \ + | sed -e '/^__odr_asan/d' \ | grep -vE '^__(start|stop)_asan_globals' \ + | grep -vxE '_init|_fini|_etext|_edata|_end|__bss_start|__data_start|data_start' \ | grep -vE '^_?(asdf_|ASDF_|libasdf_)' \ | sort -u)