Development resources

This page covers building libasdf from a git checkout, the conventions the project follows, and how releases are made. If you only want to use the library, the build instructions in the README should be sufficient.

The two build systems

libasdf ships two parallel build systems, and both are kept working at all times:

Autotools (configure.ac / Makefile.am) is the primary build system for development, and the one used to produce release tarballs. make dist and make distcheck give a canonical, self-contained source archive that builds without any of the tooling used to generate it–no autotools, no Sphinx, no git checkout. make distcheck also verifies that the tarball builds out of tree, that everything needed was actually distributed, and that it uninstalls cleanly, which is what makes it trustworthy as a release artifact.

CMake (CMakeLists.txt) exists for consumers, not for producing releases. A great deal of C and C++ tooling assumes CMake: it drops into projects that use FetchContent or find_package, into meta-build systems and package managers, and into IDEs that understand CMake projects natively. Requiring downstream users to deal with autotools in those settings would be a real obstacle.

The two are not independent: make distcheck runs the CMake build from the distribution tarball (the distcheck-cmake target in the top-level Makefile.am), so a change that breaks CMake, or that forgets to distribute a file CMake needs, fails the autotools release check. Both are also built and tested in CI, by the Build and CMake Build workflows respectively.

The practical consequences:

  • When you add a source file, add it to both build systems. Both list their sources explicitly–src_files in the top-level Makefile.am and libasdf_sources in src/CMakeLists.txt.

  • When you add a public header, add it to both install lists– include/Makefile.am and include/CMakeLists.txt. Drift between these two breaks the installed library without breaking the build tree, so it tends to be noticed late.

  • When you add a documentation page, add it to EXTRA_DIST in docs/Makefile.am, or it will be missing from the release tarball and the documentation build inside make distcheck will fail.

Building with autotools

A git checkout has no configure script; generate it first with autogen.sh (a wrapper script around autoreconf --install). This is normally only needed once, or again after editing configure.ac or any Makefile.am, though the generated makefiles normally re-run the necessary steps by themselves:

$ git clone --recurse-submodules https://github.com/asdf-format/libasdf.git
$ cd libasdf
$ ./autogen.sh
$ ./configure
$ make
$ make check

The asdf-standard submodule provides reference files used by the test suite; if you cloned without --recurse-submodules, run git submodule update --init before building (this is also run automatically by the build system, but sometimes it can be flaky).

Useful configure options:

--enable-debug

Debug build: CFLAGS=-g -O0, and DEBUG is defined.

--with-asan

Build with AddressSanitizer.

--with-ubsan

Build with UndefinedBehaviorSanitizer.

--disable-tool

Skip building the asdf command-line tool. Useful if argp is unavailable.

--enable-docs

Build the Sphinx documentation. The default is auto: docs are built if Sphinx and its extensions are found, and silently skipped otherwise.

--enable-valgrind

Enable the check-valgrind targets.

--enable-logging, --enable-log-color, --with-log-default=LEVEL, --with-log-min=LEVEL

Compile-time logging configuration; see Logging.

Autotools fully supports out-of-tree builds, and keeping several configurations side by side is the recommended way to work, since a change should pass under more than one of them:

$ mkdir build-asan && cd build-asan
$ ../configure --with-asan
$ make && make check

Anything that touches library code should pass make check in both an AddressSanitizer build and a debug build before being committed.

Building with CMake

The CMake build is documented in the README and is reproduced here for convenience:

$ mkdir build && cd build
$ cmake .. \
      -D CMAKE_BUILD_TYPE=RelWithDebInfo \
      -D ENABLE_TESTING=YES \
      -D ENABLE_ASAN=[YES/NO] \
      -D ENABLE_TOOL=[YES/NO]
$ make
$ ctest --output-on-failure

Other options of note:

-D ENABLE_TESTING_DOCS=YES

Additionally build and run the example programs embedded in the documentation (see Documentation examples).

-D ENABLE_TESTING_ALL=YES

Enable every test target, including the shell-based integration tests. This is what CI uses.

-D ENABLE_DOCS=YES

Build the Sphinx documentation (make docs).

-D FYAML_NO_PKGCONFIG=YES, -D ARGP_NO_PKGCONFIG=YES

Locate libfyaml or argp by explicit path rather than pkg-config; pair with -D FYAML_LIBDIR=/-D FYAML_INCLUDEDIR= and the ARGP_ equivalents.

make package_source and make package produce CPack archives. Note that these are not what is published for a release–the release tarball comes from make dist under autotools.

Note

On systems with a libasdf already installed under, say, ~/.local/lib, be aware that CMake links test binaries with DT_RUNPATH rather than DT_RPATH, and the dynamic loader searches LD_LIBRARY_PATH before DT_RUNPATH. An installed copy can therefore shadow the freshly built one and cause confusing test failures. Unset LD_LIBRARY_PATH before running ctest if you hit this.

Running the tests

The test suite uses the µnit framework, with some custom wrappers around it (helper macros) defined in tests/munit.h.

Test binaries are named test-<name>.unit and live in the tests/ directory of the build tree. Tests that need to reach internals link against libasdf_static.la, a static convenience archive of the whole library, which makes ASDF_LOCAL (hidden-visibility) symbols reachable from the test binary. A couple of the narrower ones instead compile the single source under test directly, so that it can be exercised in isolation.

From a build directory:

$ make check                      # everything
$ tests/test-block.unit           # one binary directly
$ tests/test-ndarray.unit --help  # munit options

munit accepts a test path to run a single case, and other useful flags:

$ tests/test-ndarray.unit /test_ndarray/test_read_all
$ tests/test-ndarray.unit --seed 0x1234  # reproduce a specific ordering
$ tests/test-ndarray.unit --no-fork      # keep the debugger attached

In particular, the --no-fork option is always enabled by default for debug builds, as it’s extremely helpful if you want to run the tests under gdb (or your debugger of choice).

Shell-based integration tests (tests/test-events.sh, tests/test-info.sh and friends) exercise the asdf command-line tool against golden output files under tests/fixtures/. When a deliberate change alters that output, regenerate the golden files with make update-fixtures from the build directory, and check the diff carefully before committing it.

Some test data comes from the asdf-standard submodule’s reference_files/ directory.

Two more targets, both requiring the corresponding configure option:

  • make check-valgrind (--enable-valgrind)

  • make check-code-coverage (--enable-code-coverage)

Documentation examples

The example programs in README.rst and under docs/usage/ are compiled and executed as part of the test suite, so they cannot drift away from the API.

A code block opts in with the :test: option, naming the test, and may declare an input file with :fixture::

.. code:: c
   :test: test-open-close-file
   :fixture: cube.asdf

   #include <asdf.h>
   ...

tests/scripts/extract_doc_examples.py pulls each marked block out into a .c file under tests/doc_examples/, compiles it against the freshly built library, and runs it with the resolved fixture path as its first argument. A block with no :fixture: is run with no arguments; a fixture of temp or temp:<name> resolves to a throwaway output path instead of an input file.

The set of files scanned is listed explicitly, as DOC_EXAMPLE_FILES in tests/Makefile.am and DOC_FILES in tests/CMakeLists.txt; a new documentation page containing examples must be added to both.

The examples are run for their exit status, not compared against the output quoted in the documentation. When you change one, re-run it and paste its real output back into the surrounding prose:

$ make check
$ ./tests/doc_examples/test-open-close-file tests/fixtures/cube.asdf

Under autotools this needs Python 3, and is skipped if none is found; under CMake it is gated on -D ENABLE_TESTING_DOCS=YES.

Code style

Formatting is enforced by clang-format; the rules live in .clang-format. Run:

$ make format

from a build directory before committing. It rewrites all library sources and headers in place.

There is also a make tidy target that runs clang-tidy, though its output is not currently clean and it is not enforced.

To have formatting applied automatically, it is recommended to install the pre-commit hook shipped in .pre-commit-config.yaml:

$ pip install pre-commit      # or: pipx install pre-commit
$ pre-commit install

That registers a git hook which runs clang-format -i over any staged files under src/ or include/.

Note

The hook uses language: system, so it runs your clang-format; a different major version may format differently from the rest of the tree. To check the whole tree at any time:

pre-commit run --all-files

Documentation

The documentation is built with Sphinx. API reference pages are generated from the doc comments in the public headers under include/asdf/ using Hawkmoth, which extracts /** ... */ comments and feeds them to Sphinx as reStructuredText. Because of that, doc comments in public headers are written in reST, using field lists (:param foo:, :return:) rather than a Doxygen-style syntax.

Build them with:

$ ./configure --enable-docs
$ make docs

The rendered output lands in docs/_build/html under the build directory. CI builds the docs with -W, so warnings are errors; if you add a page, make sure it is referenced from a toctree and that every cross-reference resolves.

Two things about hawkmoth are worth knowing before writing header comments:

  • An undocumented declaration is dropped entirely, and takes its members with it. A struct whose fields all carry /** ... */ comments will still be absent from the rendered API unless the struct itself has one. If a type you expect is missing from the output, this is almost always why.

  • A header that clang cannot parse loses its declarations silently. The parse needs the same include path the compiler gets; when a declaration vanishes from the output for no apparent reason, check that first.

Because conf.py sets nitpicky = True and uses c:expr as the default role, any bare identifier written in single backticks is looked up in the C domain. Standard C names have no inventory to resolve against and are listed in nitpick_ignore; for anything else that is not a real API symbol–a file name, a schema name, a field mentioned in passing–use

``double backticks``

instead.

The asdf(1) man page is generated from docs/usage/cli.rst but is committed to the repository (as docs/man/asdf.1), so that building or installing from a release tarball does not require Sphinx. After changing usage/cli.rst, regenerate and commit it:

$ make man-page

Changelog entries

The changelog is assembled by towncrier from individual news fragments in the changes/ directory. Each fragment is a small reStructuredText file named for the issue or pull request it relates to, with the category as its extension:

changes/123.feature
changes/456.bugfix

The available categories are general, feature, bugfix, doc, removal and misc. For a change with no associated issue number, use a descriptive name prefixed with +, for example changes/+cmake-build.misc.

Write the entry for the reader of the release notes, not for the reviewer of the diff.

Making a release

Despite not being a Python package, version numbers follow PEP 440 (because the author likes it) and are managed by bumpver, configured in .bumpver.toml.

A single bumpver invocation rewrites the version everywhere it appears (configure.ac, CMakeLists.txt, docs/conf.py, towncrier.toml, and .bumpver.toml itself), builds the changelog, commits, tags and pushes.

Tags are the bare version string, with no v prefix–0.1.0a2, not v0.1.0a2.

Note

CMake does not support PEP 440-style version tags (“a1”, “rc0”, etc.), so the full version is written in CMakeLists.txt as a PACKAGE_VERSION variable (mirroring autoconf); the CMake standard variable PROJECT_VERSION is also managed by bumpver, but only contains the MAJOR.MINOR.PATCH portion of the version.

Signing the tag

Release tags should be signed with your GPG key (bumpver does this via git tag --annotate). That command does honour git’s tag.gpgSign setting, so configure it once and release tags are signed automatically:

$ git config --local tag.gpgSign true

This should be configured before making a release; you can find more information about generating a GPG key and registering it with GitHub at Telling Git about your signing key.

Cutting the release

  1. Make sure main is up to date, the working tree is clean, and CI is passing.

  2. Check that every merged change has a news fragment in changes/, and preview the assembled changelog:

    $ towncrier build --draft --version <new version>

  3. Run bumpver with the appropriate increment. Use --dry first to see exactly what will be rewritten:

    $ bumpver update --tag-num --dry     # 0.1.0a2 -> 0.1.0a3
$ bumpver update --tag-num

    Other useful increments are --patch, --minor, --major, and --tag beta / --tag final to move along the pre-release sequence.

    This runs scripts/changelog.sh as a pre-commit hook, which consumes every file in changes/ and rewrites CHANGES.rst, then commits, tags and pushes.

  4. The rest is automated. Pushing the tag triggers the Build workflow; once it succeeds on that tag the Release workflow creates a draft GitHub release, with the release notes converted from the new CHANGES.rst section and the make dist tarball attached.

  5. Review the draft release on GitHub and publish it.

The release workflow refuses to run unless the tag matches the version recorded in .bumpver.toml and configure.ac, and unless the top section of CHANGES.rst is the one for that version, so a mistagged or half-finished release fails loudly rather than shipping.

If the Release workflow needs to be re-run against a tag that has already built successfully, for instance after fixing something in the workflow itself, it can be triggered by hand, e.g. with the GitHub CLI:

$ gh workflow run release.yml -f tag=<version>

Re-running updates the existing draft rather than failing.