In my previous two posts, I’ve gone over [what package types python has](https://bernat.tech/posts/pep-517-and-python-packaging/), and [how the package building works](https://bernat.tech/posts/pep-517-518/), especially with the introduction of the PEP-517/518. Although the changes were meant to make things more robust, we did run into a few issues while implementing and releasing them. The following are a few of those, serving as lessons learned for all of us and presenting some interesting problems to solve in the future.
Looking at the changes of PEP-517 and PEP-518, build backends (aka setuptools, flit) had very little to do, only to expose their functionality via a Python module. Most heavy work falls on the build frontend, which now needs to generate the isolated Python and then invoke the build backends in a new way. Build frontends today are pip or poetry (and tox for developers).
These projects are maintained by the community, by a handful of active developers, in their free time. They are not getting paid for it, and they need to be careful to consider the myriad ways these tools are used. Given this, it’s not much of a surprise it took almost two years after the PEP acceptance to come out with a first implementation. Planning, testing, and implementation have been going on for over a year in the background.
Against all the preparations, the first release did break a few packages, mostly where people performed operations that caught the maintainers by surprise. Let’s understand a few of these examples and how they got addressed.
PEP-518 Link to heading
The PEP introduces the TOML file format.
A format specially created to be easy to
read/write configurations. While packaging configuration is exposed under the build-system section, other tools are
free to put their configuration under the tool:name section if they own the PyPi namespace for the name. Various tools
started to take advantage of this right away (such as towncrier
,
black
, etc.).
When pip 18.0 (released 2018 July 22
) added support for PEP-518 packages
using the pyproject.toml broke, as PEP-518 mandated that all packages having the pyproject.toml must specify the
build-backend section. But packages beforehand used it only as a configuration file for these other projects since
they didn’t specify it pre-emptively; when pip ran into these files, it raised errors complaining of invalid
pyproject.toml files.
PEP-517 Link to heading
The pip wheel cache issue. Link to heading
The way pip installs in a PEP-517 world is first to generate a wheel and then extract that. To be in PEP-517 world one
must specify the build-backend key. Otherwise, all frontends per specification need to fallback to using the
setup.py commands.
When pip builds wheels, it does it by default via a caching system. This is a speed-up mechanism so that if multiple virtual environments need the same wheel, we don’t keep rebuilding it but instead re-use it. The PEP-517 wheel build operation also takes advantage of this.
This becomes troublesome, though, when you disable the cache. Now there’s no target folder where to build the wheel. So the build itself fails see the attached issue. The problem manifested early but en masse, as most CI systems run with this option turned on. A day later pip 19.0.1 fixed this.
pyproject.toml not being included into setuptools
Link to heading
There’s more work on the build backend than exposing their API as described in PEP-517. The backend also needs to ensure
that pyproject.toml is attached to the built source package. Otherwise, the build backend on the user machine will not
be able to use it. setuptools 1650
will fix this for setuptools. One can
include pyproject.toml by specifying it inside MANIFEST.in on earlier versions.
Importing the built package from within setup.py
Link to heading
Another unexpected issue was when a package was importing from within setup.py. The version of the package by
convention is exposed both as metadata for the package (in case of setuptools inside the setup.py as the version
argument to the setup function), but also under the __version__ variable at the root of the package. One could
specify the content of the variable in both places, but then it becomes troublesome to keep it in sync.
As a workaround, many packages started putting it inside a version.py at the root of the package, and then import it
as from mypy.version import __version__ as version from both the setup.py and the package root. This worked because
when someone calls a python script, the current working directory is automatically attached to the sys.path (so you
can import stuff exposed underneath it).
Adding the current working directory was never mandated. It was a side-effect of calling the build via
python setup.py sdist. Because this behavior is a side-effect (not a guarantee), all projects that import from their
setup.py should add the scripts folder to the sys.path at the start of the build.
Whether importing the built package during the packaging (when it’s not yet built/distributed) is a good idea remains
debatable (though the Python Packaging group leans towards “no”). When setuptools exposed its interface via
setuptools.build_meta, it chose not to add the current working directory to the system path.
The PEP never mandates for the frontend to do this addition, as most build backend (declarative by nature) will never
need this. Such functionality was deemed to be under the responsibility of the front end. setuptools maintainers think
if users want this functionality, they should be explicit in their setup.py and prepend the respective path to the
sys.path.
To simplify its code base, pip decided to opt all people having a pyproject.toml into the setuptools backend. With
this issue even packages that hadn’t opted in to PEP-517 started to break. To fix this, setuptools added a new build
backend (setuptools.build_meta:__legacy__) designed to be used by frontends as a default when the build backend is
left unspecified; when projects add the build-backend key, they will have to also change their setup.py to either
add the source root to their sys.path or avoid importing from the source root.
self bootstrapping backends Link to heading
Another interesting problem was raised that has a much tighter user base but exposes an interesting problem. In case we
don’t want to use wheels, and we provide only via source distribution, how should we resolve the problem of how we
provide the build backend’s build backend? For example, setuptools packages itself via setuptools. So were
setuptools specify this via PEP-517, the build frontend would be put inside an infinite loop.
To install the library pugs it would first try to create an isolated environment. This environment needs setuptools,
so the build frontend will need to build a wheel to satisfy it. The wheel build would itself trigger the creation of an
isolated environment, which has build dependency again setuptools.
How to break this loop? Mandate all build backends must be exposed as wheels? Allow backends that can build themselves? Should these self-build backends allow to take on dependencies? There’s a long discussion with various options, pros and cons, so if you’re interested, make sure to head over the python Discourse board and give your opinion.
Conclusion Link to heading
Packaging is hard. Improving a packaging system without breakage where users can write and run arbitrary code during the packaging in their free time is near impossible. With PEP-518, build dependencies are explicit and build environments easy to create. With PEP-517, we can start moving into a more declarative packaging namespace that allows less space for users to make mistakes and provides better messages when things go wrong. As we go through these changes, some packages might break, and we might disallow some practices that worked until now. We (PyPa maintainers) don’t do it in bad faith, so when errors pop up please fill in a detailed error report with what went wrong, how you tried to use it, and what your use case is.
We’re trying to improve the packaging ecosystem. We’ve created the integration-test repository, as an effort to ensure that in the future, we can catch at least some of these edge cases before they land on your machine. If you have any suggestion or requirements for any part of the packaging feel free to start a discussion on the Discuss Python forums packaging section, or open an issue for the relevant tool at hand.
That’s all for now. Thanks for reading through it all! I would like to thank Paul Ganssle for reviewing the packaging series post, and Tech At Bloomberg for allowing me to do open source contributions during my working hours.