pyproject.toml vs setup.py: build backend impact and resolution

Unexpected build failures in CI pipelines often appear when a project still relies on a legacy setup.py while the environment expects a PEP 517‑compatible build system. In production, internal package indexes receive the generated wheel, and a mismatched backend can corrupt the distribution.

# Example showing the issue
# setup.py (legacy)
from setuptools import setup, find_packages

setup(
    name="my_pkg",
    version="0.1.0",
    packages=find_packages(),
)

# pyproject.toml is missing
# Running pip in an isolated CI job
import subprocess, textwrap, sys
cmd = [sys.executable, "-m", "pip", "install", "-vvv", "."]
proc = subprocess.run(cmd, capture_output=True, text=True)
print("--- pip output snippet ---")
print("\n".join(line for line in proc.stderr.splitlines() if "PEP517" in line or "setup.py" in line)[:500])
# Expected: pip falls back to legacy "setup.py install" which fails under PEP 517 isolation

pip reads the [build-system] table in pyproject.toml to decide which backend to invoke. Without it, pip reverts to the legacy setuptools command, which runs in the source tree and can miss build‑time dependencies. This behavior follows PEP 517, the standard that separates build front‑ends from setup scripts. Related factors:

  • Missing or empty pyproject.toml
  • Incompatible setuptools version
  • CI environments that enforce isolation

To diagnose this in your code:

Run the installer with maximum verbosity:

bash
python -m pip install -vvv .

Look for lines like "PEP 517 backend" or "running setup.py install". If you see the latter, the build backend is falling back to the old path.

Fixing the Issue

The quick fix is to add a minimal pyproject.toml that tells pip to use the modern backend:

[build-system]
requires = ["setuptools>=61.0", "wheel"]
build-backend = "setuptools.build_meta"

When to use: Development or one‑off builds.

For a production‑ready setup, pin the backend versions and enable isolation so that the build environment is reproducible:

[build-system]
requires = ["setuptools==71.0.0", "wheel==0.42.0"]
build-backend = "setuptools.build_meta:__legacy__"

Add a pyproject.toml alongside setup.cfg (or migrate metadata there) and remove the old setup.py if possible. The gotcha we hit is that CI runners upgraded pip to 24.x, which started enforcing PEP 517 strictly; the old setup.py alone no longer built a wheel.

In production, also verify the wheel before publishing:

python -m build --no-isolation
if [ $? -ne 0 ]; then
    echo "Build failed – check pyproject.toml"
    exit 1
fi

This validation catches missing backend declarations early.

What Doesn’t Work

❌ Setting setup.cfg only: pip still looks for a pyproject.toml to decide the backend, so the build falls back to legacy mode.

❌ Adding build-backend = "flit_core.buildapi" without installing flit: the build crashes because the required package is missing.

❌ Running python setup.py bdist_wheel manually: this bypasses pip’s isolation and often produces wheels that miss build‑time dependencies.

  • Adding a pyproject.toml but forgetting the [build-system] table, causing pip to error out.
  • Pinning setuptools to a version lower than 61.0, which lacks PEP 517 support.
  • Leaving both setup.py and pyproject.toml with contradictory metadata, leading to mismatched version numbers.

When NOT to optimize

  • Pure source‑only distribution: If you never publish wheels and only ship sdist, the legacy path may be acceptable.
  • Legacy internal tooling: Some internal scripts invoke setup.py directly and cannot be changed without a larger migration.
  • Tiny one‑off scripts: For throw‑away packages under 100 lines, adding a full pyproject may be overkill.
  • Locked CI environment: When the CI image is frozen to an old pip that still supports the legacy flow, you can defer the change.

Frequently Asked Questions

Q: Does this behavior differ between Python 3.10 and 3.13?

No, PEP 517 handling is identical across all Python 3.10+ releases.

Q: Can I keep setup.py and just add pyproject.toml?

Yes, the pyproject.toml takes precedence; setup.py can remain for backward compatibility.

The subtle shift from setup.py to pyproject.toml is easy to miss until a CI upgrade forces PEP 517 compliance. Once the backend is declared explicitly, builds become reproducible and future‑proof. We finally added a guard in our CI that aborts if the build system table is absent – that stopped a month‑long regression in our internal package index.