Python Packaging and Dependency Management

Modern Python packaging with pyproject.toml, build backends, version pinning, and reproducible builds for reliable dependency management.

Overview

Python packaging has evolved significantly with PEP 517/518/621, standardising on pyproject.toml as the declarative configuration file and introducing pluggable build backends. The current default tool for the whole workflow is uv — it manages Python versions, virtual environments, dependencies, lockfiles, builds, and publishing, and replaces pip, pip-tools, pipx, pyenv, and most of poetry in one binary. pip, pip-tools, and poetry still work and remain widespread; they are covered below as alternatives. Commands verified against uv 0.11.x.

flowchart TB
    subgraph Project Setup
        A[pyproject.toml] --> B[Build Backend]
        B --> U[uv_build]
        B --> C[setuptools]
        B --> E[hatchling]
        B --> D[poetry-core]
    end

    subgraph Development
        F[uv venv / uv sync] --> G[Install Dependencies]
        G --> I[uv.lock]
        G --> H[requirements.txt]
    end

    subgraph Distribution
        J[uv build] --> K[wheel]
        J --> L[sdist]
        K --> M[PyPI]
        L --> M
        M --> N[uv add / pip install]
    end

    A --> F
    U --> J
    C --> J
    D --> J
    E --> J

Virtual Environments

Virtual environments provide isolated Python installations to prevent dependency conflicts between projects.

Key Concepts

Virtual environment: Isolated Python environment with separate site-packages
uv venv: Creates a virtual environment (no pip seeded by default; uv installs into it directly)
uv python: Downloads and manages standalone CPython builds — replaces pyenv
venv: Built-in stdlib module (Python 3.3+) — still fine, just slower and manual
Activation: Modifies PATH to use the environment's Python; with uv run you rarely need it

Common Patterns

# Create virtual environment with uv (instant; respects .python-version)
uv venv

# Create with a specific Python version — downloads it if missing
uv venv --python 3.12

# Activate on Linux/macOS (optional — `uv run` works without activation)
source .venv/bin/activate

# Activate on Windows
.venv\Scripts\activate

# Deactivate
deactivate

# Seed pip/setuptools into the venv if a tool insists on them
uv venv --seed

# Run a command in the project environment without activating
uv run python script.py
uv run pytest

Python version management with uv (replaces pyenv):

# Install Python versions (standalone builds from python-build-standalone)
uv python install 3.11 3.12 3.13

# List installed and available versions
uv python list

# Pin the project's Python version (writes .python-version)
uv python pin 3.12

Examples

Standard library venv (no uv available, e.g. minimal CI images):

# Create virtual environment with venv
python3 -m venv .venv

# Create with specific Python version
python3.11 -m venv .venv

# Upgrade pip in virtual environment
python -m pip install --upgrade pip

Conda environments (data science stacks with non-Python dependencies):

# Create conda environment
conda create -n myenv python=3.11

# Activate
conda activate myenv

# Export environment
conda env export > environment.yml

# Create from file
conda env create -f environment.yml

Legacy note: virtualenv and pyenv still work, but uv venv and uv python install/uv python pin cover both use cases with far less machinery — no shims, no compiling Python from source.

pyproject.toml Configuration

pyproject.toml is the modern standard for Python project configuration, replacing setup.py and setup.cfg.

Key Concepts

PEP 518: Defines pyproject.toml for build system requirements
PEP 621: Standardises project metadata in pyproject.toml
PEP 735: [dependency-groups] for dev-only dependencies that aren't published metadata
Build backend: Tool that builds the package (uv_build, setuptools, hatchling, poetry-core)
Project metadata: Name, version, dependencies, authors, etc.
uv init: Scaffolds a conformant pyproject.toml (--lib/--package adds a build system)

Common Patterns

# pyproject.toml - Minimum viable configuration
[build-system]
requires = ["setuptools>=77.0"]
build-backend = "setuptools.build_meta"

[project]
name = "mypackage"
version = "0.1.0"
description = "A sample Python package"
authors = [
    {name = "Your Name", email = "you@example.com"}
]
readme = "README.md"
requires-python = ">=3.11"
license = "MIT"  # SPDX expression (PEP 639); the {text = "..."} table form is deprecated

dependencies = [
    "requests>=2.28.0",
    "pydantic>=2.0.0",
]

# Published extras — install with `pip install mypackage[docs]`
[project.optional-dependencies]
docs = [
    "sphinx>=5.0.0",
    "sphinx-rtd-theme>=1.0.0",
]

# Dev-only dependencies (PEP 735) — not part of published metadata.
# Preferred over a `dev` extra; this is what `uv add --dev` writes.
[dependency-groups]
dev = [
    "pytest>=8.0.0",
    "ruff>=0.8.0",
]

Examples

Complete pyproject.toml with advanced features:

[build-system]
requires = ["setuptools>=77.0"]
build-backend = "setuptools.build_meta"

[project]
name = "myapi"
version = "1.2.3"
description = "Production-ready API package"
authors = [
    {name = "Team Name", email = "team@example.com"}
]
maintainers = [
    {name = "Maintainer", email = "maintainer@example.com"}
]
readme = "README.md"
requires-python = ">=3.11"
license = "MIT"
keywords = ["api", "rest", "fastapi"]
classifiers = [
    "Development Status :: 4 - Beta",
    "Intended Audience :: Developers",
    "License :: OSI Approved :: MIT License",
    "Programming Language :: Python :: 3.11",
    "Programming Language :: Python :: 3.12",
    "Programming Language :: Python :: 3.13",
]

dependencies = [
    "fastapi>=0.100.0,<1.0.0",
    "uvicorn[standard]>=0.23.0",
    "pydantic>=2.0.0",
    "sqlalchemy>=2.0.0",
]

[project.optional-dependencies]
dev = [
    "pytest>=7.4.0",
    "pytest-cov>=4.1.0",
    "black>=23.7.0",
    "ruff>=0.0.290",
    "mypy>=1.5.0",
]
test = [
    "pytest>=7.4.0",
    "pytest-asyncio>=0.21.0",
    "httpx>=0.24.0",
]

[project.urls]
Homepage = "https://github.com/username/myapi"
Documentation = "https://myapi.readthedocs.io"
Repository = "https://github.com/username/myapi.git"
Changelog = "https://github.com/username/myapi/blob/main/CHANGELOG.md"

[project.scripts]
myapi = "myapi.cli:main"
myapi-admin = "myapi.admin:main"

[tool.setuptools.packages.find]
where = ["src"]
include = ["myapi*"]
exclude = ["tests*"]

[tool.setuptools.package-data]
myapi = ["py.typed", "*.pyi"]

Dynamic version from file:

[project]
name = "mypackage"
dynamic = ["version"]

[tool.setuptools.dynamic]
version = {attr = "mypackage.__version__"}

Entry points and plugins:

[project.entry-points."mypackage.plugins"]
plugin1 = "mypackage.plugins:PluginClass"
plugin2 = "mypackage.plugins:AnotherPlugin"

[project.gui-scripts]
myapp = "mypackage.gui:main"

Managing Projects with uv

uv's project workflow is the current default: pyproject.toml declares abstract dependencies, uv.lock records the full cross-platform resolution, and the .venv is kept in sync automatically.

Key Concepts

uv init: Scaffolds a project; --lib or --package adds a build system (uv_build backend)
uv add / uv remove: Edit pyproject.toml, re-lock, and sync the environment in one step
uv lock / uv sync: Resolve to uv.lock / install exactly what the lockfile says
uv run: Run a command in the project environment, syncing first if needed
Dependency groups (PEP 735): [dependency-groups] for dev-only deps; dev is the default group
uv tool: Install CLI tools in isolated environments — replaces pipx

Common Patterns

# Create a new project (add --lib or --package for a publishable package)
uv init myproject
uv init --lib mylib

# Add dependencies (updates pyproject.toml, uv.lock, and .venv)
uv add requests "pydantic>=2.0"
uv add 'uvicorn[standard]'

# Add to the dev group / a named group / an optional extra
uv add --dev pytest
uv add --group docs sphinx
uv add --optional cli typer

# Remove a dependency
uv remove requests

# Resolve dependencies into uv.lock (uv add/run do this implicitly)
uv lock

# Install the environment exactly as locked
uv sync                  # includes the dev group by default
uv sync --no-dev         # production install
uv sync --all-groups     # everything
uv sync --frozen         # use uv.lock as-is, fail rather than re-resolve

# Verify the lockfile is current (CI guard)
uv lock --check

# Run inside the project environment (auto-syncs first)
uv run python -m myproject
uv run pytest

# Upgrade dependencies within pyproject.toml constraints
uv lock --upgrade
uv lock --upgrade-package requests

# Inspect the dependency tree
uv tree

# Export uv.lock to requirements.txt format for tools that need it
uv export --no-hashes -o requirements.txt

Examples

pyproject.toml as managed by uv:

[project]
name = "myproject"
version = "0.1.0"
requires-python = ">=3.11"
dependencies = [
    "requests>=2.32.0",
    "pydantic>=2.0.0",
]

# PEP 735 dependency groups — not published in package metadata
[dependency-groups]
dev = [
    "pytest>=8.0.0",
    "ruff>=0.8.0",
]

[build-system]
requires = ["uv_build>=0.11.0,<0.12.0"]
build-backend = "uv_build"

CLI tools in isolated environments (replaces pipx):

# Install a tool globally, isolated from every project
uv tool install ruff
uv tool install --with mkdocs-material mkdocs

# Run a tool without installing it (uvx = uv tool run)
uvx ruff check .
uvx --from 'huggingface_hub[cli]' hf auth login

# List / upgrade / remove
uv tool list
uv tool upgrade --all
uv tool uninstall ruff

Build Backends

Build backends handle package building, distribution creation, and dependency resolution. Each offers different features and workflows. Note that uv is a frontend to all of them — uv build works regardless of which backend pyproject.toml declares.

flowchart LR
    A[pyproject.toml] --> B{Build Backend}
    B --> U[uv_build]
    B --> C[setuptools]
    B --> E[hatchling]
    B --> D[poetry-core]
    B --> F[flit-core]

    U --> G[uv build]
    C --> G
    E --> G
    D --> G
    F --> G

    G --> K[Distribution Files .whl/.tar.gz]

uv_build

uv's own backend — fast, zero-config for src layouts, and what uv init --lib/--package generates. Good default for pure-Python packages; for compiled extensions use setuptools, hatchling with plugins, or maturin.

Common Patterns

# pyproject.toml with uv_build
[build-system]
requires = ["uv_build>=0.11.0,<0.12.0"]
build-backend = "uv_build"

# Build wheel and sdist into dist/ (works with any PEP 517 backend, not just uv_build)
uv build

# Build only wheel / only sdist
uv build --wheel
uv build --sdist

# Ignore tool.uv.sources overrides — recommended before publishing
uv build --no-sources

Legacy note: python -m build (the PyPA build frontend) produces the same artefacts and is still common in CI; uv build is faster and needs no pre-installed tooling.

Setuptools

The traditional, widely-compatible build backend with extensive plugin ecosystem.

Common Patterns

# pyproject.toml with setuptools
[build-system]
requires = ["setuptools>=61.0"]  # setuptools no longer needs "wheel" listed here
build-backend = "setuptools.build_meta"

[project]
name = "mypackage"
version = "0.1.0"
dependencies = ["requests>=2.28.0"]

[tool.setuptools]
# Package discovery
packages = ["mypackage", "mypackage.submodule"]

# Or use automatic discovery
[tool.setuptools.packages.find]
where = ["src"]
include = ["mypackage*"]
exclude = ["tests*", "docs*"]

# Build (uv invokes the setuptools backend declared in pyproject.toml)
uv build

# Legacy equivalent via the PyPA build frontend
pip install build && python -m build

# Install in editable mode for development
uv pip install -e .

# Install with optional dependencies
uv pip install -e ".[dev]"

Examples

setuptools with src layout:

myproject/
├── src/
│   └── mypackage/
│       ├── __init__.py
│       └── module.py
├── tests/
├── pyproject.toml
└── README.md

[tool.setuptools.packages.find]
where = ["src"]

setuptools with custom build step:

# setup.py (when customisation needed)
from setuptools import setup
from setuptools.command.build_py import build_py

class CustomBuild(build_py):
    def run(self):
        # Custom build logic
        print("Running custom build")
        super().run()

setup(
    cmdclass={'build_py': CustomBuild}
)

Poetry (alternative)

All-in-one dependency management and packaging tool with its own lockfile (poetry.lock). Still widespread in existing projects; Poetry 2.x supports standard PEP 621 [project] metadata, though the legacy [tool.poetry] table below remains common. For new projects, uv covers the same workflow.

Common Patterns

# pyproject.toml with poetry
[tool.poetry]
name = "mypackage"
version = "0.1.0"
description = "A sample package"
authors = ["Your Name <you@example.com>"]
readme = "README.md"
license = "MIT"
homepage = "https://example.com"
repository = "https://github.com/username/mypackage"

[tool.poetry.dependencies]
python = "^3.9"
requests = "^2.28.0"
pydantic = "^2.0.0"

[tool.poetry.group.dev.dependencies]
pytest = "^7.4.0"
black = "^23.7.0"
ruff = "^0.0.290"

[tool.poetry.group.docs.dependencies]
sphinx = "^5.0.0"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

# Initialise poetry project
poetry init

# Install dependencies
poetry install

# Add dependency
poetry add requests

# Add dev dependency
poetry add --group dev pytest

# Update dependencies
poetry update

# Show installed packages
poetry show

# Build distributions
poetry build

# Publish to PyPI
poetry publish

# Run command in virtual environment
poetry run python script.py
poetry run pytest

# Activate the virtual environment (Poetry 2.0+)
poetry env activate
# poetry shell was removed in 2.0; install poetry-plugin-shell to keep it

Examples

Poetry with version constraints:

[tool.poetry.dependencies]
# Caret (^): Allow compatible updates
requests = "^2.28.0"  # >=2.28.0,<3.0.0

# Tilde (~): Allow patch updates
flask = "~2.3.0"  # >=2.3.0,<2.4.0

# Exact version
django = "4.2.5"

# Multiple constraints
numpy = ">=1.20,<2.0"

# Platform-specific
pywin32 = {version = "^306", markers = "sys_platform == 'win32'"}

# Python version specific
dataclasses = {version = "^0.8", python = "~3.6"}

# From git
mylib = {git = "https://github.com/user/mylib.git", branch = "main"}

# From local path
locallib = {path = "../locallib", develop = true}

Poetry scripts and plugins:

[tool.poetry.scripts]
myapp = "mypackage.cli:main"
admin = "mypackage.admin:run"

[tool.poetry.plugins."pytest11"]
myplugin = "mypackage.pytest_plugin"

Hatchling

Modern, standards-compliant build backend from the Hatch project management tool.

Common Patterns

# pyproject.toml with hatchling
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "mypackage"
version = "0.1.0"
dependencies = ["requests>=2.28.0"]

[tool.hatch.build.targets.wheel]
packages = ["src/mypackage"]

[tool.hatch.build.targets.sdist]
exclude = [
    "/.github",
    "/docs",
    "/tests",
]

[tool.hatch.version]
path = "src/mypackage/__init__.py"

# Install hatch
pip install hatch

# Create new project
hatch new myproject

# Create environment
hatch env create

# Run command
hatch run python script.py

# Run tests
hatch run test:pytest

# Build
hatch build

# Publish
hatch publish

Examples

Hatch with multiple environments:

[tool.hatch.envs.default]
dependencies = [
    "pytest>=7.4.0",
    "pytest-cov>=4.1.0",
]

[tool.hatch.envs.default.scripts]
test = "pytest {args:tests}"
cov = "pytest --cov=mypackage {args:tests}"

[tool.hatch.envs.lint]
detached = true
dependencies = [
    "black>=23.7.0",
    "ruff>=0.0.290",
]

[tool.hatch.envs.lint.scripts]
check = [
    "black --check .",
    "ruff check .",
]
fix = [
    "black .",
    "ruff check --fix .",
]

[tool.hatch.envs.docs]
dependencies = [
    "sphinx>=5.0.0",
    "sphinx-rtd-theme>=1.0.0",
]

[tool.hatch.envs.docs.scripts]
build = "sphinx-build -b html docs docs/_build"
serve = "python -m http.server -d docs/_build"

Dynamic versioning with hatchling (requires the hatch-vcs plugin):

[build-system]
requires = ["hatchling", "hatch-vcs"]
build-backend = "hatchling.build"

[project]
name = "mypackage"
dynamic = ["version"]

[tool.hatch.version]
source = "vcs"

[tool.hatch.build.hooks.vcs]
version-file = "src/mypackage/_version.py"

Version Pinning and Constraints

Version pinning ensures reproducible installations by specifying exact or bounded dependency versions.

flowchart TD
    A[Dependency Specification] --> B{Constraint Type}

    B --> C[Loose: >=1.0.0]
    B --> D[Compatible: ^2.3.0]
    B --> E[Patch: ~1.2.3]
    B --> F[Exact: ==1.2.3]

    C --> G[Install Latest Compatible]
    D --> H[Install <3.0.0]
    E --> I[Install <1.3.0]
    F --> J[Install Exactly 1.2.3]

    G --> K{Reproducible?}
    H --> K
    I --> K
    J --> K

    K -->|No| L[Use Lockfile]
    K -->|Yes| M[Guaranteed Version]

Key Concepts

Abstract dependencies: High-level requirements (e.g., requests>=2.28.0)
Concrete dependencies: Exact versions from lockfile (e.g., requests==2.31.0)
Constraints file: Pins versions without declaring them as requirements
Version specifiers: PEP 440 syntax for version ranges

Common Patterns

# requirements.txt - Abstract dependencies
requests>=2.28.0
pydantic>=2.0.0,<3.0.0
fastapi>=0.100.0

# requirements-lock.txt - Concrete dependencies (generated)
requests==2.31.0
pydantic==2.4.2
fastapi==0.104.1
certifi==2023.7.22
charset-normalizer==3.3.0
# ... all transitive dependencies with exact versions

# constraints.txt - Version bounds
requests==2.31.0
urllib3==2.0.7

# Install from requirements (uv pip is a drop-in, faster pip replacement)
uv pip install -r requirements.txt

# Install with constraints
uv pip install -r requirements.txt -c constraints.txt

# Generate locked requirements
uv pip freeze > requirements-lock.txt

# Install from locked requirements
uv pip install -r requirements-lock.txt

# Install requirements and dev dependencies
uv pip install -r requirements.txt -r requirements-dev.txt

# Legacy: the same commands work with plain pip (pip install -r ..., pip freeze)

Examples

Multi-file requirements structure:

requirements/
├── base.txt          # Core dependencies
├── dev.txt           # Development dependencies
├── test.txt          # Testing dependencies
├── docs.txt          # Documentation dependencies
└── prod.txt          # Production dependencies

# base.txt
django>=4.2.0,<5.0.0
psycopg2-binary>=2.9.0
celery>=5.3.0
redis>=5.0.0

# dev.txt
-r base.txt
-r test.txt
black>=23.7.0
ruff>=0.0.290
ipython>=8.12.0

# test.txt
-r base.txt
pytest>=7.4.0
pytest-django>=4.5.0
pytest-cov>=4.1.0
factory-boy>=3.3.0

# prod.txt
-r base.txt
gunicorn>=21.2.0

Version specifier syntax (PEP 440):

# requirements.txt with various specifiers

# Compatible release (recommended)
requests>=2.28.0,<3.0.0  # Explicit range
django~=4.2.0            # >=4.2.0,<4.3.0

# Exact version
pillow==10.0.0

# Multiple constraints
numpy>=1.20.0,!=1.21.0,<2.0.0

# Pre-release versions
some-package>=1.0.0a1

# Exclude pre-releases by default
stable-package>=2.0.0    # Won't install 2.1.0a1

# Include pre-releases
bleeding-edge>=1.0.0.dev0

# Version exclusion
problematic-lib>=1.0.0,!=1.2.0,!=1.2.1

# Environment markers
pywin32>=306; sys_platform == "win32"
uvloop>=0.17.0; sys_platform != "win32" and python_version >= "3.7"

# URL-based dependencies
mylib @ https://github.com/user/mylib/archive/main.zip
locallib @ file:///path/to/locallib

# Git repositories
requests @ git+https://github.com/psf/requests.git@main
mypackage @ git+ssh://git@github.com/user/package.git@v1.0.0

Using uv pip compile/sync for requirements.txt lockfiles (pip-tools replacement):

# Create requirements.in (abstract dependencies)
cat > requirements.in << EOF
django>=4.2
requests
celery[redis]
EOF

# Compile to requirements.txt (concrete dependencies)
uv pip compile requirements.in -o requirements.txt

# Sync environment to match exactly (adds and removes packages)
uv pip sync requirements.txt

# Upgrade packages
uv pip compile --upgrade requirements.in -o requirements.txt

# Upgrade specific package
uv pip compile --upgrade-package django requirements.in -o requirements.txt

# Generate requirements for development
uv pip compile requirements-dev.in -o requirements-dev.txt

# Compile straight from pyproject.toml
uv pip compile pyproject.toml -o requirements.txt

Legacy note: the original pip-tools commands (pip-compile, pip-sync) take the same inputs and produce compatible output; uv pip compile/uv pip sync are drop-in, much faster replacements. For uv-managed projects, prefer uv lock/uv sync with uv.lock instead — it is cross-platform, whereas a compiled requirements.txt is resolved for the platform it was compiled on.

Lockfiles and Reproducible Builds

Lockfiles record exact versions of all dependencies (including transitive) to ensure reproducible builds across environments.

Key Concepts

Lockfile: File containing exact versions of all dependencies (uv.lock, poetry.lock, compiled requirements.txt)
Transitive dependencies: Dependencies of dependencies
Reproducibility: Guarantee same versions install everywhere
Dependency resolution: Process of determining compatible versions
Cross-platform lock: uv.lock resolves for all platforms at once; requirements.txt is per-platform

Common Patterns

uv lockfile (uv.lock):

# Generate/update lockfile (also happens implicitly on uv add/run/sync)
uv lock

# Install exactly what the lockfile says (includes dev group by default)
uv sync

# Production install: no dev group, and fail instead of re-resolving
uv sync --no-dev --frozen

# CI guard: fail if uv.lock is out of date with pyproject.toml
uv lock --check

# Update a single package / everything (within pyproject.toml constraints)
uv lock --upgrade-package requests
uv lock --upgrade

# Export to requirements.txt format for tools that can't read uv.lock
uv export -o requirements.txt          # includes hashes by default
uv export --no-hashes --no-dev -o requirements.txt

uv.lock is cross-platform: it records the resolution for all platforms and Python versions in range, so a lock generated on macOS installs identically on Linux CI. Commit it to version control.

Poetry lockfile (poetry.lock):

# Generate/update lockfile
poetry lock

# Install from lockfile
poetry install

# Install without dev dependencies
poetry install --without dev

# Update single package
poetry update requests

# Update all packages
poetry update

# Install specific groups
poetry install --with docs,test --without dev

# Export to requirements format (needs poetry-plugin-export since Poetry 2.0)
poetry export -f requirements.txt -o requirements.txt --without-hashes
poetry export --with dev -f requirements.txt -o requirements-dev.txt

Pipenv lockfile (Pipfile.lock) — legacy, still seen in older codebases:

# Create Pipfile and install packages
pipenv install requests django
pipenv install --dev pytest black

# Generate lockfile / install from it
pipenv lock
pipenv install

# Run command in virtual environment
pipenv run python script.py

# Generate requirements.txt
pipenv requirements > requirements.txt

requirements.txt lockfiles (uv pip compile, or legacy pip-tools):

# requirements.in (input)
django>=4.2
celery[redis]
requests

# Compile to lockfile (add --generate-hashes for hash-pinned installs)
uv pip compile --generate-hashes requirements.in -o requirements.txt

# Sync environment
uv pip sync requirements.txt requirements-dev.txt

# Legacy pip-tools equivalents: pip-compile / pip-sync (same file formats)

Examples

Docker with lockfiles for reproducibility:

# Using uv
FROM python:3.12-slim

# Copy the uv binary from the official image (pin the tag in production)
COPY --from=ghcr.io/astral-sh/uv:0.11 /uv /uvx /bin/

WORKDIR /app

# Copy only dependency files first (cache layer), install deps without the project
COPY pyproject.toml uv.lock ./
RUN --mount=type=cache,target=/root/.cache/uv \
    uv sync --frozen --no-dev --no-install-project

# Copy application code and install it
COPY . .
RUN --mount=type=cache,target=/root/.cache/uv \
    uv sync --frozen --no-dev

# Use the project venv directly
ENV PATH="/app/.venv/bin:$PATH"

CMD ["python", "-m", "myapp"]

# Using a requirements.txt lockfile (pip — works on any base image)
FROM python:3.12-slim

WORKDIR /app

# Copy requirements lockfile
COPY requirements.txt ./

# Install exact versions
RUN pip install --no-cache-dir -r requirements.txt

# Copy application
COPY . .

CMD ["python", "-m", "myapp"]

Reproducible CI/CD pipeline:

# .github/workflows/test.yml
name: Tests

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v6

      - name: Install uv
        uses: astral-sh/setup-uv@v8
        with:
          enable-cache: true

      # uv installs the pinned Python version automatically
      - name: Install dependencies from lockfile
        run: uv sync --frozen

      - name: Check lockfile is up to date
        run: uv lock --check

      - name: Run tests
        run: uv run pytest

Poetry equivalent: cache ~/.cache/pypoetry keyed on hashFiles('poetry.lock'), pip install poetry, then poetry install, poetry check --lock, and poetry run pytest.

Verifying reproducible builds:

# uv verification
uv lock --check          # Fail if uv.lock is stale relative to pyproject.toml
uv sync --frozen         # Install from uv.lock without re-resolving
uv export -o requirements.txt   # Hash-pinned export by default

# Hash-based requirements.txt (uv or legacy pip-compile)
uv pip compile --generate-hashes requirements.in -o requirements.txt

# requirements.txt output includes hashes
# requests==2.31.0 \
#     --hash=sha256:58cd2187c01e70e6e26505bca751777aa9f2ee0b7f4300988b709f44e013003f

# Install with hash verification
pip install --require-hashes -r requirements.txt

# Poetry verification
poetry check --lock  # Verify lockfile is current

# Pipenv verification
pipenv verify  # Check Pipfile.lock matches Pipfile

Publishing to PyPI

Distribute packages to Python Package Index (PyPI) or private indexes for sharing and installation.

flowchart LR
    A[Source Code] --> B[Build]
    B --> C[wheel .whl]
    B --> D[sdist .tar.gz]

    C --> E{Upload To}
    D --> E

    E --> F[TestPyPI]
    E --> G[PyPI]
    E --> H[Private Index]

    F --> I[Test Install]
    G --> J[Public Install]
    H --> K[Internal Install]

Key Concepts

PyPI: Public Python Package Index (pypi.org)
TestPyPI: Sandbox for testing package uploads
wheel: Built distribution format (.whl)
sdist: Source distribution (.tar.gz)
Trusted Publishing: PyPI's OIDC-based auth — CI proves its identity, no long-lived tokens
uv publish / twine: Tools for uploading distributions to an index

Common Patterns

# Build distributions into dist/ (use --no-sources before publishing)
uv build --no-sources

# Bump the version in pyproject.toml first if needed
uv version --bump patch    # also: major, minor, alpha, beta, rc, post, dev

# Upload to TestPyPI (test first!)
uv publish --index testpypi    # with a [[tool.uv.index]] entry, see below

# Install from TestPyPI to test
uv pip install --index-url https://test.pypi.org/simple/ mypackage

# Upload to PyPI — no credentials needed under Trusted Publishing in CI;
# locally, pass a token via --token or UV_PUBLISH_TOKEN
uv publish

# Check built distributions before upload (twine still useful for this)
uvx twine check dist/*

# pyproject.toml — named index for TestPyPI publishing
[[tool.uv.index]]
name = "testpypi"
url = "https://test.pypi.org/simple/"
publish-url = "https://test.pypi.org/legacy/"
explicit = true

Examples

Publishing with GitHub Actions and Trusted Publishing (OIDC) — current best practice:

First register the publisher on PyPI (project → Settings → Publishing): repository owner/name, workflow filename, and optionally a GitHub environment. No token is created or stored anywhere.

# .github/workflows/publish.yml
name: Publish to PyPI

on:
  release:
    types: [published]

jobs:
  publish:
    runs-on: ubuntu-latest
    environment: pypi          # match the environment configured on PyPI
    permissions:
      id-token: write          # REQUIRED for OIDC token exchange

    steps:
      - uses: actions/checkout@v6

      - name: Install uv
        uses: astral-sh/setup-uv@v8

      - name: Build package
        run: uv build --no-sources

      # Exchanges the OIDC token for a short-lived PyPI token automatically
      - name: Publish to PyPI
        uses: pypa/gh-action-pypi-publish@release/v1

uv publish also supports Trusted Publishing directly — with id-token: write set, replace the last step with run: uv publish (detection is automatic on GitHub Actions; force it with --trusted-publishing always). The pypa/gh-action-pypi-publish route additionally generates and uploads PEP 740 attestations.

Legacy: token-based publishing. API tokens are still required for registries without Trusted Publishing support — private indexes, Artifactory, DevPI, some self-hosted mirrors — and for uploads from outside CI:

      # Legacy token flow (store the token as a repository secret)
      - name: Publish to PyPI
        env:
          UV_PUBLISH_TOKEN: ${{ secrets.PYPI_API_TOKEN }}
        run: uv publish
        # twine equivalent:
        #   TWINE_USERNAME: __token__
        #   TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
        #   run: twine upload dist/*

Legacy: configure PyPI credentials locally (~/.pypirc, used by twine):

[distutils]
index-servers =
    pypi
    testpypi

[pypi]
username = __token__
password = pypi-AgEIcHlwaS...  # Your API token

[testpypi]
username = __token__
password = pypi-AgENdGVzdC...  # TestPyPI token

Publishing with Poetry (alternative):

# Configure PyPI token
poetry config pypi-token.pypi pypi-AgEIcHlwaS...

# Build and publish in one command
poetry publish --build

# Publish to TestPyPI
poetry config repositories.testpypi https://test.pypi.org/legacy/
poetry config pypi-token.testpypi pypi-AgENdGVzdC...
poetry publish --repository testpypi

# Dry run (check without uploading)
poetry publish --dry-run

Publishing to private index (token/credential auth — Trusted Publishing is PyPI-side only):

# pyproject.toml — named private index for uv
[[tool.uv.index]]
name = "company"
url = "https://pypi.company.com/simple/"
publish-url = "https://pypi.company.com/upload/"
explicit = true

# Upload with uv (credentials via env vars or --token/--username/--password)
UV_PUBLISH_TOKEN=... uv publish --index company

# Legacy: twine to a private index (e.g., Artifactory, DevPI)
twine upload --repository-url https://pypi.company.com/upload/ dist/*

# Install from private index
uv pip install --index-url https://pypi.company.com/simple/ mypackage

# Use private index with fallback to PyPI
uv pip install --index-url https://pypi.company.com/simple/ \
    --extra-index-url https://pypi.org/simple/ \
    mypackage

Package versioning strategy:

# Semantic versioning in __init__.py
__version__ = "1.2.3"

# MAJOR.MINOR.PATCH
# MAJOR: Breaking changes
# MINOR: New features (backwards compatible)
# PATCH: Bug fixes

# Pre-release versions
__version__ = "2.0.0a1"  # Alpha
__version__ = "2.0.0b1"  # Beta
__version__ = "2.0.0rc1"  # Release candidate

# Development versions
__version__ = "1.2.3.dev0"

# Post-release
__version__ = "1.2.3.post1"

# Version bumping with uv
uv version --bump patch  # 1.2.3 -> 1.2.4
uv version --bump minor  # 1.2.4 -> 1.3.0
uv version --bump major  # 1.3.0 -> 2.0.0
uv version --bump patch --bump beta  # 1.3.0 -> 1.3.1b1
uv version --bump stable  # 1.3.1b1 -> 1.3.1
uv version 2.0.0          # set explicitly
uv version --bump minor --dry-run  # preview only

# Build and publish new version
uv version --bump patch
uv build --no-sources
uv publish

# Poetry equivalents: poetry version patch|minor|major|prerelease

Automated releases with changelog:

# .github/workflows/release.yml
name: Release

on:
  push:
    tags:
      - 'v*'

jobs:
  release:
    runs-on: ubuntu-latest
    permissions:
      contents: write          # create the GitHub release
      id-token: write          # PyPI Trusted Publishing (OIDC)

    steps:
      - uses: actions/checkout@v6

      - name: Install uv
        uses: astral-sh/setup-uv@v8

      - name: Extract version
        id: version
        run: echo "VERSION=${GITHUB_REF#refs/tags/v}" >> $GITHUB_OUTPUT

      - name: Create release notes
        id: release_notes
        run: |
          # Extract changelog for this version
          sed -n '/## \[${{ steps.version.outputs.VERSION }}\]/,/## \[/p' CHANGELOG.md | head -n -1 > notes.md

      # Build BEFORE creating the release — dist/* must exist to be attached
      - name: Build package
        run: uv build --no-sources

      - name: Create GitHub Release
        uses: softprops/action-gh-release@v3
        with:
          body_path: notes.md
          files: dist/*

      - name: Publish to PyPI
        run: uv publish   # uses Trusted Publishing via the id-token permission

Package Structure Best Practices

Organising package structure for maintainability, testing, and distribution.

Common Patterns

Flat layout:

mypackage/
├── mypackage/
│   ├── __init__.py
│   ├── module1.py
│   ├── module2.py
│   └── subpackage/
│       ├── __init__.py
│       └── submodule.py
├── tests/
│   ├── __init__.py
│   ├── test_module1.py
│   └── test_module2.py
├── docs/
├── pyproject.toml
├── README.md
├── LICENSE
└── CHANGELOG.md

Src layout (recommended):

mypackage/
├── src/
│   └── mypackage/
│       ├── __init__.py
│       ├── py.typed
│       ├── module1.py
│       └── module2.py
├── tests/
│   ├── test_module1.py
│   └── test_module2.py
├── pyproject.toml
├── README.md
└── LICENSE

Benefits of src layout:

# pyproject.toml with src layout
[tool.setuptools.packages.find]
where = ["src"]
include = ["mypackage*"]

[tool.setuptools.package-data]
mypackage = ["py.typed", "*.pyi"]

Examples

Including data files:

[tool.setuptools.package-data]
mypackage = [
    "data/*.json",
    "templates/*.html",
    "static/css/*.css",
]

# Or with MANIFEST.in (if needed)
# MANIFEST.in
include README.md
include LICENSE
recursive-include src/mypackage/data *.json
recursive-include src/mypackage/templates *.html

Namespace packages:

# src/mycompany/mypackage/__init__.py
# For native namespace packages (PEP 420), omit __init__.py from the
# namespace directory (src/mycompany/) entirely; only the inner
# package (mypackage/) has an __init__.py

[tool.setuptools.packages.find]
where = ["src"]
include = ["mycompany.*"]

[tool.setuptools.package-dir]
"" = "src"

Type hints and py.typed:

# src/mypackage/__init__.py
from mypackage.module1 import function1
from mypackage.module2 import Class2

__all__ = ["function1", "Class2"]
__version__ = "1.0.0"

# Create py.typed marker for PEP 561 compliance
touch src/mypackage/py.typed

[tool.setuptools.package-data]
mypackage = ["py.typed"]

Quick Reference

uv Project Commands

Task	Command
New project / library	`uv init myproject` / `uv init --lib mylib`
Add dependency	`uv add requests`
Add dev dependency	`uv add --dev pytest`
Remove dependency	`uv remove requests`
Lock dependencies	`uv lock`
Install from lockfile	`uv sync` (`--no-dev --frozen` for prod)
Check lockfile current	`uv lock --check`
Run in project env	`uv run pytest`
Upgrade one package	`uv lock --upgrade-package requests`
Bump version	`uv version --bump patch`
Export requirements.txt	`uv export --no-hashes -o requirements.txt`
Install Python	`uv python install 3.12`
Pin Python version	`uv python pin 3.12`
Install a CLI tool	`uv tool install ruff` (or one-off: `uvx ruff`)

Virtual Environment Commands

Task	Command
Create venv (uv)	`uv venv`
Create with Python version	`uv venv --python 3.12`
Create venv (stdlib)	`python3 -m venv .venv`
Activate (Unix)	`source .venv/bin/activate`
Activate (Windows)	`.venv\Scripts\activate`
Deactivate	`deactivate`
Install in venv	`uv pip install package`

Build and Install Commands

Task	Command
Build distributions	`uv build` (legacy: `python -m build`)
Build wheel only	`uv build --wheel`
Build sdist only	`uv build --sdist`
Build for publishing	`uv build --no-sources`
Install editable	`uv pip install -e .`
Install with extras	`uv pip install -e ".[dev,test]"`
Check distributions	`uvx twine check dist/*`

Poetry Commands (alternative)

Task	Command
Initialise project	`poetry init`
Add dependency	`poetry add requests`
Add dev dependency	`poetry add --group dev pytest`
Install dependencies	`poetry install`
Update dependencies	`poetry update`
Build	`poetry build`
Publish	`poetry publish`
Run command	`poetry run python script.py`
Lock dependencies	`poetry lock`
Export requirements	`poetry export -f requirements.txt -o requirements.txt`

requirements.txt Compilation Commands (uv pip; legacy pip-tools in brackets)

Task	Command
Compile lockfile	`uv pip compile requirements.in -o requirements.txt` (`pip-compile`)
Sync environment	`uv pip sync requirements.txt` (`pip-sync`)
Upgrade all	`uv pip compile --upgrade requirements.in -o requirements.txt`
Upgrade package	`uv pip compile --upgrade-package requests requirements.in -o requirements.txt`
With hashes	`uv pip compile --generate-hashes requirements.in -o requirements.txt`

Publishing Commands

Task	Command
Upload to PyPI	`uv publish` (Trusted Publishing in CI, or `UV_PUBLISH_TOKEN`)
Upload to TestPyPI	`uv publish --index testpypi` (with `[[tool.uv.index]]` entry)
Check before upload	`uvx twine check dist/*`
Legacy twine upload	`twine upload dist/*`
Poetry publish	`poetry publish --build`

Common Version Specifiers

package>=1.0.0        # Minimum version
package>=1.0.0,<2.0.0 # Range
package~=1.4.2        # Compatible (>=1.4.2,<1.5.0)
package==1.2.3        # Exact version
package!=1.3.0        # Exclude version
package>=1.0.0,!=1.2.0,<2.0.0  # Complex constraint

pyproject.toml Essential Sections

[build-system]
requires = ["uv_build>=0.11.0,<0.12.0"]   # or setuptools/hatchling
build-backend = "uv_build"

[project]
name = "mypackage"
version = "0.1.0"
requires-python = ">=3.11"
dependencies = ["requests>=2.28.0"]

[project.optional-dependencies]
cli = ["typer>=0.12.0"]      # published extras

[dependency-groups]
dev = ["pytest>=8.0.0"]      # dev-only (PEP 735), what `uv add --dev` writes

[project.scripts]
myapp = "mypackage.cli:main"

Common Issues and Solutions

Issue	Solution
Package not found after install	Ensure package is in PYTHONPATH; check `pip show package` to verify installation location
Import error in editable install	Use src layout; ensure `pyproject.toml` has correct package discovery configuration
Dependencies conflict	Use constraints file; pin conflicting packages to compatible versions; check with `pip check`
Lockfile out of sync	Regenerate: `uv lock` (or `poetry lock`, `pipenv lock`); commit lockfile to version control; gate CI with `uv lock --check`
Build fails with missing files	Add files to `MANIFEST.in` or `package-data`; verify with `python -m build --sdist`
Upload fails authentication	Prefer Trusted Publishing (OIDC) in CI — check `id-token: write` and the publisher config on PyPI; for token flows, verify `UV_PUBLISH_TOKEN`/`~/.pypirc` and token scope
Package version conflict in Docker	Use lockfiles; copy `uv.lock`/requirements before code; `uv sync --frozen` or `--no-cache-dir` with pip
Cannot install from private index	Add index URL: `uv pip install --index-url https://private.pypi/simple/ package`
Poetry install slow	Parallel installs are on by default (`installer.parallel`); avoid `--no-cache` unless debugging
Editable install breaks imports	Ensure package has `__init__.py`; check package discovery in `pyproject.toml`
Wrong Python version in venv	`uv venv --python 3.12` (downloads it if missing), or `uv python pin 3.12`; stdlib: `python3.11 -m venv .venv`
Transitive dependency issue	Pin problematic version in direct dependencies; use constraints file
Hash mismatch error	Regenerate lockfile: `uv pip compile --generate-hashes` (or `pip-compile`); ensure requirements.txt is up to date
Module not found after build	Check `[tool.setuptools.packages.find]` includes correct paths
Large wheel size	Exclude unnecessary files in `[tool.setuptools.exclude-package-data]` or `.gitignore`

Debugging Tips

# Verify package contents in wheel
unzip -l dist/mypackage-1.0.0-py3-none-any.whl

# Check installed package files
pip show -f mypackage

# Verify dependency resolution
uv pip install --dry-run package

# Show the resolved dependency tree
uv tree

# Debug build process
uv build --verbose          # or: python -m build --verbose

# Check package metadata
uv build && tar -tzf dist/mypackage-1.0.0.tar.gz

# Validate pyproject.toml
pip install validate-pyproject
validate-pyproject pyproject.toml

# Test installation and import from the built package without polluting the project
uv run --with mypackage --no-project -- python -c "import mypackage"

# Test installation from built wheel
uv pip install dist/mypackage-1.0.0-py3-none-any.whl

# Check for dependency conflicts
uv pip check

# Verify reproducibility
uv pip freeze > snapshot.txt
uv pip sync snapshot.txt

Python Packaging and Dependency Management

Overview

Virtual Environments

Key Concepts

Common Patterns

Examples

pyproject.toml Configuration

Key Concepts

Common Patterns

Examples

Managing Projects with uv

Key Concepts

Common Patterns

Examples

Build Backends

uv_build

Common Patterns

Setuptools

Common Patterns

Examples

Poetry (alternative)

Common Patterns

Examples

Hatchling

Common Patterns

Examples

Version Pinning and Constraints

Key Concepts

Common Patterns

Examples

Lockfiles and Reproducible Builds

Key Concepts

Common Patterns

Examples

Publishing to PyPI

Key Concepts

Common Patterns

Examples

Package Structure Best Practices

Common Patterns

Examples

Quick Reference

uv Project Commands

Virtual Environment Commands

Build and Install Commands

Poetry Commands (alternative)

requirements.txt Compilation Commands (uv pip; legacy pip-tools in brackets)

Publishing Commands

Common Version Specifiers

pyproject.toml Essential Sections

Common Issues and Solutions

Debugging Tips

Related Topics