hungryroot/popsicle/contract-testing-schemathesis.org

ID: bc395965-e123-4c85-85e6-334967b5a161

Backend Project Proposal: Contract Testing with Schemathesis

Decision DECIDED

Adopt *Schemathesis* for automated OpenAPI contract validation of the backend

API. The harness validates our drf-spectacular schema against the running

implementation in CI, catching schema/implementation drift before it escapes

to clients.

This document is scoped to Schemathesis only. Consumer-driven contracts

between Sodapop and its clients (web, iOS, Android) — the work previously

sketched as "Phase 2 (Pact)" — are owned by the Web RFC and are *out of scope*

here. See [[*Follow-on: consumer contracts and contract versioning][Follow-on: consumer contracts and contract versioning]].

Executive Summary

We harden API reliability by validating the OpenAPI schema against the

implementation automatically in CI. Schemathesis generates property-based

requests from the drf-spectacular schema and asserts that responses conform to

it, surfacing annotation gaps and response mismatches that today escape to

staging or to mobile clients.

This is a backend-only effort with quick time-to-value: it reuses the existing

CI and Django/DRF test client, requires no running server, and onboards

incrementally by endpoint priority.

Context and Problem Statement

Backend API stability is constrained by incomplete or incorrect OpenAPI

documentation and by schema drift versus the implementation. This drives:

- Escaped defects when the documented schema disagrees with what the API

actually returns.

- Manual verification overhead in pre-release windows.

- Erosion of trust in the published schema as a source of truth for clients

and tooling.

The decision is a pragmatic, low-risk path: validate the OpenAPI schema

(Schemathesis) so the published contract is provably accurate. Enforcing

consumer expectations across mobile clients is a separate concern handled by

the RFC referenced below.

Decision Requirements and Criteria

- Automated detection of schema/implementation mismatches in CI.

- Clear ownership and failure signals (a distinct, labeled contract-test

stage).

- Minimal infra changes — reuse current CI and the Django/DRF test client; no

live server.

- Incremental onboarding by endpoint priority.

- Property-based coverage of request/response shapes, not just hand-written

examples.

Scope

In scope

- Schemathesis harness that loads the drf-spectacular schema in-process for

=/api/v2/= and =/api/v3/= and validates responses through the DRF test

client.

- Prioritized endpoint coverage and the schema-annotation fixes the harness

reveals.

- A distinct contract-test CI stage.

Out of scope

- Consumer-driven contracts (Pact) between Sodapop and web/iOS/Android.

- A "can I deploy?" release gate across clients.

- A Pact Broker / PactFlow workspace.

These are addressed by the Web RFC — see

[[*Follow-on: consumer contracts and contract versioning][Follow-on]].

Why Schemathesis (and why only Schemathesis here)

Schemathesis is the right first and standalone investment for the backend:

- It improves the accuracy of the published schema immediately, which is a

prerequisite for *any* downstream contract work (consumer-driven or

otherwise). A consumer contract built on an inaccurate schema inherits that

inaccuracy.

- It is backend-only and low-coordination: no cross-team rollout, no broker,

no release-gate negotiation.

- It reuses infrastructure we already have (drf-spectacular, the DRF test

client, the existing CI test job).

Consumer-driven contract testing is valuable but is a different problem with

different owners (the client teams and Sodapop) and a different mechanism. It

is deliberately decoupled from this effort and tracked in the RFC.

Plan

Objective: validate and harden the OpenAPI schema; catch request/response

mismatches automatically in CI.

1. Add =schemathesis= to =pyproject.toml= =[project.optional-dependencies].dev=.

The repo has no =requirements-test.txt=; test/dev dependencies live here.

Pin to the =3.x= line: Schemathesis =4.x= requires =tenacity>=9.1.2=, which

conflicts with our =tenacity~=8.3.0=.

2. Create =app/tests/rest/contract/= for contract tests.

3. Load the drf-spectacular schema in-process for each API version (generate

directly from the version url-patterns; no running server) and load it into

Schemathesis.

4. Drive property-based request/response validation through the DRF =APIClient=.

5. Target top-priority endpoints, seeding required data with existing Factory

Boy / =seed_*= helpers.

6. Fix the schema-annotation gaps the harness reveals (e.g. undocumented error

responses on filtered list endpoints).

7. Add the contract suite as a distinct, clearly labeled stage in CI.

Status

- Step 1–4 (harness scaffold) landed under [[https://hungryroot.atlassian.net/browse/BE-7835][BE-7835]]: =schemathesis~=3.39= in

the dev deps, the =app/tests/rest/contract/= package, in-process per-version

schema loading, and =APIClient=-driven response validation. The harness

validates =/api/v2/tags/= against the schema and already surfaces a real gap

(an undocumented =400= on the =tag_group= filter).

- Steps 5–7 (priority-endpoint coverage, annotation fixes, CI stage) are the

remaining work.

Deliverables

- A green, clearly labeled CI contract-test stage with prioritized endpoint

coverage.

- An updated, accurate drf-spectacular OpenAPI schema.

- Documentation for running and triaging Schemathesis locally and in CI.

CI Integration

The contract suite runs through the standard Django test runner as its own

job, so failures are attributable to a labeled "Contract Tests" stage rather

than buried in the general test run.

#+begin_src yaml

# django-tests.yml (excerpt)

jobs:

contract-tests:

runs-on: ubuntu-latest

steps:

- uses: actions/checkout@v4

- uses: actions/setup-python@v5

with:

python-version: '3.10'

- run: uv sync --extra dev

- run: uv run python manage.py test app.tests.rest.contract --keepdb

#+end_src

Technical Approach Details

Target endpoints (initial coverage)

- Authentication: login, refresh, logout.

- Core domain read paths with the highest client use (top endpoints by

traffic / error budget).

Repository changes

- New package: =app/tests/rest/contract/= (schema loader, base test case,

concrete per-endpoint tests).

- Dependency: =schemathesis~=3.39= in =pyproject.toml= =[dev]=.

- A distinct "Contract Tests" CI job.

Success Metrics

- ≥ 90% of the top 20 endpoints covered by Schemathesis.

- Zero schema/implementation mismatches escaping to staging once coverage is

in place.

- The published schema is treated as authoritative by client and tooling

teams.

Risks and Mitigations

- *Schema gaps balloon the effort.* Mitigation: triage daily; prioritize

high-traffic endpoints; fix annotations incrementally rather than blocking on

full coverage.

- *Property-based flakiness / nondeterminism.* Mitigation: deterministic

seeded test data; bounded example counts; stable provider state per test.

- *Dependency friction (Schemathesis 4.x vs. our pins).* Mitigation: pinned to

the =3.x= line, which has no =tenacity= dependency and accepts our

=jsonschema~=4.19.2=.

Follow-on: consumer contracts and contract versioning

Enforcing that clients and the provider agree on payloads — and evolving those

payloads safely — is the consumer-driven side of contract testing. That work

is owned by the Web RFC, not this document:

[[https://hungryroot.atlassian.net/wiki/spaces/WD/pages/1584168961][RFC: Client-Safe Sodapop UI Attachments and Suggested Action Contracts]]

In particular, the policy for evolving contracts once they have shipped — never

mutating a shipped contract in place, versioning breaking changes, and gating

deprecation on client usage — is defined in that RFC's

[[https://hungryroot.atlassian.net/wiki/spaces/WD/pages/1584168961/RFC+Client-Safe+Sodapop+UI+Attachments+and+Suggested+Action+Contracts#Maintaining-Contract-Versions]["Maintaining Contract Versions"]] section. The RFC also proposes Pact as the

mechanism for making the client/provider capability matrix evidence-backed

rather than a manual assumption.

The accurate schema this Schemathesis effort produces is the foundation that

consumer-contract work builds on.

Tickets

Phase 1 (Schemathesis) work, tracked under the =contract-testing= label

([[https://hungryroot.atlassian.net/issues?jql=labels%20%3D%20%22contract-testing%22][view in Jira]]):

|--------------------------------------------------+----------------------------------------------------------------------------+-----------+-------------|