Org Web Adapter

Editing hungryroot/popsicle/contract-testing-schemathesis.org

:PROPERTIES:
:ID:       bc395965-e123-4c85-85e6-334967b5a161
:END:
#+title: Backend Project Proposal: Contract Testing with Schemathesis

* Decision DECIDED

Adopt *Schemathesis* for automated OpenAPI contract validation of the backend
API. The harness validates our drf-spectacular schema against the running
implementation in CI, catching schema/implementation drift before it escapes
to clients.

This document is scoped to Schemathesis only. Consumer-driven contracts
between Sodapop and its clients (web, iOS, Android) — the work previously
sketched as "Phase 2 (Pact)" — are owned by the Web RFC and are *out of scope*
here. See [[*Follow-on: consumer contracts and contract versioning][Follow-on: consumer contracts and contract versioning]].

* Executive Summary

We harden API reliability by validating the OpenAPI schema against the
implementation automatically in CI. Schemathesis generates property-based
requests from the drf-spectacular schema and asserts that responses conform to
it, surfacing annotation gaps and response mismatches that today escape to
staging or to mobile clients.

This is a backend-only effort with quick time-to-value: it reuses the existing
CI and Django/DRF test client, requires no running server, and onboards
incrementally by endpoint priority.

* Context and Problem Statement

Backend API stability is constrained by incomplete or incorrect OpenAPI
documentation and by schema drift versus the implementation. This drives:

- Escaped defects when the documented schema disagrees with what the API
  actually returns.
- Manual verification overhead in pre-release windows.
- Erosion of trust in the published schema as a source of truth for clients
  and tooling.

The decision is a pragmatic, low-risk path: validate the OpenAPI schema
(Schemathesis) so the published contract is provably accurate. Enforcing
consumer expectations across mobile clients is a separate concern handled by
the RFC referenced below.

* Decision Requirements and Criteria

- Automated detection of schema/implementation mismatches in CI.
- Clear ownership and failure signals (a distinct, labeled contract-test
  stage).
- Minimal infra changes — reuse current CI and the Django/DRF test client; no
  live server.
- Incremental onboarding by endpoint priority.
- Property-based coverage of request/response shapes, not just hand-written
  examples.

* Scope

** In scope

- Schemathesis harness that loads the drf-spectacular schema in-process for
  =/api/v2/= and =/api/v3/= and validates responses through the DRF test
  client.
- Prioritized endpoint coverage and the schema-annotation fixes the harness
  reveals.
- A distinct contract-test CI stage.

** Out of scope

- Consumer-driven contracts (Pact) between Sodapop and web/iOS/Android.
- A "can I deploy?" release gate across clients.
- A Pact Broker / PactFlow workspace.

These are addressed by the Web RFC — see
[[*Follow-on: consumer contracts and contract versioning][Follow-on]].

* Why Schemathesis (and why only Schemathesis here)

Schemathesis is the right first and standalone investment for the backend:

- It improves the accuracy of the published schema immediately, which is a
  prerequisite for *any* downstream contract work (consumer-driven or
  otherwise). A consumer contract built on an inaccurate schema inherits that
  inaccuracy.
- It is backend-only and low-coordination: no cross-team rollout, no broker,
  no release-gate negotiation.
- It reuses infrastructure we already have (drf-spectacular, the DRF test
  client, the existing CI test job).

Consumer-driven contract testing is valuable but is a different problem with
different owners (the client teams and Sodapop) and a different mechanism. It
is deliberately decoupled from this effort and tracked in the RFC.

* Plan

Objective: validate and harden the OpenAPI schema; catch request/response
mismatches automatically in CI.

1. Add =schemathesis= to =pyproject.toml= =[project.optional-dependencies].dev=.
   The repo has no =requirements-test.txt=; test/dev dependencies live here.
   Pin to the =3.x= line: Schemathesis =4.x= requires =tenacity>=9.1.2=, which
   conflicts with our =tenacity~=8.3.0=.
2. Create =app/tests/rest/contract/= for contract tests.
3. Load the drf-spectacular schema in-process for each API version (generate
   directly from the version url-patterns; no running server) and load it into
   Schemathesis.
4. Drive property-based request/response validation through the DRF =APIClient=.
5. Target top-priority endpoints, seeding required data with existing Factory
   Boy / =seed_*= helpers.
6. Fix the schema-annotation gaps the harness reveals (e.g. undocumented error
   responses on filtered list endpoints).
7. Add the contract suite as a distinct, clearly labeled stage in CI.

** Status

- Step 1–4 (harness scaffold) landed under [[https://hungryroot.atlassian.net/browse/BE-7835][BE-7835]]: =schemathesis~=3.39= in
  the dev deps, the =app/tests/rest/contract/= package, in-process per-version
  schema loading, and =APIClient=-driven response validation. The harness
  validates =/api/v2/tags/= against the schema and already surfaces a real gap
  (an undocumented =400= on the =tag_group= filter).
- Steps 5–7 (priority-endpoint coverage, annotation fixes, CI stage) are the
  remaining work.

* Deliverables

- A green, clearly labeled CI contract-test stage with prioritized endpoint
  coverage.
- An updated, accurate drf-spectacular OpenAPI schema.
- Documentation for running and triaging Schemathesis locally and in CI.

* CI Integration

The contract suite runs through the standard Django test runner as its own
job, so failures are attributable to a labeled "Contract Tests" stage rather
than buried in the general test run.

#+begin_src yaml
# django-tests.yml (excerpt)
jobs:
  contract-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.10'
      - run: uv sync --extra dev
      - run: uv run python manage.py test app.tests.rest.contract --keepdb
#+end_src

* Technical Approach Details

** Target endpoints (initial coverage)

- Authentication: login, refresh, logout.
- Core domain read paths with the highest client use (top endpoints by
  traffic / error budget).

** Repository changes

- New package: =app/tests/rest/contract/= (schema loader, base test case,
  concrete per-endpoint tests).
- Dependency: =schemathesis~=3.39= in =pyproject.toml= =[dev]=.
- A distinct "Contract Tests" CI job.

* Success Metrics

- ≥ 90% of the top 20 endpoints covered by Schemathesis.
- Zero schema/implementation mismatches escaping to staging once coverage is
  in place.
- The published schema is treated as authoritative by client and tooling
  teams.

* Risks and Mitigations

- *Schema gaps balloon the effort.* Mitigation: triage daily; prioritize
  high-traffic endpoints; fix annotations incrementally rather than blocking on
  full coverage.
- *Property-based flakiness / nondeterminism.* Mitigation: deterministic
  seeded test data; bounded example counts; stable provider state per test.
- *Dependency friction (Schemathesis 4.x vs. our pins).* Mitigation: pinned to
  the =3.x= line, which has no =tenacity= dependency and accepts our
  =jsonschema~=4.19.2=.

* Follow-on: consumer contracts and contract versioning

Enforcing that clients and the provider agree on payloads — and evolving those
payloads safely — is the consumer-driven side of contract testing. That work
is owned by the Web RFC, not this document:

[[https://hungryroot.atlassian.net/wiki/spaces/WD/pages/1584168961][RFC: Client-Safe Sodapop UI Attachments and Suggested Action Contracts]]

In particular, the policy for evolving contracts once they have shipped — never
mutating a shipped contract in place, versioning breaking changes, and gating
deprecation on client usage — is defined in that RFC's
[[https://hungryroot.atlassian.net/wiki/spaces/WD/pages/1584168961/RFC+Client-Safe+Sodapop+UI+Attachments+and+Suggested+Action+Contracts#Maintaining-Contract-Versions]["Maintaining Contract Versions"]] section. The RFC also proposes Pact as the
mechanism for making the client/provider capability matrix evidence-backed
rather than a manual assumption.

The accurate schema this Schemathesis effort produces is the foundation that
consumer-contract work builds on.

* Tickets

Phase 1 (Schemathesis) work, tracked under the =contract-testing= label
([[https://hungryroot.atlassian.net/issues?jql=labels%20%3D%20%22contract-testing%22][view in Jira]]):

| Key                                              | Summary                                                                    | Plan step | Status      |
|--------------------------------------------------+----------------------------------------------------------------------------+-----------+-------------|
| [[https://hungryroot.atlassian.net/browse/BE-7835][BE-7835]] | Scaffold Schemathesis OpenAPI contract-testing harness                     | Steps 1–4 | In Progress |
| [[https://hungryroot.atlassian.net/browse/BE-7836][BE-7836]] | Cover priority endpoints in contract suite with Factory Boy auth fixtures  | Step 5    | Refined     |
| [[https://hungryroot.atlassian.net/browse/BE-7837][BE-7837]] | Fix OpenAPI schema annotation gaps revealed by Schemathesis                | Step 6    | Refined     |
| [[https://hungryroot.atlassian.net/browse/BE-7838][BE-7838]] | Add contract tests as a distinct CI step in django-tests.yml               | Step 7    | Refined     |
| [[https://hungryroot.atlassian.net/browse/BE-7839][BE-7839]] | Document running and triaging Schemathesis locally and in CI               | Closeout  | Refined     |