Org Web Adapter

Editing hungryroot/jira/be_7217_add_structured_logging_to_healthcheck_endpoint_to_diagnose_500_errors.org

:PROPERTIES:
:ID:       05214436-0345-46f5-a022-3ced41afe04e
:END:
#+title: BE-7217 Add structured logging to /healthcheck/ endpoint to diagnose 500 errors

- source :: https://hungryroot.atlassian.net/browse/BE-7217
- tags   :: [[id:3a477c02-e26d-40f7-a719-0aa1cd169dd8][Hungryroot API Project]]
- epic   :: [[id:86327fb4-0f66-4bc6-9068-c414cea77042][EP-411 Eng Parking Lot]]
- sprint :: [[id:211bb652-f918-4774-9611-ec6c3a77c72b][HR BE Sprint 105]]
- status :: [[id:ec92a960-cac1-4b05-b4f5-25a2935eea75][Status - To Do]]

* Description
[[https://app.datadoghq.com/logs?query=service%3Adjango%20container_id%3Aff9f9bd2ba8b4d4ba47c7903636f0606-3079480798&agg_m=count&agg_m_source=base&agg_t=count&clustering_pattern_field_path=message&cols=%40event%2C%40route&context_event=AZx85_WCAADT-66rR1arrQAG&event=AwAAAZx85_Jfddpd1QAAABhBWng4NV9XQ0FBRFQtNjZyUjFhcnJRQUcAAAAkMDE5YzdjZjAtMDA0OS00YjM5LWE5YTMtZDhmY2QwMTEzMDMwAAAs9A&fromUser=true&messageDisplay=inline&refresh_mode=sliding&saved-view-id=1931212&storage=hot&stream_sort=time%2Cdesc&to_event=AwAAAZx85_5h6Sk1kgAAABhBWng4NkFFNkFBRFoxR0ozaFpMRDBnQUYAAAAkMDE5YzdjZjAtMDA0OS00YjM5LWE5YTMtZDhmY2QwMTEzMDMwAAFwRQ&viz=&from_ts=1771621801599&to_ts=1771622104674&live=false][Datadog]]

The /healthcheck/ endpoint (route health_check_home) occasionally returns 500
errors, but there is no logging to indicate which specific health check backend
failed. This makes it difficult to diagnose and triage the root cause.

Current Setup

The endpoint is served by django-health-check==3.17.0 and is configured with the
following checks:

Database (health_check.db)

Cache (health_check.cache)

Migrations (health_check.contrib.migrations)

Celery ping (health_check.contrib.celery_ping)

S3 (app.backends.S3HealthCheckBackend) — custom backend

Celery queue length (app.backends.CeleryQueueLengthBackend) — custom backend

The health check URL is defined in hrplatform/urls/api.py:

path("healthcheck/", include("health_check.urls"))

The custom backends are registered in app/apps.py::_register_health_checks().

Proposed Work

Add structured logging around the health check view so that when a check fails,
we log which backend(s) failed, the error message, and how long each check took.

Options to investigate:

Subclass or wrap the django-health-check view (health_check.views.MainView) to
add logging before/after the check runs.

Add a Django middleware scoped to the /healthcheck/ path that logs the response
status and any error details.

Add try/except logging in each custom backend's check_status() method (the
CeleryQueueLengthBackend already does this; S3HealthCheckBackend does not).

Ensure the S3 backend logs errors — currently it silently re-raises
HealthCheckException without logging the underlying exception.

Relevant Files

hrplatform/urls/api.py — URL registration

app/backends.py — Custom health check backends (S3HealthCheckBackend,
CeleryQueueLengthBackend)

app/apps.py — Backend registration in _register_health_checks()

hrplatform/settings/components/common.py — INSTALLED_APPS with health_check apps

Acceptance Criteria

When /healthcheck/ returns a 500, logs clearly indicate which backend(s) failed
and why.

Existing health check behavior is unchanged (same response format, same checks).

Logging uses structlog consistent with the rest of the codebase.