Org Web Adapter

Editing hungryroot/incidents/INT-1120-adhoc-shuffles-deploy.org

:PROPERTIES:
:ID:       1cac5f72-0f97-402e-a0f3-99ad5eabe528
:END:
#+title: INT-1120 — P1 Production Deploy Error: Adhoc Shuffles
#+filetags: :project:

* Executive Summary

On 2026-05-12, a routine PROD deploy failed because an empty-string value for the
=MEMORYDB_CLUSTER_ENDPOINT_PORT= env var (set via Terraform) caused
=settings/components/common.py= to raise =ValueError= during boot, preventing
Celery workers from starting. Adhoc shuffles and other background tasks did
not process for ~2h 50m. A premature/incorrect fix was deployed at 3:50p before
the correct fix landed at 4:48p; queues recovered by 5:11p. The on-call incident
Slack workflow also failed (separate bug, since fixed). Confirmed CX impact: at
least 3 cancellations.

- *Severity:* P1 (Highest)
- *Status:* In Review (assignee: Colin Powell; reporter: Stephen Johnston)
- *Jira:* [[https://hungryroot.atlassian.net/browse/INT-1120][INT-1120]]
- *Total outage:* 2h 50m of Celery processing
- *Root cause:* Terraform-set empty string for =MEMORYDB_CLUSTER_ENDPOINT_PORT=
  cast to =int()= in =common.py= → boot failure
- *Resolution:* Default-on-empty handling deployed at 4:48p; queue purged via
  shell during war room

* Timeline (EDT, 2026-05-12)

| Time  | Event                                                                                  |
|-------+----------------------------------------------------------------------------------------|
| 2:21p | Initial PROD deploy started                                                            |
| 2:52p | Initial deploy errored on last few ECS tasks                                           |
| 3:00p | Celery urgent failures begin                                                           |
| 3:38p | Root cause found: env var setting in =common.py=                                       |
| 3:50p | First *incorrect* fix deployed                                                         |
| 4:15p | Proper fix identified                                                                  |
| 4:28p | Incorrect fix re-deploy attempt; failed                                                |
| 4:30p | War room started; incident channel *not* auto-created (on-call workflow bug)           |
| 4:48p | Proper fix deployed (PRs [[https://github.com/hungryroot/hungryroot/pull/13994][#13994]], [[https://github.com/hungryroot/hungryroot/pull/13998][#13998]])                                       |
| 5:10p | Celery urgent recovering; shuffle queue still backlogged → purged via shell            |
| 5:11p | Queues recovered; incident closed                                                      |

* Impact

- ECS task rollout blocked; Celery worker fleet would not boot.
- Background processing lag/outage for ~2h 50m (adhoc shuffles, smartgrocery
  queue, other tasks).
- Confirmed business impact: at least 3 customer cancellations attributed by CX.
- Secondary problem: adhoc shuffle queue couldn't drain even after fix because
  retry cadence (30s) was far shorter than expiration (12h) — feedback loop kept
  re-queuing faster than workers could process. Resolved by manually purging the
  =smartgrocery= queue.

* Root Cause

Terraform difference in =MEMORYDB_CLUSTER_ENDPOINT_PORT= setting caused an empty
string to be passed in; =django-environ= then tried =int('')= and crashed
settings load:

#+BEGIN_SRC text
  File "hrplatform/settings/components/common.py", line 459, in <module>
    MEMORYDB_CLUSTER_ENDPOINT_PORT = env("MEMORYDB_CLUSTER_ENDPOINT_PORT", default=6379)
  ...
ValueError: invalid literal for int() with base 10: ''
#+END_SRC

Open question (not caught in prior envs): why this didn't surface until PROD.

** Contributing Factors

- Premature/incorrect fix deployed once, then a failed deploy attempt before the
  good release.
- On-call =Report Production Incident= Slack workflow aborted because adding a
  specific user (BZ) to the auto-created channel failed → rest of flow aborted.
- Adhoc shuffle retry/expiration mismatch amplified backlog after fix landed.

* Resolution

- 4:48p: Correct env var default-on-empty deployed (PRs
  [[https://github.com/hungryroot/hungryroot/pull/13994][#13994]], [[https://github.com/hungryroot/hungryroot/pull/13998][#13998]]).
- 5:10p: Cleared =smartgrocery= queue via shell to break the requeue feedback
  loop:

#+begin_src python
    from django.conf import settings
    from kombu import Connection
    with Connection(settings.CELERY_BROKER_URL) as conn:
        count = conn.default_channel.queue_purge('smartgrocery')
        print(f'Purged {count} messages from smartgrocery')

#+end_src

- 5:11p: Celery queues recovered; incident operationally closed.

* Action Items

** DONE

- [ ] *Root cause identified* and documented in ticket.
- [ ] *Proper fix deployed* — env var default-on-empty handling
  ([[https://github.com/hungryroot/hungryroot/pull/13994][PR #13994]], [[https://github.com/hungryroot/hungryroot/pull/13998][PR #13998]]).
- [ ] *Celery deploy health checks added* during incident (Jag).
- [ ] *Smartgrocery queue purged* to break requeue feedback loop.
- [ ] *On-call Slack workflow fixed* (Ellen) — was failing because BZ couldn't
  be auto-added to the incident channel, which aborted the rest of the flow.
- [ ] *Follow-up PRs merged* (Jag): [[https://github.com/hungryroot/hungryroot/pull/14017][#14017]], [[https://github.com/hungryroot/hungryroot/pull/14019][#14019]].

** TODO

- [ ] *Postmortem document* — exact env var, why =common.py= allowed bad state,
  guardrails (tests, config validation, staged rollout).
- [ ] *Celery queue-depth alert* — no task failures fired during this incident,
  just queue growth. Need an alert when Celery queues back up. (Colin)
- [ ] *Task-in-queue-time alert* — likely an anomaly check, not a static
  monitor. (Colin)
- [ ] *Move existing test alerts to production* — Andrew has DD monitor
  [[https://app.datadoghq.com/monitors/278415339][278415339]] in =datadog-test=; decide whether to move to on-call alerts or
  build new ones.
- [ ] *Prevent rollout under Terraform drift* — investigate stopping deploys
  when Terraform vars differ across tasks/services; "TODO PREVENTION" flagged
  in ticket.
- [ ] *Deploy safety review* — rollback criteria and clearer
  abort/rollback playbooks for partial ECS rollouts after a bad config deploy.
- [ ] *[[https://hungryroot.atlassian.net/browse/ISA-2829][ISA-2829]]* — Live ad-hoc shuffles should have expiration matching requeue
  cadence. Currently 12h expiration + 30s requeue → cascading overload.
  Proposals: (a) shorten =CELERY_TASK_EXPIRATION_SECONDS= or tie 30s cache
  timeout to a DC; (b) match retry time and timeout, make both configurable
  from our side. /Status: Ready for Testing — Jag./

* References

- Jira: [[https://hungryroot.atlassian.net/browse/INT-1120][INT-1120]], linked: [[https://hungryroot.atlassian.net/browse/ISA-2829][ISA-2829]]
- PRs: [[https://github.com/hungryroot/hungryroot/pull/13994][#13994]], [[https://github.com/hungryroot/hungryroot/pull/13998][#13998]], [[https://github.com/hungryroot/hungryroot/pull/14017][#14017]], [[https://github.com/hungryroot/hungryroot/pull/14019][#14019]]
- Initial deploy run: [[https://github.com/hungryroot/hungryroot/actions/runs/25753774980/job/75639605612][actions/runs/25753774980]]
- Incorrect fix run: [[https://github.com/hungryroot/hungryroot/actions/runs/25758847709][actions/runs/25758847709]]
- Final fix run: [[https://github.com/hungryroot/hungryroot/actions/runs/25761689208][actions/runs/25761689208]]
- DD monitor (test): [[https://app.datadoghq.com/monitors/278415339][278415339]]