hungryroot/incidents/INT-1120-adhoc-shuffles-deploy.org

ID: 1cac5f72-0f97-402e-a0f3-99ad5eabe528

INT-1120 — P1 Production Deploy Error: Adhoc Shuffles

Executive Summary

On 2026-05-12, a routine PROD deploy failed because an empty-string value for the

=MEMORYDB_CLUSTER_ENDPOINT_PORT= env var (set via Terraform) caused

=settings/components/common.py= to raise =ValueError= during boot, preventing

Celery workers from starting. Adhoc shuffles and other background tasks did

not process for ~2h 50m. A premature/incorrect fix was deployed at 3:50p before

the correct fix landed at 4:48p; queues recovered by 5:11p. The on-call incident

Slack workflow also failed (separate bug, since fixed). Confirmed CX impact: at

least 3 cancellations.

- *Severity:* P1 (Highest)

- *Status:* In Review (assignee: Colin Powell; reporter: Stephen Johnston)

- *Jira:* [[https://hungryroot.atlassian.net/browse/INT-1120][INT-1120]]

- *Total outage:* 2h 50m of Celery processing

- *Root cause:* Terraform-set empty string for =MEMORYDB_CLUSTER_ENDPOINT_PORT=

cast to =int()= in =common.py= → boot failure

- *Resolution:* Default-on-empty handling deployed at 4:48p; queue purged via

shell during war room

Timeline (EDT, 2026-05-12)

| Time | Event |

|-------+----------------------------------------------------------------------------------------|

| 2:21p | Initial PROD deploy started |

| 2:52p | Initial deploy errored on last few ECS tasks |

| 3:00p | Celery urgent failures begin |

| 3:38p | Root cause found: env var setting in =common.py= |

| 3:50p | First *incorrect* fix deployed |

| 4:15p | Proper fix identified |

| 4:28p | Incorrect fix re-deploy attempt; failed |

| 4:30p | War room started; incident channel *not* auto-created (on-call workflow bug) |

| 4:48p | Proper fix deployed (PRs [[https://github.com/hungryroot/hungryroot/pull/13994][#13994]], [[https://github.com/hungryroot/hungryroot/pull/13998][#13998]]) |

| 5:10p | Celery urgent recovering; shuffle queue still backlogged → purged via shell |

| 5:11p | Queues recovered; incident closed |

Impact

- ECS task rollout blocked; Celery worker fleet would not boot.

- Background processing lag/outage for ~2h 50m (adhoc shuffles, smartgrocery

queue, other tasks).

- Confirmed business impact: at least 3 customer cancellations attributed by CX.

- Secondary problem: adhoc shuffle queue couldn't drain even after fix because

retry cadence (30s) was far shorter than expiration (12h) — feedback loop kept

re-queuing faster than workers could process. Resolved by manually purging the

=smartgrocery= queue.

Root Cause

Terraform difference in =MEMORYDB_CLUSTER_ENDPOINT_PORT= setting caused an empty

string to be passed in; =django-environ= then tried =int('')= and crashed

settings load:

#+BEGIN_SRC text

File "hrplatform/settings/components/common.py", line 459, in <module>

MEMORYDB_CLUSTER_ENDPOINT_PORT = env("MEMORYDB_CLUSTER_ENDPOINT_PORT", default=6379)

...

ValueError: invalid literal for int() with base 10: ''

#+END_SRC

Open question (not caught in prior envs): why this didn't surface until PROD.

Contributing Factors

- Premature/incorrect fix deployed once, then a failed deploy attempt before the

good release.

- On-call =Report Production Incident= Slack workflow aborted because adding a

specific user (BZ) to the auto-created channel failed → rest of flow aborted.

- Adhoc shuffle retry/expiration mismatch amplified backlog after fix landed.

Resolution

- 4:48p: Correct env var default-on-empty deployed (PRs

[[https://github.com/hungryroot/hungryroot/pull/13994][#13994]], [[https://github.com/hungryroot/hungryroot/pull/13998][#13998]]).

- 5:10p: Cleared =smartgrocery= queue via shell to break the requeue feedback

loop:

#+begin_src python

from django.conf import settings

from kombu import Connection

with Connection(settings.CELERY_BROKER_URL) as conn:

count = conn.default_channel.queue_purge('smartgrocery')

print(f'Purged {count} messages from smartgrocery')

#+end_src

- 5:11p: Celery queues recovered; incident operationally closed.

Action Items

DONE

- [ ] *Root cause identified* and documented in ticket.

- [ ] *Proper fix deployed* — env var default-on-empty handling

([[https://github.com/hungryroot/hungryroot/pull/13994][PR #13994]], [[https://github.com/hungryroot/hungryroot/pull/13998][PR #13998]]).

- [ ] *Celery deploy health checks added* during incident (Jag).

- [ ] *Smartgrocery queue purged* to break requeue feedback loop.

- [ ] *On-call Slack workflow fixed* (Ellen) — was failing because BZ couldn't

be auto-added to the incident channel, which aborted the rest of the flow.

- [ ] *Follow-up PRs merged* (Jag): [[https://github.com/hungryroot/hungryroot/pull/14017][#14017]], [[https://github.com/hungryroot/hungryroot/pull/14019][#14019]].

TODO

- [ ] *Postmortem document* — exact env var, why =common.py= allowed bad state,

guardrails (tests, config validation, staged rollout).

- [ ] *Celery queue-depth alert* — no task failures fired during this incident,

just queue growth. Need an alert when Celery queues back up. (Colin)

- [ ] *Task-in-queue-time alert* — likely an anomaly check, not a static

monitor. (Colin)

- [ ] *Move existing test alerts to production* — Andrew has DD monitor

[[https://app.datadoghq.com/monitors/278415339][278415339]] in =datadog-test=; decide whether to move to on-call alerts or

build new ones.

- [ ] *Prevent rollout under Terraform drift* — investigate stopping deploys

when Terraform vars differ across tasks/services; "TODO PREVENTION" flagged

in ticket.

- [ ] *Deploy safety review* — rollback criteria and clearer

abort/rollback playbooks for partial ECS rollouts after a bad config deploy.

- [ ] *[[https://hungryroot.atlassian.net/browse/ISA-2829][ISA-2829]]* — Live ad-hoc shuffles should have expiration matching requeue

cadence. Currently 12h expiration + 30s requeue → cascading overload.

Proposals: (a) shorten =CELERY_TASK_EXPIRATION_SECONDS= or tie 30s cache

timeout to a DC; (b) match retry time and timeout, make both configurable

from our side. /Status: Ready for Testing — Jag./

References

- Jira: [[https://hungryroot.atlassian.net/browse/INT-1120][INT-1120]], linked: [[https://hungryroot.atlassian.net/browse/ISA-2829][ISA-2829]]

- PRs: [[https://github.com/hungryroot/hungryroot/pull/13994][#13994]], [[https://github.com/hungryroot/hungryroot/pull/13998][#13998]], [[https://github.com/hungryroot/hungryroot/pull/14017][#14017]], [[https://github.com/hungryroot/hungryroot/pull/14019][#14019]]

- Initial deploy run: [[https://github.com/hungryroot/hungryroot/actions/runs/25753774980/job/75639605612][actions/runs/25753774980]]

- Incorrect fix run: [[https://github.com/hungryroot/hungryroot/actions/runs/25758847709][actions/runs/25758847709]]

- Final fix run: [[https://github.com/hungryroot/hungryroot/actions/runs/25761689208][actions/runs/25761689208]]

- DD monitor (test): [[https://app.datadoghq.com/monitors/278415339][278415339]]